Fong, T. L. | Search bioRxiv

bioRxiv2025

Towards Video-LLM Driven Workflow for Behavioral Segmentation and Scoring in Mice Performing a Skilled Water Reaching Task: An Evaluation of Recent LLM Models

SignificanceBehavior scoring is labor-intensive and subjective, introducing variability in results. Large Language Models (LLMs) capable of video understanding offer a transformative solution to manual scoring, crucial for accelerating and standardizing neuroscience workflows. AimWe sought to benchmark state-of-the-art video LLMs (Gemini 2.5 Pro, Qwen3-VL, and VideoLLaMA3) for automated behavioural segmentation and scoring of mice performing a water-reaching task. ApproachVideos of mice performing water reaching from the front view were analysed by the LLMs. Accuracy was compared across different models and against prompt adjustments within Gemini. To assess classification determinants, video fidelity was altered through pixel interpolation and key regions blurred (paws/snout-mouth). In addition, the models were asked to describe the mouses actions over time. ResultsGemini 2.5 Pro (0.74 {+/-} 0.12 accuracy) and Qwen3-VL-30B (0.67 {+/-} 0.13) exhibited ability to classify trial outcomes. Reliable classification required a minimum pixel resolution of 0.28 mm per pixel. Accuracy is significantly reduced upon obscuring the snout-mouth area. In 549/1058 of videos, Gemini 2.5 Pro also provided completely accurate frame-to-frame behaviour segmentations. ConclusionsVideo-LLMs offer potential to accelerate neuroscience by providing scalable, objective quantification of goal-directed behaviors. By producing temporal annotations, Gemini enables fast first-pass labelling that markedly streamlines manual dataset curation.

neuroscience↗

bioRxiv2025

PyMouse Lifter: Real Time 3-D Pose Estimation for Mice with Only 2-D Annotation Via Data Synthesis

Neural-network-based pose estimation models have become increasingly popular for quantitative analysis of mouse behavior, yet most recordings still use a single 2-D camera view and therefore lack the depth cues needed for accurate 3-D kinematics. Existing open-source 3-D mouse datasets for training deep-learning models cover only a narrow range of environments and do not generalize well to various laboratory settings. To overcome these limitations, we introduce PyMouse Lifter, a pipeline that automatically reconstructs 3-D mouse poses from ordinary 2-D top-view videos with minimal manual 2-D annotation. PyMouse Lifter combines (i) an anatomically realistic 3-D mouse model for automated data synthesis, (ii) a monocular depth estimation model, and (iii) a 2-D key-point estimation model, enabling accurate 3-D reconstruction (model-based 3D inference) in virtually any open-field arena without using depth or multiple camera views for reconstruction. We validate the system on multiple datasets against depth-camera ground truth and show that the lifted 3D trajectories yield improved behavior classification over 2-D data and can be implemented in real time.

neuroscience↗