PyMouse Lifter: Real Time 3-D Pose Estimation for Mice with Only 2-D Annotation Via Data Synthesis

Abstract

Neural-network-based pose estimation models have become increasingly popular for quantitative analysis of mouse behavior, yet most recordings still use a single 2-D camera view and therefore lack the depth cues needed for accurate 3-D kinematics. Existing open-source 3-D mouse datasets for training deep-learning models cover only a narrow range of environments and do not generalize well to various laboratory settings. To overcome these limitations, we introduce PyMouse Lifter, a pipeline that automatically reconstructs 3-D mouse poses from ordinary 2-D top-view videos with minimal manual 2-D annotation. PyMouse Lifter combines (i) an anatomically realistic 3-D mouse model for automated data synthesis, (ii) a monocular depth estimation model, and (iii) a 2-D key-point estimation model, enabling accurate 3-D reconstruction (model-based 3D inference) in virtually any open-field arena without using depth or multiple camera views for reconstruction. We validate the system on multiple datasets against depth-camera ground truth and show that the lifted 3D trajectories yield improved behavior classification over 2-D data and can be implemented in real time.