Marker Data Enhancement For Markerless Motion Capture

Abstract

ObjectiveHuman pose estimation models can measure movement from videos at a large scale and low cost; however, open-source pose estimation models typically detect only sparse keypoints, which leads to inaccurate joint kinematics. OpenCap, a freely available service for researchers to measure movement from videos, addresses this issue using a deep learning model-- the marker enhancer--that transforms sparse keypoints into dense anatomical markers. However, OpenCap performs poorly on movements not included in the training data. Here, we create a much larger and more diverse training dataset and develop a more accurate and generalizable marker enhancer. MethodsWe compiled marker-based motion capture data from 1176 subjects and synthesized 1433 hours of keypoints and anatomical markers to train the marker enhancer. We evaluated its accuracy in computing kinematics using both benchmark movement videos and synthetic data representing unseen, diverse movements. ResultsThe marker enhancer improved kinematic accuracy on benchmark movements (mean error: 4.1{degrees}, max: 8.7{degrees}) compared to using video keypoints (mean: 9.6{degrees}, max: 43.1{degrees}) and OpenCaps original enhancer (mean: 5.3{degrees}, max: 11.5{degrees}). It also better generalized to unseen, diverse movements (mean: 4.1{degrees}, max: 6.7{degrees}) than OpenCaps original enhancer (mean: 40.4{degrees}, max: 252.0{degrees}). ConclusionOur marker enhancer demonstrates both accuracy and generalizability across diverse movements. SignificanceWe integrated the marker enhancer into OpenCap, thereby offering its thousands of users more accurate measurements across a broader range of movements.