Biomechanical Arm and Hand Tracking with Multiview Markerless Motion Capture

Human arm and hand function is extremely complex with many degrees of freedom. It is also a common target for clinical interventions. However, precisely measuring upper extremity movement in both clinical and research settings is logistically challenging. We overcame this challenge through a novel approach to reconstructing arm biomechanics from markerless motion capture from multiple synchronized videos. Our approach directly opti-mizes the kinematics of an accurate biomechanical arm and hand that allows end-to-end minimization of the errors between the reconstructed movements and keypoints detected by computer vision. Key to this is an implicit function that maps from time to joint kinematics, which provides a learnable trajectory representation that can be differentiated through the biomechanical model, and supports GPU acceleration using MuJoCo-MJX. This optimization solves for the inverse kinematic solution consistent with the measured keypoints, consistent with biomechanical constraints, in addition to scaling the model while solving for the kinematics. We compare different hand keypoint detectors and find the best produces a fit with only several millimeters of reconstruction error. We also find that end-to-end optimization outperforms a two-stage fitting procedure, equivalent to more traditional biomechanical pipelines, where we first compute 3D marker trajectories and then perform inverse kinematics fitting in OpenSim. We anticipate this framework will reduce the barriers to biomechanical analysis of the arm and hand in both clinical and research settings.