FERAL: A Video-Understanding System for Direct Video-to-Behavior Mapping

Abstract

Quantifying animal behavior often requires segmenting continuous actions into discrete, interpretable states, yet most automated pipelines infer actions from keypoint dynamics and are limited by keypoint tracking quality. Here we present FERAL (Feature Extraction for Recognition of Animal Locomotion), a supervised video-understanding toolkit that maps raw video directly to frame-level behavioral labels, bypassing the keypoint-extraction and pose-classification stages of conventional pipelines. Across benchmarks, FERAL matches or exceeds state-of-the-art pose- and video-based baselines. On the CalMS21 mouse social-interaction benchmark, it exceeds Googles VideoPrism using only a quarter of the training data. FERAL generalizes across species (apes, zebras, mice, ants, flies, and nematodes), recording conditions, and levels of organization, from single animals to social interactions and colony-scale collective behavior. Released as a user-friendly, open-source package, it integrates with existing analysis pipelines. By enabling scalable, species-agnostic behavioral quantification directly from raw video, FERAL broadens the experimental paradigms available for studying animal behavior.