SingleBehavior Lab: behavioral sequencing and phenotyping with lightweight task specific adaptation

Abstract

Understanding behavioral differences between experimental groups and quantifying action structure in behavioral experiments remains challenging. Currently, most approaches rely on pose estimation followed by downstream classification, resulting in assay specific pipelines with substantial annotation requirements. Here we present SingleBehavior Lab (SBL), a framework for modeling behavior across experimental contexts using a standardized graphical interface. SBL leverages spatiotemporal embeddings from large video foundation models and combines them with lightweight contrastive adapters, a multi-head attention pooling (MAP) module and a temporal decoder to enable behavior sequencing and task-specific refinement. The framework supports few-shot learning, allowing small models trained on pretrained embeddings to improve action segmentation and classification with limited labeled data, without fine-tuning the underlying video model. In parallel, a large segmentation model with motion-aware memory is used to extract object-centered representations that, together with shared spatiotemporal embeddings, enable unsupervised clustering of behavioral states and analysis of their structure, including cluster prioritization, transition dynamics and attention-based interpretability. Across multiple assays and species, SBL supports identification of group-level differences and rare behaviors, and provides a basis for integrating behavioral representations across experimental contexts.