ABEL: an active-learning behavior estimation and labeling platform

Abstract

Detailed behavior analysis is essential for thorough characterization of ethologically relevant behaviors in model organisms, yet manual annotation of the full behavioral repertoire remains subjective, time intensive, and susceptible to observer error. Advances in machine learning have enabled high-throughput pose estimation on recorded video, but tools for behavior classification from pose and video data are still developing. Instead of hand-scoring every frame of video, experimenters can instead label a small subset of video frames and software trained through machine learning makes predictions on the rest. Here, we present an Active-learning Behavior Estimation and Labeling (ABEL) platform that uses clip-level active learning (i.e., human labeling of short video snippets) with multimodal features (pose, video, context/ROI) to train robust behavior classifiers. We rigorously validated ABEL-derived behavior predictions against expert human observers and field-standard automated software, across diverse rodent behavioral assays. Across eight assays and 45 behaviors, model training required 19.5 hours of human annotation in total, with the reviewer scoring ~8% of available video. Models trained in ABEL achieved a mean precision-recall area under the curve (PR-AUC - a 0-1 score of how well a model balances missed detections against false alarms, with 1 being perfect) of 0.90 (SD 0.09, range 0.60-0.99), with no association between performance and behavior prevalence (r = 0.20). This was aided by custom tools, Essence Extractor and UMAP Interactive Selection, for targeted discovery of high probability clips which reduce the clip review needed to find a rare behavior 6-fold relative to random sampling and 10-fold relative to labeling whole videos. As a biological validation, we assessed how ABEL-derived behaviors relate to underlying neuronal calcium dynamics. Behavior labels were tightly synced with neuronal signatures distinct from ambiguous behavior and randomly chosen, behavior-unrelated time windows (shuffle control). Together, these data indicate that ABEL provides an efficient platform for frame-precise classification of distinct ethologically relevant behaviors.