Abstract
We introduce MABe22, a large-scale, multi-agent video and trajectory benchmark to assess the quality of learned behavior representations. This dataset is collected from a variety of biology experiments, and includes triplets of interacting mice (4.7 million frames video+pose tracking data, 10 million frames pose only), symbiotic beetle-ant interactions (10 million frames video data), and groups of interacting flies (4.4 million frames of pose tracking data). Accompanying these data, we introduce a panel of real-life downstream analysis tasks to assess the quality of learned representations by evaluating how well they preserve information about the experimental conditions (e.g. strain, time of day, optogenetic stimulation) and animal behavior. We test multiple state-of-the-art self-supervised video and trajectory representation learning methods to demonstrate the use of our benchmark, revealing that methods developed using human action datasets do not fully translate to animal datasets. We hope that our benchmark and dataset encourage a broader exploration of behavior representation learning methods across species and settings.
Original language | English (US) |
---|---|
Pages (from-to) | 32936-32990 |
Number of pages | 55 |
Journal | Proceedings of Machine Learning Research |
Volume | 202 |
State | Published - 2023 |
Event | 40th International Conference on Machine Learning, ICML 2023 - Honolulu, United States Duration: Jul 23 2023 → Jul 29 2023 |
Funding
Acquisition of behavioral data was supported by Army Research Office MURI award W911NF1910269 (JP) and a US National Science Foundation CAREER award (2047472) (JP). Acquisition of behavioral data was supported by NIH grants DA041668 (NIDA), DA048034 (NIDA), and Simons Foundation SFARI Director’s Award (to VK). Curation of data task design was funded by NIMH award #R00MH117264 (to AK) and NSERC Award #PGSD3-532647-2019 (to JJS). Acquisition of behavioral data was funded by the Howard Hughes Medical Institute. This work was generously supported by the Simons Collaboration on the Global Brain grant 543025 (to PP), NIH Award #R00MH117264 (to AK), NSF Award #1918839 (to YY), NIH 1R34NS118470-01 (to JP), NSERC Award #PGSD3-532647-2019 (to JJS), as well as a gift from Charles and Lily Trimble (to PP). We would like to thank Tom Sproule for mouse breeding and dataset collection. The mouse dataset was supported by the National Institute of Health DA041668 (NIDA), DA048634 (NIDA, and Simons Foundation SFARI Director's Award) (to VK). We also greatly appreciate Google, Amazon, HHMI, and the Simons Foundation for sponsoring the MABe22 Challenge & Workshop. This work was generously supported by the Simons Collaboration on the Global Brain grant 543025 (to PP), NIH Award #R00MH117264 (to AK), NSF Award #1918839 (to YY), NIH 1R34NS118470-01 (to JP), NSERC Award #PGSD3-532647-2019 (to JJS), as well as a gift from Charles and Lily Trimble (to PP). We would like to thank Tom Sproule for mouse breeding and dataset collection. The mouse dataset was supported by the National Institute of Health DA041668 (NIDA), DA048634 (NIDA, and Simons Foundation SFARI Director’s Award) (to VK). We also greatly appreciate Google, Amazon, HHMI, and the Simons Foundation for sponsoring the MABe22 Challenge & Workshop.
ASJC Scopus subject areas
- Artificial Intelligence
- Software
- Control and Systems Engineering
- Statistics and Probability