The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice

Cristina Segalin; Jalani Williams; Tomomi Karigo; May Hui; Moriel Zelikowsky; Jennifer J. Sun; Pietro Perona; David J. Anderson; Ann Kennedy

doi:10.7554/eLife.63720

The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice

Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J. Sun, Pietro Perona, David J. Anderson, Ann Kennedy^*

^*Corresponding author for this work

Neuroscience

Research output: Contribution to journal › Article › peer-review

102 Scopus citations

Abstract

The study of naturalistic social behavior requires quantification of animals’ interactions. This is generally done through manual annotation—a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging. We introduce the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. We compare MARS’s annotations to human annotations and find that MARS’s pose estimation and behavior classification achieve human-level performance. We also release the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, we introduce the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.

Original language	English (US)
Article number	e63720
Journal	eLife
Volume	10
DOIs	https://doi.org/10.7554/eLife.63720
State	Published - Nov 2021

Funding

National Institute of Mental HealthK99MH117264, Ann Kennedy, National Institute of Mental HealthK99MH108734, Moriel Zelikowsky, Simons Foundation Autism Research Initiative401141. We are grateful to Grant Van Horn for providing the original TensorFlow implementation of the MSC-MultiBox detection library, Matteo Ronchi for his pose error diagnosis code, and Mark Zylka for providing the Cul3 and Chd8 mouse lines. Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Number R01MH123612 and 5R01MH070053 (DJA), K99MH108734 (MZ) and K99MH117264 (AK), and by the Human Frontier Science Program (TK), the Helen Hay Whitney Foundation (AK), the Simons Foundation Autism Research Initiative (DJA), the Gordon and Betty Moore Foundation (PP), and a gift from Liying Huang and Charles Trimble (to PP). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

ASJC Scopus subject areas

General Neuroscience
General Biochemistry, Genetics and Molecular Biology
General Immunology and Microbiology

Access to Document

10.7554/eLife.63720

Cite this

@article{e692bd4887ef4f4680f1ac619197b49f,

title = "The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice",

abstract = "The study of naturalistic social behavior requires quantification of animals{\textquoteright} interactions. This is generally done through manual annotation—a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging. We introduce the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. We compare MARS{\textquoteright}s annotations to human annotations and find that MARS{\textquoteright}s pose estimation and behavior classification achieve human-level performance. We also release the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, we introduce the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.",

author = "Cristina Segalin and Jalani Williams and Tomomi Karigo and May Hui and Moriel Zelikowsky and Sun, {Jennifer J.} and Pietro Perona and Anderson, {David J.} and Ann Kennedy",

note = "Publisher Copyright: {\textcopyright} Segalin et al.",

year = "2021",

month = nov,

doi = "10.7554/eLife.63720",

language = "English (US)",

volume = "10",

journal = "eLife",

issn = "2050-084X",

publisher = "eLife Sciences Publications Ltd",

}

TY - JOUR

T1 - The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice

AU - Segalin, Cristina

AU - Williams, Jalani

AU - Karigo, Tomomi

AU - Hui, May

AU - Zelikowsky, Moriel

AU - Sun, Jennifer J.

AU - Perona, Pietro

AU - Anderson, David J.

AU - Kennedy, Ann

PY - 2021/11

Y1 - 2021/11

N2 - The study of naturalistic social behavior requires quantification of animals’ interactions. This is generally done through manual annotation—a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging. We introduce the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. We compare MARS’s annotations to human annotations and find that MARS’s pose estimation and behavior classification achieve human-level performance. We also release the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, we introduce the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.

AB - The study of naturalistic social behavior requires quantification of animals’ interactions. This is generally done through manual annotation—a highly time-consuming and tedious process. Recent advances in computer vision enable tracking the pose (posture) of freely behaving animals. However, automatically and accurately classifying complex social behaviors remains technically challenging. We introduce the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely interacting mice. We compare MARS’s annotations to human annotations and find that MARS’s pose estimation and behavior classification achieve human-level performance. We also release the pose and annotation datasets used to train MARS to serve as community benchmarks and resources. Finally, we introduce the Behavior Ensemble and Neural Trajectory Observatory (BENTO), a graphical user interface for analysis of multimodal neuroscience datasets. Together, MARS and BENTO provide an end-to-end pipeline for behavior data extraction and analysis in a package that is user-friendly and easily modifiable.

UR - http://www.scopus.com/inward/record.url?scp=85120851603&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85120851603&partnerID=8YFLogxK

U2 - 10.7554/eLife.63720

DO - 10.7554/eLife.63720

M3 - Article

C2 - 34846301

AN - SCOPUS:85120851603

SN - 2050-084X

VL - 10

JO - eLife

JF - eLife

M1 - e63720

ER -

The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice

Abstract

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this