Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning

Weizhe Hong, Ann Kennedy, Xavier P. Burgos-Artizzu, Moriel Zelikowsky, Santiago G. Navonne, Pietro Perona, David J. Anderson*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

195 Scopus citations


A lack of automated, quantitative, and accurate assessment of social behaviors in mammalian animalmodels has limited progress toward understanding mechanisms underlying social interactions and their disorders such as autism. Here we present a new integrated hardware and software system that combines video tracking, depth sensing, and machine learning for automatic detection and quantification of social behaviors involving close and dynamic interactions between two mice of different coat colors in their home cage. We designed a hardware setup that integrates traditional video cameras with a depth camera, developed computer vision tools to extract the body "pose" of individual animals in a social context, and used a supervised learning algorithm to classify several well-described social behaviors. We validated the robustness of the automated classifiers in various experimental settings and used them to examine how genetic background, such as that of Black and Tan Brachyury (BTBR) mice (a previously reported autism model), influences social behavior. Our integrated approach allows for rapid, automated measurement of social behaviors across diverse experimental designs and also affords the ability to develop new, objective behavioral metrics.

Original languageEnglish (US)
Pages (from-to)E5351-E5360
JournalProceedings of the National Academy of Sciences of the United States of America
Issue number38
StatePublished - Sep 22 2015
Externally publishedYes


  • Behavioral tracking
  • Depth sensing
  • Machine vision
  • Social behavior
  • Supervised machine learning

ASJC Scopus subject areas

  • General


Dive into the research topics of 'Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning'. Together they form a unique fingerprint.

Cite this