DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels
- Harvard Medical School, United States
- Boston Children's Hospital, United States
- Massachusetts General Hospital, United States
- Champalimaud Center for the Unknown, Portugal
Abstract
Videos of animal behavior are used to quantify researcher-defined behaviors-of-interest to study neural function, gene mutations, and pharmacological therapies. Behaviors-of-interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors-of-interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram's rapid, automatic, and reproducible labeling of researcher-defined behaviors-of-interest may accelerate and enhance supervised behavior analysis.
Data availability
Code is posted publicly on Github and linked in the paper. Video datasets and human annotations are publicly available and linked in the paper.
Article and author information
Author details
Christopher D Harvey
Funding
National Institutes of Health (R01MH107620)
- Christopher D Harvey
National Science Foundation (GRFP)
- Nivanthika K Wimalasena
Fundacao para a Ciencia ea Tecnologia (PD/BD/105947/2014)
- Tomás Cruz
Harvard Medical School Dean's Innovation Award
- Christopher D Harvey
Harvard Medical School Goldenson Research Award
- Christopher D Harvey
National Institutes of Health (DP1 MH125776)
- Christopher D Harvey
National Institutes of Health (R01NS089521)
- Christopher D Harvey
National Institutes of Health (R01NS108410)
- Christopher D Harvey
National Institutes of Health (F31NS108450)
- James P Bohnslav
National Institutes of Health (R35NS105076)
- Clifford J Woolf
National Institutes of Health (R01AT011447)
- Clifford J Woolf
National Institutes of Health (R00NS101057)
- Lauren L Orefice
National Institutes of Health (K99DE028360)
- David A Yarmolinsky
European Research Council (ERC-Stg-759782)
- M Eugenia Chiappe
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Ethics
Animal experimentation: All experimental procedures were approved by the Institutional Animal Care and Use Committees at Boston Children's Hospital (protocol numbers 17-06-3494R and 19-01-3809R) or Massachusetts General Hospital (protocol number 2018N000219) and were performed in compliance with the Guide for the Care and Use of Laboratory Animals.
Copyright
© 2021, Bohnslav et al.
This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.
Metrics
-
- 13,640
- views
-
- 1,301
- downloads
-
- 110
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels
eLife 10:e63377.
https://doi.org/10.7554/eLife.63377
Further reading
-
- Neuroscience
Non-linear summation of synaptic inputs to the dendrites of pyramidal neurons has been proposed to increase the computation capacity of neurons through coincidence detection, signal amplification, and additional logic operations such as XOR. Supralinear dendritic integration has been documented extensively in principal neurons, mediated by several voltage-dependent conductances. It has also been reported in parvalbumin-positive hippocampal basket cells, in dendrites innervated by feedback excitatory synapses. Whether other interneurons, which support feed-forward or feedback inhibition of principal neuron dendrites, also exhibit local non-linear integration of synaptic excitation is not known. Here, we use patch-clamp electrophysiology, and two-photon calcium imaging and glutamate uncaging, to show that supralinear dendritic integration of near-synchronous spatially clustered glutamate-receptor mediated depolarization occurs in NDNF-positive neurogliaform cells and oriens-lacunosum moleculare interneurons in the mouse hippocampus. Supralinear summation was detected via recordings of somatic depolarizations elicited by uncaging of glutamate on dendritic fragments, and, in neurogliaform cells, by concurrent imaging of dendritic calcium transients. Supralinearity was abolished by blocking NMDA receptors (NMDARs) but resisted blockade of voltage-gated sodium channels. Blocking L-type calcium channels abolished supralinear calcium signalling but only had a minor effect on voltage supralinearity. Dendritic boosting of spatially clustered synaptic signals argues for previously unappreciated computational complexity in dendrite-projecting inhibitory cells of the hippocampus.
-
- Neuroscience
Complex structural and functional changes occurring in typical and atypical development necessitate multidimensional approaches to better understand the risk of developing psychopathology. Here, we simultaneously examined structural and functional brain network patterns in relation to dimensions of psychopathology in the Adolescent Brain Cognitive Development (ABCD) dataset. Several components were identified, recapitulating the psychopathology hierarchy, with the general psychopathology (p) factor explaining most covariance with multimodal imaging features, while the internalizing, externalizing, and neurodevelopmental dimensions were each associated with distinct morphological and functional connectivity signatures. Connectivity signatures associated with the p factor and neurodevelopmental dimensions followed the sensory-to-transmodal axis of cortical organization, which is related to the emergence of complex cognition and risk for psychopathology. Results were consistent in two separate data subsamples and robust to variations in analytical parameters. Although model parameters yielded statistically significant brain–behavior associations in unseen data, generalizability of the model was rather limited for all three latent components (r change from within- to out-of-sample statistics: LC1within = 0.36, LC1out = 0.03; LC2within = 0.34, LC2out = 0.05; LC3within = 0.35, LC3out = 0.07). Our findings help in better understanding biological mechanisms underpinning dimensions of psychopathology, and could provide brain-based vulnerability markers.
