Real-time, low-latency closed-loop feedback using markerless posture tracking

  1. Gary A Kane
  2. Gonçalo Lopes
  3. Jonny L Saunders
  4. Alexander Mathis
  5. Mackenzie W Mathis  Is a corresponding author
  1. Harvard University, United States
  2. NeuroGears, United Kingdom
  3. University of Oregon, United States
  4. EPFL, Switzerland

Abstract

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here we provide a new DeepLabCut-Live! package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called DLC-Live! GUI, and integration into (2) Bonsai and (3) AutoPilot. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.

Data availability

All models, data, test scripts and software is already released and made freely available on GitHub: https://github.com/DeepLabCut/DeepLabCut-live

Article and author information

Author details

  1. Gary A Kane

    The Rowland Institute at Harvard, Harvard University, Cambridge, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-7703-5055
  2. Gonçalo Lopes

    NeuroGears, London, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-0731-4945
  3. Jonny L Saunders

    Institute of Neuroscience, Department of Psychology, University of Oregon, Eugene, United States
    Competing interests
    The authors declare that no competing interests exist.
  4. Alexander Mathis

    Life Sciences, EPFL, Geneva, Switzerland
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-3777-2202
  5. Mackenzie W Mathis

    Brain Mind Institute, EPFL, Genève, Switzerland
    For correspondence
    mackenzie.mathis@epfl.ch
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-7368-4456

Funding

Chan Zuckerberg Initiative (EOSS)

  • Alexander Mathis
  • Mackenzie W Mathis

National Science Foundation (1309047)

  • Jonny L Sanders

The Rowland Institute at Harvard, Harvard University

  • Gary A Kane
  • Alexander Mathis
  • Mackenzie W Mathis

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Animal experimentation: All mouse work was carried out under the permission of the IACUC at Harvard University (#17-07-309). Dog videos and feedback was exempt from IACUC approval (with conformation with IACUC).

Reviewing Editor

  1. Gordon J Berman, Emory University, United States

Publication history

  1. Received: August 8, 2020
  2. Accepted: December 6, 2020
  3. Accepted Manuscript published: December 8, 2020 (version 1)
  4. Accepted Manuscript updated: December 9, 2020 (version 2)
  5. Version of Record published: January 4, 2021 (version 3)

Copyright

© 2020, Kane et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 9,365
    Page views
  • 1,079
    Downloads
  • 35
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, Scopus, PubMed Central.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Gary A Kane
  2. Gonçalo Lopes
  3. Jonny L Saunders
  4. Alexander Mathis
  5. Mackenzie W Mathis
(2020)
Real-time, low-latency closed-loop feedback using markerless posture tracking
eLife 9:e61909.
https://doi.org/10.7554/eLife.61909

Further reading

    1. Computational and Systems Biology
    2. Physics of Living Systems
    Spencer Farrell, Alice E Kane ... Andrew D Rutenberg
    Research Article Updated

    As an organism ages, its health-state is determined by a balance between the processes of damage and repair. Measuring these processes requires longitudinal data. We extract damage and repair transition rates from repeated observations of binary health attributes in mice and humans to explore robustness and resilience, which respectively represent resisting or recovering from damage. We assess differences in robustness and resilience using changes in damage rates and repair rates of binary health attributes. We find a conserved decline with age in robustness and resilience in mice and humans, implying that both contribute to worsening aging health – as assessed by the frailty index (FI). A decline in robustness, however, has a greater effect than a decline in resilience on the accelerated increase of the FI with age, and a greater association with reduced survival. We also find that deficits are damaged and repaired over a wide range of timescales ranging from the shortest measurement scales toward organismal lifetime timescales. We explore the effect of systemic interventions that have been shown to improve health, including the angiotensin-converting enzyme inhibitor enalapril and voluntary exercise for mice. We have also explored the correlations with household wealth for humans. We find that these interventions and factors affect both damage and repair rates, and hence robustness and resilience, in age and sex-dependent manners.

    1. Computational and Systems Biology
    Jeffrey Molendijk, Ronnie Blazev ... Benjamin L Parker
    Research Article

    Improving muscle function has great potential to improve the quality of life. To identify novel regulators of skeletal muscle metabolism and function, we performed a proteomic analysis of gastrocnemius muscle from 73 genetically distinct inbred mouse strains, and integrated the data with previously acquired genomics and >300 molecular/phenotypic traits via quantitative trait loci mapping and correlation network analysis. These data identified thousands of associations between protein abundance and phenotypes and can be accessed online (https://muscle.coffeeprot.com/) to identify regulators of muscle function. We used this resource to prioritize targets for a functional genomic screen in human bioengineered skeletal muscle. This identified several negative regulators of muscle function including UFC1, an E2 ligase for protein UFMylation. We show UFMylation is up-regulated in a mouse model of amyotrophic lateral sclerosis, a disease that involves muscle atrophy. Furthermore, in vivo knockdown of UFMylation increased contraction force, implicating its role as a negative regulator of skeletal muscle function.