1. Neuroscience

DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

  1. Jacob M Graving  Is a corresponding author
  2. Daniel Chae
  3. Hemal Naik
  4. Liang Li
  5. Benjamin Koger
  6. Blair R Costelloe
  7. Iain D Couzin
  1. Max Planck Institute of Animal Behavior, Germany
  2. Princeton University, United States
https://doi.org/10.7554/eLife.47994
Share this article
Cite this article
  1. Jacob M Graving
  2. Daniel Chae
  3. Hemal Naik
  4. Liang Li
  5. Benjamin Koger
  6. Blair R Costelloe
  7. Iain D Couzin
(2019)
DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
eLife 8:e47994.
https://doi.org/10.7554/eLife.47994
  2. Download BibTeX
  3. Download .RIS

Abstract

Quantitative behavioral measurements are important for answering questions across scientific disciplines-from neuroscience to ecology. State-of-the-art deep-learning methods offer major advances in data quality and detail by allowing researchers to automatically estimate locations of an animal's body parts directly from images or videos. However, currently-available animal pose estimation methods have limitations in speed and robustness. Here we introduce a new easy-to-use software toolkit, DeepPoseKit, that addresses these problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. These advances improve processing speed >2× with no loss in accuracy compared to currently-available methods. We demonstrate the versatility of our methods with multiple challenging animal pose estimation tasks in laboratory and field settings-including groups of interacting individuals. Our work reduces barriers to using advanced tools for measuring behavior and has broad applicability across the behavioral sciences.

Data availability

Data used and generated for experiments and model comparisons are included in the supporting files. Posture datasets can be found at: https://github.com/jgraving/deepposekit-dataThe code for DeepPoseKit is publicly available at the URL we provided in the paper: https://github.com/jgraving/deepposekit/The reviewers should follow the provided instructions for installation in the README file https://github.com/jgraving/deepposekit/blob/master/README.md#installation. Example Jupyter notebooks for how to use the code are provided here: https://github.com/jgraving/deepposekit/tree/master/examples

The following data sets were generated
The following previously published data sets were used

Article and author information

Author details

  1. Jacob M Graving

    Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
    For correspondence
    jgraving@gmail.com
    Competing interests
    No competing interests declared.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-5826-467X

  2. Daniel Chae

    Department of Computer Science, Princeton University, Princeton, United States
    Competing interests
    No competing interests declared.

  3. Hemal Naik

    Department for Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
    Competing interests
    No competing interests declared.

  4. Liang Li

    Department for Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
    Competing interests
    No competing interests declared.

  5. Benjamin Koger

    Department for Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
    Competing interests
    No competing interests declared.

  6. Blair R Costelloe

    Department for Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
    Competing interests
    No competing interests declared.

  7. Iain D Couzin

    Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
    Competing interests
    Iain D Couzin, Reviewing editor, eLife.

Funding

National Science Foundation (IOS-1355061)

  • Iain D Couzin

Horizon 2020 Framework Programme (Marie Sklodowska-Curie grant agreement No. 748549)

  • Blair R Costelloe

Nvidia (GPU Grant)

  • Blair R Costelloe

Office of Naval Research (N00014-09-1-1074)

  • Iain D Couzin

Office of Naval Research (N00014-14-1-0635)

  • Iain D Couzin

Army Research Office (W911NG-11-1-0385)

  • Iain D Couzin

Army Research Office (W911NF14-1-0431)

  • Iain D Couzin

Deutsche Forschungsgemeinschaft (DFG Centre of Excellence 2117)

  • Iain D Couzin

University of Konstanz (Zukunftskolleg Investment Grant)

  • Blair R Costelloe

The Strukture-und Innovations fonds fur die Forschung of the State of Baden-Wurttemberg

  • Iain D Couzin

Max Planck Society

  • Iain D Couzin

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Animal experimentation: All procedures for collecting the zebra (E. grevyi) dataset were reviewed and approved by Ethikrat, the independent Ethics Council of the Max Planck Society. The zebra dataset was collected with the permission of Kenya's National Commission for Science, Technology and Innovation (NACOSTI/P/17/59088/15489 and NACOSTI/P/18/59088/21567) using drones operated by B.R.C. with the permission of the Kenya Civil Aviation Authority (authorization numbers: KCAA/OPS/2117/4 Vol. 2 (80), KCAA/OPS/2117/4 Vol. 2 (81), KCAA/OPS/2117/5 (86) and KCAA/OPS/2117/5 (87); RPAS Operator Certificate numbers: RPA/TP/0005 AND RPA/TP/000-0009).

Copyright

© 2019, Graving et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 24,885
    views
  • 2,418
    downloads
  • 398
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Jacob M Graving
  2. Daniel Chae
  3. Hemal Naik
  4. Liang Li
  5. Benjamin Koger
  6. Blair R Costelloe
  7. Iain D Couzin
(2019)
DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
eLife 8:e47994.
https://doi.org/10.7554/eLife.47994

Share this article

https://doi.org/10.7554/eLife.47994

Categories and tags

Further reading

    1. Neuroscience

    Specific presynaptic functions require distinct Drosophila Cav2 splice isoforms

    Christopher Bell, Lukas Kilo ... Stefanie Ryglewski
    Research Article

    At many vertebrate synapses, presynaptic functions are tuned by expression of different Cav2 channels. Most invertebrate genomes contain only one Cav2 gene. The Drosophila Cav2 homolog, cacophony (cac), induces synaptic vesicle release at presynaptic active zones (AZs). We hypothesize that Drosophila cac functional diversity is enhanced by two mutually exclusive exon pairs that are not conserved in vertebrates, one in the voltage sensor and one in the loop binding Caβ and Gβγ subunits. We find that alternative splicing in the voltage sensor affects channel activation voltage. Only the isoform with the higher activation voltage localizes to AZs at the glutamatergic Drosophila larval neuromuscular junction and is imperative for normal synapse function. By contrast, alternative splicing at the other alternative exon pair tunes multiple aspects of presynaptic function. While expression of one exon yields normal transmission, expression of the other reduces channel number in the AZ and thus release probability. This also abolishes presynaptic homeostatic plasticity. Moreover, reduced channel number affects short-term plasticity, which is rescued by increasing the external calcium concentration to match release probability to control. In sum, in Drosophila alternative splicing provides a mechanism to regulate different aspects of presynaptic functions with only one Cav2 gene.

    1. Neuroscience

    Sleep need driven oscillation of glutamate synaptic phenotype

    Kaspar E Vogt, Ashwinikumar Kulkarni ... Robert W Greene
    Research Article

    Sleep loss increases AMPA-synaptic strength and number in the neocortex. However, this is only part of the synaptic sleep loss response. We report an increased AMPA/NMDA EPSC ratio in frontal-cortical pyramidal neurons of layers 2–3. Silent synapses are absent, decreasing the plastic potential to convert silent NMDA to active AMPA synapses. These sleep loss changes are recovered by sleep. Sleep genes are enriched for synaptic shaping cellular components controlling glutamate synapse phenotype, overlap with autism risk genes, and are primarily observed in excitatory pyramidal neurons projecting intra-telencephalically. These genes are enriched with genes controlled by the transcription factor, MEF2c, and its repressor, HDAC4. Sleep genes can thus provide a framework within which motor learning and training occur mediated by the sleep-dependent oscillation of glutamate-synaptic phenotypes.

    1. Neuroscience
    2. Structural Biology and Molecular Biophysics

    CryoEM structures of Kv1.2 potassium channels, conducting and non-conducting

    Yangyu Wu, Yangyang Yan ... Fred J Sigworth
    Research Article

    We present near-atomic-resolution cryoEM structures of the mammalian voltage-gated potassium channel Kv1.2 in open, C-type inactivated, toxin-blocked and sodium-bound states at 3.2 Å, 2.5 Å, 3.2 Å, and 2.9 Å. These structures, all obtained at nominally zero membrane potential in detergent micelles, reveal distinct ion-occupancy patterns in the selectivity filter. The first two structures are very similar to those reported in the related Shaker channel and the much-studied Kv1.2–2.1 chimeric channel. On the other hand, two new structures show unexpected patterns of ion occupancy. First, the toxin α-Dendrotoxin, like Charybdotoxin, is seen to attach to the negatively-charged channel outer mouth, and a lysine residue penetrates into the selectivity filter, with the terminal amine coordinated by carbonyls, partially disrupting the outermost ion-binding site. In the remainder of the filter two densities of bound ions are observed, rather than three as observed with other toxin-blocked Kv channels. Second, a structure of Kv1.2 in Na+ solution does not show collapse or destabilization of the selectivity filter, but instead shows an intact selectivity filter with ion density in each binding site. We also attempted to image the C-type inactivated Kv1.2 W366F channel in Na+ solution, but the protein conformation was seen to be highly variable and only a low-resolution structure could be obtained. These findings present new insights into the stability of the selectivity filter and the mechanism of toxin block of this intensively studied, voltage-gated potassium channel.