1. Neuroscience
Download icon

DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

  1. Semih Günel  Is a corresponding author
  2. Helge Rhodin
  3. Daniel Morales
  4. João H Campagnolo
  5. Pavan Ramdya  Is a corresponding author
  6. Pascal Fua
  1. EPFL, Switzerland
Tools and Resources
  • Cited 15
  • Views 4,659
  • Annotations
Cite this article as: eLife 2019;8:e48571 doi: 10.7554/eLife.48571

Abstract

Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in 3-dimensional (3D) space. Deep neural networks can estimate 2-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.

Data availability

All data generated and analyzed during this study are included in the DeepFly3D GitHub site: https://github.com/NeLy-EPFL/DeepFly3D and in the Harvard Dataverse.

The following data sets were generated

Article and author information

Author details

  1. Semih Günel

    School of Computer and Communication Sciences, Computer Vision Laboratory, EPFL, Lausanne, Switzerland
    For correspondence
    semih.gunel@epfl.ch
    Competing interests
    The authors declare that no competing interests exist.

  2. Helge Rhodin

    School of Computer and Communication Sciences, Computer Vision Laboratory, EPFL, Lausanne, Switzerland
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2692-0801

  3. Daniel Morales

    School of Life Sciences, Brain Mind Institute and Interfaculty Institute of Bioengineering, Neuroengineering Laboratory, EPFL, Lausanne, Switzerland
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-7469-0898

  4. João H Campagnolo

    School of Life Sciences, Brain Mind Institute and Interfaculty Institute of Bioengineering, Neuroengineering Laboratory, EPFL, Lausanne, Switzerland
    Competing interests
    The authors declare that no competing interests exist.

  5. Pavan Ramdya

    School of Life Sciences, Brain Mind Institute and Interfaculty Institute of Bioengineering, Neuroengineering Laboratory, EPFL, Lausanne, Switzerland
    For correspondence
    pavan.ramdya@epfl.ch
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-5425-4610

  6. Pascal Fua

    School of Computer and Communication Sciences, Computer Vision Laboratory, EPFL, Lausanne, Switzerland
    Competing interests
    The authors declare that no competing interests exist.

Funding

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (175667)

  • Daniel Morales
  • Pavan Ramdya

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (181239)

  • Daniel Morales
  • Pavan Ramdya

EPFL (iPhD)

  • Semih Günel

Microsoft Research (JRC Project)

  • Helge Rhodin

Swiss Government Excellence Postdoctoral Scholarship (2018.0483)

  • Daniel Morales

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Reviewing Editor

  1. Timothy O'Leary, University of Cambridge, United Kingdom

Publication history

  1. Received: May 18, 2019
  2. Accepted: September 28, 2019
  3. Accepted Manuscript published: October 4, 2019 (version 1)
  4. Version of Record published: November 4, 2019 (version 2)

Copyright

© 2019, Günel et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 4,659
    Page views
  • 538
    Downloads
  • 15
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, Scopus, PubMed Central.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)

Categories and tags

Research organism

Further reading

    1. Neuroscience

    GABAB receptor auxiliary subunits modulate Cav2.3-mediated release from medial habenula terminals

    Pradeep Bhandari et al.
    Research Article Updated

    The synaptic connection from medial habenula (MHb) to interpeduncular nucleus (IPN) is critical for emotion-related behaviors and uniquely expresses R-type Ca2+ channels (Cav2.3) and auxiliary GABAB receptor (GBR) subunits, the K+-channel tetramerization domain-containing proteins (KCTDs). Activation of GBRs facilitates or inhibits transmitter release from MHb terminals depending on the IPN subnucleus, but the role of KCTDs is unknown. We therefore examined the localization and function of Cav2.3, GBRs, and KCTDs in this pathway in mice. We show in heterologous cells that KCTD8 and KCTD12b directly bind to Cav2.3 and that KCTD8 potentiates Cav2.3 currents in the absence of GBRs. In the rostral IPN, KCTD8, KCTD12b, and Cav2.3 co-localize at the presynaptic active zone. Genetic deletion indicated a bidirectional modulation of Cav2.3-mediated release by these KCTDs with a compensatory increase of KCTD8 in the active zone in KCTD12b-deficient mice. The interaction of Cav2.3 with KCTDs therefore scales synaptic strength independent of GBR activation.

    1. Neuroscience

    Response-based outcome predictions and confidence regulate feedback processing and learning

    Romy Frömer et al.
    Research Article Updated

    Influential theories emphasize the importance of predictions in learning: we learn from feedback to the extent that it is surprising, and thus conveys new information. Here, we explore the hypothesis that surprise depends not only on comparing current events to past experience, but also on online evaluation of performance via internal monitoring. Specifically, we propose that people leverage insights from response-based performance monitoring – outcome predictions and confidence – to control learning from feedback. In line with predictions from a Bayesian inference model, we find that people who are better at calibrating their confidence to the precision of their outcome predictions learn more quickly. Further in line with our proposal, EEG signatures of feedback processing are sensitive to the accuracy of, and confidence in, post-response outcome predictions. Taken together, our results suggest that online predictions and confidence serve to calibrate neural error signals to improve the efficiency of learning.

    1. Neuroscience

    Distinct protocerebral neuropils associated with attractive and aversive female-produced odorants in the male moth brain

    Jonas Hansen Kymre et al.
    Research Article

    The pheromone system of heliothine moths is an optimal model for studying principles underlying higher-order olfactory processing. In Helicoverpa armigera, three male-specific glomeruli receive input about three female-produced signals, the primary pheromone component, serving as an attractant, and two minor constituents, serving a dual function, i.e. attraction versus inhibition of attraction. From the antennal-lobe glomeruli, the information is conveyed to higher olfactory centers, including the lateral protocerebrum, via three main paths – of which the medial tract is the most prominent. In this study, we traced physiologically identified medial-tract projection neurons from each of the three male‑specific glomeruli with the aim of mapping their terminal branches in the lateral protocerebrum. Our data suggest that the neurons’ wide-spread projections are organized according to behavioral significance, including a spatial separation of signals representing attraction versus inhibition – however, with a unique capacity of switching behavioral consequence based on the amount of the minor components.