DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

Abstract
Data availability
Article and author information
Metrics

Abstract

Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in 3-dimensional (3D) space. Deep neural networks can estimate 2-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.

Data availability

All data generated and analyzed during this study are included in the DeepFly3D GitHub site: https://github.com/NeLy-EPFL/DeepFly3D and in the Harvard Dataverse.

The following data sets were generated

1. Gunel S
2. Rhodin H
3. Morales D
4. Campagnolo J
5. Ramdya P
6. Fua P
(2019) aDN-GAL4 Control
Harvard Dataverse, doi:10.7910/DVN/PKKXOE.

https://doi.org/10.7910/DVN/PKKXOE
1. Gunel S
2. Rhodin H
3. Morales D
4. Campagnolo J
5. Ramdya P
6. Fua P
(2019) MDN-GAL4 Control
Harvard Dataverse, doi:10.7910/DVN/HOLXOR.

https://doi.org/10.7910/DVN/HOLXOR
1. Gunel S
2. Rhodin H
3. Morales D
4. Campagnolo J
5. Ramdya P
6. Fua P
(2019) aDN-GAL4 UAS-CsChrimson
Harvard Dataverse, doi:10.7910/DVN/S4L4KX.

https://doi.org/10.7910/DVN/S4L4KX
1. Gunel S
2. Rhodin H
3. Morales D
4. Campagnolo J
5. Ramdya P
6. Fua P
(2019) MDN-GAL4 UAS-CsChrimson
Harvard Dataverse, doi:10.7910/DVN/8SUC9U.

https://doi.org/10.7910/DVN/8SUC9U

Article and author information

Author details

Semih Günel

School of Computer and Communication Sciences, Computer Vision Laboratory, EPFL, Lausanne, Switzerland

For correspondence
semih.gunel@epfl.ch

Competing interests
The authors declare that no competing interests exist.
Helge Rhodin

School of Computer and Communication Sciences, Computer Vision Laboratory, EPFL, Lausanne, Switzerland

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0003-2692-0801
Daniel Morales

School of Life Sciences, Brain Mind Institute and Interfaculty Institute of Bioengineering, Neuroengineering Laboratory, EPFL, Lausanne, Switzerland

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0002-7469-0898
João H Campagnolo

School of Life Sciences, Brain Mind Institute and Interfaculty Institute of Bioengineering, Neuroengineering Laboratory, EPFL, Lausanne, Switzerland

Competing interests
The authors declare that no competing interests exist.
Pavan Ramdya

School of Life Sciences, Brain Mind Institute and Interfaculty Institute of Bioengineering, Neuroengineering Laboratory, EPFL, Lausanne, Switzerland

For correspondence
pavan.ramdya@epfl.ch

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0001-5425-4610
Pascal Fua

School of Computer and Communication Sciences, Computer Vision Laboratory, EPFL, Lausanne, Switzerland

Competing interests
The authors declare that no competing interests exist.

Funding

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (175667)

Daniel Morales
Pavan Ramdya

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (181239)

Daniel Morales
Pavan Ramdya

EPFL (iPhD)

Semih Günel

Microsoft Research (JRC Project)

Helge Rhodin

Swiss Government Excellence Postdoctoral Scholarship (2018.0483)

Daniel Morales

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Copyright

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.