DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

The precise quantification of movements is critical for understanding how neurons, biomechanics, and the environment influence and give rise to animal behaviors. For organisms with skeletons and exoskeletons, these measurements are naturally made with reference to 3D joint and appendage locations. Paired with modern approaches to simultaneously record the activity of neural populations in tethered, behaving animals (Dombeck et al., 2007; Seelig et al., 2010; Chen et al., 2018), 3D joint and appendage tracking promises to accelerate the discovery of neural control principles, particularly in the genetically tractable and numerically simple nervous system of the fly, Drosophila melanogaster.

However, algorithms for reliably estimating 3D pose in such small Drosophila-sized animals have not yet been developed. Instead, multiple alternative approaches have been taken. For example, one can affix and use small markers—reflective, colored, or fluorescent particles—to identify and reconstruct keypoints from video data (Bender et al., 2010; Kain et al., 1910; Todd et al., 2017). Although this approach works well on humans (Moeslund and Granum, 2000), in smaller, Drosophila-sized animals markers likely hamper movements and are difficult to mount on sub-millimeter scale limbs. Most importantly, measurements of one or even two markers for each leg (Todd et al., 2017) cannot fully describe 3D limb kinematics. Another strategy has been to use computer vision techniques that operate without markers. However, these measurements have been restricted to 2D pose in freely behaving flies. Before the advent of deep learning, this was accomplished by matching the contours of animals seen against uniform backgrounds (Isakov et al., 2016), measuring limb tip positions using complex TIRF-based imaging approaches (Mendes et al., 2013), or measuring limb segments using active contours (Uhlmann et al., 2017). In addition to being limited to 2D rather than 3D pose, these methods are complex, time-consuming, and error-prone in the face of long data sequences, cluttered backgrounds, fast motion, and occlusions that naturally occur when animals are observed from a single 2D perspective.

As a result, in recent years the computer vision community has largely forsaken these techniques in favor of deep learning-based methods. Consequently, the efficacy of monocular 3D human pose estimation algorithms has greatly improved. This is especially true when capturing human movements for which there is enough annotated data to train deep networks effectively. Walking and upright poses are prime examples of this, and state-of-the-art algorithms (Pavlakos et al., 2017a; Tome et al., 2017; Popa et al., 2017; Moreno-noguer, 2017; Martinez et al., 2017; Mehta et al., 2017; Rogez et al., 2017; Pavlakos et al., 2017b; Zhou et al., 2017; Tekin et al., 2017; Sun et al., 2017) now deliver impressive real-time results in uncontrolled environments. Increased robustness to occlusions can be obtained by using multi-camera setups (Elhayek et al., 2015; Rhodin et al., 2016; Simon et al., 2017; Pavlakos et al., 2017b) and triangulating the 2D detections. This improves accuracy while making it possible to eliminate false detections.

These advances in 2D pose estimation have also recently been used to measure behavior in laboratory animals. For example, DeepLabCut provides a user-friendly interface to DeeperCut, a state-of-the-art human pose estimation network (Mathis et al., 2018), and LEAP (Pereira et al., 2019) can successfully track limb and appendage landmarks using a shallower network. Still, 2D pose provides an incomplete representation of animal behavior: important information can be lost due to occlusions, and movement quantification is heavily influenced by perspective.

Approaches used to translate human 2D to 3D pose have also been applied to larger animals, like lab mice and cheetahs (Nath et al., 2019), but require the use of calibration boards. These techniques cannot be easily transferred for the study of small animals like Drosophila: adult flies are approximately 2.5 mm long and precisely registering multiple camera viewpoints using traditional approaches would require the fabrication of a prohibitively small checkerboard pattern, along with the tedious labor of using a small, external calibration pattern. Moreover, flies have many appendages and joints, are translucent, and in most laboratory experiments are only illuminated using infrared light (to avoid visual stimulation)—precluding the use of color information.

To overcome these challenges, we introduce DeepFly3D, a deep learning-based software pipeline that achieves comprehensive, rapid, and reliable 3D pose estimation in tethered, behaving adult Drosophila (Figure 1, Figure 1—video 1). DeepFly3D is applied to synchronized videos acquired from multiple cameras. It first uses a state-of-the-art deep network (Newell et al., 2016) and then enforces consistency across views. This makes it possible to eliminate spurious detections, achieve high 3D accuracy, and use 3D pose errors to further fine-tune the deep network to achieve even better accuracy. To register the cameras, DeepFly3D uses a novel calibration mechanism in which the fly itself is the calibration target. During the calibration process, we also employ sparse bundle adjustment methods, as previously used for human pose estimation (Takahashi et al., 2018; Triggs et al., 2000; Puwein et al., 2014). Thus, the user does not need to manufacture a prohibitively small calibration pattern, or repeat cumbersome calibration protocols. We explain how users can modify the codebase to extend DeepFly3D for 3D pose estimation in other animals (see Materials and methods). Finally, we demonstrate that unsupervised behavioral embedding of 3D joint angle data is robust against problematic artifacts present in embeddings of 2D pose data. In short, DeepFly3D delivers 3D pose estimates reliably, accurately, and with minimal manual intervention while also providing a critical tool for automated behavioral data analysis.

Figure 1 with 1 supplement see all

Download asset Open asset

We have developed DeepFly3D, a deep learning-based 3D pose estimation system that is optimized for quantifying limb and appendage movements in tethered, behaving Drosophila. By using multiple synchronized cameras and exploiting multiview redundancy, our software delivers robust and accurate pose estimation at the sub-millimeter scale. Ultimately, we may work solely with monocular images by lifting the 2D detections (Pavlakos et al., 2017b) to 3D or by directly regressing to 3D (Tekin et al., 2017) as has been achieved in human pose estimation studies. Our approach relies on supervised deep learning to train a neural network that detects 2D joint locations in individual camera images. Importantly, our network becomes increasingly competent as it runs: By leveraging the redundancy inherent to a multiple-camera setup, we iteratively reproject 3D pose to automatically detect and correct 2D errors, and then use these corrections to further train the network without user intervention.

None of the techniques we have put together—an approach for multiple-camera calibration that uses the animal itself rather than an external apparatus, an iterative approach to inferring 3D pose using graphical models as well as optimization based on dynamic programming and belief propagation, and a graphical user interface and active learning policy for interacting with, annotating, and correcting 3D pose data—are fly-specific. They could easily be adapted to other limbed animals, from mice to primates and humans. The only thing that would have to change significantly are the dimensions of the experimental setup. This would remove the need to deal with the very small scales Drosophila requires and would, in practice, make pose estimation easier. In the Materials and methods section, we explain in detail how organism-specific features of DeepFly3D—bone segment length, number of legs, and camera focal distance—can be modified to study, for example, humans, primates, rodents, or other insects.

As in the past, we anticipate that the development of new technologies for quantifying behavior will open new avenues and enhance existing lines of investigation. For example, deriving 3D pose using DeepFly3D can improve the resolution of studies examining how neuronal stimulation influences animal behavior (Cande et al., 2018; McKellar et al., 2019), the precision and predictive power of efforts to define natural action sequences (Seeds et al., 2014; McKellar et al., 2019), the assessment of interventions that target models of human disease (Feany and Bender, 2000; Hewitt and Whitworth, 2017), and links between neural activity and animal behavior—when coupled with recording technologies like 2-photon microscopy (Seelig et al., 2010; Chen et al., 2018). Importantly, 3D pose improves the robustness of unsupervised behavioral classification approaches. Therefore, DeepFly3D is a critical step toward the ultimate goal of achieving fully-automated, high-fidelity behavioral data analysis.

With synchronized Drosophila video sequences from seven cameras in hand, the first task for DeepFly3D is to detect the 2D location of 38 landmarks. These 2D locations of the same landmarks seen across multiple views are then triangulated to generate 3D pose estimates. This pipeline is depicted in Figure 6. First, we will describe our deep learning-based approach to detect landmarks in images. Then, we will explain the triangulation process that yields full 3D trajectories. Finally, we will describe how we identify and correct erroneous 2D detections automatically.

Figure 6

Download asset Open asset

2D pose estimation

Deep network architecture

Request a detailed protocol

We aim to detect five joints on each limb, six on the abdomen, and one on each antenna, giving a total of 38 keypoints per time instance. To achieve this, we adapted a state-of-the-art Stacked Hourglass human pose estimation network (Newell et al., 2016) by changing the input and output layers to accommodate a new input image resolution and a different number of tracked points. A single hourglass stack consists of residual bottleneck modules with max pooling, followed by up-sampling layers and skip connections. The first hourglass network begins with a convolutional layer and a pooling layer to reduce the input image size from 256 × 512 to 64 × 128 pixels. The remaining hourglass input and output tensors are 64 × 128. We used 8 stacks of hourglasses in our final implementation. The output of the network is a stack of probability maps, also known as heatmaps or confidence maps. Each probability map encodes the location of one keypoint, as the belief of the network that a given pixel contains that particular tracked point. However, probability maps do not formally define a probability distribution; their sum over all pixels does not equal 1.

2D pose training dataset

Request a detailed protocol

We trained our network for 19 keypoints, resulting in the tracking of 38 points when both sides of the fly are taken into account. Determining which images to use for training purposes is critical. The intuitively simple approach—training with randomly selected images—may lead to only marginal improvements in overall network performance. This is because images for which network predictions can already be correctly made give rise to only small gradients during training. On the other hand, manually identifying images that may lead to incorrect network predictions is highly laborious. Therefore, to identify such challenging images, we exploited the redundancy of having multiple camera views (see section 3D pose correction). Outliers in individual camera images were corrected automatically using images from other cameras, and frames that still exhibited large reprojection errors on multiple camera views were selected for manual annotation and network retraining. This combination of self supervision and active learning permits faster training using a smaller manually annotated dataset (Simon et al., 2017). The full annotation and iterative training pipeline is illustrated in Figure 6. In total, 40,063 images were annotated: 5,063 were labeled manually in the first iteration, 29,000 by automatic correction, and 6,000 by manually correcting those proposed by the active learning strategy.

Deep network training procedure

Request a detailed protocol

We trained our Stacked Hourglass network to regress from 256 × 512 pixel grayscale video images to multiple 64 × 128 probability maps. Specifically, during training and testing, networks output a 19 × 64 × 128 tensor; one 64 × 128 probability map per tracked point. During training, we created probability maps by embedding a 2D Gaussian with mean at the ground-truth point and 1px symmetrical extent (i.e. $\sigma =1px$) on the diagonal of the covariance matrix. We calculated the loss as the $L_2$ distance between the ground-truth and predicted probability maps. During testing, the final network prediction for a given point was the probability map pixel with maximum probability. We started with a learning rate of 0.0001 and then multiplied the learning rate by a factor of 0.1 once the loss function plateaued for more than five epochs. We used an RMSPROP optimizer for gradient descent, following the original Stacked Hourglass implementation, with a batch-size of eight images. Using 37,000 training images, the Stacked Hourglass network usually converges to a local minimum after 100 epochs (20 h on a single GPU).

Network training details

Request a detailed protocol

Variations in each fly’s position across experiments are handled by the translational invariance of the convolution operation. In addition, we artificially augment training images to improve network generalization for further image variables. These variables include (i) illumination conditions – we randomly changed the brightness of images using a gamma transformation, (ii) scale – we randomly rescaled images between 0.80x - 1.20x, and (iii) rotation – we randomly rotated images and corresponding probability maps ±15°. This augmentation was enough to compensate for real differences in the size and orientation of tethered flies across experiments. Furthermore, as per general practice, the mean channel intensity was subtracted from each input image to distribute annotations symmetrically around zero. We began network training using pretrained weights from the MPII human pose dataset (Andriluka et al., 2014). This dataset consists of more than 25,000 images with 40,000 annotations, possibly with multiple ground-truth human pose labels per image. Starting with a pretrained network results in faster convergence. However, in our experience, this does not affect final network accuracy in cases with a large amount of training data. We split the dataset into 37,000 training images, 2,063 testing images, and 1,000 validation images. None of these subsets shared common images or common animals, to ensure that the network could generalize across animals, and experimental setups. 5,063 of our training images were manually annotated, and the remaining data were automatically collected using belief propagation, graphical models, and active learning, (see section 3D pose correction). Deep neural network parameters need to be trained on a dataset with manually annotated ground-truth key point positions. To initialize the network, we collected annotations using a custom multicamera annotation tool that we implemented in JavaScript using Google Firebase (Figure 7). The DeepFly3D annotation tool operates on a simple web-server, easing the distribution of annotations across users and making these annotations much easier to inspect and control.

Figure 7

Download asset Open asset

Computing hardware and software

Request a detailed protocol

We trained our model on a desktop computing workstation running on an Intel Core i9-7900X CPU, 32 GB of DDR4 RAM, and a GeForce GTX 1080. With 37,000 manually and automatically labeled images, training takes nearly 20 h on a single GeForce GTX 1080 GPU. Our code is implemented with Python 3.6, Pytorch 0.4 and CUDA 9.2. Using this desktop configuration, our network can run at 100 Frames-Per-Second (FPS) using the 8-stack variant of the Hourglass network, and can run at 420 FPS using the smaller 2-stack version. Thanks to an effective initialization step, calibration takes 3–4 s. Error checking and error correction can be performed at 100 FPS and 10 FPS, respectively. Error correction is only performed in response to large reprojection errors and does not create a bottleneck in the overall speed of the pipeline.

Accuracy analysis

Request a detailed protocol

Consistent with the human pose estimation literature, we report accuracy as Percentage of Correct Keypoints (PCK) and Root Mean Squared Error (RMSE). PCK refers to the percentage of detected points lying within a specific radius from the ground-truth label. We set this threshold as 50 pixels, which is roughly one third of the 3D length of the femur. The final estimated position of each keypoint was obtained by selecting the pixel with the largest probability value on the relevant probability map. We compared DeepFly3D’s annotations with manually annotated ground-truth labels to test our model’s accuracy. For RMSE, we report the square root of average pixel distance between the prediction and the ground-truth location of the tracked point. We remove trivial points such as the body-coxa and coxa-femur—which remain relatively stationary—to fairly evaluate our algorithms and to prevent these points from dominating our accuracy measurements.

From 2D landmarks to 3D trajectories

In the previous section, we described our approach to detect 38 2D landmarks. Let ${\mathbf{𝐱}}_{c,j}\in {ℝ}^2$ denote the 2D position of landmark $j$ in the image acquired by camera $c$. For each landmark, our task is now to estimate the corresponding 3D position, ${\mathbf{𝐗}}_j\in {ℝ}^3$. To accomplish this, we used triangulation and bundle-adjustment (Hartley and Zisserman, 2000) to compute 3D locations, and we used pictorial structures (Felzenszwalb and Huttenlocher, 2005) to enforce geometric consistency and to eliminate potential errors caused by misdetections. We present these steps below.

Pinhole camera model

Request a detailed protocol

The first step is to model the projection operation that relates a specific ${\mathbf{𝐗}}_j$ to its seven projections in each camera view ${\mathbf{𝐱}}_{c,j}$. To make this easier, we follow standard practice and convert all Cartesian coordinates $\left[\begin{array}{c}\hfill x_c,y_c,z_c\hfill \end{array}\right]$ to homogeneous ones $\left[\begin{array}{c}\hfill x_h,y_h,z_h,s\hfill \end{array}\right]$ such that $x_c=x_h/s$, $y_c=y_h/s$, $z_c=z_h/s$. From now on, we will assume that all points are expressed in homogeneous coordinates and omit the $h$ subscript. Assuming that these coordinates are expressed in a coordinate system whose origin is in the optical center of the camera and whose z-axis is its optical axis, the 2D image projection $\left[\begin{array}{c}\hfill u,v\hfill \end{array}\right]$ of a 3D homogeneous point $\left[\begin{array}{c}\hfill x,y,z,1\hfill \end{array}\right]$ can be written as

(1) ${\displaystyle \begin{array}{l}{\displaystyle u={\displaystyle U/W,}}\\ v{\displaystyle =V/W,}\\ {\displaystyle \left[\begin{array}{c}U\\ V\\ W\end{array}\right]{\displaystyle =\mathbf{K}\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right],\text{ with }\mathbf{K}=\left[\begin{array}{cccc}{f}_x& 0& c_x& 0\\ 0& f_y& c_y& 0\\ 0& 0& 1& 0\end{array}\right],}}\end{array}}$

where the 3 × 4 matrix $\mathbf{𝐊}$ is known as the intrinsic parameters matrix—scaling in the $x$ and $y$ direction and image coordinates of the principal point $c_x$ and $c_y$—that characterizes the camera settings.

In practice, the 3D points are not expressed in a camera fixed coordinate system, especially in our application where we use seven different cameras. Therefore, we use a world coordinate system that is common to all cameras. For each camera, we must therefore convert 3D coordinates expressed in this world coordinate system to camera coordinates. This requires rotating and translating the coordinates to account for the position of the camera’s optical center and its orientation. When using homogeneous coordinates, this is accomplished by multiplying the coordinate vector by a 4 × 4 extrinsic parameters matrix

(2) $\mathbf{𝐌}=\left[\begin{array}{cc}\hfill \mathbf{𝐑}\hfill & \hfill \mathbf{𝐓}\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right],$

where $\mathbf{𝐑}$ is a 3 × 3 rotation matrix and $\mathbf{𝐓}$ a 3 × 1 translation vector. Combining Equation 1 and Equation 2 yields

(3) ${\displaystyle \begin{array}{l}{\displaystyle u={\displaystyle U/W,}}\\ v{\displaystyle =V/W,}\\ {\displaystyle \left[\begin{array}{c}U\\ V\\ W\end{array}\right]{\displaystyle =\mathbf{P}\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right],}}\end{array}}$

where ${\displaystyle \mathbf{\text{P}}=\mathbf{\text{MK}}}$ is a 3 × 4 matrix.

Camera distortion

Request a detailed protocol

The pinhole camera model described above is an idealized one. The projections of real cameras deviate from it. These deviations are referred to as distortions and must be accounted for. The most significant distortion is known as radial distortion because the error grows with the distance from the image center. For the cameras we use, radial distortion can be expressed as

(4) ${\displaystyle \begin{array}{l}{\displaystyle u_{\text{ pinhole }}{\displaystyle =u\left(1+{\mathrm{k}}_1^x{\mathrm{r}}^2+{\mathrm{k}}_2^x{\mathrm{r}}^4\right),}}\\ {\displaystyle v_{\text{ pinhole }}{\displaystyle =v\left(1+{\mathrm{k}}_1^y{\mathrm{r}}^2+{\mathrm{k}}_2^y{\mathrm{r}}^4\right),}}\end{array}}$

where $\left[\begin{array}{c}\hfill u,v\hfill \end{array}\right]$ is the actual projection of a 3D point and $\left[\begin{array}{c}\hfill u_{\text{pinhole}},v_{\text{pinhole}}\hfill \end{array}\right]$ is the one the pinhole model predicts. In other words, the four parameters $\{k_1^x,k_2^x,k_1^y,k_2^y\}$ characterize the distortion. From now on, we will therefore write the full projection as

(5) ${\displaystyle \begin{array}{ll}\mathbf{X}{\displaystyle =}& \pi (\mathbf{x})=f_d(f_p(\mathbf{x})),\\ \mathbf{X}=& [x,y,z],\\ \mathbf{X}=& [u,v],\end{array}}$

where $f_p$ denotes the ideal pinhole projection of Equation 3 and $f_d$ the correction of Equation 4.

Triangulation

Request a detailed protocol

We can associate to each of the seven cameras a projection function ${\pi }_c$ like the one in Equation 5, where $c$ is the camera number. Given a 3D point and its projections ${\mathbf{𝐱}}_c$ in the images, its 3D coordinates can be estimated by minimizing the reprojection error

(6) ${\displaystyle \underset{\mathbf{X}\in {\mathbb{R}}^4}{\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}\sum _{c=1}^7{e}_c\Vert {\pi }_c(\mathbf{X})-{\mathbf{x}}_c{\Vert }^2,}$

where $e_c$ is one if the point was visible in image $c$ and zero otherwise. In the absence of camera distortion, that is, when the projection $\pi$ is a purely linear operation in homogeneous coordinates, this can be done for any number of cameras by solving a Singular Value Decomposition (SVD) problem (Hartley and Zisserman, 2000). In the presence of distortions, we replace the observed $u$ and $v$ coordinates of the projections by the corresponding $u_{\text{pinhole}}$ and $u_{\text{pinhole}}$ values of Equation 5 before performing the SVD.

Camera calibration

Request a detailed protocol

Triangulating as described above requires knowing the projection matrices ${\mathbf{𝐏}}_c$ of Equation 3 for each camera $c$, corresponding distortion parameters $\{k_1^x,k_2^x,k_1^y,k_2^y\}$ of Equation 4, together with the intrinsic parameters of focal length and principal point offset. In practice, we use the focal length and principal point offset provided by the manufacturer and estimate the remaining parameters automatically: the three translations and three rotations for each camera that define the corresponding matrix $\mathbf{𝐌}$ of extrinsic parameters along with the distortion parameters.

To avoid having to design the exceedingly small calibration pattern that more traditional methods use to estimate these parameters, we use the fly itself as calibration pattern and minimize the reprojection error of Equation 6 for all joints simultaneously while allowing the camera parameters to also change. In other words we look for

(7) ${\displaystyle \begin{array}{ll}& \mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}\sum _{c=1}^7\sum _{j=1}^m{e}_{c,j}\rho ({\pi }_c({\mathbf{X}}_j)-{\mathbf{x}}_{c,j}),\\ & {\pi }_{c_{1\le c\le 7}}\\ & {{\mathbf{X}}_{\mathbf{j}}}_{1\le j\le m}\end{array}}$

where ${\mathbf{𝐗}}_{\mathbf{𝐣}}$ and ${\mathbf{𝐱}}_{c,j}$ are the 3D locations and 2D projections of the landmarks introduced above and $\rho$ denotes the Huber loss. Equation 7 is known as bundle-adjustment (Hartley and Zisserman, 2000). Huber loss is defined as

${\displaystyle {\rho }_{\delta }(a)=\{\begin{array}{ll}\frac{1}{2}{a}^2& \text{ for }|a|\le \delta \\ \delta \left(|a|-\frac{1}{2}\delta \right)& \text{ otherwise }\end{array}.}$

Replacing the squared loss by the Huber loss makes our approach more robust to erroneous detections ${\mathbf{𝐱}}_{c,j}$. We empirically set $\delta$ to 20 pixels. Note that we perform this minimization with respect to ten degrees-of-freedom per camera: three translations, three rotations, and four distortions.

For this optimization to work properly, we need to initialize these 10 parameters and we need to reduce the number of outliers. To achieve this, the initial distortion parameters are set to zero. We also produce initial estimates for the three rotation and three translation parameters by measuring the distances between adjacent cameras and their relative orientations. To initialize the rotation and translation vectors, we measure the distance and the angle between adjacent cameras, from which we infer rough initial estimates. Finally, we rely on epipolar geometry (Hartley and Zisserman, 2000) to automate outlier rejection. Because the cameras form a rough circle and look inward, the epipolar lines are close to being horizontal Figure 8A. Thus, corresponding 2D projections must belong to the same image rows, or at most a few pixels higher or lower. In practice, this means checking if all 2D predictions lie in nearly the same rows and discarding a priori those that do not.

Figure 8

Download asset Open asset

3D pose correction

The triangulation procedure described above can produce erroneous results when the 2D estimates of landmarks are wrong. Additionally, it may result in implausible 3D poses for the entire animal because it treats each joint independently. To enforce more global geometric constraints, we rely on pictorial structures (Felzenszwalb and Huttenlocher, 2005) as described in Figure 9. Pictorial structures encode the relationship between a set of variables (in this case the 3D location of separate tracked points) in a probabilistic setting using a graphical model. This makes it possible to consider multiple 2D locations ${\mathbf{𝐱}}_{c,j}$ for each landmark ${\mathbf{𝐗}}_c$ instead of only one. This increases the likelihood of finding the true 3D pose.

Figure 9

Download asset Open asset

Generating multiple candidates

Request a detailed protocol

Instead of selecting landmarks as the locations with the maximum probability in maps output by our Stacked Hourglass network, we generate multiple candidate 2D landmark locations $x_{c,j}$. From each probability map, we select 10 local probability maxima that are at least one pixel apart from one another. Then, we generate 3D candidates by triangulating 2D candidates in every tuple of cameras. Because a single point is visible from at most four cameras, this results in at most ${\displaystyle \left(\begin{array}{c}4\\ 2\end{array}\right)\times {10}^2}$ candidates for each tracked point.

Choosing the best candidates

Request a detailed protocol

To identify the best subset of resulting 3D locations, we introduce the probability distribution $P(L|I,\theta )$ that assigns a probability to each solution $L$, consisting of 38 sets of 2D points observed from each camera. Our goal is then to find the most likely one. More formally, $P$ represents the likelihood of a set of tracked points $L$, given the images, model parameters, camera calibration, and geometric constraints. In our formulation, $I$ denotes the seven camera images $I={\{I_c\}}_{1\le c\le 7}$ and $\theta$ represents the set of projection functions ${\pi }_c$ for camera $c$ along with a set of length distributions $S_{i,j}$ between each pair of points $i$ and $j$ that are connected by a limb. $L$ consists of a set of tracked points ${\{L_i\}}_{1\le i\le n}$, where each $L_i$ describes a set of 2D observations $l_{i,c}$ from multiple camera views. These are used to triangulate the corresponding 3D point locations $\overline{l_i}$. If the set of 2D observations is incomplete, as some points are totally occluded in some camera views, we triangulate the 3D point ${\overline{l}}_i$ using the available ones and replace the missing observations by projecting the recovered 3D positions into the images, ${\pi }_c({\overline{l}}_i)$ in Equation 3. In the end, we aim to find the solution $\widehat{L}=\underset{L}{argmax}P(L|I,\theta ).$ This is known as Maximum a Posteriori (MAP) estimation. Using Bayes rule, we write

(8) $P(L|I,\theta )\propto P(I|L,\theta )P(L|\theta ),$

where the two terms can be computed separately. We compute $P(I|J,\theta )$ using the probability maps $H_{j,c}$ generated by the Stacked Hourglass network for the tracked point $j$ for camera $c$. For a single joint $j$ seen by camera $c$, we model the likelihood of observing that particular point using $P(H_{j,c}|l_{j,c})$, which can be directly read from the probability maps as the pixel intensity. Ignoring the dependency between the cameras, we write the overall likelihood as the product of the individual likelihood terms

(9) $P(I|L,\theta )=P(H|L)\propto \prod _{i=1}^n\prod _{c=1}^{7}P(H_{j,c}|l_{i,c}),$

which can be read directly from the probability maps as pixel intensities and represent the network’s confidence that a particular keypoint is located at a particular pixel. When a point is not visible from a particular camera, we assume the probability map only contains a constant non-zero probability, which does not affect the final solution. We express $P(L|\theta )$ as

(10) $P(L|\theta )=P(L|\pi ,S)=\prod _{(i,j)\in E}P({\overline{l}}_i,{\overline{l}}_j|S_{i,j})\prod _{j=1}^n\prod _{c=1}^7{e}_{c,j}{\parallel {\pi }_c({\overline{l}}_j)-l_{c,j}\parallel }_2^{-1},$

where pairwise dependencies $P({\overline{l}}_i,{\overline{l}}_j|S_{i,j})$ between two variables respect the segment length constraint when the variables are connected by a limb. The length of segments defined by pairs of connected 3D points follows a normal distribution. Specifically, we model $P({\overline{l}}_i,{\overline{l}}_j|S_{i,j})$ as $S_{i,j}({\overline{l}}_i,{\overline{l}}_j)=𝒩(\parallel {\overline{l}}_i-{\overline{l}}_j\parallel -{\mu }_{i,j},{\sigma }_{i,j})$. We model the reprojection error for a particular point $j$ as ${\prod }_{c=1}^7{e}_{c,j}{\parallel {\pi }_c({\overline{l}}_j)-l_{c,j}\parallel }_2^{-1}$ which is set to zero using the variable $e_{c,j}$ denoting the visibility of the point $j$ from camera $c$. If a 2D observation for a particular camera is manually set by a user with the DeepFly3D GUI, we take it to be the only possible candidate for that particular image and we set $P(L_j|H)$ to 1, where $j$ denotes the manually assigned pixel location.

Solving the MAP problem using the Max-Sum algorithm

Request a detailed protocol

For general graphs, MAP estimation with pairwise dependencies is NP-hard and therefore intractable. However, in the specific case of non-cyclical graphs, it is possible to solve the inference problem using belief propagation (Bishop, 2006). Since the fly’s skeleton has a root and contains no loops, we can use a message passing approach (Felzenszwalb and Huttenlocher, 2005). It is closely related to Viterbi recurrence and propagates the unary probabilities $P(L_j|L_i)$ between the edges of the graph starting from the root and ending at the leaf nodes. This first propagation ends with the computation of the marginal distribution for the leaf node variables. During the subsequent backward iteration, as $P(L_j)$ for leaf node is computed, the point $L_j$ with maximum posterior probability is selected in $O(k)$ time, where $k$ is the upper bound on the number of proposals for a single tracked point. Next, the distribution $P(L_i|L_j)$ is calculated, adjacent nodes for the leaf node. Continuing this process on all the remaining points results in a MAP solution for the overall distribution $P(L)$, as shown in Figure 9, with overall $O(k^2)$ computational complexity.

Learning the parameters

Request a detailed protocol

We learn the parameters for the set of pairwise distributions $S_{i,j}$ using a maximum likelihood process and assuming the distributions to be Gaussian. We model the segment length $S_{i,j}$ as the euclidean distance between the points ${\overline{l}}_j$ and ${\overline{l}}_j$. We then solve for $\underset{S}{argmax}P(S|L,\theta )$, assuming segments have a Gaussian distribution resulting from the Gaussian noise in point observations $L$. This gives us the mean and variance, defining each distribution $S_{i.j}$. We exclude the same points that we removed from the calibration procedure, that exhibit high reprojection error.

In practice, we observe a large variance for pretarsus values (Figure 10). This is because occlusions occasionally shorten visible tarsal segments. To eliminate the resulting bias, we treat these limbs differently from the others and model the distribution of tibia-tarsus and tarsus-tip points as a Beta distribution, with parameters found using a similar Maximum Likelihood Estimator (MLE) formulation. Assuming the observation errors to be Gaussian and zero-centered, the bundle adjustment procedure can also be understood as an MLE of the calibration parameters (Triggs et al., 2000). Therefore, the entire set of parameters for the formulation can be learned using MLE. Thus, prior information about potentially occluded targets can be used to guide inference. For example, in a head-fixed rodent, the left eye may not always be visible from the right-side of the animal. This information can be incorporated into DeepFly3D’s inference system in the file, skeleton.py, by editing the function camera_see_joint. Afterwards, predictions from occluded cameras will not be used to triangulate a given 3D point. If no such information is provided, every prediction will be used to triangulate a given 3D point.

Figure 10

Download asset Open asset

The pictorial structure formulation can be further expanded using temporal information, penalizing large movements of a single tracked point between two consecutive frames. However, we abstained from using temporal information more extensively for several reasons. First, temporal dependencies would introduce loops in our pictorial structures, thus making exact inference NP-hard as discussed above. This can be handled using loopy belief propagation algorithms (Murphy et al., 1999) but requires multiple message passing rounds, which prevents real-time inference without any theoretical guarantee of optimal inference. Second, the rapidity of Drosophila limb movements makes it hard to assign temporal constraints, even with fast video recording. Finally, we empirically observed that the current formulation, enforcing structured poses in a single temporal frame, already eliminates an overwhelming majority of false-positives inferred during the pose estimation stage of the algorithm.

Modifying DeepFly3D to study other animals

Request a detailed protocol

DeepFly3D does not assume a circular camera arrangement or that there is one degree of freedom in the camera network. Therefore, it could easily be adapted for 3D pose estimation in other animals, ranging from rodents to primates and humans. We illustrate this flexibility by using DeepFly3D to capture human 3D pose in the Human 3.6M Dataset (http://vision.imar.ro/human3.6m/description.php) very popular, publicly available computer vision benchmarking dataset generated using four synchronized cameras (Ionescu et al., 2014; Ionescu et al., 2011) (Figure 11).

Figure 11

Download asset Open asset

Generally, for any new dataset, the user first needs to provide an initial set of manual annotations. The user would describe the number of tracked points and their relationships to one another in a python setup file. Then, in a configuration file, the user specifies the number of cameras along with the resolutions of input images and output probability maps. DeepFly3D will then use these initial manual annotations to (i) train the 2D Stacked Hourglass network, (ii) perform camera calibration without an external calibration pattern, (iii) learn the epipolar geometry to perform outlier detection, and (iv) learn the segment length distributions $S_{i,j}$. After this initial bootstrapping, DeepFly3D can be then used with pictorial structures and active learning to iteratively improve pose estimation accuracy.

The initial manual annotations can be performed using the DeepFly3D Annotation GUI. Afterwards, these annotations can be downloaded from the Annotation GUI as a CSV file using the Save button (Figure 7). Once the CSV file is placed in the images folder, DeepFly3D will automatically read and display the annotations. To train the Stacked Hourglass network, use the csv-path flag while running pose2d.py (found in deepfly/pose2d/). DeepFly3D will then train the Stacked Hourglass network by performing transfer learning using the large MPII dataset and the smaller set of user manual annotations.

To perform camera calibration, the user should select the Calibration button on the GUI Figure 12. DeepFly3D will then perform bundle adjustment (Equation 7) and save the camera parameters in calibration.pickle (found in the images folder). The path of this file should then be added to Config.py to initialize calibration. These initial calibration parameters will then be used in further experiments for fast and accurate convergence. If the number of annotations is insufficient for accurate calibration, or if bundle adjustment is converging too slowly, an initial rough estimate of the camera locations can be set in Config.py. As long as a calibration is set in Config.py, DeepFly3D will use it as a projection matrix to calculate the epipolar geometry between cameras. This step is necessary to perform outlier detection on further calibration operations.

Figure 12

Download asset Open asset

DeepFly3D will also learn the distribution $S_{i,j}$, whose non-zero entries are found in skeleton.py. One can easily calculate these segment length distribution parameters using the functions provided with DeepFly3D. CameraNetwork class (found under deepfly/GUI/), will then automatically load the points and calibration parameters from the images folder. The function CameraNetwork.triangulate will convert 2D annotation points into 3D points using the calibration parameters. The $S_{i,j}$ parameters can then be saved using the pickle library (the save path can be set in Config.py). The calcBoneParams method will then output the segment lengths’ mean and variance. These values will then be used with pictorial structures (Equation 8).

We provide further technical details for how to adapt DeepFly3D to other multi-view datasets online (https://github.com/NeLy-EPFL/DeepFly3D [Günel et al., 2019] copy archived at https://github.com/elifesciences-publications/DeepFly3D).

Experimental setup

We positioned seven Basler acA1920-155um cameras (FUJIFILM AG, Niederhaslistrasse, Switzerland) 94 mm away from the tethered fly, resulting in a circular camera network with the animal in the center (Figure 13). We acquired 960 × 480 pixel video data at 100 FPS under 850 nm infrared ring light illumination (Stemmer Imaging, Pfäffikon Switzerland). Cameras were mounted with 94 mm W.D./1.00 x InfiniStix lenses (Infinity Photo-Optical GmbH, Göttingen). Optogenetic stimulation LED light was filtered out using 700 nm longpass optical filters (Edmund Optics, York UK). Each camera’s depth of field was increased using 5.8 mm aperture retainers (Infinity Photo-Optical GmbH). To automate the timing of optogenetic LED stimulation and camera acquisition triggering, we use an Arduino (Arduino, Sommerville, MA) and custom software written using the Basler camera API.

Figure 13

Download asset Open asset

We assessed the optimal number of cameras for DeepFly3D and concluded that increasing the number of cameras increases accuracy by stabilizing triangulation. Specifically, we observed the following. (i) Calibration is not a significant source of error: calibrating with fewer than seven cameras does not dramatically increase estimation error. (ii) Having more cameras improves triangulation. Reducing the number of cameras down to four, even having calibrated with seven cameras, results in an increase of 0.05 mm triangulation error. This may be because the camera views are sufficiently different, having largely non-overlapping 2D-detection failure cases. Thus, the redundancy provided by having more cameras mitigates detection errors by finding a 3D pose that is consistent across at least two camera views.

Drosophila transgenic lines

Request a detailed protocol

UAS-CsChrimson (Klapoetke et al., 2014) animals were obtained from the Bloomington Stock Center (Stock $\mathrm{\#}55135$). MDN-1-Gal4 (Bidaye et al., 2014) (VT44845-DBD; VT50660-AD) was provided by B. Dickson (Janelia Research Campus, Ashburn). aDN-Gal4 (Hampel et al., 2015)(R76F12-AD; R18C11-DBD), was provided by J. Simpson (University of California, Santa Barbara). Wild-type, PR animals were provided by M. Dickinson (California Institute of Technology, Pasadena).

Optogenetic stimulation experiments

Request a detailed protocol

Experiments were performed in the late morning or early afternoon Zeitgeber time (Z.T.), inside a dark imaging chamber. An adult female animal 2–3 days-post-eclosion (dpe), was mounted onto a custom stage (Chen et al., 2018) and allowed to acclimate for 5 min on an air-supported spherical treadmill (Chen et al., 2018). Optogenetic stimulation was performed using a 617 nm LED (Thorlabs, Newton, NJ) pointed at the dorsal thorax through a hole in the stage, and focused with a lens (LA1951, 01" f = 25.4 mm, Thorlabs, Newton, NJ). Tethered flies were otherwise allowed to behave spontaneously. Data were acquired in 9 s epochs: 2 s baseline, 5 s with optogenetic illumination, and 2 s without stimulation. Individual flies were recorded for five trials each, with one-minute intervals. Data were excluded from analysis if flies pushed their abdomens onto the spherical treadmill—interfering with limb movements—or if flies struggled during optogenetic stimulation, pushing their forelimbs onto the stage for prolonged periods of time.

All data generated and analyzed during this study are included in the DeepFly3D GitHub site: https://github.com/NeLy-EPFL/DeepFly3D (copy archived at https://github.com/elifesciences-publications/DeepFly3D) and in the Harvard Dataverse.

1. 1. Gunel S
   2. Rhodin H
   3. Morales D
   4. Campagnolo J
   5. Ramdya P
   6. Fua P

   (2019) Harvard Dataverse

   aDN-GAL4 Control.

   https://doi.org/10.7910/DVN/PKKXOE
2. 1. Gunel S
   2. Rhodin H
   3. Morales D
   4. Campagnolo J
   5. Ramdya P
   6. Fua P

   (2019) Harvard Dataverse

   MDN-GAL4 Control.

   https://doi.org/10.7910/DVN/HOLXOR
3. 1. Gunel S
   2. Rhodin H
   3. Morales D
   4. Campagnolo J
   5. Ramdya P
   6. Fua P

   (2019) Harvard Dataverse

   aDN-GAL4 UAS-CsChrimson.

   https://doi.org/10.7910/DVN/S4L4KX
4. 1. Gunel S
   2. Rhodin H
   3. Morales D
   4. Campagnolo J
   5. Ramdya P
   6. Fua P

   (2019) Harvard Dataverse

   MDN-GAL4 UAS-CsChrimson.

   https://doi.org/10.7910/DVN/8SUC9U

1. Conference

   1. Andriluka M
   2. Pishchulin L
   3. Gehler P
   4. Schiele B

   (2014) 2d human pose estimation: new benchmark and state of the art analysis

   Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3686–3693.

   https://doi.org/10.1109/CVPR.2014.471

   • Google Scholar
2. 1. Bender JA
   2. Simpson EM
   3. Ritzmann RE

   (2010) Computer-assisted 3D kinematic analysis of all leg joints in walking insects

   PLOS ONE 5:e13617.

   https://doi.org/10.1371/journal.pone.0013617

   • PubMed
   • Google Scholar
3. 1. Berman GJ
   2. Choi DM
   3. Bialek W
   4. Shaevitz JW

   (2014) Mapping the stereotyped behaviour of freely moving fruit flies

   Journal of the Royal Society Interface 11:20140672.

   https://doi.org/10.1098/rsif.2014.0672

   • Google Scholar
4. 1. Bidaye SS
   2. Machacek C
   3. Wu Y
   4. Dickson BJ

   (2014) Neuronal control of Drosophila walking direction

   Science 344:97–101.

   https://doi.org/10.1126/science.1249964

   • PubMed
   • Google Scholar
5. Book

   1. Bishop CM

   (2006)

   Pattern Recognition and Machine Learning

   Springer.

   • Google Scholar
6. 1. Cande J
   2. Namiki S
   3. Qiu J
   4. Korff W
   5. Card GM
   6. Shaevitz JW
   7. Stern DL
   8. Berman GJ

   (2018) Optogenetic dissection of descending behavioral control in Drosophila

   eLife 7:e34275.

   https://doi.org/10.7554/eLife.34275

   • PubMed
   • Google Scholar
7. Conference

   1. Casiez G
   2. Roussel N
   3. Vogel D

   (2012)

   1€ filter: a simple speed-based low-pass filter for noisy input in interactive systems

   Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM. pp. 2527–2530.

   • Google Scholar
8. Conference

   1. Chavdarova T
   2. Baqué P
   3. Bouquet S
   4. Maksai A
   5. Jose C
   6. Lettry L
   7. Fua P
   8. Gool LV
   9. Fleuret F

   (2018)

    WILDTRACK: A Multi-Camera HD Dataset for Dense Unscripted Pedestrian Detection

   The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5030–5039.

   • Google Scholar
9. 1. Chen CL
   2. Hermans L
   3. Viswanathan MC
   4. Fortun D
   5. Aymanns F
   6. Unser M
   7. Cammarato A
   8. Dickinson MH
   9. Ramdya P

   (2018) Imaging neural activity in the ventral nerve cord of behaving adult Drosophila

   Nature Communications 9:4390.

   https://doi.org/10.1038/s41467-018-06857-z

   • PubMed
   • Google Scholar
10. 1. Dombeck DA
    2. Khabbaz AN
    3. Collman F
    4. Adelman TL
    5. Tank DW

    (2007) Imaging large-scale neural activity with cellular resolution in awake, mobile mice

    Neuron 56:43–57.

    https://doi.org/10.1016/j.neuron.2007.08.003

    • PubMed
    • Google Scholar
11. Conference

    1. Elhayek A
    2. Aguiar E
    3. Jain A
    4. Tompson J
    5. Pishchulin L
    6. Andriluka M
    7. Bregler C
    8. Schiele B
    9. Theobalt C

    (2015) Efficient Convnet-Based Marker-Less motion capture in general scenes with a low number of cameras

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

    https://doi.org/10.1109/CVPR.2015.7299005

    • Google Scholar
12. 1. Feany MB
    2. Bender WW

    (2000) A Drosophila model of Parkinson's disease

    Nature 404:394–398.

    https://doi.org/10.1038/35006074

    • PubMed
    • Google Scholar
13. 1. Felzenszwalb PF
    2. Huttenlocher DP

    (2005) Pictorial structures for object recognition

    International Journal of Computer Vision 61:55–79.

    https://doi.org/10.1023/B:VISI.0000042934.15159.49

    • Google Scholar
14. Software

    1. Günel S
    2. Harbulot J
    3. Ramdya P

    (2019) DeepFly3D, version f185c48

    GitHub.

    https://github.com/NeLy-EPFL/DeepFly3D
15. 1. Hampel S
    2. Franconville R
    3. Simpson JH
    4. Seeds AM

    (2015) A neural command circuit for grooming movement control

    eLife 4:e08758.

    https://doi.org/10.7554/eLife.08758

    • PubMed
    • Google Scholar
16. Book

    1. Hartley R
    2. Zisserman A

    (2000)

    Multiple View Geometry in Computer Vision

    Cambridge University Press.

    • Google Scholar
17. 1. Hewitt VL
    2. Whitworth AJ

    (2017) Mechanisms of Parkinson's Disease: Lessons from Drosophila

    Current Topics in Developmental Biology 121:173–200.

    https://doi.org/10.1016/bs.ctdb.2016.07.005

    • PubMed
    • Google Scholar
18. Conference

    1. Ionescu C
    2. Li F
    3. Sminchisescu C

    (2011) Latent structured models for human pose estimation

    2011 International Conference on Computer Vision IEEE. pp. 2220–2227.

    https://doi.org/10.1109/ICCV.2011.6126500

    • Google Scholar
19. 1. Ionescu C
    2. Papava D
    3. Olaru V
    4. Sminchisescu C

    (2014) Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

    IEEE Transactions on Pattern Analysis and Machine Intelligence 36:1325–1339.

    https://doi.org/10.1109/TPAMI.2013.248

    • PubMed
    • Google Scholar
20. 1. Isakov A
    2. Buchanan SM
    3. Sullivan B
    4. Ramachandran A
    5. Chapman JK
    6. Lu ES
    7. Mahadevan L
    8. de Bivort B

    (2016) Recovery of locomotion after injury in Drosophila Melanogaster depends on proprioception

    The Journal of Experimental Biology 219:1760–1771.

    https://doi.org/10.1242/jeb.133652

    • PubMed
    • Google Scholar
21. 1. Kain J
    2. Stokes C
    3. Gaudry Q
    4. Song X
    5. Foley J
    6. Wilson R
    7. De Bivort B

    (1910) Leg-tracking and automated behavioural classification in Drosophila

    Nature Communications 2013:4.

    https://doi.org/10.1038/ncomms2908

    • Google Scholar
22. 1. Klapoetke NC
    2. Murata Y
    3. Kim SS
    4. Pulver SR
    5. Birdsey-Benson A
    6. Cho YK
    7. Morimoto TK
    8. Chuong AS
    9. Carpenter EJ
    10. Tian Z
    11. Wang J
    12. Xie Y
    13. Yan Z
    14. Zhang Y
    15. Chow BY
    16. Surek B
    17. Melkonian M
    18. Jayaraman V
    19. Constantine-Paton M
    20. Wong GK-S
    21. Boyden ES

    (2014) Independent optical excitation of distinct neural populations

    Nature Methods 11:338–346.

    https://doi.org/10.1038/nmeth.2836

    • Google Scholar
23. 1. Maaten LJP

    (2008)

     Visualizing High Dimensional Data Using t-SNE

    Journal of Machine Learning Research pp. 2579–2605.

    • Google Scholar
24. Conference

    1. Martinez J
    2. Hossain R
    3. Romero J
    4. Little JJ

    (2017)

    A simple yet effective baseline for 3D human pose estimation

    ICCV.

    • Google Scholar
25. 1. Mathis A
    2. Mamidanna P
    3. Cury KM
    4. Abe T
    5. Murthy VN
    6. Mathis MW
    7. Bethge M

    (2018) DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

    Nature Neuroscience 21:1281–1289.

    https://doi.org/10.1038/s41593-018-0209-y

    • PubMed
    • Google Scholar
26. 1. McKellar CE
    2. Lillvis JL
    3. Bath DE
    4. Fitzgerald JE
    5. Cannon JG
    6. Simpson JH
    7. Dickson BJ

    (2019) Threshold-Based ordering of sequential actions during Drosophila courtship

    Current Biology 29:426–434.

    https://doi.org/10.1016/j.cub.2018.12.019

    • PubMed
    • Google Scholar
27. Conference

    1. Mehta D
    2. Sridhar S
    3. Sotnychenko O
    4. Rhodin H
    5. Shafiei M
    6. Seidel H
    7. Xu W
    8. Casas D
    9. Theobalt C

    (2017)

    Vnect: Real-Time3D Human Pose Estimation with a Single RGB Camera

    SIGGRAPH.

    • Google Scholar
28. 1. Mendes CS
    2. Bartos I
    3. Akay T
    4. Márka S
    5. Mann RS

    (2013) Quantification of gait parameters in freely walking wild type and sensory deprived Drosophila Melanogaster

    eLife 2:e00231.

    https://doi.org/10.7554/eLife.00231

    • PubMed
    • Google Scholar
29. Conference

    1. Moeslund TB
    2. Granum E

    (2000) Multiple cues used in model-based human motion capture.

    Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580). pp. 362–367.

    https://doi.org/10.1109/AFGR.2000.840660

    • Google Scholar
30. Conference

    1. Moreno-noguer F

    (2017)

    3d human pose estimation from a single image via distance matrix regression

    CVPR.

    • Google Scholar
31. Conference

    1. Murphy KP
    2. Weiss Y
    3. Jordan MI

    (1999)

    Loopy belief propagation for approximate inference: an empirical study

    Onference on Uncertainty in Artificial Intelligence. pp. 467–475.

    • Google Scholar
32. 1. Nath T
    2. Mathis A
    3. Chen AC
    4. Patel A
    5. Bethge M
    6. Mathis MW

    (2019) Using DeepLabCut for 3D markerless pose estimation across species and behaviors

    Nature Protocols 14:2152–2176.

    https://doi.org/10.1038/s41596-019-0176-0

    • PubMed
    • Google Scholar
33. 1. Newell A
    2. Yang K
    3. Deng J

    (2016)

    Stacked Hourglass Networks for Human Pose Estimation

    483–499, European Conference on Computer Vision, Stacked Hourglass Networks for Human Pose Estimation, Springer.

    • Google Scholar
34. Conference

    1. Pavlakos G
    2. Zhou X
    3. Derpanis K
    4. Konstantinos G
    5. Daniilidis K

    (2017a)

    Coarse-To-Fine volumetric prediction for Single-Image 3D human pose

    CVPR.

    • Google Scholar
35. Conference

    1. Pavlakos G
    2. Zhou X
    3. Konstantinos KDG
    4. Kostas D

    (2017b)

    Harvesting multiple views for Marker-Less 3D human pose annotations

    In: CVPR.

    • Google Scholar
36. 1. Pereira TD
    2. Aldarondo DE
    3. Willmore L
    4. Kislin M
    5. Wang SS
    6. Murthy M
    7. Shaevitz JW

    (2019) Fast animal pose estimation using deep neural networks

    Nature Methods 16:117–125.

    https://doi.org/10.1038/s41592-018-0234-5

    • PubMed
    • Google Scholar
37. Conference

    1. Popa AI
    2. Zanfir M
    3. Sminchisescu C

    (2017)

    Deep multitask architecture for integrated 2D and 3D human sensing

    In: CVPR.

    • Google Scholar
38. 1. Puwein J
    2. Ballan L
    3. Ziegler R
    4. Pollefeys M

    (2014)

    Joint Camera Pose Estimation and 3D Human Pose Estimation in a Multi-Camera Setup

    473–487, Accelerated Kmeans Clustering Using Binary Random Projection, Joint Camera Pose Estimation and 3D Human Pose Estimation in a Multi-Camera Setup, springer.

    • Google Scholar
39. Conference

    1. Rhodin H
    2. Robertini N
    3. Casas D
    4. Richardt C
    5. Seidel HP
    6. Theobalt C

    (2016)

    General automatic human shape and motion capture using volumetric contour cues

    ECCV.

    • Google Scholar
40. Conference

    1. Rogez G
    2. Weinzaepfel P
    3. Schmid C

    (2017)

    Lcr-Net: localization-classification-regression for human pose

    In: CVPR.

    • Google Scholar
41. 1. Seeds AM
    2. Ravbar P
    3. Chung P
    4. Hampel S
    5. Midgley FM
    6. Mensh BD
    7. Simpson JH

    (2014) A suppression hierarchy among competing motor programs drives sequential grooming in Drosophila

    eLife 3:e02951.

    https://doi.org/10.7554/eLife.02951

    • PubMed
    • Google Scholar
42. 1. Seelig JD
    2. Chiappe ME
    3. Lott GK
    4. Dutta A
    5. Osborne JE
    6. Reiser MB
    7. Jayaraman V

    (2010) Two-photon calcium imaging from head-fixed Drosophila during optomotor walking behavior

    Nature Methods 7:535–540.

    https://doi.org/10.1038/nmeth.1468

    • PubMed
    • Google Scholar
43. Conference

    1. Simon T
    2. Joo H
    3. Matthews I
    4. Sheikh Y

    (2017)

    Hand keypoint detection in single images using multiview bootstrapping

    In: CVPR.

    • Google Scholar
44. Conference

    1. Sun X
    2. Shang J
    3. Liang S
    4. Wei Y

    (2017)

    Compositional human pose regression

    ICCV.

    • Google Scholar
45. Conference

    1. Takahashi K
    2. Mikami D
    3. Isogawa M
    4. Kimata H

    (2018)

    Human pose as calibration pattern; 3D human pose estimation with multiple unsynchronized and uncalibrated cameras

    The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.

    • Google Scholar
46. Conference

    1. Tekin B
    2. Marquez-neila P
    3. Salzmann M
    4. Fua P

    (2017)

    Learning to fuse 2D and 3D image cues for monocular body pose estimation

    ICCV.

    • Google Scholar
47. 1. Todd JG
    2. Kain JS
    3. de Bivort BL

    (2017) Systematic exploration of unsupervised methods for mapping behavior

    Physical Biology 14:015002.

    https://doi.org/10.1088/1478-3975/14/1/015002

    • PubMed
    • Google Scholar
48. Preprint

    1. Tome D
    2. Russell C
    3. Agapito L

    (2017) Lifting from the deep: convolutional 3D pose estimation from a single image

    arXiv.

    https://arxiv.org/abs/1701.00295

    • Google Scholar
49. Book

    1. Triggs B
    2. Mclauchlan P
    3. Hartley R
    4. Fitzgibbon A

    (2000)

    Vision Algorithms: Theory and Practice

    Springer.

    • Google Scholar
50. 1. Uhlmann V
    2. Ramdya P
    3. Delgado-Gonzalo R
    4. Benton R
    5. Unser M

    (2017) FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila

    PLOS ONE 12:e0173433.

    https://doi.org/10.1371/journal.pone.0173433

    • PubMed
    • Google Scholar
51. Conference

    1. Zhou X
    2. Huang Q
    3. Sun X
    4. Xue X
    5. Wei Y

    (2017)

    Weakly-supervised transfer for 3d human pose estimation in the wild

    IEEE International Conference on Computer Vision.

    • Google Scholar

Author details

1. Semih Günel

   1. Computer Vision Laboratory, School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
   2. Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, Switzerland

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing—original draft, Writing—review and editing

   For correspondence

   semih.gunel@epfl.ch

   Competing interests

   No competing interests declared

2. Helge Rhodin

   1. Computer Vision Laboratory, School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
   2. Department of Computer Science, UBC, Vancouver, Canada

   Contribution

   Conceptualization, Software, Formal analysis, Supervision, Methodology, Writing—original draft, Project administration, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2692-0801

3. Daniel Morales

   Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, Switzerland

   Contribution

   Data curation, Investigation, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-7469-0898

4. João Campagnolo

   Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, Switzerland

   Contribution

   Data curation, Software, Writing—review and editing

   Competing interests

   No competing interests declared

5. Pavan Ramdya

   Neuroengineering Laboratory, Brain Mind Institute & Interfaculty Institute of Bioengineering, School of Life Sciences, EPFL, Lausanne, Switzerland

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Methodology, Writing—original draft, Project administration, Writing—review and editing

   Contributed equally with

   Pascal Fua

   For correspondence

   pavan.ramdya@epfl.ch

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5425-4610

6. Pascal Fua

   Computer Vision Laboratory, School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Methodology, Project administration, Writing—review and editing

   Contributed equally with

   Pavan Ramdya

   Competing interests

   No competing interests declared

• 8,381

  views

• 913

  downloads

• 144

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Citations by DOI

• 144

  citations for umbrella DOI https://doi.org/10.7554/eLife.48571