Studying animal behavior can reveal how animals make decisions based on what they sense in their environment, but measuring behavior can be difficult and time-consuming. Computer programs that measure and analyze animal movement have made these studies faster and easier to complete. These tools have also made more advanced behavioral experiments possible, which have yielded new insights about how the brain organizes behavior.

Recently, scientists have started using new machine learning tools called deep neural networks to measure animal behavior. These tools learn to measure animal posture – the positions of an animal’s body parts in space – directly from real data, such as images or videos, without being explicitly programmed with instructions to perform the task. This allows deep learning algorithms to automatically track the locations of specific animal body parts in videos faster and more accurately than previous techniques. This ability to learn from images also removes the need to attach physical markers to animals, which may alter their natural behavior.

Now, Graving et al. have created a new deep learning toolkit for measuring animal behavior that combines components from previous tools with the latest advances in computer science. Simple modifications to how the algorithms are trained can greatly improve their performance. For example, adding connections between layers, or ‘neurons’, in the deep neural network and training the algorithm to learn the full geometry of the body – by drawing lines between body parts – both enhance its accuracy. As a result of adding these changes, the new toolkit can measure an animal's pose from previously unseen images with high speed and accuracy, after being trained on just 100 examples. Graving et al. tested their model on videos of fruit flies, zebras and locusts, and found that, after training, it was able to accurately track the animals’ movements. The new toolkit has an easy-to-use software interface and is freely available for other scientists to use and build on.

The new toolkit may help scientists in many fields including neuroscience and psychology, as well as other computer scientists. For example, companies like Google and Apple use similar algorithms to recognize gestures, so making those algorithms faster and more efficient may make them more suitable for mobile devices like smartphones or virtual-reality headsets. Other possible applications include diagnosing and tracking injuries, or movement-related diseases in humans and livestock.

Understanding the relationships between individual behavior, brain activity (reviewed by Krakauer et al., 2017), and collective and social behaviors (Rosenthal et al., 2015; Strandburg-Peshkin et al., 2013; Jolles et al., 2017; Klibaite et al., 2017; Klibaite and Shaevitz, 2019) is a central goal of the behavioral sciences—a field that spans disciplines from neuroscience to psychology, ecology, and genetics. Measuring and modelling behavior is key to understanding these multiple scales of complexity, and, with this goal in mind, researchers in the behavioral sciences have begun to integrate theory and methods from physics, computer science, and mathematics (Anderson and Perona, 2014; Berman, 2018; Brown and de Bivort, 2018). A cornerstone of this interdisciplinary revolution is the use of state-of-the-art computational tools, such as computer vision algorithms, to automatically measure locomotion and body posture (Dell et al., 2014). Such a rich description of animal movement then allows for modeling, from first principles, the full behavioral repertoire of animals (Stephens et al., 2011; Berman et al., 2014b; Berman et al., 2016; Wiltschko et al., 2015; Johnson et al., 2016b; Todd et al., 2017; Klibaite et al., 2017; Markowitz et al., 2018; Klibaite and Shaevitz, 2019; Costa et al., 2019). Tools for automatically measuring animal movement represent a vital first step toward developing unified theories of behavior across scales (Berman, 2018; Brown and de Bivort, 2018). Therefore, technical factors like scalability, robustness, and usability are issues of critical importance, especially as researchers across disciplines begin to increasingly rely on these methods.

Two of the latest contributions to the growing toolbox for quantitative behavioral analysis are from Mathis et al. (2018) and Pereira et al. (2019), who make use of a popular type of machine learning model called convolutional neural networks, or CNNs (LeCun et al., 2015; Appendix 2), to automatically measure detailed representations of animal posture—structural keypoints, or joints, on the animal’s body—directly from images and without markers. While these methods offer a major advance over conventional methods with regard to data quality and detail, they have disadvantages in terms of speed and robustness, which may limit their practical applications. To address these problems, we introduce a new software toolkit, called DeepPoseKit, with methods that are fast, robust, and easy-to-use. We run experiments using multiple datasets to compare our new methods with those from Mathis et al. (2018) and Pereira et al. (2019), and we find that our approach offers considerable improvements. These results also demonstrate the flexibility of our toolkit for both laboratory and field situations and exemplify the wide applicability of our methods across a range of species and experimental conditions.

Animal pose estimation using deep learning

In the past, conventional methods for measuring posture with computer vision relied on species-specific algorithms (Uhlmann et al., 2017), highly specialized or restrictive experimental setups (Mendes et al., 2013; Kain et al., 2013), attaching intrusive physical markers to the study animal (Kain et al., 2013), or some combination thereof. These methods also typically required expert computer-vision knowledge to use, were limited in the number or type of body parts that could be tracked (Mendes et al., 2013), involved capturing and handling the study animals to attach markers (Kain et al., 2013)—which is not possible for many species—and despite best efforts to minimize human involvement, often required manual intervention to correct errors (Uhlmann et al., 2017). These methods were all built to work for a small range of conditions and typically required considerable effort to adapt to novel contexts.

In contrast to conventional computer-vision methods, modern deep-learning–based methods can be used to achieve near human-level accuracy in almost any scenario by manually annotating data (Figure 1)—known as a training set—and training a general-purpose image-processing algorithm—a convolutional neural network or CNN—to automatically estimate the locations of an animal’s body parts directly from images (Figure 2). State-of-the-art machine learning methods, like CNNs, use these training data to parameterize a model describing the statistical relationships between a set of input data (i.e., images) and the desired output distribution (i.e., posture keypoints). After adequate training, a model can be used to make predictions on previously-unseen data from the same dataset—inputs that were not part of the training set, which is known as inference. In other words, these models are able to generalize human-level expertise at scale after having been trained on only a relatively small number of examples. We provide more detailed background information on using CNNs for pose estimation in Appendices 2–6.

Figure 1 with 1 supplement see all

Download asset Open asset

Figure 2

Download asset Open asset

Similar to conventional pose estimation methods, the task of implementing deep-learning models in software and training them on new data is complex and requires expert knowledge. However, in most cases, once the underlying model and training routine are implemented, a high-accuracy pose estimation model for a novel context can be built with minimal modification—often just by changing the training data. With a simplified toolkit and high-level software interface designed by an expert, even scientists with limited computer-vision knowledge can begin to apply these methods to their research. Once the barriers for implementing and training a model are sufficiently reduced, the main bottleneck for using these methods becomes collecting an adequate training set—a labor-intensive task made less time-consuming by techniques described in Appendix 3.

Mathis et al. (2018) and Pereira et al. (2019) were the first to popularize the use of CNNs for animal pose estimation. These researchers built on work from the human pose estimation literature (e.g., Andriluka et al., 2014; Insafutdinov et al., 2016; Newell et al., 2016) using a type of fully-convolutional neural network or F-CNN (Long et al., 2015; Appendix 4) often referred to as an encoder-decoder model (Appendix 4: 'Encoder-decoder models'). These models are used to measure animal posture by training the network to transform images into probabilistic estimates of keypoint locations, known as confidence maps (shown in Figure 2), that describe the body posture for one or more individuals. These confidence maps are processed to produce the 2-D spatial coordinates of each keypoint, which can then be used for further analysis.

While deep-learning models typically need large amounts of training data, both Mathis et al. (2018) and Pereira et al. (2019) have demonstrated that near human-level accuracy can be achieved with few training examples (Appendix 3). In order to ensure generalization to large datasets, both groups of researchers introduced ideas related to iteratively refining the training set used for model fitting (Mathis et al., 2018; Pereira et al., 2019). In particular, Pereira et al. (2019) describe a technique known as active learning where a trained model is used to initialize new training data and reduce annotation time (Appendix 3). Mathis et al. (2018) describe multiple techniques that can be used to further refine training data and minimize errors when making predictions on the full dataset. Simple methods to accomplish this include filtering data or selecting new training examples based on confidence scores or the entropy of the confidence maps from the model output. Nath et al. (2019) also introduced the use temporal derivatives (i.e., speed and acceleration) and autoregressive models to identify outlier frames, which can then be labeled to refine the training set or excluded from further analysis on the final dataset (Figure 1).

Here, we introduce fast, flexible, and robust pose estimation methods, with a software interface—a high-level programming interface (API) and graphical user-interface (GUI) for annotations—that emphasizes usability. Our methods build on the state-of-the-art for individual pose estimation (Newell et al., 2016; Appendix 5), convolutional regression models (Jégou et al., 2017; Appendix 4: 'Encoder-decoder models'), and conventional computer vision algorithms (Guizar-Sicairos et al., 2008) to improve model efficiency and achieve faster, more accurate results on multiple challenging pose estimation tasks. We developed two model implementations—including a new model architecture that we call Stacked DenseNet—and a new method for processing confidence maps called subpixel maxima that provides fast and accurate peak detection for estimating keypoint locations with subpixel precision—even at low spatial resolutions. We also discuss a modification to incorporate a hierarchical posture graph for learning the multi-scale geometry between keypoints on the animal’s body, which increases accuracy when training pose estimation models. We ran experiments to optimize our approach and compared our new models to the models from Mathis et al. (2018) (DeepLabCut) and Pereira et al. (2019) (LEAP) in terms of speed, accuracy, training time, and generalization ability. We benchmarked these models using three image datasets recorded in the laboratory and the field—including multiple interacting individuals that were first localized and cropped from larger, multi-individual images (see 'Materials and methods’ for details).

Stacked DenseNet is fast and robust

We benchmarked our new model implementations against the models Pereira et al. (2019) and Mathis et al. (2018). We find that our Stacked DenseNet model outperforms both the LEAP model (Pereira et al., 2019) and the DeepLabCut model (Mathis et al., 2018) in terms of speed while also achieving much higher accuracy than the LEAP model (Pereira et al., 2019) with similar accuracy to the DeepLabCut model (Mathis et al., 2018; Figure 3a). We found that both the Stacked Hourglass and Stacked DenseNet models outperformed the LEAP model (Pereira et al., 2019). Notably our Stacked DenseNet model achieved approximately 2× faster inference speeds with 3× higher mean accuracy. Not only were our models average prediction error significantly improved, but also, importantly, the variance was lower—indicating that our models produced fewer extreme prediction errors. At $\frac{1}{4}\times$ resolution, our Stacked DenseNet model consistently achieved prediction accuracy nearly identical to the DeepLabCut model (Mathis et al., 2018) while running inference at nearly 2× the speed and using only ∼5% of the parameters—1.5 million vs. ∼26 million. Detailed results of our model comparisons are shown in Figure 3—figure supplement 1.

Figure 3 with 1 supplement see all

Download asset Open asset

While the Stacked DenseNet model used for comparisons is already fast, inference speed could be further improved by using a $\frac{1}{8}\times$ output without much increase in error (Appendix 1—figure 1) or by further adjusting the hyperparameters to constrain the size of the model. Our Stacked Hourglass implementation followed closely behind the performance of our Stacked DenseNet model and the DeepLabCut model (Mathis et al., 2018) but consistently performed more poorly than our Stacked DenseNet model in terms of prediction accuracy, so we excluded this model from further analysis. We were also able to reproduce the results reported by Pereira et al. (2019) that the LEAP model and the Stacked Hourglass model (Newell et al., 2016) have similar average prediction error for the fly dataset. However, we also find that the LEAP model (Pereira et al., 2019) has much higher variance, which suggests it is more prone to extreme prediction errors—a problem for further data analysis.

Here, we have presented a new software toolkit, called DeepPoseKit, for estimating animal posture using deep learning models. We built on the state-of-the-art for individual pose estimation using convolutional neural networks to achieve fast inference without reducing accuracy or generalization ability. Our new pose estimation model, called Stacked DenseNet, offers considerable improvements (Figure 3a; Figure 3—figure supplement 1) over the models from Mathis et al. (2018) (DeepLabCut) and Pereira et al. (2019) (LEAP), and our software framework also provides a simplified interface (Figure 3b) for using these advanced tools to measure animal behavior and locomotion. We tested our methods across a range of datasets from controlled laboratory environments with single individuals to challenging field situations with multiple interacting individuals and variable lighting conditions. We found that our methods perform well for all these situations and require few training examples to achieve good predictive performance on new data—without the use of transfer learning. We ran experiments to optimize our approach and discovered that some straightforward modifications can greatly improve speed and accuracy. Additionally, we demonstrated that these modifications improve not the just the average error but also help to reduce extreme prediction errors—a key determinant for the reliability of subsequent statistical analysis.

While our results offer a good-faith comparison of the available methods for animal pose estimation, there is inherent uncertainty that we have attempted to account for but may still bias our conclusions. For example, deep learning models are trained using stochastic optimization algorithms that give different results with each replicate, and the Bayesian statistical methods we use for comparison are explicitly probabilistic in nature. There is also great variability across hardware and software configurations when using these models in practice (Mathis and Warren, 2018), so performance may change across experimental setups and datasets. Additionally, we demonstrated that some models may perform better than others for specific applications (Figure 3—figure supplement 1), and to account for this, our toolkit offers researchers the ability to choose the model that best suits their requirements—including the LEAP model (Pereira et al., 2019) and the DeepLabCut model (Mathis et al., 2018).

We highlighted important considerations when using CNNs for pose estimation and reviewed the progress of fully convolutional regression models from the literature. The latest advancements for these models have been driven mostly by a strategy of adding more connections between layers to increase performance and efficiency (e.g., Jégou et al., 2017). Future progress for this class of models may require better loss functions (Goodfellow et al., 2014; Johnson et al., 2016a; Chen et al., 2017; Zhang et al., 2018), models that more explicitly incorporate the spatial dependencies within a scene (Van den Oord et al., 2016b), and temporal structure of the data (Seethapathi et al., 2019), as well as more mathematically principled approaches (e.g., Weigert et al., 2018; Roy et al., 2019) such as the application of formal probabilistic concepts (Kendall and Gal, 2017) and Bayesian inference at scale (Tran et al., 2018).

Measuring behavior is a critical factor for many studies in neuroscience (Krakauer et al., 2017). Understanding the connections between brain activity and behavioral output requires detailed and objective descriptions of body posture that match the richness and resolution neural measurement technologies have provided for years (Anderson and Perona, 2014; Berman, 2018; Brown and de Bivort, 2018), which our methods and other deep-learning–based tools provide (Mathis et al., 2018; Pereira et al., 2019). We have also demonstrated the possibility that our toolkit could be used for real-time inference, which allows for closed-loop experiments where sensory stimuli or optogenetic stimulation are controlled in response to behavioral measurements (e.g., Bath et al., 2014; Stowers et al., 2017). Using real-time measurements in conjunction with optogenetics or thermogenetics may be key to disentangling the causal structure of motor output from the brain—especially given that recent work has shown an animal’s response to optogenetic stimulation can differ depending on the behavior it is currently performing (Cande et al., 2018). Real-time behavioral quantification is also particularly important as closed-loop virtual reality is quickly becoming an indispensable tool for studying sensorimotor relationships in individuals and collectives (Stowers et al., 2017).

Quantifying individual movement is essential for revealing the genetic (Kain et al., 2012; Brown et al., 2013; Ayroles et al., 2015) and environmental (Bierbach et al., 2017; Akhund-Zade et al., 2019; Versace et al., 2019) underpinnings of phenotypic variation in behavior—as well as the phylogeny of behavior (e.g., Berman et al., 2014a). Measuring individual behavioral phenotypes requires tools that are robust, scaleable, and easy-to-use, and our approach offers the ability to quickly and accurately quantify the behavior of many individuals in great detail. When combined with tools for genetic manipulations (Ran et al., 2013; Doudna and Charpentier, 2014), high-throughput behavioral experiments (Alisch et al., 2018; Javer et al., 2018; Werkhoven et al., 2019), and behavioral analysis (e.g., Berman et al., 2014b; Wiltschko et al., 2015), our methods could help to provide the data resolution and statistical power needed for dissecting the complex relationships between genes, environment, and behavioral variation.

When used together with other tools for localization and tracking (e.g., Pérez-Escudero et al., 2014; Crall et al., 2015; Graving, 2017; Romero-Ferrero et al., 2019; Wild et al., 2018; Boenisch et al., 2018), our methods are capable of reliably measuring posture for multiple interacting individuals. The importance of measuring detailed representations of individual behavior when studying animal collectives has been well established (Strandburg-Peshkin et al., 2013; Rosenthal et al., 2015; Strandburg-Peshkin et al., 2015; Strandburg-Peshkin et al., 2017). Estimating body posture is an essential first step for unraveling the sensory networks that drive group coordination, such as vision-based networks measured via raycasting (Strandburg-Peshkin et al., 2013; Rosenthal et al., 2015). Additionally, using body pose estimation in combination with computational models of behavior (e.g., Costa et al., 2019; Wiltschko et al., 2015) and unsupervised behavioral classification methods (e.g., Berman et al., 2014b; Pereira et al., 2019) may allow for further dissection of how information flows through groups by revealing the networks of behavioral contagion across multiple timescales and sensory modalities. While we have provided a straightforward solution for applying existing pose estimation methods to measure collective behavior, there still remain many challenging scenarios where these methods would fail. For example, tracking posture in a densely packed bee hive or school of fish would require novel solutions to deal with the 3-D nature of individual movement, which includes maintaining individual identities and dealing with the resulting occlusions that go along with imaging these types of biological systems.

When combined with unmanned aerial vehicles (UAVs; Schiffman, 2014) or other field-based imaging (Francisco et al., 2019), applying these methods to the study of individuals and groups in the wild can provide high-resolution behavioral data that goes beyond the capabilities of current GPS and accelerometry-based technologies (Nagy et al., 2010; Nagy et al., 2013; Kays et al., 2015; Strandburg-Peshkin et al., 2015; Strandburg-Peshkin et al., 2017; Flack et al., 2018)—especially for species that are impractical to study with tags or collars. Additionally, by applying these methods in conjunction with 3-D habitat reconstruction—using techniques from photogrammetry (Strandburg-Peshkin et al., 2017; Francisco et al., 2019)—field-based studies can begin to integrate fine-scale behavioral measurements with the full 3-D environment in which the behavior evolved. Future advances will likely allow for the calibration and synchronizaton of imaging devices across multiple UAVs (e.g., Price et al., 2018; Saini et al., 2019). This would make it possible to measure the full 3-D posture of wild animals (e.g., Zuffi et al., 2019) in scenarios where fixed camera systems (e.g., Nath et al., 2019) would not be tractable, such as during migratory or predation events. When combined, these technologies could allow researchers to address questions about the behavioral ecology of animals that were previously impossible to answer.

Computer vision algorithms for measuring behavior at the scale of posture have rapidly advanced in a very short time; nevertheless, the task of pose estimation is far from solved. There are hard limitations to this current generation of pose estimation methods that are primarily related to the requirement for human annotations and user-defined keypoints—both in terms of the number of keypoints, the specific body parts being tracked, and the inherent difficulty of incorporating temporal information into the annotation and training procedures. Often the body parts chosen for annotation are an obvious fit for the experimental design and have reliably visible reference points on the animal’s body that make them easy to annotate. However, in many cases the required number and type of body parts needed for data analysis may not be so obvious—such as in the case of unsupervised behavior classification methods (Berman et al., 2014b; Pereira et al., 2019). Additionally, the reference points for labeling images with keypoints can be hard to define and consistently annotate across images, which is often the case for soft or flexible-bodied animals like worms and fish. Moreover, due to the laborious nature of annotating keypoints, the current generation of methods also rarely takes into account the natural temporal structure of the data, instead treating each video frame as a statistically independent event, which can lead to extreme prediction errors (reviewed by Seethapathi et al., 2019). Extending these methods to track the full three-dimensional posture of animals also typically requires the use of multiple synchronized cameras (Nath et al., 2019; Günel et al., 2019), which increases the cost and complexity of creating an experimental setup, as well as the manual labor required for annotating a training set, which must include labeled data from every camera view.

These limitations make it clear that fundamentally-different methods may be required to move the field forward. New pose estimation methods are already replacing human annotations with fully articulated volumetric 3-D models of the animal’s body (e.g., the SMAL model from Zuffi et al., 2017 or the SMALST model from Zuffi et al., 2019), and the 3-D scene can be estimated using unsupervised, semi-supervised, or weakly-supervised methods (e.g., Jaques et al., 2019; Zuffi et al., 2019), where the shape, position, and posture of the animal’s body, the camera position and lens parameters, and the background environment and lighting conditions are jointly learned directly from 2-D images by a deep-learning model (Valentin et al., 2019; Zuffi et al., 2019). These inverse graphics models (Kulkarni et al., 2015; Sabour et al., 2017; Valentin et al., 2019) take advantage of recently developed differentiable graphics engines that allow 3-D rendering parameters to be controlled using standard optimization methods (Zuffi et al., 2019; Valentin et al., 2019). After optimization, the volumetric 3-D timeseries data predicted by the deep learning model could be used directly for behavioral analysis or specific keypoints or body parts could be selected for analysis post-hoc. In order to more explicitly incorporate the natural statistical properties of the data, these models also apply perceptual loss functions (Johnson et al., 2016a; Zhang et al., 2018; Zuffi et al., 2019) and could be extended to use adversarial (Goodfellow et al., 2014; Chen et al., 2017) loss functions, both of which incorporate spatial dependencies within the scene rather than modelling each video frame as a set of statistically independent pixel distributions—as is the case with current methods that use likelihood functions such as pixel-wise mean squared error (e.g., Pereira et al., 2019) or cross-entropy loss (e.g., Mathis et al., 2018). Because there is limited or no requirement for human-provided labels with these new methods, these models could also be easily modified to incorporate the temporal structure of the data using autoregressive representations (e.g., Van den Oord et al., 2016a; Van den Oord et al., 2016b; Kumar et al., 2019), rather than modeling the scene in each video frame as a statistically independent event. Together these advances could lead to larger, higher-resolution, more reliable behavioral datasets that could revolutionize our understanding of relationships between behavior, the brain, and the environment.

In conclusion, we have presented a new toolkit, called DeepPoseKit, for automatically measuring animal posture from images. We combined recent advances from the literature to create methods that are fast, robust, and widely applicable to a range of species and experimental conditions. When designing our framework we emphasized usability across the entire software interface, which we expect will help to make these advanced tools accessible to a wider range of researchers. The fast inference and real-time capabilities of our methods should also help further reduce barriers to previously intractable questions across many scientific disciplines—including neuroscience, ethology, and behavioral ecology—both in the laboratory and the field.

We ran three main experiments to test and optimize our approach. First, we compared our new subpixel maxima layer to an integer-based global maxima with downsampled outputs ranging from 1× to $\frac{1}{16}\times$ the input resolution using our Stacked DenseNet model. Next, we tested if training our Stacked DenseNet model to predict the multi-scale geometry of the posture graph improves accuracy. Finally, we compared our model implementations of Stacked Hourglass and Stacked DenseNet to the models from Pereira et al. (2019) (LEAP) and Mathis et al. (2018) (DeepLabCut), which we also implemented in our framework (see Appendix 8 for details). We assessed both the inference speed and prediction accuracy of each model as well as training time and generalization ability. When comparing these models we incorporated the relevant improvements from our experiments—including subpixel maxima and predicting multi-scale geometry between keypoints—unless otherwise noted (see Appendix 8).

While we do make comparisons to the DeepLabCut model (Mathis et al., 2018) we do not use the same training routine as Mathis et al. (2018) and Nath et al. (2019), who use binary cross-entropy loss for optimizing the confidence maps in addition to the location refinement maps described by Insafutdinov et al. (2016). We made this modification in order to hold the training routine constant for each model while only varying the model itself. However, we find that these differences between training routines effectively have no impact on performance when the models are trained using the same dataset and data augmentations (Appendix 8—figure 1). We also provide qualitative comparisons to demonstrate that, when trained with our DeepPoseKit framework, our implementation of the DeepLabCut model (Mathis et al., 2018) appears to produce fewer prediction errors than the original implementation from Mathis et al. (2018) and Nath et al. (2019) when applied to a novel video (Appendix 8—figure 1—figure supplements 1 and 2; Appendix 8—figure 1—video 1).

Datasets

Request a detailed protocol

We performed experiments using the vinegar or 'fruit’ fly (Drosophila melanogaster) dataset (Figure 4, Figure 4—video 1) provided by Pereira et al. (2019), and to demonstrate the versatility of our methods we also compared model performance across two previously unpublished posture data sets from groups of desert locusts (Schistocerca gregaria) recorded in a laboratory setting (Figure 4, Figure 4—video 2), and herds of Grévy’s zebras (Equus grevyi) recorded in the wild (Figure 4, Figure 4—video 3). The locust and zebra datasets are particularly challenging for pose estimation as they feature multiple interacting individuals—with focal individuals centered in the frame—and the latter with highly-variable environments and lighting conditions. These datasets are freely-available from https://github.com/jgraving/deepposekit-data (Graving et al., 2019; copy archived at https://github.com/elifesciences-publications/DeepPoseKit-Data).

Figure 4 with 3 supplements see all

Download asset Open asset

Our locust dataset consisted of a group of 100 locusts in a circular plastic arena 1 m in diameter. The locust group was recorded from above using a high-resolution camera (Basler ace acA2040-90umNIR) and video recording system (Motif, loopbio GmbH). Locusts were localized and tracked using 2-D barcode markers (Graving, 2017) attached to the thorax with cyanoacrylate glue, and any missing localizations (<0.02% of the total dataset) between successful barcode reads were interpolated with linear interpolation. Our zebra dataset consisted of variably sized groups in the wild recorded from above using a commercially available quadcopter drone (DJI Phantom 4 Pro). Individual zebra were localized using custom deep-learning software based on Faster R-CNN (Ren et al., 2015) for predicting bounding boxes. The positions of each zebra were then tracked across frames using a linear assignment algorithm (Munkres, 1957) and data were manually verified for accuracy.

After positional tracking, the videos were then cropped using the egocentric coordinates of each individual and saved as separate videos—one for each individual. The images used for each training set were randomly selected using the k-means sampling procedure (with k = 10) described by Pereira et al. (2019) (Appendix 3) to reduce correlation between sampled images. After annotating the images with keypoints, we rotationally and translationally aligned the images and keypoints using the central body axis of the animal in each labeled image. This step allowed us to more easily perform data augmentations (see 'Model training’) that allow the model to make accurate predictions regardless of the animal’s body size and orientation (see Appendix 6). However, this preprocessing step is not a strict requirement for training, and there is no requirement for this preprocessing step when making predictions on new unlabeled data, such as with the methods described by Pereira et al. (2019) (Appendix 6). Before training each model we split each annotated dataset into randomly selected training and validation sets with 90% training examples and 10% validation examples, unless otherwise noted. The details for each dataset are described in Table 1.

Table 1

Name	Species	Resolution	# Images	# Keypoints	Individuals	Source
Vinegar fly	Drosophila melanogaster	192 × 192	1500	32	Single	Pereira et al., 2019
Desert locust	Schistocerca gregaria	160 × 160	800	35	Multiple	This paper
Grévy’s zebra	Equus grevyi	160 × 160	900	9	Multiple	This paper

Data used and generated for experiments and model comparisons are included in the supporting files. Posture datasets can be found at https://github.com/jgraving/deepposekit-data (copy archived at https://github.com/elifesciences-publications/DeepPoseKit-Data). The code for DeepPoseKit is publicly available at the URL we provided in the paper: https://github.com/jgraving/deepposekit/ (copy archived at https://github.com/elifesciences-publications/DeepPoseKit).

1. 1. Graving JM
   2. Chae D
   3. Naik H
   4. Li L
   5. Koger B
   6. Costelloe BR
   7. Couzin IA

   (2019) Zenodo

   Example Datasets for DeepPoseKit (Version v0.1-doi) [Data set].

   https://doi.org/10.5281/zenodo.3366908

1. 1. Pereira TD
   2. Aldarondo DE
   3. Willmore L
   4. Kislin M
   5. Wang SS-H
   6. Murthy M
   7. Shaevitz JW

   (2018) DataSpace

   Fast animal pose estimation using deep neural networks.

   https://dataspace.princeton.edu/jspui/handle/88435/dsp01pz50gz79z

1. 1. Abadi M
   2. Agarwal A
   3. Barham P
   4. Brevdo E
   5. Chen Z
   6. Citro C
   7. Corrado GS
   8. Davis A
   9. Dean J
   10. Devin M
   11. Ghemawat S
   12. Goodfellow I
   13. Harp A
   14. Irving G
   15. Isard M
   16. Jia Y
   17. Jozefowicz R
   18. Kaiser L
   19. Kudlur M
   20. Levenberg J
   21. Mané D
   22. Monga R
   23. Moore S
   24. Murray D
   25. Olah C
   26. Schuster M
   27. Shlens J
   28. Steiner B
   29. Sutskever I
   30. Talwar K
   31. Tucker P
   32. Vanhoucke V
   33. Vasudevan V
   34. Viégas F
   35. Vinyals O
   36. Warden P
   37. Wattenberg M
   38. Wicke M
   39. Yu Y
   40. Zheng X

   (2015) TensorFlow: Large-scale machine learning on heterogeneous systems

   TensorFlow: Large-scale machine learning on heterogeneous systems, https://www.tensorflow.org.

   https://www.tensorflow.org

   • Google Scholar
2. 1. Akhund-Zade J
   2. Ho S
   3. O'Leary C
   4. de Bivort B

   (2019) The effect of environmental enrichment on behavioral variability depends on genotype, behavior, and type of enrichment

   The Journal of Experimental Biology 222:jeb.202234.

   https://doi.org/10.1242/jeb.202234

   • Google Scholar
3. 1. Alisch T
   2. Crall JD
   3. Kao AB
   4. Zucker D
   5. de Bivort BL

   (2018) MAPLE (modular automated platform for large-scale experiments), a robot for integrated organism-handling and phenotyping

   eLife 7:e37166.

   https://doi.org/10.7554/eLife.37166

   • PubMed
   • Google Scholar
4. 1. Anderson DJ
   2. Perona P

   (2014) Toward a science of computational ethology

   Neuron 84:18–31.

   https://doi.org/10.1016/j.neuron.2014.09.005

   • PubMed
   • Google Scholar
5. Conference

   1. Andriluka M
   2. Pishchulin L
   3. Gehler P
   4. Schiele B

   (2014) 2d human pose estimation: new benchmark and state of the art analysis

   IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

   https://doi.org/10.1109/cvpr.2014.471

   • Google Scholar
6. Conference

   1. Andriluka M
   2. Iqbal U
   3. Insafutdinov E
   4. Pishchulin L
   5. Milan A
   6. Gall J
   7. Schiele B

   (2018) Posetrack: a benchmark for human pose estimation and tracking

   Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5167–5176.

   https://doi.org/10.1109/cvpr.2018.00542

   • Google Scholar
7. Preprint

   1. Ayinde BO
   2. Zurada JM

   (2018) Building efficient convnets using redundant feature pruning

   arXiv.

   https://arxiv.org/abs/1802.07653

   • Google Scholar
8. 1. Ayroles JF
   2. Buchanan SM
   3. O'Leary C
   4. Skutt-Kakaria K
   5. Grenier JK
   6. Clark AG
   7. Hartl DL
   8. de Bivort BL

   (2015) Behavioral idiosyncrasy reveals genetic control of phenotypic variability

   PNAS 112:6706–6711.

   https://doi.org/10.1073/pnas.1503830112

   • PubMed
   • Google Scholar
9. Preprint

   1. Badrinarayanan V
   2. Kendall A
   3. Cipolla R

   (2015) Segnet: a deep convolutional encoder-decoder architecture for image segmentation

   arXiv.

   https://arxiv.org/abs/1511.00561

   • Google Scholar
10. 1. Bath DE
    2. Stowers JR
    3. Hörmann D
    4. Poehlmann A
    5. Dickson BJ
    6. Straw AD

    (2014) FlyMAD: rapid thermogenetic control of neuronal activity in freely walking Drosophila

    Nature Methods 11:756–762.

    https://doi.org/10.1038/nmeth.2973

    • PubMed
    • Google Scholar
11. Preprint

    1. Berman GJ
    2. Choi DM
    3. Bialek W
    4. Shaevitz JW

    (2014a) Mapping the structure of drosophilid behavior

    bioRxiv.

    https://doi.org/10.1101/002873

    • Google Scholar
12. 1. Berman GJ
    2. Choi DM
    3. Bialek W
    4. Shaevitz JW

    (2014b) Mapping the stereotyped behaviour of freely moving fruit flies

    Journal of the Royal Society Interface 11:20140672.

    https://doi.org/10.1098/rsif.2014.0672

    • Google Scholar
13. 1. Berman GJ
    2. Bialek W
    3. Shaevitz JW

    (2016) Predictability and hierarchy in Drosophila behavior,

    PNAS 11:948.

    https://doi.org/10.1101/052928

    • Google Scholar
14. 1. Berman GJ

    (2018) Measuring behavior across scales

    BMC Biology 16:23.

    https://doi.org/10.1186/s12915-018-0494-7

    • PubMed
    • Google Scholar
15. 1. Bierbach D
    2. Laskowski KL
    3. Wolf M

    (2017) Behavioural individuality in clonal fish arises despite near-identical rearing conditions

    Nature Communications 8:15361.

    https://doi.org/10.1038/ncomms15361

    • PubMed
    • Google Scholar
16. 1. Boenisch F
    2. Rosemann B
    3. Wild B
    4. Dormagen D
    5. Wario F
    6. Landgraf T

    (2018) Tracking all members of a honey bee colony over their lifetime using learned models of correspondence

    Frontiers in Robotics and AI 5:35.

    https://doi.org/10.3389/frobt.2018.00035

    • Google Scholar
17. 1. Brown AE
    2. Yemini EI
    3. Grundy LJ
    4. Jucikas T
    5. Schafer WR

    (2013) A dictionary of behavioral motifs reveals clusters of genes affecting Caenorhabditis elegans locomotion

    PNAS 110:791–796.

    https://doi.org/10.1073/pnas.1211447110

    • PubMed
    • Google Scholar
18. 1. Brown AEX
    2. de Bivort B

    (2018) Ethology as a physical science

    Nature Physics 14:653–657.

    https://doi.org/10.1038/s41567-018-0093-0

    • Google Scholar
19. 1. Cande J
    2. Namiki S
    3. Qiu J
    4. Korff W
    5. Card GM
    6. Shaevitz JW
    7. Stern DL
    8. Berman GJ

    (2018) Optogenetic dissection of descending behavioral control in Drosophila

    eLife 7:e34275.

    https://doi.org/10.7554/eLife.34275

    • PubMed
    • Google Scholar
20. Conference

    1. Cao Z
    2. Simon T
    3. Wei S-E
    4. Sheikh Y

    (2017) Realtime Multi-Person 2d pose estimation using part affinity fields,

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7291–7299.

    https://doi.org/10.1109/cvpr.2017.143

    • Google Scholar
21. 1. Carpenter B
    2. Lee D
    3. Brubaker MA
    4. Riddell A
    5. Gelman A
    6. Goodrich B
    7. Guo J
    8. Hoffman M
    9. Betancourt M
    10. Li P

    (2017)

    Stan: a probabilistic programming language

    Journal of Statistical Software 76:v076i01.

    • Google Scholar
22. 1. Cauchy A

    (1847) Méthode générale pour la résolution des systemes d’équations simultanées

    Comp Rend Sci Paris 25:536–538.

    https://doi.org/10.1017/cbo9780511702396.063

    • Google Scholar
23. Conference

    1. Chen Y
    2. Shen C
    3. Wei X-S
    4. Liu L
    5. Yang J

    (2017) Adversarial posenet: a structure-aware convolutional network for human pose estimation,

    Proceedings of the IEEE International Conference on Computer Vision. pp. 1221–1230.

    https://doi.org/10.1109/iccv.2017.137

    • Google Scholar
24. Conference

    1. Chollet F

    (2017) Xception: deep learning with depthwise separable convolutions

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1251–1258.

    https://doi.org/10.1109/cvpr.2017.195

    • Google Scholar
25. 1. Costa AC
    2. Ahamed T
    3. Stephens GJ

    (2019) Adaptive, locally linear models of complex dynamics

    PNAS 116:1501–1510.

    https://doi.org/10.1073/pnas.1813476116

    • Google Scholar
26. 1. Crall JD
    2. Gravish N
    3. Mountcastle AM
    4. Combes SA

    (2015) BEEtag: a Low-Cost, Image-Based tracking system for the study of animal behavior and locomotion

    PLOS ONE 10:e0136487.

    https://doi.org/10.1371/journal.pone.0136487

    • PubMed
    • Google Scholar
27. 1. Dell AI
    2. Bender JA
    3. Branson K
    4. Couzin ID
    5. de Polavieja GG
    6. Noldus LP
    7. Pérez-Escudero A
    8. Perona P
    9. Straw AD
    10. Wikelski M
    11. Brose U

    (2014) Automated image-based tracking and its application in ecology

    Trends in Ecology & Evolution 29:417–428.

    https://doi.org/10.1016/j.tree.2014.05.004

    • PubMed
    • Google Scholar
28. Conference

    1. Deng J
    2. Dong W
    3. Socher R
    4. Li K
    5. Fei-Fei L

    (2009) Imagenet: a large-scale hierarchical image database

    IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255.

    https://doi.org/10.1109/CVPR.2009.5206848

    • Google Scholar
29. 1. Doudna JA
    2. Charpentier E

    (2014) Genome editing. The new frontier of genome engineering with CRISPR-Cas9

    Science 346:1258096.

    https://doi.org/10.1126/science.1258096

    • PubMed
    • Google Scholar
30. 1. Duane S
    2. Kennedy AD
    3. Pendleton BJ
    4. Roweth D

    (1987) Hybrid monte carlo

    Physics Letters B 195:216–222.

    https://doi.org/10.1016/0370-2693(87)91197-X

    • Google Scholar
31. Conference

    1. Dugas C
    2. Bengio Y
    3. Bélisle F
    4. Nadeau C
    5. Garcia R

    (2001) Incorporating second-order functional knowledge for better option pricing

    Advances in Neural Information Processing Systems. pp. 472–478.

    http://papers.nips.cc/paper/2024-incorporating-invariances-in-non-linear-support-vector-machines

    • Google Scholar
32. 1. Flack A
    2. Nagy M
    3. Fiedler W
    4. Couzin ID
    5. Wikelski M

    (2018) From local collective behavior to global migratory patterns in white storks

    Science 360:911–914.

    https://doi.org/10.1126/science.aap7781

    • PubMed
    • Google Scholar
33. Preprint

    1. Francisco FA
    2. Nührenberg P
    3. Jordan AL

    (2019) A low-cost, open-source framework for tracking and behavioural analysis of animals in aquatic ecosystems

    bioRxiv.

    https://doi.org/10.1101/571232

    • Google Scholar
34. Conference

    1. Goodfellow I
    2. Pouget-Abadie J
    3. Mirza M
    4. Xu B
    5. Warde-Farley D
    6. Ozair S
    7. Courville A
    8. Bengio Y

    (2014) Generative adversarial nets

    Advances in Neural Information Processing Systems.

    http://papers.nips.cc/paper/5423-generative-adversarial-nets

    • Google Scholar
35. Book

    1. Goodfellow I
    2. Bengio Y
    3. Courville A

    (2016)

    Deep Learning

    MIT press.

    • Google Scholar
36. Software

    1. Graving JM

    (2017) Pinpoint: behavioral tracking using 2D barcode tags v0.0.1-alpha

    Zenodo.

    https://doi.org/10.5281/zenodo.3366908
37. Software

    1. Graving JM
    2. Chae D
    3. Naik H
    4. Li L
    5. Koger B
    6. Costelloe BR
    7. Couzin ID

    (2019) Example datasets for DeepPoseKit, version c6964d2

    Github.

    https://github.com/jgraving/deepposekit-data
38. 1. Guizar-Sicairos M
    2. Thurman ST
    3. Fienup JR

    (2008) Efficient subpixel image registration algorithms

    Optics Letters 33:156–158.

    https://doi.org/10.1364/OL.33.000156

    • PubMed
    • Google Scholar
39. 1. Günel S
    2. Rhodin H
    3. Morales D
    4. Campagnolo JH
    5. Ramdya P
    6. Fua P

    (2019) DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

    eLife 8:e48571.

    https://doi.org/10.7554/eLife.48571

    • PubMed
    • Google Scholar
40. Conference

    1. He K
    2. Zhang X
    3. Ren S
    4. Sun J

    (2016) Deep residual learning for image recognition

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778.

    https://doi.org/10.1109/cvpr.2016.90

    • Google Scholar
41. 1. Hoffman MD
    2. Gelman A

    (2014)

    The No-U-Turn sampler: adaptively setting path lengths in hamiltonian monte carlo

    Journal of Machine Learning Research 15:1593–1623.

    • Google Scholar
42. Conference

    1. Huang G
    2. Liu Z
    3. Van Der Maaten L
    4. Weinberger KQ

    (2017a) Densely connected convolutional networks

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4700–4708.

    https://doi.org/10.1109/cvpr.2017.243

    • Google Scholar
43. Conference

    1. Huang J
    2. Rathod V
    3. Sun C
    4. Zhu M
    5. Korattikara A
    6. Fathi A
    7. Fischer I
    8. Wojna Z
    9. Song Y
    10. Guadarrama S

    (2017b) Speed/accuracy trade-offs for modern convolutional object detectors

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7310–7311.

    https://doi.org/10.1109/cvpr.2017.351

    • Google Scholar
44. Conference

    1. Insafutdinov E
    2. Pishchulin L
    3. Andres B
    4. Andriluka M
    5. Schiele B

    (2016) Deepercut: a deeper, stronger, and faster multi-person poseestimation model

    European Conference on Computer Vision. pp. 34–50.

    https://doi.org/10.1007/978-3-319-46466-4_3

    • Google Scholar
45. Conference

    1. Iqbal U
    2. Milan A
    3. Gall J

    (2017) Posetrack: joint multi-person pose estimation and tracking

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2011–2020.

    https://doi.org/10.1109/cvpr.2017.495

    • Google Scholar
46. Preprint

    1. Jaques M
    2. Burke M
    3. Hospedales T

    (2019) Physics-as-inverse-graphics: joint unsupervised learning of objects and physics from video

    arXiv.

    https://arxiv.org/abs/1905.11169

    • Google Scholar
47. 1. Javer A
    2. Currie M
    3. Lee CW
    4. Hokanson J
    5. Li K
    6. Martineau CN
    7. Yemini E
    8. Grundy LJ
    9. Li C
    10. Ch'ng Q
    11. Schafer WR
    12. Nollen EAA
    13. Kerr R
    14. Brown AEX

    (2018) An open-source platform for analyzing and sharing worm-behavior data

    Nature Methods 15:645–646.

    https://doi.org/10.1038/s41592-018-0112-1

    • PubMed
    • Google Scholar
48. Conference

    1. Jégou S
    2. Drozdzal M
    3. Vázquez D
    4. Romero A
    5. Bengio Y

    (2017)

    The one hundred layers tiramisu: fully convolutional densenets for semantic segmentation

    IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 1175–1183.

    • Google Scholar
49. Conference

    1. Johnson J
    2. Alahi A
    3. Fei-Fei L

    (2016a) Perceptual losses for real-time style transfer and super-resolution

    European Conference on Computer Vision. 694–711.

    https://doi.org/10.1007/978-3-319-46475-6_43

    • Google Scholar
50. Conference

    1. Johnson M
    2. Duvenaud DK
    3. Wiltschko A
    4. Adams RP
    5. Datta SR

    (2016b) Composing graphical models with neural networks for structured representations and fast inference

    Advances in Neural Information Processing Systems. 2946–2954.

    https://papers.nips.cc/paper/6379-composing-graphical-models-with-neural-networks-for-structured-representations-and-fast-inference

    • Google Scholar
51. 1. Jolles JW
    2. Boogert NJ
    3. Sridhar VH
    4. Couzin ID
    5. Manica A

    (2017) Consistent individual differences drive collective behavior and group functioning of schooling fish

    Current Biology 27:2862–2868.

    https://doi.org/10.1016/j.cub.2017.08.004

    • PubMed
    • Google Scholar
52. 1. Jung A

    (2018) imgaug

    imgaug, https://github.com/aleju/imgaug.

    https://github.com/aleju/imgaug

    • Google Scholar
53. 1. Kain JS
    2. Stokes C
    3. de Bivort BL

    (2012) Phototactic personality in fruit flies and its suppression by serotonin and white

    PNAS 109:19834–19839.

    https://doi.org/10.1073/pnas.1211988109

    • PubMed
    • Google Scholar
54. 1. Kain J
    2. Stokes C
    3. Gaudry Q
    4. Song X
    5. Foley J
    6. Wilson R
    7. de Bivort B

    (2013) Leg-tracking and automated behavioural classification in Drosophila

    Nature Communications 4:1910.

    https://doi.org/10.1038/ncomms2908

    • PubMed
    • Google Scholar
55. 1. Kays R
    2. Crofoot MC
    3. Jetz W
    4. Wikelski M

    (2015) ECOLOGY. Terrestrial animal tracking as an eye on life and planet

    Science 348:aaa2478.

    https://doi.org/10.1126/science.aaa2478

    • PubMed
    • Google Scholar
56. Conference

    1. Ke L
    2. Chang M-C
    3. Qi H
    4. Lyu S

    (2018) Multi-scale structure-aware network for human pose estimation

    The European Conference on Computer Vision (ECCV). pp. 731–746.

    https://doi.org/10.1007/978-3-030-01216-8_44

    • Google Scholar
57. Conference

    1. Kendall A
    2. Gal Y

    (2017) What uncertainties do we need in bayesian deep learning for computer vision?

    Advances in Neural Information Processing Systems. pp. 5574–5584.

    https://papers.nips.cc/paper/7141-what-uncertainties-do-we-need-in-bayesian-deep-learning-for-computer-vision

    • Google Scholar
58. 1. keras team

    (2015) Keras

    Keras, https://github.com/fchollet/keras.

    https://github.com/fchollet/keras

    • Google Scholar
59. 1. Kiefer J
    2. Wolfowitz J

    (1952) Stochastic estimation of the maximum of a regression function

    The Annals of Mathematical Statistics 23:462–466.

    https://doi.org/10.1214/aoms/1177729392

    • Google Scholar
60. Preprint

    1. Kingma DP
    2. Ba J

    (2014) Adam: a method for stochastic optimization

    arXiv.

    https://arxiv.org/abs/1412.6980

    • Google Scholar
61. Conference

    1. Klambauer G
    2. Unterthiner T
    3. Mayr A
    4. Hochreiter S

    (2017) Self-normalizing neural networks

    Advances in Neural Information Processing Systems. 971–980.

    http://papers.nips.cc/paper/6698-self-normalizing-neural-networks

    • Google Scholar
62. 1. Klibaite U
    2. Berman GJ
    3. Cande J
    4. Stern DL
    5. Shaevitz JW

    (2017) An unsupervised method for quantifying the behavior of paired animals

    Physical Biology 14:015006.

    https://doi.org/10.1088/1478-3975/aa5c50

    • PubMed
    • Google Scholar
63. Preprint

    1. Klibaite U
    2. Shaevitz JW

    (2019) Interacting fruit flies synchronize behavior

    bioRxiv.

    https://doi.org/10.1101/545483

    • Google Scholar
64. 1. Krakauer JW
    2. Ghazanfar AA
    3. Gomez-Marin A
    4. MacIver MA
    5. Poeppel D

    (2017) Neuroscience needs behavior: correcting a reductionist Bias

    Neuron 93:480–490.

    https://doi.org/10.1016/j.neuron.2016.12.041

    • PubMed
    • Google Scholar
65. Book

    1. Kuhn M
    2. Johnson K

    (2013) Applied Predictive Modeling

    Springer.

    https://doi.org/10.1007/978-1-4614-6849-3

    • Google Scholar
66. Conference

    1. Kulkarni TD
    2. Whitney WF
    3. Kohli P
    4. Tenenbaum J

    (2015) Deep convolutional inverse graphics network

    Advances in Neural Information Processing Systems. pp. 2539–2547.

    https://papers.nips.cc/paper/5851-deep-convolutional-inverse-graphics-network

    • Google Scholar
67. Preprint

    1. Kumar M
    2. Babaeizadeh M
    3. Erhan D
    4. Finn C
    5. Levine S
    6. Dinh L
    7. Kingma D

    (2019) Videoflow: a flow-based generative model for video

    arXiv.

    https://arxiv.org/abs/1903.01434

    • Google Scholar
68. 1. LeCun Y
    2. Bengio Y
    3. Hinton G

    (2015) Deep learning

    Nature 521:436–444.

    https://doi.org/10.1038/nature14539

    • PubMed
    • Google Scholar
69. Conference

    1. Li H
    2. Xu Z
    3. Taylor G
    4. Studer C
    5. Goldstein T

    (2018) Visualizing the loss landscape of neural nets

    Advances in Neural Information Processing Systems. pp. 6391–6401.

    http://papers.nips.cc/paper/7875-visualizing-the-loss-landscape-of-neural-nets

    • Google Scholar
70. Conference

    1. Long J
    2. Shelhamer E
    3. Darrell T

    (2015) Fully convolutional networks for semantic segmentation,

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3431–3440.

    https://doi.org/10.1109/cvpr.2015.7298965

    • Google Scholar
71. 1. Markowitz JE
    2. Gillis WF
    3. Beron CC
    4. Neufeld SQ
    5. Robertson K
    6. Bhagat ND
    7. Peterson RE
    8. Peterson E
    9. Hyun M
    10. Linderman SW
    11. Sabatini BL
    12. Datta SR

    (2018) The striatum organizes 3D behavior via Moment-to-Moment action selection

    Cell 174:44–58.

    https://doi.org/10.1016/j.cell.2018.04.019

    • PubMed
    • Google Scholar
72. 1. Mathis A
    2. Mamidanna P
    3. Cury KM
    4. Abe T
    5. Murthy VN
    6. Mathis MW
    7. Bethge M

    (2018) DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

    Nature Neuroscience 21:1281–1289.

    https://doi.org/10.1038/s41593-018-0209-y

    • Google Scholar
73. Preprint

    1. Mathis A
    2. Warren R

    (2018) On the inference speed and video-compression robustness of DeepLabCut

    bioRxiv.

    https://doi.org/10.1101/457242

    • Google Scholar
74. 1. Mendes CS
    2. Bartos I
    3. Akay T
    4. Márka S
    5. Mann RS

    (2013) Quantification of gait parameters in freely walking wild type and sensory deprived Drosophila Melanogaster

    eLife 2:e00231.

    https://doi.org/10.7554/eLife.00231

    • PubMed
    • Google Scholar
75. 1. Munkres J

    (1957) Algorithms for the assignment and transportation problems

    Journal of the Society for Industrial and Applied Mathematics 5:32–38.

    https://doi.org/10.1137/0105003

    • Google Scholar
76. 1. Nagy M
    2. Akos Z
    3. Biro D
    4. Vicsek T

    (2010) Hierarchical group dynamics in pigeon flocks

    Nature 464:890–893.

    https://doi.org/10.1038/nature08891

    • PubMed
    • Google Scholar
77. 1. Nagy M
    2. Vásárhelyi G
    3. Pettit B
    4. Roberts-Mariani I
    5. Vicsek T
    6. Biro D

    (2013) Context-dependent hierarchies in pigeons

    PNAS 110:13049–13054.

    https://doi.org/10.1073/pnas.1305552110

    • PubMed
    • Google Scholar
78. 1. Nath T
    2. Mathis A
    3. Chen AC
    4. Patel A
    5. Bethge M
    6. Mathis MW

    (2019) Using DeepLabCut for 3D markerless pose estimation across species and behaviors

    Nature Protocols 14:2152–2176.

    https://doi.org/10.1038/s41596-019-0176-0

    • PubMed
    • Google Scholar
79. Conference

    1. Newell A
    2. Yang K
    3. Deng J

    (2016) Stacked hourglass networks for human pose estimation

    European Conference on Computer Vision. pp. 483–499.

    https://doi.org/10.1007/978-3-319-46484-8_29

    • Google Scholar
80. 1. Pereira TD
    2. Aldarondo DE
    3. Willmore L
    4. Kislin M
    5. Wang SS
    6. Murthy M
    7. Shaevitz JW

    (2019) Fast animal pose estimation using deep neural networks

    Nature Methods 16:117–125.

    https://doi.org/10.1038/s41592-018-0234-5

    • PubMed
    • Google Scholar
81. 1. Pérez-Escudero A
    2. Vicente-Page J
    3. Hinz RC
    4. Arganda S
    5. de Polavieja GG

    (2014) idTracker: tracking individuals in a group by automatic identification of unmarked animals

    Nature Methods 11:743–748.

    https://doi.org/10.1038/nmeth.2994

    • PubMed
    • Google Scholar
82. Conference

    1. Pratt LY

    (1992) Discriminability-based transfer between neural networks

    Advances in Neural Information Processing Systems. pp. 204–211.

    https://papers.nips.cc/paper/641-discriminability-based-transfer-between-neural-networks

    • Google Scholar
83. 1. Prechelt L

    (1998) Automatic early stopping using cross validation: quantifying the criteria

    Neural Networks 11:761–767.

    https://doi.org/10.1016/S0893-6080(98)00010-0

    • PubMed
    • Google Scholar
84. 1. Price E
    2. Lawless G
    3. Ludwig R
    4. Martinovic I
    5. Bulthoff HH
    6. Black MJ
    7. Ahmad A

    (2018) Deep neural Network-Based cooperative visual tracking through multiple micro aerial vehicles

    IEEE Robotics and Automation Letters 3:3193–3200.

    https://doi.org/10.1109/LRA.2018.2850224

    • Google Scholar
85. 1. Ran FA
    2. Hsu PD
    3. Wright J
    4. Agarwala V
    5. Scott DA
    6. Zhang F

    (2013) Genome engineering using the CRISPR-Cas9 system

    Nature Protocols 8:2281–2308.

    https://doi.org/10.1038/nprot.2013.143

    • PubMed
    • Google Scholar
86. Conference

    1. Ren S
    2. He K
    3. Girshick R
    4. Sun J

    (2015) Faster R-CNN: towards real-time object detection with region proposal networks

    Advances in Neural Information Processing Systems. pp. 91–99.

    https://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks

    • Google Scholar
87. 1. Robbins H
    2. Monro S

    (1951) A stochastic approximation method

    The Annals of Mathematical Statistics 22:400–407.

    https://doi.org/10.1214/aoms/1177729586

    • Google Scholar
88. 1. Romero-Ferrero F
    2. Bergomi MG
    3. Hinz RC
    4. Heras FJH
    5. de Polavieja GG

    (2019) Idtracker.ai: tracking all individuals in small or large collectives of unmarked animals

    Nature Methods 16:179–182.

    https://doi.org/10.1038/s41592-018-0295-5

    • PubMed
    • Google Scholar
89. Conference

    1. Ronneberger O
    2. Fischer P
    3. Brox T

    (2015) U-net: convolutional networks for biomedical image segmentation

    International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 234–241.

    https://doi.org/10.1007/978-3-319-24574-4_28

    • Google Scholar
90. 1. Rosenthal SB
    2. Twomey CR
    3. Hartnett AT
    4. Wu HS
    5. Couzin ID

    (2015) Revealing the hidden networks of interaction in mobile animal groups allows prediction of complex behavioral contagion

    PNAS 112:4690–4695.

    https://doi.org/10.1073/pnas.1420068112

    • PubMed
    • Google Scholar
91. 1. Roy AG
    2. Conjeti S
    3. Navab N
    4. Wachinger C

    (2019) Bayesian QuickNAT: model uncertainty in deep whole-brain segmentation for structure-wise quality control

    NeuroImage 195:11–22.

    https://doi.org/10.1016/j.neuroimage.2019.03.042

    • Google Scholar
92. Conference

    1. Sabour S
    2. Frosst N
    3. Hinton GE

    (2017) Dynamic routing between capsules

    Advances in Neural Information Processing Systems. pp. 3856–3866.

    https://papers.nips.cc/paper/6975-dynamic-routing-between-capsules

    • Google Scholar
93. Conference

    1. Saini N
    2. Price E
    3. Tallamraju R
    4. Enficiaud R
    5. Ludwig R
    6. Martinovia I
    7. Ahmad A
    8. Black M

    (2019)

    Markerless outdoor human motion capture using multiple autonomous micro aerial vehicles

    In: International Conference on Computer Vision.

    • Google Scholar
94. Conference

    1. Sandler M
    2. Howard A
    3. Zhu M
    4. Zhmoginov A
    5. Chen L-C

    (2018) Mobilenetv2: inverted residuals and linear bottlenecks,

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4510–4520.

    https://doi.org/10.1109/cvpr.2018.00474

    • Google Scholar
95. 1. Schiffman R

    (2014) Drones flying high as new tool for field biologists

    Science 344:459–460.

    https://doi.org/10.1126/science.344.6183.459

    • Google Scholar
96. Preprint

    1. Seethapathi N
    2. Wang S
    3. Saluja R
    4. Blohm G
    5. Kording KP

    (2019) Movement science needs different pose tracking algorithms

    arXiv.

    https://arxiv.org/abs/1907.10226

    • Google Scholar
97. 1. Stephens GJ
    2. Bueno de Mesquita M
    3. Ryu WS
    4. Bialek W

    (2011) Emergence of long timescales and stereotyped behaviors in Caenorhabditis elegans

    PNAS 108:7286–7289.

    https://doi.org/10.1073/pnas.1007868108

    • PubMed
    • Google Scholar
98. 1. Stowers JR
    2. Hofbauer M
    3. Bastien R
    4. Griessner J
    5. Higgins P
    6. Farooqui S
    7. Fischer RM
    8. Nowikovsky K
    9. Haubensak W
    10. Couzin ID
    11. Tessmar-Raible K
    12. Straw AD

    (2017) Virtual reality for freely moving animals

    Nature Methods 14:995–1002.

    https://doi.org/10.1038/nmeth.4399

    • PubMed
    • Google Scholar
99. 1. Strandburg-Peshkin A
    2. Twomey CR
    3. Bode NW
    4. Kao AB
    5. Katz Y
    6. Ioannou CC
    7. Rosenthal SB
    8. Torney CJ
    9. Wu HS
    10. Levin SA
    11. Couzin ID

    (2013) Visual sensory networks and effective information transfer in animal groups

    Current Biology 23:R709–R711.

    https://doi.org/10.1016/j.cub.2013.07.059

    • PubMed
    • Google Scholar
100. 1. Strandburg-Peshkin A
     2. Farine DR
     3. Couzin ID
     4. Crofoot MC

     (2015) GROUP DECISIONS. Shared decision-making drives collective movement in wild baboons

     Science 348:1358–1361.

     https://doi.org/10.1126/science.aaa5099

     • PubMed
     • Google Scholar
101. 1. Strandburg-Peshkin A
     2. Farine DR
     3. Crofoot MC
     4. Couzin ID

     (2017) Habitat and social factors shape individual decisions and emergent group structure during baboon collective movement

     eLife 6:e19505.

     https://doi.org/10.7554/eLife.19505

     • PubMed
     • Google Scholar
102. 1. Todd JG
     2. Kain JS
     3. de Bivort BL

     (2017) Systematic exploration of unsupervised methods for mapping behavior

     Physical Biology 14:015002.

     https://doi.org/10.1088/1478-3975/14/1/015002

     • PubMed
     • Google Scholar
103. Conference

     1. Tran D
     2. Hoffman MW
     3. Moore D
     4. Suter C
     5. Vasudevan S
     6. Radul A

     (2018) Simple, distributed, and accelerated probabilistic programming.

     Advances in Neural Information Processing Systems.

     https://papers.nips.cc/paper/7987-simple-distributed-and-accelerated-probabilistic-programming

     • Google Scholar
104. 1. Uhlmann V
     2. Ramdya P
     3. Delgado-Gonzalo R
     4. Benton R
     5. Unser M

     (2017) FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila

     PLOS ONE 12:e0173433.

     https://doi.org/10.1371/journal.pone.0173433

     • PubMed
     • Google Scholar
105. 1. Valentin J
     2. Keskin C
     3. Pidlypenskyi P
     4. Makadia A
     5. Sud A
     6. Bouaziz S

     (2019) Tensorflow graphics: Computer graphics meets deep learning

     Tensorflow graphics: Computer graphics meets deep learning, https://github.com/tensorflow/graphics.

     https://github.com/tensorflow/graphics

     • Google Scholar
106. Preprint

     1. Van den Oord A
     2. Dieleman S
     3. Zen H
     4. Simonyan K
     5. Vinyals O
     6. Graves A
     7. Kalchbrenner N
     8. Senior A
     9. Kavukcuoglu K

     (2016a) Wavenet: a generativemodel for raw audio

     arXiv.

     https://arxiv.org/abs/1609.03499

     • Google Scholar
107. Conference

     1. Van den Oord A
     2. Kalchbrenner N
     3. Espeholt L
     4. Vinyals O
     5. Graves A

     (2016b) Conditional image generation with pixelcnn decoders

     Advances in Neural Information Processing Systems. pp. 4790–4798.

     https://papers.nips.cc/paper/6527-conditional-image-generation-with-pixelcnn-decoders

     • Google Scholar
108. Preprint

     1. Versace E
     2. Caffini M
     3. Werkhoven Z
     4. de Bivort BL

     (2019) Individual, but not population asymmetries, are modulated by social environment and genotype in Drosophila Melanogaster

     bioRxiv.

     https://doi.org/10.1101/694901

     • Google Scholar
109. 1. Weigert M
     2. Schmidt U
     3. Boothe T
     4. Müller A
     5. Dibrov A
     6. Jain A
     7. Wilhelm B
     8. Schmidt D
     9. Broaddus C
     10. Culley S
     11. Rocha-Martins M
     12. Segovia-Miranda F
     13. Norden C
     14. Henriques R
     15. Zerial M
     16. Solimena M
     17. Rink J
     18. Tomancak P
     19. Royer L
     20. Jug F
     21. Myers EW

     (2018) Content-aware image restoration: pushing the limits of fluorescence microscopy

     Nature Methods 15:1090–1097.

     https://doi.org/10.1038/s41592-018-0216-7

     • PubMed
     • Google Scholar
110. Preprint

     1. Werkhoven Z
     2. Rohrsen C
     3. Qin C
     4. Brembs B
     5. de Bivort B

     (2019) MARGO (Massively automated Real-time GUI for Object-tracking), aplatform for high-throughput ethology

     bioRxiv.

     https://doi.org/10.1101/593046

     • Google Scholar
111. Preprint

     1. Wild B
     2. Sixt L
     3. Landgraf T

     (2018) Automatic localization and decoding of honeybee markers using deep convolutional neural networks

     arXiv.

     https://arxiv.org/abs/1802.04557

     • Google Scholar
112. 1. Wiltschko AB
     2. Johnson MJ
     3. Iurilli G
     4. Peterson RE
     5. Katon JM
     6. Pashkovski SL
     7. Abraira VE
     8. Adams RP
     9. Datta SR

     (2015) Mapping Sub-Second Structure in Mouse Behavior

     Neuron 88:1121–1135.

     https://doi.org/10.1016/j.neuron.2015.11.031

     • PubMed
     • Google Scholar
113. Conference

     1. Zhang R
     2. Isola P
     3. Efros AA
     4. Shechtman E
     5. Wang O

     (2018) The unreasonable effectiveness of deep features as a perceptual metric

     Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 586–595.

     https://doi.org/10.1109/cvpr.2018.00068

     • Google Scholar
114. Conference

     1. Zuffi S
     2. Kanazawa A
     3. Jacobs DW
     4. Black MJ

     (2017) 3D menagerie: modeling the 3D shape and pose of animals,

     Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6365–6373.

     https://doi.org/10.1109/cvpr.2017.586

     • Google Scholar
115. Conference

     1. Zuffi S
     2. Kanazawa A
     3. Berger-Wolf T
     4. Black MJ

     (2019)

     Three-D safari: learning to estimate zebra pose, shape, and texture from images" In the Wild

     International Conference on Computer Vision.

     • Google Scholar

Author details

1. Jacob M Graving

   1. Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
   2. Department of Biology, University of Konstanz, Konstanz, Germany
   3. Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   jgraving@gmail.com

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5826-467X

2. Daniel Chae

   Department of Computer Science, Princeton University, Princeton, United States

   Contribution

   Software, Formal analysis, Methodology, Writing—original draft

   Competing interests

   No competing interests declared

3. Hemal Naik

   1. Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
   2. Department of Biology, University of Konstanz, Konstanz, Germany
   3. Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany
   4. Chair for Computer Aided Medical Procedures, Technische Universität München, Munich, Germany

   Contribution

   Conceptualization, Methodology, Writing—review and editing

   Competing interests

   No competing interests declared

4. Liang Li

   1. Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
   2. Department of Biology, University of Konstanz, Konstanz, Germany
   3. Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany

   Contribution

   Data curation, Validation, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-2447-6295

5. Benjamin Koger

   1. Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
   2. Department of Biology, University of Konstanz, Konstanz, Germany
   3. Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany

   Contribution

   Data curation, Investigation, Writing—review and editing

   Competing interests

   No competing interests declared

6. Blair R Costelloe

   1. Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
   2. Department of Biology, University of Konstanz, Konstanz, Germany
   3. Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany

   Contribution

   Data curation, Supervision, Project administration, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5291-788X

7. Iain D Couzin

   1. Department of Collective Behaviour, Max Planck Institute of Animal Behavior, Konstanz, Germany
   2. Department of Biology, University of Konstanz, Konstanz, Germany
   3. Centre for the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz, Germany

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Project administration, Writing—review and editing

   For correspondence

   icouzin@ab.mpg.de

   Competing interests

   Reviewing editor, eLife

   "This ORCID iD identifies the author of this article:" 0000-0001-8556-4558

• 24,679

  views

• 2,392

  downloads

• 391

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.