Understanding how the brain encodes behaviour is the ultimate goal of neuroscience and the ability to objectively and reproducibly describe and quantify behaviour is a necessary milestone on this path. Recent technological progresses in machine learning and computational power have boosted the development and adoption of systems leveraging on high-resolution video recording to track an animal pose and describe behaviour in all four dimensions. However, the high temporal and spatial resolution that these systems offer must come as a compromise with their throughput and accessibility. Here we describe coccinella, an open-source reductionist framework combining high-throughput analysis of behaviour using real-time tracking on a distributed mesh of microcomputers (ethoscopes) with resource-lean statistical learning (HCTSA/Catch22). Coccinella is a reductionist system, yet outperforms state-of-the-art alternatives when exploring the pharmacobehaviour in Drosophila melanogaster.
This study presents an important open-source resource for high-throughput behavioral screening. The protocols employ inexpensive, off the shelf hardware, and allow real-time analysis of hundreds of behaving flies. Although these protocols were developed using Drosophila melanogaster, they could easily be applied to other models. The evidence in support of the conclusions is solid and the revisions carried out by the authors go a long way towards providing the user with an integrated system that is also more user-friendly.
The nervous system integrates stimuli, internal states, expectations, and previous experience to regulate behavioural output. Describing, quantifying, and modulating behaviour are critical aspects of modern neuroscience and, ever since its inception, the field has spent considerable effort into building and sharing paradigms or tools aimed at objectively and reproducibly quantify behaviours in the most disparate animal models, to the point that this exercise is now recognised as an exciting subfield of neuroscience in its own right: ethomics1,2. As the portmanteau name itself suggests, ethomics is not just about describing behaviour (etho-) but also about doing so in a high-throughput fashion (-omics), collecting data simultaneously from a large number of individuals, which can remain undisturbed throughout recording. Irrespective of the behaviour or the animal model to be analysed, the first compromise a researcher will face when choosing a tool for behavioural quantification will always be between throughput and resolution: a high-throughput analysis will allow for powerful experimental manipulations – such as genetics or pharmacological screens – offering unbiased approaches in identifying neuronal circuits, genes, molecules underpinning behaviour; high-resolution analysis, on the other hand, promises to identify and discriminate even minuscule differences that may not be immediately visible to the human eye, and to label behaviours into identifiable classes (e.g.: “grooming”, “courting”, “shaking”) that may be more relevant to researchers interested in modelling disease or in anthropomorphic descriptions. In the past years, the field has generally converged towards the adoption of high-resolution video recording of activity, in some cases adopting cameras that have milliseconds temporal resolution or developing setups that provide depth information for three-dimensional reconstruction of motion or posture3–7. Given the recent evolution in machine learning and progresses in computational power, even these high-resolution analyses can be at least in part compatible with high-throughput approaches8, especially when employed on small invertebrate animal models9–12 or when aided by robotic handling13. These systems, however, can still be prohibitively expensive for most laboratories, and not easily compatible with throughput in the ‘omics scale. Moreover, besides the technical urge of removing entry barrier and make ethomics an accessible tool, an equally important underlying question concerns what is the minimal amount of information that needs to be extracted to identify and classify behaviour. Do we always necessarily gain information from extracting micro-postural features or by analysing activity in three dimensions? To what extent this may actually add counterproductive biological noise to some assays?
Here we introduce coccinella: a new experimental framework that combines high-throughput, inexpensive, real-time ethomics14 with state-of-the-art statistical analysis15,16 to characterise and discriminate complex behaviours using a reductionist approach based solely on one simple feature (https://lab.gilest.ro/coccinella). Coccinella builds on ethoscopes14, an accessible open-source platform, to extract, in real-time, minimalist information from flies. Despite its minimalist nature, coccinella outperforms state-of-the-art alternatives in recognising the pharmacobehavioural space, providing better discernibility at a fraction of the cost, thus opening a new path to high-throughput ethomics.
Drosophila ethomics studies generally rely on image acquisition through so called industrial-cameras, able to collect videos with high temporal and spatial resolution and featuring mounts for a large selection of lenses. Some of the cameras commonly used for these purposes (e.g.: FLIR, Point Grey, Basler)17 are expensive and normally employed in close-up imaging that microscopically highlights the smallest anatomical features of the animal but at the same time greatly limits the number of experimental subjects that can be recorded by a single device. Normally, one or few more camera would be connected to a dedicated powerful computer for acquisition and storage of videos. The cost and physical footprint of these setups makes them incompatible, at least for most laboratories, with high-throughput simultaneous acquisition. To lower this barrier, we created a framework that employs the distributing computing power of ethoscopes14, thus allowing for inexpensive analysis of activity in hundreds or thousands of flies at once. Ethoscopes are open source and can be manufactured by a skilled end-user at a cost of about £75 per machine, mostly building on two off-the-shelf component: a Raspberry Pi microcomputer and a Raspberry Pi NoIR camera overlooking a bespoke 3D printed arena hosting freely moving flies. The temporal and spatial resolution of the collected images depends on the working modality the user chooses. When operating in offline mode, ethoscopes are capable to acquire 720p videos at 60 fps, which is a convenient option with fast moving animals. In this study, we instead opted for the default ethoscope working settings, providing online tracking and real-time parametric extraction, meaning that images are analysed by each raspberry Pi at the very moment they are acquired (Figure 1b). This latter modality limits the temporal resolution of information being processed (one frame every 444 ms ± 127 ms, equivalent to 2.2 fps on a Raspberry Pi3 at a resolution of 1280x960 pixels with each animal being constricted in an ellipse measuring 25.8 ± 1.4 x 9.85 ± 1.4 pixels – Figure 1a) but provides the most affordable and high-throughput solution, dispensing the researcher from organising video storage or asynchronous video processing for tracking the animals. In the work here described, flies moved freely in a circular two-dimensional space with a diameter of 11.5 mm designed to maintain the animal in the walking position18 while venturing on solidified agar providing nutrients alone or nutrients and drugs. In previous analysis of activity and sleep14,19, we found that the maximal velocity of the fly over a period of 10 seconds best described the basic motion features of the animal, allowing us to accurately differentiate between different activity patterns, such as walking, grooming, feeding14. We therefore adopted this measure for coccinella too, ultimately producing monodimensional time-series of a behavioural correlate, which were then digested using highly comparative time-series analysis (HCTSA)15, a computational framework that effortlessly subjects time-series to more than 7700 literature-relevant statistical tests, looking for meaningful discriminative features. Features successfully extracted through HCTSA were then used to classify behaviour using a linear support vector machine (SVMlinear)20 (Figure 1b) and here presented and compared using confusion matrices (Figure 1 - Figure Supplement 1a). To test the ability of this system to discriminate behaviour, we started by exploring the pharmacobehavioral space of flies fed with a panel of known or putative neurotropic chemicals, comprising molecules previously described in the literature along with uncharacterised ones being considered as potential insecticides (Figure 1c, 1d and Supplementary Table 1). Using an initial panel of 17 treatments (16 drugs and one solvent control) we were able to discern compounds with an accuracy of 71.4% (vs. 5.8% of a random classifier – Figure 1c). Some compounds induced behaviours with a particularly high predictive fingerprint: dieldrin, for instance, was predicted with an accuracy of 94% and flonicamid with an accuracy of 87%. For others, our system fared more poorly (e.g. tetramethrin showed 41% accuracy). In all cases, however, the relative confusion was negligible, with all compounds being correctly identified as the first choice and with the first predicted compounds having, on average, a score that was 15-times greater compared to the second-best choices (Figure 1c – min.: 3.3x; max.: 65x). To validate our framework and exclude artifacts operated by overfitting biologically irrelevant information we followed two lines of control. Firstly, we fed flies with lower concentrations of the same compounds (Figure 1c). Feeding flies with different concentrations of drugs unsurprisingly showed a different effect on short term lethality (Figure 1 - Figure Supplement 1b), with several compounds hitting a 25% lethality rate before the end of the experiment when fed at the highest concentration (1000ppm – Figure 1 - Figure Supplement 1b). Lowering the compound concentration, the predictive accuracy of the system decreased from 71.4% (1000ppm) to 61.8% (100ppm), falling to 36.1% with the lowest concentrations (1ppm), indicating that the system does operate on pharmacologically induced, biologically meaningful behavioural correlates (Figure 1c). A similar drop in accuracy was observed using a smaller panel of 12 treatments (Figure 1 - Figure Supplement 1c). As a second line of work to test specificity, we obtained genetic mutants known to be resistant to specific pharmacological treatments: the paraL1029F allele encodes for a version of the α-subunit of voltage-gated sodium channel conferring resistance to DDT and pyrethroids21; the RdlA301Sallele encodes for a version of the ligand-gated chloride channel conferring resistance to dieldrin and fiproles22. Challenging these mutants with their respective compounds created confusion in the clustering algorithm for which discerning between drug treatment and solvent control became a harder task, especially in the case of DDT and paraL1029F (Figure 1 - Figure Supplement 2a). The observed drop in accuracy suggests again that coccinella is working on biologically relevant behavioural signatures and the fact that some discrimination can still be observed with targets harbouring point mutations – which should severely affect compound efficacy – is indicative of high sensitivity.
Having established the accuracy and sensitivity of the system, we next wanted to test its usefulness in a genuine high-throughput scenario. We subjected a total of 2192 flies to a panel of 40 treatments (Figure 1d), mostly featuring known compounds but also two unexplored molecules (Supplementary Table 1). Given that the 100ppm intermediate concentrations showed the best compromise between accuracy and lethality in the previous pilot experiment (Figure 1 - Figure Supplement 1b), we performed this larger screen using compounds diluted at 100ppm only. Even with such a large panel, the system was able to first-guess 39 out of 40 of the tested treatments (the only exception being the Syngenta Compound #5) with an overall accuracy of 44.5% vs 2.5% of the random classifier.
A reductionist approach served us well so far, identifying with remarkable accuracy even subtle changes when we explored the pharmacobehavioural space of a large number of neuroactive compounds in wild type and mutant flies. But how does it compare to other more established paradigms? Coccinella is arguably to be preferred in terms of accessibility and throughput, but what is the amount of useful information that we are sacrificing by adopting a reductionist approach? To quantify any possible loss in information content, we ran a series of parallel experiments in which flies were fed with a selected panel of 12 treatments (11 drugs and a solvent control, Figure 2) and their behaviour analysed either using coccinella or using other widely adopted state-of-the-art methods, which started with high-resolution imaging and employed supervised machine learning for pose estimation (DeepLabCut17) followed by unsupervised identification of behavioural grammar (B-SOiD4). To widen the range of comparisons, data were then either immediately clustered using different common clustering algorithms (K-nearest neighbours, random forest) or first processed through a smaller, selected subsample of the HCTSA array (Catch2216) before being clustered using SVMlinear (Figure 2a). In this challenge, coccinella unambiguously identified 10 out of 12 compounds, with poor performance only for two of them (flubendiamide and tetramethrin) and an overall accuracy of 42.4%, vs 8.3% of the random classifier (Figure 2b). Surprisingly, none of the state-of-the-art high-resolution paths did better than this. The combination of pose-estimation → grammar extraction → random forest classification scored as the second best, with an accuracy of 25.4% but with only three compounds being unambiguously identified (Figure 2c). The same experimental dataset clustered with even poorer performance when using K-nearest neighbours (Figure 2e) and even the application of HCTSA features extraction to the B-SoID output still could not match the accuracy observed with coccinella’s reductionist approach (Figure 2d). This analysis is not meant to be conclusive. We expect that some alternative combination of state-of-the-art approaches will probably manage to match or likely improve over coccinella’s performance, yet the fact we could obtain such an impressive result with a system that is arguably unmatched in terms of throughput and economic cost is, alone, an argument that gives new weight to this (and future) reductionist approaches.
Finally, to push the system to its limit, we asked coccinella to find qualitative differences not in pharmacologically induced changes in activity, but in a type of spontaneous behaviour mostly characterised by lack of movement: sleep. In particular, we wondered whether coccinella could provide biological insights comparing conditions of sleep rebound observed after different regimes of sleep deprivation. Drosophila melanogaster is known to show a strong, conserved homeostatic regulation of sleep that forces flies to recover at least in part lost sleep, for instance after a night of forceful sleep deprivation23,24. We previously showed that the extent of sleep rebound observed after sleep deprivation only loosely correlates with the amount of lost sleep25 and it remains an open question whether similar amounts of sleep rebound may in fact differ from each other in some inscrutable feature that would underpin a different “sleep depth”26,27, similarly to what it is believed to happen in mammals. Here, we analysed a dataset of 727 flies that experienced different regimes of mechanically enforced sleep deprivation during the 12 hours of the night (Figure 3). Flies were housed in tubes that would rotate after a set time of inactivity ranging from 20 to 1000 seconds leading to different degrees of sleep restriction (Figure 3a, dataset from25). In this experimental paradigm, all treatments led to a statistically significant rebound compared to the undisturbed control animals (Figure 3b). We then ran coccinella on the two subsets of the panel: the baseline data, acquired the morning before the sleep deprivation (Figure 3c), and the rebound data, on the morning after (Figure 3d). Unsurprisingly, we could not detect any internal biological difference in the pre sleep deprivation control set, featuring flies of identical genotype and age housed in different tubes before the sleep deprivation treatment. In these conditions, coccinella could not discern, and performed exactly as a random classifier would (9% vs 9% – Figure 3c). However, analysis of those same animals during rebound after sleep deprivation showed a clear clustering, segregating the samples in two subsets with separation around the 300 seconds inactivity trigger (Figure 3d). This result is important for two reasons: on one hand, it provides, for the third time, strong evidence that the system is not simply overfitting data of nought biological significance, given that it could not perform any better than a random classifier on the baseline control. On the other hand, coccinella could find biologically relevant differences on rebound data after different regimes of sleep deprivation. Interestingly enough, the 300 seconds threshold that coccinella independently identified has a deep intrinsic significance for the field, for it is considered to be the threshold beyond which flies lose arousal response to external stimuli, defining a “sleep quantum” (i.e.: the minimum amount of time required for transforming inactivity bouts into sleep bouts23,24,28). Coccinella’s analysis ran agnostic of the arbitrary 5-minutes threshold and yet identified the same value as the one able to segregate the two clusters, thus providing an independent confirmation of the five-minutes rule in D. melanogaster.
System neuroscience is living a period of renaissance, and Drosophila is driving this revolution strong of the first full-brain connectome, a plethora of new genetic reagents that allow thermo- and opto-genetic manipulations, a galore of genetic transformants for circuit tracking and manipulation, and multiple tools for large-scale quantification of behaviour. Progresses in machine learning and computer power have had a massive impact on the field of ethomics, especially in achieving levels of anatomical tracking that allow mapping of the tiniest movements on an experimental animal model with the highest temporal resolution and with little human supervision6,17. Most of these systems, however, rely on relatively expensive setups and do not scale easily to high-throughput experimental paradigms. They are ideal – and irreplaceable – to identify behavioural patterns and study fine motor control but may be undue for other uses. Here we introduce a new framework, coccinella, that merges an open-source, economically accessible hardware platform (ethoscopes14,29) with a powerful toolbox for statistical analysis and clustering (HCTSA15 / Catch2216). Coccinella is a reductionist tool, not meant to replace the behavioural categorization that other tools can offer but to complement it. It relies on raspberry PIs as main acquisition devices, with associated advantages and limitations. Ethoscopes are inexpensive and versatile but are limited in terms of computing power and acquisition rates. Their online acquisition speed is fast enough to successfully capture the motor activity of different species of Drosophilae28, but may not be sufficient for other animals moving more swiftly, such as zebrafish larvae. Moreover, coccinella cannot – and is not meant to – apply labels to behaviour (“courting”, “lounging”, “sipping”, “jumping” etc.) but it can successfully identify large behavioural phenotypes and generate unbiased hypothesis on how behaviour, and a nervous system at large, can be influenced by chemicals, genetics, artificial manipulations in general. Here, we provided evidence that coccinella can be used to successfully explore and compartmentalise the pharmacobehavioural space and also showed that a reductionist approach can be employed to discern otherwise invisible shades of a very subtle naturally occurring behavior: sleep. The success of Drosophila as experimental model was built on the many genetic screens of the 1900s. We propose coccinella as an accessible, pivotal tool to boost again this important line of work in any laboratory, without funding or access to technology being a discriminative factor.
We thank the Gilestro lab at Imperial College London and Robert Lind at Syngenta for useful discussions. HJ was supported by a BBSRC/CASE studentship in partnership with Syngenta (project reference BB/M011178/1/1958700). Stocks obtained from the Bloomington Drosophila Stock Center (NIH P40OD018537) were used in this study.
Fly lines were maintained on a 12-hour light: 12-hour dark (LD) cycle and raised on polenta and yeast-based fly media (agar 96 g, polenta 240 g, fructose 960 g and Brewer’s yeast 1,200 g in 12 litres of water). Canton-Special (CS) Drosophila melanogaster were used as the wild-type line for all experiments.
Drug resistant mutants
RdlA301S is derived from RdlMDRR (RRID:BDSC_1675), an Rdl allele isolated from a natural population in Maryland30, and underwent isogenization and selection on dieldrin to eliminate the metabolic resistance and maintain the dieldrin target site resistance31. The RdlMDRR was obtained from the Bloomington Drosophila Stock Center. For para: the L1029F mutation, located in the voltage gated sodium channel paralytic, has been extensively reported to confer resistance to DDT and pyrethroids in many other insect species (called kdr, knockdown resistance, reviewed in32). In the Drosophila gene, kdr maps to L1029F and is equivalent to the often cited L1014F in other insects (e.g. Musca domestica33). The kdr L1029F mutation in Drosophila Para was introduced via CRISPR/Cas9-mediated genome editing (see below). This genome edited generated mutation resulted in a similar resistance to DDT as previously reported34.
Generation of ParaL1029F via CRISPR-CAS9 based genome editing
CRISPR/Cas9-mediated genome editing was used to introduce a point mutation L1029F in para-PBG isoform, CTT to TTT, L to F by homology-dependent repair (HDR) using one guide RNA and a dsDNA plasmid donor. The strategy design, molecular biology and screening was completed by Well Genetics Inc., Taiwan (R.O.C.). The cassette PBacDsRed contains Piggy Bac 3’ terminal repeats, the selection marker 3xP3-DsRed, and Piggy Bac 5’ terminal repeats. The selection marker 3xP3-DsRed contains Piggy Bac 3’ terminal repeats, 3x Pax3 and hsp70 promoter, DsRed2, SV40 3’UTR, and Piggy Bac 5’ terminal repeats. The dsRed marker facilitates the genetic screening and was excised by Piggy Bac transposase. Only one TTAA motif was left after transposition embedded in mutated intron sequence, and create a mutation G to A on X:16,486,649; X:16,486,649∼ X:16,486,646, CTAA to T TAA in intron. The CRISPR Target Site [PAM]: CACAAGATTGCCGATGACAA[CGG]. Guide RNA Primers: Sense oligo5’-CTTCGCACAAGATTGCCGATGACAA and Antisense oligo5’ - AAACTTGTCATCGGCAATCTTGTGC. Upstream Homology Arm: 1,027bp,the +34,097nt to +35,123 nt from ATG of para. Forward Oligo5’-GTTCACCAAACTCGGAATCG; Reverse Oligo5’- GTGGCCAAGAAGAAGGGAAT. Downstream Homology Arm: 1,022 bp, the +35,128nt to +3,6149 nt from ATG of para Forward Oligo: 5’-CCATGGCTTTAAGCATCGCA; Reverse Oligo: 5’- TTATGACGGATACGGTTACGG. Synthesis fragment: 5’– GGTTGTCATCGGCAATTTTGTGgtgagtactcttatcgaactgctgacttgtaaacgatgtttactggctataatgctgacttatcgcct
The Drosophila injection strain was white1118. 206 embryos were injected. 36 G0 crosses were established. Of the 78 positive lines crosses, in the F1 screen 25 positive lines were identified. Seven lines were positively validated by PCR and 1 line was sequenced confirming no unexpected changes in para. Lines were isogenized and balanced. DsRed was excised using PiggyBac (PBac) Transposase Bloomington Stock RRID:BDSC_8285. Excision was validated by genomic PCR and sequencing. Resulting lines were hemizygous viable. The line used in study had internal identifier: 20256ex1.
Choice, handling and preparation of drugs
The initial preliminary analysis was conducted using a group of 12 compounds “proof of principle” compounds and a solvent control. These compounds were initially used to compare both the video method and ethoscope method. After testing these initial compounds, it was found that the ethoscope methodology was more successful, and then the compound list was expanded to 17 (including the control) only using the ethoscope method. As a final test, we included additional compounds for a single concentration, bringing up the total to 40 (including control), also for the ethoscope method. All insecticide compounds were supplied by Syngenta Ltd. from their in-house stock (see Supplementary Table 1 for a full list of compounds used). Compounds were received in solid form and diluted in solvent containing 5% ethanol (VWR, 20821), 5% acetone (Sigma,179124) and 10% dimethylsulfoxide (DMSO) (D2650, Sigma) in distilled water to 1000ppm initially, then further diluted in the solvent mixture to 100ppm and 1ppm (where 1mL/L = 1000ppm). For insecticide assays, 0.5mL of 5% sucrose (Sigma, S0389), 1% agarose (Sigma, A6236) solution was pipetted into each well and allowed to set. Following this, 2µL of compound solution were placed on the surface and allowed to dry for 30 minutes or more. Male flies were then placed on the surface with a small glass cover slip placed on top (13mm circular cover slip, VWR631-0150). Flies were briefly anaesthetised (>1 minute) before being placed onto the surface of the plate. Once each well had been filled with a single male fly, arenas were placed into the ethoscope and recorded for a minimum of two days. All experiments were started between ZT0 and ZT1 and within 30 minutes of the flies being placed in the wells. For each compound in Figures 1c and Figure 1 - Figure Supplement 1c, three repeats were done at different time points. For Figure 1D, two repeats to three biological repeats per compound.
Data acquisition and processing
Ethoscope data were first processed in R using rethomics29. Each time series was exported (.csv) and converted using Python to individual time series in a file format (.dat) compatible with MatLab. A metadata (.txt) file served as a reference file of each individual time series with keywords outlining compound groups and concentrations for processing data using HCTSA.
HCTSA / Catch22
Following this process, HCTSA feature extraction was performed on the time-series data (for figures 1c, 1d, 3c, 3d, Figure 1 – Figure Supplements 1 and 2a). After the features were extracted, outputs of error-producing operations were removed through a normalization process using a sigmoidal transformation. HCTSA inbuilt functions were then used to classify data using a linear SVM classifier and a confusion matrix comparing the time series was generated. For some of the time-series data (that in Figure 2B, 2D, Figure 1 - Figure Supplement 2b), a smaller feature set of HCTSA, Catch-22 was used for feature extraction. Due to the smaller number of features used with this method, normalization was not required before using a linear SVM classifier to generate a confusion matrix comparing the results. A time series of 12 hours was used for the HCTSA analysis in Figure 1 and Figure 1 - Figure Supplement 1,2. A length of 3 hours was used for the HCTSA analysis in Figure 3. All video data in Figure 2 are from time series of 15 minutes. By always running the full set of features on aggregate to train a classifier (e.g., TS_Classify in HCTSA), no post-hoc correction is necessary because the trained classifier only ever makes a single prediction (only one test is performed).
Video generation for flies on insecticides
Custom 3D-printed squares were designed using the online CAD software Onshape and printed using Ultimaker 2+ 3D printers using PLA plastic. For insecticide assays, 0.5mL of 5% sucrose (Sigma, S0389), 1% agarose (Sigma, A6236) solution was pipetted into each well and allowed to set. Following this, 2µL of compound solution were placed on the surface and allowed to dry for 30 minutes or more before male flies were placed on the surface with a small glass cover slip placed on top (13mm circular cover slip, VWR631-0150). Flies were briefly anaesthetised (>1 minute) before being placed onto the surface of the square. Once each well had been filled with a single male fly, squares were placed in the arena and a video was recorded for a minimum of 12 hours using an ELP 8 megapixel camera with an IMX179 Sensor and 2.8-12 mm variable focus manual lens. All video recordings were started between ZT0 and ZT1. Recordings of flies exposed to compounds were done in a randomised manner. Video data was then broken down into shorter segments of 15-minute videos for processing. The first 15-minutes following 1 hour of fly exposure to compound or control was used for pose extraction.
Deeplabcut / B-SoID
The use of DeepLabCut (version 2.1) followed the detailed protocol outlined by5. Briefly, frames for labelling were extracted from 3 representative videos using a K-means algorithm and frames were labelled with 22 unique body parts (head, left eye, right eye, thorax top, thorax bottom, abdomen top, abdomen middle, abdomen bottom, left wing tip, right wing tip, left foreleg tip, left foreleg middle, right foreleg tip, right foreleg middle, left middle leg tip, left middle leg middle, right middle leg tip, right middle leg middle, left back leg tip, left back leg middle, right back leg tip, right back leg middle). These frames were labelled locally with a DeepLabCut graphical user interface (GUI) before the project file was uploaded to Google Drive for training and video analysis to be done using Google Colab. The data was split into a 9:1 test:train dataset and training was run for more than 150,000 iterations before the average Euclidean error was computed between labels and predictions. The model at the best performing checkpoint was used to predict pose in novel videos. Following this, B-SOID was used to de-structure behaviour using the output of DeepLabCut and generate fly-specific time-series of behavioural grammar as in4.
Maximal velocity time series data generated from recording flies exposed to either control conditions (no SD) or incrementally increasing immobility-triggered SD conditions were taken from the dataset generated by Geissmann et al25. Only data for female flies were taken, and where groups consisted of more than 60 individuals, a sample of 60 random individual flies were chosen to take time series data from. Time series data were processed in the same way as described in section 4.2 using the raw ethoscope data.
Flies were fed with the specified compounds at the desired concentrations and concomitantly analysed in ethoscopes for 24 hours. Time of death was calculated as the last moment of detected motion.
Code and data availability
A notebook version of the source code used to generate all figures is available on the Zenodo public repository, along with all the metadata and the raw data collected in this study (DOIs: 10.5281/zenodo.7335575 and 10.5281/zenodo.7393689). Data were analysed using rethomics35 and ethoscopy36. Two notebooks showing how to use the system for multiple uses are provided as supplementary material.
Bibliography for references listed in supplementary table 1 only (see column “Ref”).
Supplementary Notebooks. Two Jupyter notebooks guiding the user through the integration of ethoscope data with HCTSA (notebook 1) and Catch22 (notebook2).
- 1.Ethology as a physical scienceNat. Phys 1https://doi.org/10.1038/s41567-018-0093-0
- 2.Computational Neuroethology: A Call to ActionNeuron 104:11–24
- 3.Mapping Sub-Second Structure in Mouse BehaviorNeuron 88:1121–1135
- 4.B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviorsNat. Commun 12
- 5.Using DeepLabCut for 3D markerless pose estimation across species and behaviorsNat. Protoc 14:2152–2176
- 6.Fast animal pose estimation using deep neural networksNat. Methods 16:117–125
- 7.LiftPose3D, a deep learning-based approach for transforming two-dimensional to three-dimensional poses in laboratory animalsNat. Methods 18:975–981
- 8.Revealing the structure of pharmacobehavioral space through motion sequencingNat. Neurosci 23:1433–1443
- 9.Behavioral fingerprints predict insecticide and anthelmintic mode of actionMol. Syst. Biol 17
- 10.Behavioral idiosyncrasy reveals genetic control of phenotypic variabilityProc. Natl. Acad. Sci 112:6706–6711
- 11.High-throughput ethomics in large groups of DrosophilaNat. Methods 6:451–457
- 12.JAABA: interactive machine learning for automatic annotation of animal behaviorNat. Methods 10:64–67
- 13.MAPLE (modular automated platform for large-scale experiments), a robot for integrated organism-handling and phenotypingeLife 7
- 14.Ethoscopes: An Open Platform For High-Throughput EthomicsPLOS Biol 15
- 15.S. hctsa: A Computational Framework for Automated Time-Series Phenotyping Using Massive Feature ExtractionCell Syst 5:527–531
- 16.catch22: CAnonical Time-series CHaracteristicsData Min. Knowl. Discov 33:1821–1852
- 17.DeepLabCut: markerless pose estimation of user-defined body parts with deep learningNat. Neurosci 21:1281–1289
- 18.A New Chamber for Studying the Behavior of DrosophilaPLOS ONE 5
- 19.Geissmann, Q. High-throughput recording, analysis and manipulation of sleep in drosophila. (2018) doi:10.25560/69514.High-throughput recording, analysis and manipulation of sleep in drosophila https://doi.org/10.25560/69514
- 20.Chapter 4 - Feature selection and classificationRecent Trends in Computer-Aided Diagnostic Systems for Skin Diseases :95–135https://doi.org/10.1016/B978-0-323-91211-2.00001-9
- 21.Reversing insecticide resistance with allelic-drive in Drosophila melanogasterNat. Commun 13
- 22.The role of Rdl in resistance to phenylpyrazoles in Drosophila melanogasterInsect Biochem. Mol. Biol 54:11–21
- 23.Correlates of Sleep and Waking in Drosophila melanogasterScience 287:1834–1837
- 24.Rest in Drosophila Is a Sleep-like StateNeuron 25:129–138
- 25.Most sleep does not serve a vital function: Evidence from Drosophila melanogasterSci. Adv 5
- 26.Sensory processing during sleep in Drosophila melanogasterNature https://doi.org/10.1038/s41586-021-03954-w
- 27.Covert sleep-related biological processes are revealed by probabilistic analysis in DrosophilaProc. Natl. Acad. Sci 117:10024–10034
- 28.Joyce, M. et al. Divergent evolution of sleep functions. 2023.05.27.541573 Preprint at 10.1101/2023.05.27.541573 (2023).Divergent evolution of sleep functions https://doi.org/10.1101/2023.05.27.541573
- 29.Rethomics: an R framework to analyse high-throughput behavioural databioRxiv https://doi.org/10.1101/305664
- 30.Isolation of Dieldrin Resistance from Field Populations of Drosophila melanogaster (Diptera: Drosophilidae)J. Econ. Entomol 83:1733–1737
- 31.The mode of action of isocycloseram: A novel isoxazoline insecticidePestic. Biochem. Physiol 187
- 32.Expression of a glutamate-activated chloride current in Xenopus oocytes injected with Caenorhabditis elegans RNA: evidence for modulation by avermectinMol. Brain Res 15:339–348
- 33.Insect sodium channels and insecticide resistanceInvert. Neurosci 7
- 34.‘What I cannot create, I do not understand’: functionally validated synergism of metabolic and target site insecticide resistanceProc. R. Soc. B Biol. Sci 287
- 35.Rethomics: An R framework to analyse high-throughput behavioural dataPLOS ONE 14
- 36.Blackhurst, L. & Gilestro, G. F. ethoscopy & ethoscope-lab: a framework for behavioural analysis to lower entrance barrier and aid reproducibility. 2022.11.28.517675 Preprint at 10.1101/2022.11.28.517675 (2022).ethoscopy & ethoscope-lab: a framework for behavioural analysis to lower entrance barrier and aid reproducibility https://doi.org/10.1101/2022.11.28.517675
- 1.DDT: A New Hypothesis of Its Mode of ActionScience 146:657–658
- 2.Agricultural Uses of Juglone: Opportunities and ChallengesAgronomy 10
- 3.Juglone and 1,4-Naphthoquinone—Promising Nematicides for Sustainable Control of the Root Knot Nematode Meloidogyne luciFront. Plant Sci 13
- 4.Inhibition of Tribolium gut chitin synthetasePestic. Biochem. Physiol 13:129–136
- 5.Inhibition of Cellular Proliferation of Imaginal Epidermal Cells by Diflubenzuron in Pupae of the Stable FlyScience 207:985–987
- 6.Studies on the Respiratory Chain-linked Reduced Nicotinamide Adenine Dinucleotide DehydrogenaseJ. Biol. Chem 243:844–847
- 7.Inhibition of dihydrofolate reductase from bacterial and vertebrate sources by folate, aminopterin, methotrexate and their 5-deaza analoguesBiochem. Pharmacol 33:175–179
- 8.The Carbodiimide Product of Diafenthiuron Inhibits Mitochondria in VivoPestic. Biochem. Physiol 46:96–106
- 9.Fenoxycarb, a fairly new Insect Growth Regulato: review of its effects on insectsAnn. Appl. Biol 122:369–403
- 10.The thiourea insecticide diafenthiuron inhibits mitochondrial ATPase in vitro and in vivo by its carbodiimide productBiochem. Soc. Trans 22:241–244
- 11.Lufenuron, a chitin-synthesis inhibitor, interrupts development ofDrosophila melanogasterJ. Exp. Zool 278:37–44
- 12.Insecticidal Action and Mitochondrial Uncoupling Activity of AC-303,630 and Related Halogenated PyrrolesPestic. Biochem. Physiol 50:115–128
- 13.Characterization of nicotinic acetylcholine receptors from the insectsAphis craccivora,Myzus persicae, andLocusta migratoria by radioligand binding assays: Relation to thiamethoxam actionJ. Biochem. Mol. Toxicol 14:221–230
- 14.Mode of action of the ecdysone agonist tebufenozide (RH-5992), and an exclusion mechanism to explain resistance to itPest Manag. Sci 57:951–957
- 15.Toxicity, mode of action, and synergist potential of flonicamid against mosquitoesPestic. Biochem. Physiol 151:3–9
- 16.Insecticidal spider toxins are high affinity positive allosteric modulators of the nicotinic acetylcholine receptorFEBS Lett 593:1336–1350
- 17.Muscarinic Agonists as Insecticides and AcaricidesPestic. Sci 49:268–276
- 18.Indoxacarb, an oxadiazine insecticide, blocks insect neuronal sodium channels: DCJW blocks neuronal Na + channelsBr. J. Pharmacol 132:587–595
- 19.Fluazinam targets mitochondrial complex I to induce reactive oxygen species-dependent cytotoxicity in SH-SY5Y cellsNeurochem. Int 60:773–781
- 20.Uncoupling activity of a newly developed fungicide, fluazinam [3-chloro-N-(3-chloro-2,6-dinitro-4-trifluoromethylphenyl)-5-trifluoromethyl-2-pyridinamine]Biochim. Biophys. Acta BBA - Bioenerg 1056:89–92
- 21.Characterization of the Drosophila Methoprene - tolerant gene product: Juvenile hormone binding and ligand-dependent gene regulationFEBS J 272:1169–1178
- 22.The Juvenile Hormone Signaling Pathway in Insect DevelopmentAnnu. Rev. Entomol 58:181–204
- 23.The molecular interactions of pyrethroid insecticides with insect and mammalian sodium channelsPest Manag. Sci 57:877–888
- 24.Metaflumizone is a novel sodium channel blocker insecticideVet. Parasitol 150:182–189
- 25.Strategy for discovery of a novel miticide Cyenopyrafen which is one of electron transport chain inhibitorsJ. Pestic. Sci 36:511–515
- 26.Downregulation of carboxylesterase contributes to cyflumetofen resistance in Tetranychus cinnabarinus (Boisduval): Downregulated esterase genes in cyflumetofen resistance in mitesPest Manag. Sci 75:2166–2173
- 27.Synthesis and Insecticidal Activity of New Benzyl- and Indanyl-Oxazolines, Thiazolines and Alkoxy-Alkyl-ImidazolinesACS Symposium Series :411–430
- 28.Identification of a Drosophila melanogaster Glutamate-gated Chloride Channel Sensitive to the Antiparasitic Agent AvermectinJ. Biol. Chem 271:20187–20191
- 29.Expression of a glutamate-activated chloride current in Xenopus oocytes injected with Caenorhabditis elegans RNA: evidence for modulation by avermectinMol. Brain Res 15:339–348
- 30.Anthranilic diamides: A new class of insecticides with a novel mode of action, ryanodine receptor activationPestic. Biochem. Physiol 84:196–214
- 31.Discovery of cyantraniliprole, a potent and selective anthranilic diamide ryanodine receptor activator with cross-spectrum insecticidal activityBioorg. Med. Chem. Lett 23:6341–6345
- 32.A point mutation in a Drosophila GABA receptor confers insecticide resistanceNature 363:449–451
- 33.Flubendiamide, a Novel Ca 2+ Channel Modulator, Reveals Evidence for Functional Cooperation between Ca 2+ Pumps and Ca 2+ ReleaseMol. Pharmacol 69:1733–1739
- 34.Transmembrane Potential Polarization, Calcium Influx, and Receptor Conformational State Modulate the Sensitivity of the Imidacloprid-Insensitive Neuronal Insect Nicotinic Acetylcholine Receptor to Neonicotinoid InsecticidesJ. Pharmacol. Exp. Ther 341:326–339
- 35.Effects of mutations in Drosophila nicotinic acetylcholine receptor subunits on sensitivity to insecticides targeting nicotinic acetylcholine receptorsPestic. Biochem. Physiol 102:56–60
- 36.TRP Channels in Insect Stretch Receptors as Insecticide TargetsNeuron 86:665–671
- 37.Localization of Ca2+ release channels with ryanodine in junctional terminal cisternae of sarcoplasmic reticulum of fast skeletal muscleProc. Natl. Acad. Sci 82:7256–7259
- 38.Spiroindolines Identify the Vesicular Acetylcholine Transporter as a Novel Target for Insecticide ActionPLoS ONE 7
- 39.Modification of single sodium channels by the insecticide tetramethrinBrain Res 274:344–349