Drosophila develops through four developmental stages (embryo, larva, pupa, and adult), subsequently finishing its own life. The static phases (embryo, pupa, and death) alternate with the dynamic phases (larva and adult; Figure 1A). Single-image processing between continuous images distinguishes both phases: dynamic phases are detected as positive signals while static phases simultaneously present no signal (Figure 1B). Therefore, we can precisely identify the transition points between phases by monitoring the static and dynamic status and separately detecting fly phases on all plates captured simultaneously in the same image.

Figure 1 with 3 supplements see all

Download asset Open asset

DIAMonDS consists of an automated time-lapse imaging system and our novel image analysis software named ‘Sapphire’ (Figure 1B). We used a combination of a flatbed CCD scanner and VueScan software (https://www.hamrick.com), which enables multiple scanner units to capture images continuously at certain intervals (Smith et al., 2014). To identify the phase change time points for each fly event, single flies are inserted into each well of a 96- or 384-well microplate containing suitable fly media. Up to three microplates are then set on the scanner surface, and time-lapse images are acquired at appropriate intervals until the fly event is completed. A single scanner can monitor 288 or 1152 individuals in three 96-well or 384-well microplates, respectively for DIAMonDS. Because the well-size of 384-well microplate is too small to succeed in normal development and lifespan, we only use the 384-well microplate for measuring adult survivorships in the short term (within about 2 weeks) such as stress or drug resistance assays. Sapphire then automatically analyzes and detects the transition points of pupariation, eclosion, and death in the newly acquired time-lapse images (Figure 1B,C). Our system can simultaneously monitor multiple scanners with a single personal computer.

As positional bias and fluctuations in environmental factors such as ambient temperature and relative humidity (RH) might markedly reduce the reliability of our system, all experiments were conducted in a plant growth chamber (LPH-410NS) with automatic temperature and humidity regulation and additional USB fans (Appendix 6). The scan surface temperature was continuously recorded with button-sized temperature data loggers (NK Labs LLC, Cambridge, MA, USA). Temperatures widely varied among locations under external illumination on/off conditions possibly because of irregular and uneven irradiation. Thus, we acquired time-lapse images without external illumination to maintain a steady temperature (Figure 1—figure supplement 1).

Sapphire automatically determines the static-to-dynamic and dynamic-to-static phase changes for all flies according to the time-lapse images acquired by the aforementioned scanner system. The following four processes were implemented in Sapphire to enable automatic life event detection based on individual Drosophila images (Figure 1C):

1. Images of single animals were segregated by image processing (Figure 1C, Step 1), enabling the system to readily target individual flies.

2. Semantic segmentation was performed to capture each insect in each well (Figure 1C, Step 2). Recently, artificial intelligence techniques have substantially improved both image classification (Goodfellow et al., 2016) and segmentation (Badrinarayanan et al., 2017; Ronneberger et al., 2015). Here, we designed a fully convolutional network (FCN) specifying larval and adult segmentation. The FCN has encoder-decoder architecture (Figure 1—figure supplement 2) comprising three blocks, each including a convolution layer with a 3 × 3 filter, up/down sampling layer with a 2 × 2 filter, and dropout layer with a 25% ratio. The encoder and decoder parts were mutually connected by two convolution layers with 3 × 3 filters. The convolution layers in the encoder and decoder were fitted with rectifier linear units. The inputs to the layers were applied by batch normalization. The output layer was also a convolution layer and included reshaping and softmax functions.

The annotation data were manually created for larval and adult Drosophila and increased by general data augmentation techniques such as vertical and horizontal flips, rotation, and luminance modulation (Materials and methods).

After training with augmented data, the segmentation inference was calculated as a probability. If it was > 0.5, the system regarded it as the target region (animal body). If it was < 0.5, the system treated it as background. Consequently, the system obtained binary images wherein the animal body corresponded to one and the background was described as 0. Semantic segmentation was applied to all individuals and every sequential population image.

All trainings and inferences were performed on a Linux PC (Ubuntu) with four GPUs (GTX 1080Ti). All scripts were written in Python using the deep learning libraries keras (v. 2.0.9) and tensorflow-gpu (v. 1.4.0).

1. The system converted the labeled data to time series data by subtracting every two consecutive labeled images (Figure 1C, Step 3). After signaling, the system applied the ChangeFinder (CF) algorithm developed to detect turning points (Takeuchi and Yamanishi, 2006) and evaluated total individual animal activity from the central gravity distance of the segmented area between two consecutive images (Video 1).

2. The system automatically determined life event transitions with single-animal resolutions from the CF signals (Figure 1C, Step 4). Life event transitions were estimated based on the transition points from dynamic to static or vice-versa. These were designated the maximum points in the CF signals. Death and pupariation corresponded to maximum CF points reflecting dynamic-to-static phase changes. In contrast, eclosion was characterized as a maximum CF point corresponding to a static-to-dynamic change. After transition point determination, the system summarized the automatic detection results using various visualization styles (Figure 1C, Step five and Figure 1—figure supplement 3). All algorithms and visualization tools were user-friendly web applications developed with the Dash framework in Python (Figure 1—figure supplement 3).

Video 1

Download asset

Apparent dynamic-to-static and vice-versa phase changes occur at pupariation and eclosion (Figure 2A). We attempted to detect individual fly pupariation and eclosion with DIAMonDS to validate it. Newly hatched first-instar (L1) larvae were placed into 96-well microplates containing 100 µL well⁻¹ standard fly media (Figure 2—figure supplement 1). Three microplates were fixed to the scanner surface, and the scanner was placed inverted in the incubator to prevent fly media leakage. The scanner was powered on, and time-lapse scanning was performed with VueScan (Materials and methods) until all flies eclosed. Thence, time-lapse images were analyzed with Sapphire (Figure 2A,B).

Figure 2 with 5 supplements see all

Download asset Open asset

Figure 2—source data 1

Data of individual pupariation timing obtained by Sapphire and visual handling (for Figure 2C and Figure 2—figure supplement 2A).

https://cdn.elifesciences.org/articles/58630/elife-58630-fig2-data1-v2.xlsx

Download elife-58630-fig2-data1-v2.xlsx

Figure 2—source data 2

Data of individual eclosion timing obtained by Sapphire and visual handling (for Figure 2D and Figure 2—figure supplement 2B).

https://cdn.elifesciences.org/articles/58630/elife-58630-fig2-data2-v2.xlsx

Download elife-58630-fig2-data2-v2.xlsx

Figure 2—source data 3

Data for scatterplot between pupariation and eclosion timing of individual flies and for box plots of pupariation and eclosion timing in males and females (for Figure 2E–G).

https://cdn.elifesciences.org/articles/58630/elife-58630-fig2-data3-v2.xlsx

Download elife-58630-fig2-data3-v2.xlsx

The L1 larvae dove into the media until mid-L3 (~2 d). As time-lapse scanning could not detect larval movements here, this so-called feeding stage was designated as static. In contrast, the larvae typically left the media and moved around the well surface during late L3 (Figure 2A). This so-called wandering stage was designated as dynamic, as there were consecutive high-activity wandering larval signals (Figure 2B), and its duration was 12–24 hr. Thereafter, L3 larval activity gradually decreased, and pupariation followed. During the pupal stage (~100 hr), the activity was not detected in a static phase. The second signal wave activity was detected just after dynamic eclosion.

Sapphire independently and automatically detects maximum peaks at the pupariation and eclosion transition points using machine learning based on an FCN to detect larval and adult animals separately (Figures 1C and 2B). To verify Sapphire's accuracy, we compared pupariation and eclosion timings analyzed by Sapphire and those manually detected (visual handling) from the same images. The Sapphire and visual handling values were nearly identical (Figure 2C,D), 73.8% on pupariation and 84.2% on eclosion were only a slight difference within ± 10 frames between them (Figure 2—figure supplement 2). Film surface contamination after long-term rearing in small wells might have accounted for the observed decrease in Sapphire detection accuracy. Therefore, to investigate whether Sapphire's detection accuracy improves in a more transparent state of the lid, flies were transferred to the new microplate just after the time that all flies pupated (P14 stage) and scanned time-lapse scanning of the plates. The experiment resulted in 94.7% of values from Sapphire showed nearly identical (within ± 10 frames) to visual handling values, indicating that Sapphire's accuracy was greatly improved relative to the experimental setup in which the same microplate was used both for pupariation and eclosion detections (Figure 2—figure supplement 3A,B).

Arbitral image sequences such as raw images and the segmentation images could be analyzed in Sapphire (see Appendix 8). Sapphire calculates the CF signal obtained from the subtraction of consecutive segmentation images as fully automated method (CF method). Sapphire determined the timing of life-event by applying just thresholding on the arbitral time series data specified by user (TH method). At first, threshold is automatically calculated as an average of maximum and minimum value of the signal (auto-TH) and the threshold is modifiable by hands depending on user’s demand (manual-TH). Accuracy of CF method and auto- and manual- TH method on raw image subtraction was quantified in (Appendix 8). We compared CF and TH using the same time-lapse image set (Figure 2—figure supplement 3). The output of CF was superior to that of TH, as the former is relatively more sensitive to phase shifts. Thus, Sapphire is functional and highly invaluable in automatic analyses.

The relationship between pupariation and eclosion determined by DIAMonDS clearly revealed sexual dimorphism during development (Figure 2E). On average, females eclosed 4 hr earlier than males. A previous study corroborated this observation (Bainbridge and Bownes, 1981). The pupariation transition time points did not differ between sexes. Therefore, sexual dimorphism in the eclosion time points reflects the difference in pupal duration between sexes (Figure 2E–G). No significant differences were found between plates 1 and 3 used here, indicating a minimal scan surface positional effect in DIAMonDS during the detection of the pupariation and eclosion time points (Figure 2—figure supplement 4). Altogether, these results indicate that DIAMonDS is suitable for automatic measurement of both the pupariation and eclosion time points at high temporal resolution.

To understand the effect of chamber size on fly development, we used three different sizes of microplates (96-well, 48-well, and 24-well) contained normal fly media for measuring pupariation and eclosion. We observed that timings of pupariation and eclosion were slightly but significantly shorten both in the 48-well and 24-well conditions and the pupal duration might have the fewer effects of well-size, suggesting that the chamber size might affect fly’s development (Figure 2—figure supplement 5).

DIAMonDS showed excellent performance in large-scale pupariation timing analyses (Figure 3—figure supplement 1). To evaluate its performance, we explored whether two distinct genetic and environmental conditions affect larval development. First, we used larvae with delayed pupariation at 29°C (genotype: R29H01-GAL4 > UAS TeTxLC), in which tetanus toxin light chain (TeTxLC) impaired serotonergic SE0_PG neuron activity (Shimada-Niwa and Niwa, 2014). To evaluate the influences of initial larval age on DIAMonDS measurement accuracy, we used L2, early L3, and late L3 larvae in the microplate assays. DIAMonDS successfully detected pupariation delays in larvae of all ages (Figure 3A). Thus, DIAMonDS performance remains highly stable over a wide range of conditions.

Figure 3 with 1 supplement see all

Download asset Open asset

Figure 3—source data 1

Box plot analysis of pupariation timing in flies with impaired ecdysteroid biosynthesis at 29°C (for Figure 3A).

https://cdn.elifesciences.org/articles/58630/elife-58630-fig3-data1-v2.xlsx

Download elife-58630-fig3-data1-v2.xlsx

Figure 3—source data 2

Effect of sucrose concentration in media on pupariation and eclosion (for Figure 3C–F).

https://cdn.elifesciences.org/articles/58630/elife-58630-fig3-data2-v2.xlsx

Download elife-58630-fig3-data2-v2.xlsx

We also used DIAMonDS to clarify the relationship between sugar concentration and development in the w¹¹¹⁸ fruit fly strain (Figure 3B–F). A previous report stated that Drosophila larvae that were administered a high-sugar diet presented a type 2 diabetes-like phenotype and developmental delay (Musselman et al., 2011). L1 larvae were placed in wells containing normal to high-sugar concentrations, scanned by time-lapse imaging, and the pupariation and eclosion time points were detected by Sapphire. Both pupariation and eclosion were gradually delayed in a sugar concentration-dependent manner. Therefore, excess sugar adversely affects larval growth (Figure 3B–E). When we compared the pupariation and eclosion time points among individual larvae, the pupal durations were nearly constant regardless of sugar concentration. Thus, the sugar concentration determines the time spent as a larva until pupariation, but it does not influence the time spent as a pupa (Figure 3F). Interestingly, Northrop reported that prolongation of the pre-imago stage of D. melanogaster by yeast supplementation had no impact on pupal stage duration (Northrop, 1917a; Northrop, 1917b). The process by which pupal duration is determined may be independent of larval dietary intake. Overall, DIAMonDS is a powerful toolkit for detecting the pupariation and eclosion time points and discloses the effects of several endogenous and exogenous factors on individual fly development.

Adult death time points can be measured at high temporal resolution with DIAMonDS. This tool efficiently detects sudden shifts from a dynamic to a static phase. Depending on the objective, experiments are performed over a broad range of time scales extending to nearly 3 months. Small- and large-scale impact assessment targets may include mutant phenotype, environmental stress, and chemicals and drugs (Afschar et al., 2016; Harshman et al., 1999; Lin et al., 1998; Parkes et al., 1998; Piper and Partridge, 2016; Tsuda et al., 2010; Ziehm et al., 2015). DIAMonDS may be implemented using 96- and 384-well microplates to accommodate various parameters in death timing detection. The 96-well type is used in long-term detection, whereas the 384-well plate is better suited for short-term (≤10 d) detection and high-throughput assays.

We collected time-lapse scanning images at 1 min intervals for adult flies (4 day post-eclosion) maintained under starvation conditions (70 µL 1% agar/water (w/v)/well) in order to optimize DIAMonDS for 384-well microplates (Figure 4A). After the image data series consisted of dying individual flies, death time points were determined manually (visual manipulation) and by Sapphire. Both procedures generated nearly identical survival curves and were highly correlated (R = 0.9913), indicating the reliability of Sapphire (Figure 4A–D and Figure 4—figure supplement 1). Next, we examined whether the well positions within a microplate influence the death time points. No significant differences were found between each plate or among the sub-areas within each plate (Figure 4E and Figure 4—figure supplement 2). We also evaluated adult w¹¹¹⁸ fly starvation tolerance in 96-well microplates and obtained results similar to those acquired from 384-well microplates (Figure 4F). Next, we tried to test whether the DIAMonDS can effectively detect the death time points even in flies showed a very reduced amount of activity. As a hypoactive fly model, we used decapitated females who keep motor skills but hardly moved until their death (Ejima and Griffith, 2008). DIAMonDS showed relatively good results that were comparable to the visual results although there was a tendency for the accuracy to reduce slightly in comparison with the case of wild-type flies (Figure 4—figure supplement 3). Altogether, these results indicate that DIAMonDS is highly effective at detecting fruit fly death events.

Figure 4 with 3 supplements see all

Download asset Open asset

Figure 4—source data 1

Evaluation of DIAMonDS for detection of death of individual adults.

https://cdn.elifesciences.org/articles/58630/elife-58630-fig4-data1-v2.xlsx

Download elife-58630-fig4-data1-v2.xlsx

To assess the efficacy of DIAMonDS at evaluating fly survival under various stress conditions, we performed a starvation assay on male and female w¹¹¹⁸ flies (Figure 5A). They presented sexual dimorphism in terms of starvation resistance. Moreover, the temporal resolution used here (15 min intervals) was higher than those reported in previous studies (Figure 5B and Figure 5—figure supplement 1A,B; Grönke et al., 2010; Li et al., 2018).

Figure 5 with 1 supplement see all

Download asset Open asset

Figure 5—source data 1

Adult survival curve detection by DIAMonDS (for Figure 5 and Figure 5—figure supplement 1).

https://cdn.elifesciences.org/articles/58630/elife-58630-fig5-data1-v2.xlsx

Download elife-58630-fig5-data1-v2.xlsx

We then subjected the flies to various concentrations of dichlorodiphenyltrichloroethane (DDT) to identify possible resistance (Figure 5C,D; Afschar et al., 2016). To a 384-well microplate, we added media consisting of 5% (w/v) sucrose and 0.5% (w/v) yeast as described in Appendix 1. The 48 w¹¹¹⁸ males in each well were exposed to 0, 0.005, 0.01, 0.05, or 0.1% DDT (Figure 5C). DIAMonDS showed distinct variance in DDT resistance even between only slightly differing DDT concentrations (Figure 5D and Figure 5—figure supplement 1C,D).

We also investigated whether DIAMonDS can detect mutant fly phenotypes in stress resistance assays. Flies with the loss-of-function mutation methuselah (mth) present extended lifespan and resistance to several stressors including paraquat (Lin et al., 1998). To test whether DIAMonDS can discriminate mth¹ mutant stress tolerance phenotypes, we backcrossed the mth¹ mutant five times with w¹¹¹⁸ and compared the responses of both types of males to different paraquat concentrations (Figure 5E,F). The mth¹ mutant presented significantly greater resistance to several paraquat concentrations than w¹¹¹⁸. The results of this assay demonstrated that paraquat resistance could be detected in the mth¹ mutant using lower paraquat concentrations than those tested in an earlier study (20 mM)(Lin et al., 1998). Thus, DIAMonDS can readily identify optimal concentrations at high temporal resolution in drug resistance assays.

DIAMonDS successfully measured lifespans for individual w¹¹¹⁸ flies. For long-term analysis, flies must be transferred to new microplates. We performed sequential pupariation, eclosion, and lifespan measurements for individual w¹¹¹⁸ as shown in Figure 6A–C. The mean lifespans calculated and survival curves plotted for individual males and females were consistent with those of previous reports (Figure 6C; Liu et al., 2009; Schriner et al., 2014; Suh et al., 2008; Trostnikov et al., 2019).

Figure 6 with 1 supplement see all

Download asset Open asset

Figure 6—source data 1

Tracing entire life events of individual flies by DIAMonDS.

https://cdn.elifesciences.org/articles/58630/elife-58630-fig6-data1-v2.xlsx

Download elife-58630-fig6-data1-v2.xlsx

Several studies described the relationships between developmental time and adult lifespan among various species (Marchionni et al., 2020). Certain reports showed that prolonging developmental duration by yeast supplementation did not affect lifespan (Hunter, 1959; Northrop, 1917b). However, few studies focused on the associations between developmental timing and adult lifespan in Drosophila. To establish whether developmental stages and adult lifespans are interdependent in individual flies, we analyzed the following relationships: larval and pupal duration (Figure 6D,G), larval duration and adult lifespan (Figure 6E,H), and pupal duration and adult lifespan (Figure 6F,I). No significant correlations between the data for these pairs of parameters in either sex (Figure 6D–I) and those reported in previous studies were found. As the flies had the same genetic background and were exposed to the same diet and environmental conditions throughout all their life stages, it was unlikely that their genotypes influenced the relationship between development time and lifespan, demonstrating that DIAMonDS can detect unique relationships between developmental time and lifespan under different intrinsic and extrinsic conditions.

In this study, we were unsuccessful at using DIAMonDS to run long-term lifespan assays on other wild type fruit fly strains such as Oregon-R and Canton-S. Lethality accidentally escalated because the rearing conditions deteriorated. For instance, water droplets condensed and coalesced on the plate surfaces. However, these errors might have reflected relative metabolic and/or genetic differences among fly strains. To find common optimum conditions for measuring the lifespan of other laboratory strains, we have tried to observe the viability of three strains (w¹¹¹⁸, Oregon-R, and Canton-S) in condition with several microplate well-size (96-well, 48-well, and 24-well) (Figure 6—figure supplement 1). The viabilities were measured at the time point at 1, 2, 3, and 4 weeks with once plate replacement a week. We observed that the reduced viability was not rescued by changing to bigger microplate-well size, and somewhat, increasing well-size might have harmful effects on the survival of aged flies (Figure 6—figure supplement 1A–C). We next tested the two different plate replacement cycles (once a week, or once 2–3 days) using Oregon-R and Canton-S in 96-well microplate. We observed a shorter plate replacement cycle could effectively improve the viability (Figure 6—figure supplement 1B,C). However, because plate replacement is required a lot of work time, it seems to be not suitable for the high-throughput experiments. Further optimization of the experimental conditions will be necessary to effectively use DIAMonDS to conduct lifespan tests on a wide range of fly strains.

The wild-type strain used here was mainly w¹¹¹⁸, Oregon-R, and Canton-S. The mth¹ (BDSC #27896) mutant males were used in the paraquat resistance test after backcrossing six-fold with w¹¹¹⁸. R29H01-GAL4 (BDSC #47343) and UAS-TeTxLC (BDSC #28838) were obtained from the Bloomington Drosophila Stock Center, Bloomington, IN, USA. All fly strains were maintained on standard medium at 25°C (Appendix 1).

Most experiments were conducted in a plant growth chamber (LPH-410NS; NK System Co. Ltd., Nagoya, Japan) maintained at 25°C and 60% RH. For the experiment using R29H01-GAL4 > UAS TeTxLC, the larvae were reared at 29°C to increase GAL4 activity.

Detailed DIAMonDS hardware and layout are described in Appendix 6.

Microplates with 96 or 384 wells were filled with standard fruit fly culture media (Appendix 1). A handmade acrylic lid was used for the 384-well microplate (Appendix 2). Titer stick film (Watson, Tokyo, Japan) was used for the 96-well microplate (Appendix 3). To determine the pupariation and eclosion timings, the film seal on the 96-well microplate was perforated with air holes (0.35 mm diameter; eight holes/well) over each well. For the stress resistance and lifespan tests, the 96-well microplate was sealed with film, which was then cut to form a cross shape over each well. Individual adult flies were transferred to each well without anesthesia. For the 384-well microplate, the flies were anesthetized with triethylamine and placed individually into each well (Appendix 4). The acrylic lid was then securely affixed to the microplate with screws and masking tape (Appendix 2).

Three microplates were secured with glass slides to the flatbed scanner surface as described in Appendix 5. The scanner was then connected to a personal computer. ‘VueScan’ scanner software in the PC was launched, and time-lapse images were acquired at 1–15 min intervals (Appendix 6).

For the drug resistance assay, the wells of a microplate were filled with yeast-sucrose medium containing the appropriate chemicals as shown in Appendix 1. For the starvation test, the wells of a microplate were filled with 1% (w/v) agar (Appendix 1). Virgin adult flies were collected under CO₂ anesthesia, stored in a vial with normal food for 3–5 days, placed individually into each microplate well, and subjected to time-lapse image acquisition.

To measure pupariation and eclosion timing, newly hatched L1 larvae (24–25 hr AEL) were used as the lethality was too high when embryos were used (Figure 2—figure supplement 1A). The large media volume in each well (150 µL) also increased lethality because the embryos suffocated when the media became very viscous. Therefore, we used 100 µL media for this approach (Figure 2—figure supplement 1B). L2 or L3 larvae were also used in the DIAMonDS experiments. After the larvae were loaded into the microplate wells, the microplate was covered with Titer stick film (Watson, Tokyo, Japan), inverted on a predetermined area of the flatbed scanner surface, and fastened with tape. The scanner was then inverted in the incubator to prevent liquefied culture medium from falling onto the film surface. Time-lapse scanning was then run in VueScan in the PC. Scanning was terminated when all individuals eclosed to adults. The time-lapse images were analyzed in Sapphire (Figure 1B,C).

Microplates were prepared with synchronized L1 larva (one/well), and images were acquired as previously described. Developmental stages for individuals were analyzed according to the images saved to the PC. After all individuals reached the P14 stage pupa, the pupae were transferred to the same well positions in a new microplate. The wells were sealed with new film, and scanning was resumed. After all flies completely eclosed, the individual adults were transferred to the same well positions in a new microplate, and scanning was restarted to measure individual lifespans. Adult flies were transferred weekly to new microplates until all flies died. The acquired images were then analyzed in Sapphire. Finally, we introduced the data obtained by Sapphire into the DIAMonDS analysis templates (Supplementary file 1) and calculated the time of each life-event (Appendix 7).

The present system included an automated, high-accuracy life event detection algorithm and image and signal processing (Figure 1C). The software could be obtained from (https://github.com/kanglab/Sapphire/tree/master; Seong et al., 2020; copy archived at https://github.com/elifesciences-publications/Sapphire). The contents in the address also describe the installation, license, and system requirements such as directory tree and external dependencies. Quantitative algorithm summary and robustness in various parameter spaces are described in Appendix 8.

The present system performed semantic segmentation via a deep neural network whose architecture is described in Figure 1—figure supplement 2. Sufficient annotation data increases inference accuracy in deep learning. For network training, handmade supervised data were prepared and applied to image data augmentation. For adult body segmentation in death detection, the annotation data for 178 adults were introduced and magnified up to 71,200 by general image data augmentation (vertical and horizontal flips, rotation, and luminance modulation). For adult body segmentation in pupariation and eclosion detection, annotation data for 300 adults and 5760 larvae (as distractors) were prepared and magnified by ≤ 60,000 and≤57,600, respectively. For larval body segmentation in pupariation and eclosion detection, annotation data for 1839 larvae and 183,900 augmented data were introduced to train the network. The present system detected rough outlines of the animal bodies (Appendix 8).

In the present system, subtractions between consecutive segmentation images were converted to signals in CF, which is a change point detection algorithm (Takeuchi and Yamanishi, 2006).

The CF signal was obtained as follows:

1. Considering the following AR model for time series data $x_t$:

2. Inference of parameters $\theta =(A_1,\cdots ,A_k,\mu ,\sigma )$ by maximizing I:

3. Calculation of scores:

4. Time averaging of scores:

5. Recalculating scores for $y_t$:

For death detection, the CF signal was calculated from the adult body segmentation and the death timing and determined as a maximum CF signal point capturing dynamic-to-static changes (Appendix 8). On the other hand, two CF signals were obtained from the adult and larval body segmentations in pupariation and eclosion detection. Pupariation is defined as the maximum dynamic-to-static transition point for larvae. Eclosion is defined as the maximum static-to-dynamic transition point for adults (Appendix 8). After time-lapse scanning, we can easily discriminate ‘out-of-event’, such as larval, and, pupal dead individuals. Sapphire can exclude ‘out-of-event’ wells by making the ‘black lists’ file (Appendix 7).

Data were analyzed, and graphs were plotted in GraphPad Prism v. 8 (GraphPad Software, San Diego, CA, USA). A student’s unpaired, two-tailed t-test was performed to compare differences between groups in each experiment and Dunnett's method of one-way ANOVA was used for multiple comparison test (***p<0.001; **p<0.01; *p<0.05; n.s., no significant).

384-well microplate: Greiner Bio-One 781186 Clear Polystyrene 384 well Microplate

We have made a lid for 384-well microplate from an acrylic sheet (2 mm thick) as shown in Appendix 2—figure 1a. Holes (diameter = 0.7 mm) were drilled in each well position with an electric drill. The four corners were fixed by screws, and all edges were sealed by masking tape (Appendix 2—figure 1b).

Appendix 2—figure 1

Download asset Open asset

96-well microplate: TrueLine TR5003 96well culture plate

Titer stick HC film: WATOSON 547-KTS-HC

The use of anesthetics can reduce the labor dramatically for inserting individual flies in each well, especially in the case of a 384-well microplate. We tested two anesthetics, carbon dioxide and triethylamine (TEA, also called as FlyNap) (Fuyama, 1977). Short exposure of CO2 (10, 30, and 60 s) or TEA (10, 20, 30 s) did not show a significant difference in comparison with no treated flies (Appendix 4—figure 1a and b). The narcotism by short exposure of TEA vapor continues for 30 min (Fuyama, 1977). Therefore, we have been preferentially using TEA for the preparation of a fly plate.

Appendix 4—figure 1

Download asset Open asset

To put microplates on the fixed position every time, glass slides (MATSUNAMI Glass, S-2215) were arranged and glued on the scan surface as shown in Appendix 5—figure 1a (red squares indicate positions of glass slides). The microplates were placed along the guide of slide glasses and sealed by masking tape (Appendix 5—figure 1b and c). To reduce the reflection during scanning, black painted sheet was put on the surface of a scanner reflector (Appendix 5—figure 1d).

Appendix 5—figure 1

Download asset Open asset

1. Create new folder for storing images. An example of a folder name is ‘20190707-Seong-w1118.pupariation-scanner1’ (‘date-researcher’s name-information of experiment-scanner number’).

2. Connect the scanner to the PC and turn on the scanner.

3. Open the VueScan.

4. If you already saved the ‘options…”, you can load the option which recorded previously used whole parameters for acquisition of images.

5. Press the ‘preview’ button, (press ‘cancel’ button after completion of the preview), and set a region of interest (ROI) on the previewed image.

6. Set parameters in each tab as shown in Appendix 6—figure 2 and Appendix 6—figure 3. In the tab ‘Input’: we use ‘eight bit Gray’, and ‘150 dpi’ both as scan and preview resolutions. Press ‘Auto repeat’ drop-down box and choose repeat interval as shown in Appendix 6—figure 2b. Select the folder to save images. Input the JPEG file name like as ‘000001+.jpg’. In the tab ‘Color’: Input the appropriate values in ‘Black point (%)”, ‘white point (%)”, ‘Curve low’, and ‘Curve high’ to take a high contrast image. In the tab ‘Output’: confirm each parameter as shown in Appendix 6—figure 3b.

7. Press ‘File’/”Save options…’ and save the parameters as same folder name (e.g.: ‘20190707-Seong-w1118.pupariation-scanner1’).

8. Press ‘scan’ button.

9. (Option) To perform multiple scanners using one PC, repeat steps 1 to 8 for each scanner sequentially. Before the multiple scanner running, you need to replicate and assign the VueScan application for each scanner. Modify the application name of replicated each VueScan as like ‘VueScan-1’, ‘VueScan-2’... Then, open each VueScan application independently correspond to each scanner respectively.

Appendix 6—figure 2

Download asset Open asset

Appendix 6—figure 3

Download asset Open asset

We have prepared templates for data analysis of results from the Sapphire. The Sapphire analyzes the time-lapse images and, as a result, determines the event changing point as the frame number. Next, we should input each values of data (‘Blacklists’, ‘Timestamp’, and ‘Event timings (auto)”) into the appropriate positions of the Excel templates. The procedure is following.

1. Download each data set (‘Blacklists’, ‘Timestamp’, and ‘Event timings (auto)”) from the Sapphire viewer (Appendix 7—figure 1).

   • ‘Blacklists’ contains the information of omitted wells from the analysis. We can select omitted wells in the main tab on the Sapphire viewer.

   • ‘Timestamp’ is a list of date of each image.

   • ‘Event timings (auto)’ is frame number list of each well calculated by Sapphire.

2. Copy and paste downloaded data lists into appropriate position on the template (Appendix 7—figure 2 and Appendix 7—figure 3).

3. If necessary, input ‘M’ or ‘F’. You can separate the male (‘M’) and female (‘F’) using the sex information of each well (Appendix 7—figure 3).

Appendix 7—figure 1

Download asset Open asset

Appendix 7—figure 2

Download asset Open asset

Appendix 7—figure 3

Download asset Open asset

Present system provides automatic high-accurate life-event detection algorithm including the image processing and signal processing shown in Figure 1c. Following text describes quantitative summary and robustness of the algorithm in various parameter spaces.

8.2. Animal body segmentation

Present system could detect rough outline of animal body (Appendix 8-figure 1).

Appendix 8—figure 1

Download asset Open asset

In some cases, segmentation errors still appeared. The errors in segmentation could be improved by the increase of annotation data for training of network. In other word, present system showed high performance even though datasets shown in the manuscript were obtained through several imaging condition as well as the number of annotation data was small. In consistent imaging condition (same shape and size of well, plate number, and so on), the system will robustly detect entire life-event. Additional annotation data for every experiment might be also effective in order to promote network training. In other word, present algorithm could reach sufficient accuracy if an appropriate signal processing was applied on the image data even with rough segmentation.

8.3. Robustness of detection for CF parameters

In the present system, subtraction between consecutive segmentation images were converted to ChangeFinder (CF) signal which is one of change point detection algorithms1 (see Materials and methods).

For the death detection, the system calculates the CF signal from adult body segmentation (Appendix 8—figure 2a). For the pupariation and eclosion detections, two CF signals were calculated from adult and larva segmentations, respectively (Appendix 8—figure 2b).

Appendix 8—figure 2

Download asset Open asset

In the present system, the timing of life-event transitions is determined as the maximum point of corresponding CF scores (‘Max-CF’).

To evaluate robustness of detection, we compared several detection methods. Most simple way is just thresholding of the activity signal without change point detection such as CF. We calculated two types of signal from difference between consecutive images. One is the luminance difference (‘Luminance diff.’) obtained by subtraction between raw images, and another one is the subtraction between segmentation images before the CF (‘Seg. diff.’). Threshold for event timing were calculated as total average of the signals (‘auto’) and manual tuning (‘manual’). In comparison of these event detection methods, max-CF exhibited high accuracy in almost all datasets (Appendix 8—table 2).

Appendix 8—table 2

Pupariation
	Dataset 1 (allevent)	Dataset 2 (160416)	Dataset 3 (180223)
Luminance diff. thre (auto)	36.6% (102/279)	0.0% (0/255)	37.4% (92/246)
Luminance diff. thre (manual)	78.5% (219/279)	39.6% (101/255)	39.0% (96/246)
Segmentation diff. thre (auto)	96.4%** (269/279)	82.4%* (210/255)	94.7%** (233/246)
Segmentation diff. thre (manual)	96.8%** (270/279)	91.8%** (234/255)	96.3%** (237/246)
Maximum point of CF signal	97.5%** (271/279)	96.1%** (245/255)	92.3%** (227/246)
Eclosion
	Dataset 1 (allevent)	Dataset 2 (160416)	Dataset 3 (180223)
Luminance diff. thre (auto)	14.6% (34/233)	0.0% (0/241)	98.8%** (161/163)
Luminance diff. thre (manual)	60.1% (140/233)	2.9% (7/241)	98.8%** (161/163)
Segmentation diff. thre (auto)	82.0%* (191/233)	95.9%** (231/241)	99.4%** (162/163)
Segmentation diff. thre (manual)	85.4%** (199/233)	97.1%** (234/241)	99.4%** (162/163)
Maximum point of CF signal	86.3%* (201/233)	97.5%** (235/241)	99.4%** (162/163)
Death
	Dataset 1 (150625)	Dataset 2 (171013)	Dataset 3 (180223)
Luminance diff. thre (auto)	62.0% (237/382)	97.9%** (281/287)	10.3% (14/136)
Luminance diff. thre (manual)	94.0%** (359/382)	97.9%** (281/287)	10.3% (14/136)
Segmentation diff. thre (auto)	76.4% (292/382)	97.2%** (279/287)	99.3%** (135/136)
Segmentation diff. thre (manual)	93.5%** (357/382)	97.6%** (280/287)	100.0%** (136/136)
Maximum point of CF signal	94.8%** (362/382)	93.4%** (268/287)	98.5%** (134/136)

CF has two parameters, r and T. The r is a discounting rate and T is the length of a time window for averaging. We examined in robustness of detection accuracy for these two parameters in max-CF. For all event target and all datasets, max-CF demonstrated high-accuracy in wide range of the parameter, even the accuracy is relatively sensitive for discounting r than smoothing T (Appendix 8—figure 3–5).

Appendix 8—figure 3

Download asset Open asset

Present system used r = 0.003 except death detection in the datasets 1 (150626). In the data, r = 0.009 because the datasets has significantly few images.

8.4. Application of the algorithm

The system captures the animal body and its trajectory. Therefore, one could estimate behavioral history and total activity of individual Drosophila by displaying of trajectory and its summation in a well. (Video 1).

In addition, the system could demonstrate dynamic visualization of life-event transitions of Drosophila population (Appendix 8—video 1).

Appendix 8—video 1

Download asset

Through the visualization, the users could easily recognize timing differences between populations they were bled on different plates.

Appendix 8—figure 4

Download asset Open asset

Appendix 8—figure 5

Download asset Open asset

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

We are deeply grateful to Tadashi Uemura (Kyoto University) for comments, discussion and helpful supports. We thank Yuko Iijima, Eisuke Imura and Hsin Kuang Lin for their technical assistance. We are also grateful to Seino Masaki, Maki Otori, Mayu Kudo, Masaya Hata and Hiroki Watanabe for their technical support in image processing. I received generous support from Yoichi Shinkai (RIKEN CPR) and Shunsuke Ishii (RIKEN CPR).

Animal experimentation: This study was performed in strict accordance with the recommendations in the Guide for the Care and Use of Laboratory Animals of the RIKEN Institute.

© 2020, Seong et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.