Research in biology generates many image datasets, mostly from microscopy. These images have to be analyzed, and much of this analysis relies on a human expert looking at the images and manually annotating features. Image datasets are often large, and human annotation can be subjective, so automating image analysis is highly desirable. This is where machine learning algorithms, such as deep learning, have proven to be useful. In order for deep learning algorithms to work first they have to be ‘trained’. Deep learning algorithms are trained by being given a training dataset that has been annotated by human experts. The algorithms extract the relevant features to look out for from this training dataset and can then look for these features in other image data.

However, it is also worth noting that because these models try to mimic the annotation behavior presented to them during training as well as possible, they can sometimes also mimic an expert’s subjectivity when annotating data. Segebarth, Griebel et al. asked whether this was the case, whether it had an impact on the outcome of the image data analysis, and whether it was possible to avoid this problem when using deep learning for imaging dataset analysis.

For this research, Segebarth, Griebel et al. used microscopy images of mouse brain sections, where a protein called cFOS had been labeled with a fluorescent tag. This protein typically controls the rate at which DNA information is copied into RNA, leading to the production of proteins. Its activity can be influenced experimentally by testing the behaviors of mice. Thus, this experimental manipulation can be used to evaluate the results of deep learning-based image analyses.

First, the fluorescent images were interpreted manually by a group of human experts. Then, their results were used to train a large variety of deep learning models. Models were trained either on the results of an individual expert or on the results pooled from all experts to come up with a consensus model, a deep learning model that learned from the personal annotation preferences of all experts. This made it possible to test whether training a model on multiple experts reduces the risk of subjectivity. As the training of deep learning models is random, Segebarth, Griebel et al. also tested whether combining the predictions from multiple models in a so-called model ensemble improves the consistency of the analyses. For evaluation, the annotations of the deep learning models were compared to those of the human experts, to ensure that the results were not influenced by the subjective behavior of one person. The results of all bioimage annotations were finally compared to the experimental results from analyzing the mice’s behaviors in order to check whether the models were able to find the behavioral effect on cFOS.

Segebarth, Griebel et al. concluded that combining the expert knowledge of multiple experts reduces the subjectivity of bioimage annotation by deep learning algorithms. Combining such consensus information in a group of deep learning models improves the quality of bioimage analysis, so that the results are reliable, transparent and less subjective.

Modern microscopy methods enable researchers to capture images that describe cellular and molecular features in biological samples at an unprecedented scale. One of the most frequently used imaging methods is fluorescent labeling of biological macromolecules, both in vitro and in vivo. In order to test a biological hypothesis, fluorescent features have to be interpreted and analyzed quantitatively, a process known as bioimage analysis (Meijering et al., 2016). However, fluorescence does not provide clear signal-to-noise borders, forcing human experts to utilize individual heuristic criteria, such as morphology, size, or signal intensity to classify fluorescent signals as background, or to, often manually, annotate them as a region of interest (ROI). This cognitive decision process depends on the graphical perception capabilities of the individual annotator (Cleveland and McGill, 1985). Constant technological advances in fluorescence microscopy facilitate the automatized acquisition of large image datasets, even at high resolution and with high throughput (Li et al., 2010; McDole et al., 2018; Osten and Margrie, 2013). The ever increasing workload associated with image feature annotation therefore calls for computer-aided automated bioimage analysis. However, attempts to replace human experts and to automate the annotation process using traditional image thresholding techniques (e.g. histogram shape-, entropy-, or clustering-based methods [Sezgin and Sankur, 2004]) frequently lack flexibility, as they rely on a high signal-to-noise ratio in the images or require computational expertise for user-based adaptation to individual datasets (von Chamier et al., 2019). In recent years, deep learning (DL) and in particular deep convolutional neural networks have shown remarkable capacities in image recognition tasks, opening new possibilities to perform automatized image analysis. DL-based approaches have emerged as an alternative to conventional feature annotation or segmentation methods (Caicedo et al., 2019) and are even capable of performing complex tasks such as artificial labeling of plain bright-field images (von Chamier et al., 2019; Christiansen et al., 2018; Ounkomol et al., 2018). The main difference between conventional and DL algorithms is that conventional algorithms follow predefined rules (hard-coded), while DL algorithms are flexible to learn the respective task on base of a training dataset (LeCun et al., 2015). Yet, deployment of DL approaches necessitates both computational expertise and suitable computing resources. These requirements frequently prevent non-AI experts from applying DL to routine image analysis tasks. Initial efforts have already been made to break down these barriers, for instance, by integration into prevalent bioimaging tools such as ImageJ (Falk et al., 2019) and CellProfiler (McQuin et al., 2018), or using cloud-based approaches (Haberl et al., 2018). To harness the potentials of these DL-based methods, they require integration into the bioimage analysis pipeline. We argue that such an integration into the scientific process ultimately necessitates DL-based approaches to meet the same standards as any method in an empirical study. We can derive these standards from the general quality criteria of qualitative and quantitative research: objectivity, reliability, and validity (Frambach et al., 2013).

Objectivity refers to the neutrality of evidence, with the aim to reduce personal preferences, emotions, or simply limitations introduced by the context in which manual feature annotation is performed (Frambach et al., 2013). Manual annotation of fluorescent features has long been known to be subjective, especially in the case of weak signal-to-noise thresholds (Schmitz et al., 1999; Collier et al., 2003; Niedworok et al., 2016). Notably, there is no objective ground truth reference in the particular case of fluorescent label segmentation, causing a critical problem for training and evaluation of DL algorithms. As multiple studies have pointed out that annotations of low quality can cause DL algorithms to either fail to train or to reproduce inconsistent annotations on new data (von Chamier et al., 2019; Falk et al., 2019), this is a crucial obstacle for applying DL to bioimage analysis processes.

Reliability is concerned with the consistency of evidence (Frambach et al., 2013). To allow an unambiguous understanding of this concept, we further distinguish between repeatability and reproducibility. Repeatability or test-retest reliability is defined as 'closeness of the agreement between the results of successive measurements of the same measure and carried out under the same conditions' (Taylor and Kuyatt, 1994, 14), which is guaranteed for any deterministic DL model. Reproducibility, on the other hand, is specified as 'closeness of the agreement between the results of measurements of the same measure and carried out under changed conditions' (Taylor and Kuyatt, 1994, 14), for example, different observer, or different apparatus. This is a critical point, since the output of different DL models trained on the same training dataset can vary significantly. This is caused by the stochastic training procedure (e.g. random initialization, random sampling and data augmentation [Ronneberger et al., 2015]), the choice of model parameters (e.g. model architecture, weights, activation functions), and the choice of hyperparameters (e.g. learning rate, mini-batch size, training epochs). Consequently, the reproducibility of DL models merits careful investigation.

Finally, validity relates to the truth value of evidence, that is, whether we in fact measured what we intended to. Moreover, validity implies reliability - but not vice versa (Frambach et al., 2013). On a basis of a given ground truth, validity is typically measured using appropriate similarity measures such as F1 score for detection and Intersection over Union (IoU) for segmentation purposes (Ronneberger et al., 2015; Falk et al., 2019; Caicedo et al., 2019). In addition, the DL community has established widely accepted standards for training models. These comprise, among other things, techniques to avoid overfitting (regularization techniques and cross-validation), tuning hyperparameters, and selecting appropriate metrics for model evaluation. However, these standards do not apply for the training and evaluation of a DL model in the absence of a ground truth, like in the case of fluorescent features.

Taken together and with regard to the discussion about a reproducibility crisis in the fields of biology, medicine and artificial intelligence (Siebert et al., 2015; Baker, 2016; Ioannidis, 2016; Hutson, 2018; Fanelli, 2018; Chen et al., 2019), these limitations indicate that DL could aggravate this crisis by adding even more unknowns and uncertainties to bioimage analyses.

However, the present study asks whether DL, if instantiated in an appropriate manner, also holds the potential to instead enhance the objectivity, reproducibility and validity of bioimage analysis. To tackle this conundrum, we investigated different DL-based strategies on five fluorescence image datasets. We show that training of DL models on the pooled input of multiple human experts utilizing ground truth estimation (consensus models) increases objectivity of fluorescent feature segmentation. Furthermore, we demonstrate that ensembles of consensus models are even capable of enhancing the reliability and validity of bioimage analysis of ambiguous image data, such as fluorescence features in histological tissue sections.

In order to evaluate the impact of DL on bioimage analysis results, we instantiated three exemplary DL-based strategies (Figure 1; strategies color-coded in gray, blue, and orange) and investigate them in terms of objectivity, reliability, and validity of fluorescent feature annotation. The first strategy, expert models (gray), reflects mere automation of the annotation process of fluorescent features in microscopy images. Here, manual annotations of a single human expert are used to train an individual (and hence expert-specific) DL model with a U-Net (Ronneberger et al., 2015) architecture. U-Net and its variants have emerged as the de facto standard for biomedical image segmentation purposes (McQuin et al., 2018; Falk et al., 2019; Caicedo et al., 2019). The second strategy, consensus models (blue), addresses the objectivity of signal annotations. Contrary to the first strategy, simultaneous truth and performance level estimation (STAPLE) (Warfield et al., 2004) is used to estimate a ground truth and create consensus annotations. The estimated ground truth (est. GT) annotation reflects the pooled input of multiple human experts and is therefore thought to be less affected by a potential subjective bias of a single expert. We then train a single U-Net model to create a consensus model. The third strategy, consensus ensembles (orange), seeks to ensure reliability and eventually validity. Going beyond the second strategy, we train multiple consensus U-Net models to create a consensus ensemble. Such model ensembles are known to be more robust to noise (Dietterich, 2000). Hence, we hypothesize that the consensus ensembles mitigate the randomness in the training process. Moreover, deep ensembles are supposed to yield high-quality predictive uncertainty estimates (Lakshminarayanan et al., 2017).

Figure 1 with 2 supplements see all

Download asset Open asset

For each of the three strategies, we complete the bioimage analysis by performing quantification and hypothesis testing on a typical fluorescent microscopy image dataset (Figure 1—figure supplement 2). These images describe changes in fluorescence signal abundance of a protein called cFOS in brain sections of mice. cFOS is an activity-dependent transcription factor and its expression in the brain can be modified experimentally by behavioral testing of the animals (Gallo et al., 2018). The low signal-to-noise ratio of this label, its broad usage in neurobiology and the well-established correlation of its abundance with behavioral paradigms render it an ideal bioimage dataset to test our hypotheses (Shuvaev et al., 2017; Gallo et al., 2018).

Consensus ensembles yield the best results for validity and reproducibility metrics

The primary goal in bioimage analysis is to rigorously test a biological hypothesis. To leverage the potentials of DL models within this procedure, we need to trust our model – by establishing objectivity, reliability, and validity. Pertaining to the case of fluorescent labels, validity (measuring what is intended to be measured) requires objectivity to know what exactly we intend to measure in the absence of a ground truth. Similarly, reliability in terms of repeatability and reproducibility is a prerequisite for a valid and trustworthy model. Starting from the expert model strategy, we seek to establish objectivity (consensus models) and, successively, reliability and validity in the consensus ensemble strategy. In the following analysis, we first turn toward a comprehensive evaluation of the objectivity and its relation to validity before moving on to the concept of reliability.

To assess the three different strategies, a training dataset of 36 images and a test set of nine microscopy images (1024 × 1024 px, 1.61 px/µm, on average ∼35 nuclei per image, see also Figure 1—figure supplement 2) showing cFOS immunoreactivity were manually annotated by five independent experts (experts 1–5). In absence of a rigorously objective ground truth, we used STAPLE (Warfield et al., 2004) to compute an estimated ground truth (est. GT) based on all expert annotations for each image. First, we trained a set of DL models on the 36 training images and corresponding annotations, either made by an individual human expert or as reflected in the est. GT (see Materials and methods for the data set and detailed training, evaluation and model selection strategy). Then, we used our test set to evaluate the segmentation (Mean IoU) and detection (F1 score) performance of human experts and all trained models by means of similarity analysis on the level of individual images.

For the pairwise comparison of annotations (segmentation masks), we calculated the intersection over union (IoU) for all overlapping pairs of ROIs between two segmentation masks (Figure 2A; see 7.9.1 Segmentation and detection). Following Maška et al., 2014, we consider two ROIs with an IoU of at least 0.5 as matching and calculated the F1 score $M_{\text{F1 score}}$ as the harmonic mean of precision and recall (Figure 2B; see 7.9.1 Segmentation and detection). As bioimaging studies predominantly use measures related to counting ROIs in their analyses, we also focused on the feature detection performance ($M_{\text{F1 score}}$). The color coding (gray, blue, orange) introduced in Figure 2C refers to the different strategies depicted in Figure 1 and applies to all figures, if not indicated otherwise.

Figure 2 with 4 supplements see all

Download asset Open asset

To better grasp the difficulties in annotating cFOS-positive nuclei as fluorescent features in these images, we first compared manual expert annotations (Figure 2D). The analysis revealed substantial differences between the annotations of the different experts and shows varying inter-rater agreement (Schmitz et al., 1999; Collier et al., 2003; Niedworok et al., 2016). The level of inter-rater variability was inversely correlated with the relative signal intensities (Figure 2—figure supplement 1; Niedworok et al., 2016).

By comparing the annotations of the expert models (gray) to the annotations of the respective expert (Figure 2E), we observed a higher $M_{\text{F1 score}}$ median compared to the inter-rater agreement (Figure 2D) in the majority of cases. Furthermore, comparing the similarity analysis results of human experts with those of their respective expert-specific models revealed that they are closely related (Figure 2F, Figure 2—figure supplement 3, and Figure 2—figure supplement 4). As pointed out by von Chamier et al., 2019, this indicates that our expert models are able to learn and reproduce the annotation behavior of the individual experts. This becomes particularly evident in the annotations of the DL models trained on expert 1 (Figure 2F, Figure 2—figure supplement 3, and Figure 2—figure supplement 4).

Overall, the expert models yield a lower similarity to the est. GT compared to the consensus models (blue) or consensus ensembles (orange). Notably, both consensus models and consensus ensembles perform on par with human experts. Hereby, the consensus ensembles outperform all other strategies, even at varying IoU thresholds (Figure 2F and Figure 2G).

In order to test for reliability of our analysis, we measured the repeatability and reproducibility of fluorescent feature annotation of our DL strategies. We assumed that the repeatability is assured for all our strategies due to the deterministic nature of our DL models (unchanged conditions imply unchanged model weights). Hence, our evaluation was focused on the reproducibility, meaning the impact of the stochastic training process on the output. Inter-expert and inter-model comparisons within each strategy unveiled a better performance of the consensus ensembles strategy concerning both detection ($M_{\text{F1 score}}$) and segmentation (${\overline{M}}_{\text{IoU}}$) of the fluorescent features (Figure 2H). Calculating the Fleiss’ kappa value (Fleiss and Cohen, 1973) revealed that consensus ensemble annotations show a high reliability of agreement (Figure 2H). Following the Fleiss’ kappa interpretation from Landis and Koch, 1977, the results for the consensus ensembles indicate a substantial or almost perfect agreement. In contrast, the Fleiss’ kappa values for human experts refer to a fair agreement while the results for the alternative DL strategies lead to a moderate agreement (Figure 2H).

In summary, the similarity analysis of the three strategies shows that training of DL models solely on the input of a single human expert imposes a high risk of incorporating an intrinsic bias and therefore resembles, as hypothesized, a mere automation of manual image annotation. Both consensus models and consensus ensembles perform on par with human experts regarding the similarity to the est. GT, but the consensus ensembles yield by far the best results regarding their reproducibility. We conclude that, in terms of similarity metrics, only the consensus ensemble strategy meet the bioimaging standards for objectivity, reliability, and validity.

Consensus ensembles yield reliable bioimage analysis results

Similarity analysis is inevitable to assess the quality of a model’s output, that is, the predicted segmentations (Ronneberger et al., 2015; Caicedo et al., 2019; Falk et al., 2019). However, the primary goal of bioimage analysis is the unbiased quantification of distinct image features that correlate with experimental conditions. So far, it has remained unclear whether objectivity, reliability, and validity for bioimage analysis can be inferred directly from similarity metrics.

In order to systematically address this question, we used our image dataset to quantify the abundance of cFOS in brain sections of mice after Pavlovian contextual fear conditioning. It is well established in the neuroscientific literature that rodents show changes in the distribution and abundance of cFOS in a specific brain region, namely the hippocampus, after processing information about places and contexts (Keiser et al., 2017; Campeau et al., 1997; Huff et al., 2006; Ramamoorthi et al., 2011; Tayler et al., 2013; Murawski et al., 2012; Guzowski et al., 2001). Consequently, our experimental dataset offered us a second line of evidence, the objective analysis of mouse behavior, in addition to the changes of fluorescent features to validate the bioimage analyses results of our DL-based strategies.

Our dataset comprised three experimental groups (Figure 3A). In one group, mice were directly taken from their homecage as naive learning controls (H). In the second group, mice were re-exposed to a previously explored training context as context controls (C-). Mice in the third group underwent Pavlovian fear conditioning and were also re-exposed to the training context (C+) (Figure 3A). These three groups of mice showed different behavioral responses. For instance, fear (threat; LeDoux, 2014) conditioned mice (C+) showed increased freezing behavior after fear acquisition and showed strong freezing responses when re-exposed to the training context 24 hr later (Figure 3—figure supplement 1). After behavioral testing, brain sections of the different groups of mice were prepared and labeled for the neuronal activity-related protein cFOS by indirect immunofluorescence. Sections were also labeled with the neuronal marker NeuN (Fox3), allowing the anatomical identification of hippocampal subregions of interest. Images were acquired as confocal microscopy image stacks (x,y-z) and maximum intensity projections were used for subsequent bioimage analysis (Figure 1—figure supplement 2). Overall, we quantified the number of cFOS-positive nuclei and their mean signal intensity in five regions of the dorsal hippocampus (DG as a whole, suprapyramidal DG, infrapyramidal DG, CA3, and CA1), and tested for significant differences between the three experimental groups (Figure 3B–D). To extend this analysis beyond hypothesis testing at a certain significance level, we calculated the effect size (${\eta }^2$) for each of these 30 pairwise comparisons.

Figure 3 with 1 supplement see all

Download asset Open asset

Figure 3—source data 1

Source files for analyses of cFOS-positive nuclei in CA1.

https://cdn.elifesciences.org/articles/59780/elife-59780-fig3-data1-v2.zip

Download elife-59780-fig3-data1-v2.zip

We illustrate our metrics with the detailed quantification of cFOS-positive nuclei in the stratum pyramidale of CA1 as a representative example and show two analyses for each DL strategy (Figure 3E). These two examples represent those two models of each strategy that yielded the lowest and the highest effect sizes, respectively (Figure 3E). Despite a general consensus of all models and ensembles on a context-dependent increase in the number of cFOS-positive nuclei, these quantifications already indicate that the variability of effect sizes decreases from expert models to consensus models and is lowest for consensus ensembles (Figure 3E).

The analysis in Figure 4 allows us to further explore the impact of the different DL strategies on the bioimage analysis results for each hippocampal subregion. Here, we display a high-level comparison of the effect sizes and corresponding significance levels of 20 independently trained expert models (4 per expert), 36 consensus models, and 9 consensus ensembles (each derived from four consensus models). In contrast to the detailed illustration of selected models in Figure 3E, Figure 4A, for instance, summarizes the results for all analyses of the stratum pyramidale of CA1. As indicated before, all models and ensembles show a highly significant context-dependent increase in the number of cFOS-positive nuclei, but also a notable variation in effect sizes for both expert and consensus models. Moreover, we identify a significant context-dependent increase in the mean signal intensity of cFOS-positive nuclei for all consensus models and ensembles. The expert models, by contrast, yield a high variation in effect sizes at different significance levels. Interestingly, all four expert models trained on the annotations of expert 1 (and two other expert models only in the case of H vs. C+) did not yield a significant increase, indicating that expert 1’s annotation behavior was incorporated into the expert-1-specific models and that this also affects the bioimage analysis results (Figure 4A).

Figure 4

Download asset Open asset

Figure 4—source data 1

Source files for the analysis of cFOS positive nuclei in the hippocampal subregions.

https://cdn.elifesciences.org/articles/59780/elife-59780-fig4-data1-v2.zip

Download elife-59780-fig4-data1-v2.zip

Figure 4—source data 2

Statistical data for the variation per effect.

https://cdn.elifesciences.org/articles/59780/elife-59780-fig4-data2-v2.zip

Download elife-59780-fig4-data2-v2.zip

The meta analysis discloses a context-dependent increase of cFOS in almost all analyzed hippocampal regions (Figure 4A–D), except for the infrapyramidal blade of the dentate gyrus (Figure 4E). Notably, the majority votes of all three strategies at a significance level of p ≤ 0.05 (after Bonferroni correction for multiple comparisons) are identical for each pairwise comparison (Figure 4A–E). However, the results can vary between individual models or ensembles (Figure 4A–E).

In order to assess the reliability of bioimage analysis results of the individual strategies, we further examined the variation per effect and variation per model in Figure 4F. For the variation per effect, we calculated the standard deviation of reported effect sizes within each strategy for every pairwise comparison (effect). This confirmed the visual impression from Figure 4A–E as the consensus ensembles yield a significantly lower standard deviation compared to both alternative strategies (Figure 4F). To illustrate the variation per model, we show the interaction between the number of biological effects that the corresponding model (or ensemble) reported differently compared to the congruent majority votes versus the standard deviation of its centered effect sizes across all 30 analyzed effects. This analysis shows that no expert model detected all biological effects in the microscopy images that were defined by the majority votes of all models. This is in stark contrast to the consistency of effect interpretation across the consensus ensembles (Figure 4F).

Consequently, we conclude that the consensus ensemble strategy is best suited to satisfy the bioimaging standards for objectivity, reliability, and validity.

Applicability of consensus ensemble strategy for the bioimage analysis of external data sets

Bioimage analysis of fluorescent labels comes with a huge variability in terms of investigated model organisms, analyzed fluorescent features and applied image acquisition techniques (Meijering et al., 2016). In order to assess our consensus ensemble strategy across these varying parameters, we tested it on four external datasets that were created in a fully independent manner and according to individual protocols (Lab-Mue, Lab-Inns1, Lab-Inns2, and Lab-Wue2; see Materials and methods and Figure 5—source data 2). Due to limited dataset sizes, the lab-specific training datasets consisted of just five microscopy images each and the corresponding est. GT based on the annotations from multiple experts. In the biomedical research field, the limited availability of training data is a common problem when training DL algorithms. For this reason, extensive data augmentation and regularization techniques, as well as transfer learning strategies are widely used to cope with small datasets (Ronneberger et al., 2015; Christiansen et al., 2018; Falk et al., 2019). Transfer learning is a technique that enables DL models to reuse the image feature representations learned on another source, such as a task (e.g. image segmentation) or a domain (e.g. the fluorescent feature, here cFOS-positive nuclei). This is particularly advantageous when applied to a task or domain where limited training data is available (Yosinski et al., 2014; Oquab et al., 2014). Moreover, transfer learning might be used to reduce observer variability and to increase feature annotation objectivity (Bayramoglu and Heikkilä, 2016). There are typically two ways to implement transfer learning for DL models, either by fine-tuning or by freezing features (i.e. model weights) (Yosinski et al., 2014). The latter approach, if applied to the same task (e.g. image segmentation), does not require any further model training. These out-of-the-box models reduce time and hardware requirements and may further increase objectivity of image analysis, by altogether excluding the need for any additional manual input.

Consequently, we hypothesized that transfer learning from pretrained model ensembles would substantially reduce the training efforts (Falk et al., 2019) and might even increase objectivity of bioimage analysis. To test this, we followed three different initialization variants of the consensus ensemble strategy (Figure 5A). In addition to starting the training of DL models with randomly initialized weights (Figure 5A - from scratch), we reused the consensus ensemble weights from the previous evaluation (Lab-Wue1) by means of fine-tuning (Figure 5A - fine-tuned) and freezing of all model layers (Figure 5A - frozen). Although no training of the frozen model is required, we tested and evaluated the performance of frozen models to ensure their validity. After performing the similarity analysis, we compared the full bioimage analyses, including quantification and hypothesis testing, of the different initialization variants. Finally, to establish a notion of external validity, we also compared these results with the manually and independently performed bioimage analysis of a lab-specific expert (Figure 5, Figure 5—figure supplement 1, and Figure 5—figure supplement 2).

Figure 5 with 5 supplements see all

Download asset Open asset

Figure 5—source data 1

Statistical data for Lab-Mue, Lab-Wue2, Lab-Inns1, and Lab-Inns2.

https://cdn.elifesciences.org/articles/59780/elife-59780-fig5-data1-v2.zip

Download elife-59780-fig5-data1-v2.zip

Figure 5—source data 2

Characteristics of all five bioimage datasets.

https://cdn.elifesciences.org/articles/59780/elife-59780-fig5-data2-v2.docx

Download elife-59780-fig5-data2-v2.docx

Figure 5—source data 3

Model performance with selection criterion for Lab-Mue, Lab-Wue2, Lab-Inns1, and Lab-Inns2.

https://cdn.elifesciences.org/articles/59780/elife-59780-fig5-data3-v2.zip

Download elife-59780-fig5-data3-v2.zip

Dataset characteristics

The first dataset (Lab-Mue) represents very similar image parameters compared to our original Lab-Wue1 dataset (Figure 5C - Lab-Mue and Figure 5—source data 2). Mice experienced restraint stress and subsequent Pavlovian fear conditioning (cue-conditioning, tone-footshock association) and the number of cFOS-positive cells in the paraventricular thalamus (PVT) was compared between early (eRet) and late (lRET) phases of fear memory retrieval. In the context of transfer learning, this dataset originates from a very similar domain and requires the same task (image segmentation). Another two external datasets are focused on the quantification of cFOS abundance (similar domain), albeit showing less similarity in image parameters to our initial dataset (Figure 5—figure supplement 1, Figure 5—figure supplement 2 and Figure 5—source data 2). In Lab-Inns1, mice underwent Pavlovian fear conditioning and extinction in the same context. The image dataset of Lab-Inns2 shows cFOS immunoreactivity in the infralimbic cortex (IL) following fear renewal, meaning return of extinguished fear in a context different from the extinction training context. Since heterogeneity in this behavioral response was observed, mice were classified as responders (Resp) or non-responders (nResp), based on freezing responses (see Materials and methods). The image dataset of Lab-Wue2 shows the least similarity of image parameters to the dataset of Lab-Wue1. This dataset represents another commonly used model organism in neurobiology, the zebrafish. Here, cell bodies of specific neurons (GABAergic neurons) instead of nuclei were fluorescently labeled (Figure 5C - Lab-Wue2 and Figure 5—source data 2). Hence, this dataset originates from a different domain but was acquired using the same technique.

Similarity analysis

As only limited training data was available, we executed the similarity analysis for all external datasets by means of a k-fold cross-validation. We observed that the inter-rater variability differed between laboratories and different experts but remained comparable as previously for Lab-Wue1 (Figure 5D, Figure 5—figure supplement 3, and Figure 5—figure supplement 4.) Both from scratch and fine-tuned initiation variants resulted in individual consensus models that reached human expert level performance (Figure 5D, Figure 5—figure supplement 1, Figure 5—figure supplement 2). However, models adapted from pretrained weights yielded a higher validity in terms of similarity to the estimated ground truth. They either exceeded the maximal $M_{\text{F1 score}}$ reached by from scratch models (Figure 5D - Lab-Mue, Figure 5—figure supplement 1, Figure 5—figure supplement 2) or reached them after less training iterations (Figure 5D - Lab-Wue2). As expected, the performance of frozen Lab-Wue1-specific consensus models was highly dependent on the image similarity between the original and the new dataset. Consequently, the out-of-the-box segmentation performance of the frozen Lab-Wue1 models was very poor on dissimilar images (Figure 5D - Lab-Wue2), but we found it to be on par with human experts and adapted models on images that are highly similar to the original dataset (Figure 5D - Lab-Mue - very similar domain and the same task).

Bioimage analysis results

To further strengthen the validity of our workflow, we compared all DL-based bioimage analyses to the manual analysis of a human expert from the individual laboratory (Figure 5E, Figure 5—figure supplement 1, Figure 5—figure supplement 2, and Table 1).

Table 1

Lab	Groups	Initialization variant	${\mu }_1$	${\mu }_2$	U	Significance level (p)	${\eta }^2$
Mue	eRet ∼ lRet	Manual	1.00	1.65	19.0	** (0.002)	0.39
		From scratch	1.00	1.70	25.0	** (0.007)	0.31
		Fine-tuned	1.00	1.68	24.0	** (0.006)	0.32
Inns1	Ctrl ∼ Ext	Manual	1.00	3.92	10.0	** (0.005)	0.43
		From scratch	1.00	2.26	13.0	* (0.010)	0.35
		Fine-tuned	1.00	1.85	14.0	* (0.013)	0.33
Inns2	Sal ∼ Resp	Manual	1.00	1.83	5.0	** (0.002)	0.59
		From scratch	1.00	1.96	0.0	*** (<0.001)	0.71
		Fine-tuned	1.00	2.07	0.0	*** (<0.001)	0.71
	Sal ∼ nResp	Manual	1.00	1.05	27.0	n.s. (1.000)	0.00
		From scratch	1.00	1.63	8.0	n.s. (0.130)	0.29
		Fine-tuned	1.00	1.42	12.0	n.s. (0.377)	0.16
	Res ∼ nRes	Manual	1.83	1.05	42.0	n.s. (0.130)	0.29
		From scratch	1.96	1.63	41.0	n.s. (0.173)	0.26
		Fine-tuned	2.07	1.42	42.0	n.s. (0.130)	0.29
Wue2	wt ∼ kd	Manual	1.00	0.28	227.5	* (0.010)	0.19
		From scratch	1.00	0.45	220.0	* (0.021)	0.16
		Fine-tuned	1.00	0.37	216.0	* (0.029)	0.14

For Lab-Mue, the bioimage analyses of all DL-based approaches, including the frozen consensus models and ensembles pretrained on Lab-Wue1, revealed a significantly higher number of cFOS-positive cells in the PVT of mice 24 hr after fear conditioning (lRET), which was confirmed by the manual expert analysis (Figure 5E - Lab-Mue, Table 1). Yet again, the formation of model ensembles increased the reproducibility of results by yielding less or almost no variation in the effect sizes (Figure 5E - Lab-Mue).

The manual expert analysis of the Lab-Inns1 dataset revealed a significantly higher number of cFOS-positive nuclei in the basolateral amygdala (BLA) after extinction of a previously learned fear, which was also reliably detected by all consensus ensembles, regardless of initiation variant (Figure 5—figure supplement 1, Table 1). However, this significant difference was only present in the analyses of most individual consensus models, both from scratch and fine-tuned (Figure 5—figure supplement 1). Again, this could be attributed to a higher variability between the effect sizes of individual models, compared to a higher homogeneity among ensembles (Figure 5—figure supplement 1).

For Lab-Inns2, the manual expert analysis as well as all DL-based approaches that were adapted to the Lab-Inns2 dataset show increased numbers of cFOS-positive cells in the infralimbic cortex of L-DOPA/MS-275 responders (Resp) compared to control (Sal) mice (Figure 5—figure supplement 2, Table 1). However, in L-DOPA/MS-275 non-responders (nResp), we did not observe a significant increase of cFOS-positive nuclei (Figure 5—figure supplement 2, Table 1). Furthermore, the high effect sizes of the comparison between L-DOPA/MS-275 responders and non-responders further indicate that the differences observed in the behavioral responses of Resp and nResp mice were also reflected in the abundance of cFOS in the infralimbic cortex (Figure 5—figure supplement 2, Table 1).

Manual expert analysis of the fourth external dataset revealed a significantly lower amount of GABA-positive somata in gad1b knock-down zebrafish, compared to wildtypes (Figure 5E - Lab-Wue2, Table 1). Again, this effect was reliably detected by all deep-learning-based approaches that included training on the Lab-Wue2-specific training dataset and the effect sizes of ensembles showed less variability (Figure 5E - Lab-Wue2). Despite its poor segmentation performance and hence, poor validity, this effect was also present in the bioimage analysis of the frozen consensus models and ensembles pretrained on Lab-Wue1 (Figure 5E - Lab-Wue2).

As with our initial dataset, we assessed reliability by calculating the variation per effect as the standard deviation of the reported effect sizes within each group and pooled these results across all external datasets. Consistent with the higher reliability of from scratch and fine-tuned ensemble annotations (Figure 5—figure supplement 5), this analysis shows that the formation of model ensembles reduced the variation per effect in both variants, compared to the respective individual models (Figure 5B). The frozen models and ensembles exhibit a similar effect, but need to be considered with caution as they are based on models that did not meet the selection criterion (reliably performing on par with human experts; see 7.10.4 - Training, evaluation and model selection for a detailed explanation).

In summary, we assessed the reproducibility of our consensus ensemble strategy by using four external datasets. These datasets were acquired with different image acquisition techniques, investigate two common model organisms, and analyze the two main cellular compartments (nuclei and somata) at varying resolutions (Figure 5—source data 2). In-line with the results obtained on our initial dataset, we observed an increased reproducibility for the consensus ensembles compared to individual consensus models after training on all four external datasets (Figure 5B).

Moreover, our data also suggests that pretrained consensus models can even be deployed out-of-the-box, but only when carefully validated. Thus, sharing pretrained model weights across different laboratories reduces lab-specific biases within the bioimage analysis and may further increase objectivity and validity.

Ultimately, we conclude that our proposed ensemble consensus workflow is reproducible for different datasets and laboratories and increases objectivity, reliability, and validity of DL-based bioimage analyses.

The present study contributes to bridging the gap between ‘methods’ and ‘biology’ oriented studies in image feature analysis (Meijering et al., 2016). We explored the potentials and limitations of DL models utilizing the general quality criteria for quantitative research: objectivity, reliability, and validity. Thereby, we put forward an effective but easily implementable strategy that aims to establish reproducible, DL-based bioimage analysis within the life science community.

The number of DL-based tools for bioimage annotations and their accessibility for non-AI specialists is gradually increasing (McQuin et al., 2018; Haberl et al., 2018; Falk et al., 2019). DL models can hold advantages over conventional algorithms (Caicedo et al., 2019) and have the potential to be commonly used for bioimage analysis tasks throughout the life sciences. Usually, the performance of new bioimage analysis tools or methods is assessed by means of similarity measures to a certain ground truth (Ronneberger et al., 2015; McQuin et al., 2018; Haberl et al., 2018; Falk et al., 2019; Caicedo et al., 2019). However, this is rarely sufficient to establish trust in the use of DL models for bioimage analysis, as the vast amount of parameters and flexibility to adapt DL models to virtually any task renders them prone to internalize unintended, but subjective human biases (von Chamier et al., 2019). This is particularly true in the case of fluorescent feature analysis in bioimage datasets, as an objective ground truth is not available. In conjunction with the stochastic training process, this is a very critical point, because it holds the potential for intended or unintended tampering similar to p-hacking (Head et al., 2015), for example by training DL models until non-significant results become significant.

To investigate the effects of DL-based strategies on the bioimage analysis of fluorescent features, we acquired a typical bioimage dataset (Lab-Wue1) and five experts manually annotated corresponding ROIs (here cFOS-positive nuclei) in a representative subset of images. Then, we tested three DL-based strategies for automatized feature segmentation. DL models were either trained on the manual annotations of a single expert (expert models) or on the input of multiple experts pooled by ground truth estimation (consensus models). In addition, we formed ensembles of consensus models (consensus ensembles).

Data sets regarding animal behavior, immunofluorescence analysis and image acquisition were performed in five independent laboratories using lab-specific protocols. Experiments were not planned together to ensure the individual character of the datasets. We refer to the lab-specific protocols as follows:

• Lab-Mue: Institute of Physiology I, University of Münster, Germany

• Lab-Inns1: Department of Pharmacology, Medical University of Innsbruck, Austria

• Lab-Inns2: Department of Pharmacology and Toxicology, Institute of Pharmacy and Center for Molecular Biosciences Innsbruck, University of Innsbruck

• Lab-Wue1: Institute of Clinical Neurobiology, University Hospital, Würzburg, Germany

• Lab-Wue2: Department of Child and Adolescent Psychiatry, Center of Mental Health, University Hospital of Würzburg, Würzburg, Germany

The official repository of our study "On the objectivity, reliability, and validity of deep learning enabled bioimage analyses" can be found at Dryad (https://doi.org/10.5061/dryad.4b8gtht9d). In addition, we also provide all code in our GitHub repository (https://github.com/matjesg/bioimage_analysis). A copy is archived at https://archive.softwareheritage.org/swh:1:rev:eafeb5f8e1312ab29416144df0212761ddf4cfc4/.

1. 1. Segebarth D
   2. Griebel M
   3. Stein N
   4. Martin C
   5. Fiedler D
   6. Comeras LB
   7. Sah A
   8. Schoeffler V
   9. Lüffe T
   10. Dürr A
   11. Gupta R
   12. Sasi M
   13. Lillesaar C
   14. Lange MD
   15. Tasan RO
   16. Singewald N
   17. Pape HC
   18. Flath CM
   19. Blum R
   20. von Collenberg CR

   (2020) Dryad Digital Repository

   Data from: On the objectivity, reliability, and validity of deep learning enabled bioimage analyses.

   https://doi.org/10.5061/dryad.4b8gtht9d

1. Preprint

   1. Abadi M
   2. Agarwal A
   3. Barham P
   4. Brevdo E
   5. Chen Z
   6. Citro C
   7. Corrado GS
   8. Davis A
   9. Dean J
   10. Devin M
   11. Ghemawat S
   12. Goodfellow I
   13. Harp A
   14. Irving G
   15. Isard M
   16. Jia Y
   17. Jozefowicz R
   18. Kaiser L
   19. Kudlur M
   20. Levenberg J
   21. Mane D
   22. Monga R
   23. Moore S
   24. Murray D
   25. Olah C
   26. Schuster M
   27. Shlens J
   28. Steiner B
   29. Sutskever I
   30. Talwar K
   31. Tucker P
   32. Vanhoucke V
   33. Vasudevan V
   34. Viegas F
   35. Vinyals O
   36. Warden P
   37. Wattenberg M
   38. Wicke M
   39. Yu Y
   40. Zheng X

   (2016) Tensorflow: large-scale machine learning on heterogeneous distributed systems

   arXiv.

   https://arxiv.org/abs/1603.04467

   • Google Scholar
2. 1. Baker M

   (2016)

   Reproducibility crisis?

   Nature 533:353–366.

   • Google Scholar
3. Conference

   1. Bayramoglu N
   2. Heikkilä J

   (2016)

   Transfer learning for cell nuclei classification in histopathology images

   European Conference on Computer Vision. pp. 532–539.

   • Google Scholar
4. 1. Caicedo JC
   2. Goodman A
   3. Karhohs KW
   4. Cimini BA
   5. Ackerman J
   6. Haghighi M
   7. Heng C
   8. Becker T
   9. Doan M
   10. McQuin C
   11. Rohban M
   12. Singh S
   13. Carpenter AE

   (2019) Nucleus segmentation across imaging experiments: the 2018 data science bowl

   Nature Methods 16:1247–1253.

   https://doi.org/10.1038/s41592-019-0612-7

   • PubMed
   • Google Scholar
5. 1. Campeau S
   2. Falls WA
   3. Cullinan WE
   4. Helmreich DL
   5. Davis M
   6. Watson SJ

   (1997) Elicitation and reduction of fear: behavioural and neuroendocrine indices and brain induction of the immediate-early gene c-fos

   Neuroscience 78:1087–1104.

   https://doi.org/10.1016/S0306-4522(96)00632-X

   • PubMed
   • Google Scholar
6. 1. Chauveau F
   2. Lange MD
   3. Jüngling K
   4. Lesting J
   5. Seidenbecher T
   6. Pape HC

   (2012) Prevention of stress-impaired fear extinction through neuropeptide s action in the lateral amygdala

   Neuropsychopharmacology 37:1588–1599.

   https://doi.org/10.1038/npp.2012.3

   • PubMed
   • Google Scholar
7. 1. Chen X
   2. Dallmeier-Tiessen S
   3. Dasler R
   4. Feger S
   5. Fokianos P
   6. Gonzalez JB
   7. Hirvonsalo H
   8. Kousidis D
   9. Lavasa A
   10. Mele S
   11. Rodriguez DR
   12. Šimko T
   13. Smith T
   14. Trisovic A
   15. Trzcinska A
   16. Tsanaktsidis I
   17. Zimmermann M
   18. Cranmer K
   19. Heinrich L
   20. Watts G
   21. Hildreth M
   22. Lloret Iglesias L
   23. Lassila-Perini K
   24. Neubert S

   (2019) Open is not enough

   Nature Physics 15:113–119.

   https://doi.org/10.1038/s41567-018-0342-2

   • Google Scholar
8. Website

   1. Chollet F

   (2015) Keras

   Accessed November 19, 2020.

   https://keras.io
9. 1. Christiansen EM
   2. Yang SJ
   3. Ando DM
   4. Javaherian A
   5. Skibinski G
   6. Lipnick S
   7. Mount E
   8. O'Neil A
   9. Shah K
   10. Lee AK
   11. Goyal P
   12. Fedus W
   13. Poplin R
   14. Esteva A
   15. Berndl M
   16. Rubin LL
   17. Nelson P
   18. Finkbeiner S

   (2018) In Silico labeling: predicting fluorescent labels in unlabeled images

   Cell 173:792–803.

   https://doi.org/10.1016/j.cell.2018.03.040

   • PubMed
   • Google Scholar
10. 1. Cleveland WS
    2. McGill R

    (1985) Graphical perception and graphical methods for analyzing scientific data

    Science 229:828–833.

    https://doi.org/10.1126/science.229.4716.828

    • PubMed
    • Google Scholar
11. 1. Collier DC
    2. Burnett SS
    3. Amin M
    4. Bilton S
    5. Brooks C
    6. Ryan A
    7. Roniger D
    8. Tran D
    9. Starkschall G

    (2003) Assessment of consistency in contouring of normal-tissue anatomic structures

    Journal of Applied Clinical Medical Physics 4:17–24.

    https://doi.org/10.1120/jacmp.v4i1.2538

    • PubMed
    • Google Scholar
12. Conference

    1. Dietterich TG

    (2000)

    Ensemble methods in machine learning

     International Workshop on Multiple Classifier Systems. pp. 1–15.

    • Google Scholar
13. 1. Driscoll MK
    2. Welf ES
    3. Jamieson AR
    4. Dean KM
    5. Isogai T
    6. Fiolka R
    7. Danuser G

    (2019) Robust and automated detection of subcellular morphological motifs in 3D microscopy images

    Nature Methods 16:1037–1044.

    https://doi.org/10.1038/s41592-019-0539-z

    • PubMed
    • Google Scholar
14. 1. Falk T
    2. Mai D
    3. Bensch R
    4. Çiçek Ö
    5. Abdulkadir A
    6. Marrakchi Y
    7. Böhm A
    8. Deubner J
    9. Jäckel Z
    10. Seiwald K
    11. Dovzhenko A
    12. Tietz O
    13. Dal Bosco C
    14. Walsh S
    15. Saltukoglu D
    16. Tay TL
    17. Prinz M
    18. Palme K
    19. Simons M
    20. Diester I
    21. Brox T
    22. Ronneberger O

    (2019) U-Net: deep learning for cell counting, detection, and morphometry

    Nature Methods 16:67–70.

    https://doi.org/10.1038/s41592-018-0261-2

    • PubMed
    • Google Scholar
15. 1. Fanelli D

    (2018) Opinion: is science really facing a reproducibility crisis, and do we need it to?

    PNAS 115:2628–2631.

    https://doi.org/10.1073/pnas.1708272114

    • PubMed
    • Google Scholar
16. 1. Fanselow MS

    (1980) Conditioned and unconditional components of post-shock freezing

    The Pavlovian Journal of Biological Science 15:177–182.

    https://doi.org/10.1007/BF03001163

    • PubMed
    • Google Scholar
17. 1. Feldman-Stewart D
    2. Kocovski N
    3. McConnell BA
    4. Brundage MD
    5. Mackillop WJ

    (2000) Perception of quantitative information for treatment decisions

    Medical Decision Making 20:228–238.

    https://doi.org/10.1177/0272989X0002000208

    • PubMed
    • Google Scholar
18. 1. Fitzgerald PJ
    2. Whittle N
    3. Flynn SM
    4. Graybeal C
    5. Pinard CR
    6. Gunduz-Cinar O
    7. Kravitz AV
    8. Singewald N
    9. Holmes A

    (2014) Prefrontal single-unit firing associated with deficient extinction in mice

    Neurobiology of Learning and Memory 113:69–81.

    https://doi.org/10.1016/j.nlm.2013.11.002

    • PubMed
    • Google Scholar
19. 1. Fleiss JL
    2. Cohen J

    (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability

    Educational and Psychological Measurement 33:613–619.

    https://doi.org/10.1177/001316447303300309

    • Google Scholar
20. 1. Frambach JM
    2. van der Vleuten CP
    3. Durning SJ

    (2013) Am last page: quality criteria in qualitative and quantitative research

    Academic Medicine : Journal of the Association of American Medical Colleges 88:552.

    https://doi.org/10.1097/ACM.0b013e31828abf7f

    • PubMed
    • Google Scholar
21. Book

    1. Franklin KBJ
    2. Paxinos G

    (2008)

    The Mouse Brain in Stereotaxic Coordinates

    Elsevier.

    • Google Scholar
22. 1. Gallo FT
    2. Katche C
    3. Morici JF
    4. Medina JH
    5. Weisstaub NV

    (2018) Immediate early genes, memory and psychiatric disorders: focus on c-Fos, Egr1 and arc

    Frontiers in Behavioral Neuroscience 12:79.

    https://doi.org/10.3389/fnbeh.2018.00079

    • PubMed
    • Google Scholar
23. 1. Gruene TM
    2. Flick K
    3. Stefano A
    4. Shea SD
    5. Shansky RM

    (2015) Sexually divergent expression of active and passive conditioned fear responses in rats

    eLife 4:e11352.

    https://doi.org/10.7554/eLife.11352

    • PubMed
    • Google Scholar
24. Conference

    1. Guan MY
    2. Gulshan V
    3. Dai AM
    4. Hinton GE

    (2018)

    Who said what: modeling individual labelers improves classification

    Thirty-Second AAAI Conference on Artificial Intelligence. pp. 3109–3118.

    • Google Scholar
25. 1. Guzowski JF
    2. Setlow B
    3. Wagner EK
    4. McGaugh JL

    (2001)

    Experience-dependent gene expression in the rat Hippocampus after spatial learning: a comparison of the immediate-early genes arc, c-fos, and zif268

    The Journal of Neuroscience 21:5089–5098.

    • PubMed
    • Google Scholar
26. 1. Haberl MG
    2. Churas C
    3. Tindall L
    4. Boassa D
    5. Phan S
    6. Bushong EA
    7. Madany M
    8. Akay R
    9. Deerinck TJ
    10. Peltier ST
    11. Ellisman MH

    (2018) CDeep3M-Plug-and-Play cloud-based deep learning for image segmentation

    Nature Methods 15:677–680.

    https://doi.org/10.1038/s41592-018-0106-z

    • PubMed
    • Google Scholar
27. Conference

    1. He K
    2. Zhang X
    3. Ren S
    4. Sun J

    (2015) Delving deep into rectifiers: surpassing human-level performance on imagenet classification

    Proceedings of the IEEE International Conference on Computer Vision. pp. 1026–1034.

    https://doi.org/10.1109/ICCV.2015.123

    • Google Scholar
28. 1. Head ML
    2. Holman L
    3. Lanfear R
    4. Kahn AT
    5. Jennions MD

    (2015) The extent and consequences of p-hacking in science

    PLOS Biology 13:e1002106.

    https://doi.org/10.1371/journal.pbio.1002106

    • PubMed
    • Google Scholar
29. 1. Huff NC
    2. Frank M
    3. Wright-Hardesty K
    4. Sprunger D
    5. Matus-Amat P
    6. Higgins E
    7. Rudy JW

    (2006) Amygdala regulation of immediate-early gene expression in the Hippocampus induced by contextual fear conditioning

    Journal of Neuroscience 26:1616–1623.

    https://doi.org/10.1523/JNEUROSCI.4964-05.2006

    • PubMed
    • Google Scholar
30. Conference

    1. Hutson M

    (2018)

    Artificial intelligence faces reproducibility crisis

    American Association for the Advancement of Science.

    • Google Scholar
31. 1. Ioannidis JP

    (2016) Why most clinical research is not useful

    PLOS Medicine 13:e1002049.

    https://doi.org/10.1371/journal.pmed.1002049

    • PubMed
    • Google Scholar
32. Preprint

    1. Ioffe S
    2. Szegedy C

    (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift

    arXiv.

    https://arxiv.org/abs/1502.03167

    • Google Scholar
33. Report

    1. Jones E
    2. Oliphant T
    3. Peterson P

    (2001)

    SciPy: Open Source Scientific Tools for Python

    Technical report.

    • Google Scholar
34. 1. Keiser AA
    2. Turnbull LM
    3. Darian MA
    4. Feldman DE
    5. Song I
    6. Tronson NC

    (2017) Sex differences in context fear generalization and recruitment of Hippocampus and amygdala during retrieval

    Neuropsychopharmacology 42:397–407.

    https://doi.org/10.1038/npp.2016.174

    • PubMed
    • Google Scholar
35. 1. Kimmel CB
    2. Ballard WW
    3. Kimmel SR
    4. Ullmann B
    5. Schilling TF

    (1995) Stages of embryonic development of the zebrafish

    Developmental Dynamics 203:253–310.

    https://doi.org/10.1002/aja.1002030302

    • PubMed
    • Google Scholar
36. Preprint

    1. Kingma DP
    2. Ba J

    (2014) Adam: a method for stochastic optimization

    arXiv.

    https://arxiv.org/abs/1412.6980

    • Google Scholar
37. 1. Kuhn HW

    (1955) The hungarian method for the assignment problem

    Naval Research Logistics Quarterly 2:83–97.

    https://doi.org/10.1002/nav.3800020109

    • Google Scholar
38. Conference

    1. Lakshminarayanan B
    2. Pritzel A
    3. Blundell C

    (2017)

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Advances in Neural Information Processing Systems. pp. 6402–6413.

    • Google Scholar
39. 1. Lampert TA
    2. Stumpf A
    3. Gancarski P

    (2016) An empirical study into annotator agreement, ground truth estimation, and algorithm evaluation

    IEEE Transactions on Image Processing 25:2557–2572.

    https://doi.org/10.1109/TIP.2016.2544703

    • PubMed
    • Google Scholar
40. 1. Landis JR
    2. Koch GG

    (1977) The measurement of observer agreement for categorical data

    Biometrics 33:159–174.

    https://doi.org/10.2307/2529310

    • PubMed
    • Google Scholar
41. 1. LeCun Y
    2. Bengio Y
    3. Hinton G

    (2015) Deep learning

    Nature 521:436–444.

    https://doi.org/10.1038/nature14539

    • PubMed
    • Google Scholar
42. 1. LeDoux JE

    (2014) Coming to terms with fear

    PNAS 111:2871–2878.

    https://doi.org/10.1073/pnas.1400335111

    • Google Scholar
43. 1. Li A
    2. Gong H
    3. Zhang B
    4. Wang Q
    5. Yan C
    6. Wu J
    7. Liu Q
    8. Zeng S
    9. Luo Q

    (2010) Micro-optical sectioning tomography to obtain a high-resolution atlas of the mouse brain

    Science 330:1404–1408.

    https://doi.org/10.1126/science.1191776

    • PubMed
    • Google Scholar
44. 1. Lowekamp BC
    2. Chen DT
    3. Ibáñez L
    4. Blezek D

    (2013) The design of SimpleITK

    Frontiers in Neuroinformatics 7:45.

    https://doi.org/10.3389/fninf.2013.00045

    • PubMed
    • Google Scholar
45. 1. Maška M
    2. Ulman V
    3. Svoboda D
    4. Matula P
    5. Matula P
    6. Ederra C
    7. Urbiola A
    8. España T
    9. Venkatesan S
    10. Balak DM
    11. Karas P
    12. Bolcková T
    13. Streitová M
    14. Carthel C
    15. Coraluppi S
    16. Harder N
    17. Rohr K
    18. Magnusson KE
    19. Jaldén J
    20. Blau HM
    21. Dzyubachyk O
    22. Křížek P
    23. Hagen GM
    24. Pastor-Escuredo D
    25. Jimenez-Carretero D
    26. Ledesma-Carbayo MJ
    27. Muñoz-Barrutia A
    28. Meijering E
    29. Kozubek M
    30. Ortiz-de-Solorzano C

    (2014) A benchmark for comparison of cell tracking algorithms

    Bioinformatics 30:1609–1617.

    https://doi.org/10.1093/bioinformatics/btu080

    • PubMed
    • Google Scholar
46. 1. McDole K
    2. Guignard L
    3. Amat F
    4. Berger A
    5. Malandain G
    6. Royer LA
    7. Turaga SC
    8. Branson K
    9. Keller PJ

    (2018) In toto imaging and reconstruction of Post-Implantation mouse development at the Single-Cell level

    Cell 175:859–876.

    https://doi.org/10.1016/j.cell.2018.09.031

    • PubMed
    • Google Scholar
47. 1. McQuin C
    2. Goodman A
    3. Chernyshev V
    4. Kamentsky L
    5. Cimini BA
    6. Karhohs KW
    7. Doan M
    8. Ding L
    9. Rafelski SM
    10. Thirstrup D
    11. Wiegraebe W
    12. Singh S
    13. Becker T
    14. Caicedo JC
    15. Carpenter AE

    (2018) CellProfiler 3.0: next-generation image processing for biology

    PLOS Biology 16:e2005970.

    https://doi.org/10.1371/journal.pbio.2005970

    • PubMed
    • Google Scholar
48. 1. Meijering E
    2. Carpenter AE
    3. Peng H
    4. Hamprecht FA
    5. Olivo-Marin JC

    (2016) Imagining the future of bioimage analysis

    Nature Biotechnology 34:1250–1255.

    https://doi.org/10.1038/nbt.3722

    • PubMed
    • Google Scholar
49. 1. Murawski NJ
    2. Klintsova AY
    3. Stanton ME

    (2012) Neonatal alcohol exposure and the Hippocampus in developing male rats: effects on behaviorally induced CA1 c-Fos expression, CA1 pyramidal cell number, and contextual fear conditioning

    Neuroscience 206:89–99.

    https://doi.org/10.1016/j.neuroscience.2012.01.006

    • PubMed
    • Google Scholar
50. 1. Murphy CP
    2. Li X
    3. Maurer V
    4. Oberhauser M
    5. Gstir R
    6. Wearick-Silva LE
    7. Viola TW
    8. Schafferer S
    9. Grassi-Oliveira R
    10. Whittle N
    11. Hüttenhofer A
    12. Bredy TW
    13. Singewald N

    (2017) MicroRNA-Mediated rescue of fear extinction memory by miR-144-3p in Extinction-Impaired mice

    Biological Psychiatry 81:979–989.

    https://doi.org/10.1016/j.biopsych.2016.12.021

    • PubMed
    • Google Scholar
51. 1. Niedworok CJ
    2. Brown AP
    3. Jorge Cardoso M
    4. Osten P
    5. Ourselin S
    6. Modat M
    7. Margrie TW

    (2016) aMAP is a validated pipeline for registration and segmentation of high-resolution mouse brain data

    Nature Communications 7:11879.

    https://doi.org/10.1038/ncomms11879

    • PubMed
    • Google Scholar
52. Conference

    1. Oquab M
    2. Bottou L
    3. Laptev I
    4. Sivic J

    (2014) Learning and transferring mid-level image representations using convolutional neural networks

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1717–1724.

    https://doi.org/10.1109/CVPR.2014.222

    • Google Scholar
53. 1. Osten P
    2. Margrie TW

    (2013) Mapping brain circuitry with a light microscope

    Nature Methods 10:515–523.

    https://doi.org/10.1038/nmeth.2477

    • PubMed
    • Google Scholar
54. 1. Ounkomol C
    2. Seshamani S
    3. Maleckar MM
    4. Collman F
    5. Johnson GR

    (2018) Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy

    Nature Methods 15:917–920.

    https://doi.org/10.1038/s41592-018-0111-2

    • PubMed
    • Google Scholar
55. 1. Pedregosa F
    2. Varoquaux G
    3. Gramfort A
    4. Michel V
    5. Thirion B
    6. Grisel O
    7. Blondel M
    8. Prettenhofer P
    9. Weiss R
    10. Dubourg V
    11. Vanderplas J
    12. Passos A
    13. Cournapeau D
    14. Brucher M
    15. Perrot M
    16. Duchesnay E

    (2011)

    Scikit-learn: machine learning in Python

    Journal of Machine Learning Research 12:2825–2830.

    • Google Scholar
56. 1. Perkel JM

    (2018) Why Jupyter is data scientists' computational notebook of choice

    Nature 563:145–146.

    https://doi.org/10.1038/d41586-018-07196-1

    • PubMed
    • Google Scholar
57. 1. Ramamoorthi K
    2. Fropf R
    3. Belfort GM
    4. Fitzmaurice HL
    5. McKinney RM
    6. Neve RL
    7. Otto T
    8. Lin Y

    (2011) Npas4 regulates a transcriptional program in CA3 required for contextual memory formation

    Science 334:1669–1675.

    https://doi.org/10.1126/science.1208049

    • PubMed
    • Google Scholar
58. Conference

    1. Ronneberger O
    2. Fischer P
    3. Brox T

    (2015) U-net: convolutional networks for biomedical image segmentation

    International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 234–241.

    https://doi.org/10.1007/978-3-319-24574-4_28

    • Google Scholar
59. Book

    1. Rosenthal R
    2. DiMatteo MR

    (2002) Stevens’ Handbook of Experimental Psychology

    Wiley.

    https://doi.org/10.1002/9781119170174

    • Google Scholar
60. 1. Rueden CT
    2. Schindelin J
    3. Hiner MC
    4. DeZonia BE
    5. Walter AE
    6. Arena ET
    7. Eliceiri KW

    (2017) ImageJ2: imagej for the next generation of scientific image data

    BMC Bioinformatics 18:529.

    https://doi.org/10.1186/s12859-017-1934-z

    • PubMed
    • Google Scholar
61. 1. Schmitz C
    2. Korr H
    3. Heinsen H

    (1999) Design-based counting techniques: the real problems

    Trends in Neurosciences 22:345.

    https://doi.org/10.1016/S0166-2236(99)01418-6

    • PubMed
    • Google Scholar
62. 1. Schneider CA
    2. Rasband WS
    3. Eliceiri KW

    (2012) NIH image to ImageJ: 25 years of image analysis

    Nature Methods 9:671–675.

    https://doi.org/10.1038/nmeth.2089

    • PubMed
    • Google Scholar
63. Software

    1. Segebarth D

    (2020) bioimage_analysis, version eafeb5f

    GitHub.

    https://github.com/matjesg/bioimage_analysis
64. 1. Sezgin M
    2. Sankur B

    (2004) Survey over image thresholding techniques and quantitative performance evaluation

    Journal of Electronic Imaging 13:146–166.

    https://doi.org/10.1117/1.1631315

    • Google Scholar
65. 1. Shuvaev SA
    2. Lazutkin AA
    3. Kedrov AV
    4. Anokhin KV
    5. Enikolopov GN
    6. Koulakov AA

    (2017) DALMATIAN: an algorithm for automatic cell detection and counting in 3D

    Frontiers in Neuroanatomy 11:117.

    https://doi.org/10.3389/fnana.2017.00117

    • PubMed
    • Google Scholar
66. 1. Siebert S
    2. Machesky LM
    3. Insall RH

    (2015) Overflow in science and its implications for trust

    eLife 4:e10825.

    https://doi.org/10.7554/eLife.10825

    • Google Scholar
67. Preprint

    1. Smith LN

    (2018) A disciplined approach to neural network hyper-parameters: part 1–learning rate, batch size, momentum, and weight decay

    arXiv.

    https://arxiv.org/abs/1803.09820

    • Google Scholar
68. 1. Tayler KK
    2. Tanaka KZ
    3. Reijmers LG
    4. Wiltgen BJ

    (2013) Reactivation of neural ensembles during the retrieval of recent and remote memory

    Current Biology 23:99–106.

    https://doi.org/10.1016/j.cub.2012.11.019

    • PubMed
    • Google Scholar
69. Report

    1. Taylor BN
    2. Kuyatt CE

    (1994)

    Guidelines for Evaluating and Expressing the Uncertainty of Nist Measurement Results

    Technical report, US Department of Commerce, Technology Administration, National Institute of Standards and Technology.

    • Google Scholar
70. 1. Vallat R

    (2018) Pingouin: statistics in python

    Journal of Open Source Software 3:1026.

    https://doi.org/10.21105/joss.01026

    • Google Scholar
71. 1. Van De Werd HJ
    2. Rajkowska G
    3. Evers P
    4. Uylings HB

    (2010) Cytoarchitectonic and chemoarchitectonic characterization of the prefrontal cortical Areas in the mouse

    Brain Structure and Function 214:339–353.

    https://doi.org/10.1007/s00429-010-0247-z

    • PubMed
    • Google Scholar
72. 1. van der Walt S
    2. Schönberger JL
    3. Nunez-Iglesias J
    4. Boulogne F
    5. Warner JD
    6. Yager N
    7. Gouillart E
    8. Yu T
    9. scikit-image contributors

    (2014) scikit-image: image processing in Python

    PeerJ 2:e453.

    https://doi.org/10.7717/peerj.453

    • PubMed
    • Google Scholar
73. 1. von Chamier L
    2. Laine RF
    3. Henriques R

    (2019) Artificial intelligence for microscopy: what you should know

    Biochemical Society Transactions 47:1029–1040.

    https://doi.org/10.1042/BST20180391

    • PubMed
    • Google Scholar
74. 1. Warfield SK
    2. Zou KH
    3. Wells WM

    (2004) Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation

    IEEE Transactions on Medical Imaging 23:903–921.

    https://doi.org/10.1109/TMI.2004.828354

    • PubMed
    • Google Scholar
75. 1. Whittle N
    2. Hauschild M
    3. Lubec G
    4. Holmes A
    5. Singewald N

    (2010) Rescue of impaired fear extinction and normalization of cortico-amygdala circuit dysfunction in a genetic mouse model by dietary zinc restriction

    Journal of Neuroscience 30:13586–13596.

    https://doi.org/10.1523/JNEUROSCI.0849-10.2010

    • PubMed
    • Google Scholar
76. 1. Whittle N
    2. Maurer V
    3. Murphy C
    4. Rainer J
    5. Bindreither D
    6. Hauschild M
    7. Scharinger A
    8. Oberhauser M
    9. Keil T
    10. Brehm C
    11. Valovka T
    12. Striessnig J
    13. Singewald N

    (2016) Enhancing dopaminergic signaling and histone acetylation promotes long-term rescue of deficient fear extinction

    Translational Psychiatry 6:e974.

    https://doi.org/10.1038/tp.2016.231

    • PubMed
    • Google Scholar
77. Conference

    1. Yosinski J
    2. Clune J
    3. Bengio Y
    4. Lipson H

    (2014)

    How transferable are features in deep neural networks?

    Advances in Neural Information Processing Systems. pp. 3320–3328.

    • Google Scholar

Author details

1. Dennis Segebarth

   Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

   Contributed equally with

   Matthias Griebel

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-3806-9324

2. Matthias Griebel

   Department of Business and Economics, University of Würzburg, Würzburg, Germany

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

   Contributed equally with

   Dennis Segebarth

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-1959-0242

3. Nikolai Stein

   Department of Business and Economics, University of Würzburg, Würzburg, Germany

   Contribution

   Conceptualization, Validation, Visualization, Methodology, Writing - original draft, Project administration

   Competing interests

   No competing interests declared

4. Cora R von Collenberg

   Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany

   Contribution

   Data curation, Validation, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

5. Corinna Martin

   Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany

   Contribution

   Data curation, Validation, Writing - review and editing

   Competing interests

   No competing interests declared

6. Dominik Fiedler

   Institute of Physiology I, Westfälische Wilhlems-Universität, Münster, Germany

   Contribution

   Data curation, Validation, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

7. Lucas B Comeras

   Department of Pharmacology, Medical University of Innsbruck, Innsbruck, Austria

   Contribution

   Data curation, Validation, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2445-3605

8. Anupam Sah

   Department of Pharmacology and Toxicology, Institute of Pharmacy and Center for Molecular Biosciences Innsbruck, University of Innsbruck, Innsbruck, Austria

   Contribution

   Data curation, Validation, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8298-6501

9. Victoria Schoeffler

   Department of Child and Adolescent Psychiatry, Center of Mental Health, University Hospital Würzburg, Würzburg, Germany

   Contribution

   Data curation, Validation, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

10. Teresa Lüffe

    Department of Child and Adolescent Psychiatry, Center of Mental Health, University Hospital Würzburg, Würzburg, Germany

    Contribution

    Data curation, Validation, Investigation, Methodology, Writing - review and editing

    Competing interests

    No competing interests declared

11. Alexander Dürr

    Department of Business and Economics, University of Würzburg, Würzburg, Germany

    Contribution

    Software, Writing - review and editing

    Competing interests

    No competing interests declared

12. Rohini Gupta

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany

    Contribution

    Data curation, Funding acquisition, Writing - review and editing

    Competing interests

    No competing interests declared

13. Manju Sasi

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany

    Contribution

    Data curation, Funding acquisition, Writing - review and editing

    Competing interests

    No competing interests declared

14. Christina Lillesaar

    Department of Child and Adolescent Psychiatry, Center of Mental Health, University Hospital Würzburg, Würzburg, Germany

    Contribution

    Resources, Data curation, Funding acquisition, Validation, Methodology, Project administration, Writing - review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-5166-2851

15. Maren D Lange

    Institute of Physiology I, Westfälische Wilhlems-Universität, Münster, Germany

    Contribution

    Conceptualization, Resources, Data curation, Funding acquisition, Validation, Investigation, Methodology, Project administration, Writing - review and editing

    Competing interests

    No competing interests declared

16. Ramon O Tasan

    Department of Pharmacology, Medical University of Innsbruck, Innsbruck, Austria

    Contribution

    Resources, Funding acquisition, Validation, Methodology, Project administration, Writing - review and editing

    Competing interests

    No competing interests declared

17. Nicolas Singewald

    Department of Pharmacology and Toxicology, Institute of Pharmacy and Center for Molecular Biosciences Innsbruck, University of Innsbruck, Innsbruck, Austria

    Contribution

    Resources, Funding acquisition, Validation, Methodology, Project administration, Writing - review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-0166-3370

18. Hans-Christian Pape

    Institute of Physiology I, Westfälische Wilhlems-Universität, Münster, Germany

    Contribution

    Conceptualization, Resources, Supervision, Funding acquisition, Validation, Methodology, Writing - original draft, Project administration

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-6874-8224

19. Christoph M Flath

    Department of Business and Economics, University of Würzburg, Würzburg, Germany

    Contribution

    Conceptualization, Resources, Supervision, Funding acquisition, Validation, Methodology, Writing - original draft, Project administration, Writing - review and editing

    Contributed equally with

    Robert Blum

    For correspondence

    christoph.flath@uni-wuerzburg.de

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-1761-9833

20. Robert Blum

    1. Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany
    2. Comprehensive Anxiety Center, Würzburg, Germany

    Contribution

    Conceptualization, Resources, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

    Contributed equally with

    Christoph M Flath

    For correspondence

    Blum_R@ukw.de

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-5270-3854

• 4,405

  Page views

• 462

  Downloads

• 19

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.