1. Computational and Systems Biology
  2. Neuroscience
Download icon

On the objectivity, reliability, and validity of deep learning enabled bioimage analyses

  1. Dennis Segebarth
  2. Matthias Griebel
  3. Nikolai Stein
  4. Cora R von Collenberg
  5. Corinna Martin
  6. Dominik Fiedler
  7. Lucas B Comeras
  8. Anupam Sah
  9. Victoria Schoeffler
  10. Teresa Lüffe
  11. Alexander Dürr
  12. Rohini Gupta
  13. Manju Sasi
  14. Christina Lillesaar
  15. Maren D Lange
  16. Ramon O Tasan
  17. Nicolas Singewald
  18. Hans-Christian Pape
  19. Christoph M Flath  Is a corresponding author
  20. Robert Blum  Is a corresponding author
  1. University Hospital Würzburg, Germany
  2. University of Würzburg, Germany
  3. Westfälische Wilhlems-Universität, Germany
  4. University of Innsbruck, Austria
  5. University of Inssbruck, Austria
Research Article
  • Cited 0
  • Views 1,305
  • Annotations
Cite this article as: eLife 2020;9:e59780 doi: 10.7554/eLife.59780

Abstract

Bioimage analysis of fluorescent labels is widely used in the life sciences. Recent advances in deep learning (DL) allow automating time-consuming manual image analysis processes based on annotated training data. However, manual annotation of fluorescent features with a low signal-to-noise ratio is somewhat subjective. Training DL models on subjective annotations may be instable or yield biased models. In turn, these models may be unable to reliably detect biological effects. An analysis pipeline integrating data annotation, ground truth estimation, and model training can mitigate this risk. To evaluate this integrated process, we compared different DL-based analysis approaches. With data from two model organisms (mice, zebrafish) and five laboratories, we show that ground truth estimation from multiple human annotators helps to establish objectivity in fluorescent feature annotations. Furthermore, ensembles of multiple models trained on the estimated ground truth establish reliability and validity. Our research provides guidelines for reproducible DL-based bioimage analyses.

Article and author information

Author details

  1. Dennis Segebarth

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-3806-9324
  2. Matthias Griebel

    Department of Business and Economics, University of Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-1959-0242
  3. Nikolai Stein

    Department of Business and Economics, University of Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  4. Cora R von Collenberg

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  5. Corinna Martin

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  6. Dominik Fiedler

    Institute of Physiology, Westfälische Wilhlems-Universität, Münster, Germany
    Competing interests
    The authors declare that no competing interests exist.
  7. Lucas B Comeras

    Department of Pharmacology, University of Innsbruck, Innsbruck, Austria
    Competing interests
    The authors declare that no competing interests exist.
  8. Anupam Sah

    Department of Pharmacology and Toxicology, University of Innsbruck, Innsbruck, Austria
    Competing interests
    The authors declare that no competing interests exist.
  9. Victoria Schoeffler

    Department of Child and Adolescent Psychiatry, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  10. Teresa Lüffe

    Department of Child and Adolescent Psychiatry, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  11. Alexander Dürr

    Department of Business and Economics, University of Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  12. Rohini Gupta

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  13. Manju Sasi

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
  14. Christina Lillesaar

    Department of Child and Adolescent Psychiatry, University Hospital Würzburg, Würzburg, Germany
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-5166-2851
  15. Maren D Lange

    Institute of Physiology, Westfälische Wilhlems-Universität, Münster, Germany
    Competing interests
    The authors declare that no competing interests exist.
  16. Ramon O Tasan

    Department of Pharmacology, University of Inssbruck, Innsbruck, Austria
    Competing interests
    The authors declare that no competing interests exist.
  17. Nicolas Singewald

    Department of Pharmacology and Toxicology, University of Innsbruck, Innsbruck, Austria
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-0166-3370
  18. Hans-Christian Pape

    Institute of Physiology, Westfälische Wilhlems-Universität, Münster, Germany
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-6874-8224
  19. Christoph M Flath

    Department of Business and Economics, University of Würzburg, Würzburg, Germany
    For correspondence
    christoph.flath@uni-wuerzburg.de
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-1761-9833
  20. Robert Blum

    Institute of Clinical Neurobiology, University Hospital Würzburg, Würzburg, Germany
    For correspondence
    Blum_R@ukw.de
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-5270-3854

Funding

Deutsche Forschungsgemeinschaft (ID 44541416 - TRR58,A10)

  • Robert Blum

Deutsche Forschungsgemeinschaft (ID 44541416 - TRR58,A03)

  • Hans-Christian Pape

Deutsche Forschungsgemeinschaft (ID 44541416 - TRR58,B08)

  • Maren D Lange

Graduate School of Life Sciences Wuerzburg (fellowship)

  • Rohini Gupta
  • Manju Sasi

Austrian Science Fund (P29952 & P25851)

  • Ramon O Tasan

Austrian Science Fund (I2433-B26,DKW-1206,SFB F4410)

  • Nicolas Singewald

Interdisziplinaeres Zentrum fuer Klinische Zusammenarbeit Wuerzburg (N-320)

  • Christina Lillesaar

Deutsche Forschungsgemeinschaft (ID 424778381)

  • Robert Blum

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Animal experimentation: All effort was taken to minimize the number of animals used and their suffering.Lab-Wue1: All experiments with C57BL/6J wildtype mice were in accordance with the Guidelines set by the European Union and approved by our institutional Animal Care, the Utilization Committee and the Regierung von Unterfranken, Würzburg, Germany (License number: 55.2-2531.01-95/13). C57BL/6J wildtype mice were bred in the animal facility of the Institute of Clinical Neurobiology, University Hospital of Würzburg, Germany.Lab Mue: Male All animal experiments with male C57Bl/6J mice (Charles River, Sulzfeld, Germany) were carried out in accordance with European regulations on animal experimentation and protocols were approved by the local authorities (Landesamt für Natur, Umwelt und Verbraucherschutz Nordrhein-Westfalen).Lab-Inns1: Experiments were performed in adult, male C57Bl/6NCrl mice (Charles River, Sulzfeld, Germany). They were bred in the Department of Pharmacology at the Medical University Innsbruck, Austria. All procedures involving animals and animal care were conducted in accordance with international laws and policies (Directive 2010/63/EU of the European parliament and of the council of 22 September 2010 on the protection of animals used for scientific purposes; Guide for the Care and Use of Laboratory Animals, U.S. National Research Council, 2011) and were approved by the Austrian Ministry of Science.Lab-Inns2: Male 129S1/SvImJ (S1) mice (Charles River, Sulzfeld, Germany) were used for experimentation. The Austrian Animal Experimentation Ethics Board (Bundesministerium für Wissenschaft Forschung und Wirtschaft, Kommission für Tierversuchsangelegenheiten) approved all experimental procedures.

Reviewing Editor

  1. Scott E Fraser, University of Southern California, United States

Publication history

  1. Received: June 15, 2020
  2. Accepted: October 16, 2020
  3. Accepted Manuscript published: October 19, 2020 (version 1)

Copyright

© 2020, Segebarth et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 1,305
    Page views
  • 203
    Downloads
  • 0
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)

Further reading

    1. Computational and Systems Biology
    2. Immunology and Inflammation
    Antonio Cappuccio et al.
    Tools and Resources

    From cellular activation to drug combinations, immunological responses are shaped by the action of multiple stimuli. Synergistic and antagonistic interactions between stimuli play major roles in shaping immune processes. To understand combinatorial regulation, we present the immune Synergistic/Antagonistic Interaction Learner (iSAIL). iSAIL includes a machine learning classifier to map and interpret interactions, a curated compendium of immunological combination treatment datasets, and their global integration into a landscape of ~30,000 interactions. The landscape is mined to reveal combinatorial control of interleukins, checkpoints, and other immune modulators. The resource helps elucidate the modulation of a stimulus by interactions with other cofactors, showing that TNF has strikingly different effects depending on co-stimulators. We discover new functional synergies between TNF and IFNβ controlling dendritic cell-T cell crosstalk. Analysis of laboratory or public combination treatment studies with this user-friendly web-based resource will help resolve the complex role of interaction effects on immune processes.

    1. Computational and Systems Biology
    2. Neuroscience
    Richard Gao et al.
    Research Article

    Complex cognitive functions such as working memory and decision-making require information maintenance over seconds to years, from transient sensory stimuli to long-term contextual cues. While theoretical accounts predict the emergence of a corresponding hierarchy of neuronal timescales, direct electrophysiological evidence across the human cortex is lacking. Here, we infer neuronal timescales from invasive intracranial recordings. Timescales increase along the principal sensorimotor-to-association axis across the entire human cortex, and scale with single-unit timescales within macaques. Cortex-wide transcriptomic analysis shows direct alignment between timescales and expression of excitation- and inhibition-related genes, as well as genes specific to voltage-gated transmembrane ion transporters. Finally, neuronal timescales are functionally dynamic: prefrontal cortex timescales expand during working memory maintenance and predict individual performance, while cortex-wide timescales compress with aging. Thus, neuronal timescales follow cytoarchitectonic gradients across the human cortex, and are relevant for cognition in both short- and long-terms, bridging microcircuit physiology with macroscale dynamics and behavior.