Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019

  1. Kelly D Cobey  Is a corresponding author
  2. Christophe A Fehlmann
  3. Marina Christ Franco
  4. Ana Patricia Ayala
  5. Lindsey Sikora
  6. Danielle B Rice
  7. Chenchen Xu
  8. John PA Ioannidis
  9. Manoj M Lalu
  10. Alixe Ménard
  11. Andrew Neitzel
  12. Bea Nguyen
  13. Nino Tsertsvadze
  14. David Moher
  1. University of Ottawa, Canada
  2. Federal University of Pelotas, Brazil
  3. University of Toronto, Canada
  4. McGill University, Canada
  5. Stanford University, United States
  6. Ottawa Hospital Research Institute, Canada

Abstract

Introduction: Reproducibility is a central tenant of research. We aimed to synthesize the literature on reproducibility and describe its epidemiological characteristics, including how reproducibility is defined and assessed. We also aimed to determine and compare estimates for reproducibility across different fields.

Methods: We conducted a scoping review to identify English language replication studies published between 2018-2019 in economics, education, psychology, health sciences and biomedicine. We searched Medline, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature – CINAHL, Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences (IBSS), and EconLit. Documents retrieved were screened in duplicate against our inclusion criteria. We extracted year of publication, number of authors, country of affiliation of the corresponding author, and whether the study was funded. For the individual replication studies, we recorded whether a registered protocol for the replication study was used, whether there was contact between the reproducing team and the original authors, what study design was used, and what the primary outcome was. Finally, we recorded how reproducibilty was defined by the authors, and whether the assessed study(ies) successfully reproduced based on this definition. Extraction was done by a single reviewer and quality controlled by a second reviewer.

Results: Our search identified 11,224 unique documents, of which 47 were included in this review. Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single reproducibility study while the remaining 11 reported at least two reproducibility studies in the same paper. Less than the half of the studies referred to a registered protocol. There was variability in the definitions of reproduciblity success. In total, across the 47 documents 177 studies were reported. Based on the definition used by the author of each study, 95 of 177 (53.7%) studies reproduced.

Conclusion: This study gives an overview of research across five disciplines that explicitly set out to reproduce previous research. Such reproducibility studies are extremely scarce, the definition of a successfully reproduced study is ambiguous, and the reproducibility rate is overall modest.

Funding: No external funding was received for this work.

Data availability

Data and materials are available on the Open Science Framework (https://osf.io/wn7gm/).

The following data sets were generated

Article and author information

Author details

  1. Kelly D Cobey

    Heart Institute, University of Ottawa, Ottawa, Canada
    For correspondence
    kcobey@ottawaheart.ca
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2797-1686
  2. Christophe A Fehlmann

    School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  3. Marina Christ Franco

    School of Dentistry, Federal University of Pelotas, Pelotas, Brazil
    Competing interests
    The authors declare that no competing interests exist.
  4. Ana Patricia Ayala

    Gerstein Science Information Centre, University of Toronto, Toronto, Canada
    Competing interests
    The authors declare that no competing interests exist.
  5. Lindsey Sikora

    Health Sciences Library, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  6. Danielle B Rice

    Department of Psychology, McGill University, Montreal, Canada
    Competing interests
    The authors declare that no competing interests exist.
  7. Chenchen Xu

    Department of Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  8. John PA Ioannidis

    Meta-Research Innovation Center at Stanford, Stanford University, Stanford, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3118-6859
  9. Manoj M Lalu

    Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-0322-382X
  10. Alixe Ménard

    Centre for Journalology, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  11. Andrew Neitzel

    Department of Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  12. Bea Nguyen

    Department of Medicine, University of Ottawa, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  13. Nino Tsertsvadze

    Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
  14. David Moher

    Centre for Journalology, Ottawa Hospital Research Institute, Ottawa, Canada
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2434-4206

Funding

The authors declare that there was no funding for this work.

Copyright

© 2023, Cobey et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 1,066
    views
  • 99
    downloads
  • 5
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Kelly D Cobey
  2. Christophe A Fehlmann
  3. Marina Christ Franco
  4. Ana Patricia Ayala
  5. Lindsey Sikora
  6. Danielle B Rice
  7. Chenchen Xu
  8. John PA Ioannidis
  9. Manoj M Lalu
  10. Alixe Ménard
  11. Andrew Neitzel
  12. Bea Nguyen
  13. Nino Tsertsvadze
  14. David Moher
(2023)
Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019
eLife 12:e78518.
https://doi.org/10.7554/eLife.78518

Share this article

https://doi.org/10.7554/eLife.78518

Further reading

    1. Medicine
    Pinanong Na-Phatthalung, Shumin Sun ... Fudi Wang
    Research Article

    The zinc transporter Slc30a1 plays an essential role in maintaining cellular zinc homeostasis. Despite this, its functional role in macrophages remains largely unknown. Here, we examine the function of Slc30a1 in host defense using mice models infected with an attenuated stain of Salmonella enterica Typhimurium and primary macrophages infected with the attenuated Salmonella. Bulk transcriptome sequencing in primary macrophages identifies Slc30a1 as a candidate in response to Salmonella infection. Whole-mount immunofluorescence and confocal microscopy imaging of primary macrophage and spleen from Salmonella-infected Slc30a1flag-EGFP mice demonstrate Slc30a1 expression is increased in infected macrophages with localization at the plasma membrane and in the cytosol. Lyz2-Cre-driven Slc30a1 conditional knockout mice (Slc30a1fl/fl;Lyz2-Cre) exhibit increased susceptibility to Salmonella infection compared to control littermates. We demonstrate that Slc30a1-deficient macrophages are defective in intracellular killing, which correlated with reduced activation of nuclear factor kappa B and reduction in nitric oxide (NO) production. Notably, the model exhibits intracellular zinc accumulation, demonstrating that Slc30a1 is required for zinc export. We thus conclude that zinc export enables the efficient NO-mediated antibacterial activity of macrophages to control invading Salmonella.

    1. Chromosomes and Gene Expression
    2. Medicine
    Xianghong Xie, Mingyue Gao ... Xiaojun Liu
    Research Article

    LncRNAs are involved in modulating the individual risk and the severity of progression in metabolic dysfunction-associated fatty liver disease (MASLD), but their precise roles remain largely unknown. This study aimed to investigate the role of lncRNA Snhg3 in the development and progression of MASLD, along with the underlying mechanisms. The result showed that Snhg3 was significantly downregulated in the liver of high-fat diet-induced obesity (DIO) mice. Notably, palmitic acid promoted the expression of Snhg3 and overexpression of Snhg3 increased lipid accumulation in primary hepatocytes. Furthermore, hepatocyte-specific Snhg3 deficiency decreased body and liver weight, alleviated hepatic steatosis and promoted hepatic fatty acid metabolism in DIO mice, whereas overexpression induced the opposite effect. Mechanistically, Snhg3 promoted the expression, stability and nuclear localization of SND1 protein via interacting with SND1, thereby inducing K63-linked ubiquitination modification of SND1. Moreover, Snhg3 decreased the H3K27me3 level and induced SND1-mediated chromatin loose remodeling, thus reducing H3K27me3 enrichment at the Pparg promoter and enhancing PPARγ expression. The administration of PPARγ antagonist T0070907 improved Snhg3-aggravated hepatic steatosis. Our study revealed a new signaling pathway, Snhg3/SND1/H3K27me3/PPARγ, responsible for mice MASLD and indicates that lncRNA-mediated epigenetic modification has a crucial role in the pathology of MASLD.