Reproducibility is a central tenant of research. Reproducing previously published studies helps us to discern discoveries from false leads. The lexicon around reproducibility studies is diverse and poorly defined (Goodman et al., 2016). Here, we loosely use Nosek and Errington’s definition: ‘a study for which any outcome would be considered diagnostic evidence about a claim from prior research’ (Nosek and Errington, 2020a). Most scientific studies are never formally reproduced and some disciplines have lower rates of reproducibility attempts than others. For example, in education research, an analysis published in 2014 of all publications in the discipline’s top 100 journals found that only 0.13% (221 out of 164,589) of the published articles described an independent reproducibility study (Makel and Plucker, 2014). There is rising concern about the reproducibility of research and increasing interest in enhancing research transparency (e.g. Buck, 2015; Munafò et al., 2017; Collins and Tabak, 2014; Begley and Ioannidis, 2015).

Knowledge about rates of reproducibility is currently dominated by a handful of well-known projects examining the reproducibility of a group of studies within a field. For example, a project estimating the reproducibility of 100 psychology studies published in three leading journals found that just 36% had statistically significant results, compared to 97% of the original studies. Just 47% of the studies had effect sizes that were within the bounds of the 95% confidence interval of the original study (Open Science Collaboration, 2015). Estimates of reproducibility in economics are similarly modest or low: a large-scale study attempting to reproduce 67 papers was only able to reproduce 22 (33%) of these (Chang and Li, 2015). This same study showed that when teams attempting to reproduce research involved one of the original study authors as a co-author on the project, rates of reproducibility increased. This may suggest that detailed familiarity with the original study method increases the likelihood of reproducing the research findings. A cancer biology reproducibility project launched in 2013 to independently reproduce several high-profile papers has produced rather sobering results a decade later. Most of the selected studies could not even be attempted to be reproduced (e.g. it was not clear what had been done originally) and among those that an attempt to reproduce was made, most did not seem to produce consistent results (although the exact reproducibility rate depends on the definition of reproducibility) and the effect of the reproduced studies was only 15% of the original effect (Errington et al., 2021a; Kane and Kimmelman, 2021; Errington et al., 2021b).

In medicine, studies that do not reproduce in clinic may exaggerate patient benefits and harms (Le Noury et al., 2015) especially when clinical decisions are based on a single study. Despite this and other potential consequences we know very little about what predicts research reproducibility. No data exists which provides systematic estimates for reproducibility across multiple disciplines or addresses why disciplines might vary in their rates of reproducibility. Failure to empirically examine reproducibility is regrettable: without research we can’t identify actions to take that could drive improvements in research reproducibility. This contributes to research waste (Nasser et al., 2017; Ioannidis et al., 2014; Freedman et al., 2015).

We set out to broadly examine the reproducibility literature across five disciplines and report on characteristics including how reproducibility is defined, assessed, and document prevalence rates for reproducibility. We focused on studies that explicitly described themselves as reproducibility or replication studies addressing reproductions of previously published work.

Our study was conducted using the framework proposed by Arksey and O’Malley, 2005 and the related update by Levac et al., 2010, and follows a five stage process: (1) identifying the research question, (2) identifying relevant studies, (3) study selection, (4) charting the data, (5) collating, summarizing and reporting the results.

The primary objective of this study was to describe the characteristics of reproducibility studies, including how reproducibility is defined and assessed. We found 47 individual documents reporting reproducibility studies in 2018–2019 that met our inclusion criteria. Our included documents reported 177 individual reproducibility studies. Most reproducibility studies were in the disciplines of psychology and health science (>72%), with 86 and 42 studies, respectively. This may suggest unique cultures around reproducibility in distinct disciplines, future research is needed to determine if such differences truly exist given the limitations of our search and approach. Some disciplines have routinely embedded replication as part of the original discovery and validation efforts, for example replications are routinely done for genomics and other -omics findings as part of the original studies rather than as separate efforts. Consistent with previous research (Makel et al., 2012; Sukhtankar, 2017; Hubbard and Vetter, 1992), our findings suggest that overall only a very small fraction of research in any of these discipline published in a given year focuses on reproducing existing research published in previous papers. A recent evaluation of 349 randomly selected research articles from the biomedical literature published in 2015-2018 (Serghiou et al., 2021) found that 33 (10%) included a reproducibility component in their research (e.g. validating previously published experiments, running a similar clinical trial in a different population, etc.). However, the vast majority of these efforts would not qualify as separate, independent replication studies in our assessment.

Most of the documents included in our study had corresponding/lead authors who were from the United States (57.4%) and most papers reported receiving funding (68.1%); papers reporting multiple studies (N=9, 81.8%) were more likely to report funding than single replication studies (N=23, 63.9%). Together this may suggest that at least some funders recognize the value of reproducibility studies, and that USA based researchers have taken a leadership role in reproducibility research. We note that just 11 of the 47 papers reported to be reproducing more than one original study, suggesting most reproducibility studies reproduce a single previous study. We note that two of the ‘single studies’ included reported being part of a larger reproducibility project (e.g. a study part of the broader cancer reproducibility project). Four of ‘single studies’ reported more than 1 reproduction of the same original article in the document (e.g. different labs reproducing the same experiment). Future qualitative research could shed light on the motivations of researchers to conduct a single versus multiple reproducibility study. This will be important to understand what, if any, supports are needed to facilitate large-scale reproducibility studies.

When we examined the 177 individual studies reproduced in the 47 documents, we found only a minority of them referred to registered protocols. In psychology, rates were highest, with 45.8% of studies referring to a registered protocol. Registered protocols are a core open science practice, they can help to enhance transparency, mitigate publication bias and selective reporting biases (Nosek et al., 2018; Nosek et al., 2019). Importantly, they may specify what analyses were planned a priori and which were post hoc. Registration would seem especially relevant to reproducibility projects in order to pre-specify approaches to reduce perceptions of bias. We acknowledge, however, that mandates for registration are rare and exist only in particular disciplines and for specific study designs.

There was a wide range of study designs. For example, in psychology observational (e.g. cohort study) and experimental studies dominated, while data analysis studies (e.g. re-analysis) were most prominent in health sciences and economics. Reproducing an observational or experimental study may pose a greater resource challenge as compared to reproducing a data analysis, which may explain the higher rates of reproducibility success observed among data analysis studies. When no new data are generated, it may be difficult in the current research environment, which tends to favor novelty, to publish a re-analysis of existing data that shows the exact same result (Ebrahim et al., 2014; Naudet et al., 2018).

Across our five disciplines of interest the norm was that author teams did not overlap or contact the team of the original study they were attempting to reproduce. This finding may not be generalizable, because by definition we did not consider documents where the original study was reproduced within the same paper, a practice that is commonplace in many disciplines, for example genomics. Nosek and Errington describe confusion and disagreement that occurred during large scale reproducibility projects they were involved in which produced results that failed to replicate the original findings, calling for original researchers and those conducting reproducibility projects to “argue about what a replication means before you do it”. Nosek and Errington, 2020b Our finding that teams don’t overlap or communicate, suggests that this practice is not typically implemented, despite its potential value to improve reproducibility Chang and Li, 2015. Conversely, involvement of the original authors as authors in the reproducibility efforts may increase the impact of allegiance and confirmation biases. In the published experience from the Reproducibility Project: Cancer Biology, a large share of original authors did not respond to efforts to reach them to obtain information about their study (Errington et al., 2021a; Kane and Kimmelman, 2021; Errington et al., 2021b).

Of the 177 individual studies reproduced, based on the authors’ reported definition, 53.7% reproduced successfully. When examining definitions for reproducibility, we found that studies related to psychology and health sciences tended to use a comparison of effect sizes to define reproducibility success. The number of included studies across other disciplines was too low to yield meaningful comparison of differences in definitions across disciplines. Rates of reproducibility based on the authors definition were highest in health sciences (N=36/42, 85.7%; 24/33, 72.7%) including ‘other’ and lower in biomedicine (N=4/6, 66.7%), education (N=2/5, 40%), and psychology (N=25, 29.1%). Low rates in psychology were driven by a single document reporting 37 studies that failed to report outcomes. When this document was removed, rates improved to 51.0% in this discipline. When we applied p-values from the 116 studies where these were reported, 70.7% of studies had p-values less than the commonly used 0.05 threshold. There is an increasing literature of different definitions of reproducibility and ‘success’ rates will unavoidably depend on how replication is defined (Errington et al., 2021a; Errington et al., 2021b; Held et al., 2022; Pawel and Held, 2020).

To our knowledge, this is the first study to provide a broad comparison of the characteristics of explicit reproducibility studies across disciplines. This comparative approach may help to identify features to better support further reproducibility research projects. This study used a formal search strategy, including grey literature searching, to identify potential documents. It is possible that the terminology we used, which was broad to apply across disciplines, may not have captured all potential studies in this area. It is also possible that the databases used do not equally represent the distinct disciplines we investigated, meaning that the searches are not directly comparable cross-disciplinarily. We also were not able to locate the full-text of all included documents which may have impacted on the results. The impact of these missing texts may not have been equal across disciplines. For these reasons, generalizations about disciplines should not be made. A further limitation is that we only considered two years of research. This allowed for a contemporary view on characteristics of replication studies, but it prevented the ability to examine temporal changes. Indeed, several well-known large-scale replication studies would not have been captured in our search. Ultimately, the number of included documents in some disciplines was relatively modest, suggesting that inclusion of articles across a larger timeframe is needed to address the objective to compare more meaningfully across disciplines.

For feasibility we also only extracted information about the primary outcome listed for each paper, or if no primary outcome was specified, the first listed outcome. It is possible that rates of reproducibility differ across outcomes. Future research could consider all outcomes listed. While we conducted our screening and extraction using two reviewers, to foster quality control, the reporting of the studies captured was sometimes extremely poor. This impacted the extraction process as in some cases extraction was challenging and in others resulted in missing data. Further, our screeners and extractors were not naïve to the aims of the study, which may have created implicit bias. Future research could include training coders and extractors who are unaware of the project aims. Collectively these study design decisions and practical challenges present limitations on the overall generalizability of the findings beyond our dataset. Finally, we acknowledge that these explicit ‘reproducibility check’ documents that we targeted, are only one part of the much larger scientific literature where some reproducibility features may be embedded. Random samples of biomedical papers with empirical data published in 2015–2018 have shown that reproducibility concepts are not uncommon (Wallach et al., 2018). In the psychological sciences, similarly 5% of a random sample of 188 papers with empirical data published in 2014–2017 were replications (Hardwicke et al., 2020). Conversely, in the social sciences, among a random sample of 156 papers published in 2014–2017, only 2 were replication studies (1%) (Hardwicke et al., 2020). Moreover, as mentioned above, some fields explicitly require replications to be included as part of the original publication, and the large and blossoming literature of tens of thousands of meta-analyses (Ioannidis, 2016) suggests that for many topics there are multiple studies that address similar enough questions so that meta-analysts would combine them. Eventually, the relative advantages and disadvantages of different replication practices (e.g. reproducibility embedded in the original publication versus done explicitly in a subsequent stage versus done as part of a wider agenda that mixes replication and novel efforts) needs further empirical study in diverse scientific fields.

Our finding that, only about half of the reproducibility studies reproduced across five fields of interest is concerning, though consistent with other studies. These estimates may not necessarily represent appropriately the reproducibility rates of entire fields since the choice of what specific studies to try to replicate may include selection factors that introduce strong bias towards higher or lower replication rates. Moreover, while estimates of reproducibility vary across fields in our modest sample, so too do norms in definitions used to define reproducibility. Choice of these definitions (especially when these definitions are not clear, pre-specified and valid) mayaffect the interpretation of these results to fit various narratives of replication success or failure. This suggests the need for discipline and interdisciplinary specific exchange on how to best approach reproducibility studies. Discussion on definitions for reproducibility, but also about methodological best practices when conducting a reproducibility study (e.g. using registered reports) will help to foster integrity and quality. To ensure reliability, multiple and diverse reproducibility studies with converging evidence are needed. At present, and as illustrated by out sampling, explicit reproducibility studies done as targeted reproducibility checks are rare. To enhance research reliability, reproducibility studies need to be encouraged, incentivized, and supported.

Data and materials are available on the Open Science Framework (https://osf.io/wn7gm/).

1. 1. Cobey KD

   (2022) Open Science Framework

   Scoping review data on reproducibility of research in a 2018-2019 multi-discipline sample.

   https://doi.org/10.17605/OSF.IO/WN7GM

1. 1. Arksey H
   2. O’Malley L

   (2005) Scoping studies: towards a methodological framework

   International Journal of Social Research Methodology 8:19–32.

   https://doi.org/10.1080/1364557032000119616

   • Google Scholar
2. 1. Begley CG
   2. Ioannidis JPA

   (2015) Reproducibility in science: improving the standard for basic and Preclinical research

   Circulation Research 116:116–126.

   https://doi.org/10.1161/CIRCRESAHA.114.303819

   • PubMed
   • Google Scholar
3. 1. Buck S

   (2015) Solving reproducibility

   Science 348:1403.

   https://doi.org/10.1126/science.aac8041

   • PubMed
   • Google Scholar
4. Book

   1. Chang AC
   2. Li P

   (2015) Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say”Usually Not”

   Washington: Board of Governors of the Federal Reserve System.

   https://doi.org/10.17016/FEDS.2015.083

   • Google Scholar
5. 1. Collins FS
   2. Tabak LA

   (2014) NIH plans to enhance reproducibility

   Nature 505:612–613.

   https://doi.org/10.1038/505612a

   • Google Scholar
6. Website

   1. Distiller SR

   (2023) DistillerSR: Literature Review Software Smarter Reviews: Trusted Evidence

   Accessed June 29, 2023.

   https://www.evidencepartners.com/products/distillersr-systematic-review-software/
7. 1. Ebrahim S
   2. Sohani ZN
   3. Montoya L
   4. Agarwal A
   5. Thorlund K
   6. Mills EJ
   7. Ioannidis JPA

   (2014) Reanalyses of randomized clinical trial data

   JAMA 312:1024–1032.

   https://doi.org/10.1001/jama.2014.9646

   • PubMed
   • Google Scholar
8. 1. Errington TM
   2. Denis A
   3. Perfito N
   4. Iorns E
   5. Nosek BA

   (2021a) Challenges for assessing Replicability in Preclinical cancer biology

   eLife 10:e67995.

   https://doi.org/10.7554/eLife.67995

   • PubMed
   • Google Scholar
9. 1. Errington TM
   2. Mathur M
   3. Soderberg CK
   4. Denis A
   5. Perfito N
   6. Iorns E
   7. Nosek BA

   (2021b) Investigating the Replicability of Preclinical cancer biology

   eLife 10:e71601.

   https://doi.org/10.7554/eLife.71601

   • PubMed
   • Google Scholar
10. 1. Freedman LP
    2. Cockburn IM
    3. Simcoe TS

    (2015) The Economics of reproducibility in Preclinical research

    PLOS Biology 13:e1002165.

    https://doi.org/10.1371/journal.pbio.1002165

    • PubMed
    • Google Scholar
11. 1. Goodman SN
    2. Fanelli D
    3. Ioannidis JPA

    (2016) What does research reproducibility mean?

    Science Translational Medicine 8:341ps12.

    https://doi.org/10.1126/scitranslmed.aaf5027

    • PubMed
    • Google Scholar
12. 1. Hardwicke TE
    2. Wallach JD
    3. Kidwell MC
    4. Bendixen T
    5. Crüwell S
    6. Ioannidis JPA

    (2020) An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017)

    Royal Society Open Science 7:190806.

    https://doi.org/10.1098/rsos.190806

    • PubMed
    • Google Scholar
13. 1. Held L
    2. Matthews R
    3. Ott M
    4. Pawel S

    (2022) Reverse-Bayes methods for evidence assessment and research synthesis

    Research Synthesis Methods 13:295–314.

    https://doi.org/10.1002/jrsm.1538

    • PubMed
    • Google Scholar
14. 1. Hubbard R
    2. Vetter DE

    (1992) The publication incidence of Replications and critical commentary in economics

    The American Economist 36:29–34.

    https://doi.org/10.1177/056943459203600106

    • Google Scholar
15. Website

    1. IBM Support

    (2023) IBM SPSS Statistics 26

    Accessed June 29, 2023.

    https://www.ibm.com/support/pages/downloading-ibm-spss-statistics-26
16. 1. Ioannidis JPA
    2. Greenland S
    3. Hlatky MA
    4. Khoury MJ
    5. Macleod MR
    6. Moher D
    7. Schulz KF
    8. Tibshirani R

    (2014) Increasing value and reducing waste in research design, conduct, and analysis

    The Lancet 383:166–175.

    https://doi.org/10.1016/S0140-6736(13)62227-8

    • Google Scholar
17. 1. Ioannidis JPA

    (2016) The mass production of redundant, misleading, and Conflicted systematic reviews and meta-analyses

    The Milbank Quarterly 94:485–514.

    https://doi.org/10.1111/1468-0009.12210

    • PubMed
    • Google Scholar
18. 1. Kane PB
    2. Kimmelman J

    (2021) Is Preclinical research in cancer biology reproducible enough

    eLife 10:e67527.

    https://doi.org/10.7554/eLife.67527

    • PubMed
    • Google Scholar
19. 1. Le Noury J
    2. Nardo JM
    3. Healy D
    4. Jureidini J
    5. Raven M
    6. Tufanaru C
    7. Abi-Jaoude E

    (2015) Restoring study 329: Eficacy and harms of paroxetine and Imipramine in treatment of major depression in adolescence

    BMJ 351:h4320.

    https://doi.org/10.1136/bmj.h4320

    • Google Scholar
20. 1. Levac D
    2. Colquhoun H
    3. O’Brien KK

    (2010) Scoping studies: advancing the methodology

    Implementation Science 5:69.

    https://doi.org/10.1186/1748-5908-5-69

    • PubMed
    • Google Scholar
21. 1. Makel MC
    2. Plucker JA
    3. Hegarty B

    (2012) Replications in psychology research: how often do they really occur

    Perspectives on Psychological Science 7:537–542.

    https://doi.org/10.1177/1745691612460688

    • PubMed
    • Google Scholar
22. 1. Makel MC
    2. Plucker JA

    (2014) Facts are more important than novelty: replication in the education sciences

    Educational Researcher 43:304–316.

    https://doi.org/10.3102/0013189X14545513

    • Google Scholar
23. 1. McGowan J
    2. Sampson M
    3. Salzwedel DM
    4. Cogo E
    5. Foerster V
    6. Lefebvre C

    (2016) Guideline statement PRESS peer review of electronic search strategies: 2015 guideline statement

    Journal of Clinical Epidemiology 75:40–46.

    https://doi.org/10.1016/j.jclinepi.2016.01.021

    • PubMed
    • Google Scholar
24. 1. Munafò MR
    2. Nosek BA
    3. Bishop DVM
    4. Button KS
    5. Chambers CD
    6. du Sert NP
    7. Simonsohn U
    8. Wagenmakers EJ
    9. Ware JJ
    10. Ioannidis JPA

    (2017) A manifesto for reproducible science

    Nature Human Behaviour 1:0021.

    https://doi.org/10.1038/s41562-016-0021

    • PubMed
    • Google Scholar
25. 1. Nasser M
    2. Clarke M
    3. Chalmers I
    4. Brurberg KG
    5. Nykvist H
    6. Lund H
    7. Glasziou P

    (2017) What are Funders doing to minimise waste in research? lancet

    Lancet 389:1006–1007.

    https://doi.org/10.1016/S0140-6736(17)30657-8

    • PubMed
    • Google Scholar
26. 1. Naudet F
    2. Sakarovitch C
    3. Janiaud P
    4. Cristea I
    5. Fanelli D
    6. Moher D
    7. Ioannidis JPA

    (2018) Data sharing and Reanalysis of randomized controlled trials in leading BIOMEDICAL journals with a full data sharing policy: survey of studies published in the BMJ and PLOS Medicine

    BMJ 360:k400.

    https://doi.org/10.1136/bmj.k400

    • PubMed
    • Google Scholar
27. 1. Nosek BA
    2. Ebersole CR
    3. DeHaven AC
    4. Mellor DT

    (2018) The Preregistration revolution

    PNAS 115:2600–2606.

    https://doi.org/10.1073/pnas.1708274114

    • PubMed
    • Google Scholar
28. 1. Nosek BA
    2. Beck ED
    3. Campbell L
    4. Flake JK
    5. Hardwicke TE
    6. Mellor DT
    7. van ’t Veer AE
    8. Vazire S

    (2019) Preregistration is hard, and worthwhile

    Trends in Cognitive Sciences 23:815–818.

    https://doi.org/10.1016/j.tics.2019.07.009

    • PubMed
    • Google Scholar
29. 1. Nosek BA
    2. Errington TM

    (2020a) What is replication

    PLOS Biology 18:e3000691.

    https://doi.org/10.1371/journal.pbio.3000691

    • PubMed
    • Google Scholar
30. 1. Nosek BA
    2. Errington TM

    (2020b) The best time to argue about what a replication means? before you do it

    Nature 583:518–520.

    https://doi.org/10.1038/d41586-020-02142-6

    • PubMed
    • Google Scholar
31. 1. Open Science Collaboration

    (2015) PSYCHOLOGY. Estimating the reproducibility of psychological science

    Science 349:aac4716.

    https://doi.org/10.1126/science.aac4716

    • PubMed
    • Google Scholar
32. 1. Pawel S
    2. Held L

    (2020) Probabilistic forecasting of replication studies

    PLOS ONE 15:e0231416.

    https://doi.org/10.1371/journal.pone.0231416

    • PubMed
    • Google Scholar
33. 1. Rethlefsen ML
    2. Kirtley S
    3. Waffenschmidt S
    4. Ayala AP
    5. Moher D
    6. Page MJ
    7. Koffel JB
    8. PRISMA-S Group

    (2021) PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews

    Systematic Reviews 10:39.

    https://doi.org/10.1186/s13643-020-01542-z

    • PubMed
    • Google Scholar
34. 1. Serghiou S
    2. Contopoulos-Ioannidis DG
    3. Boyack KW
    4. Riedel N
    5. Wallach JD
    6. Ioannidis JPA

    (2021) Assessment of transparency indicators across the BIOMEDICAL literature: how open is open

    PLOS Biology 19:e3001107.

    https://doi.org/10.1371/journal.pbio.3001107

    • PubMed
    • Google Scholar
35. Book

    1. Sukhtankar S

    (2017)

    Online Appendix for Replications in Development Economics

    American Economic Association.

    • Google Scholar
36. 1. Tricco AC
    2. Lillie E
    3. Zarin W
    4. O’Brien KK
    5. Colquhoun H
    6. Levac D
    7. Moher D
    8. Peters MDJ
    9. Horsley T
    10. Weeks L
    11. Hempel S
    12. Akl EA
    13. Chang C
    14. McGowan J
    15. Stewart L
    16. Hartling L
    17. Aldcroft A
    18. Wilson MG
    19. Garritty C
    20. Lewin S
    21. Godfrey CM
    22. Macdonald MT
    23. Langlois EV
    24. Soares-Weiser K
    25. Moriarty J
    26. Clifford T
    27. Tunçalp Ö
    28. Straus SE

    (2018) PRISMA extension for Scoping reviews (PRISMA-SCR): checklist and explanation

    Annals of Internal Medicine 169:467–473.

    https://doi.org/10.7326/M18-0850

    • PubMed
    • Google Scholar
37. 1. Wallach JD
    2. Boyack KW
    3. Ioannidis JPA

    (2018) Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017

    PLOS Biology 16:e2006930.

    https://doi.org/10.1371/journal.pbio.2006930

    • PubMed
    • Google Scholar

Decision letter after peer review:

[Editors’ note: the authors submitted for reconsideration following the decision after peer review. What follows is the decision letter after the first round of review.]

Thank you for submitting your article "Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Mone Zaidi as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Colby J Vorland (Reviewer #1); Arthur Lupia (Reviewer #2); Jon Agley (Reviewer #3).

I believe that the reviewers' comments are clear and should be helpful to you.

Reviewer #1 (Recommendations for the authors):

Cobey et al. conducted an exhaustive survey of a number of five disciplines to catalogue and characterize replications. The authors employed best practices in performing their review and reporting, and the results provide an important snapshot of replications. Kudos to the authors for their transparent practices – preregistering their protocol, and specifying protocol amendments.

A limitation of this work is that it only surveys 2018-2019, and many replication projects were not published then, which limits comparisons across disciplines. This is understandable given the herculean screening task for just these two years, and the authors do emphasize this limitation in their discussion. Including the years 2018-2019 in the title seems appropriate to make this clear in searches.

The manuscript is generally well written, but throughout the text the authors use the terms 'reproducibility' and 'replicability' interchangeably (sometimes within the same sentence), which is confusing. Even more so is that the authors attempt to distinguish the definitions of each in the introduction:

Line 102: [Defining 'reproducibility'] "Here, we loosely use Nosek and Errington's definition: "a study for which any outcome would be considered diagnostic evidence about a claim from prior research"".

– However, this quote refers to the term 'replication'. Here is the full quote: "Replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research." An additional quote from that paper defines reproducibility and replication like so: "Credibility of scientific claims is established with evidence for their replicability using new data [1]. This is distinct from retesting a claim using the same analyses and same data (usually referred to as reproducibility or computational reproducibility) …" These definitions are in alignment with the 2019 NASEM report (https://nap.nationalacademies.org/catalog/25303/reproducibility-and-replicability-in-science). As far as I can understand, the authors studied *both* concepts in this paper; most published reports were replications, but it is noted on line 330 that "The remainder of studies captured were data studies e.g., re-analysis using previous data (35, 19.8%)…". If it is the case and that this means that these data studies re-analyzed the *same* dataset as the original work, the manuscript should be rewritten to distinguish replication from reproducibility throughout. The authors may wish to note the fact that different disciplines may use these terms in different ways (e.g., expand the discussion of reference #1), and that reproducibility is sometimes used to refer to transparency, and in other contexts replicability.

In the introduction, the authors note the percentages of studies that replicate for various replication projects. Yet, that is taking each project's definition of what a successful replication is. As the authors point out on line 131, some may define in different ways and thus simplified conclusions are difficult. This point could be moved earlier to emphasize that this is the case for all of these projects.

The data statement states: "Data and materials are available on the Open Science Framework (https://osf.io/wn7gm/)." I see the protocol there, but no extracted data; will these be added?

Could some of the results in tables 1-3 also be presented graphically?

Reviewer #2 (Recommendations for the authors):

This article offers an important discussion, and datapoint, relevant to helping scientific researchers more closely reconcile what they observe with what they claim. There is tremendous potential public and scientific value in the endeavor. The value arises from the fact that many academic reward systems are tilted towards mass production of statistically significant claims and away from truer representations of what these claims mean to readers or end users. While there is greater recognition of these problems, and many attempts to improve practices, this article raises tough questions about how well actual practice aligns with publicly stated RR goals. The authors seek a constructive way forward. They offer an empirical analysis that documents and compares practices across several disciplines. Tables 2 and 3 are particularly instructive and a model for future work of this kind.

My biggest concern with the paper is that there is a gap between the article's findings and several generalizations that are made. I find the method and empirical results interesting, but have limited confidence in how much these claims generalize – even to representative samples of articles in the fields on which the paper focuses. Edits that more closely align the text with the observations will help readers more accurately interpret the authors' findings.

For example, the authors list a series of discipline-specific article databases that they use to identify articles for comparison. If a goal is to make discipline-wide generalizations from these set of articles, it would help readers to know more about the comparability of the databases. For example, are the databases equally representative of the fields that they are characterized as covering (i.e., do some of the databases favor subfields within a discipline or lack full coverage in other ways)? As the article is written, the reader is asked to take this premise on faith. The truth-value of this premise is not apparent to me. If there is no empirical or structural basis for assuming direct comparability, the fact should be noted, cross-disciplinary comparisons or conclusions should reference the caveat, and generalizations beyond this caveat should not be made.

Another aspect of the analysis that may impede generalization is the decision to list the first reported RR outcome in studies that do not list a primary outcome. Given known incentives that lead to publication biases, and against null results, isn't it likely that first reported outcomes are more likely to be non-replications? If this is a possibility, could they compare RR success rates from first reported outcomes to last-reported outcomes? If there is no difference, the fact can be noted and a particular type of generalization becomes possible. If there is a difference, then the caveat should be added in subsequent claims.

On a related note, the authors conclude that RR frequencies and success rates are modest. But what is the relevant base rate? This standard may be easier to define for RR outcomes. On the optimal number of RR attempts, I think that there is less of a consensus. To state the question in a leading manner, if one replication/reproduction is better than none, why aren't two better than one? This question, and others like it, imply that observed rates of RR attempts that are less than 100% may not be suboptimal in a broader sense.

On a different note, I am concerned about how the coding was done. Specifically, it appears that co-authors served as "screeners" of how to categorize certain articles and article attributes. In other words, they made decision that influenced the data that they were analyzing. The risk is that people who know the hypotheses and desired outcomes implicitly bring that knowledge to coding decisions. An established best practice in fields where this type of coding is done is to first train independent coders who are unaware of the researchers; hypotheses and then conduct rigorous inter-coder reliability assessments to document the extent to which the combination of coders and categorical framework produce coding outcomes that parallel the underlying conceptual framework. Such practices increase the likelihood that data generating processes are independent of hypothesis evaluation processes.

For these reasons, I am very interested in the questions that the authors ask, and what they find, but I am not convinced that a number of the empirical claims pertaining to comparisons and magnitudes will generalize even to larger populations of articles in the stated disciplines.

I would like to see the Discussion section rewritten to focus on what the findings, and methodological challenges, mean for future work. In the current version, there are a number of speculative claims and generalizations that do not follow from the empirical work in this paper. These facts and some variation in the focus of the text in that section make the discussion longer than it needs to be and may have the effect of diminishing the excellent work that was done in the earlier pages.

Reviewer #3 (Recommendations for the authors):

The authors provide a fairly clear and accurate summary of the current questions around reproducibility in the scientific literature. I was impressed that they thoughtfully included a description of the areas where their utilized methodology was different than what they had proposed in their protocol and explained the reasons for those differences. That kind of documentation is valuable and fosters transparency in the research process.

The results of their analysis are provided in both narrative format and in tables and appendices. The authors effectively characterize the different, and sometimes difficult, decisions made in parsing the results of their search. While much attention has been paid to replication and reproducibility in recent years, the nature of the results reflects the reality that the replication studies themselves may have reporting issues, and it may not always be possible to ascertain how to link the results of such studies with the original work. The study results meaningfully add to the current, small body of literature examining the meta-scientific issues of reproducibility and replication of scientific findings.

The paper is cautious in its approach to interpretation, appropriately using language such as "this may suggest that…" to better ensure that readers do not draw inappropriately firm conclusions. This is a descriptive paper, and so readers, like the authors, would best be served by limiting strong inferences based on the findings. At the same time, the authors helpfully suggest numerous meaningful pathways forward to advance reproducibility research in multiple scientific domains. Such studies would in turn facilitate more powerful inferences.

In preparing this review, the substantive majority of recommendations that I had for the authors related to points of clarity conveyed in private comments rather than overt weaknesses or concerns.

LL50-52: The meaning of this sentence is not entirely clear (though the paragraph can still be understood when skipping the sentence). Please consider rewriting to capture your point more closely.

LL106-108: Since that analysis was published in 2014, and given the recent (albeit anecdotal) increased emphasis on reproducibility, it may be worth noting the date of the study in the text itself (or the termination date of the search used for the 0.13% statistic).

L126: In this context, "some effect in inbreeding" is unclear.

LL135-136: There may be value in disentangling the link between reproducibility and harms in the Introduction more explicitly. In the cited study (Le Noury et al), medications were utilized in a trial where reanalysis indicated that they were unlikely to produce a clinical benefit and each resulted in plausibly increased risk of harm. But it was not the failure to replicate, per se, that led to possible harms. Rather, the concern was clinical decision making predicated on the results of the original study, the results of which "stood" pending reanalysis. This may seem like a pedantic point but I think it speaks to the role not only of the original study itself, but also the degree to which decisions were made based on the findings of a single study.

L240: Can you provide a short clarification of what you mean by calibration test – such as inserting a parenthetical example "…performed a calibration test (e.g., did X) prior to…"

Table 1: The box indicating a cross section of number of authors and all studies appears to have inaccurate/incomplete information.

Table 1: The way the data are structured is a little unclear. For example, for "All Studies" the "discipline" row uses a denominator of total replication studies, whereas "year of publication" in the same column uses a denominator of the number of papers. While I see why this was done, I think additional clarity or uniformity in how the data are presented would be helpful.

LL300-301: Could you briefly indicate whether you think that excluding documents unavailable in full via your library may have affected the results (e.g., was this 11 quantitative replications, as assessed by abstracts? or was it a mixed bag?).

L384: Does the count for psychology include the three dozen or so replication studies that were unusable for this paper? If so, do the authors think that their inclusion here serves to accurately represent the replication efforts in psychology or may artificially inflate the frequency? I don't have the perspective to know which is the case, but I think the question is worth considering.

LL397-398: Would you be willing to share any information (if it exists) about whether independent and separate replications differ from those embedded within experimental protocols in meaningful ways (not just in terms of the obvious process differences)?

L417: Are you referring to registered protocols for the replication studies? Or registered protocols for the original study that can be used in completing the replication?

LL430-432: It might be helpful for readers to be briefly informed about what "the current environment" means in practice, as there is a lot of possible variability. For example, I might assume that there is an overemphasis on novelty at the desk editing stage, but it's not clear whether the authors were even thinking of that in particular.

LL434-448: There is a lot of interesting discussion in this paragraph that gets somewhat "jumbled together" in this paragraph. First, I'm not sure that it makes sense to characterize the finding as biased in L436. Rather, you found that for replication studies that were not part of the same paper, lack of overlap was common, and you do not make any assertions about studies where that was not the case (but presume the overlap might be higher – and in fact would by definition be nearly 100% since authors do not change partway through papers). Second, the larger question about what constitutes a replication is related to, but seemingly separate from, the initial discussion. Even if the authors agree on what a replication would look like for a specific study, it may not be coherent in the context of the broader field's common understanding of reproducibility.

[Editors’ note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review of articles published in 2018-2019" for further consideration by eLife. Your revised article has been evaluated by Mone Zaidi (Senior Editor) and a Reviewing Editor.

The manuscript has been improved but there are some modest remaining issues that need to be addressed, as outlined below:

Reviewer #3 (Recommendations for the authors):

I would like to thank the authors for their response to my original comments. In most cases, I found that the authors' revisions were responsive to my concerns. In the instances where I did not find that to be entirely the case, I have noted it below using the new line numbers.

L126: Thank you for revising the line about inbreeding. However, I think that the replacement phrase is still a little unclear. The sentence is structured as an 'either/or' and the first part suggests familiarity with the study method may increase likelihood of reproducing the same findings. So I am not sure how "researcher familiarity" functions as the alternative. The sense I get is that the authors may be hedging around suggesting that in some cases, the author's approach to research (perhaps unusually rigorous, or unusually sloppy) may carry over between studies and explain some of the variability in findings. However, that is an assumption. In any case, the authors might benefit from being very direct about their theory here.

L110: The phrase "…and some disciplines do worse in this regard" might be interpreted to have two meanings. I interpret it to mean that some disciplines have fewer studies where reproductions have been attempted. However, "do worse" might also be taken to mean that some disciplines' studies are less likely to successfully have their results reproduced, which is a different concept entirely.

LL404-405: I appreciate this revision made in response to reviewer #1. I am unsure about whether an assertion of "unique cultures" is significantly different than the original assertion of which cultures have it as a normative value. I think the line may be fine to retain, but should be constrained with a caveat e.g., "This may suggest unique cultures around reproducibility across different disciplines, but further study is needed to determine whether this is truly the case, and our study should not be taken as proof of differences between disciplines."

[Editors’ note: The authors appealed the original decision. What follows is the authors’ response to the first round of review.]

Reviewer #1 (Recommendations for the authors):

Cobey et al. conducted an exhaustive survey of a number of five disciplines to catalogue and characterize replications. The authors employed best practices in performing their review and reporting, and the results provide an important snapshot of replications. Kudos to the authors for their transparent practices – preregistering their protocol, and specifying protocol amendments.

A major limitation of this work is that it only surveys 2018-2019, and many replication projects were not published then, which limits comparisons across disciplines. This is understandable given the herculean screening task for just these two years, and the authors do emphasize this limitation in their discussion. Including the years 2018-2019 in the title seems appropriate to make this clear in searches.

We have updated the title to incorporate this feedback. It now reads: Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review of articles published in 2018-2019.

The manuscript is generally well written, but throughout the text the authors use the terms 'reproducibility' and 'replicability' interchangeably (sometimes within the same sentence), which is confusing. Even more so is that the authors attempt to distinguish the definitions of each in the introduction:

Line 102: [Defining 'reproducibility'] "Here, we loosely use Nosek and Errington's definition: "a study for which any outcome would be considered diagnostic evidence about a claim from prior research"".

– However, this quote refers to the term 'replication'. Here is the full quote: "Replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research." An additional quote from that paper defines reproducibility and replication like so: "Credibility of scientific claims is established with evidence for their replicability using new data [1]. This is distinct from retesting a claim using the same analyses and same data (usually referred to as reproducibility or computational reproducibility) …" These definitions are in alignment with the 2019 NASEM report (https://nap.nationalacademies.org/catalog/25303/reproducibility-and-replicability-in-science). As far as I can understand, the authors studied *both* concepts in this paper; most published reports were replications, but it is noted on line 330 that "The remainder of studies captured were data studies e.g., re-analysis using previous data (35, 19.8%)…". If it is the case and that this means that these data studies re-analyzed the *same* dataset as the original work, the manuscript should be rewritten to distinguish replication from reproducibility throughout. The authors may wish to note the fact that different disciplines may use these terms in different ways (e.g., expand the discussion of reference #1), and that reproducibility is sometimes used to refer to transparency, and in other contexts replicability.

Thanks for this thoughtful reflection. We agree the terminology could be clarified further. We have adjusted this section, it now reads:

“Reproducibility is a central tenant of research. Reproducing previously published studies to determine if results are consistent helps us to discern discoveries from false leads. The lexicon around the terms reproducibility and replication is diverse and poorly defined and may differ between disciplines ¹. Here, we loosely use Nosek and Errington’s definition for a successful replication, namely: “a study for which any outcome would be considered diagnostic evidence about a claim from prior research”².”

In the introduction, the authors note the percentages of studies that replicate for various replication projects. Yet, that is taking each project's definition of what a successful replication is. As the authors point out on line 131, some may define in different ways and thus simplified conclusions are difficult. This point could be moved earlier to emphasize that this is the case for all of these projects.

We have taken on this suggestion by adding the line “Comparison of rates of replication prove challenging because results will depend on the definition of replication success.” further ahead in the paper, around line 118.

The data statement states: "Data and materials are available on the Open Science Framework (https://osf.io/wn7gm/)." I see the protocol there, but no extracted data; will these be added?

Could some of the results in tables 1-3 also be presented graphically?

All study data and materials have been made available at this link now.

Reviewer #2 (Recommendations for the authors):

This article offers an important discussion, and datapoint, relevant to helping scientific researchers more closely reconcile what they observe with what they claim. There is tremendous potential public and scientific value in the endeavor. The value arises from the fact that many academic reward systems are tilted towards mass production of statistically significant claims and away from truer representations of what these claims mean to readers or end users. While there is greater recognition of these problems, and many attempts to improve practices, this article raises tough questions about how well actual practice aligns with publicly stated RR goals. The authors seek a constructive way forward. They offer an empirical analysis that documents and compares practices across several disciplines. Tables 2 and 3 are particularly instructive and a model for future work of this kind.

My biggest concern with the paper is that there is a gap between the article's findings and several generalizations that are made. I find the method and empirical results interesting, but have limited confidence in how much these claims generalize – even to representative samples of articles in the fields on which the paper focuses. Edits that more closely align the text with the observations will help readers more accurately interpret the authors' findings.

For example, the authors list a series of discipline-specific article databases that they use to identify articles for comparison. If a goal is to make discipline-wide generalizations from these set of articles, it would help readers to know more about the comparability of the databases. For example, are the databases equally representative of the fields that they are characterized as covering (i.e., do some of the databases favor subfields within a discipline or lack full coverage in other ways)? As the article is written, the reader is asked to take this premise on faith. The truth-value of this premise is not apparent to me. If there is no empirical or structural basis for assuming direct comparability, the fact should be noted, cross-disciplinary comparisons or conclusions should reference the caveat, and generalizations beyond this caveat should not be made.

Thanks for this comment. We have modified the paper to address this in the discussion. We have added: “It is also possible that the databases used do not equally represent the distinct disciplines we investigated, meaning that the searches are not directly comparable cross-disciplinarily”.

Another aspect of the analysis that may impede generalization is the decision to list the first reported RR outcome in studies that do not list a primary outcome. Given known incentives that lead to publication biases, and against null results, isn't it likely that first reported outcomes are more likely to be non-replications? If this is a possibility, could they compare RR success rates from first reported outcomes to last-reported outcomes? If there is no difference, the fact can be noted and a particular type of generalization becomes possible. If there is a difference, then the caveat should be added in subsequent claims.

We have now noted this possibility in the discussion. We have added: “For feasibility we also only extracted information about the primary outcome listed for each paper, or if no primary outcome was specified, the first listed outcome. It is possible that rates of replication differ across outcomes. Future research could consider all outcomes listed.”

On a related note, the authors conclude that RR frequencies and success rates are modest. But what is the relevant base rate? This standard may be easier to define for RR outcomes. On the optimal number of RR attempts, I think that there is less of a consensus. To state the question in a leading manner, if one replication/reproduction is better than none, why aren't two better than one? This question, and others like it, imply that observed rates of RR attempts that are less than 100% may not be suboptimal in a broader sense.

The paper specifies “When we examined the 177 individual studies replicated in the 47 documents, we found only a minority of them referred to registered protocols”. We feel this accurately reflects the data; the term modest is not used. We have added to the discussion to consider the points raised: “We acknowledge, however, that mandates for registration are rare and exist only in particular disciplines and for specific study designs.”

On a different note, I am concerned about how the coding was done. Specifically, it appears that co-authors served as "screeners" of how to categorize certain articles and article attributes. In other words, they made decision that influenced the data that they were analyzing. The risk is that people who know the hypotheses and desired outcomes implicitly bring that knowledge to coding decisions. An established best practice in fields where this type of coding is done is to first train independent coders who are unaware of the researchers; hypotheses and then conduct rigorous inter-coder reliability assessments to document the extent to which the combination of coders and categorical framework produce coding outcomes that parallel the underlying conceptual framework. Such practices increase the likelihood that data generating processes are independent of hypothesis evaluation processes.

Piloting was indeed undertaken to train reviewers to ensure consistency. This has been further specified by adding: “Prior to extraction a series of iterative pilot tests were done on included documents to ensure consistency between extractors.”

For these reasons, I am very interested in the questions that the authors ask, and what they find, but I am not convinced that a number of the empirical claims pertaining to comparisons and magnitudes will generalize even to larger populations of articles in the stated disciplines.

We have expanded the discussion about generalizability by adding: “Collectively these study design decisions and practical challenges present limitations on the overall generalizability of the findings beyond our dataset.”

I would like to see the Discussion section rewritten to focus on what the findings, and methodological challenges, mean for future work. In the current version, there are a number of speculative claims and generalizations that do not follow from the empirical work in this paper. These facts and some variation in the focus of the text in that section make the discussion longer than it needs to be and may have the effect of diminishing the excellent work that was done in the earlier pages.

The reviewer has provided helpful discussion above which we have addressed as per our rebuttal notes. We feel the issues regarding potential concerns about generalizability are now stressed in the discussion.

Reviewer #3 (Recommendations for the authors):

The authors provide a fairly clear and accurate summary of the current questions around reproducibility in the scientific literature. I was impressed that they thoughtfully included a description of the areas where their utilized methodology was different than what they had proposed in their protocol and explained the reasons for those differences. That kind of documentation is valuable and fosters transparency in the research process.

The results of their analysis are provided in both narrative format and in tables and appendices. The authors effectively characterize the different, and sometimes difficult, decisions made in parsing the results of their search. While much attention has been paid to replication and reproducibility in recent years, the nature of the results reflects the reality that the replication studies themselves may have reporting issues, and it may not always be possible to ascertain how to link the results of such studies with the original work. The study results meaningfully add to the current, small body of literature examining the meta-scientific issues of reproducibility and replication of scientific findings.

The paper is cautious in its approach to interpretation, appropriately using language such as "this may suggest that…" to better ensure that readers do not draw inappropriately firm conclusions. This is a descriptive paper, and so readers, like the authors, would best be served by limiting strong inferences based on the findings. At the same time, the authors helpfully suggest numerous meaningful pathways forward to advance reproducibility research in multiple scientific domains. Such studies would in turn facilitate more powerful inferences.

In preparing this review, the substantive majority of recommendations that I had for the authors related to points of clarity conveyed in private comments rather than overt weaknesses or concerns.

LL50-52: The meaning of this sentence is not entirely clear (though the paragraph can still be understood when skipping the sentence). Please consider rewriting to capture your point more closely.

Agree. We have deleted this line.

LL106-108: Since that analysis was published in 2014, and given the recent (albeit anecdotal) increased emphasis on reproducibility, it may be worth noting the date of the study in the text itself (or the termination date of the search used for the 0.13% statistic).

We have specified 2014 now.

L126: In this context, "some effect in inbreeding" is unclear.

We have reworded to “some effect of researcher familiarity”.

LL135-136: There may be value in disentangling the link between reproducibility and harms in the Introduction more explicitly. In the cited study (Le Noury et al), medications were utilized in a trial where reanalysis indicated that they were unlikely to produce a clinical benefit and each resulted in plausibly increased risk of harm. But it was not the failure to replicate, per se, that led to possible harms. Rather, the concern was clinical decision making predicated on the results of the original study, the results of which "stood" pending reanalysis. This may seem like a pedantic point but I think it speaks to the role not only of the original study itself, but also the degree to which decisions were made based on the findings of a single study.

Great point to consider. We have amended this to read: “In medicine, studies that do not reproduce in clinic may exaggerate patient benefits and harms¹⁴ especially when clinical decisions are based on a single study.”

L240: Can you provide a short clarification of what you mean by calibration test – such as inserting a parenthetical example "…performed a calibration test (e.g., did X) prior to…"

We have specified: “Specifically, a series of included documents were then extracted independently. The team then met to discuss differences in extraction between team members and challenges encountered before extracting from subsequent documents. This was repeated until consensus was reached.”

Table 1: The box indicating a cross section of number of authors and all studies appears to have inaccurate/incomplete information.

Thanks for catching this. This has been updated.

Table 1: The way the data are structured is a little unclear. For example, for "All Studies" the "discipline" row uses a denominator of total replication studies, whereas "year of publication" in the same column uses a denominator of the number of papers. While I see why this was done, I think additional clarity or uniformity in how the data are presented would be helpful.

This distinction has been reflected in the footnotes.

LL300-301: Could you briefly indicate whether you think that excluding documents unavailable in full via your library may have affected the results (e.g., was this 11 quantitative replications, as assessed by abstracts? or was it a mixed bag?).

We have added this to the limitation section by specifying: “We also were not able to locate the full-text of all included documents which may have impacted the results.”

L384: Does the count for psychology include the three dozen or so replication studies that were unusable for this paper? If so, do the authors think that their inclusion here serves to accurately represent the replication efforts in psychology or may artificially inflate the frequency? I don't have the perspective to know which is the case, but I think the question is worth considering.

It is a good point, but hard to know without accessing the full-text. In addition to the line above, we now specify: “The impact of these missing texts may not have been equal across disciplines.”

LL397-398: Would you be willing to share any information (if it exists) about whether independent and separate replications differ from those embedded within experimental protocols in meaningful ways (not just in terms of the obvious process differences)?

We did not examine this.

L417: Are you referring to registered protocols for the replication studies? Or registered protocols for the original study that can be used in completing the replication?

We refer to registered protocol of the replication studies. This has not been further clarified by modifying the line to read “we recorded whether a registered protocol for the replication study was used”.

LL430-432: It might be helpful for readers to be briefly informed about what "the current environment" means in practice, as there is a lot of possible variability. For example, I might assume that there is an overemphasis on novelty at the desk editing stage, but it's not clear whether the authors were even thinking of that in particular.

That’s correct. We have modified this line to read: “Of note, when no new data are generated, it may be difficult in the current research environment, which tends to favor novelty, to publish a re-analysis of existing data that shows the exact same result^31,32.”

LL434-448: There is a lot of interesting discussion in this paragraph that gets somewhat "jumbled together" in this paragraph. First, I'm not sure that it makes sense to characterize the finding as biased in L436. Rather, you found that for replication studies that were not part of the same paper, lack of overlap was common, and you do not make any assertions about studies where that was not the case (but presume the overlap might be higher – and in fact would by definition be nearly 100% since authors do not change partway through papers). Second, the larger question about what constitutes a replication is related to, but seemingly separate from, the initial discussion. Even if the authors agree on what a replication would look like for a specific study, it may not be coherent in the context of the broader field's common understanding of reproducibility.

We have re-read this section to address the jumble and streamline.

[Editors’ note: what follows is the authors’ response to the second round of review.]

The manuscript has been improved but there are some modest remaining issues that need to be addressed, as outlined below:

Reviewer #3 (Recommendations for the authors):I would like to thank the authors for their response to my original comments. In most cases, I found that the authors' revisions were responsive to my concerns. In the instances where I did not find that to be entirely the case, I have noted it below using the new line numbers.

L126: Thank you for revising the line about inbreeding. However, I think that the replacement phrase is still a little unclear. The sentence is structured as an 'either/or' and the first part suggests familiarity with the study method may increase likelihood of reproducing the same findings. So I am not sure how "researcher familiarity" functions as the alternative. The sense I get is that the authors may be hedging around suggesting that in some cases, the author's approach to research (perhaps unusually rigorous, or unusually sloppy) may carry over between studies and explain some of the variability in findings. However, that is an assumption. In any case, the authors might benefit from being very direct about their theory here.

We have amended the line to read: “This may suggest that detailed familiarity with the original study method increases the likelihood of reproducing the research findings.”

L110: The phrase "…and some disciplines do worse in this regard" might be interpreted to have two meanings. I interpret it to mean that some disciplines have fewer studies where reproductions have been attempted. However, "do worse" might also be taken to mean that some disciplines' studies are less likely to successfully have their results reproduced, which is a different concept entirely.

We have amended this line to read: “Most scientific studies are never formally reproduced and some disciplines have lower rates of reproducibility attempts than others.”

LL404-405: I appreciate this revision made in response to reviewer #1. I am unsure about whether an assertion of "unique cultures" is significantly different than the original assertion of which cultures have it as a normative value. I think the line may be fine to retain, but should be constrained with a caveat e.g., "This may suggest unique cultures around reproducibility across different disciplines, but further study is needed to determine whether this is truly the case, and our study should not be taken as proof of differences between disciplines."

We have amended this line to read: “This may suggest unique cultures around reproducibility in distinct disciplines, future research is needed to determine if such differences truly exist given the limitations of our search and approach.”

Author details

1. Kelly D Cobey

   1. Heart Institute, University of Ottawa, Ottawa, Canada
   2. School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada

   Contribution

   Conceptualization, Data curation, Formal analysis, Supervision, Investigation, Methodology, Writing - original draft, Project administration, Writing – review and editing

   For correspondence

   kcobey@ottawaheart.ca

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2797-1686

2. Christophe A Fehlmann

   1. School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
   2. Department of Anaesthesiology, Clinical Pharmacology, Intensive Care and Emergency Medicine, Geneva University Hospitals, Geneva, Switzerland

   Contribution

   Data curation, Formal analysis, Investigation, Visualization, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

3. Marina Christ Franco

   1. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
   2. School of Dentistry, Federal University of Pelotas, Pelotas, Brazil

   Contribution

   Data curation, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

4. Ana Patricia Ayala

   Gerstein Science Information Centre, University of Toronto, Toronto, Canada

   Contribution

   Conceptualization, Data curation, Validation, Investigation, Methodology, Project administration, Writing – review and editing

   Competing interests

   No competing interests declared

5. Lindsey Sikora

   Health Sciences Library, University of Ottawa, Ottawa, Canada

   Contribution

   Conceptualization, Data curation, Validation, Investigation, Methodology, Project administration, Writing – review and editing

   Competing interests

   No competing interests declared

6. Danielle B Rice

   1. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
   2. Department of Psychology, McGill University, Montreal, Canada

   Contribution

   Data curation, Validation, Investigation, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

7. Chenchen Xu

   1. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
   2. Department of Medicine, University of Ottawa, Ottawa, Canada

   Contribution

   Validation, Investigation, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

8. John PA Ioannidis

   Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford, Stanford University, Stanford, United States

   Contribution

   Conceptualization, Validation, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3118-6859

9. Manoj M Lalu

   1. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
   2. Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, Canada
   3. Regenerative Medicine Program, Ottawa Hospital, Ottawa, Canada

   Contribution

   Validation, Investigation, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0322-382X

10. Alixe Ménard

    Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada

    Contribution

    Validation, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

11. Andrew Neitzel

    1. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
    2. Department of Medicine, University of Ottawa, Ottawa, Canada

    Contribution

    Validation, Investigation, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

12. Bea Nguyen

    1. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
    2. Department of Medicine, University of Ottawa, Ottawa, Canada

    Contribution

    Validation, Investigation, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

13. Nino Tsertsvadze

    Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada

    Contribution

    Validation, Investigation, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

14. David Moher

    1. School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
    2. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada

    Contribution

    Conceptualization, Supervision, Validation, Investigation, Methodology, Writing – review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-2434-4206

• 713

  Page views

• 44

  Downloads

• 1

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.