Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019
- University of Ottawa, Canada
- Federal University of Pelotas, Brazil
- University of Toronto, Canada
- McGill University, Canada
- Stanford University, United States
- Ottawa Hospital Research Institute, Canada
Abstract
Introduction: Reproducibility is a central tenant of research. We aimed to synthesize the literature on reproducibility and describe its epidemiological characteristics, including how reproducibility is defined and assessed. We also aimed to determine and compare estimates for reproducibility across different fields.
Methods: We conducted a scoping review to identify English language replication studies published between 2018-2019 in economics, education, psychology, health sciences and biomedicine. We searched Medline, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature – CINAHL, Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences (IBSS), and EconLit. Documents retrieved were screened in duplicate against our inclusion criteria. We extracted year of publication, number of authors, country of affiliation of the corresponding author, and whether the study was funded. For the individual replication studies, we recorded whether a registered protocol for the replication study was used, whether there was contact between the reproducing team and the original authors, what study design was used, and what the primary outcome was. Finally, we recorded how reproducibilty was defined by the authors, and whether the assessed study(ies) successfully reproduced based on this definition. Extraction was done by a single reviewer and quality controlled by a second reviewer.
Results: Our search identified 11,224 unique documents, of which 47 were included in this review. Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single reproducibility study while the remaining 11 reported at least two reproducibility studies in the same paper. Less than the half of the studies referred to a registered protocol. There was variability in the definitions of reproduciblity success. In total, across the 47 documents 177 studies were reported. Based on the definition used by the author of each study, 95 of 177 (53.7%) studies reproduced.
Conclusion: This study gives an overview of research across five disciplines that explicitly set out to reproduce previous research. Such reproducibility studies are extremely scarce, the definition of a successfully reproduced study is ambiguous, and the reproducibility rate is overall modest.
Funding: No external funding was received for this work.
Data availability
Data and materials are available on the Open Science Framework (https://osf.io/wn7gm/).
-
Scoping review data on reproducibility of research in a 2018-2019 multi-discipline sampleOpen Science FrameworkDOI 10.17605/OSF.IO/WN7GM.
Article and author information
Author details
Funding
The authors declare that there was no funding for this work.
Copyright
© 2023, Cobey et al.
This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.
Metrics
-
- 1,090
- views
-
- 104
- downloads
-
- 5
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019
eLife 12:e78518.
https://doi.org/10.7554/eLife.78518
Further reading
-
- Medicine
Estrogen significantly impacts women’s health, and postmenopausal hypertension is a common issue characterized by blood pressure fluctuations. Current control strategies for this condition are limited in efficacy, necessitating further research into the underlying mechanisms. Although metabolomics has been applied to study various diseases, its use in understanding postmenopausal hypertension is scarce. Therefore, an ovariectomized rat model was used to simulate postmenopausal conditions. Estrogen levels, blood pressure, and aortic tissue metabolomics were analyzed. Animal models were divided into Sham, OVX, and OVX +E groups. Serum estrogen levels, blood pressure measurements, and aortic tissue metabolomics analyses were performed using radioimmunoassay, UHPLC-Q-TOF, and bioinformatics techniques. Based on the above research content, we successfully established a correlation between low estrogen levels and postmenopausal hypertension in rats. Notable differences in blood pressure parameters and aortic tissue metabolites were observed across the experimental groups. Specifically, metabolites that were differentially expressed, particularly L-alpha-aminobutyric acid (L-AABA), showed potential as a biomarker for postmenopausal hypertension, potentially exerting a protective function through macrophage activation and vascular remodeling. Enrichment analysis revealed alterations in sugar metabolism pathways, such as the Warburg effect and glycolysis, indicating their involvement in postmenopausal hypertension. Overall, this current research provides insights into the metabolic changes associated with postmenopausal hypertension, highlighting the role of L-AABA and sugar metabolism reprogramming in aortic tissue. The findings suggest a potential link between low estrogen levels, macrophage function, and vascular remodeling in the pathogenesis of postmenopausal hypertension. Further investigations are needed to validate these findings and explore their clinical implications for postmenopausal women.
-
- Medicine
- Neuroscience
Background:
Subarachnoid hemorrhage (SAH) is characterized by intense central inflammation, leading to substantial post-hemorrhagic complications such as vasospasm and delayed cerebral ischemia. Given the anti-inflammatory effect of transcutaneous auricular vagus nerve stimulation (taVNS) and its ability to promote brain plasticity, taVNS has emerged as a promising therapeutic option for SAH patients. However, the effects of taVNS on cardiovascular dynamics in critically ill patients, like those with SAH, have not yet been investigated. Given the association between cardiac complications and elevated risk of poor clinical outcomes after SAH, it is essential to characterize the cardiovascular effects of taVNS to ensure this approach is safe in this fragile population. Therefore, this study assessed the impact of both acute and repetitive taVNS on cardiovascular function.
Methods:
In this randomized clinical trial, 24 SAH patients were assigned to either a taVNS treatment or a sham treatment group. During their stay in the intensive care unit, we monitored patient electrocardiogram readings and vital signs. We compared long-term changes in heart rate, heart rate variability (HRV), QT interval, and blood pressure between the two groups. Additionally, we assessed the effects of acute taVNS by comparing cardiovascular metrics before, during, and after the intervention. We also explored acute cardiovascular biomarkers in patients exhibiting clinical improvement.
Results:
We found that repetitive taVNS did not significantly alter heart rate, QT interval, blood pressure, or intracranial pressure (ICP). However, repetitive taVNS increased overall HRV and parasympathetic activity compared to the sham treatment. The increase in parasympathetic activity was most pronounced from 2 to 4 days after initial treatment (Cohen’s d = 0.50). Acutely, taVNS increased heart rate, blood pressure, and peripheral perfusion index without affecting the corrected QT interval, ICP, or HRV. The acute post-treatment elevation in heart rate was more pronounced in patients who experienced a decrease of more than one point in their modified Rankin Score at the time of discharge.
Conclusions:
Our study found that taVNS treatment did not induce adverse cardiovascular effects, such as bradycardia or QT prolongation, supporting its development as a safe immunomodulatory treatment approach for SAH patients. The observed acute increase in heart rate after taVNS treatment may serve as a biomarker for SAH patients who could derive greater benefit from this treatment.
Funding:
The American Association of Neurological Surgeons (ALH), The Aneurysm and AVM Foundation (ALH), The National Institutes of Health R01-EB026439, P41-EB018783, U24-NS109103, R21-NS128307 (ECL, PB), McDonnell Center for Systems Neuroscience (ECL, PB), and Fondazione Neurone (PB).
Clinical trial number:
