1. Medicine

Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review of articles published in 2018-2019

  1. Kelly D Cobey  Is a corresponding author
  2. Christophe A Fehlmann
  3. Marina Christ Franco
  4. Ana Patricia Ayala
  5. Lindsey Sikora
  6. Danielle B Rice
  7. Chenchen Xu
  8. John PA Ioannidis
  9. Manoj M Lalu
  10. Alixe Ménard
  11. Andrew Neitzel
  12. Bea Nguyen
  13. Nino Tsertsvadze
  14. David Moher
  1. Heart Institute, University of Ottawa, Canada
  2. School of Epidemiology and Public Health, University of Ottawa, Canada
  3. Department of Anaesthesiology, Clinical Pharmacology, Intensive Care and Emergency Medicine, Geneva University Hospitals, Switzerland
  4. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Canada
  5. School of Dentistry, Federal University of Pelotas, Brazil
  6. Gerstein Science Information Centre, University of Toronto, Canada
  7. Health Sciences Library, University of Ottawa, Canada
  8. Department of Psychology, McGill University, Canada
  9. Department of Medicine, University of Ottawa, Canada
  10. Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford, Stanford University, United States
  11. Department of Anesthesiology and Pain Medicine, University of Ottawa, Canada
  12. Regenerative Medicine Program, Ottawa Hospital, Canada
https://doi.org/10.7554/eLife.78518
Share this article
Cite this article
  1. Kelly D Cobey
  2. Christophe A Fehlmann
  3. Marina Christ Franco
  4. Ana Patricia Ayala
  5. Lindsey Sikora
  6. Danielle B Rice
  7. Chenchen Xu
  8. John PA Ioannidis
  9. Manoj M Lalu
  10. Alixe Ménard
  11. Andrew Neitzel
  12. Bea Nguyen
  13. Nino Tsertsvadze
  14. David Moher
(2023)
Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review of articles published in 2018-2019
eLife 12:e78518.
https://doi.org/10.7554/eLife.78518
  2. Download BibTeX
  3. Download .RIS
1 figure, 3 tables and 1 additional file

Figures

Figure 1
Download asset Open asset
Flow diagram of articles.

Tables

Table 1
Characteristics of included documents.
CharacteristicCategoriesAll studiesSingle replication papers(N=36)Multiple replication papers(N=11)
N (%) (unless otherwise indicated)
What discipline does the work best fit in?*Biomedicine
Economics
Education
Health sciences
Psychology
Other (mixture of two or more of the abov1e)		6 (3.4)
5 (2.8)
5 (2.8)
42 (23.7)
86 (48.6)
33 (18.6)		6 (16.7)
5 (13.9)
1 (2.8)
9 (25.0)
15 (41.7)
-		-
-
1 (9.1)
4 (36.4)
4 (36.4)
2 (18.2)
Year of publication2018
2019		28 (59.6)
19 (40.4)		21 (58.3)
15 (41.7)		7 (63.6)
4 (36.4)
Country of corresponding author (reported based on Top 3 overall)USA
The Netherlands
Australia		27 (57.4)
4 (8.5)
3 (6.4)		19 (52.8)
3 (8.3)
3 (8.3)		8 (72.7)
1 (9.1)
-
Number of authors†Median
Range		3
1–172		3
1–124		4
1–172
FundingYes
No
Not reported		32 (68.1)
6 (12.8)
9 (19.1)		23 (63.9)
5 (13.9)
8 (22.2)		9 (81.8)
1 (9.1)
1 (9.1)
Funding source‡Government
Academic
Non-profit
Unsure		19 (59.4)
15 (46.9)
14 (43.8)
1 (3.1)		17 (73.9)
9 (39.1)
9 (39.1)
-		2 (22.2)
6 (66.7)
5 (55.6)
1 (11.1)
Ethics approvalYes
No
Ethics approval not relevant		23 (48.9)
10 (21.3)
14 (29.8)		17 (47.2)
8 (22.2)
11 (30.6)		6 (54.5)
2 (18.2)
3 (27.3)
  1. *

    Data reported at the study level.

  2. †

    Data reports median and range.

  3. ‡

    Data refers to funded studies only, some studies report multiple funding sources.

Table 2
Study replication methods characteristics.
CharacteristicCategoriesAll discipline studies(N=177)BiomedicineN (%)EconomicsN (%)EducationN (%)Health sciences*N (%)PsychologyN (%)Other (mixture of two or more of the above)N (%)
Did the replication study team specify that they contacted the original study project team?Yes, the author teams overlapped
Yes, there was contact
No, the teams did not overlap or contact		16 (9.0)
44 (24.9)
117 (66.1)		2 (33.3)
-
4 (66.7)		-
-
5 (100)		-
-
5 (100)		4 (9.5)
14 (33.3)
24 (57.1)		10 (11.6)
9 (10.5)
67 (77.9)		-
21 (63.6)
12 (36.4)
Does the replication study refer to a protocol that was registered prior to data collection?Yes
No		81 (45.8)
96 (54.2)		2 (33.3)
4 (66.7)		-
5 (100)		1 (20.0)
4 (80.0)		18 (42.9)
24 (57.1)		39 (45.3)
47 (54.7)		21 (63.6)
12 (36.4)
Do the authors specify that they used an identical protocol?Yes
No †
Not reported
Unsure		41 (23.2)
70 (39.5)
64 (36.2)
2 (1.1)		2 (33.3)
1 (16.7)
-
2 (33.3)		1 (20.0)
3 (60.0)
1 (20.0)
-		-
3 (60.0)
2 (40.0)
-		9 (21.4)
15 (35.7)
17 (40.5)
-		8 (9.3)
34 (39.5)
44 (51.2)
-		21 (63.6)
12 (36.64)
-
-
Does the study indicate that data is shared publicly?Yes‡
No		112 (63.3)
65 (36.7)		2 (33.3)
4 (66.7)		-
5 (100)		1 (20.0)
4 (80.0)		18 (42.9)
24 (57.1)		70 (81.4)
16 (18.6)		21 (63.6)
12 (36.4)
What is the study design used?Data re-analysis
Experimental
Observational
Trial		35 (19.8)
52 (29.4)
85 (48.0)
5 (2.8)		-
2 (33.3)
3 (50.0)
1 (16.7)		3 (60.0)
1 (20.0)
1 (20.0)
-		1 (20.0)
-
3 (60.0)
1 (20.0)		26 (61.9)
10 (23.8)
3 (7.1)
3 (7.1)		5 (5.8)
39 (45.3)
42 (48.8)
-		33 (100)
-
Did the study specify a primary outcome?Yes
No		43 (24.3)
134 (75.7)		-
6 (100)		-
5 (100)		-
5 (100)		26 (61.9)
16 (38.1)		13 (15.1)
73 (84.9)		-
33 (100)
  1. *

    One study provided results by outcome not by studies being replicated, in this instance we were unable to determine how the results corresponded to the studies the author listed they replicated so these data are missing.

  2. †

    In these instances authors specified deviations between their protocol and that of the original research team.

  3. ‡

    This was not verified. We simply recorded what the authors reported. It is possible that self-reported sharing and rates of actual sharing are not identical.

Table 3
Reproducibility characteristics of studies replicated overall and across disciplines.
CharacteristicCategoriesOverallBiomedicineEconomicsEducationHealth sciencesPsychologyOther
How did the authors assess reproducibility?Effect sizes
Meta analysis of original effect size
Null hypothesis testing using p-value
Subjective assessment
Other		116 (65.5)
33 (18.6)
17 (9.6)
5 (2.8)
6 (3.4)		1 (16.7)
2 (33.3)
2 (33.3)
-
1 (16.7)		1 (20.0)
-
2 (40.0)
1 (20.0)
1 (20.0)		1 (20.0)
-
-
2 (40.0)
2 (40.0)		25 (59.5)
9 (21.4)
5 (11.9)
2 (4.8)
1 (2.4)		76 (88.4)
1 (1.2)
8 (9.3)
-
1 (1.2)		12 (36.4)
21 (63.6)
-
-
-
Based on the authors definition of reproducibility, did the study replicate?Yes
No
Mixed
Unclear		95 (53.7)
36 (20.3)
8 (4.5)
38 (21.5)		4 (66.7)
1 (16.7)
1 (16.7)
-		4 (80.0)
1 (20.0)
-
-		2 (40.0)
2 (40.0)
1 (20.0)
-		36 (85.7)
4 (9.5)
1 (2.4)
1 (2.4)		25 (29.1)
19 (22.1)
5 (5.8)
37 (43.0)		24 (72.7)
9 (27.3)
-
-
Was the p-value reported on the statistical test conducted on the primary outcome?Yes
No/unclear		116 (65.5)
61 (34.5)		3 (50.0)
3 (50.0)		3 (60.0)
2 (40.0)		4 (80.0)
1 (20.0)		33 (78.6)
9 (21.4)		45 (52.3)
41 (47.7)		28 (84.8)
5 (15.2)

Additional files

MDAR checklist
https://cdn.elifesciences.org/articles/78518/elife-78518-mdarchecklist1-v2.docx

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Kelly D Cobey
  2. Christophe A Fehlmann
  3. Marina Christ Franco
  4. Ana Patricia Ayala
  5. Lindsey Sikora
  6. Danielle B Rice
  7. Chenchen Xu
  8. John PA Ioannidis
  9. Manoj M Lalu
  10. Alixe Ménard
  11. Andrew Neitzel
  12. Bea Nguyen
  13. Nino Tsertsvadze
  14. David Moher
(2023)
Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review of articles published in 2018-2019
eLife 12:e78518.
https://doi.org/10.7554/eLife.78518