Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review of articles published in 2018-2019
- Heart Institute, University of Ottawa, Canada
- School of Epidemiology and Public Health, University of Ottawa, Canada
- Department of Anaesthesiology, Clinical Pharmacology, Intensive Care and Emergency Medicine, Geneva University Hospitals, Switzerland
- Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Canada
- School of Dentistry, Federal University of Pelotas, Brazil
- Gerstein Science Information Centre, University of Toronto, Canada
- Health Sciences Library, University of Ottawa, Canada
- Department of Psychology, McGill University, Canada
- Department of Medicine, University of Ottawa, Canada
- Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford, Stanford University, United States
- Department of Anesthesiology and Pain Medicine, University of Ottawa, Canada
- Regenerative Medicine Program, Ottawa Hospital, Canada
Figures
Figure 1
Flow diagram of articles.
Tables
Table 1
Characteristics of included documents.
| Characteristic | Categories | All studies | Single replication papers(N=36) | Multiple replication papers(N=11) |
|---|---|---|---|---|
| N (%) (unless otherwise indicated) | ||||
| What discipline does the work best fit in?* | Biomedicine Economics Education Health sciences Psychology Other (mixture of two or more of the abov1e) | 6 (3.4) 5 (2.8) 5 (2.8) 42 (23.7) 86 (48.6) 33 (18.6) | 6 (16.7) 5 (13.9) 1 (2.8) 9 (25.0) 15 (41.7) - | - - 1 (9.1) 4 (36.4) 4 (36.4) 2 (18.2) |
| Year of publication | 2018 2019 | 28 (59.6) 19 (40.4) | 21 (58.3) 15 (41.7) | 7 (63.6) 4 (36.4) |
| Country of corresponding author (reported based on Top 3 overall) | USA The Netherlands Australia | 27 (57.4) 4 (8.5) 3 (6.4) | 19 (52.8) 3 (8.3) 3 (8.3) | 8 (72.7) 1 (9.1) - |
| Number of authors† | Median Range | 3 1–172 | 3 1–124 | 4 1–172 |
| Funding | Yes No Not reported | 32 (68.1) 6 (12.8) 9 (19.1) | 23 (63.9) 5 (13.9) 8 (22.2) | 9 (81.8) 1 (9.1) 1 (9.1) |
| Funding source‡ | Government Academic Non-profit Unsure | 19 (59.4) 15 (46.9) 14 (43.8) 1 (3.1) | 17 (73.9) 9 (39.1) 9 (39.1) - | 2 (22.2) 6 (66.7) 5 (55.6) 1 (11.1) |
| Ethics approval | Yes No Ethics approval not relevant | 23 (48.9) 10 (21.3) 14 (29.8) | 17 (47.2) 8 (22.2) 11 (30.6) | 6 (54.5) 2 (18.2) 3 (27.3) |
-
*
Data reported at the study level.
-
†
Data reports median and range.
-
‡
Data refers to funded studies only, some studies report multiple funding sources.
Table 2
Study replication methods characteristics.
| Characteristic | Categories | All discipline studies(N=177) | BiomedicineN (%) | EconomicsN (%) | EducationN (%) | Health sciences*N (%) | PsychologyN (%) | Other (mixture of two or more of the above)N (%) |
|---|---|---|---|---|---|---|---|---|
| Did the replication study team specify that they contacted the original study project team? | Yes, the author teams overlapped Yes, there was contact No, the teams did not overlap or contact | 16 (9.0) 44 (24.9) 117 (66.1) | 2 (33.3) - 4 (66.7) | - - 5 (100) | - - 5 (100) | 4 (9.5) 14 (33.3) 24 (57.1) | 10 (11.6) 9 (10.5) 67 (77.9) | - 21 (63.6) 12 (36.4) |
| Does the replication study refer to a protocol that was registered prior to data collection? | Yes No | 81 (45.8) 96 (54.2) | 2 (33.3) 4 (66.7) | - 5 (100) | 1 (20.0) 4 (80.0) | 18 (42.9) 24 (57.1) | 39 (45.3) 47 (54.7) | 21 (63.6) 12 (36.4) |
| Do the authors specify that they used an identical protocol? | Yes No † Not reported Unsure | 41 (23.2) 70 (39.5) 64 (36.2) 2 (1.1) | 2 (33.3) 1 (16.7) - 2 (33.3) | 1 (20.0) 3 (60.0) 1 (20.0) - | - 3 (60.0) 2 (40.0) - | 9 (21.4) 15 (35.7) 17 (40.5) - | 8 (9.3) 34 (39.5) 44 (51.2) - | 21 (63.6) 12 (36.64) - - |
| Does the study indicate that data is shared publicly? | Yes‡ No | 112 (63.3) 65 (36.7) | 2 (33.3) 4 (66.7) | - 5 (100) | 1 (20.0) 4 (80.0) | 18 (42.9) 24 (57.1) | 70 (81.4) 16 (18.6) | 21 (63.6) 12 (36.4) |
| What is the study design used? | Data re-analysis Experimental Observational Trial | 35 (19.8) 52 (29.4) 85 (48.0) 5 (2.8) | - 2 (33.3) 3 (50.0) 1 (16.7) | 3 (60.0) 1 (20.0) 1 (20.0) - | 1 (20.0) - 3 (60.0) 1 (20.0) | 26 (61.9) 10 (23.8) 3 (7.1) 3 (7.1) | 5 (5.8) 39 (45.3) 42 (48.8) - | 33 (100) - |
| Did the study specify a primary outcome? | Yes No | 43 (24.3) 134 (75.7) | - 6 (100) | - 5 (100) | - 5 (100) | 26 (61.9) 16 (38.1) | 13 (15.1) 73 (84.9) | - 33 (100) |
-
*
One study provided results by outcome not by studies being replicated, in this instance we were unable to determine how the results corresponded to the studies the author listed they replicated so these data are missing.
-
†
In these instances authors specified deviations between their protocol and that of the original research team.
-
‡
This was not verified. We simply recorded what the authors reported. It is possible that self-reported sharing and rates of actual sharing are not identical.
Table 3
Reproducibility characteristics of studies replicated overall and across disciplines.
| Characteristic | Categories | Overall | Biomedicine | Economics | Education | Health sciences | Psychology | Other |
|---|---|---|---|---|---|---|---|---|
| How did the authors assess reproducibility? | Effect sizes Meta analysis of original effect size Null hypothesis testing using p-value Subjective assessment Other | 116 (65.5) 33 (18.6) 17 (9.6) 5 (2.8) 6 (3.4) | 1 (16.7) 2 (33.3) 2 (33.3) - 1 (16.7) | 1 (20.0) - 2 (40.0) 1 (20.0) 1 (20.0) | 1 (20.0) - - 2 (40.0) 2 (40.0) | 25 (59.5) 9 (21.4) 5 (11.9) 2 (4.8) 1 (2.4) | 76 (88.4) 1 (1.2) 8 (9.3) - 1 (1.2) | 12 (36.4) 21 (63.6) - - - |
| Based on the authors definition of reproducibility, did the study replicate? | Yes No Mixed Unclear | 95 (53.7) 36 (20.3) 8 (4.5) 38 (21.5) | 4 (66.7) 1 (16.7) 1 (16.7) - | 4 (80.0) 1 (20.0) - - | 2 (40.0) 2 (40.0) 1 (20.0) - | 36 (85.7) 4 (9.5) 1 (2.4) 1 (2.4) | 25 (29.1) 19 (22.1) 5 (5.8) 37 (43.0) | 24 (72.7) 9 (27.3) - - |
| Was the p-value reported on the statistical test conducted on the primary outcome? | Yes No/unclear | 116 (65.5) 61 (34.5) | 3 (50.0) 3 (50.0) | 3 (60.0) 2 (40.0) | 4 (80.0) 1 (20.0) | 33 (78.6) 9 (21.4) | 45 (52.3) 41 (47.7) | 28 (84.8) 5 (15.2) |
