Investigating the replicability of preclinical cancer biology

Science is a system for accumulating knowledge. Independent researchers and teams study topics and make claims about nature based on the evidence that they gather. They also share their work so that others can evaluate, extend, or challenge the evidence and claims. The accumulation of evidence weeds out claims that are unreliable, and supports the development of new models and theories. Over time, uncertainty declines and the accumulated knowledge provides useful descriptions, effective predictions, and a better understanding of nature. Humanity applies this system in service of creating knowledge, treating disease, and advancing society.

An important feature of this system is replication (Hempel, 1968; Musgrave, 1970; Nosek and Errington, 2020a; Salmon and Glymour, 1999). A scientific claim is said to be replicable if it is supported by new data. However, it is often not straightforward to decide if a claim is supported by new data or not. Moreover, the success or failure of an attempt to replicate rarely provides a definitive answer about the credibility of an original claim. When the replication attempt is successful, confidence in the reliability of the claim increases, but that does not mean that the claim is valid: a finding can be both replicable and invalid at the same time. Repeated successful replications can help to eliminate alternative explanations and potential confounding influences, and therefore increase confidence in both reliability and validity, but they might not eliminate all confounding influences. It is possible that the original experiment and all the replication attempts could be invalidated by a common shortcoming in experimental design.

When a replication attempt is not successful, it is possible that the original was a false positive – noise mistaken as a signal. It is possible that the original claim was overly generalized and is only replicable under a much narrower range of conditions than was originally believed. It is also possible that the methodology necessary to produce the evidence is not sufficiently defined or understood, or that the theoretical explanation for why the finding occurred is incorrect. Failures in implementing experimental protocols may also result in replication attempts being uninformative.

All of these possibilities are ordinary and can occur when researchers have been rigorous in their work. What should not be ordinary is persistent failure to recognize that some scientific claims are not replicable. Science advances via self-correction, the progressive identification and elimination of error. Self-correction requires a healthy verification process that recognizes non-replicability, eliminates unproductive paths, and redirects attention and resources to promising directions. A failure to recognize that some claims are not replicable can foster overconfidence, underestimate uncertainty, and hinder scientific progress.

There is accumulating evidence that non-replicability may be occurring at higher rates than recognized, potentially undermining credibility and self-correction. Theoretical analyses point to a system of incentives that prioritizes innovation over verification, leading to infrequent efforts to replicate findings and to behaviors that could reduce the replicability of published findings such as selective reporting, presenting exploratory findings as confirmatory tests, and failures of documentation, transparency, and sharing (Casadevall and Fang, 2012; Gelman and Loken, 2013; Greenwald, 1975; Kimmelman et al., 2014; Makel et al., 2012; Nosek et al., 2012; Rosenthal, 1979). For instance, one theoretical analysis estimated that more than half of research findings are false (Ioannidis, 2005). And in a survey, 60% of biologists who responded reported that they had failed to replicate their own results, and more than 75% had failed to replicate results from a different lab (Baker, 2016).

Large-scale replication studies in the social and behavioral sciences provide evidence of replicability challenges (Camerer et al., 2016; Camerer et al., 2018; Ebersole et al., 2016; Ebersole et al., 2020; Klein et al., 2014; Klein et al., 2018; Open Science Collaboration, 2015). In psychology, across 307 systematic replications and multisite replications, 64% reported statistically significant evidence in the same direction and effect sizes 68% as large as the original experiments (Nosek et al., 2021).

In the biomedical sciences, the ALS Therapy Development Institute observed no effectiveness of more than 100 potential drugs in a mouse model in which prior research reported effectiveness in slowing down disease, and eight of those compounds were tried and failed in clinical trials costing millions and involving thousands of participants (Perrin, 2014). Of 12 replications of preclinical spinal cord injury research in the FORE-SCI program, only two clearly replicated the original findings – one under constrained conditions of the injury and the other much more weakly than the original (Steward et al., 2012). And, in cancer biology and related fields, two drug companies (Bayer and Amgen) reported failures to replicate findings from promising studies that could have led to new therapies (Prinz et al., 2011; Begley and Ellis, 2012). Their success rates (25% for the Bayer report, and 11% for the Amgen report) provided disquieting initial evidence that preclinical research may be much less replicable than recognized. Unfortunately, because of proprietary concerns, very little information was made available on the studies that failed to replicate, on the replication methodology, or on the particular barriers encountered for replicating the findings. This lack of transparency makes it difficult to ascertain the reasons for failures to replicate and critique the basis of the claims.

In the Reproducibility Project: Cancer Biology, we sought to acquire evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from 53 high-impact papers published in 2010, 2011, and 2012 (Errington et al., 2014). We describe in a companion paper (Errington et al., 2021b) the challenges we encountered while repeating these experiments. These barriers include: shortcomings in documentation of the original methodology; failures of transparency in original findings and protocols; failures to share original data, reagents, and other materials; methodological challenges encountered during the execution of the replication experiments. These challenges meant that we only completed 50 of the 193 experiments (26%) we planned to repeat. The 50 experiments that we were able to complete included a total of 158 effects that could be compared with the same effects in the original paper. It was common for experiments to have multiple effects, such as assessing whether an intervention affected both tumor burden and overall survival, or assessing the impact that depleting different genes has on cellular proliferation.

In this paper, we report the results of a meta-analysis of all these comparisons. There is no single method for assessing the success or failure of replication attempts (Mathur and VanderWeele, 2019; Open Science Collaboration, 2015; Valentine et al., 2011), so we used seven different methods to compare the effect reported in the original paper and the effect observed in the replication attempt (see Results). Six of these methods were dichotomous (i.e., replication success/failure) and one was not.

In total, 136 of the 158 effects (86%) reported in the original papers were positive effects – the original authors interpreted their data as showing that a relationship between variables existed or that an intervention had an impact on the biological system being studied. The other 22 (14%) were null effects – the original authors interpreted their data as not showing evidence for a meaningful relationship or impact of an intervention. Furthermore, 117 of the effects reported in the original papers (74%) were supported by a numerical result (such as graphs of quantified data or statistical tests), and 41 (26%) were supported by a representative image or similar. For effects where the original paper reported a numerical result for a positive effect, it was possible to use all seven methods of comparison. However, for cases where the original paper relied on a representative image (without a numerical result) as evidence for a positive effect, or when the original paper reported a null effect, it was not possible to use all seven methods.

In this section we discuss the seven different methods that we used to assess replication attempts, and report what we found when we used these methods to compare the effects reported in the original papers and the effects observed in the replications. The results are reported in Table 1. We display the results of original positive effects and original null effects separately; we also display cases where the original effect was reported as a numerical value separate from cases where the original effect was reported as a representative image. In some cases we conducted two or more internal replication experiments for the same original effect, which increased the total number of outcomes from 158 to 188 (see Materials and methods). In the text of this article we mostly report and discuss our results in terms of effects, the relevant tables and figures report the results by outcome, effect, experiment, and paper.

The nested structure of outcomes within effects, effects within experiments, and experiments within papers provides different ways to characterize the results, and it is possible for some effects within an experiment to replicate successfully while other effects in the same experiment fail to replicate. However, the results are similar irrespective of whether we look at them by paper (23 in total), by experiment (50 in total), by effect (158 in total), or by outcome (188 in total). We also use a number of strategies for aggregating data across effects and experiments, but observe very similar findings regardless of method used for aggregation (see Tables S1–S3 in Supplementary file 1).

Evaluating replications against a null hypothesis

Null hypothesis significance testing is used to test for evidence that an observed effect size or something larger would have been unlikely to occur under the null hypothesis. For positive original results, it is straightforward to assess whether a replication effect observes a statistically significant result in the same direction as the original effect. The simplicity of this indicator has led to it being a common replication criterion despite its dichotomous nature, its dependence on the power of the replication experiment, and challenges for proper interpretation (Andrews and Kasy, 2019; Camerer et al., 2016; Camerer et al., 2018; Open Science Collaboration, 2015; Patil et al., 2016; Valentine et al., 2011). Of the 112 original effects with associated statistical significance tests, 97 were interpreted as positive effects, and 15 were interpreted as null effects (Figure 1).

Figure 1 with 1 supplement see all

Download asset Open asset

For original positive effects, 42 of the 97 (43%) replication effects were statistically significant and in the same direction as the original effect; 48 (49%) were null results; and 7 (7%) were statistically significant in the opposite direction. Based on the power of the experiments, if the replications were all statistically consistent with the original experiments, we would expect approximately 87% of replications to be statistically significant and positive (Mathur and VanderWeele, 2020b), which is considerably higher than what we observed (43% [95% CI: 25%, 62%]). A sensitivity analysis that approximately accounts for possible heterogeneity within pairs also yielded a value (85%) that was considerably higher than what was observed.

For the original null effects, 11 (73%) replication effects were null results and 4 (27%) were statistically significant. Combining positive effects that remained positive and null effects that remained null, 53 of 112 (47%) of the replications were consistent with the original effects.

For cases in which the original findings were reported as representative images, we were able to conduct statistical significance tests for the replications: of the 22 effects that were positive in the original experiments, 14 (64%) replications were statistically significant in the same direction. And of the five null effects in the original experiments, 4 replications were also null (Table 1).

A weakness of this approach to assessing replication results is that it treats p = 0.05 as a bright-line criterion between replication success and failure. For example, if an excess of findings fell just above p = 0.05 it could indicate false negatives are present in the non-statistically significant outcomes of original positive results. p-values for non-statistically significant replication effects were widely distributed (Figure 1—figure supplement 1), and do not statistically differ from the approximately uniform distribution that would be expected if all were true null results whether examining the findings that had p-values for both original and replication effects (Fisher’s exact test: χ²(118) = 135.7, p = 0.127), or also including the replication effects for which the original effects were based on a representative image (χ²(138) = 155.1, p = 0.152). Therefore, we cannot reject the hypothesis that the observed null effects come from a population of true negatives.

Comparing effect sizes in the original experiments and the replications

Another way to assess replications is to compare the original effect size and the replication effect size. For the 97 effects that were positive in the original experiments, the effect size was lower in 94 (97%) of the replications (Table 1): if the original effect sizes were accurately estimated, one would expect this percentage to be about 50%, and the probability of 94 of the 97 replication effect sizes being lower than the original effect sizes would be vanishingly low (binomial test: p = 1.92 × 10^–24). And for the 21 cases in which the original evidence for a positive effect was a representative image, the effect size in the replication was smaller than the effect size estimated for the original in 14 cases (67%). Combining these results, the effect sizes in the replications were smaller than the effect sizes in the original findings in 108 of 118 (92%) cases.

We also compared the mean and median values of the effect sizes for positive effects (Table 3): in both cases the value was considerably larger for the original effect. Comparing means, the value for the original effects was 6.15 (SD = 12.39, 95% CI: [1.83, 10.47]), and the value for the replications was 1.37 (SD = 3.01, 95% CI: [0.42, 2.32]). Comparing medians, the value for the original effects was 2.96 (interquartile range [IQR] = 1.71–5.70), and the value for the replications was 0.43 (IQR = 0.15–2.06).

Table 3

	Papers	Experiments	Effects	All outcomes
ORIGINAL POSITIVE RESULTS
Number of outcomes	19	33	97	112
Mean (SD) original experiment effect size	7.35 (18.77)	6.36 (14.62)	6.15 (12.39)	5.56 (11.63)
Median [IQR] original experiment effect size	2.07 [1.68–5.03]	2.45 [1.42–4.58]	2.96 [1.71–5.70]	2.57 [1.60–5.49]
Mean (SD) replication experiment effect size	1.38 (2.02)	1.55 (3.31)	1.37 (3.01)	1.30 (2.83)
Median [IQR] replication experiment effect size	0.53 [0.18–1.80]	0.37 [0.10–1.31]	0.43 [0.15–2.06]	0.47 [0.17–1.67]
Meta-analytic mean (SD) estimate	1.68 (1.81)	1.79 (2.90)	1.66 (2.47)	1.61 (2.32)
Meta-analytic median [IQR] estimate	0.98 [0.57–2.20]	1.00 [0.28–2.03]	0.92 [0.36–2.43]	1.05 [0.36–2.11]
Sample sizes
Median [IQR] of original	46.0 [20.0–100]	24.0 [9.0–48.0]	8.0 [6.0–13.0]	8.5 [6.0–18.0]
Median [IQR] of replication	50.0 [28.0–128]	32.0 [12.0–50.0]	12.0 [8.0–23.0]	12.0 [8.0–18.0]
ORIGINAL NULL RESULTS
Number of outcomes	11	12	15	20
Mean (SD) original experiment effect size	0.70 (0.64)	0.72 (0.61)	0.63 (0.59)	0.51 (0.55)
Median [IQR] original experiment effect size	0.61 [0.15–1.03]	0.68 [0.15–1.03]	0.61 [0.16–0.97]	0.18 [0.15–0.79]
Mean (SD) replication experiment effect size	–0.08 (0.75)	–0.02 (0.74)	0.02 (0.69)	0.01 (0.86)
Median [IQR] replication experiment effect size	0.13 [-0.27–0.24]	0.13 [-0.23–0.39]	0.16 [-0.24–0.47]	0.16 [-0.21–0.39]
Meta-analytic mean (SD) estimate	0.20 (0.31)	0.25 (0.34)	0.24 (0.34)	0.20 (0.39)
Meta-analytic median [IQR] estimate	0.17 [0.06–0.40]	0.23 [0.07–0.43]	0.16 [0.06–0.44]	0.16 [0.07–0.43]
Sample sizes
Median [IQR] of original	16.0 [8.0–25.0]	12.0 [7.0–22.5]	18.0 [8.0–32.0]	19.0 [11.0–514]
Median [IQR] of replication	24.0 [16.0–69.0]	22.5 [8.0–61.5]	30.0 [12.0–72.5]	27.0 [17.5–573]

1. Comparing original effect sizes and effect sizes in the replications for original positive results (top) and null results (bottom) when treating internal replications individually (all outcomes; column 5) and aggregated by effects (column 4), experiments (column 3), and papers (column 2). The mean and median of the effect sizes in the original results were considerably larger than those for the replications. SD = standard deviation; IQR = interquartile range.

The pattern was similar when we compared mean and median values for null effects. Comparing means, the value for the original effects was 0.63 (SD = 0.59), and the value for replications was 0.02 (SD = 0.69). Comparing medians, the value for the original effects was 0.61 (IQR = 0.16–0.97), and the value for replications was 0.16 (IQR = –0.24–0.47).

Although the original and replication effect sizes had very different effect magnitudes, larger effect sizes in the original results tended to be associated with larger effect sizes in the replication (Spearman’s r = 0.47, p = 1.83 × 10^–7; Figure 2). This indicates that observed effect sizes are not all random, and that some findings retain their rank ordering in effect size, despite the clear differences between the original and replication effect sizes. To illustrate the comparability of these findings across different levels of aggregation, Figure 3 presents density plots of original effect sizes compared to replication effect sizes by individual outcomes, effects, experiments, and papers.

Figure 2 with 3 supplements see all

Download asset Open asset

Figure 3 with 1 supplement see all

Download asset Open asset

Comparing animal vs. non-animal experiments

Animal experiments have special significance in understanding biological mechanisms and in translating basic science into potential clinical application. We explored whether the patterns of replicability differed between the animal and non-animal experiments included in this meta-analysis (Table 4). Descriptively, animal experiments with positive effects were less likely to replicate than non-animal experiments with positive effects on every replication criterion. For example, 12% of replication effects were in the same direction as the original and statistically significant for animal experiments, compared with 54% for non-animal experiments. Likewise, 44% of replication effects were in the 95% prediction interval for animal experiments, compared with 63% for non-animal experiments.

Table 4

	Animal	Non-animal	Total
Total number of effects	36	122	158
ORIGINAL POSITIVE EFFECTS
Numerical results
Same direction	17 of 27 (63%)	63 of 74 (85%)	80 of 101 (79%)
Direction and statistical significance	3 of 25 (12%)	39 of 72 (54%)	42 of 97 (43%)
Original ES in replication CI	4 of 25 (16%)	13 of 72 (18%)	17 of 97 (18%)
Replication ES in original CI	9 of 25 (36%)	33 of 72 (46%)	42 of 97 (43%)
Replication ES in PI (porig)	11 of 25 (44%)	45 of 72 (63%)	56 of 97 (58%)
Replication ES≥ original ES	0 of 25 (0%)	3 of 72 (4%)	3 of 97 (3%)
Meta-analysis (p < 0.05)	13 of 25 (52%)	47 of 72 (65%)	60 of 97 (62%)
Representative images
Same direction	1 of 4 (25%)	27 of 31 (87%)	28 of 35 (80%)
Direction and statistical significance	0 of 2 (0%)	14 of 20 (70%)	14 of 22 (64%)
Original image in replication CI	0 of 1 (0%)	10 of 20 (50%)	10 of 21 (48%)
Replication effect ≥ original image	0 of 1 (0%)	7 of 20 (35%)	7 of 21 (33%)
Sample sizes
Median [IQR] of original	14.0 [10.0–20.0]	7.0 [6.0–11.2]	8.0 [6.0–13.0]
Median [IQR] of replication	15.0 [13.0–21.8]	10.0 [8.0–22.0]	12.0 [8.0–22.2]
ORIGINAL NULL EFFECTS
Numerical results
Same direction	N/A	N/A	N/A
Direction and statistical significance	4 of 5 (80%)	7 of 10 (70%)	11 of 15 (73%)
Original ES in replication CI	4 of 5 (80%)	7 of 10 (70%)	11 of 15 (73%)
Replication ES in original CI	5 of 5 (100%)	7 of 10 (70%)	12 of 15 (80%)
Replication ES in PI (porig)	5 of 5 (100%)	7 of 10 (70%)	12 of 15 (80%)
Replication ES≤ original ES	N/A	N/A	N/A
Meta-analysis (p > 0.05)	3 of 5 (60%)	7 of 10 (70%)	10 of 15 (67%)
Representative images
Same direction	N/A	N/A	N/A
Direction and statistical significance	N/A	7 of 5 (80%)	4 of 5 (80%)
Original image in replication CI	N/A	3 of 5 (60%)	3 of 5 (60%)
Replication effect ≤ original image	N/A	N/A	N/A
Sample sizes
Median [IQR] of original	21.0 [20.0–30.0]	8.0 [5.0–266]	15.0 [7.5–31.0]
Median [IQR] of replication	35.0 [30.0–61.0]	16.0 [7.0–604]	27.0 [8.0–66.8]

1. Comparing replication rates for animal experiments (column 2) and non-animal experiments (column 3) according to the seven criteria used in Table 1. For statistical significance, if original effects were interpreted as a positive effect but were not significant at p < 0.05, then they were treated as a positive effect (7 cases), and likewise if they were interpreted as a null effect but were significant at p < 0.05 they were treated as a null effect (3 cases). Standardized mean difference (SMD) effect sizes are reported. CI = 95% confidence interval; PI = 95% prediction interval; ES = effect size; IQR = interquartile range.

We used multi-level models to explore the association of five possible moderators with the replication rate: (1) animal experiments vs. non-animal (i.e., in vitro) experiments; (2) the use of contract research organizations to conduct replications; (3) the use of academic research core facilities to conduct replications; (4) whether the original authors shared materials with the replicating labs; (5) the quality of methodological clarifications made by the original authors upon request from the replicating labs (Errington et al., 2021b). None of the five moderators showed a consistent, significant association with replication success (see Table S7 in Supplementary file 1), though the moderators were variably correlated with one another (ranging from r = –0.68 to r = 0.53; Figure 4). We cannot say whether any of these moderators influence replication success in general, but this analysis suggests that they do not account for much variation in replication success in this sample.

Other factors have been identified that could improve replicability, such as blinding, randomization, and sample size planning (Landis et al., 2012; Macleod and Mohan, 2019). However, these aspects were very rarely reported in the original experiments so they could not be examined as candidate moderators. For example, for the 36 animal effects across 15 experiments, none of the original experiments reported blinding, one experiment (for two effects) reported randomization, and none reported determining sample size a priori. By comparison, the replications reported blinding for five experiments (11 effects), randomization for 13 experiments (28 effects), and all 15 experiments (36 effects) reported calculating sample size a priori.

Figure 4

Download asset Open asset

The multi-level analysis does not support the conclusion that there is a meaningful difference in the replication rates of animal and non-animal experiments. As can be seen in Table 5, median effect sizes were 84% smaller than the original findings for animal replications, and 78% smaller for non-animal replications. The reason that animal experiments had such a low replication rate, particularly according to the statistical significance criterion (12%), is that the effect sizes in the original experiments (Mdn = 1.61) were notably smaller than the effect sizes in the original non-animal experiments (Mdn = 3.65; Figure 5). In sum, original findings with smaller effect sizes were less likely to replicate than original findings with larger effect sizes, and animal experiments tended to have smaller effect sizes than non-animal experiments. In other words, when seeking to predict if a replication will be successful it is more useful to know the original effect size than to know whether the original experiment was an animal experiment or not.

Figure 5

Download asset Open asset

Table 5

	Animal	Non-animal	Total
ORIGINAL POSITIVE EFFECTS
Number of outcomes	25	72	97
Mean (SD) original experiment effect size	1.88 (1.61)	7.63 (14.07)	6.15 (12.39)
Median [IQR] original experiment effect size	1.61 [0.81–2.30]	3.65 [2.45–6.43]	2.96 [1.71–5.70]
Mean (SD) replication experiment effect size	0.19 (0.50)	1.78 (3.39)	1.37 (3.01)
Median [IQR] replication experiment effect size	0.25 [−0.06–0.41]	0.79 [0.20–2.27]	0.43 [0.15–2.06]
Meta-analytic mean (SD) estimate	0.65 (0.54)	2.02 (2.77)	1.66 (2.47)
Meta-analytic median [IQR] estimate	0.83 [0.11–1.05]	1.06 [0.46–2.79]	0.92 [0.36–2.43]
ORIGINAL NULL EFFECTS
Number of outcomes	5	10	15
Mean (SD) original experiment effect size	0.34 (0.29)	0.78 (0.65)	0.63 (0.59)
Median [IQR] original experiment effect size	0.19 [0.10–0.61]	0.84 [0.17–1.08]	0.61 [0.16–0.97]
Mean (SD) replication experiment effect size	0.21 (0.48)	–0.08 (0.78)	0.02 (0.69)
Median [IQR] replication experiment effect size	0.13 [−0.18–0.65]	0.16 [−0.27–0.28]	0.16 [−0.24–0.47]
Meta-analytic mean (SD) estimate	0.21 (0.31)	0.25 (0.37)	0.24 (0.34)
Meta-analytic median [IQR] estimate	0.12 [0.04–0.37]	0.17 [0.10–0.45]	0.16 [0.06–0.44]

1. Comparing original and replication effect sizes (means and medians) for animal experiments (column 2) and non-animal experiments (column 3), along with meta-analytic means and medians for the effect size obtained by combining data from the original effects and the replications. SD = standard deviation; IQR = interquartile range.

Summarizing replications across five criteria

The criteria described above returned a range of replication rates due to the different assumptions made by each, particularly how they handle the estimation of uncertainty. To provide an overall picture, we combined the replication rates by five of these criteria, selecting criteria that could be meaningfully applied to both positive and null effects, which meant excluding the ‘same direction’ and ‘comparing effect size’ criteria, as neither works for null effects.

For replications of original positive effects, 13 of 97 (13%) replications succeeded on all five criteria, 15 succeeded on four, 11 succeeded on three, 22 failed on three, 15 failed on four, and 21 (22%) failed on all five (Table 6 and Figure 6). For original null effects, 7 of 15 (47%) replications succeeded on all five criteria, 2 succeeded on four, 3 succeeded on three, 0 failed on three, 2 failed on four, and 1 (7%) failed on all five. If we consider a replication to be successful overall if it succeeded on a majority of criteria (i.e., three or more), original null effects (80%) were twice as likely to replicate as original positive effects (40%). Combining positive and null effects, 51 of 112 (46%) replications succeeded on more criteria than they failed, and 61 (54%) replications failed on more criteria than they succeeded. We also found these five criteria to be positively correlated with one another ranging from 0.15 to 0.78 and a median of 0.345 suggesting that they provide related but distinct information (Figure 7).

Figure 6

Download asset Open asset

Table 6

	Papers		Experiments		Effects		All outcomes
ORIGINAL POSITIVE RESULTS
Successful replication on all five criteria	2	11%	2	6%	13	13%	20	18%
Success on 4; failure on 1	1	5%	5	15%	15	15%	13	12%
Success on 3; failure on 2	3	16%	1	3%	11	11%	13	12%
Success on 2; failure on 3	5	26%	15	45%	22	23%	26	23%
Success on 1, failure on 4	6	32%	6	18%	15	15%	19	17%
Success on 0, failure on 5	2	11%	4	12%	21	22%	21	19%
Total	19		33		97		112
ORIGINAL NULL RESULTS
Successful replication on all five criteria	5	45%	7	58%	7	47%	6	30%
Success on 4; failure on 1	2	18%	1	8%	2	13%	2	10%
Success on 3; failure on 2	2	18%	2	17%	3	20%	5	25%
Success on 2; failure on 3	2	18%	2	17%	2	13%	2	10%
Success on 1; failure on 4	0	0%	0	0%	0	0%	3	15%
Success on 0; failure on 5	0	0%	0	0%	1	7%	2	10%
Total	11		12		15		20

1. Five of the criteria we used to assess replications could be used for both positive results and null results. The number of papers, experiments, effects, and outcomes where replications were successful on various numbers of these criteria are shown for positive results (top) and null results (bottom). The five criteria were: (i) direction and statistical significance (p < 0.05); (ii) original effect size in replication 95% confidence interval; (iii) replication effect size in original 95% confidence interval; (iv) replication effect size in original 95% prediction interval; (v) meta-analysis combining original and replication effect sizes is statistically significant (p < 0.05). The data in this table are based on standardized mean difference (SMD) effect sizes. Very similar results are obtained when alternative strategies are used to aggregate the data (see Tables S8–S10 in Supplementary file 1).

Figure 7

Download asset Open asset

The ‘same direction’ and ‘comparing effect size’ criteria were not included as neither works for null effects. Also, these two criteria cannot be directly compared with the other five criteria because they both have a minimum replication success rate of 50% under ordinary assumptions, compared with 0% for the other criteria. For example, if all the original effects were due to noise, then the ‘same direction’ criterion would return a value of 50% for the replication rate indicating the worst possible performance. The observation that 79% of replications were in the same direction as the original effect indicates some signal being detected. Conversely, if all the original and replication effect sizes were equivalent and estimated without bias, then the ‘comparing effect size’ criterion would return a value of 50% indicating the best possible performance. The observation that just 3% of replications had larger effect sizes than original positive effects indicates that the original effect sizes were overestimated.

We used seven criteria to assess the replicability of 158 effects in a selection of 23 papers reporting the results of preclinical research in cancer biology. Across multiple criteria, the replications provided weaker evidence for the findings than the original papers. For original positive effects that were reported as numerical values, the median effect size for the replications was 0.43, which was 85% smaller than the median of the original effect sizes (2.96). And although 79% of the replication effects were in the same direction as the original finding (random would be 50%), 92% of replication effect sizes were smaller than the original (combining numeric and representative images). Across five dichotomous criteria for assessing replicability, original null results were twice as likely as original positive results to mostly replicate successfully (80% vs. 40%). Combining original positive and null effects for each of the five criteria, the replication success rates were 47% for same direction and statistical significance, 25% for the original effect size being inside the 95% confidence interval (CI) of the replication, 48% for the replication effect size being inside the 95% CI of the original, 61% for the replication effect size being inside the 95% prediction interval, and 63% for a criterion based on a meta-analytic combination of the data from the original experiment and the replication. Replication rates were relatively consistent whether examining the effects in isolation, combining across internal replications, combining across all the effects of each experiment, or combining across all the experiments of each paper. Animal and non-animal replications both had similarly weaker effect sizes compared to original findings, but animal experiments were much less likely to replicate (probably because the original effect size tended to be smaller in animal experiments). In this section we discuss some of the implications of our results.

All experimental details (e.g., additional protocol details, data, analysis files) of the individual replications and data, code, and materials for the overall project are openly available at https://osf.io/collections/rpcb/. Master data files, containing the aggregate coded variables, are available for exploratory analysis at https://osf.io/e5nvr/.

1. 1. Errington TM
   2. Denis A

   (2021) Open Science Framework

   Replication Data from the Reproducibility Project: Cancer Biology.

   https://doi.org/10.17605/osf.io/e5nvr

1. 1. Amaral OB
   2. Neves K
   3. Wasilewska-Sampaio AP
   4. Carneiro CF

   (2019) The Brazilian Reproducibility Initiative

   eLife 8:e41602.

   https://doi.org/10.7554/eLife.41602

   • Google Scholar
2. 1. Anderson CJ
   2. Bahník Š
   3. Barnett-Cowan M
   4. Bosco FA
   5. Chandler J
   6. Chartier CR
   7. Cheung F
   8. Christopherson CD
   9. Cordes A
   10. Cremata EJ
   11. Della Penna N
   12. Estel V
   13. Fedor A
   14. Fitneva SA
   15. Frank MC
   16. Grange JA
   17. Hartshorne JK
   18. Hasselman F
   19. Henninger F
   20. van der Hulst M
   21. Jonas KJ
   22. Lai CK
   23. Levitan CA
   24. Miller JK
   25. Moore KS
   26. Meixner JM
   27. Munafò MR
   28. Neijenhuijs KI
   29. Nilsonne G
   30. Nosek BA
   31. Plessow F
   32. Prenoveau JM
   33. Ricker AA
   34. Schmidt K
   35. Spies JR
   36. Stieger S
   37. Strohminger N
   38. Sullivan GB
   39. van Aert RCM
   40. van Assen M
   41. Vanpaemel W
   42. Vianello M
   43. Voracek M
   44. Zuni K

   (2016) Response to Comment on “Estimating the reproducibility of psychological science.”

   Science 351:1037.

   https://doi.org/10.1126/science.aad9163

   • Google Scholar
3. 1. Andrews I
   2. Kasy M

   (2019) Identification of and Correction for Publication Bias

   American Economic Review 109:2766–2794.

   https://doi.org/10.1257/aer.20180310

   • Google Scholar
4. 1. Baker M

   (2016)

   Is there a reproducibility crisis

   Nature 533:353–366.

   • Google Scholar
5. 1. Baker M

   (2019) Animal registries aim to reduce bias

   Nature 573:297–298.

   https://doi.org/10.1038/d41586-019-02676-4

   • PubMed
   • Google Scholar
6. 1. Begley CG
   2. Ellis LM

   (2012) Drug development: Raise standards for preclinical cancer research

   Nature 483:531–533.

   https://doi.org/10.1038/483531a

   • PubMed
   • Google Scholar
7. 1. Bell RM
   2. McCaffrey DF

   (2002)

   Bias reduction in standard errors for linear regression with multi-stage samples

   Survey Methodology 28:169–182.

   • Google Scholar
8. 1. Benjamin DJ
   2. Berger JO
   3. Johannesson M
   4. Nosek BA
   5. Wagenmakers E-J
   6. Berk R
   7. Bollen KA
   8. Brembs B
   9. Brown L
   10. Camerer C
   11. Cesarini D
   12. Chambers CD
   13. Clyde M
   14. Cook TD
   15. De Boeck P
   16. Dienes Z
   17. Dreber A
   18. Easwaran K
   19. Efferson C
   20. Fehr E
   21. Fidler F
   22. Field AP
   23. Forster M
   24. George EI
   25. Gonzalez R
   26. Goodman S
   27. Green E
   28. Green DP
   29. Greenwald AG
   30. Hadfield JD
   31. Hedges LV
   32. Held L
   33. Hua Ho T
   34. Hoijtink H
   35. Hruschka DJ
   36. Imai K
   37. Imbens G
   38. Ioannidis JPA
   39. Jeon M
   40. Jones JH
   41. Kirchler M
   42. Laibson D
   43. List J
   44. Little R
   45. Lupia A
   46. Machery E
   47. Maxwell SE
   48. McCarthy M
   49. Moore DA
   50. Morgan SL
   51. Munafó M
   52. Nakagawa S
   53. Nyhan B
   54. Parker TH
   55. Pericchi L
   56. Perugini M
   57. Rouder J
   58. Rousseau J
   59. Savalei V
   60. Schönbrodt FD
   61. Sellke T
   62. Sinclair B
   63. Tingley D
   64. Van Zandt T
   65. Vazire S
   66. Watts DJ
   67. Winship C
   68. Wolpert RL
   69. Xie Y
   70. Young C
   71. Zinman J
   72. Johnson VE

   (2018) Redefine statistical significance

   Nature Human Behaviour 2:6–10.

   https://doi.org/10.1038/s41562-017-0189-z

   • PubMed
   • Google Scholar
9. 1. Button KS
   2. Ioannidis JPA
   3. Mokrysz C
   4. Nosek BA
   5. Flint J
   6. Robinson ESJ
   7. Munafò MR

   (2013) Power failure: why small sample size undermines the reliability of neuroscience

   Nature Reviews Neuroscience 14:365–376.

   https://doi.org/10.1038/nrn3475

   • PubMed
   • Google Scholar
10. 1. Camerer C. F
    2. Dreber A
    3. Forsell E
    4. Ho TH
    5. Huber J
    6. Johannesson M
    7. Kirchler M
    8. Almenberg J
    9. Altmejd A
    10. Chan T
    11. Heikensten E
    12. Holzmeister F
    13. Imai T
    14. Isaksson S
    15. Nave G
    16. Pfeiffer T
    17. Razen M
    18. Wu H

    (2016) Evaluating replicability of laboratory experiments in economics

    Science 351:1433–1436.

    https://doi.org/10.1126/science.aaf0918

    • PubMed
    • Google Scholar
11. 1. Camerer CF
    2. Dreber A
    3. Holzmeister F
    4. Ho T-H
    5. Huber J
    6. Johannesson M
    7. Kirchler M
    8. Nave G
    9. Nosek BA
    10. Pfeiffer T
    11. Altmejd A
    12. Buttrick N
    13. Chan T
    14. Chen Y
    15. Forsell E
    16. Gampa A
    17. Heikensten E
    18. Hummer L
    19. Imai T
    20. Isaksson S
    21. Manfredi D
    22. Rose J
    23. Wagenmakers E-J
    24. Wu H

    (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

    Nature Human Behaviour 2:637–644.

    https://doi.org/10.1038/s41562-018-0399-z

    • PubMed
    • Google Scholar
12. 1. Casadevall A
    2. Fang FC

    (2012) Reforming science: methodological and cultural reforms

    Infection and Immunity 80:891–896.

    https://doi.org/10.1128/IAI.06183-11

    • PubMed
    • Google Scholar
13. 1. Casadevall A
    2. Ellis LM
    3. Davies EW
    4. McFall-Ngai M
    5. Fang FC

    (2016) A Framework for Improving the Quality of Research in the Biological Sciences

    MBio 7:e01256-16.

    https://doi.org/10.1128/mBio.01256-16

    • PubMed
    • Google Scholar
14. 1. Chakroborty S
    2. Sharma A
    3. Mancevska K
    4. Martin Z
    5. Yuan J
    6. Sheffield C
    7. Petanceska SS
    8. Refolo L

    (2020) AlzPED: Improving the predictive power and translational validity of preclinical testing of candidate therapeutics in Alzheimer’s disease animal models: Development of new models and analysis methods/validation of pre‐clinical methods

    Alzheimer’s & Dementia 16:S3.

    https://doi.org/10.1002/alz.036763

    • Google Scholar
15. 1. Chalmers I
    2. Bracken MB
    3. Djulbegovic B
    4. Garattini S
    5. Grant J
    6. Gülmezoglu AM
    7. Howells DW
    8. Ioannidis JPA
    9. Oliver S

    (2014) How to increase value and reduce waste when research priorities are set

    The Lancet 383:156–165.

    https://doi.org/10.1016/S0140-6736(13)62229-1

    • Google Scholar
16. 1. Chambers C

    (2019) What’s next for Registered Reports?

    Nature 573:187–189.

    https://doi.org/10.1038/d41586-019-02674-6

    • PubMed
    • Google Scholar
17. 1. Cragin MH
    2. Palmer CL
    3. Carlson JR
    4. Witt M

    (2010) Data sharing, small science and institutional repositories

    Philosophical Transactions of the Royal Society A 368:4023–4038.

    https://doi.org/10.1098/rsta.2010.0165

    • Google Scholar
18. 1. Drude NI
    2. Martinez Gamboa L
    3. Danziger M
    4. Dirnagl U
    5. Toelch U

    (2021) Improving preclinical studies through replications

    eLife 10:e62101.

    https://doi.org/10.7554/eLife.62101

    • PubMed
    • Google Scholar
19. 1. Ebersole CR
    2. Atherton OE
    3. Belanger AL
    4. Skulborstad HM
    5. Allen JM
    6. Banks JB
    7. Baranski E
    8. Bernstein MJ
    9. Bonfiglio DBV
    10. Boucher L
    11. Brown ER
    12. Budiman NI
    13. Cairo AH
    14. Capaldi CA
    15. Chartier CR
    16. Chung JM
    17. Cicero DC
    18. Coleman JA
    19. Conway JG
    20. Davis WE
    21. Devos T
    22. Fletcher MM
    23. German K
    24. Grahe JE
    25. Hermann AD
    26. Hicks JA
    27. Honeycutt N
    28. Humphrey B
    29. Janus M
    30. Johnson DJ
    31. Joy-Gaba JA
    32. Juzeler H
    33. Keres A
    34. Kinney D
    35. Kirshenbaum J
    36. Klein RA
    37. Lucas RE
    38. Lustgraaf CJN
    39. Martin D
    40. Menon M
    41. Metzger M
    42. Moloney JM
    43. Morse PJ
    44. Prislin R
    45. Razza T
    46. Re DE
    47. Rule NO
    48. Sacco DF
    49. Sauerberger K
    50. Shrider E
    51. Shultz M
    52. Siemsen C
    53. Sobocko K
    54. Weylin Sternglanz R
    55. Summerville A
    56. Tskhay KO
    57. van Allen Z
    58. Vaughn LA
    59. Walker RJ
    60. Weinberg A
    61. Wilson JP
    62. Wirth JH
    63. Wortman J
    64. Nosek BA

    (2016) Many Labs 3: Evaluating participant pool quality across the academic semester via replication

    Journal of Experimental Social Psychology 67:68–82.

    https://doi.org/10.1016/j.jesp.2015.10.012

    • Google Scholar
20. 1. Ebersole CR
    2. Mathur MB
    3. Baranski E
    4. Bart-Plange D-J
    5. Buttrick NR
    6. Chartier CR
    7. Corker KS
    8. Corley M
    9. Hartshorne JK
    10. IJzerman H
    11. Lazarević LB
    12. Rabagliati H
    13. Ropovik I
    14. Aczel B
    15. Aeschbach LF
    16. Andrighetto L
    17. Arnal JD
    18. Arrow H
    19. Babincak P
    20. Bakos BE
    21. Baník G
    22. Baskin E
    23. Belopavlović R
    24. Bernstein MH
    25. Białek M
    26. Bloxsom NG
    27. Bodroža B
    28. Bonfiglio DBV
    29. Boucher L
    30. Brühlmann F
    31. Brumbaugh CC
    32. Casini E
    33. Chen Y
    34. Chiorri C
    35. Chopik WJ
    36. Christ O
    37. Ciunci AM
    38. Claypool HM
    39. Coary S
    40. Čolić MV
    41. Collins WM
    42. Curran PG
    43. Day CR
    44. Dering B
    45. Dreber A
    46. Edlund JE
    47. Falcão F
    48. Fedor A
    49. Feinberg L
    50. Ferguson IR
    51. Ford M
    52. Frank MC
    53. Fryberger E
    54. Garinther A
    55. Gawryluk K
    56. Ashbaugh K
    57. Giacomantonio M
    58. Giessner SR
    59. Grahe JE
    60. Guadagno RE
    61. Hałasa E
    62. Hancock PJB
    63. Hilliard RA
    64. Hüffmeier J
    65. Hughes S
    66. Idzikowska K
    67. Inzlicht M
    68. Jern A
    69. Jiménez-Leal W
    70. Johannesson M
    71. Joy-Gaba JA
    72. Kauff M
    73. Kellier DJ
    74. Kessinger G
    75. Kidwell MC
    76. Kimbrough AM
    77. King JPJ
    78. Kolb VS
    79. Kołodziej S
    80. Kovacs M
    81. Krasuska K
    82. Kraus S
    83. Krueger LE
    84. Kuchno K
    85. Lage CA
    86. Langford EV
    87. Levitan CA
    88. de Lima TJS
    89. Lin H
    90. Lins S
    91. Loy JE
    92. Manfredi D
    93. Markiewicz Ł
    94. Menon M
    95. Mercier B
    96. Metzger M
    97. Meyet V
    98. Millen AE
    99. Miller JK
    100. Montealegre A
    101. Moore DA
    102. Muda R
    103. Nave G
    104. Nichols AL
    105. Novak SA
    106. Nunnally C
    107. Orlić A
    108. Palinkas A
    109. Panno A
    110. Parks KP
    111. Pedović I
    112. Pękala E
    113. Penner MR
    114. Pessers S
    115. Petrović B
    116. Pfeiffer T
    117. Pieńkosz D
    118. Preti E
    119. Purić D
    120. Ramos T
    121. Ravid J
    122. Razza TS
    123. Rentzsch K
    124. Richetin J
    125. Rife SC
    126. Rosa AD
    127. Rudy KH
    128. Salamon J
    129. Saunders B
    130. Sawicki P
    131. Schmidt K
    132. Schuepfer K
    133. Schultze T
    134. Schulz-Hardt S
    135. Schütz A
    136. Shabazian AN
    137. Shubella RL
    138. Siegel A
    139. Silva R
    140. Sioma B
    141. Skorb L
    142. de Souza LEC
    143. Steegen S
    144. Stein LAR
    145. Sternglanz RW
    146. Stojilović D
    147. Storage D
    148. Sullivan GB
    149. Szaszi B
    150. Szecsi P
    151. Szöke O
    152. Szuts A
    153. Thomae M
    154. Tidwell ND
    155. Tocco C
    156. Torka A-K
    157. Tuerlinckx F
    158. Vanpaemel W
    159. Vaughn LA
    160. Vianello M
    161. Viganola D
    162. Vlachou M
    163. Walker RJ
    164. Weissgerber SC
    165. Wichman AL
    166. Wiggins BJ
    167. Wolf D
    168. Wood MJ
    169. Zealley D
    170. Žeželj I
    171. Zrubka M
    172. Nosek BA

    (2020) Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

    Advances in Methods and Practices in Psychological Science 3:309–331.

    https://doi.org/10.1177/2515245920958687

    • Google Scholar
21. 1. Errington TM
    2. Iorns E
    3. Gunn W
    4. Tan FE
    5. Lomax J
    6. Nosek BA

    (2014) An open investigation of the reproducibility of cancer biology research

    eLife 3:e04333.

    https://doi.org/10.7554/eLife.04333

    • Google Scholar
22. 1. Errington TM
    2. Denis A
    3. Allison AB
    4. Araiza R
    5. Aza-Blanc P
    6. Bower LR
    7. Campos J
    8. Chu H
    9. Denson S
    10. Donham C
    11. Harr K
    12. Haven B
    13. Iorns E
    14. Kwok J
    15. McDonald E
    16. Pelech S
    17. Perfito N
    18. Pike A
    19. Sampey D
    20. Settles M
    21. Scott DA
    22. Sharma V
    23. Tolentino T
    24. Trinh A
    25. Tsui R
    26. Willis B
    27. Wood J
    28. Young L

    (2021a) Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology

    eLife 10:e73430.

    https://doi.org/10.7554/eLife.73430

    • Google Scholar
23. 1. Errington TM
    2. Denis A
    3. Perfito N
    4. Iorns E
    5. Nosek BA

    (2021b) Challenges for assessing replicability in preclinical cancer biology

    eLife 10:e67995.

    https://doi.org/10.7554/eLife.67995

    • Google Scholar
24. 1. Franco A
    2. Malhotra N
    3. Simonovits G

    (2014) Social science. Publication bias in the social sciences: unlocking the file drawer

    Science 345:1502–1505.

    https://doi.org/10.1126/science.1255484

    • PubMed
    • Google Scholar
25. Website

    1. Gelman A
    2. Loken E

    (2013) The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time

    Department of Statistics, Columbia University. Accessed October 22, 2021.

    http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
26. 1. Gilbert DT
    2. King G
    3. Pettigrew S
    4. Wilson TD

    (2016) Comment on “Estimating the reproducibility of psychological science.”

    Science 351:1037.

    https://doi.org/10.1126/science.aad7243

    • Google Scholar
27. 1. Greenwald AG

    (1975) Consequences of prejudice against the null hypothesis

    Psychological Bulletin 82:1–20.

    https://doi.org/10.1037/h0076157

    • Google Scholar
28. 1. Hasselblad V
    2. Hedges LV

    (1995) Meta-analysis of screening and diagnostic tests

    Psychological Bulletin 117:167–178.

    https://doi.org/10.1037/0033-2909.117.1.167

    • PubMed
    • Google Scholar
29. 1. Hatch A
    2. Curry S

    (2020) Changing how we evaluate research is difficult, but not impossible

    eLife 9:e58654.

    https://doi.org/10.7554/eLife.58654

    • PubMed
    • Google Scholar
30. 1. Heinl C
    2. Chmielewska J
    3. Olevska A
    4. Grune B
    5. Schönfelder G
    6. Bert B

    (2020) Rethinking the incentive system in science: animal study registries: Preregistering experiments using animals could greatly improve transparency and reliability of biomedical studies and improve animal welfare

    EMBO Reports 21:e49709.

    https://doi.org/10.15252/embr.201949709

    • PubMed
    • Google Scholar
31. 1. Hempel CG

    (1968) Maximal Specificity and Lawlikeness in Probabilistic Explanation

    Philosophy of Science 35:116–133.

    https://doi.org/10.1086/288197

    • Google Scholar
32. 1. Horai H
    2. Arita M
    3. Kanaya S
    4. Nihei Y
    5. Ikeda T
    6. Suwa K
    7. Ojima Y
    8. Tanaka K
    9. Tanaka S
    10. Aoshima K
    11. Oda Y
    12. Kakazu Y
    13. Kusano M
    14. Tohge T
    15. Matsuda F
    16. Sawada Y
    17. Hirai MY
    18. Nakanishi H
    19. Ikeda K
    20. Akimoto N
    21. Maoka T
    22. Takahashi H
    23. Ara T
    24. Sakurai N
    25. Suzuki H
    26. Shibata D
    27. Neumann S
    28. Iida T
    29. Tanaka K
    30. Funatsu K
    31. Matsuura F
    32. Soga T
    33. Taguchi R
    34. Saito K
    35. Nishioka T

    (2010) MassBank: a public repository for sharing mass spectral data for life sciences

    Journal of Mass Spectrometry 45:703–714.

    https://doi.org/10.1002/jms.1777

    • PubMed
    • Google Scholar
33. 1. Ioannidis JPA

    (2005) Why Most Published Research Findings Are False

    PLOS Medicine 2:e124.

    https://doi.org/10.1371/journal.pmed.0020124

    • Google Scholar
34. 1. John LK
    2. Loewenstein G
    3. Prelec D

    (2012) Measuring the prevalence of questionable research practices with incentives for truth telling

    Psychological Science 23:524–532.

    https://doi.org/10.1177/0956797611430953

    • PubMed
    • Google Scholar
35. 1. Kaplan RM
    2. Irvin VL

    (2015) Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time

    PLOS ONE 10:e0132382.

    https://doi.org/10.1371/journal.pone.0132382

    • PubMed
    • Google Scholar
36. 1. Kidwell MC
    2. Lazarević LB
    3. Baranski E
    4. Hardwicke TE
    5. Piechowski S
    6. Falkenberg L-S
    7. Kennett C
    8. Slowik A
    9. Sonnleitner C
    10. Hess-Holden C
    11. Errington TM
    12. Fiedler S
    13. Nosek BA
    14. Macleod MR

    (2016) Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency

    PLOS Biology 14:e1002456.

    https://doi.org/10.1371/journal.pbio.1002456

    • Google Scholar
37. 1. Kimmelman J
    2. Mogil JS
    3. Dirnagl U

    (2014) Distinguishing between exploratory and confirmatory preclinical research will improve translation

    PLOS Biology 12:e1001863.

    https://doi.org/10.1371/journal.pbio.1001863

    • PubMed
    • Google Scholar
38. 1. Klein RA
    2. Ratliff KA
    3. Vianello M
    4. Adams RB
    5. Bahník Š
    6. Bernstein MJ
    7. Bocian K
    8. Brandt MJ
    9. Brooks B
    10. Brumbaugh CC
    11. Cemalcilar Z
    12. Chandler J
    13. Cheong W
    14. Davis WE
    15. Devos T
    16. Eisner M
    17. Frankowska N
    18. Furrow D
    19. Galliani EM
    20. Nosek BA

    (2014) Investigating Variation in Replicability: A “Many Labs” Replication Project

    Social Psychology 45:142–152.

    https://doi.org/10.1027/1864-9335/a000178

    • Google Scholar
39. 1. Klein RA
    2. Vianello M
    3. Hasselman F
    4. Adams BG
    5. Adams RB
    6. Alper S
    7. Aveyard M
    8. Axt JR
    9. Babalola MT
    10. Bahník Š
    11. Batra R
    12. Berkics M
    13. Bernstein MJ
    14. Berry DR
    15. Bialobrzeska O
    16. Binan ED
    17. Bocian K
    18. Brandt MJ
    19. Busching R
    20. Rédei AC
    21. Cai H
    22. Cambier F
    23. Cantarero K
    24. Carmichael CL
    25. Ceric F
    26. Chandler J
    27. Chang JH
    28. Chatard A
    29. Chen EE
    30. Cheong W
    31. Cicero DC
    32. Coen S
    33. Coleman JA
    34. Collisson B
    35. Conway MA
    36. Corker KS
    37. Curran PG
    38. Cushman F
    39. Dagona ZK
    40. Dalgar I
    41. Dalla Rosa A
    42. Davis WE
    43. de Bruijn M
    44. De Schutter L
    45. Devos T
    46. de Vries M
    47. Doğulu C
    48. Dozo N
    49. Dukes KN
    50. Dunham Y
    51. Durrheim K
    52. Ebersole CR
    53. Edlund JE
    54. Eller A
    55. English AS
    56. Finck C
    57. Frankowska N
    58. Freyre MÁ
    59. Friedman M
    60. Galliani EM
    61. Gandi JC
    62. Ghoshal T
    63. Giessner SR
    64. Gill T
    65. Gnambs T
    66. Gómez Á
    67. González R
    68. Graham J
    69. Grahe JE
    70. Grahek I
    71. Green EGT
    72. Hai K
    73. Haigh M
    74. Haines EL
    75. Hall MP
    76. Heffernan ME
    77. Hicks JA
    78. Houdek P
    79. Huntsinger JR
    80. Huynh HP
    81. IJzerman H
    82. Inbar Y
    83. Innes-Ker ÅH
    84. Jiménez-Leal W
    85. John MS
    86. Joy-Gaba JA
    87. Kamiloğlu RG
    88. Kappes HB
    89. Karabati S
    90. Karick H
    91. Keller VN
    92. Kende A
    93. Kervyn N
    94. Knežević G
    95. Kovacs C
    96. Krueger LE
    97. Kurapov G
    98. Kurtz J
    99. Lakens D
    100. Lazarević LB
    101. Levitan CA
    102. Lewis NA
    103. Lins S
    104. Lipsey NP
    105. Losee JE
    106. Maassen E
    107. Maitner AT
    108. Malingumu W
    109. Mallett RK
    110. Marotta SA
    111. Međedović J
    112. Mena-Pacheco F
    113. Milfont TL
    114. Morris WL
    115. Murphy SC
    116. Myachykov A
    117. Neave N
    118. Neijenhuijs K
    119. Nelson AJ
    120. Neto F
    121. Lee Nichols A
    122. Ocampo A
    123. O’Donnell SL
    124. Oikawa H
    125. Oikawa M
    126. Ong E
    127. Orosz G
    128. Osowiecka M
    129. Packard G
    130. Pérez-Sánchez R
    131. Petrović B
    132. Pilati R
    133. Pinter B
    134. Podesta L
    135. Pogge G
    136. Pollmann MMH
    137. Rutchick AM
    138. Saavedra P
    139. Saeri AK
    140. Salomon E
    141. Schmidt K
    142. Schönbrodt FD
    143. Sekerdej MB
    144. Sirlopú D
    145. Skorinko JLM
    146. Smith MA
    147. Smith-Castro V
    148. Smolders K
    149. Sobkow A
    150. Sowden W
    151. Spachtholz P
    152. Srivastava M
    153. Steiner TG
    154. Stouten J
    155. Street CNH
    156. Sundfelt OK
    157. Szeto S
    158. Szumowska E
    159. Tang ACW
    160. Tanzer N
    161. Tear MJ
    162. Theriault J
    163. Thomae M
    164. Torres D
    165. Traczyk J
    166. Tybur JM
    167. Ujhelyi A
    168. van Aert RCM
    169. van Assen M
    170. van der Hulst M
    171. van Lange PAM
    172. van ’t Veer AE
    173. Vásquez- Echeverría A
    174. Ann Vaughn L
    175. Vázquez A
    176. Vega LD
    177. Verniers C
    178. Verschoor M
    179. Voermans IPJ
    180. Vranka MA
    181. Welch C
    182. Wichman AL
    183. Williams LA
    184. Wood M
    185. Woodzicka JA
    186. Wronska MK
    187. Young L
    188. Zelenski JM
    189. Zhijia Z
    190. Nosek BA

    (2018) Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

    Advances in Methods and Practices in Psychological Science 1:443–490.

    https://doi.org/10.1177/2515245918810225

    • Google Scholar
40. 1. Landis SC
    2. Amara SG
    3. Asadullah K
    4. Austin CP
    5. Blumenstein R
    6. Bradley EW
    7. Crystal RG
    8. Darnell RB
    9. Ferrante RJ
    10. Fillit H
    11. Finkelstein R
    12. Fisher M
    13. Gendelman HE
    14. Golub RM
    15. Goudreau JL
    16. Gross RA
    17. Gubitz AK
    18. Hesterlee SE
    19. Howells DW
    20. Huguenard J
    21. Kelner K
    22. Koroshetz W
    23. Krainc D
    24. Lazic SE
    25. Levine MS
    26. Macleod MR
    27. McCall JM
    28. Narasimhan K
    29. Noble LJ
    30. Perrin S
    31. Porter JD
    32. Steward O
    33. Unger E
    34. Utz U
    35. Silberberg SD

    (2012) A call for transparent reporting to optimize the predictive value of preclinical research

    Nature 490:187–191.

    https://doi.org/10.1038/nature11556

    • PubMed
    • Google Scholar
41. 1. Langan D
    2. Higgins JPT
    3. Jackson D
    4. Bowden J
    5. Veroniki AA
    6. Kontopantelis E
    7. Viechtbauer W
    8. Simmonds M

    (2019) A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses

    Research Synthesis Methods 10:83–98.

    https://doi.org/10.1002/jrsm.1316

    • PubMed
    • Google Scholar
42. 1. Lemmon VP
    2. Abeyruwan S
    3. Visser U
    4. Bixby JL

    (2014) Facilitating transparency in spinal cord injury studies using data standards and ontologies

    Neural Regeneration Research 9:6–7.

    https://doi.org/10.4103/1673-5374.125322

    • PubMed
    • Google Scholar
43. 1. Lindsay DS
    2. Simons DJ
    3. Lilienfeld SO

    (2016)

    Research preregistration 101

    APS Observer 29:10.

    • Google Scholar
44. 1. Macleod MR
    2. Michie S
    3. Roberts I
    4. Dirnagl U
    5. Chalmers I
    6. Ioannidis JPA
    7. Salman RAS
    8. Chan AW
    9. Glasziou P

    (2014) Biomedical research: increasing value, reducing waste

    The Lancet 383:101–104.

    https://doi.org/10.1016/S0140-6736(13)62329-6

    • Google Scholar
45. 1. Macleod MR
    2. Lawson McLean A
    3. Kyriakopoulou A
    4. Serghiou S
    5. de Wilde A
    6. Sherratt N
    7. Hirst T
    8. Hemblade R
    9. Bahor Z
    10. Nunes-Fonseca C
    11. Potluru A
    12. Thomson A
    13. Baginskitae J
    14. Egan K
    15. Vesterinen H
    16. Currie GL
    17. Churilov L
    18. Howells DW
    19. Sena ES

    (2015) Risk of Bias in Reports of In Vivo Research: A Focus for Improvement

    PLOS Biology 13:e1002273.

    https://doi.org/10.1371/journal.pbio.1002273

    • Google Scholar
46. 1. Macleod M
    2. Mohan S

    (2019) Reproducibility and Rigor in Animal-Based Research

    ILAR Journal 60:17–23.

    https://doi.org/10.1093/ilar/ilz015

    • PubMed
    • Google Scholar
47. 1. Makel MC
    2. Plucker JA
    3. Hegarty B

    (2012) Replications in Psychology Research: How Often Do They Really Occur?

    Perspectives on Psychological Science 7:537–542.

    https://doi.org/10.1177/1745691612460688

    • PubMed
    • Google Scholar
48. 1. Mathur MB
    2. VanderWeele TJ

    (2019) Challenges and suggestions for defining replication “success” when effects may be heterogeneous: Comment on Hedges and Schauer (2019)

    Psychological Methods 24:571–575.

    https://doi.org/10.1037/met0000223

    • PubMed
    • Google Scholar
49. 1. Mathur MB
    2. VanderWeele TJ

    (2020a) A Simple, Interpretable Conversion from Pearson’s Correlation to Cohen’s for d Continuous Exposures

    Epidemiology 31:e16–e18.

    https://doi.org/10.1097/EDE.0000000000001105

    • Google Scholar
50. 1. Mathur MB
    2. VanderWeele TJ

    (2020b) New statistical metrics for multisite replication projects

    Journal of the Royal Statistical Society 183:1145–1166.

    https://doi.org/10.1111/rssa.12572

    • Google Scholar
51. 1. McShane BB
    2. Böckenholt U
    3. Hansen KT

    (2016) Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes

    Perspectives on Psychological Science 11:730–749.

    https://doi.org/10.1177/1745691616662243

    • PubMed
    • Google Scholar
52. Book

    1. Musgrave A

    (1970) Criticism and the Growth of Knowledge

    Cambridge University Press.

    https://doi.org/10.2307/2412896

    • Google Scholar
53. 1. Nosek BA
    2. Spies JR
    3. Motyl M

    (2012) Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability

    Perspectives on Psychological Science 7:615–631.

    https://doi.org/10.1177/1745691612459058

    • PubMed
    • Google Scholar
54. 1. Nosek BA
    2. Lakens D

    (2014) Registered Reports

    Social Psychology 45:137–141.

    https://doi.org/10.1027/1864-9335/a000192

    • Google Scholar
55. 1. Nosek BA
    2. Alter G
    3. Banks GC
    4. Borsboom D
    5. Bowman SD
    6. Breckler SJ
    7. Buck S
    8. Chambers CD
    9. Chin G
    10. Christensen G
    11. Contestabile M
    12. Dafoe A
    13. Eich E
    14. Freese J
    15. Glennerster R
    16. Goroff D
    17. Green DP
    18. Hesse B
    19. Humphreys M
    20. Ishiyama J
    21. Karlan D
    22. Kraut A
    23. Lupia A
    24. Mabry P
    25. Madon T
    26. Malhotra N
    27. Mayo-Wilson E
    28. McNutt M
    29. Miguel E
    30. Paluck EL
    31. Simonsohn U
    32. Soderberg C
    33. Spellman BA
    34. Turitto J
    35. VandenBos G
    36. Vazire S
    37. Wagenmakers EJ
    38. Wilson R
    39. Yarkoni T

    (2015) Promoting an open research culture

    Science 348:1422–1425.

    https://doi.org/10.1126/science.aab2374

    • Google Scholar
56. Preprint

    1. Nosek BA
    2. Gilbert EA

    (2017) Mischaracterizing Replication Studies Leads to Erroneous Conclusions

    PsyArXiv.

    https://doi.org/10.31234/osf.io/nt4d3

    • Google Scholar
57. 1. Nosek BA
    2. Errington TM

    (2020a) What is replication?

    PLOS Biology 18:e3000691.

    https://doi.org/10.1371/journal.pbio.3000691

    • Google Scholar
58. 1. Nosek BA
    2. Errington TM

    (2020b) The best time to argue about what a replication means? Before you do it

    Nature 583:518–520.

    https://doi.org/10.1038/d41586-020-02142-6

    • Google Scholar
59. 1. Nosek BA
    2. Hardwicke TE
    3. Moshontz H
    4. Allard A
    5. Corker KS
    6. Dreber A
    7. Fidler F
    8. Hilgard J
    9. Struhl MK
    10. Nuijten MB
    11. Rohrer JM
    12. Romero F
    13. Scheel AM
    14. Scherer LD
    15. Schönbrodt FD
    16. Vazire S

    (2021) Replicability, Robustness, and Reproducibility in Psychological Science

    Annual Review of Psychology 73:114157.

    https://doi.org/10.1146/annurev-psych-020821-114157

    • Google Scholar
60. 1. Olsson-Collentine A
    2. Wicherts JM
    3. van Assen MALM

    (2020) Heterogeneity in direct replications in psychology and its association with effect size

    Psychological Bulletin 146:922–940.

    https://doi.org/10.1037/bul0000294

    • PubMed
    • Google Scholar
61. 1. Open Science Collaboration

    (2015) Estimating the reproducibility of psychological science

    Science 349:aac4716.

    https://doi.org/10.1126/science.aac4716

    • PubMed
    • Google Scholar
62. 1. Patil P
    2. Peng RD
    3. Leek JT

    (2016) What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science

    Perspectives on Psychological Science 11:539–544.

    https://doi.org/10.1177/1745691616646366

    • PubMed
    • Google Scholar
63. Preprint

    1. Pelech S
    2. Gallagher C
    3. Sutter C
    4. Yue L
    5. Kerwin J
    6. Bhargava A
    7. Iorns E
    8. Tsui R
    9. Denis A
    10. Perfito N
    11. Errington T

    (2021) Replication Study: RAF Inhibitors Prime Wild-Type RAF to Activate the MAPK Pathway and Enhance Growth

    bioRxiv.

    https://doi.org/10.1101/2021.11.30.470372

    • Google Scholar
64. 1. Perrin S

    (2014) Preclinical research: Make mouse studies work

    Nature 507:423–425.

    https://doi.org/10.1038/507423a

    • PubMed
    • Google Scholar
65. 1. Prinz F
    2. Schlange T
    3. Asadullah K

    (2011) Believe it or not: how much can we rely on published data on potential drug targets?

    Nature Reviews Drug Discovery 10:712.

    https://doi.org/10.1038/nrd3439-c1

    • PubMed
    • Google Scholar
66. 1. Pustejovsky JE
    2. Tipton E

    (2017) Small-Sample Methods for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed Effects Models

    Journal of Business & Economic Statistics 36:672–683.

    https://doi.org/10.1080/07350015.2016.1247004

    • Google Scholar
67. Software

    1. R Development Core Team

    (2021) R: A language and environment for statistical computing

    R Foundation for Statistical Computing, Vienna, Austria.

    https://www.R-project.org/
68. 1. Rice K
    2. Higgins JPT
    3. Lumley T

    (2017) A re‐evaluation of fixed effect(s) meta‐analysis

    Journal of the Royal Statistical Society 181:205–227.

    https://doi.org/10.1111/rssa.12275

    • Google Scholar
69. 1. Rosenthal R

    (1979) The file drawer problem and tolerance for null results

    Psychological Bulletin 86:638–641.

    https://doi.org/10.1037/0033-2909.86.3.638

    • Google Scholar
70. Book

    1. Salmon MH
    2. Glymour C

    (1999)

    Troduction to the Philosophy of Science

    Hackett Publishing.

    • Google Scholar
71. Preprint

    1. Scheel AM
    2. Schijen M
    3. Lakens D

    (2020) An Excess of Positive Results: Comparing the Standard Psychology Literature with Registered Reports

    PsyArXiv.

    https://doi.org/10.31234/osf.io/p6e9c

    • Google Scholar
72. 1. Serghiou S
    2. Contopoulos-Ioannidis DG
    3. Boyack KW
    4. Riedel N
    5. Wallach JD
    6. Ioannidis JPA

    (2021) Assessment of transparency indicators across the biomedical literature: How open is open?

    PLOS Biology 19:e3001107.

    https://doi.org/10.1371/journal.pbio.3001107

    • PubMed
    • Google Scholar
73. 1. Simmons JP
    2. Nelson LD
    3. Simonsohn U

    (2011) False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant

    Psychological Science 22:1359–1366.

    https://doi.org/10.1177/0956797611417632

    • PubMed
    • Google Scholar
74. 1. Soderberg CK

    (2018) Using OSF to Share Data: A Step-by-Step Guide

    Advances in Methods and Practices in Psychological Science 1:115–120.

    https://doi.org/10.1177/2515245918757689

    • Google Scholar
75. Preprint

    1. Soderberg CK
    2. Errington T
    3. Schiavone SR
    4. Bottesini JG
    5. Thorn FS
    6. Vazire S
    7. Esterling KM
    8. Nosek BA

    (2020) Research Quality of Registered Reports Compared to the Traditional Publishing Model

    MetaArXiv.

    https://doi.org/10.31222/osf.io/7x9vy6

    • Google Scholar
76. 1. Stanley TD
    2. Doucouliagos H

    (2014) Meta-regression approximations to reduce publication selection bias

    Research Synthesis Methods 5:60–78.

    https://doi.org/10.1002/jrsm.1095

    • PubMed
    • Google Scholar
77. 1. Steward O
    2. Popovich PG
    3. Dietrich WD
    4. Kleitman N

    (2012) Replication and reproducibility in spinal cord injury research

    Experimental Neurology 233:597–605.

    https://doi.org/10.1016/j.expneurol.2011.06.017

    • PubMed
    • Google Scholar
78. 1. Stodden V
    2. Seiler J
    3. Ma Z

    (2018) An empirical analysis of journal policy effectiveness for computational reproducibility

    PNAS 115:2584–2589.

    https://doi.org/10.1073/pnas.1708290115

    • Google Scholar
79. 1. Teal TK
    2. Cranston KA
    3. Lapp H
    4. White E
    5. Wilson G
    6. Ram K
    7. Pawlik A

    (2015) Data Carpentry: Workshops to Increase Data Literacy for Researchers

    International Journal of Digital Curation 10:135–143.

    https://doi.org/10.2218/ijdc.v10i1.351

    • Google Scholar
80. 1. Valentine JC
    2. Biglan A
    3. Boruch RF
    4. Castro FG
    5. Collins LM
    6. Flay BR
    7. Kellam S
    8. Mościcki EK
    9. Schinke SP

    (2011) Replication in prevention science

    Prevention Science 12:103–117.

    https://doi.org/10.1007/s11121-011-0217-6

    • PubMed
    • Google Scholar
81. 1. Van Calster B
    2. Wynants L
    3. Riley RD
    4. van Smeden M
    5. Collins GS

    (2021) Methodology over metrics: current scientific standards are a disservice to patients and society

    Journal of Clinical Epidemiology 138:219–226.

    https://doi.org/10.1016/j.jclinepi.2021.05.018

    • PubMed
    • Google Scholar
82. 1. van der Naald M
    2. Wenker S
    3. Doevendans PA
    4. Wever KE
    5. Chamuleau SAJ

    (2020) Publication rate in preclinical research: a plea for preregistration

    BMJ Open Science 4:e100051.

    https://doi.org/10.1136/bmjos-2019-100051

    • Google Scholar
83. 1. Vines TH
    2. Albert AYK
    3. Andrew RL
    4. Débarre F
    5. Bock DG
    6. Franklin MT
    7. Gilbert KJ
    8. Moore JS
    9. Renaut S
    10. Rennison DJ

    (2014) The availability of research data declines rapidly with article age

    Current Biology 24:94–97.

    https://doi.org/10.1016/j.cub.2013.11.014

    • PubMed
    • Google Scholar
84. 1. Wilson G

    (2014) Software Carpentry: lessons learned

    F1000Research 3:62.

    https://doi.org/10.12688/f1000research.3-62.v2

    • PubMed
    • Google Scholar
85. 1. Wilson DJ

    (2019) The harmonic mean p -value for combining dependent tests

    PNAS 116:1195–1200.

    https://doi.org/10.1073/pnas.1814092116

    • Google Scholar

Author details

1. Timothy M Errington

   Center for Open Science, Charlottesville, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Validation, Visualization, Writing – original draft, Writing – review and editing

   For correspondence

   tim@cos.io

   Competing interests

   Employed by the Center for Open Science, a non-profit organization that has a mission to increase openness, integrity, and reproducibility of research

   "This ORCID iD identifies the author of this article:" 0000-0002-4959-5143

2. Maya Mathur

   Quantitative Sciences Unit, Stanford University, Stanford, United States

   Contribution

   Formal analysis, Methodology, Validation, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-6698-2607

3. Courtney K Soderberg

   Center for Open Science, Charlottesville, United States

   Contribution

   Methodology, Visualization, Writing – review and editing

   Competing interests

   Was employed by the Center for Open Science, a non-profit organization that has a mission to increase openness, integrity, and reproducibility of research

4. Alexandria Denis

   Center for Open Science, Charlottesville, United States

   Present address

   Fordham University School of Law, New York, United States

   Contribution

   Data curation, Investigation

   Competing interests

   Employed by the Center for Open Science, a non-profit organization that has a mission to increase openness, integrity, and reproducibility of research

5. Nicole Perfito

   Center for Open Science, Charlottesville, United States

   Present address

   Rarebase, Palo Alto, United States

   Contribution

   Conceptualization, Project administration, Writing – review and editing

   Competing interests

   Was employed by and holds shares in Science Exchange Inc

6. Elizabeth Iorns

   Science Exchange, Palo Alto, United States

   Contribution

   Conceptualization, Funding acquisition, Project administration, Writing – review and editing

   Competing interests

   Employed by and holds shares in Science Exchange Inc

   "This ORCID iD identifies the author of this article:" 0000-0002-5515-1258

7. Brian A Nosek

   1. Center for Open Science, Charlottesville, United States
   2. University of Virginia, Charlottesville, United States

   Contribution

   Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   Employed by the nonprofit Center for Open Science that has a mission to increase openness, integrity, and reproducibility of research

   "This ORCID iD identifies the author of this article:" 0000-0001-6797-5476

• 17,450

  Page views

• 1,883

  Downloads

• 120

  Citations

Article citation count generated by polling the highest count across the following sources: Scopus, Crossref, PubMed Central.