Developing a new drug can take years, partly because preclinical research on non-human animals is required before any clinical trials with humans can take place. Nevertheless, only a fraction of cancer drugs that are put into clinical trials after showing promising results in preclinical animal studies end up proving safe and effective in human beings.

Many researchers and commentators have suggested that this high failure rate reflects flaws in the way preclinical studies in cancer are designed and reported. Now, Henderson et al. have looked at all the published animal studies of a cancer drug called sunitinib and asked how well the design of these studies attempted to limit bias and match the clinical scenarios they were intended to represent.

This systematic review and meta-analysis revealed that many common practices, like randomization, were rarely implemented. None of the published studies used ‘blinding’, whereby information about which animals are receiving the drug and which animals are receiving the control is kept from the experimenter, until after the test; this technique can help prevent any expectations or personal preferences from biasing the results. Furthermore, most tumors were tested in only one model system, namely, mice that had been injected with specific human cancer cells. This makes it difficult to rule out that any anti-cancer activity was in fact unique to that single model.

Henderson et al. went on to find evidence that suggests that the anti-cancer effects of sunitinib might have been overestimated by as much as 45% because those studies that found no or little anti-cancer effect were simply not published. Though it is known that the anti-cancer activity of the drug increases with the dose given in both human beings and animals, an evaluation of the effects of all the published studies combined did not detect such a dose-dependent response.

The poor design and reporting issues identified provide further grounds for concern about the value of many preclinical experiments in cancer. These findings also suggest that there are many opportunities for improving the design and reliability of study reports. Researchers studying certain medical conditions (such as strokes) have already developed, and now routinely implement, a set of standards for the design and reporting of preclinical research. It now appears that the cancer research community should do the same.

Preclinical experiments provide evidence of clinical promise, inform trial design, and establish the ethical basis for exposing patients to a new substance. However, preclinical research is plagued by poor design and reporting practices (van der Worp et al., 2010; Begley, 2013a; Begley and Ioannidis, 2015). Recent reports also suggest that many effects in preclinical studies fail replication (Begley and Ellis, 2012). Drug development efforts grounded on non-reproducible findings expose patients to harmful and inactive agents; they also absorb scarce scientific and human resources, the costs of which are reflected as higher drug prices.

Several studies have evaluated the predictive value of animal models in cancer drug development (Johnson et al., 2001; Voskoglou-Nomikos et al., 2003; Corpet and Pierre, 2005). However, few have systematically examined experimental design—as opposed to use of specific models—and its impact on effect sizes across different malignancies (Amarasingh et al., 2009; Hirst et al., 2013). A recent systematic review of guidelines for limiting bias in preclinical research design was unable to identify any guidelines in oncology (Henderson et al., 2013). Validity threats in preclinical oncology may be particularly important to address in light of the fact that cancer drug development has one of the highest rates of attrition (Hay et al., 2014), and oncology drug development commands billions of dollars in funding each year (Adams and Brantner, 2006).

In what follows, we conducted a systematic review and meta-analysis of features of design and outcomes for preclinical efficacy studies of the highly successful drug sunitinib. Sunitinib is a multi-targeted tyrosine kinase inhibitor sunitinib (SU11248, Sutent) and is licensed as monotherapy for three different malignancies (Chow and Eckhardt, 2007; Raymond et al., 2011). As it was introduced into clinical development around 2000 and tested against numerous malignancies, sunitinib provided an opportunity to study a large sample of preclinical studies across a broad range of malignancies—including several supporting successful translation trajectories.

Preclinical studies serve an important role in formulating clinical hypotheses and justifying the advance of a new drug into clinical testing. Our meta-analysis, which included malignancies that respond to sunitinib in human beings and those that do not, raises several questions about methods and reporting practices in preclinical oncology—at least in the context of one well-established drug.

First, reporting of design elements and data was poor and inconsistent with widely recognized standards for animal studies (Kilkenny et al., 2010). Indeed, 98 experiments (30% of qualitative sample) could not be quantitatively analyzed because sample sizes or measures of dispersion were not provided. Experimenters only sporadically addressed major internal validity threats and tended not to test indication-activity in more than one model and species. This finding is consistent with what others have observed in experimental stroke and other research areas (Macleod et al., 2004; van der Worp et al., 2005; Kilkenny et al., 2009; Glasziou et al., 2014). Some teams have shown a relationship between failure to address internal validity threats and exaggerated effect size (Crossley et al., 2008; Rooke et al., 2011); we did not observe a clear relationship. Consistent with what has been reported in stroke (O'Collins et al., 2006), our findings suggest that testing in more models tends to produce smaller effect sizes. However, since a larger sample of studies will provide a more precise estimate of effect, we cannot rule out that the trends observed for external validity reflect a regression to the mean.

Second, preclinical studies for sunitinib seem to be prone to publication bias. Notwithstanding limitations on using funnel plots to detect publication bias (Lau et al., 2006), our plots were highly asymmetrical. That all malignancy types tested showed statistically significant anti-cancer activity strains credulity. Others have reported that far more animal studies report statistical significance than would be expected (Wallace et al., 2009; Tsilidis et al., 2013), and our observations that two thirds of individual studies showed significance extends these observations.

Third, we were unable to detect a meaningful relationship between preclinical effect sizes and known clinical behavior. Although a full analysis correlating trial and preclinical effect sizes will be needed, we did not observe obvious relationships between the two. We also did not detect a dose–response effect over three orders of magnitude even within an indication—RCC—known to respond to sunitinib and even when different time points were used. It is possible that heterogeneity in cell lines or strains may have obscured the effects of dose. For example, experimenters may have delivered higher doses to xenografts known to show slow tumor growth. However, RCC patients—each of whom harbors genetically distinct tumors—show dose–response effects in trials (Faivre et al., 2006) and between trials in a meta-analysis (Houk et al., 2010). It is also possible that the toxicity of sunitinib may have limited the ability to demonstrate dose response, though this contradicts demonstration of dose response within studies (Abrams et al., 2003; Amino et al., 2006; Ko et al., 2010). Finally, the tendency for preclinical efficacy studies to report drug dose, but rarely drug exposure (i.e., serum measurement of active drug), further limits the construct validity of these studies (Peterson and Houghton, 2004).

One explanation for our findings is that human xenograft models, which dominated our meta-analytic sample, have little predictive value, at least in the context of receptor tyrosine kinase inhibitors. This is a possibility that contradicts other reports (Kerbel, 2003; Voskoglou-Nomikos et al., 2003). We disfavor this explanation in light of the suggestion of publication bias; also, xenografts should show a dose–response regardless of whether they are useful clinical models. A second explanation is that experimental methods are so varied as to mask real effects. However, we note that the observed patterns on experimental design are based purely on what was reported in ‘Materials and methods’ section. Third, experiments assessing changes in tumor volume might only be interpretable in the context of other experiments within a preclinical report, such as with mechanistic and pharmacokinetic studies. This explanation is consistent with our observation that studies testing effect along a causal pathway tended to produce smaller effect sizes. A fourth possible explanation for our findings is that the predictive value of a small number of preclinical studies was obscured by inclusion of poorly designed and executed preclinical studies in our meta-analysis. Quantitative analysis of preclinical design factors that confer greater clinical generalizability awaits side-by-side comparison with pooled effects in clinical trials. Finally, it may be that design and reporting practices are so poor in preclinical cancer research as to make interpretation of tumor volume curves useless. Or, non-reporting may be so rampant as to render meta-analysis of preclinical research impossible. If so, this raises very troubling questions for the publication economy of cancer biology: even well-designed and reported studies may be difficult to interpret if their results cannot be compared to and synthesized with other studies.

Our systematic review has several limitations. First, we relied on what authors reported in the published study. It is possible certain experimental practices, like randomization, were used but not reported in methods. Further to this, we relied only on published reports, and restriction of searches to the English language may have excluded some articles. In February of 2012, we filed a Freedom of Information Act request from the Food and Drug Administration (FDA) for additional preclinical data submitted in support of sunitinib's licensure; nearly 4 years later, the request has not been honored. Second, effect sizes were calculated using graph digitizer software from tumor volume curves: minor distortion of effect sizes may have occurred but were likely non-differential between groups. Third, subtle experimental design features—not apparent in ‘Materials and methods’ sections—may explain our failure to detect a dose–response effect. For instance, few reports provide detailed animal housing and testing conditions, perhaps leading to important inter-laboratory differences in tumor growth. It should also be emphasized that our study was exploratory in nature; findings like ours will need to be confirmed using prespecified protocols. Fourth, our study represents analysis of a single drug, and it may be our findings do not extend beyond receptor tyrosine kinase inhibitors, or sunitinib. However, many of our findings are consistent with those observed in other systematic reviews of preclinical cancer interventions (Amarasingh et al., 2009; Sugar et al., 2012; Hirst et al., 2013). Fifth, our analysis does not directly address many design elements—like duration of experiment or choice of tissue xenograft—that are likely to bear on study validity. Finally, we acknowledge that there may be funding constraints that limit implementation of validity practices described above. We note, nevertheless, that other realms, in particular, neurology, have found ways to make such methods a mainstay.

Numerous commentators have raised concerns about the design and reporting of preclinical cancer research (Sugar et al., 2012; Begley, 2013b). In one report, only 11% preclinical cancer studies submitted to a major biotechnology company withstood in-house replication (Begley and Ellis, 2012). The Center for Open Science and Science Exchange has initiated a project that will attempt to reproduce 50 of the highest impact papers in cancer biology published between 2010 and 2012 (Morrison, 2014). In a recent commentary, Smith et al. fault many researchers for performing in vitro preclinical tests using drug levels that are clinically unachievable due to toxicity (Smith and Houghton, 2013). Unaddressed preclinical validity threats like this—and the ones documented in our study—encourage futile clinical development trajectories. Many research areas, like stroke, epilepsy, and cardiology, have devised design guidelines aimed at improving the clinical generalizability of preclinical studies (Fisher et al., 2009; Galanopoulou et al., 2012; Curtis et al., 2013; Pusztai et al., 2013); and the ARRIVE guidelines (Kilkenny et al., 2010) for reporting animal experiments have been taken up by numerous journals and funding bodies. Our findings provide further impetus for developing and implementing guidelines for the design, reporting, and synthesis of preclinical studies in cancer.

1. 1. Anonymous

   (2014) Budgets up at NIH, NCI, and FDA

   Cancer Discovery 4:263.

   https://doi.org/10.1158/2159-8290.CD-NB2014-016

   • Google Scholar
2. 1. Abrams TJ
   2. Murray LJ
   3. Pesenti E
   4. Holway VW
   5. Colombo T
   6. Lee LB
   7. Cherrington JM
   8. Pryer NK

   (2003)

   Preclinical evaluation of the tyrosine kinase inhibitor SU11248 as a single agent and in combination with ‘standard of care’ therapeutic agents for the treatment of breast cancer

   Molecular Cancer Therapeutics 2:1011–1021.

   • Google Scholar
3. 1. Adams CP
   2. Brantner VV

   (2006) Estimating the cost of new drug development: is it really 802 million dollars?

   Health Affairs 25:420–428.

   https://doi.org/10.1377/hlthaff.25.2.420

   • Google Scholar
4. 1. Amarasingh S
   2. Macleod MR
   3. Whittle IR

   (2009) What is the translational efficacy of chemotherapeutic drug research in neuro-oncology? A systematic review and meta-analysis of the efficacy of BCNU and CCNU in animal models of glioma

   Journal of Neuro-Oncology 91:117–125.

   https://doi.org/10.1007/s11060-008-9697-z

   • Google Scholar
5. 1. Amino N
   2. Ideyama Y
   3. Yamano M
   4. Kuromitsu S
   5. Tajinda K
   6. Samizu K
   7. Hisamichi H
   8. Matsuhisa A
   9. Shirasuna K
   10. Kudoh M
   11. Shibasaki M

   (2006) YM-359445, an orally bioavailable vascular endothelial growth factor receptor-2 tyrosine kinase inhibitor, has highly potent antitumor activity against established tumors

   Clinical Cancer Research 12:1630–1638.

   https://doi.org/10.1158/1078-0432.CCR-05-2028

   • Google Scholar
6. 1. Begley CG

   (2013a) Six red flags for suspect work

   Nature 497:433–434.

   https://doi.org/10.1038/497433a

   • Google Scholar
7. 1. Begley CG

   (2013b) An unappreciated challenge to oncology drug discovery: pitfalls in preclinical research

   American Society of Clinical Oncology Educational Book 2013:466–468.

   https://doi.org/10.1200/EdBook_AM.2013.33.466

   • Google Scholar
8. 1. Begley CG
   2. Ellis LM

   (2012) Drug development: raise standards for preclinical cancer research

   Nature 483:531–533.

   https://doi.org/10.1038/483531a

   • Google Scholar
9. 1. Begley CG
   2. Ioannidis JP

   (2015) Reproducibility in science: improving the standard for basic and preclinical research

   Circulation Research 116:116–126.

   https://doi.org/10.1161/CIRCRESAHA.114.303819

   • Google Scholar
10. 1. Chow LQ
    2. Eckhardt SG

    (2007) Sunitinib: from rational design to clinical efficacy

    Journal of Clinical Oncology 25:884–896.

    https://doi.org/10.1200/JCO.2006.06.3602

    • Google Scholar
11. 1. Corpet DE
    2. Pierre F

    (2005) How good are rodent models of carcinogenesis in predicting efficacy in humans? A systematic review and meta-analysis of colon chemoprevention in rats, mice and men

    European Journal of Cancer 41:1911–1922.

    https://doi.org/10.1016/j.ejca.2005.06.006

    • Google Scholar
12. 1. Crossley NA
    2. Sena E
    3. Goehler J
    4. Horn J
    5. van der Worp B
    6. Bath PM
    7. Macleod M
    8. Dirnagl U

    (2008) Empirical evidence of bias in the design of experimental stroke studies: a metaepidemiologic approach

    Stroke 39:929–934.

    https://doi.org/10.1161/STROKEAHA.107.498725

    • Google Scholar
13. 1. Curtis MJ
    2. Hancox JC
    3. Farkas A
    4. Wainwright CL
    5. Stables CL
    6. Saint DA
    7. Clements-Jewery H
    8. Lambiase PD
    9. Billman GE
    10. Janse MJ
    11. Pugsley MK
    12. Ng GA
    13. Roden DM
    14. Camm AJ
    15. Walker MJ

    (2013) The Lambeth Conventions (II): guidelines for the study of animal and human ventricular and supraventricular arrhythmias

    Pharmacology & Therapeutics 139:213–248.

    https://doi.org/10.1016/j.pharmthera.2013.04.008

    • Google Scholar
14. 1. de Vries RB
    2. Hooijmans CR
    3. Tillema A
    4. Leenaars M
    5. Ritskes-Hoitinga M

    (2011) A search filter for increasing the retrieval of animal studies in Embase

    Laboratory Animals 45:268–270.

    https://doi.org/10.1258/la.2011.011056

    • Google Scholar
15. 1. DerSimonian R
    2. Laird N

    (1986) Meta-analysis in clinical trials

    Contemporary Clinical Trials 7:177–188.

    https://doi.org/10.1016/0197-2456(86)90046-2

    • Google Scholar
16. 1. Dietz G
    2. Dahabreh IJ
    3. Gurevitch J
    4. Lajeunesse MJ
    5. Schmid CH
    6. Trikalinos TA
    7. Wallace BC

    (2014)

    OpenMEE: software for ecological and evolutionary meta-analysis

    OpenMEE: software for ecological and evolutionary meta-analysis.

    • Google Scholar
17. 1. Faivre S
    2. Delbaldo C
    3. Vera K
    4. Robert C
    5. Lozahic S
    6. Lassau N
    7. Bello C
    8. Deprimo S
    9. Brega N
    10. Massimini G
    11. Armand JP
    12. Scigalla P
    13. Raymond E

    (2006) Safety, pharmacokinetic, and antitumor activity of SU11248, a novel oral multitarget tyrosine kinase inhibitor, in patients with cancer

    Journal of Clinical Oncology 24:25–35.

    https://doi.org/10.1200/JCO.2005.02.2194

    • Google Scholar
18. 1. Fisher M
    2. Feuerstein G
    3. Howells DW
    4. Hurn PD
    5. Kent TA
    6. Savitz SI
    7. Lo EH
    8. STAIR Group

    (2009) Update of the stroke therapy academic industry roundtable preclinical recommendations

    Stroke 40:2244–2250.

    https://doi.org/10.1161/STROKEAHA.108.541128

    • Google Scholar
19. 1. Galanopoulou AS
    2. Buckmaster PS
    3. Staley KJ
    4. Moshé SL
    5. Perucca E
    6. Engel J Jr
    7. Löscher W
    8. Noebels JL
    9. Pitkänen A
    10. Stables J
    11. White HS
    12. O'Brien TJ
    13. Simonato M
    14. American Epilepsy Society Basic Science Committee And The International League Against Epilepsy Working Group On Recommendations For Preclinical Epilepsy Drug Discovery

    (2012) Identification of new epilepsy treatments: issues in preclinical methodology

    Epilepsia 53:571–582.

    https://doi.org/10.1111/j.1528-1167.2011.03391.x

    • Google Scholar
20. 1. Glasziou P
    2. Altman DG
    3. Bossuyt P
    4. Boutron I
    5. Clarke M
    6. Julious S
    7. Michie S
    8. Moher D
    9. Wager E

    (2014) Reducing waste from incomplete or unusable reports of biomedical research

    Lancet 383:267–276.

    https://doi.org/10.1016/S0140-6736(13)62228-X

    • Google Scholar
21. 1. Han JY
    2. Kim HY
    3. Lim KY
    4. Han JH
    5. Lee YJ
    6. Kwak MH
    7. Kim HJ
    8. Yun T
    9. Kim HT
    10. Lee JS

    (2013) A phase II study of sunitinib in patients with relapsed or refractory small cell lung cancer

    Lung Cancer 79:137–142.

    https://doi.org/10.1016/j.lungcan.2012.09.019

    • Google Scholar
22. 1. Hay M
    2. Thomas DW
    3. Craighead JL
    4. Economides C
    5. Rosenthal J

    (2014) Clinical development success rates for investigational drugs

    Nature Biotechnology 32:40–51.

    https://doi.org/10.1038/nbt.2786

    • Google Scholar
23. 1. Henderson VC
    2. Kimmelman J
    3. Fergusson D
    4. Grimshaw JM
    5. Hackam DG

    (2013) Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments

    PLOS Medicine 10:e1001489.

    https://doi.org/10.1371/journal.pmed.1001489

    • Google Scholar
24. 1. Hill AB

    (1965)

    The environment and disease: association or causation?

    Proceedings of the Royal Society of Medicine 58:295–300.

    • Google Scholar
25. 1. Hirst TC
    2. Vesterinen HM
    3. Sena ES
    4. Egan KJ
    5. Macleod MR
    6. Whittle IR

    (2013) Systematic review and meta-analysis of temozolomide in animal models of glioma: was clinical efficacy predicted?

    British Journal of Cancer 108:64–71.

    https://doi.org/10.1038/bjc.2012.504

    • Google Scholar
26. 1. Hooijmans CR
    2. Tillema A
    3. Leenaars M
    4. Ritskes-Hoitinga M

    (2010) Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed

    Laboratory Animals 44:170–175.

    https://doi.org/10.1258/la.2010.009117

    • Google Scholar
27. 1. Houk BE
    2. Bello CL
    3. Poland B
    4. Rosen LS
    5. Demetri GD
    6. Motzer RJ

    (2010) Relationship between exposure to sunitinib and efficacy and tolerability endpoints in patients with cancer: results of a pharmacokinetic/pharmacodynamic meta-analysis

    Cancer Chemotherapy and Pharmacology 66:357–371.

    https://doi.org/10.1007/s00280-009-1170-y

    • Google Scholar
28. 1. Johnson JI
    2. Decker S
    3. Zaharevitz D
    4. Rubinstein LV
    5. Venditti JM
    6. Schepartz S
    7. Kalyandrug S
    8. Christian M
    9. Arbuck S
    10. Hollingshead M
    11. Sausville EA

    (2001) Relationships between drug activity in NCI preclinical in vitro and in vivo models and early clinical trials

    British Journal of Cancer 84:1424–1431.

    https://doi.org/10.1054/bjoc.2001.1796

    • Google Scholar
29. 1. Kerbel RS

    (2003)

    Human tumor xenografts as predictive preclinical models for anticancer drug activity in humans: better than commonly perceived-but they can be improved

    Cancer Biology & Therapy 2:S134–S139.

    • Google Scholar
30. 1. Kilkenny C
    2. Browne WJ
    3. Cuthill IC
    4. Emerson M
    5. Altman DG

    (2010) Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research

    PLOS Biology 8:e1000412.

    https://doi.org/10.1371/journal.pbio.1000412

    • Google Scholar
31. 1. Kilkenny C
    2. Parsons N
    3. Kadyszewski E
    4. Festing MF
    5. Cuthill IC
    6. Fry D
    7. Hutton J
    8. Altman DG

    (2009) Survey of the quality of experimental design, statistical analysis and reporting of research using animals

    PLOS ONE 4:e7824.

    https://doi.org/10.1371/journal.pone.0007824

    • Google Scholar
32. 1. Ko JS
    2. Rayman P
    3. Ireland J
    4. Swaidani S
    5. Li G
    6. Bunting KD
    7. Rini B
    8. Finke JH
    9. Cohen PA

    (2010) Direct and differential suppression of myeloid-derived suppressor cell subsets by sunitinib is compartmentally constrained

    Cancer Research 70:3526–3536.

    https://doi.org/10.1158/0008-5472.CAN-09-3278

    • Google Scholar
33. 1. Lau J
    2. Ioannidis JP
    3. Terrin N
    4. Schmid CH
    5. Olkin I

    (2006) The case of the misleading funnel plot

    BMJ 333:597–600.

    https://doi.org/10.1136/bmj.333.7568.597

    • Google Scholar
34. 1. Macleod MR
    2. O'Collins T
    3. Howells DW
    4. Donnan GA

    (2004) Pooling of animal experimental data reveals influence of study design and publication bias

    Stroke 35:1203–1208.

    https://doi.org/10.1161/01.STR.0000125719.25853.20

    • Google Scholar
35. 1. Maris JM
    2. Courtright J
    3. Houghton PJ
    4. Morton CL
    5. Kolb EA
    6. Lock R
    7. Tajbakhsh M
    8. Reynolds CP
    9. Keir ST
    10. Wu J
    11. Smith MA

    (2008) Initial testing (stage 1) of sunitinib by the pediatric preclinical testing program

    Pediatric Blood & Cancer 51:42–48.

    https://doi.org/10.1002/pbc.21535

    • Google Scholar
36. 1. Morrison SJ

    (2014) Time to do something about reproducibility

    eLife 3:e03981.

    https://doi.org/10.7554/eLife.03981

    • Google Scholar
37. 1. Motzer RJ
    2. Hutson TE
    3. Tomczak P
    4. Michaelson MD
    5. Bukowski RM
    6. Oudard S
    7. Negrier S
    8. Szczylik C
    9. Pili R
    10. Bjarnason GA
    11. Garcia-del-Muro X
    12. Sosman JA
    13. Solska E
    14. Wilding G
    15. Thompson JA
    16. Kim ST
    17. Chen I
    18. Huang X
    19. Figlin RA

    (2009) Overall survival and updated results for sunitinib compared with interferon alfa in patients with metastatic renal cell carcinoma

    Journal of Clinical Oncology 27:3584–3590.

    https://doi.org/10.1200/JCO.2008.20.1293

    • Google Scholar
38. 1. Motzer RJ
    2. Michaelson MD
    3. Redman BG
    4. Hudes GR
    5. Wilding G
    6. Figlin RA
    7. Ginsberg MS
    8. Kim ST
    9. Baum CM
    10. DePrimo SE
    11. Li JZ
    12. Bello CL
    13. Theuer CP
    14. George DJ
    15. Rini BI

    (2006a) Activity of SU11248, a multitargeted inhibitor of vascular endothelial growth factor receptor and platelet-derived growth factor receptor, in patients with metastatic renal cell carcinoma

    Journal of Clinical Oncology 24:16–24.

    https://doi.org/10.1200/JCO.2005.02.2574

    • Google Scholar
39. 1. Motzer RJ
    2. Rini BI
    3. Bukowski RM
    4. Curti BD
    5. George DJ
    6. Hudes GR
    7. Redman BG
    8. Margolin KA
    9. Merchan JR
    10. Wilding G
    11. Ginsberg MS
    12. Bacik J
    13. Kim ST
    14. Baum CM
    15. Michaelson MD

    (2006b) Sunitinib in patients with metastatic renal cell carcinoma

    JAMA 295:2516–2524.

    https://doi.org/10.1001/jama.295.21.2516

    • Google Scholar
40. 1. O'Collins VE
    2. Macleod MR
    3. Donnan GA
    4. Horky LL
    5. van der Worp BH
    6. Howells DW

    (2006) 1,026 experimental treatments in acute stroke

    Annals of Neurology 59:467–477.

    https://doi.org/10.1002/ana.20741

    • Google Scholar
41. 1. Peterson JK
    2. Houghton PJ

    (2004) Integrating pharmacology and in vivo cancer models in preclinical and clinical drug development

    European Journal of Cancer 40:837–844.

    https://doi.org/10.1016/j.ejca.2004.01.003

    • Google Scholar
42. 1. Pusztai L
    2. Hatzis C
    3. Andre F

    (2013) Reproducibility of research and preclinical validation: problems and solutions

    Nature Reviews Clinical Oncology 10:720–724.

    https://doi.org/10.1038/nrclinonc.2013.171

    • Google Scholar
43. 1. Raymond E
    2. Dahan L
    3. Raoul JL
    4. Bang YJ
    5. Borbath I
    6. Lombard-Bohas C
    7. Valle J
    8. Metrakos P
    9. Smith D
    10. Vinik A
    11. Chen JS
    12. Hörsch D
    13. Hammel P
    14. Wiedenmann B
    15. Van Cutsem E
    16. Patyna S
    17. Lu DR
    18. Blanckmeister C
    19. Chao R
    20. Ruszniewski P

    (2011) Sunitinib malate for the treatment of pancreatic neuroendocrine tumors

    The New England Journal of Medicine 364:501–513.

    https://doi.org/10.1056/NEJMoa1003825

    • Google Scholar
44. 1. Rooke ED
    2. Vesterinen HM
    3. Sena ES
    4. Egan KJ
    5. Macleod MR

    (2011) Dopamine agonists in animal models of Parkinson's disease: a systematic review and meta-analysis

    Parkinsonism & Related Disorders 17:313–320.

    https://doi.org/10.1016/j.parkreldis.2011.02.010

    • Google Scholar
45. 1. Smith MA
    2. Houghton P

    (2013) A proposal regarding reporting of in vitro testing results

    Clinical Cancer Research 19:2828–2833.

    https://doi.org/10.1158/1078-0432.CCR-13-0043

    • Google Scholar
46. 1. Sugar E
    2. Pascoe AJ
    3. Azad N

    (2012) Reporting of preclinical tumor-graft cancer therapeutic studies

    Cancer Biology & Therapy 13:1262–1268.

    https://doi.org/10.4161/cbt.21782

    • Google Scholar
47. 1. Tsilidis KK
    2. Panagiotou OA
    3. Sena ES
    4. Aretouli E
    5. Evangelou E
    6. Howells DW
    7. Al-Shahi Salman R
    8. Macleod MR
    9. Ioannidis JP

    (2013) Evaluation of excess significance bias in animal studies of neurological diseases

    PLOS Biology 11:e1001609.

    https://doi.org/10.1371/journal.pbio.1001609

    • Google Scholar
48. 1. van der Worp HB
    2. de Haan P
    3. Morrema E
    4. Kalkman CJ

    (2005) Methodological quality of animal studies on neuroprotection in focal cerebral ischaemia

    Journal of Neurology 252:1108–1114.

    https://doi.org/10.1007/s00415-005-0802-3

    • Google Scholar
49. 1. van der Worp HB
    2. Howells DW
    3. Sena ES
    4. Porritt MJ
    5. Rewell S
    6. O'Collins V
    7. Macleod MR

    (2010) Can animal models of disease reliably inform human studies?

    PLOS Medicine 7:e1000245.

    https://doi.org/10.1371/journal.pmed.1000245

    • Google Scholar
50. 1. Voskoglou-Nomikos T
    2. Pater JL
    3. Seymour L

    (2003)

    Clinical predictive value of the in vitro cell line, human xenograft, and mouse allograft preclinical cancer models

    Clinical Cancer Research 9:4227–4239.

    • Google Scholar
51. 1. Wallace BC
    2. Schmid CH
    3. Lau J
    4. Trikalinos TA

    (2009) Meta-analyst: software for meta-analysis of binary, continuous and diagnostic data

    BMC Medical Research Methodology 9:80.

    https://doi.org/10.1186/1471-2288-9-80

    • Google Scholar

Thank you for submitting your work entitled “A Meta-Analysis of Validity Threats and Clinical Correlates in Preclinical Research of Sunitinib” for peer review at eLife. Your submission has been favorably evaluated by Prabhat Jha (Senior Editor) and three reviewers, one of whom, Dawn Teare, is a member of our Board of Reviewing Editors. The other two reviewers, Carlijn Hooijmans and Glenn Begley, have agreed to share their identity.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

The authors describe a meta-analysis of preclinical studies performed with sunitinib. Their analysis found poor scientific methodology with for example lack of blinding, lack of evidence of power calculations and likely bias in over-reporting of positive results. It provides objective evidence to support claims that have been made by others. This is an important and valuable contribution. It addresses an issue of increasing scientific and societal importance.

Essential revisions:

1) The dose response analysis needs revision. The systematic review has collected the data from animal research but I cannot see where the clinical outcome data comes from. In fact I do not see any clinical data. Is the assumption that the clinical data does show a dose response? I find the lack of a dose response rather curious in that presumably individual studies did report an effect but somehow there is heterogeneity between the studies and hence this reduces to a random scatter? There appear to be a lot of points in Figure 5A but I cannot easily see how many studies were eligible for the dose-response. I also do not understand the statistical analysis going along with Figure 5. What sort of curve/line is being fitted? What role does the two-group t-test have in a dose response analysis? The line drawn on the figure appears to be a straight line. I think this section needs to be more clearly written.

2) It would be very helpful for the authors to comment on the number of studies that reported drug exposure rather than drug dose. It is common for preclinical studies to report the intended drug dose, but without reference to the actual levels of drug attained. Assessment of drug levels confirms (i) that drug was actually administered and (ii) to the intended cohort. Moreover, it is frequently the case that when levels rather than dose are examined, it is evident that the levels achieved in rodents are not tolerated in humans. This has specifically been recognized for sunitinib and is an additional explanation for the disconnection between preclinical versus clinical effectiveness.

3) It would also be helpful if the authors could comment on the number of studies that included a positive control. While sunitinib may have had activity, without a positive control it is difficult to place that activity into context. Was the efficacy comparable or greater than that seen with another known active agent?

4) It would be beneficial to understand how many of the studies used an appropriate statistical test when analyzing their data.

5) The authors did not comment on the ARRIVE Guidelines for pharmacology studies. These should be discussed (cited in Begley and Ioannidis., Reproducibility in Science. Improving the Standard for Basic and Preclinical Research., Circulation Research., 2015; 116: 116-126.)

6) The authors should check that their references are correctly cited. For example, the statement (in the Introduction) “A recent systematic review of guidelines for limiting bias in preclinical research design was unable to identify any guidelines in oncology (Han et al., 2013)” is not supported by Han et al., 2013.

7) The observation that “Our findings suggest that testing in more models tends to produce smaller effect sizes” has also been made previously (cited in Begley and Ioannidis, 2015).

8) Why did the authors of this review decide a) to include only papers published in English b) not to retrieve data via contacting the authors of the original papers? The authors conclude that there is a substantial risk on publication bias. But it might be possible that the observed risk on publication bias could have been much smaller when the authors contacted the authors for missing data and included also papers published in languages other than English.

9) In this review, the outcome measure of interest needs a more detailed description in the Material and methods section. In the subsection “Extraction,” the authors state that they are only interested in tumor volume (“[…] those evaluating the effect of monotherapy on tumor volume”), whereas the Abstract and Introduction talk about tumor growth (which is a much broader term). Did the authors only assess tumor volume of primary tumors? Or did they include the volumes of metastases in the body as well? Did the authors include studies in which the weight of tumors and metastases were assessed? And what about tumor incidence? The authors also state: “[…] To account for different measures of tumor growth”; which measures of tumor growth did they include?

10) More details are needed in the Materials and methods concerning the meta analyses:

a) Are data pooled in case tumor volume was determined in various regions of the body within one experiment?

Are the data pooled in the overall analyses (Figure 3) when results were assessed at various time points?

b) In animal studies the same control group is often used for multiple experimental. Did the authors correct for multiple use of the control groups?

c) What was the minimum group size to allow subgroup analyses?

11) The authors should report group sizes of all subgroups (also in figures; e.g. Figure 3), and take group size into account when interpreting the results.

12) The authors should report heterogeneity statistics (for example I²), and take these results into account when interpreting the data.

Essential revisions:

1) The dose response analysis needs revision. The systematic review has collected the data from animal research but I cannot see where the clinical outcome data comes from. In fact I do not see any clinical data. Is the assumption that the clinical data does show a dose response? I find the lack of a dose response rather curious in that presumably individual studies did report an effect but somehow there is heterogeneity between the studies and hence this reduces to a random scatter? There appear to be a lot of points in Figure 5A but I cannot easily see how many studies were eligible for the dose-response. I also do not understand the statistical analysis going along with Figure 5. What sort of curve/line is being fitted? What role does the two-group t-test have in a dose response analysis? The line drawn on the figure appears to be a straight line. I think this section needs to be more clearly written.

Dose response is discussed twice in the manuscript. First it appears when we report the fraction of studies that tested a dose response. Here, our emphasis is on dose response testing as a means of enhancing the internal validity of claims regarding clinical promise. The second mention is at the end of our Results section. There, the dose-response analysis was not meant to compare our preclinical results directly with sunitinib clinical data (although a dose-response relationship in GIST and mRCC has been demonstrated by others – see below). Our premise was that preclinical studies should demonstrate a dose-response relationship in the absence of overwhelming inter-experimental heterogeneity and publication bias. This premise was based on the known dose response in human beings, and the fact that dose-response was shown within individual preclinical experiments. The reviewers have interpreted our point correctly: our inability to show a clear dose-response after pooling all results raises some concerns about the predictive value of studies included in our meta-analysis.

Regarding sample size, there are 158 data points in Figure 5A. Each data point represents a single experiment or sub-experiment (in cases where one experiment had more than one treatment arm with a defined dose). Inclusion into the dose-response analysis required experiments or sub-experiments to have used a regular dosing schedule (i.e. sunitinib administered at a defined dose once a day instead of experiments where sunitinib was dosed more irregularly or only once). These inclusion criteria led to 26 experiments or sub-experiments to be excluded. It is worth reiterating that the number of data points possible in our dose-response analysis for all malignancies was 183. This number differs from the total number of data points possible in our other analyses, where we calculated a weighted average of the SMDs and 95% CIs in the cases of experiments that had more than one treatment arm.

Regarding the curve we fitted, we believe our simple regression (calculated using the linear regression function in Excel) is sound for descriptive purposes and assessing a simple relationship between doses. Our intent was simply to judge the presence or absence of an expected positive correlation between effect and increasing dose rather than identifying precise dose-response curves.

Regarding the source of clinical outcome data, we think this confusion stems from a poor choice of title, which we changed to “A Meta-Analysis of Threats to Valid Clinical Inference in Preclinical Research of Sunitinib”.

We modified the manuscript as follows:

We enhanced the description of dose response in Methods, Results, and Discussion sections. We also cite Houk et al (PMID: 19967539) to show that a dose-response relationship was found in GIST and mRCC in clinical studies. This citation helps support our assertion that the absence of a dose-response in preclinical models (especially when RCC was analyzed alone) is odd.

We also removed the two-group t-test, which was a remnant of a previous draft and out of place.

We made some edits to figure legend, as well as Methods section, to explain more clearly how (and why) we performed dose response curves, as well as the number of experiments included in them.

2) It would be very helpful for the authors to comment on the number of studies that reported drug exposure rather than drug dose. It is common for preclinical studies to report the intended drug dose, but without reference to the actual levels of drug attained. Assessment of drug levels confirms (i) that drug was actually administered and (ii) to the intended cohort. Moreover, it is frequently the case that when levels rather than dose are examined, it is evident that the levels achieved in rodents are not tolerated in humans. This has specifically been recognized for sunitinib and is an additional explanation for the disconnection between preclinical versus clinical effectiveness.

We agree that reporting drug exposure (and measured serum levels of drug in experimental animals) would be the optimal measurement upon which to construct dose-response curves and to later correlate with clinical data. Unfortunately, these measurements were rarely included within tumour volume experiments we extracted. Indeed, this comment prompted us to reexamine the first 20% of studies in our sample and we found no instances where levels were measured instead of dose. We had included 4 studies that included serum measurements of sunitinib post-dosing (PK). However, none of these serum levels were taken during efficacy experiments; they were only performed in studies to correlate with molecular data (e.g. FLT3 expression levels).

We protocolized and standardized eligibility criteria and dose exposure measurement to reduce variability inherent with using dose, given that drug serum levels were not available to us. For instance, in dose response analysis, we only included experiments where the dose was clearly stated and was administered once per day (as opposed to more complex or variable dosing schedules). Additionally, we used a common time point between experiments (14 days after the first day of drug administration).

The difficulty we faced constructing a robust measure of drug exposure and effect further highlights the poor reporting practices in preclinical research and the need for improvement in this regard.

In light of the above, we made the following modifications to the manuscript:

We added a statement in the Methods section as to why we used dose instead of drug exposure (“As we were unable to find experiments that reported drug exposure (e.g. drug serum levels), we calculated pooled effect sizes in OpenMeta[Analyst] and plotted against dose”).

We also added a statement in the Discussion section stating “the tendency for preclinical efficacy studies to report drug dose, but rarely drug exposure (i.e. serum measurement of active drug), further limits the construct validity of these studies,” and we buttressed this with a new reference (Peterson et al., 2004).

3) It would also be helpful if the authors could comment on the number of studies that included a positive control. While sunitinib may have had activity, without a positive control it is difficult to place that activity into context. Was the efficacy comparable or greater than that seen with another known active agent?

We agree that efficacy data are difficult to make sense of absent positive controls. Examining our dataset, we found that 93/158 experiments included in the meta-analysis included an active comparator drug arm. Note that we do not use the term “positive control” strictly here, as sometimes sunitinib was being the positive control in a study of a newer drug.

We inserted two sentences to the subsection “Design Elements Addressing Validity Threats” describing the frequency of active drug comparators in experiments, and the relationship between their inclusion and reported effect sizes.

4) It would be beneficial to understand how many of the studies used an appropriate statistical test when analyzing their data.

We agree this would be a useful analysis but it is beyond the scope of this paper. To address appropriateness adequately we would need raw subject data, subject flow, protocols, and analytical plans. While some issues are black and white, many are nuanced/subjective and require complete background info. Instead, we focused our energies in the validity criteria identified in our PLoS Medicine systematic review of preclinical design guidelines.

5) The authors did not comment on the ARRIVE Guidelines for pharmacology studies. These should be discussed (cited in Begley and Ioannidis., Reproducibility in Science. Improving the Standard for Basic and Preclinical Research., Circulation Research., 2015; 116: 116-126.)

We heartily agree that citing ARRIVE is important for this audience and we’ve now cited and singled it out for mention at the end of the Discussion. We also added a reference to the Circulation Research article, which provides a nice overview of reproducibility challenges. We also took this occasion to mention the Center for Open Science Reproducibility Project: Cancer Biology.

6) The authors should check that their references are correctly cited. For example, the statement (in the Introduction) “A recent systematic review of guidelines for limiting bias in preclinical research design was unable to identify any guidelines in oncology (Han et al., 2013)” is not supported by Han et al., 2013.

Thanks to the reviewers for catching this. It has now been corrected.

7) The observation that “Our findings suggest that testing in more models tends to produce smaller effect sizes” has also been made previously (cited in Begley and Ioannidis, 2015).

Thank you. We now cite the Circulation article (see above). For this particular claim, the referees are correct – this has been suggested before – in particular by CAMARADES. We now cite the paper (O’Collins et al, Annals Neurol, 2006).

8) Why did the authors of this review decide a) to include only papers published in English b) not to retrieve data via contacting the authors of the original papers? The authors conclude that there is a substantial risk on publication bias. But it might be possible that the observed risk on publication bias could have been much smaller when the authors contacted the authors for missing data and included also papers published in languages other than English.

This was decided mainly on feasibility and funding constraints. We did not have a budget to both identify relevant expertise and to translate articles. Contacting authors for additional data would pull in experiments and findings that have not been subject to peer review. Our experience contacting drug companies for missing data does not lead us to think this would have been a productive use of our time. We did, however, attempt to obtain Sugen/Pfizer’s preclinical dataset from FDA by filing a Freedom of Information Act. The FOIA was submitted in February of 2012, around the time the grant received funding. The 3-year grant ended in 2015, with the FOIA request not yet honoured. We added a sentence in the Discussion stating this as a limitation.

9) In this review, the outcome measure of interest needs a more detailed description in the Material and methods section. In the subsection “Extraction,” the authors state that they are only interested in tumor volume (“[…] those evaluating the effect of monotherapy on tumor volume”), whereas the Abstract and Introduction talk about tumor growth (which is a much broader term). Did the authors only assess tumor volume of primary tumors? Or did they include the volumes of metastases in the body as well? Did the authors include studies in which the weight of tumors and metastases were assessed? And what about tumor incidence? The authors also state: “[…] To account for different measures of tumor growth”; which measures of tumor growth did they include?

We expanded the Methods section to reflect the reviewers’ queries. For instance, we changed our broad references to “tumour growth” into references to tumour volume measurements (which is what we were extracting). We also provide a more detailed description of outcome measurement (see, for example, text beginning “Our primary outcome was experimental […]”, in the subsection “Extraction”).

10) More details are needed in the Materials and methods concerning the meta analyses.

a) Are data pooled in case tumor volume was determined in various regions of the body within one experiment?

We were a bit unsure what the reviewers mean by “various regions of the body within one experiment”. We did not encounter studies where tumors were injected into more than one location within a single experiment. Perhaps the referees are referring to inter-experimental variability in tumour implantation site (e.g. experimental renal tumour injected subcutaneously into the hindleg vs. injected orthotopically into the renal capsule of the mouse). If this is the case, we did record this information (as to whether the tumour implantation site was heterotopic, orthotopic, or systemic), but we did not perform separate analyses according to this variable. Our concern is that doing so would divide our sample into slices too thin to enable analysis. If referees think it useful we would be happy to add a sentence or two about the use orthotopic vs. heterotopic models in our sample of experiments.

Are the data pooled in the overall analyses (Figure 3) when results were assessed at various time points?

We clarified information of our time points to the Methods (in the subsection “Extraction”). We also clarified which time point was used for which analysis in the Results section and in each figure legend. As explained in the subsection “Study Characteristics”, the mean number of days in last common timepoint (LCT) was 31 days (± 14 days SDM).

b) In animal studies the same control group is often used for multiple experimental. Did the authors correct for multiple use of the control groups?

Indeed, we did adjust. In cases where multiple arms of treatment were reported versus the same control, a pooled estimate of the treatment arms was calculated using the inverse variance approach and then compared to the single control to produce an SMD estimate and 95% CIs.

c) What was the minimum group size to allow subgroup analyses?

We did not specify a minimum group size for meta-analysis. Instead we presented all available data. The goal of subgroup analyses in meta-analysis is to present all available data relevant to the strata even if just 1 experiment, thus a minimum subgroup size is not as relevant. Now that we have integrated sample sizes into our figures more clearly, readers can perhaps make their own judgments.

11) The authors should report group sizes of all subgroups (also in figures; e.g. Figure 3), and take group size into account when interpreting the results.

We converted Figure 3 to forest plots and have included the subgroup sizes.

12) The authors should report heterogeneity statistics (for example I²), and take these results into account when interpreting the data.

The primary purpose of our paper was to evaluate potential sources of methodological and pre-clinical heterogeneity through pre-specified items of internal, external, and construct validity. We now report I² statistics but we must note that we (and others – e.g. Paul SR, Donner A., Small sample performance of tests of homogeneity of odds ratios in k 2×2 tables., Stat Med 1992; 11: 159-65; Hardy RJ, Thompson SG., Detecting and describing heterogeneity in meta-analysis., Stat Med 1998; 17: 841-56) are somewhat skeptical about the value of heterogeneity statistics due to their extreme sensitivity in the presence of a small number of studies. As the referees will see, heterogeneity statistics, as expected, within each malignancy was generally high (see Figure 2–source data 1B), with some notable exceptions (e.g. high-grade glioma, prostate cancer). By design, our study sheds light on possible sources of methodological heterogeneity (i.e. Figure 3) and we have incorporated such observations in our interpretation and conclusions.

Author details

1. Valerie C Henderson

   Studies of Translation, Ethics and Medicine Research Group, Biomedical Ethics Unit, McGill University, Montréal, Canada

   Contribution

   VCH, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

2. Nadine Demko

   Studies of Translation, Ethics and Medicine Research Group, Biomedical Ethics Unit, McGill University, Montréal, Canada

   Contribution

   ND, Acquisition of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

3. Amanda Hakala

   Studies of Translation, Ethics and Medicine Research Group, Biomedical Ethics Unit, McGill University, Montréal, Canada

   Contribution

   AH, Acquisition of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

4. Nathalie MacKinnon

   Studies of Translation, Ethics and Medicine Research Group, Biomedical Ethics Unit, McGill University, Montréal, Canada

   Contribution

   NMK, Acquisition of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

5. Carole A Federico

   Studies of Translation, Ethics and Medicine Research Group, Biomedical Ethics Unit, McGill University, Montréal, Canada

   Contribution

   CAF, Acquisition of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

6. Dean Fergusson

   Department of Clinical Epidemiology, Ottawa Hospital Research Institute, Ottawa, Canada

   Contribution

   DF, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

7. Jonathan Kimmelman

   Studies of Translation, Ethics and Medicine Research Group, Biomedical Ethics Unit, McGill University, Montréal, Canada

   Contribution

   JK, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   For correspondence

   jonathan.kimmelman@mcgill.ca

   Competing interests

   The authors declare that no competing interests exist.

• 4,408

  Page views

• 678

  Downloads

• 31

  Citations

Article citation count generated by polling the highest count across the following sources: Scopus, Crossref, PubMed Central.