Reproducibility in Cancer Biology: Mixed outcomes for computational predictions

Experimental efforts to validate the output of a computational model that predicts new uses for existing drugs highlights the inherently complex nature of cancer biology.
  1. Chi Van Dang  Is a corresponding author
  1. University of Pennsylvania, United States

In 2011 researchers at Stanford and the Lucile Packard Children's Hospital reported a computational approach for predicting if a drug that was currently approved for the treatment of a certain disease could also be used to treat a form of cancer (Sirota et al., 2011). The researchers derived gene expression signatures for 100 diseases from data available in the Gene Expression Omnibus (GEO) and compared these signatures with gene expression measurements on 164 drugs.

Based on these comparisons Sirota et al. predicted that cimetidine, an antiulcer drug, could be used to treat a form of lung cancer called lung adenocarcinoma, and then went on to demonstrate the efficacy of cimetidine against this form of cancer both in vitro and in vivo (in xenograft experiments in which human tumor cells were transplanted into mice). They also confirmed, as predicted by their computational model, that cimetidine was not effective against renal carcinoma.

In 2015, as part of the Reproducibility Project: Cancer Biology, Kandela et al. published a Registered Report (Kandela et al., 2015) which explained in detail how they would seek to replicate selected experiments from Sirota et al. The results of these experiments have now been published as a Replication Study (Kandela et al., 2017).

In the original work, Sirota et al. demonstrated that cimetidine induced the death of the lung cancer cell line A549. To corroborate this finding, they tested three doses of cimetidine against A549 tumors that had been implanted in mice. While tumors treated with a negative control grew to 3.25 times their original volume, and tumors treated with a positive control (an established cancer drug called doxorubicin) doubled in size, tumors treated with the highest dose of cimetidine grew to 2.3 times their original volume, which was statistically significant. However, xenograft studies are inherently complex – there is a lot of variability in the length of time it takes a tumor to become established after implantation, and tumors also grow at different rates in different animals – and the effects of doxorubicin and cimetidine on the growth rates of tumors do not seem very robust from a biological point of view.

In the Replication Study, Kandela et al. found that cimetidine treatment in the lung adenocarcinoma xenograft model resulted in decreased tumor sizes compared to a negative control treatment (Kandela et al., 2017). However, while the effects were in the same direction as those reported by Sirota et al., they were not significant when a Bonferroni correction was used to adjust for multiple comparisons. Treatment with doxorubicin also reduced the size of the lung tumors compared to a control, but again the effects were not significant. In both cases, however, a statistically significant effect was observed when the dataset from the original paper and the dataset from the Replication Study were combined in a meta-analysis.

The findings of the Replication Study raise issues related to robustness, statistical methods, and effect sizes. Robustness characterizes the consistent response of a system to perturbations: the more robust the system, the less influence these perturbations have on its output. There are many factors that could influence the robustness of the xenograft models used in these experiments: batch effects on the efficacy of the drugs used; changes in the properties of cell lines over time; the strains of the mice used, and also their sex; factors related to microbiome and chow; circadian effects; temperature; and the antimicrobials that might be used in certain facilities.

It is also possible that Sirota et al. paid too much attention to statistical significance and P values and not enough attention to the actual size of the effect being investigated (Motulsky, 2014). Indeed, the actual tumor sizes observed by Sirota et al. were not diminished by cimetidine at early time points, and the reduction in tumor size at the last time point (the only time point at which the reduction was statistically significant) was only about 30%. This moderate effect size might have contributed to the fact that the effects seen in the replication were similar to those in the original experiments, but not statistically significant (although, as mentioned above, combining data from the two studies did give significant results).

There have been growing concerns about irreproducibility in the biological literature in recent years (Begley and Ellis, 2012; Errington et al., 2014; Baker, 2016). A powerful lesson to emerge from this Replication Study is that reproducibility is not black and white because, like many areas of research, cancer biology is nuanced and inherently complex.


Chi Van Dang was the eLife Reviewing Editor for the Registered Report (Kandela et al., 2015) and the Replication Study (Kandela et al., 2017).


Article and author information

Author details

  1. Chi Van Dang

    Abramson Cancer Center, Abramson Family Cancer Research Institute, University of Pennsylvania, Philadelphia, United States
    For correspondence
    Competing interests
    The author declares that no competing interests exist.

Publication history

  1. Version of Record published: January 19, 2017 (version 1)


© 2017, Dang

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.


  • 1,444
    Page views
  • 162
  • 4

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Chi Van Dang
Reproducibility in Cancer Biology: Mixed outcomes for computational predictions
eLife 6:e22661.
  1. Further reading

Further reading

    1. Cancer Biology
    Irawati Kandela, Fraser Aird, Reproducibility Project: Cancer Biology
    Replication Study

    In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper “Discovery and Preclinical Validation of Drug Indications Using Compendia of Public Gene Expression Data“ (Sirota et al., 2011). Here we report the results of those experiments. We found that cimetidine treatment in a xenograft model using A549 lung adenocarcinoma cells resulted in decreased tumor volume compared to vehicle control; however, while the effect was in the same direction as the original study (Figure 4C; Sirota et al., 2011), it was not statistically significant. Cimetidine treatment in a xenograft model using ACHN renal cell carcinoma cells did not differ from vehicle control treatment, similar to the original study (Supplemental Figure 1; Sirota et al., 2011). Doxorubicin treatment in a xenograft model using A549 lung adenocarcinoma cells did not result in a statistically significant difference compared to vehicle control despite tumor volume being reduced to levels similar to those reported in the original study (Figure 4C; Sirota et al., 2011). Finally, we report a random effects meta-analysis for each result. These meta-analyses show that the inhibition of A549 derived tumors by cimetidine resulted in a statistically significant effect, as did the inhibition of A549 derived tumors by doxorubicin. The effect of cimetidine on ACHN derived tumors was not statistically significant, as predicted.

    1. Cancer Biology
    Aojia Zhuang, Aobo Zhuang ... Chen Ding
    Research Article Updated

    The presence of lymph node metastasis (LNM) affects treatment strategy decisions in T1NxM0 colorectal cancer (CRC), but the currently used clinicopathological-based risk stratification cannot predict LNM accurately. In this study, we detected proteins in formalin-fixed paraffin-embedded (FFPE) tumor samples from 143 LNM-negative and 78 LNM-positive patients with T1 CRC and revealed changes in molecular and biological pathways by label-free liquid chromatography tandem mass spectrometry (LC-MS/MS) and established classifiers for predicting LNM in T1 CRC. An effective 55-proteins prediction model was built by machine learning and validated in a training cohort (N=132) and two validation cohorts (VC1, N=42; VC2, N=47), achieved an impressive AUC of 1.00 in the training cohort, 0.96 in VC1 and 0.93 in VC2, respectively. We further built a simplified classifier with nine proteins, and achieved an AUC of 0.824. The simplified classifier was performed excellently in two external validation cohorts. The expression patterns of 13 proteins were confirmed by immunohistochemistry, and the IHC score of five proteins was used to build an IHC predict model with an AUC of 0.825. RHOT2 silence significantly enhanced migration and invasion of colon cancer cells. Our study explored the mechanism of metastasis in T1 CRC and can be used to facilitate the individualized prediction of LNM in patients with T1 CRC, which may provide a guidance for clinical practice in T1 CRC.