Reproducibility in Cancer Biology: Mixed outcomes for computational predictions

  1. Chi Van Dang  Is a corresponding author
  1. University of Pennsylvania, United States


Experimental efforts to validate the output of a computational model that predicts new uses for existing drugs highlights the inherently complex nature of cancer biology.

Main text

In 2011 researchers at Stanford and the Lucile Packard Children's Hospital reported a computational approach for predicting if a drug that was currently approved for the treatment of a certain disease could also be used to treat a form of cancer (Sirota et al., 2011). The researchers derived gene expression signatures for 100 diseases from data available in the Gene Expression Omnibus (GEO) and compared these signatures with gene expression measurements on 164 drugs.

Based on these comparisons Sirota et al. predicted that cimetidine, an antiulcer drug, could be used to treat a form of lung cancer called lung adenocarcinoma, and then went on to demonstrate the efficacy of cimetidine against this form of cancer both in vitro and in vivo (in xenograft experiments in which human tumor cells were transplanted into mice). They also confirmed, as predicted by their computational model, that cimetidine was not effective against renal carcinoma.

In 2015, as part of the Reproducibility Project: Cancer Biology, Kandela et al. published a Registered Report (Kandela et al., 2015) which explained in detail how they would seek to replicate selected experiments from Sirota et al. The results of these experiments have now been published as a Replication Study (Kandela et al., 2017).

In the original work, Sirota et al. demonstrated that cimetidine induced the death of the lung cancer cell line A549. To corroborate this finding, they tested three doses of cimetidine against A549 tumors that had been implanted in mice. While tumors treated with a negative control grew to 3.25 times their original volume, and tumors treated with a positive control (an established cancer drug called doxorubicin) doubled in size, tumors treated with the highest dose of cimetidine grew to 2.3 times their original volume, which was statistically significant. However, xenograft studies are inherently complex – there is a lot of variability in the length of time it takes a tumor to become established after implantation, and tumors also grow at different rates in different animals – and the effects of doxorubicin and cimetidine on the growth rates of tumors do not seem very robust from a biological point of view.

In the Replication Study, Kandela et al. found that cimetidine treatment in the lung adenocarcinoma xenograft model resulted in decreased tumor sizes compared to a negative control treatment (Kandela et al., 2017). However, while the effects were in the same direction as those reported by Sirota et al., they were not significant when a Bonferroni correction was used to adjust for multiple comparisons. Treatment with doxorubicin also reduced the size of the lung tumors compared to a control, but again the effects were not significant. In both cases, however, a statistically significant effect was observed when the dataset from the original paper and the dataset from the Replication Study were combined in a meta-analysis.

The findings of the Replication Study raise issues related to robustness, statistical methods, and effect sizes. Robustness characterizes the consistent response of a system to perturbations: the more robust the system, the less influence these perturbations have on its output. There are many factors that could influence the robustness of the xenograft models used in these experiments: batch effects on the efficacy of the drugs used; changes in the properties of cell lines over time; the strains of the mice used, and also their sex; factors related to microbiome and chow; circadian effects; temperature; and the antimicrobials that might be used in certain facilities.

It is also possible that Sirota et al. paid too much attention to statistical significance and P values and not enough attention to the actual size of the effect being investigated (Motulsky, 2014). Indeed, the actual tumor sizes observed by Sirota et al. were not diminished by cimetidine at early time points, and the reduction in tumor size at the last time point (the only time point at which the reduction was statistically significant) was only about 30%. This moderate effect size might have contributed to the fact that the effects seen in the replication were similar to those in the original experiments, but not statistically significant (although, as mentioned above, combining data from the two studies did give significant results).

There have been growing concerns about irreproducibility in the biological literature in recent years (Begley and Ellis, 2012; Errington et al., 2014; Baker, 2016). A powerful lesson to emerge from this Replication Study is that reproducibility is not black and white because, like many areas of research, cancer biology is nuanced and inherently complex.


Chi Van Dang was the eLife Reviewing Editor for the Registered Report (Kandela et al., 2015) and the Replication Study (Kandela et al., 2017).


Article and author information

Author details

  1. Chi Van Dang

    Abramson Cancer Center, Abramson Family Cancer Research Institute, University of Pennsylvania, Philadelphia, United States
    For correspondence
    Competing interests
    The author declares that no competing interests exist.

Publication history

  1. Version of Record published: January 19, 2017 (version 1)


© 2017, Dang

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.


  • 1,412
    Page views
  • 157
  • 4

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Chi Van Dang
Reproducibility in Cancer Biology: Mixed outcomes for computational predictions
eLife 6:e22661.

Further reading

    1. Cancer Biology
    Irawati Kandela et al.
    Replication Study

    In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper “Discovery and Preclinical Validation of Drug Indications Using Compendia of Public Gene Expression Data“ (Sirota et al., 2011). Here we report the results of those experiments. We found that cimetidine treatment in a xenograft model using A549 lung adenocarcinoma cells resulted in decreased tumor volume compared to vehicle control; however, while the effect was in the same direction as the original study (Figure 4C; Sirota et al., 2011), it was not statistically significant. Cimetidine treatment in a xenograft model using ACHN renal cell carcinoma cells did not differ from vehicle control treatment, similar to the original study (Supplemental Figure 1; Sirota et al., 2011). Doxorubicin treatment in a xenograft model using A549 lung adenocarcinoma cells did not result in a statistically significant difference compared to vehicle control despite tumor volume being reduced to levels similar to those reported in the original study (Figure 4C; Sirota et al., 2011). Finally, we report a random effects meta-analysis for each result. These meta-analyses show that the inhibition of A549 derived tumors by cimetidine resulted in a statistically significant effect, as did the inhibition of A549 derived tumors by doxorubicin. The effect of cimetidine on ACHN derived tumors was not statistically significant, as predicted.

    1. Cancer Biology
    2. Computational and Systems Biology
    Iurii Petrov, Andrey Alexeyenko
    Research Article

    Late advances in genome sequencing expanded the space of known cancer driver genes several-fold. However, most of this surge was based on computational analysis of somatic mutation frequencies and/or their impact on the protein function. On the contrary, experimental research necessarily accounted for functional context of mutations interacting with other genes and conferring cancer phenotypes. Eventually, just such results become 'hard currency' of cancer biology. The new method, NEAdriver employs knowledge accumulated thus far in the form of global interaction network and functionally annotated pathways in order to recover known and predict novel driver genes. The driver discovery was individualized by accounting for mutations' co-occurrence in each tumour genome - as an alternative to summarizing information over the whole cancer patient cohorts. For each somatic genome change, probabilistic estimates from two lanes of network analysis were combined into joint likelihoods of being a driver. Thus, ability to detect previously unnoticed candidate driver events emerged from combining individual genomic context with network perspective. The procedure was applied to ten largest cancer cohorts followed by evaluating error rates against previous cancer gene sets. The discovered driver combinations were shown to be informative on cancer outcome. This revealed driver genes with individually sparse mutation patterns that would not be detectable by other computational methods and related to cancer biology domains poorly covered by previous analyses. In particular, recurrent mutations of collagen, laminin, and integrin genes were observed in the adenocarcinoma and glioblastoma cancers. Considering constellation patterns of candidate drivers in individual cancer genomes opens a novel avenue for personalized cancer medicine.