Experimental efforts to validate the output of a computational model that predicts new uses for existing drugs highlights the inherently complex nature of cancer biology.
In 2011 researchers at Stanford and the Lucile Packard Children's Hospital reported a computational approach for predicting if a drug that was currently approved for the treatment of a certain disease could also be used to treat a form of cancer (Sirota et al., 2011). The researchers derived gene expression signatures for 100 diseases from data available in the Gene Expression Omnibus (GEO) and compared these signatures with gene expression measurements on 164 drugs.
Based on these comparisons Sirota et al. predicted that cimetidine, an antiulcer drug, could be used to treat a form of lung cancer called lung adenocarcinoma, and then went on to demonstrate the efficacy of cimetidine against this form of cancer both in vitro and in vivo (in xenograft experiments in which human tumor cells were transplanted into mice). They also confirmed, as predicted by their computational model, that cimetidine was not effective against renal carcinoma.
In 2015, as part of the Reproducibility Project: Cancer Biology, Kandela et al. published a Registered Report (Kandela et al., 2015) which explained in detail how they would seek to replicate selected experiments from Sirota et al. The results of these experiments have now been published as a Replication Study (Kandela et al., 2017).
In the original work, Sirota et al. demonstrated that cimetidine induced the death of the lung cancer cell line A549. To corroborate this finding, they tested three doses of cimetidine against A549 tumors that had been implanted in mice. While tumors treated with a negative control grew to 3.25 times their original volume, and tumors treated with a positive control (an established cancer drug called doxorubicin) doubled in size, tumors treated with the highest dose of cimetidine grew to 2.3 times their original volume, which was statistically significant. However, xenograft studies are inherently complex – there is a lot of variability in the length of time it takes a tumor to become established after implantation, and tumors also grow at different rates in different animals – and the effects of doxorubicin and cimetidine on the growth rates of tumors do not seem very robust from a biological point of view.
In the Replication Study, Kandela et al. found that cimetidine treatment in the lung adenocarcinoma xenograft model resulted in decreased tumor sizes compared to a negative control treatment (Kandela et al., 2017). However, while the effects were in the same direction as those reported by Sirota et al., they were not significant when a Bonferroni correction was used to adjust for multiple comparisons. Treatment with doxorubicin also reduced the size of the lung tumors compared to a control, but again the effects were not significant. In both cases, however, a statistically significant effect was observed when the dataset from the original paper and the dataset from the Replication Study were combined in a meta-analysis.
The findings of the Replication Study raise issues related to robustness, statistical methods, and effect sizes. Robustness characterizes the consistent response of a system to perturbations: the more robust the system, the less influence these perturbations have on its output. There are many factors that could influence the robustness of the xenograft models used in these experiments: batch effects on the efficacy of the drugs used; changes in the properties of cell lines over time; the strains of the mice used, and also their sex; factors related to microbiome and chow; circadian effects; temperature; and the antimicrobials that might be used in certain facilities.
It is also possible that Sirota et al. paid too much attention to statistical significance and P values and not enough attention to the actual size of the effect being investigated (Motulsky, 2014). Indeed, the actual tumor sizes observed by Sirota et al. were not diminished by cimetidine at early time points, and the reduction in tumor size at the last time point (the only time point at which the reduction was statistically significant) was only about 30%. This moderate effect size might have contributed to the fact that the effects seen in the replication were similar to those in the original experiments, but not statistically significant (although, as mentioned above, combining data from the two studies did give significant results).
There have been growing concerns about irreproducibility in the biological literature in recent years (Begley and Ellis, 2012; Errington et al., 2014; Baker, 2016). A powerful lesson to emerge from this Replication Study is that reproducibility is not black and white because, like many areas of research, cancer biology is nuanced and inherently complex.
Common misconceptions about data analysis and statisticsJournal of Pharmacology and Experimental Therapeutics 351:200–205.https://doi.org/10.1124/jpet.114.219170
Discovery and preclinical validation of drug indications using compendia of public gene expression dataScience Translational Medicine 3:96ra77.https://doi.org/10.1126/scitranslmed.3001318
Downloads (link to download the article as PDF)
Download citations (links to download the citations from this article in formats compatible with various reference manager tools)
Open citations (links to open the citations from this article in various online reference manager services)
In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper “Discovery and Preclinical Validation of Drug Indications Using Compendia of Public Gene Expression Data“ (Sirota et al., 2011). Here we report the results of those experiments. We found that cimetidine treatment in a xenograft model using A549 lung adenocarcinoma cells resulted in decreased tumor volume compared to vehicle control; however, while the effect was in the same direction as the original study (Figure 4C; Sirota et al., 2011), it was not statistically significant. Cimetidine treatment in a xenograft model using ACHN renal cell carcinoma cells did not differ from vehicle control treatment, similar to the original study (Supplemental Figure 1; Sirota et al., 2011). Doxorubicin treatment in a xenograft model using A549 lung adenocarcinoma cells did not result in a statistically significant difference compared to vehicle control despite tumor volume being reduced to levels similar to those reported in the original study (Figure 4C; Sirota et al., 2011). Finally, we report a random effects meta-analysis for each result. These meta-analyses show that the inhibition of A549 derived tumors by cimetidine resulted in a statistically significant effect, as did the inhibition of A549 derived tumors by doxorubicin. The effect of cimetidine on ACHN derived tumors was not statistically significant, as predicted.