Point of View: Why clinical translation cannot succeed without failure
- Carnegie Mellon University, United States
- McGill University, Canada
Abstract
The high rates of attrition that occur in drug development are widely regarded as problematic, but the failure of well-designed studies benefits both researchers and healthcare systems by, for example, generating evidence about disease theories and demonstrating the limits of proven drugs. A wider recognition of these benefits will help the biomedical research enterprise to take full advantage of all the information generated during the drug development process.
Main text
For every hundred drugs introduced into clinical development, approximately 90 are never approved for clinical use, and success rates for secondary applications of licensed drugs are not much better (Hay et al., 2014). Many commentators view high rates of attrition—defined as a failure to show that a promising drug is useful—as symptomatic of inefficient drug development. Some attribute inefficiencies to flawed preclinical or clinical study design, others to regulatory impediments, and still others to deficits in scientific techniques (Kola and Landis, 2004; Collins, 2011). Attrition is therefore viewed as a burden for the research enterprise and, since development costs are reflected in the price of new pharmaceuticals, for healthcare systems too.
However, given the incompleteness of our knowledge of the relevant disease processes and pharmacology, attrition is actually a necessary part of a rational and efficient approach to building a robust understanding of the diseases we are trying to treat. Attrition is also crucial for refining how this knowledge is applied in the clinic. Indeed, recognizing that unsuccessful translation trajectories can contribute to the viability of research and healthcare systems, and understanding more about this process, should lead to improvements in both systems.
How failure promotes translation
First, although pharmaceutical development is aimed at producing new drugs, the most valuable product of translation efforts is information about disease and drug mechanisms. This information is valuable because it informs drug development and it guides clinical practice (Kimmelman and London, 2015). For example, theories about amyloid metabolism have guided the selection of drugs and populations for the testing of Alzheimer's treatments for the last decade. About a third of all oncology prescriptions are off-label (Conti et al., 2013), and theories of tumor physiology help clinicians to extend findings from trials in order to devise treatments for patients who might not have met the initial criteria for treatment with the drug. However, our understanding of disease causation and pharmacology are incomplete, and many translational trajectories falter because the theoretical understandings guiding the trials are incorrect. (This is especially true when failures occur in phase 2 trials.) Even when disease and drug models build on robust basic science and preclinical testing, in vivo studies in humans represent the only way of confirming or discrediting emerging theories.
Second, efficient methods for generating this information limit the extent to which early signs of promise can be vindicated in the later phases of the translation process. To see why this is so, consider that drug development activities naturally divide between exploratory investigations aimed at identifying intervention candidates, and confirmatory investigations aimed at demonstrating their clinical utility (Kimmelman et al., 2014). Identifying promising interventions is akin to exploring a vast, multidimensional landscape of agents, doses, disease indications and treatment schedules. The methods used to explore this landscape (which happens during preclinical research and early-phase trials) often rely on small sample sizes and/or surrogate endpoints. This allows large areas of the landscape to be explored quickly and at relatively low cost. However, economy and speed come at a cost, since small and less rigorous studies tend to produce more false positives (i.e., studies that show spurious clinical promise due to bias or random variation) (Button et al., 2013).
Negative results advance drug discovery by helping to improve our understanding of both diseases and treatments.
Illustration: Claudia Stocker, vividbiology.com.
Third, the rational desire to allocate resources to interventions that show the greatest promise entails a certain amount of attrition. In particular, base rates for discovering truly effective interventions are likely to be low in areas where our knowledge of disease process, mechanism and pharmacology are underdeveloped. As is well known in diagnosis, when base rates are low, false positive tests due to random variation are frequent, even if tests are sensitive. Similarly, most promising results in early-phase studies in novel areas will prove to be false positives. Drug development thus confronts the problem of ‘winners curse’ (Button et al., 2013). This refers to the tendency of randomly large effects to regress toward a mean on subsequent testing. Consequently, large treatment effects in phase 1 studies will tend to diminish on testing in phase 2, and similarly for large effects seen in phase 2 trials. Several studies have shown such statistical tendencies in clinical research (Ioannidis et al., 2001). Winner's curse is the result of an appropriate type of experimenter bias, whereby researchers follow up on treatments that show large effects and set aside treatments that show smaller effects.
Fourth, ethical constraints on drug trials mean that new interventions will frequently fail to prove superior to standard treatments. This happens because enrolling patients with serious illnesses into randomized trials is only ethical where there is genuine and evidence-based uncertainty within the expert clinical community about the comparative advantage of the interventions in a trial. This condition, which is known as clinical equipoise (London, 2007), serves two functions. First, it ensures that patients can enter trials knowing they are receiving competent care (Freedman, 1987). Second, it promotes efficiency because it ensures that randomized trials are conducted only where there is genuine uncertainty about the value of a new treatment. If the uncertainty in the expert community is based on accurate evidence, novel interventions should prove superior to standard treatments as often as standard treatments prove superior to novel interventions. Indeed, studies have shown such patterns (Djulbegovic et al., 2013), suggesting that early-phase research provides a reliable basis for designing randomized trials, and that the clinical research enterprise adequately protects the welfare of subjects.
Finally, even when drugs with high clinical promise are identified, the process of refining our understanding of how to use them requires some level of failure. Drugs are only clinically useful insofar as clinicians know which dose, schedule, timing and diagnostic eligibility criteria to apply at the bedside. For example, some drugs will be ineffective for patients below a certain diagnostic score and/or toxic for patients who show a higher value (Schmidt, 2011; Kimmelman, 2012). These limits on the clinical utility of an intervention often cannot be defined without testing regimens that prove to be outside the desired window of activity. But without such failures and negative results, clinicians lack the knowledge of how far they can extend the application of a new drug while preserving its desirable effects. Consequently, efficient healthcare—even for successfully translated drugs—depends on testing regimens that fail, but that help to define the window within which a drug can be used safely and effectively.
Learning from failure
Obviously, some failures—such as studies that fail to recruit a meaningful sample, studies that suffer fatal design flaws, or studies that fail to deal with random variation and bias (Yusuf et al., 1984)—are unmitigated failures because they are not likely to deliver any useful information. Nevertheless, the foregoing analysis suggests that many unsuccessful translation trajectories are a product of efficient, moral and rational research efforts rather than inattention to experimental design. Well-designed studies that produce negative results generate information necessary to improve our understanding of causal processes related to disease and treatment, thereby advancing both drug discovery and the efficient application of licensed interventions at the bedside.
Our analysis also points to opportunities for improving the translation process. First, the drug development enterprise needs more effective ways of capturing and utilizing the vast amounts of information it generates. Trials do more than test new interventions; they also deliver evidence on pathophysiological theories and the limits of validated treatments. Most of this information is inaccessible to the broader research community due to non-publication and poor reporting. For example, one of us (JK) and co-workers recently looked at drugs that reached phase 3 trials but were never successfully licensed; results were published for only 37% of trials for these failed drugs (Hakala et al., 2015). Another study showed that only 17% of healthy volunteer phase 1 trials are published (Decullier et al., 2009). Even where trials are published, information probing pathophysiological theories is often withheld. Less than 40% of pharmacodynamics analysis results—which probe pathophysiological theories—within cancer trials are published in full, and what is reported is often difficult to interpret due to incomplete description of methods (Freeman and Kimmelman, 2012). More needs to be done to demand the publication of results from early-phase research, data from trials of products that are not licensed, and data from pharmacodynamics studies conducted within larger research studies.
Companies tend not to publish negative or inconclusive studies or secondary analyses for a number of reasons: such findings can be exploited by competing drug companies; they can complicate a clean narrative for investors; and they can diminish off-label use of a licensed drug. Similar pressures may deter academics from pursuing publication as well. Various private (Green et al., 2015; Peart, 2015) and public initiatives (Collins and Tabak, 2014) have been established to increase the transparency of trials and the release of data: however, given the volume of data that is not captured and the incentives for companies not to disclose certain data, these efforts are not enough. Policy-makers and funding bodies should pursue measures that promote unbiased and prompt publication for unsuccessful translation trajectories. Public funding agencies might also consider withholding the last installments of grants until all the relevant data have been made public, or they might offer small grants for publishing negative results. When hosting privately funded trials, academic medical centers could insist on using contracts that commit sponsors to sharing full datasets with local investigators, and that allow researchers to publish complete datasets when private sponsors do not. Lawmakers might create policies—like extending the period where drug companies can prevent competitors from using their trial data in licensing applications—that provide a financial incentive for publishing trials submitted to drug regulators for new drugs.
Second, researchers and healthcare professionals should better exploit the information generated by unsuccessful translation trajectories. Researchers should pay more attention to negative and inconclusive findings when they are published: researchers tend to cite positive findings more frequently than negative findings, even when there are no obvious differences in the quality of the studies (Greenberg, 2009), and many clinical investigators perceive negative studies as uninformative (Smyth et al., 2011). Yet without comprehensive data about all of the investigations motivated by a common theoretical framework, it is difficult to distinguish unsuccessful translation trajectories that reflect shortcomings in the study from those that are informative about strategies for turning interventions into therapies. Moreover, when studied together, positive and negative studies for a given drug can help clinicians to refine their understanding of the drug and/or the disease it is intended to treat. Journal editors should encourage researchers to include negative and inconclusive findings in their papers and to explain what these findings mean for our understanding of the drug and/or the disease. Ethics committees can also encourage such a shift by demanding that trial brochures discuss any relevant negative and inconclusive findings that have been made public: if nothing else, this will help to distinguish proposed interventions from similar ones that have failed, and may also shed light on why the trial is being conducted on certain subpopulations.
Researchers and healthcare professionals should better exploit the information generated by unsuccessful translation trajectories.
Third, policy makers and researchers should be cautious about policies that aim to accelerate translation by granting licenses for products early in the trial process. Recent Japanese legislation will allow the commercialization of cell-based interventions once their safety has been established in in early-phase studies (Cyranoski, 2013), and a drug for the treatment of lung cancer was recently approved for use in the US after a single phase 1 study (Khozin et al., 2015). As our analysis indicates, such policies harbor liabilities. Large effects in early development are likely to regress in later trials, and truncating the development process means that we will not gather useful information about optimal dosing, diagnostic cut-points, or which subpopulations to use a drug in. Early licensing slows the collection of this evidence by discouraging patient recruitment into trials (Rettig, 2007) and by diminishing incentives for other companies to run trials. The result of such reforms may be an increase in the number of patients exposed to harmful or ineffective interventions, and/or longer timelines for the development of truly effective treatments. Such reforms are also likely to shift the cost of gathering this information from drug developers to patients and healthcare systems.
Until our knowledge of the underlying processes that cause disease improves, the best approach for increasing the efficiency of translation is to maximize the information gained from patient exposure. This means better reporting on carefully planned trials, greater uptake of evidence from unsuccessful translation trajectories, and using failure in drug development to improve the search for or the use of new interventions. Without explicit recognition of the statistical, scientific and ethical factors that limit the rate of drug development, efforts to make translation more efficient risk shifting costs and burdens without producing better options for patients and healthcare systems.
References
-
Power failure: why small sample size undermines the reliability of neuroscienceNature Reviews Neuroscience 14:365–378.https://doi.org/10.1038/nrn3475
-
Reengineering translational science: the time is rightScience Translational Medicine 3:90cm17.https://doi.org/10.1126/scitranslmed.3002747
-
Clinical development success rates for investigational drugsNature Biotechnology 32:40–51.https://doi.org/10.1038/nbt.2786
-
Replication validity of genetic association studiesNature Genetics 29:306–309.https://doi.org/10.1038/ng749
-
The structure of clinical translation: efficiency, information, and ethicsHastings Center Report 5:27–39.https://doi.org/10.1002/hast.433
-
Can the pharmaceutical industry reduce attrition rates?Nature Reviews Drug Discovery 3:711–715.https://doi.org/10.1038/nrd1470
-
BookClinical equipoise: foundational requirement or fundamental errorIn: Steinbock B, editors. The Oxford Handbook of Bioethics. Oxford: Oxford University Press. pp. 571–596.
-
Yale YODA project announces first availability of medical device trial dataYale News, Available at: http://news.yale.edu/2015/01/14/yale-yoda-project-announces-first-availability-medical-device-trial-data.
-
BookFalse Hope: Bone Marrow Transplantation for Breast CancerOxford: Oxford University Press.
-
How do you tell whether a breast cancer is HER2 positive? Ongoing studies keep debate in high gearJournal of the National Cancer Institute 103:87–89.https://doi.org/10.1093/jnci/djq557
-
Why do we need some large, simple randomized trials?Statistics in Medicine 3:409–422.https://doi.org/10.1002/sim.4780030421
Article and author information
Author details
Publication history
Copyright
© 2015, London and Kimmelman
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics
-
- 2,311
- views
-
- 329
- downloads
-
- 21
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Point of View: Why clinical translation cannot succeed without failure
eLife 4:e12844.
https://doi.org/10.7554/eLife.12844
Further reading
-
- Medicine
Background:
Clonal hematopoiesis of indeterminate potential (CHIP) was initially linked to a twofold increase in atherothrombotic events. However, recent investigations have revealed a more nuanced picture, suggesting that CHIP may confer only a modest rise in myocardial infarction (MI) risk. This observed lower risk might be influenced by yet unidentified factors that modulate the pathological effects of CHIP. Mosaic loss of the Y chromosome (mLOY), a common marker of clonal hematopoiesis in men, has emerged as a potential candidate for modulating cardiovascular risk associated with CHIP. In this study, we aimed to ascertain the risk linked to each somatic mutation or mLOY and explore whether mLOY could exert an influence on the cardiovascular risk associated with CHIP.
Methods:
We conducted an examination for the presence of CHIP and mLOY using targeted high-throughput sequencing and digital PCR in a cohort of 446 individuals. Among them, 149 patients from the CHAth study had experienced a first MI at the time of inclusion (MI(+) subjects), while 297 individuals from the Three-City cohort had no history of cardiovascular events (CVE) at the time of inclusion (MI(-) subjects). All subjects underwent thorough cardiovascular phenotyping, including a direct assessment of atherosclerotic burden. Our investigation aimed to determine whether mLOY could modulate inflammation, atherosclerosis burden, and atherothrombotic risk associated with CHIP.
Results:
CHIP and mLOY were detected with a substantial prevalence (45.1% and 37.7%, respectively), and their occurrence was similar between MI(+) and MI(-) subjects. Notably, nearly 40% of CHIP(+) male subjects also exhibited mLOY. Interestingly, neither CHIP nor mLOY independently resulted in significant increases in plasma hs-CRP levels, atherosclerotic burden, or MI incidence. Moreover, mLOY did not amplify or diminish inflammation, atherosclerosis, or MI incidence among CHIP(+) male subjects. Conversely, in MI(-) male subjects, CHIP heightened the risk of MI over a 5 y period, particularly in those lacking mLOY.
Conclusions:
Our study highlights the high prevalence of CHIP and mLOY in elderly individuals. Importantly, our results demonstrate that neither CHIP nor mLOY in isolation substantially contributes to inflammation, atherosclerosis, or MI incidence. Furthermore, we find that mLOY does not exert a significant influence on the modulation of inflammation, atherosclerosis burden, or atherothrombotic risk associated with CHIP. However, CHIP may accelerate the occurrence of MI, especially when unaccompanied by mLOY. These findings underscore the complexity of the interplay between CHIP, mLOY, and cardiovascular risk, suggesting that large-scale studies with thousands more patients may be necessary to elucidate subtle correlations.
Funding:
This study was supported by the Fondation Cœur & Recherche (the Société Française de Cardiologie), the Fédération Française de Cardiologie, ERA-CVD (« CHEMICAL » consortium, JTC 2019) and the Fondation Université de Bordeaux. The laboratory of Hematology of the University Hospital of Bordeaux benefitted of a convention with the Nouvelle Aquitaine Region (2018-1R30113-8473520) for the acquisition of the Nextseq 550Dx sequencer used in this study.
Clinical trial number:
-
- Biochemistry and Chemical Biology
- Medicine
HER2 overexpression significantly contributes to the aggressive nature and recurrent patterns observed in various solid tumors, notably gastric cancers. Trastuzumab, HER2-targeting monoclonal antibody drug, has shown considerable clinical success; however, readily emerging drug resistance emphasizes the pressing need for improved interventions in HER2-overexpressing cancers. To address this, we proposed targeting the protein-protein interaction (PPI) between ELF3 and MED23 as an alternative therapeutic approach to trastuzumab. In this study, we synthesized a total of 26 compounds consisting of 10 chalcones, 7 pyrazoline acetyl, and 9 pyrazoline propionyl derivatives, and evaluated their biological activity as potential ELF3-MED23 PPI inhibitors. Upon systematic analysis, candidate compound 10 was selected due to its potency in downregulating reporter gene activity of ERBB2 promoter confirmed by SEAP activity and its effect on HER2 protein and mRNA levels. Compound 10 effectively disrupted the binding interface between the ELF3 TAD domain and the 391–582 amino acid region of MED23, resulting in successful inhibition of the ELF3-MED23 PPI. This intervention led to a substantial reduction in HER2 levels and its downstream signals in the HER2-positive gastric cancer cell line. Subsequently, compound 10 induced significant apoptosis and anti-proliferative effects, demonstrating superior in vitro and in vivo anticancer activity overall. We found that the anticancer activity of compound 10 was not only restricted to trastuzumab-sensitive cases, but was also valid for trastuzumab-refractory clones. This suggests its potential as a viable therapeutic option for trastuzumab-resistant gastric cancers. In summary, compound 10 could be a novel alternative therapeutic strategy for HER2-overexpressing cancers, overcoming the limitations of trastuzumab.
