Robust and Efficient Assessment of Potency (REAP) as a quantitative tool for doseresponse curve estimation
Abstract
The medianeffect equation has been widely used to describe the doseresponse relationship and identify compounds that activate or inhibit specific disease targets in contemporary drug discovery. However, the experimental data often contain extreme responses, which may significantly impair the estimation accuracy and impede valid quantitative assessment in the standard estimation procedure. To improve the quantitative estimation of the doseresponse relationship, we introduce a novel approach based on robust beta regression. Substantive simulation studies under various scenarios demonstrate solid evidence that the proposed approach consistently provides robust estimation for the medianeffect equation, particularly when there are extreme outcome observations. Moreover, simulation studies illustrate that the proposed approach also provides a narrower confidence interval, suggesting a higher power in statistical testing. Finally, to efficiently and conveniently perform common lab data analyses, we develop a freely accessible webbased analytic tool to facilitate the quantitative implementation of the proposed approach for the scientific community.
Editor's evaluation
This article proposes methodology and accompanying software for robustly fitting doseresponse curves where response is a number between 0 and 1. When response is transformed using the common logistic transformation, values close to 0 or 1 become large in magnitude, unduly influencing the fitted curve after backtransformation and introducing bias in the estimate of certain parameters. As demonstrated through simulation and application to real data, the proposed approach, called Robust and Efficient Assessment of Potency, is less perturbed by these extreme measurements.
https://doi.org/10.7554/eLife.78634.sa0eLife digest
Finding a new drug which is both safe and efficient is an expensive and timeconsuming endeavour. In particular, establishing the ‘doseeffect relationship’ – how beneficial a drug is at different dosages – can be challenging. Predicting this curve requires gathering experimental data by exposing and recording how cells respond to various levels of the drug. However, extreme values are often observed at low and high dosages, potentially introducing errors that are hard to correct in the prediction process. Yet, these extreme observations are sometimes genuine so researchers cannot just ignore them.
To improve doseeffect estimation, Zhou, Liu, Fang et al. developed a new generalpurpose approach. It uses advanced statistical modelling to account for extremes in lab data. This strategy outperformed other methods when dealing with these observations while also providing higher efficiency in data analysis with more uniform data in experiments.
To facilitate implementation, Zhou, Liu, Fang et al. set up a userfriendly tool baptized ‘REAP’; this free online resource allows scientists without advanced statistical experience to harness the new approach and to perform doseeffect analysis more easily and accurately. This could boost research across many different disciplines that examine the effects of chemicals on cells.
Introduction
The medianeffect equation is a unified theory in medicine to describe the doseresponse relationship and identify agents or their combinations that activate or inhibit specific disease targets (Chou, 2006). It is a fundamental method established based on the pharmacological principle of massaction law (Chou, 1976). As the common link for many biomedical systems, it has been used extensively to analyze in vitro experimental data and evaluate the potency of related drugs (Chou and Talalay, 1984; Chou and Rideout, 1991; Greco et al., 1995; Lee and Kong, 2009).
In practice, the medianeffect equation can be estimated for drug efficacy or pathway inhibition from normalized data generated from experimental studies. Without knowing the true doseeffect curve during the experimental design and data collection, it is common to observe extreme values of (un)affected cell fraction that is close to the response of either 0 or 100% in the analytic dataset. Quantitatively, it poses a special analytic challenge to estimate the medianeffect question in practice. The standard estimation approach, often based on a linear regression model after a logit transformation (Roell et al., 2017; Gadagkar and Call, 2015), could suffer badly from poor estimation in such situations. Figure 1 illustrates a preliminary example in that the standard approach is deficient in describing the median effect curve with a perturbation in one extreme data point. The variation in real experimental data, mostly caused by unavoidable measurement error, often at a much larger degree, therefore challenges the reliability of result presentation and interpretation for many drug assessment studies.
Additionally, the modeling strategy of deleting extreme values may not be feasible in many situations (Solzin et al., 2020). For example, a meaningful drug concentration could consist of high inhibition (>90%) or low cell viability (<10%) in cancer research. It is not logical to ignore extreme observations when they are indeed biologically relevant for the target effect, not even to mention an associated loss of power and accuracy by leaving fewer data points for estimation. As illustrated in Figure 2, deleting the extreme values couldn’t eliminate the estimation bias, but only impaired the efficiency of interval estimation with wider nominal 95% confidence intervals (C.I.) and harmed the estimation accuracy with worse coverage probabilities.
Furthermore, it is dubious to apply the constant error variance, a default assumption in standard linear regression modeling, in doseresponse estimation. As an assumption can be examined with repeated measures, many doseresponse data have indicated either a constant variance before logit transformation or a positive correlation with drug dose. It is incongruous to apply linear regression if the assumption is violated due to error heteroscedasticity (Schmidheiny, 2009; Williams et al., 2007). Therefore, it is essential to develop a robust quantitative approach to estimating the medianeffect equation.
Here, we introduce a novel approach to improving the quantitative assessment of doseresponse relationship and drug potency, together with a userfriendly webbased analytic tool to facilitate the implementation. The proposed method to estimate the medianeffect equation is established in the robust beta regression framework, which not only takes the beta law to account for nonnormality and heteroskedasticity (Ferrari and CribariNeto, 2004), but also minimizes the average density power divergence (DPD) using a tuning parameter (Ghosh, 2019). We apply a datadriven approach to optimizing the tuning parameter, which further compensates for the lack of robustness against outliers. In the simulation studies, we compare the robust beta regression framework with linear regression models either in the standard normal distribution error, or in the heavytailed t distribution error with 3 degrees of freedom hopefully to downweigh the influence of extreme observations. Results from simulation studies under various scenarios confirm that the proposed approach consistently gives robust estimation for the medianeffect equation. Particularly, we examine two important measures for drug binding affinity: the Hill coefficient, which signifies the sigmoidicity of the curve, and the overall effect, indicated by dose concentration for a specified (e.g. 50%) response (Shen et al., 2008; Sampah et al., 2011). When there are extreme outcome observations, the improvement of robust beta regression in estimation accuracy could be substantial. Moreover, simulation studies further illustrate that the proposed approach provides a narrower confidence interval, which in turn suggests a higher efficiency to achieve better power in statistical testing even without acquiring additional experimental data. Illustrative examples using realworld data for cancer research and SARSCoV2 treatment are provided. The analyses are implemented using the freely accessible webbased application REAP, developed based on the Shiny package of R language, with which research scientists could conveniently upload their drug experiment dataset and perform the data analysis.
Results
REAP Shiny App
We developed a userfriendly analytic tool, coined ‘REAP’ (Robust and Efficient Assessment of Potency), for convenient application of the robust doseresponse estimation to realworld data analysis. It is established in an agile modeling framework under the parameterization of the beta law to describe a continuous response variable with values in a standard unit interval (0.1). We further exploited a robust estimation method of the beta regression, named the minimum density power divergence estimators (MDPDE) (Ghosh, 2019), for doseresponse estimation, with the tuning parameter optimized by a datadriven method (Ribeiro and Ferrari, 2020). The technical details are provided in the Materials and methods.
REAP presents a straightforward analytic environment for robust estimation of doseresponse curve and assessment of key statistics, including implementation of statistical comparisons and delivery of customized output for graphic presentation (Figure 3). The doseresponse curve is a timehonored tool to convey the pharmacological activity of a compound. Through doseresponse curves, we can compare the relative activity of a compound on different assays or the sensitivity of different compounds on an assay. REAP aims to make this job simple, estimation efficient, and results robust.
There are three sections in REAP: Introduction, Dataset and Output. Users can have both overview and instruction of REAP in the Introduction. Dataset is uploaded in the Dataset section. The input dataset is mandated to be in a csv file format and contains three columns of data respectively for drug concentration, response effect and group name, in a specific order. It is recommended that users normalize the response variable to the range of (0,1) by themselves. Otherwise, REAP automatically will truncate the values exceeding the boundaries to (0,1) using a truncation algorithm (see Appendix 1  Truncation Strategy). In the Output section, it generates a doseresponse plot, along with tabulation for effect and model estimations. A special feature of REAP is that it conveniently allows the users to specify the target effect level, rather than fixed at the common median effect (i.e., 50%), in dose estimation. We also enable hypothesis testing for comparisons of effect estimations, slopes and models (i.e. comparing both intercepts and slopes; see Materials and methods). By default, the xaxis of the doseresponse plot is logscaled. In the plot, users can choose to add mean values and sample standard deviations for data points under the same agent and dose level. Both plots and estimation tables are downloadable on REAP to plug in presentations and manuscripts for result dissemination.
The opensourced REAP is freely available and accessible at https://xinyingfang.shinyapps.io/REAP/. We demonstrated it in two realworld examples, after presenting the simulation results, to illustrate the functionality of REAP.
Simulations
We conducted simulation studies to investigate the robust beta regression model, in comparison to linear regression models with data transformation, either under a normal distribution error (implemented with R package ‘stats’) or a heavytailed t distribution error with 3 degrees of freedom (implemented with R package ‘heavy’), to characterize the medianeffect equation under different scenarios. The model assessment is established based on both the point estimation and interval estimation derived from each method. Details on the simulation setting are described in the Appendix 1  Data simulating process.
With data simulated using normal error terms, the robust beta regression provides sensible estimation of IC_{50}, IC_{90}, ${\beta}_{1}$, and ${\beta}_{0}$ from medianeffect equation (Figure 4, Appendix 1—table 1). Particularly, when there are extreme outcome observations, the robust beta regression manages much lower bias and rootmeansquare error (RMSE) for point estimates and better coverage probability for interval estimates than the linear regression model with normal distribution error. For data without extreme values, their performance is comparable in bias, RMSE and coverage probability, but the linear regression model has much wider 95% CIs (Figure 4). Indeed, the wider 95% CIs occur across all the scenarios, indicating higher estimation efficiency of the robust beta regression approach. In contrast, the heavytailed linear regression model demonstrates improved bias and RMSE in point estimation from the standard linear regression, but the nominal 95% CIs are significantly underestimated with coverage probability below 50% in most cases (Appendix 1—table 1). Therefore, the heavytailed linear regression model, although sometimes provides good point estimations, cannot maintain consistently robust and statistically efficient estimations. Overall, the robust beta regression model is the most robust and stable in estimating the medianeffect equation with reliable performance in both point estimations and 95% CI coverage probabilities.
In parallel, similar results are obtained consistently with data simulated using beta error terms, which induces heteroscedasticity (smaller variation on the two ends and bigger in the middle) at different dose levels (Appendix 1—figure 1, Appendix 1—table 2). All the results above demonstrate the sensitivity of regression models in dealing with datasets including extreme values. In addition, the result comparisons between the sevendose set and the sixdose set with the largest or smallest dose eliminated display the potential worse influence of deleting extreme values directly in modeling doseresponse using linear regression, which further notarizes the robustness and efficiency of the proposed robust beta regression.
Overall, the simulation study suggests that the robust beta regression model produces wellcalibrated doseresponse curves while being more robust and powerful than the standard regression model and the heavytailed linear regression model in estimating the median effect equation.
Bcell lymphoma data
The first example of REAP application is doseresponse curve estimation of the same agent under different cell lines. The data was originally from a study on using a drug called auranofin in treating Bcell lymphomas such as relapsed or refractory mantle cell lymphoma (MCL) (Wang et al., 2019). As an FDAapproved treatment of rheumatoid arthritis, auranofin targets thioredoxin reductase1 (Txnrd1), and was repurposed as a potential antitumor drug to effectively induce DNA damage, reactive oxygen species (ROS) production, cell growth inhibition, and apoptosis in aggressive Bcell lymphomas, especially in TP53mutated or PTENdeleted lymphomas.
In the experiment, the effect of auranofin was evaluated in six MCL cell lines (Z138, JVM2, Mino, Maver1, Jeko1, and JekoR) with auranofin in concentrations ranging from 0 to 5 μM for 72 hr and tested cell viability using a luminescent assay. The interval bars of observed doseresponse in Figure 5 show that the sample variance of error from repeated measurements decreased with the increase of auranofin concentrations. To account for the heteroscedasticity and asymmetry in the variance, we enable a dosedependent precision (proportional to inverse variance) in REAP, adding $\mathrm{log}\left(dose\right)$ as an additional regressor for the precision parameter. Figure 5 shows the fitted doseresponse curves with the dosedependent precision. The test for homogeneity (pvalue <0.0001) suggests distinct doseresponse between cell lines. The estimation of intercepts, hill coefficients and pairwise comparisons of IC_{50} estimations are provided in Appendix 1—table 3.
SARSCoV2 data
The second example is on the doseresponse curve estimation in antiviral drug development for coronavirus disease 2019 (COVID19). At the beginning of 2020, COVID19 broke out at an unprecedented pace internationally, but there were limited therapeutic options for treating this disease. Therefore, many compounds and their combinations were rapidly tested in vitro against the SARSCoV2 virus to identify potentially effective treatments and prioritize clinical investigation.
In the data (Bobrowski et al., 2021), the benchmark compound collection consists of five known antivirals, including remdesivir, E64d (aloxistatin), chloroquine, calpain Inhibitor IV and hydroxychloroquine. The in vitro experiment was performed using the same biological batch of SARSCoV2 virus and conducted in biosafety level3. In the original publication (Bobrowski et al., 2021), the doseresponse curves were fitted by linear regression, which could yield inconclusive estimation (e.g. hydroxychloroquine in Figure 1G of Bobrowski et al., 2021), while the estimated inhibition tends to exceed 1 when concentration is larger than 10 µM. REAP gives reasonable estimation for the doseresponse curves (Figure 6). The hypothesis testing results show that at least one slope estimation is different from other antivirals (pvalue = 0.0003) and at least one EC_{50} estimation is different from others (pvalue = 0.003). Calpain Inhibitor IV shows a higher potency than other agents including hydroxychloroquine (pvalue = 0.0038, Appendix 1—table 4).
Discussion
Quantifying the potency of a compelling substance is always a central topic in life sciences (Schindler, 2017). It is a vital component of research in pharmacology, but also prevalent in the fields of toxicology, environmental science, agrochemistry, and medicine, among many others. For instance, the description of doseresponse curves can provide the initial toxicological risk assessment (National Research Council, 2007), and guide in silico modeling of toxic doses to humans and the environment (Blaauboer et al., 2012). Based on proper identification of doseresponse relationship from in vitro assays, studies can successfully predict systemic toxicological effects in vivo without additional in silico modelling (Groothuis et al., 2015). Nevertheless, it necessitates accurate and reliable description of the doseresponse curve, which further demands robust and efficient modeling strategies to account for embedded variability in observed response and to derive solid inference with valid quantification of uncertainty.
The doseresponse estimation could be substantially biased by the standard regression modeling. In the illustrative example (Figure 1), the estimated IC_{50} dose indeed effects the 70% fraction of cell affected, while the estimated response at the true IC_{50} dose is only 22%. Such a large discrepancy is sourced by a small (<0.5%) single measurement error, which is common and inevitable in any regular in vivo experiment, but could engender a profound impact on the assessment of drug potency and determination of synergy in drug combinations. In addition, the modeling strategy of deleting those extreme values (e.g. Figure 2, or 6noL and 6 noS datasets in Figure 4 and Appendix 1—figure 1) is futile to improve the poor performance of standard regression model, but may further impair the estimation efficiency and accuracy. In general, it fails to reduce bias but only introduces larger uncertainty in estimation of dose concentration, especially at extreme responses (e.g. IC_{90}). On the other hand, a heavytailed error distribution may help to stabilize the point estimation, but the interval estimation could be largely underestimated with poor coverage probabilities.
We develop REAP for assessment of drug potency to address concerns in this regard. It has substantial advantages over existing methods by reducing the impact of random errors due to implicit variations in the experimental data. To our best knowledge, it is also for the first time that beta regression is introduced to doseresponse estimation. The underlying modified robust beta regression model estimated by the datadriven tuning parameter is resilient to estimation bias caused by extreme observations, which is a routinely encountered situation for deficient doseresponse estimation using the standard estimation approach. The proposed approach is also efficient in quantitative characterization of doseresponse curves with narrower confidence intervals for key estimators. Furthermore, REAP can simultaneously model the data heterogeneity with a dosedependent precision component (Figure 5). It is simply different from other doseresponse methods, in which a vector of weights have to be (possibly mis)specified externally. REAP is an opensource and userfriendly platform, developed for diverse noncomputational scientists for handson wetlaboratory data analysis in regular use, and can be hosted within R shiny environment under Windows, Linux, and Mac systems or deployed in Docker available as a web server.
Our work potentially can be useful in applications of drug screening. The proposed method and the developed REAP App allow for the robust and efficient estimation and accounting for outliers as well, making it fitted particularly in a highthroughput setting. As the result of a complex and dynamic cascade of events, exposure time is another important factor ultimately affecting the doseresponse. For in vitro experiments measured at different time points in a choice of celllines and expressed by a variety of assays (Byrne and Maher, 2019), the proposed modeling framework can be naturally extended to model timedependent cytotoxicity while controlling for fixed or random effects. Furthermore, the application of robust and efficient doseresponse estimation can be integrated into methods to identify drug interaction effect (Lee and Kong, 2009; Lee et al., 2007). There is a venerable history that multiagent combination therapies demonstrate great advantages in improving therapeutic efficacy and revolutionize patient outcomes in a wide range of diseases. Robust and efficient estimation of the doseresponse curve would be crucial in investigation of adequate drug combinations.
The developed method has limitations. We presented a model of the median effect equation for doseresponse curve estimation based on mass action law. While in specific scenarios other laws may be considered more suitable to describe the biomedical systems, the current modeling framework can be naturally adapted for other doseresponse functions like probit (via cumulative normal distribution) and Weibull model (Christensen, 1984), or any other continuous distribution functions. In addition, the medianeffect equation to characterize pharmacological activity assumes the compound can affect all the cells. From a quantitative perspective, a compound that cannot reach high binding affinity will yield an overconservative estimation for median effective dose of a drug. However, in comparison to the sensitivity of different compounds in an assay, it is not harmful because the less effective compounds will be more easily identified. If it is a concern that the maximal effects of candidate compounds are different and the aim is to accurately model the doseresponse curve, the Emax model could be a better choice (Lee et al., 2010). Furthermore, the robust beta regression approach in REAP cannot handle values equal or less than 0, or equal or greater than 1. Thus, we developed a sequential data truncation algorithm in REAP to overcome the limitation of the conventional transformation (y * (n−1)+0.5) / n, which could be too rough in doseresponse curve estimation particularly when the sample size n for each group is relatively small. Although empirically we have validated it using simulated data, the algorithm could be improved by future work to retain information more efficiently.
In summary, a good modeling strategy must effectively characterize the nature of the observed doseresponse pattern (Lyles et al., 2008). Rapid advances in novel drug development and considerable deficiency in modeling data with extreme values offer an appealing opportunity for nextgeneration quantitative approaches. While many aspects of the techniques discussed here fit in the statistical framework of robust beta regression, our aim is to clearly apply and rigorously customize the analytic considerations, to reduce bias and ameliorate efficiency in routinely used doseeffect estimation, and to facilitate the convenient analytic implementation and dissemination. Experimental conditions and candidate drug potency could inevitably vary in practice, but REAP provides a great tolerance for points with extreme values, solid support for accurate and efficient doseresponse curve estimation, and useful reference to the future development of methodology in drug investigation. Overall, we anticipate that our work will contribute more to quantitative analysis in assessment of drug potency in preclinical research.
Materials and methods
Medianeffect equation and doseresponse curve
Request a detailed protocolThe medianeffect equation describes a popular model of the doseresponse relationship based on the median effect principle of the mass action law in various biological systems (Chou, 1976). Assume ${f}_{a}$ and ${f}_{u}$ are the fractions of the system affected and unaffected by a drug concentration $d$. The medianeffect equation states that
where $m$ is the Hill coefficient signifying the sigmoidicity of the doseeffect curve and ${D}_{m}$ is the dose of a drug required to produce the median effect, which is analogous to the more familiar ${IC}_{50}$ (drug concentration that causes 50% of the maximum inhibitory effect), ${ED}_{50}$ (halfmaximum effective dose), or ${LD}_{50}$ (median lethal dose) values (Ghosh, 2019). For example, if an inhibitory substance is of interest, the parameter $m$ measures the cooperativity in the binding of multiple ligands to linked binding sites, and the parameter ${D}_{m}={IC}_{50}$ , defined by the concentration that causes 50% of the maximum inhibitory effect.
Given ${f}_{a}+{f}_{u}=1$, the medianeffect Equation 1 is equivalent to
where $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left(p\right)$ denotes the logit function $\mathrm{log}\frac{p}{1p}$ . The Equation 2 shows a loglinear relationship between the drug dose $d$ and its effect ${f}_{a}$ (or ${f}_{u}$ , if it is, for example, the % survival of interest) after a logit transformation. Because from a modeling perspective the identical strategy can be applied to model both ${f}_{a}$ and ${f}_{u}$ , for the effect on cell fraction $E$, we can rewrite Equation 2 to be:
where ${\beta}_{0}$ is the intercept and ${\beta}_{1}$ the slope of the response curve. A linear regression model (LRM) can be applied in the form of Equation 3 with a standard normal distribution error. In simulation studies, we also examine Equation 3 with a heavytailed tdistribution error, denoted by heavytailed linear regression model (HLRM).
In this presentation, the median effect dose
the Hill coefficient
and the doseresponse curve
where ${\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}}^{1}\left(x\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(x\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left(x\right)}$ is the inverselogit function.
Beta regression model for doseresponse curve estimation
Request a detailed protocolWe will review the beta regression model which for the first time will be applied in doseresponse estimation. The effect $E$ and the parameters $\beta =({\beta}_{0},{\beta}_{1})$ in Equation 3 cannot be directly observed, but they can be estimated using experimental data, in which the observed sample cell fraction $y$ produced by the drug dose $d$ is a random variable with mean $E$. It is clear that effective estimation must properly account for random variation and be based upon a model that not only matches the nature of the response variable, but adequately characterizes the observed doseresponse pattern (Lyles et al., 2008).
Among all the unknown quantities, the parameters $\beta $ could be first estimated and play a fundamental role in supporting the inference for others. In the standard estimation procedure based on linear regression, $\mathrm{logit}\left(y\right)=\mathrm{log}\frac{y}{1y}$ is regressed on $\mathrm{log}d$ to get the inference on parameters $\beta $. Subsequently, the doseresponse curve can be estimated by Equation 6, and $({D}_{m},m)$ can be derived based on Equations (4) and (5) for medianeffect Equation 2. Because the extreme values of $y$ close to 0 or 1 could yield very large values of $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\left(y\right)$ (approaching to $\infty $ or $+\infty $, respectively, if $y\to 0$ or 1), and induce significant bias in estimation of $\beta $, the accuracy of the estimated doseresponse curve and medianeffect equation is in question when there exist extreme values in the dataset.
The beta regression model describes a response variable $y$ with continuous values restricted to the open standard unit interval (Johnson et al., 1995; Simas et al., 2010). In a classic beta regression framework, the beta regression model uses a parameterization of the beta law that is indexed by the mean parameter μ, and the precision parameter $\varphi $ that controls the overall variation (Ferrari and CribariNeto, 2004). To model the doseresponse relationship for the cell fraction $E$, we assume that the response $y$ is a betadistributed random variable and its mean $\mu =E$ has the form of Equation 6, where $d$ is the dose producing effect $E$, ${\beta}_{1}$ and ${\beta}_{0}$ are the regression parameters. Estimation of regression parameters $\beta $ can be performed using maximum likelihood method to derive point estimate $\widehat{\beta}$ and covariance matrix $\Sigma $.
Beta regression is resistant to extreme values and provides reliable estimations (Figure 1). Compared with the standard approach, which applies a nonlinear transformation in the response for an approximation to the normal distribution, the beta density can take on a variety of shapes to account for nonnormality and skewness (Smithson and Verkuilen, 2006). In the presence of heteroskedasticity and asymmetry, two common problems frequently observed in limited range continuous response data, an empirical study showed that the beta regression provided the best estimation among several alternatives (Kieschnick and McCullough, 2016).
Robust beta regression model with MDPDE
Request a detailed protocolWe will present a modified robust beta regression approach in REAP implementation, which is established based on density power divergence for robust estimation (Ghosh, 2019), but further improved after we introduce a datadriven method to identify the optimal tuning parameter. The standard beta regression potentially could still be sensitive against outliers because its inference is based on the maximum likelihood estimation. Ghosh, 2019 developed the robust minimum density power divergence estimators (MDPDE) that address the problem by minimizing the average density power divergence (DPD)
between the empirical density $\widehat{g}$ and the beta model density function $g\equiv Beta\left(\mu \varphi ,\left(1\mu \right)\varphi \right)$ with ${\mu =\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}}^{1}\left({\beta}_{1}\mathrm{log}d+{\beta}_{0}\right)$ . $\alpha $ is a nonnegative tuning parameter, smoothly connecting the likelihood disparity (at $\alpha $ = 0) to the L_{2}Divergence (at $\alpha $ = 1). The parameter of interest $\beta $ is estimated by minimizing the DPD measure between ${g}_{i}$ and the density, ${\widehat{g}}_{i}$ ,
where $\mathit{\theta}={\left(\beta ,\varphi \right)}^{T}$. After mathematically simplifying Equation 8, (Ghosh, 2019), $\mathit{\theta}$ can be equivalently estimated by minimizing the objective function using the estimation equations:
where ${K}_{i,\alpha}\left(\mathit{\theta}\right)=\frac{B({\left(1+\alpha \right)\mu}_{i}\varphi ,\left(1+\alpha \right)\left(1{\mu}_{i}\right)\varphi \alpha )}{B{\left({\mu}_{i}\varphi ,\left(1{\mu}_{i}\right)\varphi \right)}^{\alpha}}$.
MDPDE improves the standard beta regression with the DPD measure and a fixed tuning parameter. The recommended α is around 0.3 to 0.4, but simply assigning a fixed α in [0.3, 0.4] is not applicable in many cases. Here we adopted a datadriven method (Ribeiro and Ferrari, 2020) to identify the optimal α. The search for the optimal α starts with a grid of α, a predefined α_{max} and grid size $\rho$, which generates a sequence of equally spaced $\{{\alpha}_{k}{\}}_{k=0}^{m}\text{}(0={\alpha}_{0}{\alpha}_{1}\cdot \cdot {\alpha}_{m}\le {\alpha}_{max})$. MDPDE calculates the corresponding θ and se(θ) with each α so that we get a vector of standardized estimates:
The standardized quadratic variations (SQV) are defined by:
We compare each $SQ{V}_{{\alpha}_{k}}$ with a predefined threshold $L\text{}\left(L0\right)$. If all ${\alpha}_{k}$ satisfy the stability condition of $SQ{V}_{{\alpha}_{k}}<L$, then the optimal $\alpha $ equals the minimal $\alpha $ in ${\alpha}_{k}$ . Otherwise, restart the search with a new grid of ${\alpha}_{k}$ . The new grid of the same size $p$ is picked from the sequence ${\left\{{\alpha}_{k}\right\}}_{k=0}^{m}$ starting from the largest ${\alpha}_{k}$ that fails the stability condition. Repeat searching until all ${\alpha}_{k}$ in the current grid satisfy the stability condition or ${\alpha}_{max}$ is reached. If the stability condition is satisfied before ${\alpha}_{max}$ is reached then optimal $\alpha $ equals the minimal value in the grid of ${\alpha}_{k}$ . If ${\alpha}_{max}$ is reached, then optimal α equals 0, which is equivalent to the maximum likelihood estimation. We denote this approach by robust beta regression model (BRM) in the simulation study.
Point estimate and its confidence interval for drug activity measurements
Request a detailed protocolThe objective of analysis is to characterize the doseresponse curves in equation (2) and quantify in vitro drug potency. Popular drug activity measurements include Hill coefficient $m$ and median effect dose ${D}_{m}$ . In some circumstances, other measurements such as instantaneous inhibitory potential (IIP), which directly quantifies the log decrease in singleround infection events caused by a drug at a clinically relevant concentration, are of special interest (Shen et al., 2009).
The MDPDE for beta regression model provides a robust strategy to estimate $\beta $, from which the point estimates and confidence intervals of relevant drug activity measurements can be derived. Mathematically, those drug activity quantities can be written as functions of parameters $\beta $ with an explicit form. Subsequently, their point estimates and confidence intervals can be derived based on the inference of $\beta $. For example, given a point estimate $\hat{\beta}=({\hat{\beta}}_{0},{\hat{\beta}}_{1})$, the point estimate for $\hat{m}$ , $\hat{D}}_{m$ as a single value, and $\hat{E}$ as a function of dose $d$ can be computed using Equations 4–6.
It is important to construct the confidence interval around the point estimate to gauge the estimation uncertainty. With different levels of measurement error from either wellmanaged or lousy experiments, the levels of evidence vary for statistical inference, even if it derives the same point estimates for the intercept ${\beta}_{0}$ , slope ${\beta}_{1}$ and the corresponding doseresponse curve. Given the point estimate $\hat{\beta}$ and its positivedefinite covariance matrix $\Sigma $ to account for variability in observed response, we apply the multivariate delta method and approximate the variance estimate after assuming asymptotic normality (Bickel and Doksum, 2015). As demonstrated in our simulation studies, the constructed $\left(1\alpha \right)\times 100\mathrm{\%}$ confidence interval consistently provides better results to quantify the $\left(1\alpha \right)\times 100\mathrm{\%}$ coverage probability. More importantly, the width of the constructed confidence interval was narrower than that from a linear regression model, suggesting that our approach is more efficient with a higher statistical power (Appendix 1—tables 1 and 2).
Comparison of the doseresponse curves
Request a detailed protocolWhen we estimate multiple doseresponse curves with the data collection experiments conducted in a similar setting, it is often of interest to statistically compare the drug potency and/or Hill coefficients. A typical comparison may occur when we examine the similarity of response from different drugs, explore the additional effect of a drug combined with certain monotherapy, or assess the homogeneity of a drug to different patient samples or cell lines. In the beta regression framework, the statistical comparison can be conducted by first comparing independent fits for each curve with a global fit that shares the common parameters among different groups. Subsequently, the likelihood ratio test can be applied to examine whether the same Hill coefficient or one doseresponse curve can adequately fit all the data. The only exception is to assess whether median effect doses are the same in different groups, while an F test is used for the single parameter testing. If the global test for potency shows a significant pvalue, a pairwise comparison can be conducted using twosided ttest for the ordered groups with BenjaminiHochberg correction for multiplicity.
Appendix 1
Truncation strategy
Based on the medianeffect equation method by Chou TC, the software “CompuSyn” was published. In the data entry illustration of this software, they pointed out the sensitivity limits of data points, for example too low (fa <0.02) and too high (fa >0.99) and suggested that such data points out of effect may be edited or deleted.
There are some data truncation algorithms in the literature. Two obvious remedies are proportionally “shrinking” the range to a subrange nearly covering the unit interval (e.g., [.00001,.99999]) or simply adding a small amount to 0valued observations and subtracting the same amount from 1valued observations while leaving the other observations unchanged. Macmillan and Creelman, 2005 mentioned a method that is frequently used in practice in areas such as signal detection is to add 1/(2n) to a 0 observation and subtract 1/(2n) from a 1 observation, where n is the total number of observations. Besides, Smithson and Verkuilen, 2006 demonstrated that a useful transformation in practice is (y * (n−1)+0.5) / n, which is also mentioned by the documentation for R Betareg package for conditions when data assumes the extremes 0 s and 1 s. In doseresponse curve estimation, this treatment could be too rough, especially when n is small.
To minimize the impact from truncation of data points, we apply the following algorithm. The first step is to shrink the data range to [1e9, 11e9]. If there still exist abnormal conditions, we will sequentially shrink the data range of abnormal ones to [1e8, 11e8], …, until [1e3, 11e3] or nonexist of abnormal conditions. Then, if it still exists, though rarely, the transformation of (y * (n−1)+0.5) / n, where n is the sample size, in the documentation of R Betareg package would be applied. We have conducted simulations to test this algorithm in various scenarios with different errors and it achieved reasonable performance in handling all conditions.
Data simulating process
In the simulation study, both robust beta regression and linear regression are applied to estimate doseresponse curves under different scenarios. The point estimations and 95% confidence intervals of IC_{50}, IC_{90}, ${\beta}_{0}$ and ${\beta}_{1}$ under each method will be obtained and then, be compared to evaluate the model performance.
To generate data for simulation studies, we define the dose set for simulation as 0.1, 0.2, 0.4, 0.8, 1.6, 3.2 and 6.4 $\mu M$, which consists of 7 doses, and choose the appropriate true curve with ${\beta}_{1}=2.2098$ and ${\beta}_{0}=0.4931$ such that the corresponding effects of the smallest and largest dose are 0.01 and 0.99, respectively. Let’s call the true curve “$E=f\left(\mathrm{log}\left(dose\right)\right)$”. Then, the following equation is applied to generate data by inducing random error into effect:
We simulated data with two types of errors, normal error term and beta error term, to examine the accuracy and sensitivity of model performance in general setting. The normal error term is implemented with different standard deviations (SDs), for example 0.005, 0.01 and 0.05, while the beta error term is under different precision parameter $\varphi $, for example 35, 15, 5. Note that the larger the $\varphi $, the smaller the variance. By implementing under different SD or $\varphi $, it allows for generation of not only wellcontrolled data which is assumed for experiments with almost no error, but also noised data which is more identical to realworld data. The generated data is 1 replicate given each dose level with the total simulation size equal 10,000 for each choice of SD or $\varphi $. Since the defined dose set is symmetric, we set up several scenarios under both error terms above: (1) full 7dose set with extreme values; (2) 6dose set after removing the largest dose; (3) 6dose set after removing the smallest dose; (4) full 7dose set with less extreme values by obtaining the smallest and largest dose levels with corresponding effect as 0.1 and 0.9 under the same true curve. The scenarios 1–4 assume constant precision parameter during data simulation and modeling process.
To mimic the realworld environment of data collections, the assumption of equal variance doesn’t always hold. Thus, we also set up the 5^{th} scenario which uses full 7dose set with extreme values with nonconstant SD or precision parameter during data simulation and modeling process, but linearly dosedependent. For normal error term, the modified SDs for data simulation have the form of ${SD}^{*}=({\gamma}_{0}+{\gamma}_{1}*log.dose)*SD$; for beta error term, the modified precisions $\varphi $ for data simulation have the form of ${\varphi}^{*}=({\gamma}_{0}+{\gamma}_{1}*log.dose)*\varphi $. Assuming the same true doseresponse curve as the previous simulation, we predefined ${\gamma}_{0}$ and ${\gamma}_{1}$ as 0.25 and 0.1378 such that the average of ${SD}^{*}$ is close to $SD$, and the average of ${\varphi}^{*}$ is close to $\varphi $, respectively.
Data availability
All datasets in illustrative examples are available at https://github.com/vivid225/REAP/blob/main/REAP/ to generate the figures and tables using REAP.
References

Synergistic and antagonistic drug combinations against SARSCoV2Molecular Therapy 29:873–885.https://doi.org/10.1016/j.ymthe.2020.12.016

Numerically modelling time and dose dependent cytotoxicityComputational Toxicology 12:100090.https://doi.org/10.1016/j.comtox.2019.100090

Derivation and properties of MichaelisMenten type and Hill type equations for reference ligandsJournal of Theoretical Biology 59:253–276.https://doi.org/10.1016/00225193(76)901697

BookThe MedianEffect Principle and the Combination Index for Quantitation of Synergism and AntagonismSynergism and Antagonism in Chemotherapy.

Beta regression for modelling rates and proportionsJournal of Applied Statistics 31:799–815.https://doi.org/10.1080/0266476042000214501

Computational tools for fitting the Hill equation to doseresponse curvesJournal of Pharmacological and Toxicological Methods 71:68–76.https://doi.org/10.1016/j.vascn.2014.08.006

Robust inference under the beta regression model with application to health care studiesStatistical Methods in Medical Research 28:871–888.https://doi.org/10.1177/0962280217738142

The search for synergy: a critical review from a response surface perspectivePharmacological Reviews 47:331–385.

Interaction index and different methods for determining drug interaction in combination therapyJournal of Biopharmaceutical Statistics 17:461–480.https://doi.org/10.1080/10543400701199593

Confidence intervals of interaction index for assessing multiple drug interactionStatistics in Biopharmaceutical Research 1:4–17.https://doi.org/10.1198/sbr.2009.0001

Emax model and interaction index for assessing drug interaction in combination studiesFrontiers in Bioscience 2:582–601.https://doi.org/10.2741/e116

Nonlinear modelbased estimates of IC(50) for studies involving continuous therapeutic doseresponse dataContemporary Clinical Trials 29:878–886.https://doi.org/10.1016/j.cct.2008.05.009

BookDetection Theory: A User’s Guide (Second Edition)Lawrence Erlbaum Associates Publishers.

BookToxicity Testing in the 21st Century: A Vision and A StrategyNational Academies Press.https://doi.org/10.17226/11970

Theory of synergistic effects: Hilltype response surfaces as “nullinteraction” models for mixturesTheoretical Biology & Medical Modelling 14:15.https://doi.org/10.1186/s129760170060y

A novel method for determining the inhibitory potential of antiHIV drugsTrends in Pharmacological Sciences 30:610–616.https://doi.org/10.1016/j.tips.2009.09.003

Improved estimators for a general class of beta regression modelsComputational Statistics & Data Analysis 54:348–366.https://doi.org/10.1016/j.csda.2009.08.017

Statistical monitoring of heteroscedastic doseresponse profiles from highthroughput screeningJournal of Agricultural, Biological, and Environmental Statistics 12:216–235.https://doi.org/10.1198/108571107X197779
Decision letter

Philip BoonstraReviewing Editor; University of Michigan, United States

Aleksandra M WalczakSenior Editor; CNRS LPENS, France

Philip BoonstraReviewer; University of Michigan, United States
Our editorial process produces two outputs: i) public reviews designed to be posted alongside the preprint for the benefit of readers; ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.
Decision letter after peer review:
Thank you for submitting your article "Robust and Efficient Assessment of Potency (REAP): A Quantitative Tool for Doseresponse Curve Estimation" for consideration by eLife. Your article has been reviewed by 2 peer reviewers, including Philip Boonstra as the Reviewing Editor and Reviewer #1, and the evaluation has been overseen by and Aleksandra Walczak as the Senior Editor.
The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.
Essential revisions:
1) Following Reviewer 2's first comment, please clearly state what LRM is.
2) Related to this and as suggested above, please consider also comparing the REAP approach to a version of LRM with a heavy tailed error distribution such as tdistribution with 3 degrees of freedom, which would also seem to possess the same robustness properties as REAP.
3) Please conduct additional stress testing and debugging of the shiny app, if it is intended to be included and advertised in the manuscript. See Reviewer 1's comments for specific suggestions.
Reviewer #1 (Recommendations for the authors):
I would suggest running additional stress tests on the web app to address some of the bugs above. You might also add some additional features. For example, there is a little help key next to 'Add effect estimation' that, when you hover over it, explains what that does. Could more such keys be added to other user inputs?
As to the methodology, it seems to me like it would be worthwhile to consider an approach such as what I describe in my public review – essentially the linear model but with a heavy tailed error distribution that will be less sensitive to extreme values.
Reviewer #2 (Recommendations for the authors):
The topic of this manuscript is very interesting to many researchers who work on drug screening tests and development. The manuscript is well developed and well written. How I do have some comments for the authors to address.
(1) The authors should clearly state what the LRM is, I assume it is model (3) based on normal error assumption. However, the authors should clearly spell it out.
(2) The parameter β is estimated by minimizing (8) or (9), are the two equations (8) and (9) equivalent? If so, the authors should make the connection clear.
(3) In the method session, the authors mentioned the stability condition for the selection of the tuning parameter. What is the stability condition? The authors stated that "If the stability condition is satisfied before αmax is reached, then optimal α equals the minimal value in the grid of αk.". Do the authors imply that the optimal α equals the minimal value in the grids which met the stability condition?
Overall, this is a nice manuscript and will make a nice contribution to the field involving invitro drug screening and testing.
https://doi.org/10.7554/eLife.78634.sa1Author response
Essential revisions:
Reviewer #1 (Recommendations for the authors):
I would suggest running additional stress tests on the web app to address some of the bugs above. You might also add some additional features. For example, there is a little help key next to 'Add effect estimation' that, when you hover over it, explains what that does. Could more such keys be added to other user inputs?
We thank the reviewer for the instructive suggestion and conducted multiple experiments in different data patterns to stress test the web app. We have fixed the following bugs and added instructions for some features:
1. Solve the triplicate model assessments and comparison when there is only one agent in the dataset
2. Add a hover box next to “Model Comparisons” to indicate how to select the two check boxes for effect, slope and model comparisons
3. Add a hover box next to “Width of error bar” indicating the range of the error bar width should be within (0, 0.1). The input width is restricted within the range. If the input is outside of (0, 0.1), an error message will appear below the input box and the input value will be capped at 0.1.
4. Solve the problem when there are negative dose levels in data.
5. Solve the problem of missing sample mean and error bar in doseresponse curve plotting given either out of range or missing dose responses.
As to the methodology, it seems to me like it would be worthwhile to consider an approach such as what I describe in my public review – essentially the linear model but with a heavy tailed error distribution that will be less sensitive to extreme values.
This is a great suggestion and we have added the simulation under both normal error term and β error term to compare the robust β regression framework with standard linear regression model with normal error distribution and the heavytailed linear model with t (df = 3) error distribution. In general, the heavytailed linear regression model demonstrates great potential to improve point estimation, but the coverage probabilities of its nominal 95% confidence interval are consistently underestimated below the level of 50% if with extreme values. The comparison results have been updated in the simulation Results section, Figure 4, Supplementary Figure 1 and Supplementary Tables 1 and 2.
Reviewer #2 (Recommendations for the authors):
The topic of this manuscript is very interesting to many researchers who work on drug screening tests and development. The manuscript is well developed and well written. How I do have some comments for the authors to address.
(1) The authors should clearly state what the LRM is, I assume it is model (3) based on normal error assumption. However, the authors should clearly spell it out.
We thank the reviewer for the instructive suggestion. LRM has been spelled out in the Methods section “A linear regression model (LRM) can be applied in the form of equation (3) with a standard normal distribution error.” Additionally, we also include the specifications on the heavytailed linear regression model (HLRM) and on the robust βregression model (BRM) in the Methods section and provide the linkage in figure captions.
(2) The parameter β is estimated by minimizing (8) or (9), are the two equations (8) and (9) equivalent? If so, the authors should make the connection clear.
We apologize for the confusion. Equation (9) is simplified from equation (8), with technical details in Section 2.2 (Pg. 875) of Ghosh (2019). In the Methods section of the manuscript, additional descriptions are added:
 In equation (8),$\theta ={\left(\beta ,\varphi \right)}^{T}$
 After mathematically simplifying equation (8), θ can be equivalently estimated by minimizing the objective function using the estimation equations (9).
(3) In the method session, the authors mentioned the stability condition for the selection of the tuning parameter. What is the stability condition? The authors stated that "If the stability condition is satisfied before αmax is reached, then optimal α equals the minimal value in the grid of αk.". Do the authors imply that the optimal α equals the minimal value in the grids which met the stability condition?
We thank the reviewer for the question for us to clarify the estimation methods. The stability condition is $\text{SQ}{V}_{{\alpha}_{k}}<L$. The default choice of $L$ is 0.02. The optimal α is chosen in different situations:
1. If the stability condition is satisfied before α_{max} is reached, then optimal α equals the minimal value in the grid of ${\alpha}_{k}$.
2. If α_{max} is reached, then optimal α equals 0, which is equivalent to the maximum likelihood estimation.
The above descriptions are also illustrated in the paragraph after the definition of SQV in the Methods section.
https://doi.org/10.7554/eLife.78634.sa2Article and author information
Author details
Funding
Penn State College of Medicine (Junior Faculty Development Award)
 Shouhao Zhou
 Xinyi Liu
University of Texas MD Anderson Cancer Center (Bcell Lymphoma Moonshot Program)
 Shouhao Zhou
National Cancer Institute (Cancer Center Support Grant P30 CA016672)
 Shouhao Zhou
 J Jack Lee
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Acknowledgements
This study was supported in part by the Penn State College of Medicine Junior Faculty Development Award, MD Anderson Bcell Lymphoma Moon Shot Program, and NIH Cancer Center Support Grant P30 CA016672.
Senior Editor
 Aleksandra M Walczak, CNRS LPENS, France
Reviewing Editor
 Philip Boonstra, University of Michigan, United States
Reviewer
 Philip Boonstra, University of Michigan, United States
Version history
 Preprint posted: November 22, 2021 (view preprint)
 Received: March 14, 2022
 Accepted: June 21, 2022
 Version of Record published: August 3, 2022 (version 1)
Copyright
© 2022, Zhou, Liu, Fang et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics

 948
 Page views

 139
 Downloads

 3
 Citations
Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading

 Cancer Biology
 Computational and Systems Biology
Pancreatic cancer is one of the deadliest cancer types with poor treatment options. Better detection of early symptoms and relevant disease correlations could improve pancreatic cancer prognosis. In this retrospective study, we used symptom and disease codes (ICD10) from the Danish National Patient Registry (NPR) encompassing 6.9 million patients from 1994 to 2018,, of whom 23,592 were diagnosed with pancreatic cancer. The Danish cancer registry included 18,523 of these patients. To complement and compare the registry diagnosis codes with deeper clinical data, we used a text mining approach to extract symptoms from free text clinical notes in electronic health records (3078 pancreatic cancer patients and 30,780 controls). We used both data sources to generate and compare symptom disease trajectories to uncover temporal patterns of symptoms prior to pancreatic cancer diagnosis for the same patients. We show that the text mining of the clinical notes was able to complement the registrybased symptoms by capturing more symptoms prior to pancreatic cancer diagnosis. For example, ‘Blood pressure reading without diagnosis’, ‘Abnormalities of heartbeat’, and ‘Intestinal obstruction’ were not found for the registrybased analysis. Chaining symptoms together in trajectories identified two groups of patients with lower median survival (<90 days) following the trajectories ‘Cough→Jaundice→Intestinal obstruction’ and ‘Pain→Jaundice→Abnormal results of function studies’. These results provide a comprehensive comparison of the two types of pancreatic cancer symptom trajectories, which in combination can leverage the full potential of the health data and ultimately provide a fuller picture for detection of early risk factors for pancreatic cancer.

 Cancer Biology
Cancer stem cells (CSCs) undergo epithelialmesenchymal transition (EMT) to drive metastatic dissemination in experimental cancer models. However, tumour cells undergoing EMT have not been observed disseminating into the tissue surrounding human tumour specimens, leaving the relevance to human cancer uncertain. We have previously identified both EpCAM and CD24 as CSC markers that, alongside the mesenchymal marker Vimentin, identify EMT CSCs in human oral cancer cell lines. This afforded the opportunity to investigate whether the combination of these three markers can identify disseminating EMT CSCs in actual human tumours. Examining disseminating tumour cells in over 12,000 imaging fields from 74 human oral tumours, we see a significant enrichment of EpCAM, CD24 and Vimentin costained cells disseminating beyond the tumour body in metastatic specimens. Through training an artificial neural network, these predict metastasis with high accuracy (crossvalidated accuracy of 8789%). In this study, we have observed single disseminating EMT CSCs in human oral cancer specimens, and these are highly predictive of metastatic disease.