Robust and Efficient Assessment of Potency (REAP) as a quantitative tool for dose-response curve estimation

  1. Shouhao Zhou  Is a corresponding author
  2. Xinyi Liu
  3. Xinying Fang
  4. Vernon M Chinchilli
  5. Michael Wang
  6. Hong-Gang Wang
  7. Nikolay V Dokholyan
  8. Chan Shen
  9. J Jack Lee
  1. Department of Public Health Sciences, Pennsylvania State University, United States
  2. Department of Lymphoma and Myeloma, University of Texas MD Anderson Cancer Center, United States
  3. Department of Pharmacology, Pennsylvania State University, United States
  4. Department of Pediatrics, Pennsylvania State University, United States
  5. Department of Biochemistry and Molecular Biology, Pennsylvania State University, United States
  6. Department of Surgery, The Pennsylvania State University, United States
  7. Department of Biostatistics, University of Texas MD Anderson Cancer Center, United States
7 figures, 4 tables and 1 additional file

Figures

Dose-response curve fitting with extreme observations.

The original data points are on the true curve. The leftmost data point is changed from 0.005 to 1e-6, referring to a small white noise that cannot be visually recognized. The change leads to the obvious departure between the estimated curve by linear regression model (dotted) and the true curve (solid), which demonstrates that standard regression is sensitive to extreme values. The response at the true IC50 (dotdashed, vertical, left) is only 22% from the estimated curve; the estimated IC50 (dotdashed, vertical, right) corresponds to the 70% fraction of cell affected, effecting a substantive 20% inflation (50% ->70%) in estimation error. In contrast, the estimated curve by beta regression model (dashed) is almost overlapped with the true curve (solid), which shows that BRM is much more robust to extreme values. LRM: linear regression model; BRM: robust beta regression model. Detailed model descriptions of LRM and BRM are provided in Materials and methods section.

Comparison of estimation efficiency and accuracy using linear regression model and beta regression model.

Deleting the extreme values could not eliminate the bias (panel A), but only harmed the accuracy with worse coverage probabilities (panel B) and impaired the efficiency of interval estimation with wider nominal 95% confidence intervals (panel C). A total of 1000 data sets were generated following the data simulating process described in Appendix 1, using the dose sets and true dose-response curve under 7 dose setting with a precision parameter of 100. Responses ≤5% or ≥95% were considered extreme responses. Dashed line in panel B denotes 95% nominal coverage probability. BRM: beta regression with extreme data points; LRM: linear regression model with extreme data points; LRM(t): linear regression model with truncated dataset after deleting extreme values. Detailed model descriptions of LRM and BRM are provided in Materials and methods section.

REAP App interface, with a highlight of Output section.

Using the robust beta regression method, REAP produces a dose-response curve plot with effect and model estimations. The left panel allows users to specify model features and design plot specifics. REAP also provides hypothesis testing results to compare effect estimations, slopes and models.

Comparison of the point estimates and 95% confidence intervals using linear regression model, heavy-tailed linear regression model and robust beta regression model, with data simulated from normal error term.

The vertical solid lines indicate the true values. The dots represent the averaged point estimates and the bars represent the averaged lower and upper bound of 95% CIs. The point estimation by robust beta regression is consistently closer to the true value with a narrower 95% CI compared to the linear regression model. The 95% CI of heavy-tailed linear regression underestimates the nominal coverage probability. LRM: linear regression model; LRM-7: LRM under 7-dose dataset with extreme data points; LRM-6noL: LRM under 6 dose dataset after removing the highest dose data point; LRM-6noS: LRM under 6-dose dataset after removing the lowest dose data point; LRM-7lessE: LRM under 7-dose dataset with less extreme data points; LRM-7NCP: LRM under 7-dose dataset with extreme data points and dose-dependent precision; HLRM: heavy-tailed linear regression model; HLRM-7: Heavy-tailed LRM under 7-dose dataset with extreme data points; HLRM-6noL: Heavy-tailed LRM under 6-dose dataset after removing the highest dose data point; HLRM-6noS: Heavy-tailed LRM under 6-dose dataset after removing the lowest dose data point; HLRM-7lessE: Heavy-tailed LRM under 7-dose dataset with less extreme data points; HLRM-7NCP: Heavy-tailed LRM under 7-dose dataset with extreme data points and dose-dependent precision; BRM: robust beta regression model; BRM-7: BRM under 7-dose dataset with extreme data points; BRM-6noL: BRM under 6-dose dataset after removing the highest dose data point; BRM-6noS: BRM under 6-dose dataset after removing the lowest dose data point; BRM-7lessE: BRM under 7-dose dataset with less extreme data points; BRM-7NCP: BRM under 7-dose dataset with extreme data points and dose-dependent precision. Detailed model descriptions of LRM, HLRM, and BRM are provided in Materials and methods section.

Dose-response curve estimation of auranofin (μM) under different MCL cell lines.

The dose-response curve was fitted with a dose-dependent precision with log(dose) as an additional regressor for the precision estimator. Observed dose effects are displayed with interval bars, which end with arrows when estimated intervals exceed (0,1). Triangles at the bottom indicate IC50 values for each MCL cell line. MCL: mantle cell lymphoma.

Dose-response curve estimation of anti-viral drugs under the same biological batch with SARS-CoV-2 data.

The robust beta regression gives reasonable estimations to the dose-response curve of hydroxychloroquine, compared to the inconclusive dose-response curve fitted by linear regression in Bobrowski et al. (2020). The plot is generated without selecting the option of mean and confidence interval for observations. Triangles indicate the estimated EC50 values for each drug.

Appendix 1—figure 1
Comparison of the point estimates and 95% confidence intervals using linear regression model, heavy-tailed linear regression model and robust beta regression model, with data simulated from beta error term.

The vertical solid lines indicate the true values. The dots represent the averaged point estimates and the bars represent the averaged lower and upper bound of 95% CIs. The point estimation by robust beta regression is consistently closer to the true value with a narrower 95% CI compared to the linear regression model. The 95% CI of heavy-tailed linear regression underestimates the nominal coverage probability. LRM: linear regression model; LRM-7: LRM under 7 dose dataset with extreme data points; LRM-6noL: LRM under 6 dose dataset after removing the highest dose data point; LRM-6noS: LRM under 6 dose dataset after removing the lowest dose data point; LRM-7lessE: LRM under 7 dose dataset with less extreme data points; LRM-7NCP: LRM under 7 dose dataset with extreme data points and dose-dependent precision; HLRM: heavy-tailed linear regression model; HLRM-7: Heavy-tailed LRM under 7 dose dataset with extreme data points; HLRM-6noL: Heavy-tailed LRM under 6 dose dataset after removing the highest dose data point; HLRM-6noS: Heavy-tailed LRM under 6 dose dataset after removing the lowest dose data point; HLRM-7lessE: Heavy-tailed LRM under 7 dose dataset with less extreme data points; HLRM-7NCP: Heavy-tailed LRM under 7 dose dataset with extreme data points and dose-dependent precision; BRM: robust beta regression model; BRM-7: BRM under 7 dose dataset with extreme data points; BRM-6noL: BRM under 6 dose dataset after removing the highest dose data point; BRM-6noS: BRM under 6 dose dataset after removing the lowest dose data point; BRM-7lessE: BRM under 7 dose dataset with less extreme data points; BRM-7NCP: BRM under 7 dose dataset with extreme data points and dose-dependent precision.

Tables

Appendix 1—table 1
Simulation result of bias, RMSE and 95% CI coverage probability corresponding to normal error terms.
ScenarioMethodBiasRMSE95% CI Coverage Probability
IC50IC90β0β1IC50IC90β0β1IC50IC90β0β1
(a)data simulated using normal error term with SD = 0.005
7 doses with extreme valuesLRM0.005–0.0470.0370.1520.0980.2980.5250.5570.9540.7730.9430.666
HLRM0.0000.0040.0020.0040.0180.1020.0630.1690.4680.3350.4630.225
BRM0.000–0.0040.0030.0110.0130.1300.0440.0880.9810.9270.9750.889
6 doses after removing largestLRM0.009–0.01–0.0240.0980.0450.1840.1290.5170.9670.9210.9710.691
HLRM–0.0010.0010.002–0.0010.0140.0630.0370.0650.4530.4080.4670.250
BRM0.0010.005–0.000.0050.0090.1320.0240.0890.9550.8920.9500.844
6 doses after removing smallestLRM–0.007–0.0360.0700.1020.0370.3060.3620.5330.9690.6230.9330.695
HLRM0.0010.007–0.002–0.0010.0140.0830.0460.0640.4560.2490.4000.259
BRM–0.0010.0000.0050.0050.0090.1510.0390.0890.9560.8720.9400.853
7 doses with less extreme valuesLRM0.000–0.0000.0000.0010.0050.0300.0160.0260.8910.8280.8830.799
HLRM0.0000.0000.0000.0000.0050.0310.0160.0280.6680.5990.6610.567
BRM0.0000.0000.0000.0000.0040.0260.0130.0240.8640.8230.8600.800
7 doses with extreme values and dose-dependent precisionLRM–0.016–0.0470.1370.1110.0690.4090.5830.5200.9650.5730.9210.685
HLRM0.0000.003–0.001–0.0010.0080.0490.0270.0360.3250.1880.2900.151
BRM0.0000.023–0.003–0.0110.0030.1960.0230.0950.9570.8490.9310.794
(b)data simulated using normal error term with SD = 0.01
7 doses with extreme valuesLRM0.030–0.2000.1660.8040.2240.5441.2341.3050.9580.6990.9500.716
HLRM0.0000.0060.0250.1050.0310.2380.1990.7950.4080.2860.4040.194
BRM0.001–0.0390.0130.0600.0300.1720.0940.1550.9810.9310.9760.892
6 doses after removing largestLRM0.040–0.088–0.1250.5490.1010.2800.3141.2450.9660.9240.9720.734
HLRM–0.0040.0070.008–0.0080.0270.1250.0750.1230.4240.3840.4450.241
BRM0.005–0.012–0.0070.0400.0190.1400.0440.1480.9540.8930.9490.850
6 doses after removing smallestLRM–0.027–0.1720.3570.5310.0800.5340.8361.2200.9660.5610.9370.734
HLRM0.0040.027–0.010–0.0050.0280.1600.0890.1250.4340.2470.3840.253
BRM–0.005–0.0350.0250.0400.0180.1780.0780.1430.9580.8770.9430.857
7 doses with less extreme valuesLRM0.000–0.0010.0010.0040.0110.0600.0320.0530.8920.8260.8830.799
HLRM0.0000.0000.0010.0020.0110.0610.0320.0560.6650.5970.6570.564
BRM0.0000.0010.0000.0010.0090.0520.0260.0480.8650.8220.8600.801
7 doses with extreme values and dose-dependent precisionLRM–0.052–0.2150.5240.4620.1300.6221.1230.9840.9650.5210.9210.726
HLRM0.0020.011–0.004–0.0030.0160.0960.0530.0710.3030.1800.2730.148
BRM0.0000.012–0.001–0.0050.0090.1420.0540.0800.9570.8510.9320.798
(c)data simulated using normal error term with SD = 0.05
7 doses with extreme valuesLRM0.079–0.4630.5602.3990.3930.9242.1281.8530.9480.6120.9420.591
HLRM0.0240.0470.3451.5240.2231.0501.6002.6380.4770.2580.4470.083
BRM0.0130.0290.0100.0950.1290.5240.3710.3770.8570.8610.8690.855
6 doses after removing largestLRM0.079–0.3380.3372.1940.2760.7492.0542.2390.9680.7790.9750.675
HLRM–0.0050.0800.2600.9260.1700.8201.3922.5470.4070.2930.4170.146
BRM0.0170.0010.0070.1600.1240.5120.3870.4300.8790.8350.9120.812
6 doses after removing smallestLRM0.022–0.4030.6662.2320.3350.9651.9362.2200.9720.6100.8980.681
HLRM0.0350.2440.1620.9190.1580.9981.3322.5260.4100.2090.3200.146
BRM0.009–0.0210.0300.1600.1190.5110.3580.4200.8740.7540.8620.819
7 doses with less extreme valuesLRM0.005–0.0770.0860.3620.0840.3830.6931.2150.9050.7950.8950.813
HLRM0.0010.0290.0090.0390.0530.3270.1850.4460.6190.5510.6100.520
BRM0.002–0.0010.0120.0620.0540.2960.1870.3110.8610.8100.8580.815
7 doses with extreme values and dose-dependent precisionLRM–0.029–0.4410.9941.6790.2970.9551.8401.7150.9450.4440.9040.678
HLRM0.0060.0370.2780.6170.1060.7101.1521.7810.3350.1450.2950.105
BRM–0.0050.0370.023–0.0020.0400.3230.1720.2710.9540.8190.9300.758
Appendix 1—table 2
Simulation result of bias, RMSE and 95% CI coverage probability corresponding to beta error terms.
ScenarioMethodBiasRMSE95% CI Coverage Probability
IC50IC90β0β1IC50IC90β0β1IC50IC90β0β1
(a)data simulated using beta error term with ϕ=35
7 doses with extreme valuesLRM0.017–0.3040.1420.6530.1630.4810.6580.6860.9240.6970.9090.566
HLRM0.008–0.1610.0850.3660.1170.4780.4020.5760.5810.4290.5710.313
BRM0.003–0.0140.0180.0730.0740.3390.2190.2750.8350.8180.8320.829
6 doses after removing largestLRM0.039–0.174–0.0270.5040.1150.4020.3540.6660.9330.8640.9360.666
HLRM0.015–0.0920.0260.2730.1090.4630.3370.4860.5810.5180.5940.368
BRM0.0080.0050.0070.0770.0770.3750.2290.2980.8120.8080.8110.803
6 doses after removing smallestLRM–0.022–0.2880.2480.5020.1080.5010.4990.6700.9300.6200.9010.664
HLRM0.000–0.1200.0940.2740.1060.5090.3660.4870.5960.3830.5470.368
BRM–0.002–0.0200.0310.0740.0750.3530.2230.2980.8150.7820.8050.800
7 doses with less extreme valuesLRM0.004–0.0450.0230.1170.0630.3320.1950.3000.9000.8280.8910.821
HLRM0.003–0.0170.0210.0960.0670.3820.2050.3230.6860.6300.6800.619
BRM0.0030.0310.0040.0280.0580.3310.1700.2720.8450.8370.8470.838
7 doses with extreme values and dose-dependent precisionLRM0.080–0.093–0.1790.4440.1470.2810.4350.5480.9040.9080.9270.639
HLRM0.061–0.200–0.0630.7530.1990.5530.6760.9670.4840.4470.5200.231
BRM0.0030.031–0.0030.0120.0670.2770.1820.2360.7550.7800.7680.772
(b)data simulated using beta error term with ϕ=15
7 doses with extreme valuesLRM0.029–0.5710.3731.6330.2330.5551.1771.1970.9410.5880.9100.406
HLRM0.016–0.3340.2250.9830.1720.6570.7251.1860.5610.3530.5440.213
BRM0.008–0.0500.0360.1740.1090.4640.3240.4030.8170.7900.8200.805
6 doses after removing largestLRM0.074–0.360–0.0431.2690.1640.5160.6351.1920.9420.8230.9390.556
HLRM0.029–0.2060.0860.7020.1640.7150.5940.9250.5520.4450.5510.272
BRM0.018–0.0280.0190.2030.1170.5270.3550.4460.7780.7830.7940.776
6 doses after removing smallestLRM–0.042–0.5640.6201.2800.1570.5960.8901.1800.9390.5030.9050.555
HLRM–0.001–0.2620.2370.7000.1600.7120.6630.9170.5460.3000.4860.268
BRM–0.001–0.0710.0660.1920.1150.4990.3450.4490.7870.7330.7820.777
7 doses with less extreme valuesLRM0.008–0.1040.0590.2860.0960.5130.3200.4890.9060.7950.8840.807
HLRM0.009–0.0300.0480.2400.1030.7260.3250.5170.6940.6020.6780.598
BRM0.0060.0740.0120.0620.0880.5530.2600.4090.8380.8360.8410.847
7 doses with extreme values and dose-dependent precisionLRM0.165–0.202–0.4251.1140.2180.3740.7710.9590.9160.8860.9410.534
HLRM0.141–0.347–0.2502.0550.2911.0271.2701.7980.5150.4400.5760.136
BRM0.0100.053–0.0080.0500.1040.4230.2810.3680.7630.7900.7640.769
(c)data simulated using beta error term with ϕ=5
7 doses with extreme valuesLRM0.042–0.8510.7483.4140.2700.5271.6111.4210.9480.4230.8980.198
HLRM0.040–0.6290.6893.1470.2781.5181.4802.1120.6810.2410.6440.094
BRM0.008–0.0110.0470.1910.1610.6060.5890.7670.8110.8000.8150.811
6 doses after removing largestLRM0.106–0.6360.1812.9620.2340.8651.4011.6040.9490.6840.9380.356
HRLM0.075–0.0980.2762.4100.2748.5991.3512.0820.6070.3620.6000.142
BRM0.007–0.1150.1120.3550.1660.6430.6210.7610.7620.7520.7730.769
6 doses after removing smallestLRM–0.039–0.8711.1262.9350.2310.6491.4281.6200.9530.3800.8830.362
HLRM0.004–0.3710.7732.3790.2615.2381.4642.0680.5990.2060.4920.149
BRM0.008–0.0870.0700.3280.1660.5880.5090.7240.7660.7550.7720.773
7 doses with less extreme valuesLRM0.008–0.4560.2751.1260.1520.5150.6791.0410.9150.6390.8740.707
HLRM0.007–0.3150.2360.9230.1640.7670.6511.0470.6810.4750.6440.491
BRM0.002–0.0880.0930.3180.1320.5880.4230.6600.8390.8120.8340.835
7 doses with extreme values and dose-dependent precisionLRM0.245–0.378–0.6462.3000.2740.4701.2071.2620.9080.8330.9470.343
HLRM0.209–0.578–0.2334.9820.3326.8522.0162.0170.6830.3810.7280.050
BRM0.0250.171–0.0320.0740.1610.7620.4260.5490.7570.7960.7610.770
Appendix 1—table 3
The first example of REAP application with B-cell lymphoma data, corresponding with Figure 5.

IC50 estimations are ranked from low to high. Hypothesis testings on equal potency (i.e., concentration for IC50) were conducted pairwise with the group right above (one rank lower). Jeko-1 has the highest potency and the difference of IC50 estimations between Jeko-1 and Jeko-R is significant with a P-value <0.0001. The B-cell lymphoma dataset is available on Github (Fang et al., 2022).

ModelInterceptSlope (m)Std. Err for mP-value for m>1IC50 estimationStd. Err for IC50 estimationPairwise comparison
Jeko-1–4.807–1.2520.1550.05190.0210.008-
Jeko-R–4.305–1.8220.112<.00010.0940.006<0.0001
Rec-1–5.304–2.630.091<.00010.1330.004<0.0001
Mino–2.684–1.4740.1410.00040.1620.0150.0656
Jeko-NO #1–2.012–1.1920.1350.07690.1850.0210.3755
MAVER-1–2.21–1.370.1250.00150.1990.0210.6312
Jeko-NO #11–1.459–1.2670.1520.03980.3160.0380.0114
JVM2–1.056–1.2710.1350.02230.4360.0550.0818
Appendix 1—table 4
The output for the estimated dose-response curve of anti-viral drugs under the same biological batch with SARS-CoV-2 data.

Calpain inhibitor IV has the highest potency (P-value = 0.0038). The reconstructed SARS-CoV-2 dataset is available on Github (Fang et al., 2022).

ModelInterceptSlope (m)Std. Err for mP-value for m>1EC50 estimationStd. Err for EC50 estimationPairwise comparison
CalpainInhibitorIV0.6780.7250.1140.99180.3930.103-
Chloroquine–1.0130.840.1350.88133.3370.880.0038
Remdesivir–1.7910.7970.120.95539.4692.6380.0282
Hydroxychloroquine–1.4850.5620.075114.0744.9940.4445
E64d (Aloxistatin)–3.2110.8610.1290.858741.6115.4730.1242

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Shouhao Zhou
  2. Xinyi Liu
  3. Xinying Fang
  4. Vernon M Chinchilli
  5. Michael Wang
  6. Hong-Gang Wang
  7. Nikolay V Dokholyan
  8. Chan Shen
  9. J Jack Lee
(2022)
Robust and Efficient Assessment of Potency (REAP) as a quantitative tool for dose-response curve estimation
eLife 11:e78634.
https://doi.org/10.7554/eLife.78634