1. Medicine
  2. Microbiology and Infectious Disease
Download icon

Early prediction of level-of-care requirements in patients with COVID-19

  1. Boran Hao
  2. Shahabeddin Sotudian
  3. Taiyao Wang
  4. Tingting Xu
  5. Yang Hu
  6. Apostolos Gaitanidis
  7. Kerry Breen
  8. George C Velmahos
  9. Ioannis Ch Paschalidis  Is a corresponding author
  1. Center for Information and Systems Engineering, Boston University, United States
  2. Division of Trauma, Emergency Services, and Surgical Critical Care Massachusetts General Hospital, Harvard Medical School, United States
Research Article
Cite this article as: eLife 2020;9:e60519 doi: 10.7554/eLife.60519
17 tables and 2 additional files

Tables

Table 1
Hospitalization prediction model (test performance).

The values inside the parentheses refer to the standard deviation of the corresponding metric. Random refers to test set results from the five random training/test splits. BWH refers to training on four other hospitals and testing on data from BWH. SVM-L1 and LR-L1 refer to the ℓ1-norm regularized SVM and LR models. For the parsimonious model, we list the LR coefficients of each variable (Coef), the correlation of the variable with the outcome (Y-corr), the mean of the variable (Y1-mean) in the positive class (hospitalized for this table), and the mean of the variable (Y0-mean) in the negative class (non-hospitalized). Binary Coef denotes the coefficient of the variables in the binarized model. We report the corresponding odds ratio (OR) and the 95% confidence intervals (CI). Thresholds used for the binarized model are provided in Appendix 1—table 5.

AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 106 features
LR-L287.0% (1.7%)85.9%81.6% (1.3%)84.2%
SVM-L187.0% (1.6%)85.8%81.5% (1.5%)83.9%
XGBoost87.8% (1.9%)87.7%80.9% (1.8%)83.3%
RF88.2% (1.6%)88.1%81.2% (1.1%)83.2%
Models using 74 statistically selected features
LR-L287.1% (1.7%)86.0%82.0% (1.3%)83.9%
SVM-L187.1% (1.7%)85.8%82.0% (1.4%)84.0%
XGBoost87.9% (1.9%)87.6%81.2% (1.9%)84.2%
RF88.0% (1.7%)88.1%80.8% (1.7%)83.9%
Parsimonious Model using 11 features
LR-L283.4% (1.7%)83.7%78.7% (0.9%)81.0%
SVM-L183.4% (1.7%)83.8%78.1% (1.1%)79.9%
Variables for the Parsimonious Model
VariableCoefY1 meanY0 meanp-valueY-corrCoef binaryOROR 95% CI
SpO2 (%)−11.9095.4497.11<0.001−0.291.745.673.978.12
Temperature10.3637.2137.06<0.0010.080.862.361.763.18
Respiratory
Rate
7.2022.8220.83<0.0010.18−0.130.880.691.13
Age5.1462.3146.02<0.0010.410.882.41.863.11
Pulse4.6090.0990.4<0.001−0.010.72.011.492.71
Diastolic
BP
−3.5673.0777.21<0.001−0.231.514.512.887.06
Adrenal
Insufficiency
3.090.0130.001<0.0010.082.5813.141.57110.37
BMI2.3031.3431.64<0.001−0.04−0.090.910.711.17
Transplantation1.900.0230.002<0.0010.11.434.191.0416.87
Dyspnea1.850.170.02<0.0010.2627.414.8511.32
CKD1.550.140.02<0.0010.250.812.251.353.74
Intercept−2.51
  1. SpO2: oxygen saturation; BP: Blood pressure; BMI: Body Mass Index; CKD: Chronic Kidney Disease.

Table 2
ICU prediction model (test performance).

Abbreviations are as in Table 1. Thresholds for the binarized model, PSI and CURB-65 scores are in the Appendix.

ICU prediction results with 2513 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost86.0% (2.8%)83.1%90.0% (1.7%)91.7%
SVM-L185.9% (2.5%)80.2%89.9% (1.0%)89.2%
LR-L184.6% (2.8%)76.8%89.7% (1.0%)89.9%
RF86.9% (2.4%)83.7%90.4% (1.1%)91.1%
Models using 56 statistically selected features
XGBoost86.8% (3.1%)82.8%90.4% (1.4%)91.3%
SVM-L186.2% (2.6%)82.6%90.6% (1.2%)90.8%
LR-L185.8% (2.9%)81.8%90.2% (1.3%)91.3%
RF86.7% (2.0%)83.2%90.5% (1.7%)91.5%
Parsimonious Model using 10 features
LR-L185.8% (2.6%)83.9%90.0% (1.4%)89.1%
LR-L1 (binarized model)84.2% (2.2%)82.5%89.8% (1.1%)88.1%
Model using PSI or CURB-65 score
PSI score72.9% (4.9%)78.8%86.8% (0.7%)88.2%
CURB-65 score67.0% (5.0%)75.4%87.0% (0.5%)88.1%
Variables for the parsimonious model
VariableCoefY1 meanY0 meanp-valueY-corrCoef binaryOROR 97.5% CI
Radiology
Opacities
0.540.760.27<0.0010.301.414.082.835.89
Respiratory
Rate
0.4624.6121.37<0.0010.160.501.661.142.41
Age0.4562.6150.58<0.0010.180.561.761.272.43
Fever0.400.640.33<0.0010.180.611.831.322.55
Male0.350.640.44<0.0010.120.501.651.212.26
Albumin−0.343.683.84<0.001−0.160.581.781.102.90
Anion
Gap
0.3316.4015.35<0.0010.13−0.050.950.461.98
SpO2 (%)−0.2294.7296.72<0.001−0.240.832.291.633.21
LDH0.22400.40327.48<0.0010.150.962.621.743.94
Calcium−0.218.849.01<0.001−0.100.551.731.212.48
Intercept−0.93
  1. SpO2: oxygen saturation; LDH: Lactate dehydrogenase.

Table 3
Restricted ICU prediction model (test performance).

Abbreviations are as in Table 1. Thresholds for the binarized model, PSI and CURB-65 scores are in the Appendix.

ICU prediction results with 628 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost82.5% (1.9%)67.3%81.4% (0.7%)72.6%
SVM-L177.8% (3.8%)72.8%79.7% (1.2%)73.6%
LR-L175.9% (3.6%)69.7%79.2% (2.5%)73.7%
RF80.9% (2.7%)76.9%78.8% (1.9%)73.6%
Models using 29 statistically selected features
XGBoost82.7% (2.7%)76.2%80.6% (2.1%)72.6%
SVM-L177.9% (3.7%)73.1%78.5% (1.4%)73.6%
LR-L178.4% (4.1%)71.5%79.5% (2.6%)74.4%
RF82.1% (2.8%)74.1%79.0% (2.4%)75.4%
Parsimonious Model using 8 features
LR-L180.1% (2.9%)74.2%80.9% (2.1%)77.2%
LR-L1 (binarized model)72.5% (5.4%)69.9%73.4% (2.8%)69.7%
Model using PSI or CURB-65 score
PSI score58.8% (7.4%)68.3%66.7% (2.2%)65.3%
CURB-65 score56.8% (4.5%)76.9%66.2% (1.5%)63.8%
Variables for the parsimonious model
VariableCoefY1 meanY0 meanp-valueY-corrCoef binaryOROR 97.5% CI
LDH0.53519.88304.40<0.0010.151.594.882.658.99
CRP (mg/L)0.47127.1767.43<0.0010.350.762.130.706.47
Calcium−0.358.839.01<0.001−0.130.712.031.253.31
IDDM0.300.250.120.0030.151.002.731.624.60
SpO2 (%)−0.2994.1395.590.003−0.220.341.410.922.16
Radiology Opacities0.250.880.71<0.0010.160.621.861.053.29
Anion Gap0.2016.6615.28<0.0010.200.341.400.484.12
Sodium−0.16136.13137.53<0.001−0.140.471.601.052.43
Intercept−0.34
  1. LDH: Lactate dehydrogenase; CRP: C-reactive protein; IDDM: Insulin-dependent diabetes mellitus; SpO2: oxygen saturation.

Table 4
Ventilation prediction model (test performance).

Abbreviations are as in Table 1. Thresholds for the binarized model, PSI and CURB-65 scores are in the Appendix.

Ventilation prediction results with 2525 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost85.8% (4.0%)83.8%91.0% (0.4%)91.6%
SVM-L182.6% (4.9%)83.8%90.9% (0.8%)91.6%
LR-L180.7% (5.4%)81.7%90.4% (1.2%)91.4%
RF85.7% (3.9%)83.7%91.2% (0.9%)91.8%
Models using 55 statistically selected features
XGBoost85.7% (3.3%)86.3%91.1% (0.6%)91.6%
SVM-L183.9% (3.7%)84.8%90.9% (1.1%)91.7%
LR-L183.3% (4.0%)83.9%90.8% (1.3%)91.4%
RF86.4% (3.4%)86.7%91.4% (1.1%)91.3%
Parsimonious Model using 8 features
LR-L185.2% (2.3%)87.0%90.3% (0.3%)90.7%
LR-L1 (binarized model)81.3% (3.1%)82.6%90.0% (0.6%)90.2%
Model using PSI or CURB-65 score
PSI score73.6% (4.1%)80.7%89.4% (0.4%)90.3%
CURB-65 score66.8% (3.1%)75.9%89.7% (0.1%)90.0%
Variables for the Parsimonious Model
VariableCoefY1 meanY0 meanp-valueY-corrCoef binaryOROR 97.5% CI
Radiology
opacities
0.860.770.28<0.0010.271.584.863.257.25
Albumin−0.453.653.83<0.001−0.161.072.911.804.72
Fever0.430.660.33<0.0010.170.722.051.422.95
Respiratory
rate
0.4224.7021.44<0.0010.150.501.641.092.47
Glucose0.38170.17138.32<0.0010.150.972.631.714.06
Male0.340.640.44<0.0010.100.431.541.092.18
LDH0.33408.56328.78<0.0010.140.912.481.583.89
Anion
gap
0.3116.5015.37<0.0010.130.271.310.533.25
Intercept−1.06
  1. LDH: Lactate dehydrogenase.

Table 5
Restricted ventilation prediction model (test performance).

Abbreviations are as in Table 1.Thresholds for the binarized, PSI and CURB-65 scores are in the Appendix.

Ventilation prediction results with 635 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost80.6% (1.9%)74.7%79.4% (2.6%)75.7%
SVM-L179.4% (5.2%)71.3%80.8% (2.0%)75.7%
LR-L176.9% (3.9%)68.2%78.6% (3.2%)73.4%
RF81.0% (3.1%)75.8%79.8% (4.2%)72.7%
Models using 29 statistically selected features
XGBoost81.6% (3.2%)76.9%79.0% (2.9%)71.7%
SVM-L179.1% (4.6%)69.4%80.6% (2.5%)75.7%
LR-L180.9% (3.6%)70.9%80.4% (2.2%)75.7%
RF81.3% (2.6%)75.4%79.2% (1.7%)69.6%
Parsimonious Model using 5 features
LR-L182.4% (3.7%)75.2%81.8% (1.7%)71.7%
LR-L1 (binarized model)71.4% (6.2%)65.5%76.6% (3.5%)68.3%
Model using PSI or CURB-65 score
PSI score57.6% (4.5%)67.4%73.2% (1.3%)71.2%
CURB-65 score56.9% (7.1%)74.0%72.4% (0.2%)68.3%
Variables for the parsimonious model
VariableCoefY1 meanY0 meanp-valueY-corrCoef binaryOROR 97.5% CI
CRP (mg/L)0.60134.5269.62<0.0010.350.421.530.514.59
LDH0.55550.41311.01<0.0010.161.876.473.1913.10
Calcium−0.398.829.00<0.001−0.130.581.791.072.98
IDDM0.360.260.120.0020.151.183.261.905.58
Anion Gap0.2916.8115.32<0.0010.1918.661.27E+080.00inf
Intercept−0.39
  1. CRP: C-reactive protein; LDH: Lactate dehydrogenase; IDDM: Insulin-dependent diabetes mellitus.

Table 6
Mean and median hours between reference date/lab results to outcomes in full/restricted ICU and ventilation model prediction.
From reference date (mean)From reference date (median)From lab results (mean)From lab results (median)
Restricted ICU38.1328.0822.559.90
Restricted intubation35.3626.4022.3710.39
Full ICU22.8617.2815.8612.99
Full intubation25.6222.2010.238.97
Appendix 1—table 1
Representative patient statistics.
Admitted (36.2%)ICU (10.6%)Intubated (8.5%)
YesNop-valueYesNop-valueYesNop-value
Age62.346.0<0.00163.350.6<0.00163.350.9<0.001
Gender (male)55.3%40.1%<0.00163.0%43.5%<0.00163.6%43.9%<0.001
Asian3.7%4.0%0.973.7%3.9%13.7%3.9%1
Black/African American15.7%17.8%0.6114.7%17.3%0.7514.3%17.3%0.74
Hispanic/Latino4.9%5.9%0.816.6%5.4%0.886.9%5.4%0.83
White45.4%43.9%0.9139.6%45.0%0.4039.6%44.9%0.53
Hypertension61.7%26.4%<0.00162.3%36.5%<0.00161.8%37.1%<0.001
Diabetes34.2%9.7%<0.00140.7%15.9%<0.00142.9%16.3%<0.001
Alzheimer6.7%0.6%<0.0012.6%2.8%13.2%2.7%0.98
Congestive Heart Failure (CHF)11.3%0.8%<0.0019.5%4.0%<0.0018.8%4.2%0.025
Chronic Kidney Disease (CKD)14.4%1.7%<0.00112.8%5.5%<0.00111.5%5.8%0.011
ACE Inhibitors (ACEIs)17.5%8.4%<0.00120.5%10.7%<0.00119.8%11.0%0.002
Acetaminophen
Tylenol
39.8%17.8%<0.00131.9%25.1%0.1230.4%25.4%0.45
Amiodarone1.6%0.1%<0.0011.5%0.5%0.320.9%0.6%0.95
Anticoagulants9.4%1.7%<0.0019.9%3.8%<0.00111.1%3.8%<0.001
Anti-depressants25.4%16.7%<0.00120.5%19.8%0.9922.6%19.6%0.77
Angiotensin Receptor Blockers (ARBs)12.0%5.2%<0.00115.4%6.8%<0.00117.1%6.8%<0.001
Aspirin related32.3%11.6%<0.00133.7%17.4%<0.00133.2%17.8%<0.001
Beta-Blockers28.1%10.4%<0.00125.6%15.7%<0.00125.8%16.0%0.003
Calcium Chanel Blockers (CCBs)2.6%0.7%0.0014.4%1.0%<0.0014.6%1.1%<0.001
Coumadin
warfarin
3.5%0.7%<0.0011.8%1.7%11.8%1.7%1
Diuretics16.0%4.5%<0.00113.9%8.1%0.01513.4%8.3%0.089
Immuno- suppressants5.3%2.6%0.0053.7%3.5%14.1%3.5%0.97
Insulin related14.6%3.5%<0.00119.0%6.2%<0.00121.2%6.3%<0.001
Metformin related19.5%8.6%<0.00123.8%11.2%<0.00124.9%11.4%<0.001
Nonsteroidal anti-inflammatory drugs (NSAIDs)21.9%21.0%0.9519.0%21.6%0.8218.0%21.6%0.66
Proton Pump Inhibitors (PPIs)26.6%15.0%<0.00124.5%18.5%0.1325.8%18.6%0.081
Statins45.1%17.3%<0.00147.6%24.9%<0.00145.6%25.7%<0.001
Steroids30.5%23.0%<0.00130.8%25.2%0.2630.4%25.3%0.44
Cough65.6%29.6%<0.00168.1%39.6%<0.00169.1%40.2%<0.001
Dyspnea16.6%2.2%<0.00121.6%5.7%<0.00123.5%5.9%<0.001
Chest pain21.1%5.6%<0.00122.0%9.9%<0.00124.4%10.0%<0.001
Fever57.4%23.7%<0.00161.2%32.9%<0.00163.6%33.4%<0.001
SpO295.297.4<0.00193.496.7<0.00193.396.7<0.001
Diastolic BP72.578.1<0.00172.075.6<0.00170.975.6<0.001
Pulse90.688.3<0.00193.388.80.00394.188.90.01
Respiratory Rate (RR)23.120.3<0.00125.621.2<0.00125.921.3<0.001
Temperature (oC)37.237.0<0.00137.337.10.00137.337.10.001
Anion Gap15.817.015.1<0.00117.115.1<0.001
Sodium137.0136.3137.4<0.001136.2137.3<0.001
Calcium9.08.89.0<0.0018.89.0<0.001
Lactic acid1.82.11.6<0.0012.11.6<0.001
Glomerular filtration rate (GFR)67.064.872.3<0.00164.771.9<0.001
Chloride98.197.298.8<0.00197.198.8<0.001
Glucose149.6171.5135.8<0.001173.9137.2<0.001
Lactate Dehydrogenase (LDH)377.2524.6303.9<0.001551.8310.6<0.001
Albumin3.83.63.9<0.0013.63.9<0.001
D-Dimer1373.51525.01223.7<0.0011614.51214.0<0.001
C-reactive Protein (CRP)89.6133.165.5<0.001140.168.1<0.001
Blood Urea Nitrogen (BUN)21.424.318.5<0.00123.818.9<0.001
Creatine Kinase (CK)385.2563.4282.7<0.001620.3285.1<0.001
Ferritin854.21349.5601.6<0.0011477.1621.8<0.001
Mean Platelet Volume (MPV)10.510.610.5<0.00110.610.5<0.001
Atelectasis19.0%4.6%<0.00115.8%9.2%0.00816.6%9.2%0.007
Consolidation5.9%0.6%<0.00110.3%1.6%<0.00111.1%1.7%<0.001
Nodule4.9%0.6%<0.0014.4%1.9%0.0723.7%2.0%0.47
Opacity64.8%13.7%<0.00178.4%26.7%<0.00180.6%27.8%<0.001
Pleural Effusion8.8%1.1%<0.00111.7%3.0%<0.00113.8%3.0%<0.001
Appendix 1—table 2
Distribution of patients in different hospitals and outcome groups.
HospitalPositiveAdmittedICUIntubated
Brigham and Women's Hospital (BWH)6481716756
Newton-Wellesley Hospital (NWH)4341453318
Massachusetts General Hospital (MGH)1195475144121
North Shore Medical Center (NSM)97631612
Faulkner Hospital (FH)192761310
Total2566930273217
Appendix 1—table 3
List of 164 features used for hospitalization, ICU, and ventilation models.
CategoryFeatures
DemographicsMarital status, Gender, Race, Age, Language, Tobacco, Alcohol, Height, Weight, BMI
VitalsSystolic BP, Diastolic BP, Temperature, Pulse, Respiratory Rate, SpO2 percentage
SymptomsFever, Cough, Dyspnea, Fatigue, Diarrhea, Nausea, Vomiting, Abdominal pain, Loss of smell, Loss of taste, Chest pain, Headache, Sore throat, Hemoptysis, Myalgia
Pre-existing medicationsSteroids, ACEIs, ARBs, NSAIDs, Anti-depressants, CCBs, Diuretics, Digoxin, Statins, Beta-Blockers, Acetaminophen Tylenol, Immunosuppressants, Anticoagulants, Aspirin related, Coumadin warfarin, Amiodarone, Insulin related, Metformin related, PPIs
ComorbiditiesHypertension, COPD, Diabetes, CKD, CAD, MI, Asthma, Osteoarthritis arthritis, SLE, HLD, Arrhythmia, Thyroid disease, Stroke, Migraine, Epilepsy, Alzheimer, Parkinson, Nephrolithiasis, Cushing, Adrenal Insufficiency, Diverticulosis, GERD, IBS, IBD, Cholelithiasis, Inguinal hernia, Hepatitis, Cirrhosis, Valvular disease, CHF, PAD, Osteoporosis, Cancer, TB, Cardiomyopathy, AAA, DVT, vWD, Anemia, Transplantation, HIV, Depression, Anxiety
RadiologyOpacity, Atelectasis, Consolidation, Pleural Effusion, Pneumothorax, Nodule
LabsRDW, PLT, MCH, HGB, MCHC, HCT, MCV, RBC, WBC, MPV, NRBC (%), GFR (estimated), Creatinine, Potassium, Chloride, Sodium, Anion Gap, BUN, Glucose, Calcium, Carbon Dioxide, Absolute Neutrophil count, Absolute Lymphocyte count, Absolute Monocyte count, Absolute Eosinophil count, Absolute Basophil count, Immature Granulocytes, ALT, Total Protein, Albumin, Globulin, AST, Bilirubin (Total), Alkaline phosphatase, NRBC Auto (#), LDH, Ferritin, CK, Magnesium, CRP, PT, D-Dimer, Lactic acid, Phosphorus, PTT, PCO2 (Venous), pH (Venous), Fibrinogen, Lipase, Bands (manual), PO2 (Venous), Base Deficit (Venous), Iron, Bilirubin (Direct), Myelocytes, HCO3 (unspecified), TIBC, Base Deficit (Arterial), PCO2 (Arterial), Metamyelocytes, Plasma cells (%), PO2 (Arterial), Ionized Calcium, pH (Arterial), Osmolality
Appendix 1—table 4
Performance of the NLP models.
Precision (%)Recall (%)F1-score (%)
NER+NLI model93.6087.9790.70
Regular expression matching99.0196.1597.56
Appendix 1—table 5
Abnormal ranges for laboratory tests and vitals.
VariableAbnormal range
Albumin<3.3
Chloride<95
Lactic acid≥2
LDH≥250
CRP (mg/L)≥10
Calcium≤8.5
Anion gap≥12
Glucose≥110
Total protein≤6.5 or ≥8.3
D-Dimer (ng/mL)≥500
GFR≤60
Sodium<135
Globulin≤2 or ≥4
SpO2≤94
Systolic blood pressure≤100
Pulse≥100
Respiratory rate≥20
Age≥65
Diastolic blood pressure≤60
BMI≥30
Temperature≥37.5 °C or ≥98.7 °F
Appendix 1—table 6
Derivation cohort performance for the hospitalization prediction model.

Abbreviations and metrics reported are as in Table 1.

AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 106 features
LR-L288.3% (0.4%)88.3%82.9% (0.5%)82.3%
SVM-L188.2% (0.4%)88.2%82.8% (0.5%)82.1%
XGBoost91.5% (2.1%)90.9%85.7% (2.3%)85.2%
RF96.0% (0.7%)95.3%92.9% (1.2%)90.8%
Models using 74 statistically selected features
LR-L287.8% (0.4%)87.8%82.4% (0.4%)81.7%
SVM-L187.8% (0.4%)87.7%82.5% (0.7%)81.7%
XGBoost91.9% (1.8%)91.9%86.0% (1.8%)86.2%
RF94.9% (0.9%)96.6%91.3% (1.3%)93.2%
Parsimonious Model using 11 features
LR-L282.6% (0.5%)82.4%77.6% (0.1%)76.9%
SVM-L182.5% (0.5%)82.3%77.5% (0.3%)76.9%
Appendix 1—table 7
Derivation cohort performance for the ICU prediction model.

Abbreviations and metrics reported are as in Table 1.

ICU prediction results (training performance) with 2513 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost94.5% (3.6%)96.1%94.0% (1.7%)94.1%
SVM-L189.7% (0.7%)91.4%91.5% (0.4%)91.9%
LR-L191.3% (0.6%)92.9%91.5% (0.5%)91.9%
RF93.4% (3.2%)97.0%94.3% (1.6%)95.4%
Models using 56 statistically selected features
XGBoost94.1% (1.5%)95.1%93.6% (0.6%)93.7%
SVM-L188.5% (0.7%)89.7%91.2% (0.4%)91.4%
LR-L189.3% (0.7%)90.4%91.2% (0.2%)91.4%
RF91.0% (1.9%)94.9%93.0% (1.0%)94.2%
Parsimonious Model using 10 features
LR-L186.2% (0.6%)83.8%90.4% (0.4%)89.1%
LR-L1
(binarized
model)
84.0% (0.6%)80.6%89.4% (0.1%)88.2%
Model using PSI or CURB-65 score
PSI score74.3% (1.2%)72.3%87.5% (0.2%)87.1%
CURB-65 score67.9% (1.3%)65.3%87.3% (0.2%)86.8%
Appendix 1—table 8
Derivation cohort performance for the restricted ICU prediction model.

Abbreviations and metrics reported are as in Table 1.

ICU prediction training performance with 628 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost89.6% (4.8%)92.5%85.4% (5.8%)87.6%
SVM-L180.1% (0.6%)80.8%79.4% (0.5%)80.4%
LR-L187.1% (0.8%)88.0%83.5% (0.5%)83.6%
RF95.6% (2.9%)95.7%91.0% (3.3%)90.2%
Models using 29 statistically selected features
XGBoost86.3% (1.0%)87.4%81.9% (0.4%)83.8%
SVM-L180.5% (0.9%)80.4%79.1% (0.5%)80.4%
LR-L180.9% (1.0%)81.6%79.0% (0.3%)80.3%
RF89.8% (2.6%)92.8%85.0% (1.9%)88.2%
Parsimonious Model using 8 features
LR-L180.4% (0.9%)81.4%79.7% (0.5%)80.0%
LR-L1
(binarized
model)
75.4% (1.1%)77.2%75.2% (0.7%)77.5%
Model using PSI or CURB-65 score
PSI score60.5% (1.7%)59.0%68.6% (0.5%)68.7%
CURB-65 score60.2% (1.2%)57.2%67.5% (0.4%)67.3%
Appendix 1—table 9
Derivation cohort performance for the ventilation prediction model.

Abbreviations and metrics reported are as in Table 1.

Ventilation prediction training performance with 2525 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost97.2% (1.5%)95.2%95.8% (1.0%)94.5%
SVM-L192.3% (0.7%)92.8%93.1% (0.1%)93.4%
LR-L193.8% (0.6%)94.3%93.3% (0.2%)93.2%
RF95.1% (0.8%)94.7%95.4% (0.5%)94.3%
Models using 55 statistically selected features
XGBoost96.9% (1.4%)98.3%95.6% (0.9%)96.6%
SVM-L190.8% (0.7%)91.3%92.7% (0.2%)93.0%
LR-L191.4% (0.7%)92.0%92.6% (0.3%)92.8%
RF94.8% (0.7%)94.1%95.5% (0.3%)94.8%
Parsimonious Model using 8 features
LR-L186.9% (0.5%)88.1%91.6% (0.2%)91.9%
LR-L1
(binarized
model)
84.4% (0.7%)86.7%91.1% (0.2%)91.2%
Model using PSI or CURB-65 score
PSI score74.0% (1.0%)71.4%89.9% (0.1%)89.6%
CURB-65 score67.6% (0.8%)64.7%89.7% (0.0%)89.6%
Appendix 1—table 10
Derivation cohort performance for the restricted ventilation prediction model. Abbreviations and metrics reported are as in Table 1.
Ventilation prediction training performance with 635 patients
AlgorithmAUCF1-weighted
RandomBWHRandomBWH
Models using all 130 features
XGBoost91.8% (2.2%)98.6%87.4% (2.0%)95.3%
SVM-L181.2% (0.7%)83.2%82.4% (1.1%)83.9%
LR-L189.7% (0.6%)89.6%86.9% (1.0%)85.8%
RF93.5% (4.2%)93.7%89.5% (3.8%)89.7%
Models using 29 statistically selected features
XGBoost89.9% (2.3%)89.9%86.1% (1.6%)86.0%
SVM-L181.5% (1.6%)84.4%82.2% (1.2%)83.7%
LR-L182.6% (0.7%)84.0%83.0% (0.9%)83.6%
RF92.3% (4.8%)94.3%88.8% (3.7%)89.3%
Parsimonious Model using 5 features
LR-L180.3% (1.0%)79.0%82.1% (0.7%)81.7%
LR-L1
(binarized
model)
73.1% (1.4%)66.5%78.3% (0.9%)73.5%
Model using PSI or CURB-65 score
PSI score58.8% (1.0%)57.2%73.9% (0.3%)74.2%
CURB-65 score58.5% (1.7%)55.8%73.2% (0.1%)73.7%
Appendix 1—table 11
AUC and weighted F1-score on an extended BWH test set, where patients with lab-to outcome time smaller than or equal to certain gaps are excluded.
Time gap6hr12 hr18 hr24 hr48 hr
Restricted ICU model - AUC86.05%84.73%86.85%86.14%84.62%
Restricted ICU model - weighted-F183.10%82.17%86.47%86.09%86.28%
Restricted intubation model - AUC68.00%64.44%63.85%63.85%64.34%
Restricted intubation model - weighted-F165.75%66.59%69.81%69.81%72.33%

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)