Healthcare providers use clinical decision support tools to assist with patient diagnosis, often to improve accuracy of diagnosis, reduce cost by avoiding unnecessary laboratory tests, and in the case of infectious diseases, deter the inappropriate prescription of antibiotics (Sintchenko et al., 2008). Typically, data entered into these tools is related directly to the patient’s individual characteristics, but data sources external to the patient can be informative for diagnosis. For example, climate, seasonality, and epidemiological data inform predictive models for communicable disease incidence (Colwell, 1996, Chao et al., 2019 Fine et al., 2011). The emergence of advanced computing and machine learning has enabled the incorporation of large data sources in the development of clinical support tools (Shortliffe and Sepúlveda, 2018) such as SMART-COP for predicting the need for intensive respiratory support for pneumonia (Charles et al., 2008) or the ALaRMS model for predicting inpatient mortality (Tabak et al., 2014).

Clinical decision support tools rely on the availability of information sources and computing at the time of patient encounter. Although increased availability of internet/mobile phones have increased access to information and computing power in low-resource settings, there may be times when connectivity, computing power, or data-collection infrastructure is unavailable. Thus, there is a need to build clinical decision support tools which can flexibly include features of external sources when available, or function without them if unavailable. Methods that enable the dynamic updating of predictive models are advantageous due to potential cyclical patterns of infectious etiologies. Furthermore, with the emergence of point-of-care (POC) tests for clinical decision-making (Price, 2001), predictive models that are able to integrate results of such diagnostic testing could enhance their usefulness.

We develop a novel method for diagnostic prediction which integrates multiple data sources by utilizing a post-test odds formulation with proof-of-concept in antibiotic stewardship for pediatric diarrhea. Our formulation first fits separate models from different sources of data, and then combines the likelihood ratios from each of these independent models into a single prediction. This method allows the multiple components to be flexibly included or excluded. We apply this method to the prediction of diarrhea etiology with data from the Global Enteric Multicenter Study (GEMS) (Kotloff et al., 2013) and assess the performance of this tool, including with the addition of a synthetic diagnostic, using two forms of internal-validation and by showing its potential effect on reducing inappropriate antibiotic use.

We present our approach to building and assessing a flexible multi-source clinical prediction tool with (1) the data sources, (2) the individual prediction models, (3) the use of the likelihood ratio for integrating predictive models, (4) validation of the method, (5) the impact of an additional diagnostic, and (6) a simulation of conditionally dependent tests. We program our prediction tool using R version 3.6.2 (R Project for Statistical Computing, RRID:SCR_001905).

Data sources

Request a detailed protocol

We apply our post-test odds model using clinical data from GEMS, a prospective, case-control study from 2007 to 2011 which took place in seven countries in Africa and Asia. Methods for the GEMS study have been described in detail (Kotloff et al., 2012). Briefly, 9439 children with moderate-to-severe diarrhea were enrolled at local health care centers along with one to three matched control-children. A fecal sample was taken from each child at enrollment to identify enteropathogens and clinical information was collected, including demographic, anthropometric, and clinical history of the child. We used the quantitative real-time PCR-based (qPCR) attribution models developed by Liu et al., 2016 in order to best characterize the cause of diarrhea. Our dependent variable was presence or absence of viral etiology, defined as a diarrhea episode with at least one viral pathogen with an episode-specific attributable fraction (AFe ≥ 0.5) and no bacterial or parasitic pathogens with an episode-specific attributable fraction. Prediction of viral attribution is clinically meaningful since it indicates that a patient would not benefit from antimicrobial therapy. We defined other known etiologies as having a majority attribution of diarrhea episode by at least one other non-viral pathogen. We exclude patients with unknown etiologies when fitting the model, though it has been previously shown that these cases have a similar distribution of viral predictions using a model with presenting patient information as those cases with known etiologies (Brintz et al., 2020).

We obtained weather data local to each site’s health centers during the GEMS study using NOAA’s Integrated Surface Database (Smith et al., 2011). The incidence of many pathogens, including rotavirus (Cook et al., 1990), norovirus (Ahmed et al., 2013), cholera (Emch et al., 2008), and Salmonella (Mohanty et al., 2006), are known to have seasonal patterns, and other analyses have established climatic factors to be associated with diarrheal diseases (Colwell, 1996, Chao et al., 2019, Farrar et al., 2019). Stations near GEMS sites such as in The Gambia exhibit seasonal patterns (Figure 1). We used daily temperature and rain data weighted most by those weather stations closest to the GEMS sites (Appendix 1).

Figure 1 with 2 supplements see all

Download asset Open asset

Use of the likelihood ratio to integrate predictive models from multiple data sources

We integrate predictive models from the multiple sources of data described above using the post-test odds formulation. Using Bayes’ Theorem, $P(A|B)=\frac{P(B|A)\cdot P(A)}{P(B)}$, to construct the post-test odds of having a viral etiology,

${\displaystyle \begin{array}{}\text{(1)}& \frac{P(V=1|T_1=t_1,T_2=t_2,\cdots ,T_k=t_k)}{P(V=0|T_1=t_1,T_2=t_2,\cdots ,T_k=t_k)}& =\frac{P(V=1,T_1=t_1,T_2=t_2,\cdots ,T_k=t_k)}{P(V=0,T_1=t_1,T_2=t_2,\cdots ,T_k=t_k)}& \text{(2)}& & =\frac{P(T_1=t_1,T_2=t_2,\cdots ,T_k=t_k|V=1)\cdot P(V=1)}{P(T_1=t_1,T_2=t_2,\cdots ,T_k=t_k|V=0)\cdot P(V=0)}& \text{(3)}& & =\frac{P(V=1)}{P(V=0)}\cdot \prod _{j=1}^k\frac{P(T_j=t_j|V=1)}{P(T_j=t_j|V=0)}& \end{array}}$

where $V=1$ represents a viral etiology and $V=0$ represents an other known etiology, $T_1,T_2,\mathrm{\cdots },T_k$ represent the k tests, the distribution of the predictions from one or more predictive models, used to obtain the post-test odds, and $\frac{P(V=1)}{P(V=0)}$ is the pre-test odds. Note that going from line (2) to line (3) requires conditional independence between the tests, that is, that ${\displaystyle P(T_i=t_i,T_j=t_j|V=1)}$ = ${\displaystyle P(T_i=t_i|V=1)\cdot P(T_j=t_j|V=1)}$ and $P(T_i=t_i,T_j=t_j|V=0)=P(T_i=t_i|V=0)\cdot P(T_j=t_j|V=0)$ for all i and j. We test for conditional independence to assess the necessity of making higher-dimensional kernel density estimates using the $ci.test$ function from the $\{bnlearn\}$ package in R (Scutari, 2010). We derive each $P(T_j=t_j|V=1)$ and $P(T_j=t_j|V=0)$ using Gaussian kernel density estimates on conditional predictions from a logistic regression model fit on the training set (Silverman, 1986). The distribution of $P(T_j|V)$ is derived using the kernel density estimator $f(t_j)=\frac{1}{nh}{\sum }_{i=1}^{n}K(\frac{t_j-x_i}{h})$ where, in our case, $K(x)=\varphi (x)$, the standard normal density function, and the bandwidth, h, is Silverman’s 'rule of thumb' and the default chosen in the $density$ function in R (Parzen, 1962).

Figure 2 shows an example of the frequency of predictions from a logistic regression model conditional on the viral-only status (V = 0 and V = 1) determined from attributable fractions. Additionally, we overlaid the estimated one-dimensional kernel density. To obtain the value of $\frac{P(T_j=t_j|V=1)}{P(T_j=t_j|V=0)}$, the predicted odds, from a model’s prediction, we divide the kernel density estimate from the $V=1$ set (right) by the kernel density estimate from the $V=0$ set (left). It is feasible to estimate a multi-dimensional kernel density so that it is not necessary to make the conditional independence assumption to move from line 2 to line 3 in the equation above. Figure 2—figure supplement 1 shows an example two-dimensional contour plot for kernel density estimates of predicted values obtained from logistic regression on GEMS seasonality and climate data in Mali which we will discuss further below. The density was created using R function $kde2d$ (Venables and Ripley, 2002).

Figure 2 with 1 supplement see all

Download asset Open asset

Pre-test odds from historical data

Request a detailed protocol

We calculated pre-test odds using historical rates of viral diarrhea by site and date. We utilize available diarrhea etiology data for a given date, regardless of year, and site using a moving average such that pre-test probability ${\pi }_d$ for date d is

${\displaystyle \begin{array}{rl}{\displaystyle {\pi }_d}& {\displaystyle =\frac{D_{d-n}+D_{d-n+1}+\cdots +D_d+\cdots +D_{d+n-1}+D_{d+n}}{k_{d-n}+k_{d-n+1}+\cdots +k_d+\cdots +k_{d+n-1}+k_{d+n}}}\\ {\displaystyle D_d}& {\displaystyle ={\mathrm{\Sigma }}_{i=1}^{k_d}{D}_{di}}\end{array}}$

where k_d is the number of observed patients on date d, $D_{di}$ is 1 if the etiology of the patients’ diarrhea is viral and 0 otherwise, and n is the number of days included on both sides of the moving average. We would expect ${\pi }_d$ to represent a pre-test probability of observing a viral diarrhea etiology on date d. Given that this rate information will likely be unavailable in new sites without established etiology studies, we provide an alternative formula based on recent patients’ presentations (Appendix 2). Additionally, we include a sensitivity analysis by calculating pre-test odds using conventional diagnostic methods data as qPCR data are unlikely to be available in high-burden settings.

Validating the method

Request a detailed protocol

Given the temporal nature of some of the tests we developed, we estimate model performance using within rolling-origin-recalibration evaluation. This method evaluates a model by sequentially moving values from a test set to a training set and re-training the model on all of the training set (Bergmeir and Benítez, 2012); for example, we train on the first 70% of the data and test on the remaining 30%, then train on the first 80% of the data and test on the remaining 20%. No data from the training set is used as part of the prediction for the test set. In each iteration of evaluation, predictions on the test set are produced and corresponding measures of performance obtained: the receiver operating characteristic (ROC) curve, and area under the ROC curve (AUC), also known as the C-statistic, along with AUC confidence intervals (LeDell et al., 2015). Figure 3 depicts one iteration of within rolling-origin-recalibration evaluation.

Figure 3

Download asset Open asset

We additionally include a joint density for the climate and seasonal data in which we estimate a two-dimensional kernel density (not shown in Figure 3). This model is called ”Joint’ in the results to follow. To assess how this model might generalize to a site that was not used for model training, we used a leave-one-site-out validation. By excluding a site and training the model’s tests at a higher level, such as on the entire continent, we get an idea of performance at a new site within one of the continents for which we have data. Lastly, we define a threshold for the predicted odds ratio based on the desired specificity of the model. We use this threshold to evaluate the effect of the model on prescription or treatment of patients with antibiotics in the GEMS data.

The management of illness in much of the world relies on clinical decisions made in the absence of laboratory diagnostics. Such empirical decision-making, including decisions to use antibiotics, are informed by variable degrees of clinical and demographic data gathered by the clinician. Traditional clinical prediction rules focus on the clinical data from the presenting patient alone. In this analysis, we present a method that allows flexible integration of multiple data sources, including climate data and clinical or historical information from prior patients, resulting in improved predictive performance over traditional predictive models utilizing a single source of data. Using this formulation, if certain sources of data such as climate or previous patient information are not available (e.g. due to a lack of internet connection or data infrastructure), the prediction can still be made using current patient information or seasonality, as appropriatethe other sources. A mobile phone application is an ideal platform for a decision support tool implemented in low-resource settings. Through internet access by wifi or cellular data, a smartphone platform could automatically download recent patient or climate data, while its portability would facilitate clinicians in entering current patient clinical information. We show that application of such a predictive model, especially with an additional diagnostic test, may translate to reductions in inappropriate antibiotic prescriptions for pediatric viral diarrhea.

The global burden of acute infectious diarrhea is highest in low- and middle-income countries (LMICs) in southeast Asia and Africa (Walker et al., 2013), where there is limited access to diagnostic testing. The care of children in these regions could greatly benefit from an accurate and flexible decision making tool. Decisions for treatment are often empiric and antibiotics are over-prescribed (Rogawski et al., 2017), although the majority of cases of diarrhea do not benefit from antibiotic use and also many instances of acute watery diarrhea are self-limiting . For example, 2653 (79%) of the 3366 patients in our study were treated with or prescribed antibiotics. Of these 806 (30%) were prescribed to those with a viral-only etiology. Unnecessary antibiotic use exposes children to significant adverse events including serious allergic reactions (Logan et al., 2016, Marra et al., 2009) and clostridium difficile infection (Jernberg et al., 2010), and contributes to increased antimicrobial resistance. We show that a predictive model can be used to discriminate between those with and without a viral-only etiology and that the inappropriate use of antibiotics can be avoided in 54% cases using our model with no additional diagnostics.

We found using within rolling-origin-recalibration evaluation that models which include either the pre-test odds calculated historical rates or the seasonal test were the best at discriminating between viral etiologies and other etiologies, a finding that held true across training and testing set sizes. However, in the leave-one-out validation, models which included the alternate pre-test odds and climate tended to perform the best. This difference is likely due to the generalizeability of the individual tests, i.e, the leave-one-out tests are trained at the continental level and the effect of climate on etiology is intuitively more generalizeable than seasonal curves which are very specific to each location. We found that our integrative model with only the historical (pre-test) information included (without additional diagnostics) would have identified a viral-only etiology in 88 (53%) patients who received antibiotics. We then show that even the use of an additional diagnostic test with modest performance (70% sensitivity and specificity) would further decrease inappropriate antibiotic use by another 14 (for a total of 102, or 61% of) patients. In the context of calls by the WHO for the development of affordable rapid diagnostic tools (RDTs) for antibiotic stewardship (Declaration, 2017), our findings suggest that development and evaluation of novel RDTs should not be performed in isolation. Potential for integration of rapid diagnostic tests into clinical prediction algorithms should be considered, although this needs to be balanced with the additional time and resources needed. The incremental improvement in discriminative performance achieved by the addition of an RDT to a clinical prediction algorithm may not be cost-effective in lower resourced settings. Finally, providing this model in the form of a decision support tool to the clinician could translate to reductions in inappropriate use of antibiotics, although further research needs to be done to explore the degrees of certainty that clinicians require to alter treatment decisions.

The novel use of kernel density estimates to derive the conditional tests when calculating the post-test odds enabled a flexibility in model input. While kernel density estimates have been used for conditional feature distributions in Naïve Bayes classifiers (John and Langley, 1995, Murakami and Mizuguchi, 2010), here we show that they can be used to derive conditional likelihoods for diagnostic tests constituting one or more features, stressing the effect of the overall test on the post-test odds and not individual features. As such, complicated machine learning models can be combined with simple diagnostics as part of the post-test odds. For example, we could have fit neural networks in lieu of logistic regression models, and in addition to these more complicated models, it is possible to incorporate the result of an RDT that make results available to the clinician at the point-of-care. Additionally, our method of using two-dimensional kernel density estimates can also be used to overcome the conditional independence assumption for tests based on potentially interrelated diagnostic information. Densities with higher than two dimensions can be considered, though, computational limitations are likely in both speed and, we expect, accuracy, as the dimensions increase.

Our study has a number of limitations. First, a robust training set of both cases and non-cases is required to adequately build the conditional kernel densities. Second, the post-test odds calculation, at the time of prediction, lacks interpretation on a feature level like a logistic regression or decision tree. Although, we do observe the effect of a test on an observation, we cannot see which features caused that effect without diving deeper into the training of the diagnostic tests.Thirdly, the prediction algorithm generated by the post-test odds model using GEMS data was only validated internally, and further studies are need for external validation and field implementation. Fourth, our estimation of antibiotic use reduction used data from a clinical research study, which may have biases inherent to such studies. Last, our study uses the AFe cut-off of greater than or equal to 0.5 to assign etiology from the qPCR data. This cutoff was selected based on expert elicitation, but the effect of using this cut-off has not been explored. Bacterial cases with AFe¡0.5 were excluded in our analysis, but may still benefit from antibiotic treatment.

In conclusion, we have developed a clinical prediction model that integrates multiple sources external to the presenting patient, through use of a post-test odds framework and showed that it improved diagnostic performance. When applied to the etiological diagnosis of pediatric diarrhea, we demonstrate its potential for reducing inappropriate antibiotic use. The flexible inclusion or exclusion of output from its components makes it ideal for decision support in lower resourced settings, when only certain data may be available due to limitations in information computation or connectivity. Additionally, the ability to incorporate new training data in real-time to update decisions allows the model to improve as more data is collected. Such a predictive model has the potential to improve the management of pediatric diarrhea, including the rational use of antibiotics in lower resourced settings.

GEMS data are available to the public from https://clinepidb.org/ce/app/. Data and code needed to reproduce all parts of this analysis are available from the corresponding author's GitHub page: https://github.com/LeungLab/GEMS_PostTestOdds [copy archived at https://archive.softwareheritage.org/swh:1:rev:67f4f5a0cdc3d569e756142f0142aaa23a9b1e03/].

1. 1. Gates Enterics Project
   2. Levine MM
   3. Kotloff K
   4. Nataro J
   5. Khan AZA
   6. Saha D
   7. Adegbola FR
   8. Sow S
   9. Alonso P
   10. Breiman R
   11. Sur D
   12. Faruque A

   (2018) ClinEpiDB

   ID DS_841a9f5259. Study GEMS1 Case Control.

   https://clinepidb.org/ce/app/record/dataset/DS_841a9f5259#Contacts

1. 1. Ahmed SM
   2. Lopman BA
   3. Levy K

   (2013) A systematic review and meta-analysis of the global seasonality of Norovirus

   PLOS ONE 8:e75922.

   https://doi.org/10.1371/journal.pone.0075922

   • PubMed
   • Google Scholar
2. 1. Bergmeir C
   2. Benítez JM

   (2012) On the use of cross-validation for time series predictor evaluation

   Information Sciences 191:192–213.

   https://doi.org/10.1016/j.ins.2011.12.028

   • Google Scholar
3. 1. Brintz BJ
   2. Howard JI
   3. Haaland B
   4. Platts-Mills JA
   5. Greene T
   6. Levine AC
   7. Nelson EJ
   8. Pavia AT
   9. Kotloff KL
   10. Leung DT

   (2020) Clinical predictors for etiology of acute diarrhea in children in resource-limited settings

   PLOS Neglected Tropical Diseases 14:e0008677.

   https://doi.org/10.1371/journal.pntd.0008677

   • PubMed
   • Google Scholar
4. 1. Chao DL
   2. Roose A
   3. Roh M
   4. Kotloff KL
   5. Proctor JL

   (2019) The seasonality of diarrheal pathogens: a retrospective study of seven sites over three years

   PLOS Neglected Tropical Diseases 13:e0007211.

   https://doi.org/10.1371/journal.pntd.0007211

   • PubMed
   • Google Scholar
5. 1. Charles PG
   2. Wolfe R
   3. Whitby M
   4. Fine MJ
   5. Fuller AJ
   6. Stirling R
   7. Wright AA
   8. Ramirez JA
   9. Christiansen KJ
   10. Waterer GW
   11. Pierce RJ
   12. Armstrong JG
   13. Korman TM
   14. Holmes P
   15. Obrosky DS
   16. Peyrani P
   17. Johnson B
   18. Hooy M
   19. Grayson ML
   20. Australian Community-Acquired Pneumonia Study Collaboration

   (2008) SMART-COP: a tool for predicting the need for intensive respiratory or vasopressor support in community-acquired pneumonia

   Clinical Infectious Diseases 47:375–384.

   https://doi.org/10.1086/589754

   • PubMed
   • Google Scholar
6. 1. Colwell RR

   (1996) Global climate and infectious disease: the cholera paradigm

   Science 274:2025–2031.

   https://doi.org/10.1126/science.274.5295.2025

   • PubMed
   • Google Scholar
7. 1. Cook SM
   2. Glass RI
   3. LeBaron CW
   4. Ho MS

   (1990)

   Global seasonality of Rotavirus infections

   Bulletin of the World Health Organization 68:171.

   • PubMed
   • Google Scholar
8. Report

   1. Declaration D

   (2017)

   Cidrap Antimicrobial Stewardship Project Policy Update, October 2017

   Policy.

   • Google Scholar
9. 1. Emch M
   2. Feldacker C
   3. Islam MS
   4. Ali M

   (2008) Seasonality of cholera from 1974 to 2005: a review of global patterns

   International Journal of Health Geographics 7:31.

   https://doi.org/10.1186/1476-072X-7-31

   • PubMed
   • Google Scholar
10. 1. Farrar DS
    2. Awasthi S
    3. Fadel SA
    4. Kumar R
    5. Sinha A
    6. Fu SH
    7. Wahl B
    8. Morris SK
    9. Jha P

    (2019) Seasonal variation and etiologic inferences of childhood pneumonia and diarrhea mortality in India

    eLife 8:e46202.

    https://doi.org/10.7554/eLife.46202

    • PubMed
    • Google Scholar
11. 1. Fine AM
    2. Brownstein JS
    3. Nigrovic LE
    4. Kimia AA
    5. Olson KL
    6. Thompson AD
    7. Mandl KD

    (2011) Integrating spatial epidemiology into a decision model for evaluation of facial palsy in children

    Archives of Pediatrics & Adolescent Medicine 165:61–67.

    https://doi.org/10.1001/archpediatrics.2010.250

    • PubMed
    • Google Scholar
12. 1. Fontana M
    2. Zuin G
    3. Paccagnini S
    4. Ceriani R
    5. Quaranta S
    6. Villa M
    7. Principi N

    (1987) Simple clinical score and laboratory-based method to predict bacterial etiology of acute diarrhea in childhood

    The Pediatric Infectious Disease Journal 6:1088–1091.

    https://doi.org/10.1097/00006454-198706120-00004

    • PubMed
    • Google Scholar
13. 1. Jernberg C
    2. Löfmark S
    3. Edlund C
    4. Jansson JK

    (2010) Long-term impacts of antibiotic exposure on the human intestinal Microbiota

    Microbiology 156:3216–3223.

    https://doi.org/10.1099/mic.0.040618-0

    • PubMed
    • Google Scholar
14. Conference

    1. John GH
    2. Langley P

    (1995)

    Estimating continuous distributions in bayesian classifiers

    Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann Publishers Inc. pp. 338–345.

    • Google Scholar
15. 1. Kotloff KL
    2. Blackwelder WC
    3. Nasrin D
    4. Nataro JP
    5. Farag TH
    6. van Eijk A
    7. Adegbola RA
    8. Alonso PL
    9. Breiman RF
    10. Faruque AS
    11. Saha D
    12. Sow SO
    13. Sur D
    14. Zaidi AK
    15. Biswas K
    16. Panchalingam S
    17. Clemens JD
    18. Cohen D
    19. Glass RI
    20. Mintz ED
    21. Sommerfelt H
    22. Levine MM

    (2012) The global enteric multicenter study (GEMS) of diarrheal disease in infants and young children in developing countries: epidemiologic and clinical methods of the case/control study

    Clinical Infectious Diseases 55 Suppl 4:S232–S245.

    https://doi.org/10.1093/cid/cis753

    • PubMed
    • Google Scholar
16. 1. Kotloff KL
    2. Nataro JP
    3. Blackwelder WC
    4. Nasrin D
    5. Farag TH
    6. Panchalingam S
    7. Wu Y
    8. Sow SO
    9. Sur D
    10. Breiman RF
    11. Faruque ASG
    12. Zaidi AKM
    13. Saha D
    14. Alonso PL
    15. Tamboura B
    16. Sanogo D
    17. Onwuchekwa U
    18. Manna B
    19. Ramamurthy T
    20. Kanungo S
    21. Ochieng JB
    22. Omore R
    23. Oundo JO
    24. Hossain A
    25. Das SK
    26. Ahmed S
    27. Qureshi S
    28. Quadri F
    29. Adegbola RA
    30. Antonio M
    31. Hossain MJ
    32. Akinsola A
    33. Mandomando I
    34. Nhampossa T
    35. Acácio S
    36. Biswas K
    37. O'Reilly CE
    38. Mintz ED
    39. Berkeley LY
    40. Muhsen K
    41. Sommerfelt H
    42. Robins-Browne RM
    43. Levine MM

    (2013) Burden and aetiology of diarrhoeal disease in infants and young children in developing countries (the global enteric multicenter study, GEMS): a prospective, case-control study

    The Lancet 382:209–222.

    https://doi.org/10.1016/S0140-6736(13)60844-2

    • PubMed
    • Google Scholar
17. 1. LeDell E
    2. Petersen M
    3. van der Laan M

    (2015) Computationally efficient confidence intervals for cross-validated area under the ROC curve estimates

    Electronic Journal of Statistics 9:1583–1607.

    https://doi.org/10.1214/15-EJS1035

    • PubMed
    • Google Scholar
18. 1. Liu J
    2. Platts-Mills JA
    3. Juma J
    4. Kabir F
    5. Nkeze J
    6. Okoi C
    7. Operario DJ
    8. Uddin J
    9. Ahmed S
    10. Alonso PL
    11. Antonio M
    12. Becker SM
    13. Blackwelder WC
    14. Breiman RF
    15. Faruque ASG
    16. Fields B
    17. Gratz J
    18. Haque R
    19. Hossain A
    20. Hossain MJ
    21. Jarju S
    22. Qamar F
    23. Iqbal NT
    24. Kwambana B
    25. Mandomando I
    26. McMurry TL
    27. Ochieng C
    28. Ochieng JB
    29. Ochieng M
    30. Onyango C
    31. Panchalingam S
    32. Kalam A
    33. Aziz F
    34. Qureshi S
    35. Ramamurthy T
    36. Roberts JH
    37. Saha D
    38. Sow SO
    39. Stroup SE
    40. Sur D
    41. Tamboura B
    42. Taniuchi M
    43. Tennant SM
    44. Toema D
    45. Wu Y
    46. Zaidi A
    47. Nataro JP
    48. Kotloff KL
    49. Levine MM
    50. Houpt ER

    (2016) Use of quantitative molecular diagnostic methods to identify causes of diarrhoea in children: a reanalysis of the GEMS case-control study

    The Lancet 388:1291–1301.

    https://doi.org/10.1016/S0140-6736(16)31529-X

    • PubMed
    • Google Scholar
19. 1. Logan AC
    2. Jacka FN
    3. Craig JM
    4. Prescott SL

    (2016) The microbiome and mental health: looking back, moving forward with lessons from allergic diseases

    Clinical Psychopharmacology and Neuroscience 14:131–147.

    https://doi.org/10.9758/cpn.2016.14.2.131

    • PubMed
    • Google Scholar
20. 1. Marra F
    2. Marra CA
    3. Richardson K
    4. Lynd LD
    5. Kozyrskyj A
    6. Patrick DM
    7. Bowie WR
    8. Fitzgerald JM

    (2009) Antibiotic use in children is associated with increased risk of asthma

    Pediatrics 123:1003–1010.

    https://doi.org/10.1542/peds.2008-1146

    • PubMed
    • Google Scholar
21. 1. Mohanty S
    2. Renuka K
    3. Sood S
    4. Das BK
    5. Kapil A

    (2006) Antibiogram pattern and seasonality of Salmonella serotypes in a north indian tertiary care hospital

    Epidemiology and Infection 134:961–966.

    https://doi.org/10.1017/S0950268805005844

    • PubMed
    • Google Scholar
22. 1. Murakami Y
    2. Mizuguchi K

    (2010) Applying the naïve Bayes classifier with kernel density estimation to the prediction of protein-protein interaction sites

    Bioinformatics 26:1841–1848.

    https://doi.org/10.1093/bioinformatics/btq302

    • PubMed
    • Google Scholar
23. 1. Parzen E

    (1962) On estimation of a probability density function and mode

    The Annals of Mathematical Statistics 33:1065–1076.

    https://doi.org/10.1214/aoms/1177704472

    • Google Scholar
24. 1. Price CP

    (2001) Point of care testing

    BMJ 322:1285–1288.

    https://doi.org/10.1136/bmj.322.7297.1285

    • PubMed
    • Google Scholar
25. 1. Rogawski ET
    2. Platts-Mills JA
    3. Seidman JC
    4. John S
    5. Mahfuz M
    6. Ulak M
    7. Shrestha SK
    8. Soofi SB
    9. Yori PP
    10. Mduma E
    11. Svensen E
    12. Ahmed T
    13. Lima AA
    14. Bhutta ZA
    15. Kosek MN
    16. Lang DR
    17. Gottlieb M
    18. Zaidi AK
    19. Kang G
    20. Bessong PO
    21. Houpt ER
    22. Guerrant RL

    (2017) Use of antibiotics in children younger than two years in eight countries: a prospective cohort study

    Bulletin of the World Health Organization 95:49–61.

    https://doi.org/10.2471/BLT.16.176123

    • PubMed
    • Google Scholar
26. 1. Scutari M

    (2010) Learning bayesian networks with the bnlearn R package

    Journal of Statistical Software 35:1–22.

    https://doi.org/10.18637/jss.v035.i03

    • Google Scholar
27. 1. Shortliffe EH
    2. Sepúlveda MJ

    (2018) Clinical decision support in the era of artificial intelligence

    JAMA 320:2199–2200.

    https://doi.org/10.1001/jama.2018.17163

    • PubMed
    • Google Scholar
28. Book

    1. Silverman BW

    (1986) Density Estimation for Statistics and Data Analysis

    CRC press.

    https://doi.org/10.1201/9781315140919

    • Google Scholar
29. 1. Sintchenko V
    2. Coiera E
    3. Gilbert GL

    (2008) Decision support systems for antibiotic prescribing

    Current Opinion in Infectious Diseases 21:573–579.

    https://doi.org/10.1097/QCO.0b013e3283118932

    • PubMed
    • Google Scholar
30. 1. Smith A
    2. Lott N
    3. Vose R

    (2011) The integrated surface database: recent developments and partnerships

    Bulletin of the American Meteorological Society 92:704–708.

    https://doi.org/10.1175/2011BAMS3015.1

    • Google Scholar
31. 1. Stolwijk AM
    2. Straatman H
    3. Zielhuis GA

    (1999) Studying seasonality by using sine and cosine functions in regression analysis

    Journal of Epidemiology & Community Health 53:235–238.

    https://doi.org/10.1136/jech.53.4.235

    • PubMed
    • Google Scholar
32. 1. Tabak YP
    2. Sun X
    3. Nunez CM
    4. Johannes RS

    (2014) Using electronic health record data to develop inpatient mortality predictive model: acute laboratory risk of mortality score (ALaRMS)

    Journal of the American Medical Informatics Association 21:455–463.

    https://doi.org/10.1136/amiajnl-2013-001790

    • PubMed
    • Google Scholar
33. Book

    1. Venables WN
    2. Ripley BD

    (2002) Modern Applied Statistics with S

    Springer.

    https://doi.org/10.1007/978-0-387-21706-2

    • Google Scholar
34. 1. Walker CLF
    2. Rudan I
    3. Liu L
    4. Nair H
    5. Theodoratou E
    6. Bhutta ZA
    7. O'Brien KL
    8. Campbell H
    9. Black RE

    (2013) Global burden of childhood pneumonia and diarrhoea

    The Lancet 381:1405–1416.

    https://doi.org/10.1016/S0140-6736(13)60222-6

    • PubMed
    • Google Scholar

Author details

1. Ben J Brintz

   1. Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, United States
   2. Division of Infectious Diseases, Department of Internal Medicine, University of Utah, Salt Lake City, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   ben.brintz@hsc.utah.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4695-0290

2. Benjamin Haaland

   Population Health Sciences, University of Utah, Salt Lake City, United States

   Contribution

   Conceptualization, Visualization, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

3. Joel Howard

   Division of Pediatric Infectious Diseases, University of Utah, Salt Lake City, United States

   Contribution

   Conceptualization, Writing - review and editing

   Competing interests

   No competing interests declared

4. Dennis L Chao

   Institute of Disease Modeling, Bill and Melinda Gates Foundation, Seattle, United States

   Contribution

   Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

5. Joshua L Proctor

   Institute of Disease Modeling, Bill and Melinda Gates Foundation, Seattle, United States

   Contribution

   Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

6. Ashraful I Khan

   International Centre for Diarrhoeal Disease Research, Bangladesh, Dhaka, Bangladesh

   Contribution

   Writing - review and editing

   Competing interests

   No competing interests declared

7. Sharia M Ahmed

   Division of Infectious Diseases, Department of Internal Medicine, University of Utah, Salt Lake City, United States

   Contribution

   Writing - review and editing

   Competing interests

   No competing interests declared

8. Lindsay T Keegan

   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, United States

   Contribution

   Writing - review and editing

   Competing interests

   No competing interests declared

9. Tom Greene

   Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, United States

   Contribution

   Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

10. Adama Mamby Keita

    Centre Pour le Développement des Vaccins-Mali, Bamako, Mali

    Contribution

    Writing - review and editing

    Competing interests

    No competing interests declared

11. Karen L Kotloff

    Division of Infectious Disease and Tropical Pediatrics, University of Maryland, Baltimore, United States

    Contribution

    Resources, Writing - review and editing

    Competing interests

    No competing interests declared

12. James A Platts-Mills

    Division of Infectious Diseases and International Health, University of Virginia, Charlottesville, United States

    Contribution

    Data curation, Writing - review and editing

    Competing interests

    No competing interests declared

13. Eric J Nelson

    1. Departments of Pediatrics, University of Florida, Gainesville, United States
    2. Departments of Environmental and Global Health, University of Florida, Gainesville, United States

    Contribution

    Writing - review and editing

    Competing interests

    No competing interests declared

14. Adam C Levine

    Department of Emergency Medicine, Brown University, Providence, United States

    Contribution

    Writing - review and editing

    Competing interests

    No competing interests declared

15. Andrew T Pavia

    Division of Pediatric Infectious Diseases, University of Utah, Salt Lake City, United States

    Contribution

    Writing - review and editing

    Competing interests

    No competing interests declared

16. Daniel T Leung

    1. Division of Infectious Diseases, Department of Internal Medicine, University of Utah, Salt Lake City, United States
    2. Division of Microbiology and Immunology, Department of Internal Medicine, University of Utah, Salt Lake City, United States

    Contribution

    Conceptualization, Resources, Supervision, Funding acquisition, Writing - review and editing

    For correspondence

    daniel.leung@utah.edu

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-8401-0801

• 1,118

  views

• 131

  downloads

• 9

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.