The Ellipse of Insignificance, a refined fragility indexfor ascertaining robustness of results indichotomous outcome trials

  1. David Robert Grimes  Is a corresponding author
  1. Dublin City University, Ireland

Abstract

There is increasing awareness throughout biomedical science that many results do not withstand the trials of repeat investigation. The growing abundance of medical literature has only increased the urgent need for tools to gauge the robustness and trustworthiness of published science. Dichotomous outcome designs are vital in randomized clinical trials, cohort studies, and observational data for ascertaining differences between experimental and control arms. It has however been shown with tools like the fragility index (FI) that many ostensibly impactful results fail to materialise when even small numbers of patients or subjects in either the control or experimental arms are recoded from event to non-event. Critics of this metric counter that there is no objective means to determine a meaningful FI. As currently used, FI is not multi-dimensional and is computationally expensive. In this work a conceptually similar geometrical approach is introduced, the ellipse of insignificance (EOI). This method yields precise deterministic values for the degree of manipulation or miscoding that can be tolerated simultaneously in both control and experimental arms, allowing for the derivation of objective measures of experimental robustness. More than this, the tool is intimately connected with sensitivity and specificity of the event / non-event tests, and is readily combined with knowledge of test parameters to reject unsound results. The method is outlined here, with illustrative clinical examples.

Data availability

The paper is a modelling study and methodology and contains no data, and code provided in the supplementary material allows reproduction of all methods.

Article and author information

Author details

  1. David Robert Grimes

    Dublin City University, Dublin, Ireland
    For correspondence
    davidrobert.grimes@dcu.ie
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3140-3278

Funding

Wellcome Trust (214461/A/18/Z)

  • David Robert Grimes

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Reviewing Editor

  1. Philip Boonstra, University of Michigan, United States

Publication history

  1. Received: April 19, 2022
  2. Accepted: September 13, 2022
  3. Accepted Manuscript published: September 20, 2022 (version 1)

Copyright

© 2022, Grimes

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 48
    Page views
  • 26
    Downloads
  • 0
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. David Robert Grimes
(2022)
The Ellipse of Insignificance, a refined fragility indexfor ascertaining robustness of results indichotomous outcome trials
eLife 11:e79573.
https://doi.org/10.7554/eLife.79573

Further reading

    1. Epidemiology and Global Health
    2. Evolutionary Biology
    Alexandra Blenkinsop, Mélodie Monod ... Oliver Ratmann
    Research Article Updated

    Background:

    More than 300 cities including the city of Amsterdam in the Netherlands have joined the UNAIDS Fast-Track Cities initiative, committing to accelerate their HIV response and end the AIDS epidemic in cities by 2030. To support this commitment, we aimed to estimate the number and proportion of Amsterdam HIV infections that originated within the city, from Amsterdam residents. We also aimed to estimate the proportion of recent HIV infections during the 5-year period 2014–2018 in Amsterdam that remained undiagnosed.

    Methods:

    We located diagnosed HIV infections in Amsterdam using postcode data (PC4) at time of registration in the ATHENA observational HIV cohort, and used HIV sequence data to reconstruct phylogeographically distinct, partially observed Amsterdam transmission chains. Individual-level infection times were estimated from biomarker data, and used to date the phylogenetically observed transmission chains as well as to estimate undiagnosed proportions among recent infections. A Bayesian Negative Binomial branching process model was used to estimate the number, size, and growth of the unobserved Amsterdam transmission chains from the partially observed phylogenetic data.

    Results:

    Between 1 January 2014 and 1 May 2019, there were 846 HIV diagnoses in Amsterdam residents, of whom 516 (61%) were estimated to have been infected in 2014–2018. The rate of new Amsterdam diagnoses since 2014 (104 per 100,000) remained higher than the national rates excluding Amsterdam (24 per 100,000), and in this sense Amsterdam remained a HIV hotspot in the Netherlands. An estimated 14% [12–16%] of infections in Amsterdan MSM in 2014–2018 remained undiagnosed by 1 May 2019, and 41% [35–48%] in Amsterdam heterosexuals, with variation by region of birth. An estimated 67% [60–74%] of Amsterdam MSM infections in 2014–2018 had an Amsterdam resident as source, and 56% [41–70%] in Amsterdam heterosexuals, with heterogeneity by region of birth. Of the locally acquired infections, an estimated 43% [37–49%] were in foreign-born MSM, 41% [35–47%] in Dutch-born MSM, 10% [6–18%] in foreign-born heterosexuals, and 5% [2–9%] in Dutch-born heterosexuals. We estimate the majority of Amsterdam MSM infections in 2014–2018 originated in transmission chains that pre-existed by 2014.

    Conclusions:

    This combined phylogenetic, epidemiologic, and modelling analysis in the UNAIDS Fast-Track City Amsterdam indicates that there remains considerable potential to prevent HIV infections among Amsterdam residents through city-level interventions. The burden of locally acquired infection remains concentrated in MSM, and both Dutch-born and foreign-born MSM would likely benefit most from intensified city-level interventions.

    Funding:

    This study received funding as part of the H-TEAM initiative from Aidsfonds (project number P29701). The H-TEAM initiative is being supported by Aidsfonds (grant number: 2013169, P29701, P60803), Stichting Amsterdam Dinner Foundation, Bristol-Myers Squibb International Corp. (study number: AI424-541), Gilead Sciences Europe Ltd (grant number: PA-HIV-PREP-16-0024), Gilead Sciences (protocol numbers: CO-NL-276-4222, CO-US-276-1712, CO-NL-985-6195), and M.A.C AIDS Fund.

    1. Epidemiology and Global Health
    2. Microbiology and Infectious Disease
    Bronner P Gonçalves, Matthew Hall ... ISARIC Clinical Characterisation Group
    Research Article

    Background:

    Whilst timely clinical characterisation of infections caused by novel SARS-CoV-2 variants is necessary for evidence-based policy response, individual-level data on infecting variants are typically only available for a minority of patients and settings.

    Methods:

    Here, we propose an innovative approach to study changes in COVID-19 hospital presentation and outcomes after the Omicron variant emergence using publicly available population-level data on variant relative frequency to infer SARS-CoV-2 variants likely responsible for clinical cases. We apply this method to data collected by a large international clinical consortium before and after the emergence of the Omicron variant in different countries.

    Results:

    Our analysis, that includes more than 100,000 patients from 28 countries, suggests that in many settings patients hospitalised with Omicron variant infection less often presented with commonly reported symptoms compared to patients infected with pre-Omicron variants. Patients with COVID-19 admitted to hospital after Omicron variant emergence had lower mortality compared to patients admitted during the period when Omicron variant was responsible for only a minority of infections (odds ratio in a mixed-effects logistic regression adjusted for likely confounders, 0.67 [95% confidence interval 0.61–0.75]). Qualitatively similar findings were observed in sensitivity analyses with different assumptions on population-level Omicron variant relative frequencies, and in analyses using available individual-level data on infecting variant for a subset of the study population.

    Conclusions:

    Although clinical studies with matching viral genomic information should remain a priority, our approach combining publicly available data on variant frequency and a multi-country clinical characterisation dataset with more than 100,000 records allowed analysis of data from a wide range of settings and novel insights on real-world heterogeneity of COVID-19 presentation and clinical outcome.

    Funding:

    Bronner P. Gonçalves, Peter Horby, Gail Carson, Piero L. Olliaro, Valeria Balan, Barbara Wanjiru Citarella, and research costs were supported by the UK Foreign, Commonwealth and Development Office (FCDO) and Wellcome [215091/Z/18/Z, 222410/Z/21/Z, 225288/Z/22/Z]; and Janice Caoili and Madiha Hashmi were supported by the UK FCDO and Wellcome [222048/Z/20/Z]. Peter Horby, Gail Carson, Piero L. Olliaro, Kalynn Kennon and Joaquin Baruch were supported by the Bill & Melinda Gates Foundation [OPP1209135]; Laura Merson was supported by University of Oxford’s COVID-19 Research Response Fund - with thanks to its donors for their philanthropic support. Matthew Hall was supported by a Li Ka Shing Foundation award to Christophe Fraser. Moritz U.G. Kraemer was supported by the Branco Weiss Fellowship, Google.org, the Oxford Martin School, the Rockefeller Foundation, and the European Union Horizon 2020 project MOOD (#874850). The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the views of the European Commission. Contributions from Srinivas Murthy, Asgar Rishu, Rob Fowler, James Joshua Douglas, François Martin Carrier were supported by CIHR Coronavirus Rapid Research Funding Opportunity OV2170359 and coordinated out of Sunnybrook Research Institute. Contributions from Evert-Jan Wils and David S.Y. Ong were supported by a grant from foundation Bevordering Onderzoek Franciscus; and Andrea Angheben by the Italian Ministry of Health “Fondi Ricerca corrente–L1P6” to IRCCS Ospedale Sacro Cuore–Don Calabria. The data contributions of J.Kenneth Baillie, Malcolm G. Semple, and Ewen M. Harrison were supported by grants from the National Institute for Health Research (NIHR; award CO-CIN-01), the Medical Research Council (MRC; grant MC_PC_19059), and by the NIHR Health Protection Research Unit (HPRU) in Emerging and Zoonotic Infections at University of Liverpool in partnership with Public Health England (PHE) (award 200907), NIHR HPRU in Respiratory Infections at Imperial College London with PHE (award 200927), Liverpool Experimental Cancer Medicine Centre (grant C18616/A25153), NIHR Biomedical Research Centre at Imperial College London (award IS-BRC-1215-20013), and NIHR Clinical Research Network providing infrastructure support. All funders of the ISARIC Clinical Characterisation Group are listed in the appendix.