Despite the crucial importance of biomedical science for human well-being, the uncomfortable reality is that swathes of published results in fields from psychology to cancer research are less robust than optimum (Ioannidis, 2005; Krawczyk, 2015; Loken and Gelman, 2017; Grimes et al., 2018; Errington et al., 2021). In cases when findings are spurious, inappropriate or errant statistical methods are often the primary cause of untrustworthy research, from incorrect interpretations of p-values to unsuitable tests to data redaction and under-reporting of overtesting, leading to research waste and unsound conclusions (Hoffmann et al., 2013; Altman and Krzywinski, 2017; Colquhoun, 2014; Glasziou et al., 2014; Grimes and Heathers, 2021a; Itaya et al., 2022; Baer et al., 2021a; Baer et al., 2021b). Across biomedical sciences, dichotomous outcome trials and studies are of paramount importance, forming the basis of everything from preclinical observational studies to randomized controlled trials. Such investigations contrast experimental and control groups for a given intervention, comparing the numbers experiencing a particular event in both arms to infer whether differences between the intervention and control arm might exist.

Such investigations are vital, but concern has been raised over the fragility of many published works, where small amounts of recoding from event to non-event in experimental arms or vice versa in control arms can create an illusion of a relationship where none truly exists. In previous work by the author (Grimes, 2022), Ellipse of Insignificance (EOI) analysis was introduced as a refined fragility index capable of handling even huge data sets analytically with ease, considering both control and experimental arms simultaneously, which traditional fragility analysis cannot. EOI analysis is robust and analytical, suitable not only for Randomized Controlled Trial (RCT) analysis but for observational trials, cohort studies, and general preclinical work. Additionally, it also links the concept of fragility to test sensitivity and specificity when these are known for the detection of events, enabling investigators to probe not only whether a result is arbitrarily fragile, but to truly probe whether consider certain results are even possible. Accordingly, it yields both objective metrics for fragility and can be employed to detect inappropriate manipulation of results if the statistical properties of the tests used are known. A web implementation of this is available at https://drg85.shinyapps.io/EOIanalysis/, replete with code in ${\displaystyle R}$ and other popular languages for general application.

While EOI analysis is a powerful method for ascertaining trial robustness, it does not explicitly consider the scenario where data is redacted. Data redaction in biomedical science creates spurious results and untrustworthy findings (Grimes and Heathers, 2021b), and can be difficult to detect. Data redaction can be accidental due to some systematic error in analysis, due to missing data, or arise through deliberate cherry-picking, and there are currently few tools for gauging its likely impact outside of direct simulation. In this technical note, we unveil a novel and powerful method for quantifying how much redaction would be required to explain a seemingly significant finding in dichotomous outcome trials, automatically finding the degree of redaction required to yield spurious results, and objective metrics for defining this. While EOI analysis produced a conic section in the form of an inclined ellipse where significance disappeared, this new tool instead produces a bounded region where significance disappears attainable by redaction and calculates the minimal vector to this regions. This technical note outlines the methodology of Region of Attainable Redaction (ROAR) analysis, including examples, R and MATLAB code for user implementations, and a web implementation for ease of deployability.

The underlying geometrical and statistical basis for EOI analysis has been previously derived and described. In brief, EOI arises from chi-squared analysis, ascertaining how many participants in experimental and control groups could be recoded from events to non-events and vice versa before apparently significance was lost. This is a powerful approach for determining robustness of outcomes, and a web implementation and code are available at https://drg85.shinyapps.io/EOIanalysis/. EOI in its current form, however, does not consider the situation where a significant result might be obtained by data redaction, where an experimenter censors or neglects observations in the final analysis.

Defining ${\displaystyle a}$ as the reported data of endpoint positive cases in the experimental or exposure arm, ${\displaystyle b}$ as the reported experimental endpoint negative, ${\displaystyle c}$ as reported endpoint positive cases in the control arm, and ${\displaystyle d}$ as the reported endpoint negative cases in the control arm, we may define ${\displaystyle x}$ and ${\displaystyle y}$ as hypothetical redacted data in the experimental and control arm, respectively. We further define the total reported sample as ${\displaystyle n=a+b+c+d}$. To account for the impact of redaction, consider that an experimenter may obtain a significant result in favour of the experimental group in several ways. When relative risk is given by ${\displaystyle \frac{a(c+d)}{c(a+b)}}$ with no significant difference, the experimental arm could still yield a greater relative risk (${\displaystyle {\displaystyle R{R}_E>1}}$) than the control arm if either ${\displaystyle x}$ endpoint negative events had been jettisoned from the experimental arm, ${\displaystyle y}$ endpoint positive events jettisoned from the control or comparison arm, or a combination of both. Equally, if there is no significant difference but a lower relative risk in the experimental group is sought (${\displaystyle {\displaystyle R{R}_E<1}}$), such a finding can be manipulated by either jettisoning ${\displaystyle x}$ endpoint positive cases from the experimental arm, ${\displaystyle y}$ endpoint negative cases from the control arm, or a combination of both. These situations are given in Table 1. Risk ratio is used in this work for simplicity in gauging the relative impact of an ostensibly significant effect in the experimental arm and can be readily converted to odds ratio if preferred.

Applying the chi-square statistic outlined previously with a threshold critical value for significance of ${\displaystyle {\nu }_c}$, the resulting identity when ${\displaystyle R{R}_E>1}$ is

and when ${\displaystyle R{R}_E<1}$, the identity is

Similar to EOI analysis, these forms can be expanded. However, the resultant equations in either case are not the conic sections of an inclined ellipse as with EOI analysis, but a more complicated cubic curve also in two variables. The resultant identity is ${\displaystyle g(x,y)}$, polynomial in ${\displaystyle x}$ and ${\displaystyle y}$, with the list of 15 coefficients in either case in given in the mathematical appendix (Supplementary file 1, Table S1). The region bound by this equation is the ROAR, and any ${\displaystyle {\displaystyle g(x,y)\le 0}}$ changes an ostensibly significant finding to the null one, with ${\displaystyle x}$ and ${\displaystyle y}$ respectively yielding the redaction from the experimental and control group required.

ROAR derivation and FOCK vector

In EOI analysis, we derived an analytical method for finding the minimum distance from the origin to the EOI. This point and vector, the Fewest Experimental/Control Knowingly Uncoded Participants (FECKUP) vector, allowed us to ascertain the minimal error which would render results insignificant. The resultant curve and bound region are inherently more complex in ROAR analysis, but the general principle remains. We seek the minimum distance from the origin to the region bound by ${\displaystyle g(x,y)=0}$, defining the vector to this point ${\displaystyle (x_e,y_e)}$ as the Fewest Observations/Censored Knowledge (FOCK) vector. Unlike EOI analysis, we cannot exploit geometric arguments to solve this analytically, and instead we proceed by the method of Lagrange multipliers. The minimum distance from the origin to a point is given by

(3) ${\displaystyle D(x,y)=\sqrt{x^2+y^2}}$

Defining the polynomial defined in Supplementary file 1, Table S1 as ${\displaystyle g(x,y)}$, we can exploit the properties of Lagrange multipliers to write

(4) ${\displaystyle {\displaystyle \frac{\mathrm{\partial }D}{\mathrm{\partial }x}=\lambda \frac{\mathrm{\partial }g}{\mathrm{\partial }x}}}$

(5) ${\displaystyle \frac{\mathrm{\partial }D}{\mathrm{\partial }y}=\lambda \frac{\mathrm{\partial }g}{\mathrm{\partial }y}.}$

As we know ${\displaystyle \lambda \ne 0}$, we can rearrange these equations for the constant scalar ${\displaystyle \lambda }$ and equate them, subject to the constraint ${\displaystyle g(x,y)=0}$. After rearrangement, we deduce that

(6) ${\displaystyle y\frac{\mathrm{\partial }g}{\mathrm{\partial }x}-x\frac{\mathrm{\partial }g}{\mathrm{\partial }y}=0.}$

This yields another unwieldy polynomial in two variables with 18 coefficients, listed in the mathematical appendix (Supplementary file 1, Table S2) for both cases. If we define the resultant function as ${\displaystyle h(x,y)}$, we seek to solve the simultaneous equations

(7) ${\displaystyle g(x_e,y_e)=0}$

(8) ${\displaystyle h(x_e,y_e)=0.}$

While analytical solutions are likely intractable, this can be readily solved numerically subject to the constraints that ${\displaystyle {\displaystyle x_e>0}}$ and ${\displaystyle {\displaystyle y_e>0}}$. By Bézout’s theorem, there are potentially up to 25 solutions to this simultaneous equation, so we restrict potential solution pairs to the real domain and select the pair yielding the minimum length FOCK vector, corresponding to (${\displaystyle x_e.y_e}$) as illustrated in Figure 1. Additionally, we solve ${\displaystyle g(x_c,0)=0}$ and ${\displaystyle g(0,y_c)=0}$ as simple cubic equations to find the minimum number of observations redacted in exclusively the experimental and control groups to lose significance. The resolution of the FOCK vector yields the minimum redacted combination of experimental and control groups, given by

(9) ${\displaystyle r_{min}=\lfloor x_e+y_e\rfloor .}$

Figure 1

Download asset Open asset

ROAR analysis outlined in this technical note extends the functionality and usefulness of EOI analysis, allowing users to estimate the likely impacts of missing data. EOI analysis and related fragility methods had the limitations that while they handled potential miscoding, they were unsuitable for inferences or quantification of impacts of redacted data or subjects lost to follow-up. Accordingly, ROAR is a powerful method of gauging the potential impacts of missing data. While more mathematically complex than EOI analysis, ROAR remains deterministic and rapid, but shares the limitation that it is only currently applicable to dichotomous outcome trials and studies, and should be applied very cautiously to time-to-event data, where it may not be suitable. Like EOI analysis, ROAR differs from typical fragility metrics by avoiding Fisher’s exact test as this is not suitable for large data sets which EOI and ROAR can readily handle. This is typically not a problem as the chi-squared test employed approximates Fisher’s test in most circumstances. However, like EOI analysis, p-values for small trials can differ slightly from chi-squared result. ROAR analysis is built upon chi-squared statistics, and it is thus possible for edge cases of small numbers to yield discordant results with Fisher’s exact test also. This can be shown from a theoretical standpoint to not typically make any appreciable difference except for rare events in very small trials (Grimes, 2022; Baer et al., 2021a).

Application of ROAR analysis is inherently context specific. For clinical trials, preregistration in principle reduces the potential for experimenter choices like redaction bias changing the outcome. But the implementation of preregistered protocols does not exclude the possibility of data dredging in the form of p-hacking or HARKing (hypothesizing after the results are known). Researchers rarely follow the precise methods, plan, and analyses that they preregistered. A recent analysis found that pre-registered studies, despite having power analysis and higher sample size than non-registered studies, do not a priori seem to prevent p-hacking and HARKing (Bakker et al., 2020; Singh et al., 2021; El‐Boghdadly et al., 2018; Sun et al., 2019), with similar proportions of positive results and effect sizes between preregistered and non-preregistered studies (van den Akker et al., 2023). A survey of 27 preregistered studies found researchers deviating from preregistered plans in all cases, most frequently in relation to planned sample size and exclusion criteria (Claesen et al., 2021). This latter aspect lends itself to potential redaction bias (Grimes and Heathers, 2021b), which can be systematic rather than deliberate and thus a means to quantify its impact is important. More importantly, EOI analysis has application for dichotomous beyond clinical trials. In preclinical work, cohort, and observational studies, the scope for redaction bias greatly increases as reporting and selection of data falls entirely on the experimenter, and thus methods like ROAR to probe potential redaction bias are important. ROAR also has potential application in case–control studies, where selection of an inappropriately fragile control group could give spurious conclusions. This again comes with caveats as studies adjusted for potential confounders and predictors might make ROAR inappropriate in such cases.

As demonstrated in this work, ROAR is under certain circumstances applicable even to large meta-analysis. In this instance, a potential redaction of just 14 subjects from over 39,000 was sufficient to overturn the null hypothesis, despite a relative risk reduction of 15% being widely reported on the basis of this meta-analysis. This of course is not to say that any redaction occurred only to quantify the vulnerability of such a study to missing data. While beyond the scope of this technical note, it is worth noting that a subsequent 2022 meta-analysis (Zhang et al., 2022) of 11 RCTS (including those in the 2019 [Zhang et al., 2019] meta-analysis) found no significant reduction in cancer mortality with vitamin D supplementation, a potential testament to the need to consider the fragility of results to missing or miscoded data, even with ostensibly large samples. There are however important caveats to applying ROAR to meta-analyses. In its naive form, it is only suitable when there is minimal confounding between studies so that the crude relative risk differs minimally from the adjust risk (${\displaystyle R{R}_C\approx R{R}_{CMH}}$), such as in the illustrative work considered herein. When this is not the case, results from individual studies cannot be crudely pooled and ROAR is not valid in these instances. As ROAR applied to meta-analyses pools studies into a simple crude measure, it does not identity the particular study or studies where hypothetical redaction might have occurred, only the global fragility metric. A full theoretical extension of ROAR and EOI for specifically for meta-analyses is beyond the scope of this work, and accordingly, ROAR must be cautiously implemented and carefully interpreted in investigations of meta-analyses.

As discussed in the EOI paper (Grimes, 2022), poor research conduct including inappropriate statistical manipulation and data redaction are not uncommon, affecting up to three quarters of all biomedical science (Fanelli, 2009). Like EOI analysis, the ROAR extension has a potential role in detecting manipulations that nudge results towards significance, and identifying inconsistencies in data and adding invaluable context. It is a demonstrable reality that even seemingly strong results can falter under inspection, and tools like ROAR and EOI analysis have a potentially important role in identifying weak results and statistical inconsistencies, with wide potential application across meta-research with the goal of furthering sustainable, reproducible research.

Sample R / MATLAB / OCTAVE code and functions for rapid implementation of EOI analysis method outlined, hosted online at https://github.com/drg85/EOIROAR_code (copy archived at Grimes, 2023). Web implementation is available at: https://drg85.shinyapps.io/EOIROAR/. All data is available in the paper and on GitHub.

1. 1. Altman N
   2. Krzywinski M

   (2017) Points of significance: interpreting p values

   Nature Methods 14:213–215.

   https://doi.org/10.1038/nmeth.4210

   • Google Scholar
2. 1. Baer BR
   2. Gaudino M
   3. Charlson M
   4. Fremes SE
   5. Wells MT

   (2021a) Fragility indices for only sufficiently likely modifications

   PNAS 118:e2105254118.

   https://doi.org/10.1073/pnas.2105254118

   • PubMed
   • Google Scholar
3. 1. Baer BR
   2. Gaudino M
   3. Fremes SE
   4. Charlson M
   5. Wells MT

   (2021b) The fragility index can be used for sample size calculations in clinical trials

   Journal of Clinical Epidemiology 139:199–209.

   https://doi.org/10.1016/j.jclinepi.2021.08.010

   • PubMed
   • Google Scholar
4. 1. Bakker M
   2. Veldkamp CLS
   3. van Assen MALM
   4. Crompvoets EAV
   5. Ong HH
   6. Nosek BA
   7. Soderberg CK
   8. Mellor D
   9. Wicherts JM

   (2020) Ensuring the quality and specificity of preregistrations

   PLOS Biology 18:e3000937.

   https://doi.org/10.1371/journal.pbio.3000937

   • PubMed
   • Google Scholar
5. 1. Claesen A
   2. Gomes S
   3. Tuerlinckx F
   4. Vanpaemel W

   (2021) Comparing dream to reality: an assessment of adherence of the first generation of preregistered studies

   Royal Society Open Science 8:211037.

   https://doi.org/10.1098/rsos.211037

   • PubMed
   • Google Scholar
6. 1. Colquhoun D

   (2014) An investigation of the false discovery rate and the misinterpretation of p-values

   Royal Society Open Science 1:140216.

   https://doi.org/10.1098/rsos.140216

   • PubMed
   • Google Scholar
7. 1. El‐Boghdadly K
   2. Wiles MD
   3. Atton S
   4. Bailey CR

   (2018) Adherence to guidance on registration of randomised controlled trials published in Anaesthesia

   Anaesthesia 73:556–563.

   https://doi.org/10.1111/anae.14103

   • Google Scholar
8. 1. Errington TM
   2. Denis A
   3. Perfito N
   4. Iorns E
   5. Nosek BA

   (2021) Reproducibility in cancer biology: challenges for assessing replicability in preclinical cancer biology

   eLife 10:e67995.

   https://doi.org/10.7554/eLife.67995

   • Google Scholar
9. 1. Fanelli D

   (2009) How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data

   PLOS ONE 4:e5738.

   https://doi.org/10.1371/journal.pone.0005738

   • PubMed
   • Google Scholar
10. 1. Glasziou P
    2. Altman DG
    3. Bossuyt P
    4. Boutron I
    5. Clarke M
    6. Julious S
    7. Michie S
    8. Moher D
    9. Wager E

    (2014) Reducing waste from incomplete or unusable reports of biomedical research

    The Lancet 383:267–276.

    https://doi.org/10.1016/S0140-6736(13)62228-X

    • Google Scholar
11. 1. Grimes DR
    2. Bauch CT
    3. Ioannidis JPA

    (2018) Modelling science trustworthiness under publish or perish pressure

    Royal Society Open Science 5:171511.

    https://doi.org/10.1098/rsos.171511

    • PubMed
    • Google Scholar
12. 1. Grimes DR
    2. Heathers J

    (2021a) Association between magnetic field exposure and miscarriage risk is not supported by the data

    Scientific Reports 11:22143.

    https://doi.org/10.1038/s41598-021-01391-3

    • PubMed
    • Google Scholar
13. 1. Grimes DR
    2. Heathers J

    (2021b) The new normal? Redaction bias in biomedical science

    Royal Society Open Science 8:211308.

    https://doi.org/10.1098/rsos.211308

    • PubMed
    • Google Scholar
14. 1. Grimes DR

    (2022) The ellipse of insignificance, a refined fragility index for ascertaining robustness of results in dichotomous outcome trials

    eLife 11:e79573.

    https://doi.org/10.7554/eLife.79573

    • PubMed
    • Google Scholar
15. Software

    1. Grimes DR

    (2023) Eoiroar_Code, version swh:1:rev:5a134703bd769565a4e4919fd99d3f908b59566c

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:c199332b0a4c251e6b012a236019fab771f8574e;origin=https://github.com/drg85/EOIROAR_code;visit=swh:1:snp:ef45dd5ad8375091b7257e8060ad3ba427857466;anchor=swh:1:rev:5a134703bd769565a4e4919fd99d3f908b59566c
16. 1. Hoffmann TC
    2. Erueti C
    3. Glasziou PP

    (2013) Poor description of non-pharmacological interventions: analysis of consecutive sample of randomised trials

    BMJ 347:f3755.

    https://doi.org/10.1136/bmj.f3755

    • PubMed
    • Google Scholar
17. 1. Ioannidis JPA

    (2005) Why most published research findings are false

    PLOS Medicine 2:e124.

    https://doi.org/10.1371/journal.pmed.0020124

    • PubMed
    • Google Scholar
18. 1. Itaya T
    2. Isobe Y
    3. Suzuki S
    4. Koike K
    5. Nishigaki M
    6. Yamamoto Y

    (2022) The fragility of statistically significant results in randomized clinical trials for COVID-19

    JAMA Network Open 5:e222973.

    https://doi.org/10.1001/jamanetworkopen.2022.2973

    • PubMed
    • Google Scholar
19. 1. Krawczyk M

    (2015) The search for significance: a few peculiarities in the distribution of p values in experimental psychology literature

    PLOS ONE 10:e0127872.

    https://doi.org/10.1371/journal.pone.0127872

    • PubMed
    • Google Scholar
20. 1. Loken E
    2. Gelman A

    (2017) Measurement error and the replication crisis

    Science 355:584–585.

    https://doi.org/10.1126/science.aal3618

    • PubMed
    • Google Scholar
21. Preprint

    1. Singh B
    2. Fairman CM
    3. Christensen JF
    4. Bolam KA
    5. Twomey R
    6. Nunan D
    7. Lahart IM

    (2021) Outcome reporting bias in exercise oncology trials (OREO): a cross-sectional study

    medRxiv.

    https://doi.org/10.1101/2021.03.12.21253378

    • Google Scholar
22. 1. Sun LW
    2. Lee DJ
    3. Collins JA
    4. Carll TC
    5. Ramahi K
    6. Sandy SJ
    7. Unteriner JG
    8. Weinberg DV

    (2019) Assessment of consistency between peer-reviewed publications and clinical trial registries

    JAMA Ophthalmology 137:552–556.

    https://doi.org/10.1001/jamaophthalmol.2019.0312

    • PubMed
    • Google Scholar
23. 1. van den Akker OR
    2. van Assen M
    3. Bakker M
    4. Elsherif M
    5. Wong TK
    6. Wicherts JM

    (2023) Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology

    Behavior Research Methods 01:1–10.

    https://doi.org/10.3758/s13428-023-02277-0

    • PubMed
    • Google Scholar
24. 1. Zhang Y
    2. Fang F
    3. Tang J
    4. Jia L
    5. Feng Y
    6. Xu P
    7. Faramand A

    (2019) Association between vitamin D supplementation and mortality: systematic review and meta-analysis

    BMJ 366:l4673.

    https://doi.org/10.1136/bmj.l4673

    • PubMed
    • Google Scholar
25. 1. Zhang R
    2. Zhang Y
    3. Liu Z
    4. Pei Y
    5. Xu P
    6. Chong W
    7. Hai Y
    8. He L
    9. He Y
    10. Yu J
    11. Wang J
    12. Fang F
    13. Peng X

    (2022) Association between vitamin D supplementation and cancer mortality: a systematic review and meta-analysis

    Cancers 14:3717.

    https://doi.org/10.3390/cancers14153717

    • PubMed
    • Google Scholar

Author details

1. David Robert Grimes

   1. School of Medicine, Trinity College Dublin, Dublin, Ireland
   2. School of Physical Sciences, Dublin City University, Dublin, Ireland

   Contribution

   Conceptualization, Resources, Software, Formal analysis, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing – original draft, Project administration

   For correspondence

   davidrobert.grimes@tcd.ie

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3140-3278

• 126

  views

• 15

  downloads

• 0

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.