Meta-Research: Why we need to report more than 'Data were Analyzed by t-tests or ANOVA'

Abstract

Transparent reporting is essential for the critical evaluation of studies. However, the reporting of statistical methods for studies in the biomedical sciences is often limited. This systematic review examines the quality of reporting for two statistical tests, t-tests and ANOVA, for papers published in a selection of physiology journals in June 2017. Of the 328 original research articles examined, 277 (84.5%) included an ANOVA or t-test or both. However, papers in our sample were routinely missing essential information about both types of tests: for example, 213 papers (95% of the papers that used ANOVA) did not contain the information needed to determine what type of ANOVA was performed, and 26.7% of papers did not specify what post-hoc test was performed. Most papers also omitted the information needed to verify ANOVA results. Essential information about t-tests was also missing in many papers. We conclude by discussing measures that could be taken to improve the quality of reporting.

Data availability

All data from the systematic review has been uploaded with the manuscript, along with the abstraction protocol.

Article and author information

Author details

  1. Tracey L Weissgerber

    Division of Nephrology and Hypertension, Mayo Clinic, Rochester, United States
    For correspondence
    weissgerber.tracey@mayo.edu
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-7490-2600
  2. Oscar Alejandro Garcia Valencia

    Division of Nephrology and Hypertension, Mayo Clinic, Rochester, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-0186-9448
  3. Vesna D Garovic

    Division of Nephrology and Hypertension, Mayo Clinic, Rochester, United States
    Competing interests
    The authors declare that no competing interests exist.
  4. Natasa M Milic

    Division of Nephrology and Hypertension, Mayo Clinic, Rochester, United States
    Competing interests
    The authors declare that no competing interests exist.
  5. Stacey J Winham

    Division of Nephrology and Hypertension, Mayo Clinic, Rochester, United States
    Competing interests
    The authors declare that no competing interests exist.

Funding

American Heart Association (16GRNT30950002)

  • Tracey L Weissgerber

National Center for Advancing Translational Sciences (UL1 TR000135)

  • Tracey L Weissgerber

Mayo Clinic (Robert W. Fulk Career Development Award)

  • Tracey L Weissgerber

National Cancer Institute (R03-CA212127)

  • Stacey J Winham

Walter and Evelyn Simmers Career Development Award for Ovarian Cancer Research

  • Stacey J Winham

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Reviewing Editor

  1. M Dawn Teare, University of Sheffield, United Kingdom

Publication history

  1. Received: February 26, 2018
  2. Accepted: December 16, 2018
  3. Accepted Manuscript published: December 21, 2018 (version 1)
  4. Version of Record published: January 9, 2019 (version 2)

Copyright

© 2018, Weissgerber et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 18,154
    Page views
  • 1,746
    Downloads
  • 18
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Tracey L Weissgerber
  2. Oscar Alejandro Garcia Valencia
  3. Vesna D Garovic
  4. Natasa M Milic
  5. Stacey J Winham
(2018)
Meta-Research: Why we need to report more than 'Data were Analyzed by t-tests or ANOVA'
eLife 7:e36163.
https://doi.org/10.7554/eLife.36163

Further reading

    1. Medicine
    2. Neuroscience
    Guido I Guberman et al.
    Research Article Updated

    Background:

    The heterogeneity of white matter damage and symptoms in concussion has been identified as a major obstacle to therapeutic innovation. In contrast, most diffusion MRI (dMRI) studies on concussion have traditionally relied on group-comparison approaches that average out heterogeneity. To leverage, rather than average out, concussion heterogeneity, we combined dMRI and multivariate statistics to characterize multi-tract multi-symptom relationships.

    Methods:

    Using cross-sectional data from 306 previously concussed children aged 9–10 from the Adolescent Brain Cognitive Development Study, we built connectomes weighted by classical and emerging diffusion measures. These measures were combined into two informative indices, the first representing microstructural complexity, the second representing axonal density. We deployed pattern-learning algorithms to jointly decompose these connectivity features and 19 symptom measures.

    Results:

    Early multi-tract multi-symptom pairs explained the most covariance and represented broad symptom categories, such as a general problems pair, or a pair representing all cognitive symptoms, and implicated more distributed networks of white matter tracts. Further pairs represented more specific symptom combinations, such as a pair representing attention problems exclusively, and were associated with more localized white matter abnormalities. Symptom representation was not systematically related to tract representation across pairs. Sleep problems were implicated across most pairs, but were related to different connections across these pairs. Expression of multi-tract features was not driven by sociodemographic and injury-related variables, as well as by clinical subgroups defined by the presence of ADHD. Analyses performed on a replication dataset showed consistent results.

    Conclusions:

    Using a double-multivariate approach, we identified clinically-informative, cross-demographic multi-tract multi-symptom relationships. These results suggest that rather than clear one-to-one symptom-connectivity disturbances, concussions may be characterized by subtypes of symptom/connectivity relationships. The symptom/connectivity relationships identified in multi-tract multi-symptom pairs were not apparent in single-tract/single-symptom analyses. Future studies aiming to better understand connectivity/symptom relationships should take into account multi-tract multi-symptom heterogeneity.

    Funding:

    Financial support for this work came from a Vanier Canada Graduate Scholarship from the Canadian Institutes of Health Research (G.I.G.), an Ontario Graduate Scholarship (S.S.), a Restracomp Research Fellowship provided by the Hospital for Sick Children (S.S.), an Institutional Research Chair in Neuroinformatics (M.D.), as well as a Natural Sciences and Engineering Research Council CREATE grant (M.D.).

    1. Epidemiology and Global Health
    2. Medicine
    Botond Antal et al.
    Research Article Updated

    Background:

    Type 2 diabetes mellitus (T2DM) is known to be associated with neurobiological and cognitive deficits; however, their extent, overlap with aging effects, and the effectiveness of existing treatments in the context of the brain are currently unknown.

    Methods:

    We characterized neurocognitive effects independently associated with T2DM and age in a large cohort of human subjects from the UK Biobank with cross-sectional neuroimaging and cognitive data. We then proceeded to evaluate the extent of overlap between the effects related to T2DM and age by applying correlation measures to the separately characterized neurocognitive changes. Our findings were complemented by meta-analyses of published reports with cognitive or neuroimaging measures for T2DM and healthy controls (HCs). We also evaluated in a cohort of T2DM-diagnosed individuals using UK Biobank how disease chronicity and metformin treatment interact with the identified neurocognitive effects.

    Results:

    The UK Biobank dataset included cognitive and neuroimaging data (N = 20,314), including 1012 T2DM and 19,302 HCs, aged between 50 and 80 years. Duration of T2DM ranged from 0 to 31 years (mean 8.5 ± 6.1 years); 498 were treated with metformin alone, while 352 were unmedicated. Our meta-analysis evaluated 34 cognitive studies (N = 22,231) and 60 neuroimaging studies: 30 of T2DM (N = 866) and 30 of aging (N = 1088). Compared to age, sex, education, and hypertension-matched HC, T2DM was associated with marked cognitive deficits, particularly in executive functioning and processing speed. Likewise, we found that the diagnosis of T2DM was significantly associated with gray matter atrophy, primarily within the ventral striatum, cerebellum, and putamen, with reorganization of brain activity (decreased in the caudate and premotor cortex and increased in the subgenual area, orbitofrontal cortex, brainstem, and posterior cingulate cortex). The structural and functional changes associated with T2DM show marked overlap with the effects correlating with age but appear earlier, with disease duration linked to more severe neurodegeneration. Metformin treatment status was not associated with improved neurocognitive outcomes.

    Conclusions:

    The neurocognitive impact of T2DM suggests marked acceleration of normal brain aging. T2DM gray matter atrophy occurred approximately 26% ± 14% faster than seen with normal aging; disease duration was associated with increased neurodegeneration. Mechanistically, our results suggest a neurometabolic component to brain aging. Clinically, neuroimaging-based biomarkers may provide a valuable adjunctive measure of T2DM progression and treatment efficacy based on neurological effects.

    Funding:

    The research described in this article was funded by the W. M. Keck Foundation (to LRMP), the White House Brain Research Through Advancing Innovative Technologies (BRAIN) Initiative (NSFNCS-FR 1926781 to LRMP), and the Baszucki Brain Research Fund (to LRMP). None of the funding sources played any role in the design of the experiments, data collection, analysis, interpretation of the results, the decision to publish, or any aspect relevant to the study. DJW reports serving on data monitoring committees for Novo Nordisk. None of the authors received funding or in-kind support from pharmaceutical and/or other companies to write this article.