Standardized mean differences cause funnel plot distortion in publication bias assessments

  1. Peter-Paul Zwetsloot  Is a corresponding author
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever  Is a corresponding author
  1. University Medical Center Utrecht, Netherlands
  2. University of Edinburgh, United Kingdom
  3. University of Tasmania, Australia
  4. Radboud Institute for Health Sciences, Radboud University Medical Center, Netherlands
  5. The University of Edinburgh, United Kingdom
  6. Radboud University Medical Center, Netherlands

Abstract

Meta-analyses are increasingly used for synthesis of evidence, and often include an assessment of publication bias based on detection of asymmetry in funnel plots. We studied the influence of different normalisation approaches, sample size and intervention effects on funnel plot asymmetry, using empirical datasets and illustrative simulations. We found that funnel plots of the Standardized Mean Difference (SMD) plotted against the standard error (SE) are susceptible to distortion, leading to overestimation of the existence and extent of publication bias. Distortion was more severe when the primary studies had a small sample size and when an intervention effect was present. We show that using the Normalised Mean Difference (when possible), or plotting the SMD against a sample size-based precision estimate, are more reliable alternatives. We conclude that funnel plots using the SMD in combination with the SE are unsuitable for publication bias assessments and can lead to false-positive results.

Article and author information

Author details

  1. Peter-Paul Zwetsloot

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    For correspondence
    P.P.M.Zwetsloot@umcutrecht.nl
    Competing interests
    The authors declare that no competing interests exist.
  2. Mira Van Der Naald

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  3. Emily S Sena

    Center for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  4. David W Howells

    School of Medicine, University of Tasmania, Hobart, Australia
    Competing interests
    The authors declare that no competing interests exist.
  5. Joanna IntHout

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  6. Joris AH De Groot

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  7. Steven AJ Chamuleau

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  8. Malcolm R MacLeod

    Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  9. Kimberley E Wever

    Systematic Review Centre for Laboratory Animal Experimentation, Radboud University Medical Center, Nijmegen, Netherlands
    For correspondence
    kim.wever@radboudumc.nl
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3635-3660

Funding

National Institute of Environmental Health Sciences (National Toxicology Program research funding)

  • Kimberley E Wever

Netherlands Cardiovascular Research Initiative (CVON-HUSTCARE)

  • Steven AJ Chamuleau

National Centre for the Replacement, Refinement and Reduction of Animals in Research (Infrastructure Award)

  • Emily S Sena
  • Malcolm R MacLeod

Alexander Suerman program (PhD student Scholarship)

  • Peter-Paul Zwetsloot

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Reviewing Editor

  1. M Dawn Teare, University of Sheffield, United Kingdom

Version history

  1. Received: December 14, 2016
  2. Accepted: August 21, 2017
  3. Accepted Manuscript published: September 8, 2017 (version 1)
  4. Version of Record published: September 29, 2017 (version 2)

Copyright

© 2017, Zwetsloot et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 5,455
    Page views
  • 404
    Downloads
  • 111
    Citations

Article citation count generated by polling the highest count across the following sources: Scopus, Crossref, PubMed Central.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Peter-Paul Zwetsloot
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever
(2017)
Standardized mean differences cause funnel plot distortion in publication bias assessments
eLife 6:e24260.
https://doi.org/10.7554/eLife.24260

Further reading

    1. Epidemiology and Global Health
    C Kim, Benjamin Chen ... RECOVER Mechanistic Pathways Task Force
    Review Article

    The NIH-funded RECOVER study is collecting clinical data on patients who experience a SARS-CoV-2 infection. As patient representatives of the RECOVER Initiative’s Mechanistic Pathways task force, we offer our perspectives on patient motivations for partnering with researchers to obtain results from mechanistic studies. We emphasize the challenges of balancing urgency with scientific rigor. We recognize the importance of such partnerships in addressing post-acute sequelae of SARS-CoV-2 infection (PASC), which includes ‘long COVID,’ through contrasting objective and subjective narratives. Long COVID’s prevalence served as a call to action for patients like us to become actively involved in efforts to understand our condition. Patient-centered and patient-partnered research informs the balance between urgency and robust mechanistic research. Results from collaborating on protocol design, diverse patient inclusion, and awareness of community concerns establish a new precedent in biomedical research study design. With a public health matter as pressing as the long-term complications that can emerge after SARS-CoV-2 infection, considerate and equitable stakeholder involvement is essential to guiding seminal research. Discussions in the RECOVER Mechanistic Pathways task force gave rise to this commentary as well as other review articles on the current scientific understanding of PASC mechanisms.

    1. Epidemiology and Global Health
    Tina Bech Olesen, Henry Jensen ... Sisse H Njor
    Research Article Updated

    Background:

    In most of the world, the mammography screening programmes were paused at the start of the pandemic, whilst mammography screening continued in Denmark. We examined the mammography screening participation during the COVID-19 pandemic in Denmark.

    Methods:

    The study population comprised all women aged 50–69 years old invited to participate in mammography screening from 2016 to 2021 in Denmark based on data from the Danish Quality Database for Mammography Screening in combination with population-based registries. Using a generalised linear model, we estimated prevalence ratios (PRs) and 95% confidence intervals (CIs) of mammography screening participation within 90, 180, and 365 d since invitation during the pandemic in comparison with the previous years adjusting for age, year and month of invitation.

    Results:

    The study comprised 1,828,791 invitations among 847,766 women. Before the pandemic, 80.2% of invitations resulted in participation in mammography screening within 90 d, 82.7% within 180 d, and 83.1% within 365 d. At the start of the pandemic, the participation in screening within 90 d was reduced to 69.9% for those invited in pre-lockdown and to 76.5% for those invited in first lockdown. Extending the length of follow-up time to 365 d only a minor overall reduction was observed (PR = 0.94; 95% CI: 0.93–0.95 in pre-lockdown and PR = 0.97; 95% CI: 0.96–0.97 in first lockdown). A lower participation was, however, seen among immigrants and among women with a low income.

    Conclusions:

    The short-term participation in mammography screening was reduced at the start of the pandemic, whilst only a minor reduction in the overall participation was observed with longer follow-up time, indicating that women postponed screening. Some groups of women, nonetheless, had a lower participation, indicating that the social inequity in screening participation was exacerbated during the pandemic.

    Funding:

    The study was funded by the Danish Cancer Society Scientific Committee (grant number R321-A17417) and the Danish regions.