Standardized mean differences cause funnel plot distortion in publication bias assessments

  1. Peter-Paul Zwetsloot  Is a corresponding author
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever  Is a corresponding author
  1. University Medical Center Utrecht, Netherlands
  2. University of Edinburgh, United Kingdom
  3. University of Tasmania, Australia
  4. Radboud Institute for Health Sciences, Radboud University Medical Center, Netherlands
  5. The University of Edinburgh, United Kingdom
  6. Radboud University Medical Center, Netherlands

Abstract

Meta-analyses are increasingly used for synthesis of evidence, and often include an assessment of publication bias based on detection of asymmetry in funnel plots. We studied the influence of different normalisation approaches, sample size and intervention effects on funnel plot asymmetry, using empirical datasets and illustrative simulations. We found that funnel plots of the Standardized Mean Difference (SMD) plotted against the standard error (SE) are susceptible to distortion, leading to overestimation of the existence and extent of publication bias. Distortion was more severe when the primary studies had a small sample size and when an intervention effect was present. We show that using the Normalised Mean Difference (when possible), or plotting the SMD against a sample size-based precision estimate, are more reliable alternatives. We conclude that funnel plots using the SMD in combination with the SE are unsuitable for publication bias assessments and can lead to false-positive results.

Article and author information

Author details

  1. Peter-Paul Zwetsloot

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    For correspondence
    P.P.M.Zwetsloot@umcutrecht.nl
    Competing interests
    The authors declare that no competing interests exist.
  2. Mira Van Der Naald

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  3. Emily S Sena

    Center for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  4. David W Howells

    School of Medicine, University of Tasmania, Hobart, Australia
    Competing interests
    The authors declare that no competing interests exist.
  5. Joanna IntHout

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  6. Joris AH De Groot

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  7. Steven AJ Chamuleau

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  8. Malcolm R MacLeod

    Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  9. Kimberley E Wever

    Systematic Review Centre for Laboratory Animal Experimentation, Radboud University Medical Center, Nijmegen, Netherlands
    For correspondence
    kim.wever@radboudumc.nl
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3635-3660

Funding

National Institute of Environmental Health Sciences (National Toxicology Program research funding)

  • Kimberley E Wever

Netherlands Cardiovascular Research Initiative (CVON-HUSTCARE)

  • Steven AJ Chamuleau

National Centre for the Replacement, Refinement and Reduction of Animals in Research (Infrastructure Award)

  • Emily S Sena
  • Malcolm R MacLeod

Alexander Suerman program (PhD student Scholarship)

  • Peter-Paul Zwetsloot

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Copyright

© 2017, Zwetsloot et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 6,026
    views
  • 440
    downloads
  • 137
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Peter-Paul Zwetsloot
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever
(2017)
Standardized mean differences cause funnel plot distortion in publication bias assessments
eLife 6:e24260.
https://doi.org/10.7554/eLife.24260

Share this article

https://doi.org/10.7554/eLife.24260

Further reading

    1. Epidemiology and Global Health
    2. Genetics and Genomics
    Rashmi Sukumaran, Achuthsankar S Nair, Moinak Banerjee
    Research Article

    Burden of stroke differs by region, which could be attributed to differences in comorbid conditions and ethnicity. Genomewide variation acts as a proxy marker for ethnicity, and comorbid conditions. We present an integrated approach to understand this variation by considering prevalence and mortality rates of stroke and its comorbid risk for 204 countries from 2009 to 2019, and Genome-wide association studies (GWAS) risk variant for all these conditions. Global and regional trend analysis of rates using linear regression, correlation, and proportion analysis, signifies ethnogeographic differences. Interestingly, the comorbid conditions that act as risk drivers for stroke differed by regions, with more of metabolic risk in America and Europe, in contrast to high systolic blood pressure in Asian and African regions. GWAS risk loci of stroke and its comorbid conditions indicate distinct population stratification for each of these conditions, signifying for population-specific risk. Unique and shared genetic risk variants for stroke, and its comorbid and followed up with ethnic-specific variation can help in determining regional risk drivers for stroke. Unique ethnic-specific risk variants and their distinct patterns of linkage disequilibrium further uncover the drivers for phenotypic variation. Therefore, identifying population- and comorbidity-specific risk variants might help in defining the threshold for risk, and aid in developing population-specific prevention strategies for stroke.

    1. Epidemiology and Global Health
    2. Evolutionary Biology
    Renan Maestri, Benoît Perez-Lamarque ... Hélène Morlon
    Research Article

    Several coronaviruses infect humans, with three, including the SARS-CoV2, causing diseases. While coronaviruses are especially prone to induce pandemics, we know little about their evolutionary history, host-to-host transmissions, and biogeography. One of the difficulties lies in dating the origination of the family, a particularly challenging task for RNA viruses in general. Previous cophylogenetic tests of virus-host associations, including in the Coronaviridae family, have suggested a virus-host codiversification history stretching many millions of years. Here, we establish a framework for robustly testing scenarios of ancient origination and codiversification versus recent origination and diversification by host switches. Applied to coronaviruses and their mammalian hosts, our results support a scenario of recent origination of coronaviruses in bats and diversification by host switches, with preferential host switches within mammalian orders. Hotspots of coronavirus diversity, concentrated in East Asia and Europe, are consistent with this scenario of relatively recent origination and localized host switches. Spillovers from bats to other species are rare, but have the highest probability to be towards humans than to any other mammal species, implicating humans as the evolutionary intermediate host. The high host-switching rates within orders, as well as between humans, domesticated mammals, and non-flying wild mammals, indicates the potential for rapid additional spreading of coronaviruses across the world. Our results suggest that the evolutionary history of extant mammalian coronaviruses is recent, and that cases of long-term virus–host codiversification have been largely over-estimated.