Standardized mean differences cause funnel plot distortion in publication bias assessments

  1. Peter-Paul Zwetsloot  Is a corresponding author
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever  Is a corresponding author
  1. University Medical Center Utrecht, Netherlands
  2. University of Edinburgh, United Kingdom
  3. University of Tasmania, Australia
  4. Radboud Institute for Health Sciences, Radboud University Medical Center, Netherlands
  5. The University of Edinburgh, United Kingdom
  6. Radboud University Medical Center, Netherlands

Abstract

Meta-analyses are increasingly used for synthesis of evidence, and often include an assessment of publication bias based on detection of asymmetry in funnel plots. We studied the influence of different normalisation approaches, sample size and intervention effects on funnel plot asymmetry, using empirical datasets and illustrative simulations. We found that funnel plots of the Standardized Mean Difference (SMD) plotted against the standard error (SE) are susceptible to distortion, leading to overestimation of the existence and extent of publication bias. Distortion was more severe when the primary studies had a small sample size and when an intervention effect was present. We show that using the Normalised Mean Difference (when possible), or plotting the SMD against a sample size-based precision estimate, are more reliable alternatives. We conclude that funnel plots using the SMD in combination with the SE are unsuitable for publication bias assessments and can lead to false-positive results.

Article and author information

Author details

  1. Peter-Paul Zwetsloot

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    For correspondence
    P.P.M.Zwetsloot@umcutrecht.nl
    Competing interests
    The authors declare that no competing interests exist.
  2. Mira Van Der Naald

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  3. Emily S Sena

    Center for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  4. David W Howells

    School of Medicine, University of Tasmania, Hobart, Australia
    Competing interests
    The authors declare that no competing interests exist.
  5. Joanna IntHout

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  6. Joris AH De Groot

    Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  7. Steven AJ Chamuleau

    Department of Cardiology, Experimental Cardiology Laboratory, University Medical Center Utrecht, Utrecht, Netherlands
    Competing interests
    The authors declare that no competing interests exist.
  8. Malcolm R MacLeod

    Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
  9. Kimberley E Wever

    Systematic Review Centre for Laboratory Animal Experimentation, Radboud University Medical Center, Nijmegen, Netherlands
    For correspondence
    kim.wever@radboudumc.nl
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3635-3660

Funding

National Institute of Environmental Health Sciences (National Toxicology Program research funding)

  • Kimberley E Wever

Netherlands Cardiovascular Research Initiative (CVON-HUSTCARE)

  • Steven AJ Chamuleau

National Centre for the Replacement, Refinement and Reduction of Animals in Research (Infrastructure Award)

  • Emily S Sena
  • Malcolm R MacLeod

Alexander Suerman program (PhD student Scholarship)

  • Peter-Paul Zwetsloot

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Copyright

© 2017, Zwetsloot et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 6,086
    views
  • 448
    downloads
  • 142
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Peter-Paul Zwetsloot
  2. Mira Van Der Naald
  3. Emily S Sena
  4. David W Howells
  5. Joanna IntHout
  6. Joris AH De Groot
  7. Steven AJ Chamuleau
  8. Malcolm R MacLeod
  9. Kimberley E Wever
(2017)
Standardized mean differences cause funnel plot distortion in publication bias assessments
eLife 6:e24260.
https://doi.org/10.7554/eLife.24260

Share this article

https://doi.org/10.7554/eLife.24260

Further reading

    1. Epidemiology and Global Health
    2. Genetics and Genomics
    Tianyu Zhao, Hui Li ... Li Chen
    Research Article

    Alzheimer’s disease (AD) is a complex degenerative disease of the central nervous system, and elucidating its pathogenesis remains challenging. In this study, we used the inverse-variance weighted (IVW) model as the major analysis method to perform hypothesis-free Mendelian randomization (MR) analysis on the data from MRC IEU OpenGWAS (18,097 exposure traits and 16 AD outcome traits), and conducted sensitivity analysis with six models, to assess the robustness of the IVW results, to identify various classes of risk or protective factors for AD, early-onset AD, and late-onset AD. We generated 400,274 data entries in total, among which the major analysis method of the IVW model consists of 73,129 records with 4840 exposure traits, which fall into 10 categories: Disease, Medical laboratory science, Imaging, Anthropometric, Treatment, Molecular trait, Gut microbiota, Past history, Family history, and Lifestyle trait. More importantly, a freely accessed online platform called MRAD (https://gwasmrad.com/mrad/) has been developed using the Shiny package with MR analysis results. Additionally, novel potential AD therapeutic targets (CD33, TBCA, VPS29, GNAI3, PSME1) are identified, among which CD33 was positively associated with the main outcome traits of AD, as well as with both EOAD and LOAD. TBCA and VPS29 were negatively associated with the main outcome traits of AD, as well as with both EOAD and LOAD. GNAI3 and PSME1 were negatively associated with the main outcome traits of AD, as well as with LOAD, but had no significant causal association with EOAD. The findings of our research advance our understanding of the etiology of AD.

    1. Epidemiology and Global Health
    Xiaoning Wang, Jinxiang Zhao ... Dong Liu
    Research Article

    Artificially sweetened beverages containing noncaloric monosaccharides were suggested as healthier alternatives to sugar-sweetened beverages. Nevertheless, the potential detrimental effects of these noncaloric monosaccharides on blood vessel function remain inadequately understood. We have established a zebrafish model that exhibits significant excessive angiogenesis induced by high glucose, resembling the hyperangiogenic characteristics observed in proliferative diabetic retinopathy (PDR). Utilizing this model, we observed that glucose and noncaloric monosaccharides could induce excessive formation of blood vessels, especially intersegmental vessels (ISVs). The excessively branched vessels were observed to be formed by ectopic activation of quiescent endothelial cells (ECs) into tip cells. Single-cell transcriptomic sequencing analysis of the ECs in the embryos exposed to high glucose revealed an augmented ratio of capillary ECs, proliferating ECs, and a series of upregulated proangiogenic genes. Further analysis and experiments validated that reduced foxo1a mediated the excessive angiogenesis induced by monosaccharides via upregulating the expression of marcksl1a. This study has provided new evidence showing the negative effects of noncaloric monosaccharides on the vascular system and the underlying mechanisms.