# Standardized mean differences cause funnel plot distortion in publication bias assessments

1. University Medical Center Utrecht, Netherlands
2. Netherlands Heart Institute, Netherlands
3. University of Edinburgh, United Kingdom
4. University of Tasmania, Australia
5. Radboud University Medical Center, Netherlands
7 figures, 4 tables and 4 additional files

## Figures

Figure 1 Hypothetical funnel plots in the absence (A) and presence (B) of bias. The precision estimate used is the standard error (SE). Dashed lines indicate the summary effect estimate. https://doi.org/10.7554/eLife.24260.002
Figure 2 Step-wise illustration of distortion in SMD versus SE funnel plots. (A) Depicted are simulated studies with a sample size of respectively 10 (large black circles), 25 (blue squares), 50 (red triangles), 100 (small green circles) and 200 (gold asterisks) subjects per group, and an SMD of zero. The SE of these studies (indicated by the dashed line for studies with n = 10) solely depends on their sample size, as SMD2 = 0 and therefore does not contribute to the equation for the SE. As expected, the SE decreases as the sample size increases. (B) Five data points from simulated studies with n = 10 and a stepwise increasing SMD are added to the plot. For these studies, the SMD2 contributes to the equation for the SE, and the SE will decrease even though the sample size is constant. The dotted line represents a hypothetical summary effect of SMD = 1 in a meta-analysis. Note that when assessing a funnel plot for asymmetry around this axis, the data points with an SMD < 1 have skewed to the upper left-hand region, whereas studies with an SMD > 1 are in the lower right region of the plot. This distortion worsens as the SMD increases. (C) Because the SMD is squared in the equation for the SE, the same distortion pattern is observed for negative SMDs. Thus, funnel plots will be distorted most when the study samples sizes are small and SMDs are either very positive or very negative. (D) The same deviation is observed for simulated studies with larger sample sizes, however, the deviation decreases as the sample size increases, because the sample size will outweigh the effect of SMD2 in the equation for the SE. https://doi.org/10.7554/eLife.24260.003
Figure 3 Reanalysis of data from Wever et al. (A,B) and Zwetsloot et al. (C,D), with funnel plots based on raw mean difference (RMD; A,D) or standardized mean difference (SMD; B,D). Filled circles = observed data points; open circles = missing data points as suggested by trim and fill analysis. https://doi.org/10.7554/eLife.24260.004
Figure 4 with 2 supplements Representative raw mean difference (RMD; A, C, E, G) and standardized mean difference (Hedges’ g SMD; B, D, F, H) funnel plots for simulated unbiased meta-analyses containing thirty (A–D) or 300 (E–H) studies with a small sample size (total study n = 12–30). Simulations were performed without an intervention effect (Δμ = 0; A–B and E–F), or with an intervention effect (Δμ = 10; C–D and G–H). Δμ = difference in normally distributed means between control and intervention group. Representative funnel plots for studies with a large sample size (total study n = 60–320) are shown in Figure 4—figure supplement 1. Representative funnel plots for the comparison between Hedges’ g and Cohen’s d are shown in Figure 4—figure supplement 2. https://doi.org/10.7554/eLife.24260.005
Figure 4—figure supplement 1
Figure 4—figure supplement 2
Figure 5 raw mean difference (RMD; A), standardized mean difference (SMD; B), normalized mean difference (NMD; C) with SE as precision estimate, and SMD funnel plots using 1/√n as precision estimate (D). All plots show the same simulated meta-analysis containing 3000 studies with small sample sizes (n = 12–30) and an overall intervention effect of Δμ = 10. Δμ = difference in normally distributed means between control and intervention group. https://doi.org/10.7554/eLife.24260.010
Figure 6 Simulation 3 funnel plots of biased meta-analyses. Representative funnel plots of simulated biased meta-analyses using a raw mean difference (RMD; A–B), a standardized mean difference (SMD; C–D), or a normalised mean difference (NMD; E–F) effect measure. The present example contains 3000 studies with a small study sample size (n = 12–30) and an intervention effect present (difference in normal distribution means between control and intervention group = 10). Publication bias was introduced stepwise, by removing 10% of primary studies in which the difference between the intervention and control group means was significant at p<0.05, 50% of studies where the significance level was p≥0.05 to p<0.10, and 90% of studies where the significance level was p≥0.10. Precision estimates are standard error (A, C, E) or sample size-based (B, D, F), where n = total primary study sample size. https://doi.org/10.7554/eLife.24260.011
Figure 7 Funnel plots of re-analysis of empirical meta-analyses. Funnel plots of empirical meta-analyses plotted as standardized mean difference (SMD) versus standard error, as in the original publications (left hand panels), and as SMD versus 1/√n after re-analysis. n = total primary study sample size; filled circles = observed data points; open circles = missing data points as suggested by trim and fill analysis. https://doi.org/10.7554/eLife.24260.013

## Tables

Table 1
Table 2
Table 3
Table 4

###### Supplementary file 1

Cohen’s d and Hedges’ g, as well as Egger’s test and Begg and Mazumdar’s test, perform similar in multiple illustrative scenario’s (simulation 1).

###### Supplementary file 2

Supplemental equations.

R scripts.

###### Transparent reporting form

A two-part list of links to download the article, or parts of the article, in various formats.

### Open citations (links to open the citations from this article in various online reference manager services)

1. Peter-Paul Zwetsloot
2. Mira Van Der Naald
3. Emily S Sena
4. David W Howells
5. Joanna IntHout
6. Joris AH De Groot
7. Steven AJ Chamuleau
8. Malcolm R MacLeod
9. Kimberley E Wever
(2017)
Standardized mean differences cause funnel plot distortion in publication bias assessments
eLife 6:e24260.
https://doi.org/10.7554/eLife.24260