Research: Publication bias and the canonization of false facts

University of Copenhagen, Denmark
University of Washington, United States
North Carolina State University, United States

Dec 20, 2016

https://doi.org/10.7554/eLife.21451

Open access
Copyright information

Download
Cite
CommentOpen annotations (there are currently 0 annotations on this page).
Share

9 figures

Figures

Figure 1

Download asset Open asset

Conducting and reporting the test of a claim.

In our model, a scientific claim is either true or false. Researchers conduct an experiment which either supports or fails to the support the claim. True claims are correctly supported with probability $1 - β$ while false claims are incorrectly supported with probability $α$ . Next, the researchers may attempt to publish their results. Positive results that support the claim are published with probability $ρ_{1}$ whereas negative results that fail to support the claim are published with probability $ρ_{0}$ . This process then repeats, with additional experiments conducted until the claim is canonized as fact or rejected as false.

https://doi.org/10.7554/eLife.21451.002

Figure 2

Download asset Open asset

A time-directed graph represents the evolution of belief over time.

In panel A, the horizontal axis indicates the number of experiments published and the vertical axis reflects the observer’s belief, quantified as the probability that the claim is true. The process begins at the single point at far left with an initial belief $q_{0}$ . Each subsequent experiment either supports the claim, moving to the next node up and right, or contradicts the claim, moving to the next node down and right. At yellow nodes, the status of the claim is as yet undecided. At green nodes, it is canonized as fact, and at blue nodes, it is rejected as false. The black horizontal lines show the evidentiary standards ( $τ_{0}$ and $τ_{1}$ ). The red path shows one possible trajectory, in which a positive experiment is followed by a negative, then two positives, then a negative, etc., ultimately becoming canonized as fact when it reaches the upper boundary. Panel B shows the same network, but with the vertical axis representing log odds and using color to indicate the probability that the process visits each node. In log-odds space, each published positive result shifts belief by the constant distance $d_{1} > 0$ and each negative result by a different distance $d_{0} < 0$ . Shown here (in both panel A and B) is a false claim with false positive rate $α = 0.2$ , false negative rate $β = 0.4$ , publication probabilities $p_{0} = 0.1$ and $p_{1} = 1$ , and initial belief $q_{0} = 0.1$ . In this case, the claim is likely to be canonized as fact, despite being false.

https://doi.org/10.7554/eLife.21451.003

Figure 3

Download asset Open asset

ROC curves reveal that true claims are almost always canonized as fact.

In the receiver operating characteristic (ROC) curves shown here, the vertical axis represents the probability that a true claim is correctly canonized as fact, and the horizontal axis represents the probability that a false one is incorrectly canonized as fact. Panel A: lax evidentiary standards $τ_{0} = 0.1$ and $τ_{1} = 0.9$ . Panel B: strict evidentiary standards $τ_{0} = 0.001$ and $τ_{1} = 0.999$ . Error rates and initial belief are $α = 0.05$ , $β = 0.2$ , and $q_{0} = 0.5$ . Each point along the ROC curve corresponds to a different value of the negative publication rate, $ρ_{0}$ , as indicated by color. Grey regions of the curve correspond to the unlikely situations in which $ρ_{0} > ρ_{1} = 1$ , i.e., negative results are more likely to be published than positive ones. The figures reveal two important points. First, when negative results are published at any rate $ρ_{0} \leq 1$ , the vast majority of true claims are canonized as fact. Second, when negative results are published at a low rate ( $ρ_{0}$ less than 0.3 or 0.2 depending on evidentiary standards), many false claims will also be canonized as true.

https://doi.org/10.7554/eLife.21451.004

Figure 4

Download asset Open asset

Publishing negative outcomes is essential for rejecting false claims.

Probability that a false claim is incorrectly canonized, as a function of the negative publication rate. Throughout, initial belief is $q_{0} = 0.5$ , and individual data series show false positive rates $α = 0.05$ (yellow), $0.10, \dots, 0.25$ (red). Top row: weak evidentiary standards $τ_{0} = 0.1$ and $τ_{1} = 0.9$ . Panel A: false negative rate $β = 0.2$ . Panel B: $β = 0.4$ . Panels C–D: similar to panels A–B, with more demanding evidentiary standards $τ_{0} = 0.001$ and $τ_{1} = 0.999$ .

https://doi.org/10.7554/eLife.21451.005

Figure 5

Download asset Open asset

False canonization rates are relatively insensitive to initial belief, unless experimental tests are inaccurate and evidentiary standards are weak.

Probability that a false claim is mistakenly canonized as a true fact vs. prior belief for various negative publication rates. Top row: weak evidentiary standards $τ_{0} = 0.1$ and $τ_{1} = 0.9$ . Panel A: false positive rate $α = 0.05$ , false negative rate $β = 0.2$ , and publication rate of negative results $ρ_{0} = 0.025$ (light green), $0.05, 0.1, 0.2, 0.4$ (dark green). Panel B: $α = 0.2$ , $β = 0.4$ , and $ρ_{0} = 0.1$ (light green), $0.3, 0.4, 0.5, 1$ (dark green). Panels C–D: similar to panels A–B, with more demanding evidentiary standards $τ_{0} = 0.001$ and $τ_{1} = 0.999$ .

https://doi.org/10.7554/eLife.21451.006

Figure 6

Download asset Open asset

Strengthening evidentiary requirements does not necessarily decrease canonization of false facts.

In panel A, the false positive rate is $α = 0.05$ , the false negative rate is $β = 0.2$ , the original belief in the claim is $q_{0} = 0.5$ , and the evidentiary standards are symmetric $τ_{1} = 1 - τ_{0}$ . In panel B, the false positive rate is increased to $α = 0.25$ while the other parameters remain unchanged. Particularly in this latter case, increasing evidentiary standards does not necessarily decrease the rate at which false claims are canonized as facts.

https://doi.org/10.7554/eLife.21451.007

Figure 7

Download asset Open asset

Scientific activity will tend to increase belief in false claims if too few negative outcomes are published.

Expected change in log odds of belief vs. negative publication rate for (A) false and (B) true claims. Lines show false positive rates $α = 0.05$ (yellow), $0.10, \dots, 0.25$ (red). Other parameter values are false negative rate $β = 0.2$ and positive publication rate $ρ_{1} = 1$ .

https://doi.org/10.7554/eLife.21451.008

Figure 8

Download asset Open asset

$p$ -hacking dramatically increases the chances of canonizing false claims.

Probability that a false claim is canonized as fact vs. fraction of negative outcomes. Throughout, all positive outcomes are published ( $p_{1} = 1$ ), and the nominal false positive rate is $α_{nom} = 0.05$ , the false negative rate is $β = 0.2$ , and evidentiary standards are strong ( $τ_{0} = 0.001$ and $τ_{1} = 0.999$ ). Curves show actual false positive rates $α_{act} = 0.05$ (yellow), $0.10, \dots, 0.25$ (red). Compared with Figure 4C, in which the nominal rates are equal to the actual rates, the probability of canonizing a false claim as fact is substantially higher.

https://doi.org/10.7554/eLife.21451.009

Figure 9

Download asset Open asset

Publishing a larger fraction of negative outcomes as belief increases lessens the chances of canonizing false claims.

Probability that a false claim is mistakenly canonized as a true fact vs. baseline probability of publishing a negative outcome. The baseline probability of publishing a negative outcome is the probability that prevails when belief in the claim is weak. The actual probability of publishing a negative outcome increases linearly from the baseline rate when belief is 0 to a value of 1 when belief is 1. All other parameters are the same as in Figure 4.

https://doi.org/10.7554/eLife.21451.010

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Article PDF

Open citations (links to open the citations from this article in various online reference manager services)

Mendeley

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

Silas Boye Nissen
Tali Magidson
Kevin Gross
Carl T Bergstrom

(2016)

Research: Publication bias and the canonization of false facts

eLife 5:e21451.

https://doi.org/10.7554/eLife.21451

Sign up for email alerts

Privacy notice