Research: Publication bias and the canonization of false facts

  1. Silas Boye Nissen
  2. Tali Magidson
  3. Kevin Gross  Is a corresponding author
  4. Carl T Bergstrom  Is a corresponding author
  1. University of Copenhagen, Denmark
  2. University of Washington, United States
  3. North Carolina State University, United States
9 figures

Figures

Conducting and reporting the test of a claim.

In our model, a scientific claim is either true or false. Researchers conduct an experiment which either supports or fails to the support the claim. True claims are correctly supported with probability 1-β while false claims are incorrectly supported with probability α. Next, the researchers may attempt to publish their results. Positive results that support the claim are published with probability ρ1 whereas negative results that fail to support the claim are published with probability ρ0. This process then repeats, with additional experiments conducted until the claim is canonized as fact or rejected as false.

https://doi.org/10.7554/eLife.21451.002
A time-directed graph represents the evolution of belief over time.

In panel A, the horizontal axis indicates the number of experiments published and the vertical axis reflects the observer’s belief, quantified as the probability that the claim is true. The process begins at the single point at far left with an initial belief q0. Each subsequent experiment either supports the claim, moving to the next node up and right, or contradicts the claim, moving to the next node down and right. At yellow nodes, the status of the claim is as yet undecided. At green nodes, it is canonized as fact, and at blue nodes, it is rejected as false. The black horizontal lines show the evidentiary standards (τ0 and τ1). The red path shows one possible trajectory, in which a positive experiment is followed by a negative, then two positives, then a negative, etc., ultimately becoming canonized as fact when it reaches the upper boundary. Panel B shows the same network, but with the vertical axis representing log odds and using color to indicate the probability that the process visits each node. In log-odds space, each published positive result shifts belief by the constant distance d1>0 and each negative result by a different distance d0<0. Shown here (in both panel A and B) is a false claim with false positive rate α=0.2, false negative rate β=0.4, publication probabilities p0=0.1 and p1=1, and initial belief q0=0.1. In this case, the claim is likely to be canonized as fact, despite being false.

https://doi.org/10.7554/eLife.21451.003
ROC curves reveal that true claims are almost always canonized as fact.

In the receiver operating characteristic (ROC) curves shown here, the vertical axis represents the probability that a true claim is correctly canonized as fact, and the horizontal axis represents the probability that a false one is incorrectly canonized as fact. Panel A: lax evidentiary standards τ0=0.1 and τ1=0.9. Panel B: strict evidentiary standards τ0=0.001 and τ1=0.999. Error rates and initial belief are α=0.05, β=0.2, and q0=0.5. Each point along the ROC curve corresponds to a different value of the negative publication rate, ρ0, as indicated by color. Grey regions of the curve correspond to the unlikely situations in which ρ0>ρ1=1, i.e., negative results are more likely to be published than positive ones. The figures reveal two important points. First, when negative results are published at any rate ρ01, the vast majority of true claims are canonized as fact. Second, when negative results are published at a low rate (ρ0 less than 0.3 or 0.2 depending on evidentiary standards), many false claims will also be canonized as true.

https://doi.org/10.7554/eLife.21451.004
Publishing negative outcomes is essential for rejecting false claims.

Probability that a false claim is incorrectly canonized, as a function of the negative publication rate. Throughout, initial belief is q0=0.5, and individual data series show false positive rates α=0.05 (yellow), 0.10,,0.25 (red). Top row: weak evidentiary standards τ0=0.1 and τ1=0.9. Panel A: false negative rate β=0.2. Panel B: β=0.4. Panels CD: similar to panels AB, with more demanding evidentiary standards τ0=0.001 and τ1=0.999.

https://doi.org/10.7554/eLife.21451.005
False canonization rates are relatively insensitive to initial belief, unless experimental tests are inaccurate and evidentiary standards are weak.

Probability that a false claim is mistakenly canonized as a true fact vs. prior belief for various negative publication rates. Top row: weak evidentiary standards τ0=0.1 and τ1=0.9. Panel A: false positive rate α=0.05, false negative rate β=0.2, and publication rate of negative results ρ0=0.025 (light green), 0.05,0.1,0.2,0.4 (dark green). Panel B: α=0.2, β=0.4, and ρ0=0.1 (light green), 0.3,0.4,0.5,1 (dark green). Panels CD: similar to panels AB, with more demanding evidentiary standards τ0=0.001 and τ1=0.999.

https://doi.org/10.7554/eLife.21451.006
Strengthening evidentiary requirements does not necessarily decrease canonization of false facts.

In panel A, the false positive rate is α=0.05, the false negative rate is β=0.2, the original belief in the claim is q0=0.5, and the evidentiary standards are symmetric τ1=1-τ0. In panel B, the false positive rate is increased to α=0.25 while the other parameters remain unchanged. Particularly in this latter case, increasing evidentiary standards does not necessarily decrease the rate at which false claims are canonized as facts.

https://doi.org/10.7554/eLife.21451.007
Scientific activity will tend to increase belief in false claims if too few negative outcomes are published.

Expected change in log odds of belief vs. negative publication rate for (A) false and (B) true claims. Lines show false positive rates α=0.05 (yellow), 0.10,,0.25 (red). Other parameter values are false negative rate β=0.2 and positive publication rate ρ1=1.

https://doi.org/10.7554/eLife.21451.008
p-hacking dramatically increases the chances of canonizing false claims.

Probability that a false claim is canonized as fact vs. fraction of negative outcomes. Throughout, all positive outcomes are published (p1=1), and the nominal false positive rate is αnom=0.05, the false negative rate is β=0.2, and evidentiary standards are strong (τ0=0.001 and τ1=0.999). Curves show actual false positive rates αact=0.05 (yellow), 0.10,,0.25 (red). Compared with Figure 4C, in which the nominal rates are equal to the actual rates, the probability of canonizing a false claim as fact is substantially higher.

https://doi.org/10.7554/eLife.21451.009
Publishing a larger fraction of negative outcomes as belief increases lessens the chances of canonizing false claims.

Probability that a false claim is mistakenly canonized as a true fact vs. baseline probability of publishing a negative outcome. The baseline probability of publishing a negative outcome is the probability that prevails when belief in the claim is weak. The actual probability of publishing a negative outcome increases linearly from the baseline rate when belief is 0 to a value of 1 when belief is 1. All other parameters are the same as in Figure 4.

https://doi.org/10.7554/eLife.21451.010

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Silas Boye Nissen
  2. Tali Magidson
  3. Kevin Gross
  4. Carl T Bergstrom
(2016)
Research: Publication bias and the canonization of false facts
eLife 5:e21451.
https://doi.org/10.7554/eLife.21451