Figures and data
Brief information about the four included journals from the publisher F1000.
Graphical summary of the study design for research question 1 showing a dummy article and two reviews.
In the first version of the article, the reviewer Smith (blue) is cited whilst Jones (purple) is not. For the second version of the article, the authors are now aware that Jones is a reviewer and Jones has been cited. The reviewers’ recommendations are the outcome and are colour-coded as Not approved (red), Reservations (orange) and Approved (green). We tested whether citations to the reviewer in the article influenced their recommendation. The matched design means that only reviewers of the same article are compared (here, Smith and Jones) and the overall effect is estimated by aggregating over multiple matched comparisons. Research question 2 used the same design but examined citations to the reviewers’ articles in their reviews.
Descriptive statistics for the articles and peer reviews.
Q1 = first quartile, Q3 = third quartile.
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation depending on whether they were cited in the article.
Top left: Odds ratios for reviewers giving a more favourable (Approved) or less favourable (Reservations or Not approved) recommendation depending on whether they were cited in the article. Reviewers cited in later versions (blue) were more likely to make a favourable recommendation (odds ratio = 1.61; adjusted 99.4% CI: 1.16 to 2.23), whereas being cited in the first version (green) did not improve their recommendation (odds ratio = 0.84; adjusted 99.4% CI: 0.69 to 1.03). Top right: Same results as top left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which they are not cited (0.51; adjusted 99.4% CI: 0.49 to 0.52); a version 1 article in which they are cited (0.46; adjusted 99.4% CI: 0.41 to 0.51); a version 2 (or higher) article in which they are not cited (0.48; adjusted 99.4% CI: 0.45 to 0.51); and a version 2 (or higher) article in which they are cited (0.60; adjusted 99.4% CI: 0.53 to 0.65). Bottom left: Same estimates as top left except that a more favourable recommendation is now Approved or Reservations and a less favourable is Not approved. There was no clear association for cited reviewers in version 1 (odds ratio = 0.84; adjusted 99.4% CI: 0.57 to 1.23) or later versions (odds ratio = 1.12; adjusted 99.4% CI: 0.59 to 2.13). Bottom right: Same results as bottom left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which they are not cited (0.51; adjusted 99.4% CI: 0.48 to 0.53); a version 1 article in which they are cited (0.47; adjusted 99.4% CI: 0.35 to 0.55); a version 2 (or higher) article in which they are not cited (0.50; adjusted 99.4% CI: 0.43 to 0.55); and a version 2 (or higher) article in which they are cited (0.53; adjusted 99.4% CI: 0.34 to 0.63). This figure is based on an analysis of 12,051 articles and 24,677 reviews for version 1 and 6090 articles and 10,196 reviews for version 2+. In all panels a dot or square represents a mean, and a horizontal line represents an adjusted 99.4% confidence interval.
Odds ratios for reviewers giving a more (OR > 1) or less (OR < 1) favourable recommendation depending on whether they were cited in the article (question 1) or included citations to their own articles (question 2).
All models were split by article version.
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation if they included a citation to their own articles in their review.
Top left: Odds ratios for reviewers giving a more favourable (Approved) or less favourable (Reservations or Not approved) recommendation depending on whether their review included a citation to their own articles. Reviewers including a citation to their own articles were less likely to make a favourable recommendation for version 1 (green; odds ratio = 0.57; adjusted 99.4% CI: 0.44 to 0.73) and later versions (blue; odds ratio = 0.15; adjusted 99.4% CI: 0.07 to 0.30). Top right: Same results as top left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not include a citation (0.51; adjusted 99.4% CI: 0.50 to 0.53); a version 1 article in which in which their review included a citation (0.37; adjusted 99.4% CI: 0.29 to 0.44); a version 2 (or higher) article in which their review did not include a citation (0.51; adjusted 99.4% CI: 0.49 to 0.53); and a version 2 (or higher) article in which in which their review included a citation (0.14; adjusted 99.4% CI: 0.01 to 0.30). Bottom left: Same estimates as top left except that a more favourable recommendation is now Approved or Reservations and a less favourable is Not approved. There was no clear association for reviewers who included a citation to their own articles in version 1 (odds ratio = 1.11; adjusted 99.4% CI: 0.77 to 1.60) or later versions (odds ratio = 0.80; adjusted 99.4% CI: 0.37 to 1.74). Bottom right: Same results as bottom left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not include a citation (0.50; adjusted 99.4% CI: 0.48 to 0.52); a version 1 article in which their review included a citation (0.53; adjusted 99.4% CI: 0.43 to 0.60); a version 2 (or higher) article in which their review did not include a citation (0.51; adjusted 99.4% CI: 0.46 to 0.54); and a version 2 (or higher) article in which their included a citation (0.45; adjusted 99.4% CI: 0.12 to 0.61). This figure is based on an analysis of 12,078 articles and 24,732 reviews for version 1 and 6101 articles and 10,213 reviews for version 2+. In all panels a dot or square represents a mean, and a horizontal line represents an adjusted 99.4% confidence interval.
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation depending on if they included citations to articles other than their own in their review.
Top left: Odds ratios for reviewers giving a more favourable (Approved) or less favourable (Reservations or Not approved) recommendation depending on whether their review included a citation to articles other than their own. Reviewers including citations to other articles were less likely to make a favourable recommendation for version 1 (green; odds ratio = 0.53; adjusted 99.4% CI: 0.44 to 0.64) and later versions (blue; odds ratio = 0.18; adjusted 99.4% CI: 0.10 to 0.30). Top right: Same results as top left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not cite other articles (0.53; adjusted 99.4% CI: 0.51 to 0.54); a version 1 article in which their review cited other articles (0.37; adjusted 99.4% CI: 0.31 to 0.42); a version 2 (or higher) article in which their review did not cite other articles (0.52; adjusted 99.4% CI: 0.50 to 0.54); and a version 2 (or higher) article in which in which their review cited other articles (0.17; adjusted 99.4% CI: 0.02 to 0.30). Bottom left: Same estimates as top left except that a more favourable recommendation is now Approved or Reservations and a less favourable is Not approved. Reviewers including citations to other articles were less likely to make a favourable recommendation for version 1 (odds ratio = 0.62; adjusted 99.4% CI: 0.46 to 0.84) and later versions (odds ratio = 0.34; adjusted 99.4% CI: 0.16 to 0.73). Bottom right: Same results as bottom left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not cite other articles (0.52; adjusted 99.4% CI: 0.49 to 0.54); a version 1 article in which their review cited other articles (0.41; adjusted 99.4% CI: 0.31 to 0.48); a version 2 (or higher) article in which their review did not cite other articles (0.52; adjusted 99.4% CI: 0.47 to 0.55); and a version 2 (or higher) article in which their review cited other articles (0.27; adjusted 99.4% CI: 0.02 to 0.45). This figure is based on an analysis of 12,078 articles and 24,732 reviews for version 1 and 6101 articles and 10,213 reviews for version 2+. In all panels a dot or square represents a mean, and a horizontal line represents an adjusted 99.4% confidence interval.
Words in the reviewers’ comments that were associated with approving the article or not for reviewers who included a citation to their own articles (n = 2, 025) and reviewers who included citations to other articles (n = 4, 350).
The words were selected using an elastic net that started with the 100 most commonly used review words. The estimates from the elastic net are shown as empty circles and the mean estimates and 95% credible intervals from a Bayesian model are shown as a solid circle and horizontal line. Words are shown if the probability of a non-zero mean was over 0.95 for either reviewers who cited their own articles or reviewers who cited other articles. Four words were selected by the elastic net for the reviewers who cited other articles but not by the elastic net for reviewers who cited their own articles. The axis label shows the stemmed word and most common whole word in brackets.
Flow chart of included reviews.
‘N’ is the number of articles and ‘n’ is the number of reviews.
Comparing the two alternatives for the citation predictor variables using either a linear variable or a binary “any versus none” variable.
A vs R/N = Approved vs Reservations/Not approved, A/R vs N = Approved/Reservations vs Not approved.
Estimated odds ratios for using linear citations as the predictor.
The reference point is zero citations.
Results with or without co-reviewers for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation if they were cited. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. The plot is designed to directly compare paired odds ratios with or without co-reviewers.
Results with or without co-reviewers for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a citation to their own articles. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. The plot is designed to directly compare paired odds ratios with or without co-reviewers.
Examining potential confounding by reviewers’ publication counts for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they were cited. We used fractional polynomials to examine a potentially non-linear association between reviewers’ publication counts and recommendation. The results for “None” are the results without the potential confounder. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. Results are missing when the model did not converge.
Examining potential confounding by reviewers’ publication counts for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a citation to their own articles. We used fractional polynomials to examine a potentially non-linear association between reviewers’ publication counts and recommendation. The results for “None” are the results without the potential confounder. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. Results are missing when the model did not converge.
Leave-one-country-out sensitivity analyses for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they were cited. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version.
Leave-one-country-out sensitivity analyses for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a citation to their own articles. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version.
Example sentences that reviewers used when suggesting citations to their own articles using a random sample of 20 reviews.
The first column shows the number of citations suggested. We have removed any references to names using [xxxx]. The results are ordered by text length.
Histogram of online view counts of published reviews.
The bins are in tens starting at [0, 10).
Distributions of the error rates.
Vaguely informative prior and posteriors for errors for not cited reviewers, cited reviewers and self-citations. The dashed vertical lines are at Pr(error ≤ x) = 90%.
