Figures and data
Brief information about the four included journals from the publisher F1000.
Graphical summary of the study design for research question 1 showing a dummy article and two reviews.
In the first version of the article, the reviewer Smith (blue) is cited whilst Jones (purple) is not. For the second version of the article, the authors are now aware that Jones is a reviewer and Jones has been cited. The reviewers’ recommendations are the outcome and are colour-coded as Not approved (red), Reservations (orange) and Approved (green). We tested whether citations to the reviewer in the article influenced their recommendation. The matched design means that only reviewers of the same article are compared (here, Smith and Jones) and the overall effect is estimated by aggregating over multiple matched comparisons. Research question 2 used the same design but examined self-citations in the reviews.
Descriptive statistics for the articles and peer reviews.
Q1 = first quartile, Q3 = third quartile.
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation depending on whether they were cited in the article.
The top row examines Approved vs Reservations or Not approved, and the bottom row examines Approved or Reservations vs Not approved. The figures show the mean (dot) and adjusted 99.4% confidence intervals (horizontal lines). All models were split by article version. The odds ratios and probabilities show the same results but on different scales.
Odds ratios for reviewers giving a more (OR > 1) or less (OR < 1) favourable recommendation depending on whether they were cited in the article (question 1) or included self-citations to their own research (question 2). All models were split by article version.
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation if they included citations to their own research in their review.
The top row examines Approved vs Reservations or Not approved, and the bottom row examines Approved or Reservations vs Not approved. The figures show the mean (dot) and adjusted 99.4% confidence intervals (horizontal lines). All models were split by article version. The odds ratios and probabilities show the same results but on different scales.
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation depending on if they included citations to other research in their review.
The top row examines Approved vs Reservations or Not approved, and the bottom row examines Approved or Reservations vs Not approved. The figures show the mean (dot) and adjusted 99.4% confidence intervals (horizontal lines). All models were split by article version. The odds ratios and probabilities show the same results but on different scales.
Words in the reviewers’ comments that were associated with approving the article or not for reviewers who included a self-citation (n = 2, 710).
The words were selected using an elastic net that started with the 100 most commonly used review words with 28 retained. The estimates from the elastic net are shown as empty circles and the mean estimates and 95% credible intervals from a Bayesian model as shown as a solid circle and horizontal line. The axis label shows the stemmed word and most common whole word in brackets.
Flow chart of included reviews. ‘N’ is the number of articles and ‘n’ is the number of reviews.
Comparing the two alternatives for the citation predictor variables using either a linear variable or a binary “any versus none” variable.
A vs R/N = Approved vs Reservations/Not approved, A/R vs N = Approve/Reservations vs Not approved.
Estimated odds ratios for using linear citations as the predictor.
Results with or without co-reviewers for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation if they were cited. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. The plot is designed to directly compare paired odds ratios with or without co-reviewers.
Results with or without co-reviewers for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a self-citation. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. The plot is designed to directly compare paired odds ratios with or without co-reviewers.
Examining potential confounding by reviewers’ publication counts for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they were cited. We used fractional polynomials to examine a potentially non-linear association between reviewers’ publication counts and recommendation. The results for “None” are the results without the potential confounder. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. Results are missing when the model did not converge.
Examining potential confounding by reviewers’ publication counts for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a self-citation. We used fractional polynomials to examine a potentially non-linear association between reviewers’ publication counts and recommendation. The results for “None” are the results without the potential confounder. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. Results are missing when the model did not converge.
Leave-one-country-out sensitivity analyses for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they were cited. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version.
Leave-one-country-out sensitivity analyses for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a self-citation. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version.
Distributions of the error rates. Vaguely informative prior and posteriors for errors for not cited reviewers, cited reviewers and self-citations.
The dashed vertical lines are at Pr(error ≤ x) = 90%.
Number of errors found in our data extraction algorithm from manual checks and the estimated 90% limit for the error rate
Histogram of online view counts of published reviews. The bins are in tens starting at [0, 10).
Example sentences that reviewers used when suggesting self-citations, using a random sample of 20 reviews.
The first column shows the number of self-citations suggested. We have removed any references to names using [xxxx]. The results are ordered by sentence length.
