Are peer reviewers influenced by their work being cited?
- School of Public Health and Social Work, Queensland University of Technology, Australia
Figures
Figure 1
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation depending on whether they were cited in the article.
Top left: Odds ratios for reviewers giving a more favourable (Approved) or less favourable (Reservations or Not approved) recommendation depending on whether they were cited in the article. Reviewers cited in later versions (blue) were more likely to make a favourable recommendation (odds ratio = 1.61; adjusted 99.4% CI: 1.16–2.23), whereas being cited in the first version (green) did not improve their recommendation (odds ratio = 0.84; adjusted 99.4% CI: 0.69–1.03). Top right: Same results as top left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which they are not cited (0.51; adjusted 99.4% CI: 0.49–0.52); a version 1 article in which they are cited (0.46; adjusted 99.4% CI: 0.41–0.51); a version 2 (or higher) article in which they are not cited (0.48; adjusted 99.4% CI: 0.45–0.51); and a version 2 (or higher) article in which they are cited (0.60; adjusted 99.4% CI: 0.53–0.65). Bottom left: Same estimates as top left except that a more favourable recommendation is now Approved or Reservations and a less favourable is Not approved. There was no clear association for cited reviewers in version 1 (odds ratio = 0.84; adjusted 99.4% CI: 0.57–1.23) or later versions (odds ratio = 1.12; adjusted 99.4% CI: 0.59–2.13). Bottom right: Same results as bottom left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which they are not cited (0.51; adjusted 99.4% CI: 0.48–0.53); a version 1 article in which they are cited (0.47; adjusted 99.4% CI: 0.35–0.55); a version 2 (or higher) article in which they are not cited (0.50; adjusted 99.4% CI: 0.43–0.55); and a version 2 (or higher) article in which they are cited (0.53; adjusted 99.4% CI: 0.34–0.63). This figure is based on an analysis of 12,051 articles and 24,677 reviews for version 1 and 6090 articles and 10,196 reviews for version 2+. In all panels, a dot or square represents a mean, and a horizontal line represents an adjusted 99.4% confidence interval.
Figure 2
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation if they included a citation to their own articles in their review.
Top left: Odds ratios for reviewers giving a more favourable (Approved) or less favourable (Reservations or Not approved) recommendation depending on whether their review included a citation to their own articles. Reviewers including a citation to their own articles were less likely to make a favourable recommendation for version 1 (green; odds ratio = 0.57; adjusted 99.4% CI: 0.44–0.73) and later versions (blue; odds ratio = 0.15; adjusted 99.4% CI: 0.08–0.30). Top right: Same results as top left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not include a citation (0.51; adjusted 99.4% CI: 0.50–0.53); a version 1 article in which in which their review included a citation (0.37; adjusted 99.4% CI: 0.29–0.44); a version 2 (or higher) article in which their review did not include a citation (0.51; adjusted 99.4% CI: 0.49–0.53); and a version 2 (or higher) article in which in which their review included a citation (0.14; adjusted 99.4% CI: 0.01–0.30). Bottom left: Same estimates as top left except that a more favourable recommendation is now Approved or Reservations and a less favourable is Not approved. There was no clear association for reviewers who included a citation to their own articles in version 1 (odds ratio = 1.11; adjusted 99.4% CI: 0.77–1.60) or later versions (odds ratio = 0.80; adjusted 99.4% CI: 0.37–1.74). Bottom right: Same results as bottom left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not include a citation (0.50; adjusted 99.4% CI: 0.48–0.52); a version 1 article in which their review included a citation (0.53; adjusted 99.4% CI: 0.43–0.60); a version 2 (or higher) article in which their review did not include a citation (0.51; adjusted 99.4% CI: 0.46–0.54); and a version 2 (or higher) article in which they included a citation (0.45; adjusted 99.4% CI: 0.12–0.61). This figure is based on an analysis of 12,078 articles and 24,732 reviews for version 1 and 6101 articles and 10,213 reviews for version 2+. In all panels, a dot or square represents a mean, and a horizontal line represents an adjusted 99.4% confidence interval.
Figure 3
Odds ratios and probabilities for reviewers giving a more or less favourable recommendation depending on if they included citations to articles other than their own in their review.
Top left: Odds ratios for reviewers giving a more favourable (Approved) or less favourable (Reservations or Not approved) recommendation depending on whether their review included a citation to articles other than their own. Reviewers including citations to other articles were less likely to make a favourable recommendation for version 1 (green; odds ratio = 0.53; adjusted 99.4% CI: 0.44–0.64) and later versions (blue; odds ratio = 0.18; adjusted 99.4% CI: 0.10–0.30). Top right: Same results as top left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not cite other articles (0.53; adjusted 99.4% CI: 0.51–0.54); a version 1 article in which their review cited other articles (0.37; adjusted 99.4% CI: 0.31–0.42); a version 2 (or higher) article in which their review did not cite other articles (0.52; adjusted 99.4% CI: 0.50–0.54); and a version 2 (or higher) article in which in which their review cited other articles (0.17; adjusted 99.4% CI: 0.02–0.30). Bottom left: Same estimates as top left except that a more favourable recommendation is now Approved or Reservations and a less favourable is Not approved. Reviewers including citations to other articles were less likely to make a favourable recommendation for version 1 (odds ratio = 0.62; adjusted 99.4% CI: 0.46–0.84) and later versions (odds ratio = 0.34; adjusted 99.4% CI: 0.16–0.73). Bottom right: Same results as bottom left displayed as conditional probabilities. From the top, the lines show the within-strata probability of a reviewer approving: a version 1 article in which their review did not cite other articles (0.52; adjusted 99.4% CI: 0.49–0.54); a version 1 article in which their review cited other articles (0.41; adjusted 99.4% CI: 0.31–0.48); a version 2 (or higher) article in which their review did not cite other articles (0.52; adjusted 99.4% CI: 0.47–0.55); and a version 2 (or higher) article in which their review cited other articles (0.27; adjusted 99.4% CI: 0.02–0.45). This figure is based on an analysis of 12,078 articles and 24,732 reviews for version 1 and 6101 articles and 10,213 reviews for version 2+. In all panels, a dot or square represents a mean, and a horizontal line represents an adjusted 99.4% confidence interval.
Figure 4
Words in the reviewers’ comments that were associated with approving the article or not for reviewers who included a citation to their own articles () and reviewers who included citations to other articles ().
The words were selected using an elastic net that started with the 100 most commonly used review words. The estimates from the elastic net are shown as empty circles and the mean estimates and 95% credible intervals from a Bayesian model are shown as a solid circle and horizontal line. Words are shown if the probability of a non-zero mean was over 0.95 for either reviewers who cited their own articles or reviewers who cited other articles. Four words were selected by the elastic net for the reviewers who cited other articles but not by the elastic net for reviewers who cited their own articles. The axis label shows the stemmed word and most common whole word in brackets.
Figure 5
Graphical summary of the study design for research question 1 showing a dummy article and two reviews.
In the first version of the article, the reviewer Smith (blue) is cited whilst Jones (purple) is not. For the second version of the article, the authors are now aware that Jones is a reviewer and Jones has been cited. The reviewers’ recommendations are the outcome and are colour-coded as Not approved (red), Reservations (orange) and Approved (green). We tested whether citations to the reviewer in the article influenced their recommendation. The matched design means that only reviewers of the same article are compared (here, Smith and Jones) and the overall effect is estimated by aggregating over multiple matched comparisons. Research question 2 used the same design but examined citations to the reviewers’ articles in their reviews.
Appendix 2—figure 1
Flow chart of included reviews.
‘N’ is the number of articles and ‘n’ is the number of reviews.
Appendix 4—figure 1
Estimated odds ratios for using linear citations as the predictor.
The reference point is zero citations.
Appendix 5—figure 1
Results with or without co-reviewers for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation if they were cited. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. The plot is designed to directly compare paired odds ratios with or without co-reviewers.
Appendix 5—figure 2
Results with or without co-reviewers for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a citation to their own articles. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. The plot is designed to directly compare paired odds ratios with or without co-reviewers.
Appendix 6—figure 1
Examining potential confounding by reviewers’ publication counts for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they were cited. We used fractional polynomials to examine a potentially non-linear association between reviewers’ publication counts and recommendation. The results for ‘None’ are the results without the potential confounder. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. Results are missing when the model did not converge.
Appendix 6—figure 2
Examining potential confounding by reviewers’ publication counts for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a citation to their own articles. We used fractional polynomials to examine a potentially non-linear association between reviewers’ publication counts and recommendation. The results for ‘None’ are the results without the potential confounder. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version. Results are missing when the model did not converge.
Appendix 6—figure 3
Leave-one-country-out sensitivity analyses for research question 1.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they were cited. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version.
Appendix 6—figure 4
Leave-one-country-out sensitivity analyses for research question 2.
Odds ratios and adjusted 99.4% confidence intervals for whether the reviewer gave a more or less favourable recommendation when they included a citation to their own articles. The results are shown for the combinations of predictor variables (linear or any vs none), outcome (Approved → Reservations → Not approved) and article version.
Appendix 8—figure 1
Histogram of online view counts of published reviews.
The bins are in tens starting at [0, 10).
Appendix 9—figure 1
Distributions of the error rates.
Vaguely informative prior and posteriors for errors for not cited reviewers, cited reviewers, and self-citations. The dashed vertical lines are at Pr(error ) = 90%.
Tables
Table 1
Descriptive statistics for the articles and peer reviews.
Q1 = first quartile, Q3 = third quartile.
| Variable | Level/statistics | Result |
|---|---|---|
| Number of reviews | n | 37,332 |
| Year | Median [Q1, Q3] | 2022 [2019, 2024] |
| Journal, n (%) | F1000Research | 24,132 (65) |
| Wellcome Open Research | 8697 (23) | |
| Open Research Europe | 2789 (7) | |
| Gates Open Research | 1714 (5) | |
| Role, n (%) | Reviewer | 34,904 (93) |
| Co-reviewer | 2428 (7) | |
| Reviewer’s recommendation, n (%) | Approved | 19,984 (54) |
| Reservations | 14,379 (38) | |
| Not approved | 2969 (8) | |
| Article version, n (%) | 1 | 26,474 (71) |
| 2 | 8995 (24) | |
| 3+ | 1863 (5) | |
| Number of papers cited in article | Median [Q1, Q3] | 24 [14, 38] |
| Any citations to reviewer, n (%) | No | 32,375 (87) |
| Yes | 4957 (13) | |
| Any papers cited by reviewer, n (%) | No | 31,546 (84) |
| Yes | 5786 (16) | |
| Any citations to the reviewer’s articles | No | 35,023 (94) |
| Yes | 2309 (6) | |
| Reviewer’s publication count | Median [Q1, Q3] | 55 [24, 118] |
| Reviewer’s country (top five only) | USA | 7655 (21%) |
| United Kingdom | 4137 (11%) | |
| India | 2472 (7%) | |
| Italy | 1368 (4%) | |
| Australia | 1349 (4%) | |
| Number of words in the review | Median [Q1, Q3] | 202 [67, 411] |
Table 2
Odds ratios for reviewers giving a more (OR >1) or less (OR <1) favourable recommendation depending on whether they were cited in the article (question 1) or included citations to their own articles (question 2).
All models were split by article version.
| Research question | Article version | Outcome | OR (adjusted 99.4% CI) |
|---|---|---|---|
| Reviewer cited by authors | Version = 1 | Approved vs Reservations/Not approved | 0.84 (0.69, 1.03) |
| Version = 1 | Approved/Reservations vs Not approved | 0.84 (0.57, 1.23) | |
| Versions = 2+ | Approved vs Reservations/Not approved | 1.61 (1.16, 2.23) | |
| Versions = 2+ | Approved/Reservations vs Not approved | 1.12 (0.59, 2.13) | |
| Reviewer cited their own articles | Version = 1 | Approved vs Reservations/Not approved | 0.57 (0.44, 0.73) |
| Version = 1 | Approved/Reservations vs Not approved | 1.11 (0.77, 1.60) | |
| Versions = 2+ | Approved vs Reservations/Not approved | 0.15 (0.08, 0.30) | |
| Versions = 2+ | Approved/Reservations vs Not approved | 0.80 (0.37, 1.74) |
Table 3
Brief information about the four included journals from the publisher F1000.
| Journal title | Year started | Field(s) of research | Articles must concern research funded by |
|---|---|---|---|
| F1000Research | 2012 | All disciplines | No restriction |
| Wellcome Open Research | 2016 | Medicine, Genomics | Wellcome |
| Gates Open Research | 2017 | Medicine | The Gates Foundation |
| Open Research Europe | 2021 | All disciplines | European Commission |
Appendix 3—table 1
Comparing the two alternatives for the citation predictor variables using either a linear variable or a binary ‘any versus none’ variable.
A vs R/N = Approved vs Reservations/Not approved, A/R vs N = Approved/Reservations vs Not approved.
| Co-reviewers included | Version | Outcome | AIC | ||
|---|---|---|---|---|---|
| Linear | Binary | Difference | |||
| No | 1 | A vs R/N | 5940.9 | 5937.4 | 3.6 |
| No | 1 | A/R vs N | 1930.3 | 1929.8 | 0.5 |
| No | 2+ | A vs R/N | 1952.4 | 1941.9 | 10.5 |
| No | 2+ | A/R vs N | 572.1 | 571.9 | 0.2 |
| Yes | 1 | A vs R/N | 5978.4 | 5975.4 | 3.0 |
| Yes | 1 | A/R vs N | 1941.1 | 1940.8 | 0.3 |
| Yes | 2+ | A vs R/N | 1963.1 | 1951.4 | 11.7 |
| Yes | 2+ | A/R vs N | 572.6 | 572.8 | –0.2 |
| No | 1 | A vs R/N | 5932.3 | 5911.1 | 21.2 |
| No | 1 | A/R vs N | 1934.1 | 1935.6 | –1.5 |
| No | 2+ | A vs R/N | 1881.4 | 1876.0 | 5.4 |
| No | 2+ | A/R vs N | 573.4 | 572.8 | 0.6 |
| Yes | 1 | A vs R/N | 5967.9 | 5944.9 | 23.0 |
| Yes | 1 | A/R vs N | 1945.4 | 1946.6 | –1.2 |
| Yes | 2+ | A vs R/N | 1917.3 | 1904.1 | 13.2 |
| Yes | 2+ | A/R vs N | 573.1 | 573.5 | –0.5 |
Appendix 7—table 1
Example sentences that reviewers used when suggesting citations to their own articles using a random sample of 20 reviews.
The first column shows the number of citations suggested. We have removed any references to names using [xxxx]. The results are ordered by text length.
| Citations suggested | Reviewer’s text |
|---|---|
| 3 | Also, the introduction, main discussion, and conclusion must be redrawn to highlight NO as a treatment option, the clinical trials discussion, the use of several NORMS (NORM-1, NORM-2, etc.), the effect of NO-carriage system, Natural NO-sources, synthetic NO-sources with limitations, Inorganic versus organic forms, etc. (e.g., in the review publications as given). |
| 1 | The term ‘true bugs’ applies to the monophyletic Heteroptera, which does include the species presented here (Acanthosoma haemorrhoidale), while aphids and mealybugs belong to the distinct lineage Sternorrhyncha, sometimes (formerly) regarded as a part of the paraphyletic Homoptera (see, for example, Figure 2 in: [xxxx]). |
| 1 | On a side note: There is already published work on population genomics of the European plaice showing that two large chromosomal rearrangements (two putative inversions) segregate in northern plaice populations (North Sea, Baltic Sea, Barents Sea, and Iceland) and distinguish different plaice populations. |
| 1 | There is some observational clinical data on how the detrusor compensates for the growing prostate and the-by consequence-increase in bladder outflow obstruction, in addition to the animal studies referred to in the commentary, to explain the pathophysiology. |
| 1 | I would like to thank the authors for including references to the work done in the [xxxx] project; I would recommend to remove the reference ([xxxx] et al., 2019b) and replace it with a reference to a much more recent and related article ([xxxx] et al.): |
| 2 | The cited literature is incomplete; it does not include all reports of studies on the presence of the snail in Colombia, and studies with relevant findings of nematodes with or without pathogenic potential in animals are omitted e.g.: |
| 1 | In the last 2 decades, our group has developed a brief instrument to assess the presence and the severity of sensory phenomena (the University of [xxxx]) to investigate OCD phenotypic subtypes and its relationship with TS/CTD. |
| 5 | The authors can find the following relevant articles to enhance their Materials and Methods section and incorporate citations to support their revised manuscript. |
| 3 | The authors should consider references from high impact journal publications on crop yield prediction. For example, the following articles by this reviewer |
| 1 | No mention to more rigorous rankings such as the Leiden Ranking are made nor to what exactly rankings are portraying. See for instance [xxxx]. |
| 1 | Maybe ‘manifest’ and ’not manifest’ would work better for example we used this terminology in [xxxx]. |
| 1 | This is especially useful if you have multiple data sets – see, for example, the [xxxx] package. |
| 3 | I would suggest the authors include some of the results of a large-scale project in Europe. |
| 2 | Please refer to some further references to revise the relevant description: |
| 1 | Here are a few additional publications you might consider referencing. |
| 1 | However, genomics resources are limited, except for parasitoid wasps. |
| 1 | Refer to this recent literature review. |
| 4 | No relevant sentence |
| 1 | No relevant sentence |
| 6 | No relevant sentence |
Appendix 9—table 1
Number of errors found in our data extraction algorithm from manual checks and the estimated 90% limit for the error rate.
| Check | Number checked | Errors found | Pr(Error rate ≤x) = 90% |
|---|---|---|---|
| Reviewer not cited | 100 | 2 | 0.051 |
| Reviewer cited | 100 | 1 | 0.037 |
| Reviewer’s citation to their own articles | 80 | 4 | 0.094 |
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Are peer reviewers influenced by their work being cited?
eLife 14:RP108748.
https://doi.org/10.7554/eLife.108748.4
