Download icon

Meta-Research: Large-scale language analysis of peer review reports

  1. Ivan Buljan  Is a corresponding author
  2. Daniel Garcia-Costa
  3. Francisco Grimaldo
  4. Flaminio Squazzoni
  5. Ana Marušić
  1. Department of Research in Biomedicine and Health, University of Split School of Medicine, Croatia
  2. Department d'Informàtica, University of Valencia, Spain
  3. Department of Social and Political Sciences, University of Milan, Italy
Feature Article
Cite this article as: eLife 2020;9:e53249 doi: 10.7554/eLife.53249
4 figures and 9 additional files

Figures

Words counts in peer review reports.

Word count (mean and 95% confidence interval; LIWC analysis) of peer review reports in four broad areas of research for double-blind review (top) and single-blind review (bottom), and for female reviewers (left) and male reviewers (right). Reports recommending accept (red) were consistently the shortest, and reports recommending major revisions (green) were consistently the longest. See Supplementary file 1 for summary data and mixed model linear regression coefficients and residuals. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.

Figure 2 with 1 supplement
Analytical tone, clout and authenticity and in peer review reports for single-blind review.

Scores returned by LIWC (mean percentages and 95% confidence interval) for analytical tone (A), clout (B) and authenticity (C) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using single-blind review. Reports recommending accept (red) consistently had the most clout, and reports recommending reject (purple) consistently had the least clout. See Supplementary files 24 for summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for analytical tone, clout and authenticity. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.

Figure 2—figure supplement 1
Analytical tone, clout and authenticity in peer review reports for double-blind review.

Scores returned by LIWC (mean percentages and 95% confidence interval) for analytical tone (A), clout (B) and authenticity (C) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using double-blind review.

Figure 3 with 1 supplement
Sentiment analysis of peer review reports for single-blind review.

Scores for sentiment analysis returned by LIWC (A; mean percentage and 95% confidence interval, CI), SentimentR (B; mean score and 95% CI), and Stanford CoreNLP (C; mean score and 95% CI) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using single-blind review. See Supplementary files 57 for summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment according to LIWC, SentimentR and Stanford CoreNLP analysis.

Figure 3—figure supplement 1
Sentiment analysis of peer review reports for double-blind review.

Scores for sentiment analysis returned by LIWC (A; mean percentage and 95% confidence interval, CI), SentimentR (B; mean score and 95% CI), and Stanford CoreNLP (C; mean score and 95% CI) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using double-blind review.

Figure 4 with 5 supplements
Moral foundations in peer review reports.

Scores returned by LIWC (mean percentage on a log scale) for general morality in peer review reports in four broad areas of research for double-blind review (top) and single-blind review (bottom), and for female reviewers (left) and male reviewers (right). Reports recommending accept (red) consistently had the highest scores. See Supplementary file 8 for lists of the ten most frequent words found in peer review reports for general morality and the five moral foundation variables. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.

Figure 4—figure supplement 1
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for care/harm, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 2
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for fairness/cheating, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 3
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for loyalty/betrayal, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 4
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for authority/subversion, one of the five foundations of Moral Foundations Theory.
Figure 4—figure supplement 5
Scores returned by LIWC (mean percentage on a log scale and 95% CI) for sanctity/degradation, one of the five foundations of Moral Foundations Theory.

Data availability

The journal dataset required a data sharing agreement to be established between authors and publishers. A protocol on data sharing entitled 'TD1306 COST Action New frontiers of peer review (PEERE) PEERE policy on data sharing on peer review' was signed by all partners involved in this research on 1 March 2017, as part of a collaborative project funded by the EU Commission. The protocol established rules and practices for data sharing from a sample of scholarly journals, which included a specific data management policy, including data minimization, retention and storage, privacy impact assessment, anonymization, and dissemination. The protocol required that data access and use were restricted to the authors of this manuscript and data aggregation and report were done in such a way to avoid any identification of publishers, journals or individual records involved. The protocol was written to protect the interests of any stakeholder involved, including publishers, journal editors and academic scholars, who could be potentially acted by data sharing, use and release. The full version of the protocol is available on the peere.org website. To request additional information on the dataset and for any claim or objection, please contact the PEERE data controller at info@peere.org.

Additional files

Supplementary file 1

Word count (Figure 1): summary data and mixed model linear regression coefficients and residuals.

https://cdn.elifesciences.org/articles/53249/elife-53249-supp1-v2.docx
Supplementary file 2

Analytical tone (Figure 2A): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC analytical tone.

https://cdn.elifesciences.org/articles/53249/elife-53249-supp2-v2.docx
Supplementary file 3

Clout (Figure 2B): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC clout.

https://cdn.elifesciences.org/articles/53249/elife-53249-supp3-v2.docx
Supplementary file 4

Authenticity (Figure 2C): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC authenticity.

https://cdn.elifesciences.org/articles/53249/elife-53249-supp4-v2.docx
Supplementary file 5

Sentiment/LIWC emotional tone (Figure 3A): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (LIWC emotional tone).

https://cdn.elifesciences.org/articles/53249/elife-53249-supp5-v2.docx
Supplementary file 6

Sentiment/SentimentR score (Figure 3B): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (SentimentR scores).

https://cdn.elifesciences.org/articles/53249/elife-53249-supp6-v2.docx
Supplementary file 7

Sentiment/Stanford CoreNLP score (Figure 3C): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (Stanford CoreNLP score).

https://cdn.elifesciences.org/articles/53249/elife-53249-supp7-v2.docx
Supplementary file 8

Ten most frequent words found in peer review reports for general morality and the five moral foundation variables.

https://cdn.elifesciences.org/articles/53249/elife-53249-supp8-v2.docx
Transparent reporting form
https://cdn.elifesciences.org/articles/53249/elife-53249-transrepform-v2.docx

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)