Meta-Research: Large-scale language analysis of peer review reports
Figures

Words counts in peer review reports.
Word count (mean and 95% confidence interval; LIWC analysis) of peer review reports in four broad areas of research for double-blind review (top) and single-blind review (bottom), and for female reviewers (left) and male reviewers (right). Reports recommending accept (red) were consistently the shortest, and reports recommending major revisions (green) were consistently the longest. See Supplementary file 1 for summary data and mixed model linear regression coefficients and residuals. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.

Analytical tone, clout and authenticity and in peer review reports for single-blind review.
Scores returned by LIWC (mean percentages and 95% confidence interval) for analytical tone (A), clout (B) and authenticity (C) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using single-blind review. Reports recommending accept (red) consistently had the most clout, and reports recommending reject (purple) consistently had the least clout. See Supplementary files 2–4 for summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for analytical tone, clout and authenticity. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.

Analytical tone, clout and authenticity in peer review reports for double-blind review.
Scores returned by LIWC (mean percentages and 95% confidence interval) for analytical tone (A), clout (B) and authenticity (C) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using double-blind review.

Sentiment analysis of peer review reports for single-blind review.
Scores for sentiment analysis returned by LIWC (A; mean percentage and 95% confidence interval, CI), SentimentR (B; mean score and 95% CI), and Stanford CoreNLP (C; mean score and 95% CI) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using single-blind review. See Supplementary files 5–7 for summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment according to LIWC, SentimentR and Stanford CoreNLP analysis.

Sentiment analysis of peer review reports for double-blind review.
Scores for sentiment analysis returned by LIWC (A; mean percentage and 95% confidence interval, CI), SentimentR (B; mean score and 95% CI), and Stanford CoreNLP (C; mean score and 95% CI) for peer review reports in four broad areas of research for female reviewers (left) and male reviewers (right) using double-blind review.

Moral foundations in peer review reports.
Scores returned by LIWC (mean percentage on a log scale) for general morality in peer review reports in four broad areas of research for double-blind review (top) and single-blind review (bottom), and for female reviewers (left) and male reviewers (right). Reports recommending accept (red) consistently had the highest scores. See Supplementary file 8 for lists of the ten most frequent words found in peer review reports for general morality and the five moral foundation variables. HMS: health and medical sciences; LS: life sciences; PS: physical sciences; SS&E: social sciences and economics.

Scores returned by LIWC (mean percentage on a log scale and 95% CI) for care/harm, one of the five foundations of Moral Foundations Theory.

Scores returned by LIWC (mean percentage on a log scale and 95% CI) for fairness/cheating, one of the five foundations of Moral Foundations Theory.

Scores returned by LIWC (mean percentage on a log scale and 95% CI) for loyalty/betrayal, one of the five foundations of Moral Foundations Theory.

Scores returned by LIWC (mean percentage on a log scale and 95% CI) for authority/subversion, one of the five foundations of Moral Foundations Theory.
Additional files
-
Supplementary file 1
Word count (Figure 1): summary data and mixed model linear regression coefficients and residuals.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp1-v2.docx
-
Supplementary file 2
Analytical tone (Figure 2A): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC analytical tone.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp2-v2.docx
-
Supplementary file 3
Clout (Figure 2B): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC clout.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp3-v2.docx
-
Supplementary file 4
Authenticity (Figure 2C): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for LIWC authenticity.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp4-v2.docx
-
Supplementary file 5
Sentiment/LIWC emotional tone (Figure 3A): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (LIWC emotional tone).
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp5-v2.docx
-
Supplementary file 6
Sentiment/SentimentR score (Figure 3B): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (SentimentR scores).
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp6-v2.docx
-
Supplementary file 7
Sentiment/Stanford CoreNLP score (Figure 3C): summary data, mixed model linear regression coefficients and residuals, and examples of reports with high and low scores for sentiment (Stanford CoreNLP score).
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp7-v2.docx
-
Supplementary file 8
Ten most frequent words found in peer review reports for general morality and the five moral foundation variables.
- https://cdn.elifesciences.org/articles/53249/elife-53249-supp8-v2.docx
-
Transparent reporting form
- https://cdn.elifesciences.org/articles/53249/elife-53249-transrepform-v2.docx