Download icon

Meta-Research: Large-scale language analysis of peer review reports

  1. Ivan Buljan  Is a corresponding author
  2. Daniel Garcia-Costa
  3. Francisco Grimaldo
  4. Flaminio Squazzoni
  5. Ana Marušić
  1. University of Split School of Medicine, Croatia
  2. Universitat de València, Spain
  3. University of Valencia, Spain
  4. University of Milan, Italy
Feature Article
  • Cited 0
  • Views 976
  • Annotations
Cite this article as: eLife 2020;9:e53249 doi: 10.7554/eLife.53249

Abstract

Peer review is often criticized for being flawed, subjective and biased, but research into peer review has been hindered by a lack of access to peer review reports. Here we report the results of a study in which text-analysis software was used to determine the linguistic characteristics of 472,449 peer review reports. A range of characteristics (including analytical tone, authenticity, clout, three measures of sentiment, and morality) were studied as a function of reviewer recommendation, area of research, type of peer review and reviewer gender. We found that reviewer recommendation had the biggest impact on the linguistic characteristics of reports, and that area of research, type of peer review and reviewer gender had little or no impact. The lack of influence of research area, type of review or reviewer gender on the linguistic characteristics is a sign of the robustness of peer review.

Article and author information

Author details

  1. Ivan Buljan

    Department for Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia
    For correspondence
    ibuljan@mefst.hr
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-8719-7277
  2. Daniel Garcia-Costa

    Departament d'Informàtica, Universitat de València, Valencia, Spain
    Competing interests
    The authors declare that no competing interests exist.
  3. Francisco Grimaldo

    Department d'Informàtica, University of Valencia, Valencia, Spain
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-1357-7170
  4. Flaminio Squazzoni

    Departament d'Informàtica, University of Milan, Milan, Italy
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-6503-6077
  5. Ana Marušić

    Department for Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-6272-0917

Funding

Spanish Ministry of Science, Innovation and Universities, the Spanish State Research Agency and the European Regional Development Fund (RTI2018-095820-B-I00)

  • Francisco Grimaldo

Croatian Science Foundation (IP-2019-04-4882)

  • Ana Marušić

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Reviewing Editor

  1. Peter Rodgers, eLife, United Kingdom

Publication history

  1. Received: November 1, 2019
  2. Accepted: July 16, 2020
  3. Accepted Manuscript published: July 17, 2020 (version 1)
  4. Version of Record published: July 29, 2020 (version 2)

Copyright

© 2020, Buljan et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 976
    Page views
  • 117
    Downloads
  • 0
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)