The growth in scientific publishing (Larsen and von Ins, 2010) makes it increasingly difficult for researchers to keep up-to-date with the newest trends in their fields (Medoff, 2006; Collins, 1998). Already in the 1960s, sociologists of science suggested that researchers, in the midst of this information overload, would search for cues as to what literature to read (Merton, 1968; Crane, 1965). One important cue was the author’s location. Academic affiliations would cast a “halo-effect” (Crane, 1967) on scholarly work that would amplify the recognition of researchers based at prestigious institutions at the expense of authors from institutions and nations of lower status. This halo effect ties closely to the idea of the “Matthew effect” in science, i.e. “the accumulation of differential advantages for certain segments of the population [of researchers] that are not necessarily bound up with demonstrated differences in capacity” (Merton, 1988).

From a social psychological perspective, country- and institution-related halo effects may arise from stereotypic beliefs about the relative performance capacity of scientists working at more or less prestigious institutions (Jost et al., 2009; Greenwald and Banaji, 1995; Ridgeway, 2001). According to status characteristics theory, a scientist’s nationality or institutional affiliation can be considered “status signals” that, when salient, implicitly influence peer-evaluations (Berger et al., 1977; Correll and Bernard, 2015). Moreover, system justification theory asserts that members of a social hierarchy, such as the scientific community, regardless of their position in this hierarchy, will feel motivated to see existing social arrangements as fair and justifiable to preserve a sense of predictability and certainty around their own position (Jost et al., 2004; Jost et al., 2002; Magee and Galinsky, 2008; Son Hing et al., 2011). As a result, both higher and lower status groups may internalize implicit behaviours and attitudes that favour higher-status groups (Jost, 2013).

Several observational studies have examined the influence of country- and institution-related halo effects in peer-reviewing (Crane, 1967; Link, 1998; Murray et al., 2018). Most of them indicate slight advantages in favor of researchers from high status countries (such as the US or UK) and universities (such as Harvard University or Oxford University). However, a key limitation of this literature concerns the unobserved heterogeneity attributable to differences in quality. Non-experimental designs do not allow us to determine the advantages enjoyed by scientists at high-status locations independent of their capabilities as scientists, or the content and character of their work.

Here we examine the impact of academic affiliations on scientific judgment, independent of manuscript quality. Specifically, we consider how information about the geographical location and institutional affiliation of authors influence how scientific abstracts are evaluated by their peers. In a preregistered survey experiment, we asked 4147 scientists from six disciplines (astronomy, cardiology, materials science, political science, psychology and public health) to rate abstracts that vary on two factors: (i) author country (high status vs. lower status scientific nation); (ii) institutional affiliation (high status vs. lower status university; see Table 1). All other content of the discipline-specific abstracts was held constant.

A few pioneering studies have already attempted to discern the influence of national and institutional location on scholarly peer-assessments, independent of manuscript quality. One study (Blank, 1991) used a randomized field experiment to examine the effects of double blind vs. single blind peer reviewing on acceptance rates in the American Economic Review. The study found no evidence that a switch from single blind to double blind peer reviewing influenced the relative ratings of papers from high-ranked and lower-ranked universities. Another study (Ross et al., 2006) examined the effect of single blind vs. double blind peer reviewing on the assessment of 13,000 abstracts submitted to the American Heart Association’s annual Scientific Sessions between 2000 and 2004. The study found that when abstracts underwent single blind compared to double blind reviewing the relative increase in acceptance rates was higher for US authored abstracts compared to non-US authored abstracts, and for abstracts from highly prestigious US institutions compared to abstracts from non-prestigious US institutions.

A recent survey experiment also found that professors at schools of public health in the US (N: 899) rated one abstract higher on likelihood of referral to a peer, when the authors’ affiliation was changed from a low-income to a high-income country (Harris et al., 2015). However, each participant was asked to rate four abstracts and the results for the remaining three abstracts were inconclusive. Likewise, the study found no evidence of country-related bias in the ratings of the strength of the evidence presented in the abstracts. In another randomized, blinded cross-over study (N: 347), the same authors found that changing the source of an abstract from a low-income to a high-income country slightly improved English clinicians’ ratings of relevance and recommendation to a peer (Harris et al., 2017). Finally, a controlled field experiment recently examined the “within subject” effect of peer-review model (single blind vs. double blind) on the acceptance rates of full-length submissions for a prestigious computer-science conference (Tomkins et al., 2017). The study allocated 974 double blind and 983 single blind reviewers to 500 papers. Two single blind and two double blind reviewers assessed each paper. The study found that single blind reviewers were more likely than double blind reviewers to accept papers from top-ranked universities compared to papers from lower-ranked universities.

Our study contributes to this pioneering work by targeting a broader range of disciplines in the social sciences, health sciences and natural sciences. This allows us to examine possible between-discipline variation in the prevalence of country- or institution-related rater bias.

Our six discipline-specific experiments show weak and inconsistent evidence of country- or institution-related status bias in abstract ratings, and mixed regression models indicate that any plausible effect must be small in size.

In accordance with our preregistered plan (see https://osf.io/4gjwa), the analysis took place in two steps. First, we used ANOVAs and logit models to conduct discipline-specific analyses of how country- and institution-related status signals influence scholarly judgments made by peer-evaluators. Second, we employed mixed regression models with disciplines as random effect parameter to estimate the direct effect and moderation effects of the presumed association between status signals and evaluative outcomes at an aggregate, cross-disciplinary level.

We used three measures to gauge the assessments of abstracts by peer-evaluators. Abstract score is our main outcome variable. It consists of four items recorded on a five-point scale (1=“very poor”, 5 = “excellent”) that ask the peer-evaluators to assess (i) the originality of the presented research, (ii) the credibility of the results, (iii) the significance for future research, and (iv) the clarity and comprehensiveness of the abstract. We computed a composite measure that specifies each participant’s total-item score for these four items (Cronbach’s α = 0.778). Open full-text is a dichotomous outcome measure that asks whether the peer-evaluator would choose to open the full text and continue reading, if s/he came across the abstract online. Include in conference is a dichotomous outcome measure that asks whether the peer-evaluator would choose to include the abstract for an oral presentation, if s/he was reviewing it as a committee member of a selective international scientific conference (the full questionnaire is included in the Supplementary file 1).

As specified in the registered plan, we used G*Power to calculate the required sample size per discipline (N = 429) for detecting a Cohen’s f = 0.15 (corresponding to a Cohen’s d = 0.30) or larger with α = 0.05 and a power of 0.80 in the cross-group comparisons with abstract score as outcome. Table 1 presents the final sample distributions for the three-way factorial design across the six disciplines.

To test for statistical equivalence in the discipline-specific comparisons of our main outcome (abstract score) across manipulations, we performed the two-one-sided tests procedure [TOST] for differences between independent means (Lakens, 2017a). We used the pre-registered ‘minimum detectable effect size’ of Cohen’s D=±0.30 for the discipline-specific equivalence tests.

The boxplots in Figure 1 display the abstracts ratings for respondents assigned to each manipulation across the six disciplines. The discipline-specific ANOVAs with the manipulations as the between-subject factor did not indicate any country- or institution-related status bias in abstract ratings (astronomy: F = 0.71, p=0.491, N = 592; cardiology: F = 1.50, p=0.225, N = 609; materials science: F = 0.73, p=0.482, N = 546; political science: F = 0.53, p=0.587, N = 1008; psychology: F = 0.19, p=0.827, N = 624; public health: F = 0.34, p=0.715, N = 732). The η² coefficients were 0.002 in astronomy, 0.005 in cardiology, 0.003 in materials science, and 0.001 in political science, psychology and public health.

Figure 1 with 5 supplements see all

Download asset Open asset

A TOST procedure with an α level of 0.05 indicated that the observed effect sizes were within the equivalence bound of d = −0.3 and d = 0.3 for 11 of the 12 between-subject comparisons at the disciplinary level (see Supplementary file 2, Tables S1–S12). In raw scores, this equivalence bound corresponds to a span from −0.8 to 0.8 abstract rating points on a scale from 4 to 20. In cardiology, the TOST test failed to reject the null-hypothesis of non-equivalence in the evaluative ratings of subjects exposed to abstracts from higher-status US universities and lower-status US universities.

A closer inspection suggests that these findings are robust across the four individual items that make up our composite abstract-score. None of these items show notable variations in abstract ratings across manipulations (Figure 1—figure supplement 1–4).

Figures 2 and 3 report the outcomes of the discipline-specific logit models with Open full-text and Include in conference as outcomes. The uncertainty of the estimates is notably larger in these models than for the one-way ANOVAs reported above, indicating wider variability in the peer-evaluators’ dichotomous assessments.

Figure 2

Download asset Open asset

Figure 3

Download asset Open asset

As displayed in Figure 2, peer-evaluators in Public Health were between 7.5% and 56.5% less likely to show interest in opening the full-text, when the author affiliation was changed from a lower-status university in the US to a lower-status university elsewhere (Odds ratio:. 634, CI: 0.435–0.925). The odds ratios for the remaining 11 comparisons across the six disciplines all had 95% confidence intervals spanning one. Moreover, in five of the 12 between-subject comparisons, the direction of the observed effects was inconsistent with the a-priori expectation that abstracts from higher-status US universities would be favoured over abstracts from lower-status US universities, and that abstracts from lower-status US universities would be favoured over abstracts from lower-status universities elsewhere.

As displayed in Figure 3, peer-evaluators in political science were between 7.0% and 59.4% less likely to consider an abstract relevant for a selective, international conference program, when the abstract’s author was affiliated with a higher-status US university compared to a lower-status US university (Odds ratio:. 613, CI:. 406-.930). This result goes against a-priori expectations concerning the influence of status signals on evaluative outcomes. The remaining five disciplines had 95% confidence intervals spanning one, and in six of the 12 comparisons, the direction of the observed effects was inconsistent with a-priori expectations. As indicated in panel b, we observe notable variation in the participants’ average responses to the Include in conference item across disciplines.

Figure 4 plots the fixed coefficients (panel a) and adjusted means (panel b) from the mixed linear regression with Abstract rating as outcome. In accordance with the pre-registered analysis plan, we report the mixed regression models with 99% confidence intervals. The fixed coefficients for the two between-group comparisons tested in this model are close to zero, and the 99% confidence intervals suggest that any plausible difference would fall within the bounds of −0.26 to 0.28 points on the 16-point abstract-rating scale. These confidence bounds can be converted into standardized effects of Cohen’s d = −0.10 to 0.11.

Figure 4

Download asset Open asset

Figure 5 displays odds ratios and 99% confidence intervals for the mixed logit regressions with Open full-text (panel a, upper display) and Include in conference (panel a, lower display) as outcomes. The odds ratios for the experimental manipulations used as predictors in these models range from 0.86 to 1.05 and have 99% confidence intervals spanning the line of no difference. The 99% confidence intervals (panel a) indicate that any plausible effect would fall within the bounds of odds ratio = 0.68 and 1.35, which corresponds to a standardized confidence bound of Cohen’s d = −0.21 to 0.17. The wide confidence bounds for the estimated proportion for Include in conference (panel b) reflect the large variations in average assessments of abstracts across disciplines.

Figure 5

Download asset Open asset

Robustness checks based on mixed linear and logit models were carried out to examine the effects of the experimental manipulations on the three outcome measures, while restricting the samples to (i) participants that responded correctly to a manipulation-check item, and (ii) participants that saw their own research as being ‘extremely close’, ‘very close’ or ‘somewhat close’ to the subject addressed in the abstract. All of these models yielded qualitatively similar results, with small residual effects and confidence intervals spanning 0 in the linear regressions and one in the logistic regressions (see Supplementary file 2, Tables S28–S33). Moreover, a pre-registered interaction analysis was conducted to examine whether the influence of country- and institution-related status signals was moderated by any of the following characteristics of the peer evaluators: (i) their descriptive beliefs in the objectivity and fairness of peer-evaluation; (ii) their structural location in the science system (in terms of institutional affiliation and scientific rank); (iii) their research accomplishments; (iv) their self-perceived scientific status. All of these two-way interactions had 99% CI intervals spanning 0 in the linear regressions and one in the logistic regressions, indicating no discernible two-way interactions (see Supplementary file 2, Tables S34–S39).

Contrary to the idea of halo effects, our study shows weak and inconsistent evidence of country- or institution-related status bias in abstract ratings. In the discipline-specific analyses, we observed statistically significant differences in two of 36 pairwise comparisons (i.e. 5.6%) and, of these, one was inconsistent with our a-priori expectation of a bias in favour of high-status sources. Moreover, the estimated confidence bounds derived from the multilevel regressions were either very small or small according to Cohen’s classification of effect sizes (Cohen, 2013). These results align with the outcomes of three out of five existing experiments, which also show small and inconsistent effects of country- and institution-related status bias in peer-evaluation (Blank, 1991; Harris et al., 2015; Harris et al., 2017). However, it should be noted that even small effects may produce large population-level variations if they accumulate over the course of scientific careers. Moreover, one could argue that small effects would be the expected outcome for a lightweight manipulation like ours in an online survey context, where the participants are asked to make decisions without real-world implications. Indeed, it is possible that the effects of country- and institution level status signals would be larger in real-world evaluation processes, where reviewers have more at stake.

Certain scope conditions also delimit the situations to which our conclusions may apply. First, our findings leave open the possibility that peer-evaluators discard research from less reputable institutions and science nations without even reading their abstracts. In other words, our study solely tests whether scientists, after being asked to carefully read an assigned abstract, on average will rate contributions from prestigious locations more favourably.

Second, even when peer-evaluators identify and read papers from “lower-status” sources, they may still omit to cite them, and instead frame their contributions in the context of more high-status sources. In the future, researchers might add to this work by examining if these more subtle processes contribute to shape the reception and uptake of scientific knowledge. Third, our conclusion of little or no bias in abstract review does not necessarily imply that biases are absent in other types of peer assessment, such as peer reviewing of journal and grant submissions, where evaluations usually follow formalized criteria and focus on full-text manuscripts and proposals. Likewise, our study does not capture how country- and institution-related status signals may influence the decisions of journal editors. Indeed, journal editors play an important role in establishing scientific merits and reputations and future experimental work should examine how halo effects may shape editorial decisions. Fourth, although we cover a broad span of research areas, it is possible that our experiment would have produced different results for other disciplines. Fifth, it should be noted that our two dichotomous outcome items (open full text and include in conference) refer to two rather different evaluative situations that may be difficult to compare. For instance, a researcher may wish to read a paper (based on its abstract) while finding it inappropriate for a conference presentation and vice versa. Moreover, the competitive aspect of selecting an abstract for a conference makes this evaluative situation quite different from the decision to open a full text.

A key limitation is that our experiment was conducted in non-probability samples with low response rates, which raises questions about selection effects. One could speculate that the scientists that are most susceptible to status bias would be less likely to participate in a study conducted by researchers from two public universities in Denmark. Moreover, we decided to include internationally outstanding universities in our high-status category (California Institute of Technology, Columbia University, Harvard University, Yale University, Johns Hopkins University, Massachusetts Institute of Technology, Princeton University, Stanford University, and University of California, Berkeley). By so doing, we aimed to ensure a relatively strong status signal in the abstract’s byline. There is a risk that this focus on outstanding institutions may have led some survey participants to discern the purpose of the experiment and censor their responses to appear unbiased. However, all participants were blinded to the study objectives (the invitation email is included in Supplementary file 1), and to reduce the salience of the manipulation, we asked each participant to review only one abstract. Moreover, in the minds of the reviewers, many other factors than the scholarly affiliations might have been manipulated in the abstracts, including the gender of the authors (Knobloch-Westerwick et al., 2013; Forscher et al., 2019), the style of writing, the source of the empirical data presented in the abstract, the clarity of the statistical reporting, the reporting of positive vs. negative and mixed results (Mahoney, 1977), the reporting of intuitive vs. counter-intuitive findings (Mahoney, 1977; Hergovich et al., 2010), and the reporting of journal information and visibility metrics (e.g. citation indicators or Altmetric scores) (Teplitskiy et al., 2020). A final limitation concerns the varying assessments of abstract quality across disciplines. This cross-disciplinary variation was particularly salient for the Include in conference item (Figure 3, panel b), which may indicate differences in the relative difficulty of getting an oral presentation at conferences in the six disciplines. In the mixed-regression models based on data from all six disciplines, we attempt to account for this variation by including discipline as random-effect parameter. In summary, this paper presents a large-scale, cross-disciplinary examination of how country- and institution-related status signals influence the reception of scientific research. In a controlled experiment that spanned six disciplines, we found no systematic evidence of status bias against abstracts from lower status universities and countries. Future research should add to this work by examining if other processes related to the social, material and intellectual organization of science contribute to producing and reproducing country- and institution-related status hierarchies.

Our study complies with all ethical regulations. Aarhus University’s Institutional Review Board approved the study (case no. 2019-616-000014). We obtained informed consent from all participants. The sampling and analysis plan was preregistered at the Open Science Framework on September 18, 2019. We have followed all of the steps presented in the registered plan, with two minor deviations. First, we did not preregister the equivalence tests reported for the discipline-specific analyses presented in Figure 1. Second, in the results section, we report the outcomes of the mixed regression models with abstract score as outcome based on linear models instead of the tobit models included in the registered plan. Tables S40–S44 in Supplementary file 2 report the outcomes of the mixed-tobit models, which are nearly identical to the results from the linear models reported in Tables S25, S28, S31, S36, S37 in Supplementary file 2.

All data and code needed to evaluate the conclusions are available here: https://osf.io/x4rj8/.

1. 1. Nielsen MW

   (2020) Open Science Framework

   ID x4rj8. Weak evidence of institutional and country-related status bias.

   https://osf.io/x4rj8/

1. Website

   1. Adler N
   2. Stewart J

   (2007) The MacArthur scale of subjective social status

   Accessed March 18, 2021.

   https://macses.ucsf.edu/research/psychosocial/subjective.php
2. 1. Anderson MS
   2. Ronning EA
   3. Devries R
   4. Martinson BC

   (2010) Extending the Mertonian norms: scientists' subscription to norms of research

   The Journal of Higher Education 81:366–393.

   https://doi.org/10.1080/00221546.2010.11779057

   • PubMed
   • Google Scholar
3. Book

   1. Berger J
   2. Fisek H
   3. Norman R
   4. Zelditch M

   (1977)

   Status Characteristics and Social Interaction

   New York: Elsevier.

   • Google Scholar
4. 1. Blank RM

   (1991)

   The effects of double-blind versus single-blind reviewing: experimental evidence from the american economic review

   The American Economic Review 81:1041–1067.

   • Google Scholar
5. Book

   1. Cohen J

   (2013) Statistical Power Analysis for the Behavioral Sciences

   Cambridge, MA: Academic Press.

   https://doi.org/10.4324/9780203771587

   • Google Scholar
6. Book

   1. Collins R

   (1998)

   The Sociology of Philosophies

   Cambridge, MA: Harvard University Press.

   • Google Scholar
7. Book

   1. Correll S
   2. Bernard S

   (2015) Biased estimators? Comparing status and statistical theories of gender discrimination

   In: Thye S. R, Lawler E. J, editors. Advances in Group Processes, 23. Bingley: Emerald Group Publishing Limited. pp. 89–116.

   https://doi.org/10.1016/S0882-6145(06)23004-2

   • Google Scholar
8. 1. Crane D

   (1965) Scientists at major and minor universities: a study of productivity and recognition

   American Sociological Review 30:699–714.

   https://doi.org/10.2307/2091138

   • PubMed
   • Google Scholar
9. 1. Crane D

   (1967)

   The gatekeepers of science: some factors affecting the selection of articles for scientific journals

   The American Sociologist 2:195–201.

   • Google Scholar
10. 1. Forscher PS
    2. Cox WTL
    3. Brauer M
    4. Devine PG

    (2019) Little race or gender bias in an experiment of initial review of NIH R01 grant proposals

    Nature Human Behaviour 3:257–264.

    https://doi.org/10.1038/s41562-018-0517-y

    • PubMed
    • Google Scholar
11. 1. Greenwald AG
    2. Banaji MR

    (1995) Implicit social cognition: attitudes, self-esteem, and stereotypes

    Psychological Review 102:4–27.

    https://doi.org/10.1037/0033-295X.102.1.4

    • PubMed
    • Google Scholar
12. 1. Harris M
    2. Macinko J
    3. Jimenez G
    4. Mahfoud M
    5. Anderson C

    (2015) Does a research article's country of origin affect perception of its quality and relevance? A national trial of US public health researchers

    BMJ Open 5:e008993.

    https://doi.org/10.1136/bmjopen-2015-008993

    • PubMed
    • Google Scholar
13. 1. Harris M
    2. Marti J
    3. Watt H
    4. Bhatti Y
    5. Macinko J
    6. Darzi AW

    (2017) Explicit bias toward high-income-country research: a randomized, blinded, crossover experiment of English clinicians

    Health Affairs 36:1997–2004.

    https://doi.org/10.1377/hlthaff.2017.0773

    • Google Scholar
14. 1. Hergovich A
    2. Schott R
    3. Burger C

    (2010) Biased evaluation of abstracts depending on topic and conclusion: further evidence of a confirmation bias within scientific psychology

    Current Psychology 29:188–209.

    https://doi.org/10.1007/s12144-010-9087-5

    • Google Scholar
15. 1. Jost JT
    2. Pelham BW
    3. Carvallo MR

    (2002) Non-conscious forms of system justification: implicit and behavioral preferences for higher status groups

    Journal of Experimental Social Psychology 38:586–602.

    https://doi.org/10.1016/S0022-1031(02)00505-X

    • Google Scholar
16. 1. Jost JT
    2. Banaji MR
    3. Nosek BA

    (2004) A decade of system justification theory: accumulated evidence of conscious and unconscious bolstering of the status quo

    Political Psychology 25:881–919.

    https://doi.org/10.1111/j.1467-9221.2004.00402.x

    • Google Scholar
17. 1. Jost JT
    2. Rudman LA
    3. Blair IV
    4. Carney DR
    5. Dasgupta N
    6. Glaser J
    7. Hardin CD

    (2009) The existence of implicit bias is beyond reasonable doubt: a refutation of ideological and methodological objections and executive summary of ten studies that no manager should ignore

    Research in Organizational Behavior 29:39–69.

    https://doi.org/10.1016/j.riob.2009.10.001

    • Google Scholar
18. Book

    1. Jost JT

    (2013)

    Cognitive social psychology

    In: Moskowitz G. B, editors. Social Cognition. Psychology Press. pp. 92–105.

    • Google Scholar
19. 1. Knobloch-Westerwick S
    2. Glynn CJ
    3. Huge M

    (2013) The Matilda effect in science communication: an experiment on gender bias in publication quality perceptions and collaboration interest

    Science Communication 35:603–625.

    https://doi.org/10.1177/1075547012472684

    • Google Scholar
20. 1. Lakens D

    (2017a) Equivalence tests: a practical primer for t tests, correlations, and meta-analyses

    Social Psychological and Personality Science 8:355–362.

    https://doi.org/10.1177/1948550617697177

    • PubMed
    • Google Scholar
21. Software

    1. Lakens D

    (2017b) TOSTER: Two One-Sided Tests (TOST) Equivalence Testing, version R package version 0. 2-648

    TOSTER: Two One-Sided Tests (TOST) Equivalence Testing.

    https://rdrr.io/cran/TOSTER/
22. 1. Larsen PO
    2. von Ins M

    (2010) The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index

    Scientometrics 84:575–603.

    https://doi.org/10.1007/s11192-010-0202-z

    • PubMed
    • Google Scholar
23. 1. Link AM

    (1998) US and non-US submissions: an analysis of reviewer bias

    JAMA 280:246–247.

    https://doi.org/10.1001/jama.280.3.246

    • PubMed
    • Google Scholar
24. Software

    1. Lüdecke D

    (2018) Esc: Effect Size Computation for Meta-Analysis, version R package version 0, 4(1)42

    Esc: Effect Size Computation for Meta-Analysis.

    https://rdrr.io/cran/esc/
25. 1. Magee JC
    2. Galinsky AD

    (2008) Social hierarchy: The self‐reinforcing nature of power and status

    Academy of Management Annals 2:351–398.

    https://doi.org/10.5465/19416520802211628

    • Google Scholar
26. 1. Mahoney MJ

    (1977) Publication prejudices: an experimental study of confirmatory bias in the peer review system

    Cognitive Therapy and Research 1:161–175.

    https://doi.org/10.1007/BF01173636

    • Google Scholar
27. 1. Medoff MH

    (2006) Evidence of a Harvard and Chicago Matthew Effect

    Journal of Economic Methodology 13:485–506.

    https://doi.org/10.1080/13501780601049079

    • Google Scholar
28. 1. Merton RK

    (1968) The Matthew effect in science

    Science 159:56–63.

    https://doi.org/10.1126/science.159.3810.56

    • Google Scholar
29. 1. Merton RK

    (1988) The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property

    Isis 79:606–623.

    https://doi.org/10.1086/354848

    • Google Scholar
30. Preprint

    1. Murray D
    2. Siler K
    3. Lariviére V
    4. Chan WM
    5. Collings AM
    6. Raymond J
    7. Sugimoto CR

    (2018) Gender and international diversity improves equity in peer review

    bioRxiv.

    https://doi.org/10.1101/400515

    • Google Scholar
31. 1. Myers KR
    2. Tham WY
    3. Yin Y
    4. Cohodes N
    5. Thursby JG
    6. Thursby MC
    7. Schiffer P
    8. Walsh JT
    9. Lakhani KR
    10. Wang D

    (2020) Unequal effects of the COVID-19 pandemic on scientists

    Nature Human Behaviour 4:880–883.

    https://doi.org/10.1038/s41562-020-0921-y

    • PubMed
    • Google Scholar
32. Book

    1. Ridgeway CL

    (2001) Blackwell Handbook of Social Psychology: Group Processes

    Blackwell.

    https://doi.org/10.1002/9780470998458

    • Google Scholar
33. 1. Ross JS
    2. Gross CP
    3. Desai MM
    4. Hong Y
    5. Grant AO
    6. Daniels SR
    7. Hachinski VC
    8. Gibbons RJ
    9. Gardner TJ
    10. Krumholz HM

    (2006) Effect of blinded peer review on abstract acceptance

    JAMA 295:1675–1680.

    https://doi.org/10.1001/jama.295.14.1675

    • PubMed
    • Google Scholar
34. 1. Son Hing LS
    2. Bobocel DR
    3. Zanna MP
    4. Garcia DM
    5. Gee SS
    6. Orazietti K

    (2011) The merit of meritocracy

    Journal of Personality and Social Psychology 101:433–450.

    https://doi.org/10.1037/a0024618

    • PubMed
    • Google Scholar
35. Preprint

    1. Teplitskiy M
    2. Duede E
    3. Menietti M
    4. Lakhani KR

    (2020) Citations systematically misrepresent the quality and impact of research articles: survey and experimental evidence from thousands of citers

    arXiv.

    https://arxiv.org/abs/2002.10033

    • Google Scholar
36. 1. Tomkins A
    2. Zhang M
    3. Heavlin WD

    (2017) Reviewer bias in single- versus double-blind peer review

    PNAS 114:12708–12713.

    https://doi.org/10.1073/pnas.1707323114

    • PubMed
    • Google Scholar

Thank you for submitting your article "Weak evidence of institutional and country-related bias in experimental study of 4,000 scientists' research assessments" to eLife for consideration as a Feature Article. Your article has been reviewed by three peer reviewers, and the following individual involved in review of your submission has agreed to reveal their identity: Bjorn Hammarfelt (Reviewer #1).

In view of the reviewers comments, you are invited to prepare a revised submission that addresses their comments, please see below.

Summary:

This is a carefully designed study and a well-written paper about bias in peer review based on authors' institutional and country location. The survey design is novel and the results seem robust. The main results show weak evidence for the claim that researchers are biased towards high-status countries and institutions. The presentation is logical and the paper is well-structured, and the main arguments are clearly articulated. However, there are also a number of points that need to be addressed to make the article suitable for publication.

Essential revisions:

1) Why choose the very top universities – like Harvard and MIT – rather than top universities that are less likely to be identified as the classic examples of “the best”. Is there a risk that participants of the survey would realize that this was part of the experiment, and thus acting accordingly in terms of “social desirability”. Perhaps a short discussion of these choices, maybe in the limitations section, would be useful.

2) Similarly, I wonder why there are no "Elite (non-US)" institutions (e.g. Oxford, ETH, etc.). Would not the inclusion of such universities make sense given that you include Elite (US) elite, and Non-elite (US) as well as Non-elite (non-US)? As it stands the selection might come off as a bit too focused on an American context, while the participants in the survey are from the entire world.

3) Please list somewhere in the article or supporting material the countries and institutions included in the survey that fall into each category and field.

4) The authors divide institutions into three categories. If I understood well, high US institutions are those in the top 20 of several discipline based rankings and in high positions in the US NRC doctorate programs ranking. Low status are those with low share of highly cited papers according to Leiden Ranking, and the same for non-US non-Elite from certain regions. While rankings may be a plausible way to go, I am not sure to what extent using the PP-top 10% score will help identifying low status institutions. In this case, the authors are looking for institutions with lower reputation (which not necessarily aligns with low citation performance), furthermore, the PP-top indicator is non-size independent, and therefore will work against larger institutions. Also, the Leiden Ranking ranks around 1000 institutions worldwide which fit certain publication threshold, probably including institutions that do not make it to the Leiden Ranking would give a better idea of low status.

5) I wonder to what extent the difference between the three categories is sufficiently detailed as to be able to show any differences. My concern is that you divide in so few groups that in the end differences may be hidden within those three large groups of institutions.

6) Effect sizes

The authors find that effect of manipulations on abstract quality ratings is likely not larger than the band -0.8 to 0.8 (16 pt scale) (Results). That does seem small, although in Cohen's d terms, d=-0.3 to 0.3 isn't all that small. Overall, we would expect effects of a light-weight manipulation in an online survey context to be tiny, but it's really hard to generalize such interventions to the field, where status effects may be large and subtle. So I'd like a little more language emphasizing low-ish (but informative) external validity here, and that the prior on light weight manipulation is very small effects.

7) You control for the reliability on the peers' judgment (how close is the abstract to their area of expertise). Why did you not do something similar with the institution as a means to control on the appropriateness of the category assignment of each institution? Could this be affecting also the results?

8) I understand that the six fields/disciplines were selected in order to cover a range of different disciplines, but a few paragraphs about why these particular fields were chosen would be useful.

9) Please discuss further the difference between choosing to read the full text and to include it in a conference. These are rather different evaluative situations: one may want to read a paper that one would not accept for a conference and vice versa. For example, a researcher might deem a paper as interesting, yet have doubts about the quality, or judge it as “worthy” of presentation while not being interested in reading it. You discuss that the disciplinary differences were higher when choosing to include in a conference or not (Discussion), and I would add that the competitive aspect of selecting for a conference makes this choice rather different than choosing to read a paper or not. This could be elaborated upon in the text and perhaps be something to study closer in future research.

10) In the framing of the paper the authors refer to biases on peer review but also discuss the Matthew Effect. While there is a relation between the halo effect that the authors are exploring and the Matthew Effect, the latter refers to the reception of publications (in terms of citations and prestige) while the former refers to biases in peer review. I think this difference is not sufficiently clear in the text, especially when reading the Appendix Text. I'd say that the fact that an halo effect may not be present in peer review would not necessarily reflect on a Matthew Effect or vice versa.

11) Situating this work in the existing literature.

I think the part of the literature review section that tries to distinguish this contribution from others isn't entirely fair. Here are the distinguishing points, followed by my comments on them:

a) Separating out location status vs. institution status

Comment: Why would this separating out really matter? Surely status of country is highly correlated with status of institution, and, presumably, the mechanism at work are inferences of quality from status. It seems fine to me to separate them out, but would be good to have an argument for why this is valuable to do

b) Previous RCTs in the peer review context may have been hampered by treatment leakage to the control group.

Comment: This surely happens ( https://arxiv.org/abs/1709.01609 ). But it just means the reported effects in those RCTs are lower bounds, as those papers I believe readily admit. So I don't think this has been a major methodological problem in dire need of fixing.

So I don't think what this experiment is doing is super novel and fixes glaring oversights in the literature. BUT, that isn't a bad thing, in my view. Simply adding observations, particularly those that are well executed, adequately powered, and cross-disciplinary, is important enough. It's an important topic, so we should have more than just one or two RCTs in it. Crucially, we know there are file-drawer effects out there, likely in this space too, so this study is useful for making sure the published record is unbiased.

Essential revisions:

1) Why choose the very top universities – like Harvard and MIT – rather than top universities that are less likely to be identified as the classic examples of “the best”. Is there a risk that participants of the survey would realize that this was part of the experiment, and thus acting accordingly in terms of “social desirability”. Perhaps a short discussion of these choices, maybe in the limitations section, would be useful.

This is an excellent point. We now reflect on why we made this choice, and the possible limitations of this choice, in the Discussion section.

2) Similarly, I wonder why there are no "Elite (non-US)" institutions (e.g. Oxford, ETH, etc). Would not the inclusion of such universities make sense given that you include Elite (U.S) elite, and Non-elite (US) as well as Non-elite (non-US)? As it stands the selection might come off as a bit too focused on an American context, while the participants in the survey are from the entire world.

This is a fair point. We decided to restrict the category of elite universities to the U.S. for the following reasons. (1) Our design allowed us to compare within-country variations in university status, and hereby tease apart the effect of country status and institution status. If we had included Oxford, Cambridge, ETH and University of Amsterdam, this would have required a more comprehensive set of lower-ranked universities in the UK, Switzerland and Netherlands as well, which in turn would have made our setup more complex. In other words, we wanted to keep the experiment simple. (2) 15 of the top 20 universities in the 2019 THE ranking, and 10 of the top 20 institutions in the 2019 US News ranking were located the U.S. Given this ratio, we found it justifiable to restrict our measure of institutional halo effects to elite universities in the U.S.

3) Please list somewhere in the article or supporting material the countries and institutions included in the survey that fall into each category and field.

Thank you for this request. We have included information on countries and institutions in Supplementary file 2.

4) The authors divide institutions into three categories. If I understood well, high US institutions are those in the top 20 of several discipline based rankings and in high positions in the US NRC doctorate programs ranking. Low status are those with low share of highly cited papers according to Leiden Ranking, and the same for non-US non-Elite from certain regions. While rankings may be a plausible way to go, I am not sure to what extent using the PP-top 10% score will help identifying low status institutions. In this case, the authors are looking for institutions with lower reputation (which not necessarily aligns with low citation performance), furthermore, the PP-top indicator is non-size independent, and therefore will work against larger institutions. Also, the Leiden Ranking ranks around 1000 institutions worldwide which fit certain publication threshold, probably including institutions that do not make it to the Leiden Ranking would give a better idea of low status.

Thank you for raising this point. There is certainly a status differential between institutions included in the Leiden Ranking and those not included. However, there is also a concern that using institutions that are not internationally recognized, e.g. by being present on such a ranking, could create other sentiments, such as distrust. Our argument is that we measure a status differential because the institutions are “on the same scale”, since they can be found on these rankings, while if we had included those outside the rankings, we might end up measuring on two different scales. In this case, we decided to rely on the PP-top 10% indicator because it measures a university’s share of publications among the 10% most cited as opposed to the number of publications among the 10% most cited (i.e. P-top 10%). In the revised Materials and methods section, we reflect in more detail on these considerations.

5) I wonder to what extent the difference between the three categories is sufficiently detailed as to be able to show any differences. My concern is that you divide in so few groups that in the end differences may be hidden within those three large groups of institutions.

Thank you for raising this important point. As an attempt to address this issue, we have computed the distribution of the average abstract ratings across universities, per discipline. As shown in Supplementary table 49 in Supplementary file 2, the within-group variations are quite modest. We chose to operate with the three categories to ensure reasonable statistical power in the analysis. As shown in Supplementary table 49 in Supplementary file 2 the Ns for individual universities are too small for any meaningful statistical comparison.

6) Effect sizes

The authors find that effect of manipulations on abstract quality ratings is likely not larger than the band -0.8 to 0.8 (16 pt scale) (Results). That does seem small, although in Cohen's d terms, d=-0.3 to 0.3 isn't all that small. Overall, we would expect effects of a light-weight manipulation in an online survey context to be tiny, but it's really hard to generalize such interventions to the field, where status effects may be large and subtle. So I'd like a little more language emphasizing low-ish (but informative) external validity here, and that the prior on light weight manipulation is very small effects.

Thank you for pointing this out. We now reflect on this issue in the first paragraph of the Discussion section.

7) You control for the reliability on the peers' judgment (how close is the abstract to their area of expertise). Why did you not do something similar with the institution as a means to control on the appropriateness of the category assignment of each institution? Could this be affecting also the results?

This is an interesting point. However, we are not entirely sure how such a control measure would be operationalized in practice? As we state in the Materials and methods section, “We carried out manual checks to ensure that all of the selected universities had active research environments within the relevant research disciplines.” Moreover, our selection of high-status institutions were all ranked highly in the U.S. National Research Council’s discipline-specific rankings of doctorate programs.

8) I understand that the six fields/disciplines were selected in order to cover a range of different disciplines, but a few paragraphs about why these particular fields were chosen would be useful.

This is a great point. In the updated Materials and methods section, we outline our reasons for targeting these particular disciplines.

9) Please discuss further the difference between choosing to read the full text and to include it in a conference. These are rather different evaluative situations: one may want to read a paper that one would not accept for a conference and vice versa. For example, a researcher might deem a paper as interesting, yet have doubts about the quality, or judge it as “worthy” of presentation while not being interested in reading it. You discuss that the disciplinary differences were higher when choosing to include in a conference or not (Discussion), and I would add that the competitive aspect of selecting for a conference makes this choice rather different than choosing to read a paper or not. This could be elaborated upon in the text and perhaps be something to study closer in future research.

We completely agree. We designed the two dichotomous outcome items to capture additional types of evaluative situations. While the two dichotomous outcomes may not be comparable, we find both of them suitable for the purposes of our experiment. We have revised our discussion of these outcomes in accordance with your suggestions (please see the paragraph on scope conditions).

10) In the framing of the paper the authors refer to biases on peer review but also discuss the Matthew Effect. While there is a relation between the halo effect that the authors are exploring and the Matthew Effect, the latter refers to the reception of publications (in terms of citations and prestige) while the former refers to biases in peer review. I think this difference is not sufficiently clear in the text, especially when reading the Appendix Text. I'd say that the fact that an halo effect may not be present in peer review would not necessarily reflect on a Matthew Effect or vice versa.

This is an excellent point. In the revised Introduction, we have limited our focus to studies that relate directly to the question of halo effects.

11) Situating this work in the existing literature.

I think the part of the literature review section that tries to distinguish this contribution from others isn't entirely fair. Here are the distinguishing points, followed by my comments on them:

a) Separating out location status vs. institution status

Comment: Why would this separating out really matter? Surely status of country is highly correlated with status of institution, and, presumably, the mechanism at work are inferences of quality from status. It seems fine to me to separate them out, but would be good to have an argument for why this is valuable to do

b) Previous RCTs in the peer review context may have been hampered by treatment leakage to the control group.

Comment: This surely happens ( https://arxiv.org/abs/1709.01609 ). But it just means the reported effects in those RCTs are lower bounds, as those papers I believe readily admit. So I don't think this has been a major methodological problem in dire need of fixing.

So I don't think what this experiment is doing is super novel and fixes glaring oversights in the literature. BUT, that isn't a bad thing, in my view. Simply adding observations, particularly those that are well executed, adequately powered, and cross-disciplinary, is important enough. It's an important topic, so we should have more than just one or two RCTs in it. Crucially, we know there are file-drawer effects out there, likely in this space too, so this study is useful for making sure the published record is unbiased.

Thank you for raising these important points. We have revised the last part of the Introduction in accordance with your suggestions.

Author details

1. Mathias Wullum Nielsen

   Mathias Wullum Nielsen is in the Department of Sociology, University of Copenhagen, Copenhagen, Denmark

   Contribution

   Conceptualization, Resources, Data curation, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

   For correspondence

   mwn@soc.ku.dk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8759-7150

2. Christine Friis Baker

   Christine Friis Baker is in the Danish Centre for Studies in Research and Research Policy, Department of Political Science, Aarhus University, Aarhus, Denmark

   Contribution

   Formal analysis, Investigation, Visualization, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-3370-021X

3. Emer Brady

   Emer Brady is in the Danish Centre for Studies in Research and Research Policy, Department of Political Science, Aarhus University, Aarhus, Denmark

   Contribution

   Formal analysis, Investigation, Visualization, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-6065-8096

4. Michael Bang Petersen

   Michael Bang Petersen is in the Department of Political Science, Aarhus University, Aarhus, Denmark

   Contribution

   Formal analysis, Validation, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-6782-5635

5. Jens Peter Andersen

   Jens Peter Andersen is in the Danish Centre for Studies in Research and Research Policy, Department of Political Science, Aarhus University, Aarhus, Denmark

   Contribution

   Conceptualization, Data curation, Formal analysis, Funding acquisition, Validation, Investigation, Visualization, Methodology, Project administration, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2444-6210

• 1,959

  views

• 129

  downloads

• 6

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.