Meta-Research: A Collection of Articles

Meta-research is research that uses the methods of science to study science itself. Also known as meta-science or the science of science, it involves studying the processes and decisions that shape the evolution of scientific research. This collection of articles highlights the breadth of meta-research with articles on topics as diverse as gender bias in peer review, statistical power in clinical trials and the readability of the scientific literature. Information on submitting meta-research papers to eLife is available in eLife Latest: Highlighting meta-research.

Collection

1. 1. Physics of Living Systems

   Female-dominated disciplines have lower evaluated research quality and funding success rates, for men and women

   Alex James, Franca Buelow ... Ann Brower

   There is a systemic bias in research evaluation, i.e., disciplines with a higher proportion of women have lower evaluations and funding success for all researchers, regardless of gender.
2. 1. Biochemistry and Chemical Biology

   Meta-Research: Blinding reduces institutional prestige bias during initial review of applications for a young investigator award

   Anne E Hultgren, Nicole MF Patras, Jenna Hicks

   Blinding the initial review of applications for Beckman Young Investigator awards reduces prestige bias, but has no impact on the success rates for different genders.
3. 1. Physics of Living Systems

   Gendered hiring and attrition on the path to parity for academic faculty

   Nicholas LaBerge, Kenneth Hunter Wapman ... Daniel B Larremore

   Achieving gender parity among U.S. tenured and tenure-track faculty will require changes to hiring, which has substantially greater impacts on faculty gender representation than gendered differences in attrition rates.
4. 1. Genetics and Genomics

   Meta-Research: Understudied genes are lost in a leaky pipeline between genome-wide assays and reporting of results

   Reese Richardson, Heliodoro Tejedor Navarro ... Thomas Stoeger

   Find My Understudied Genes (FMUG) is a data-driven tool created to counteract the abandonment of understudied genes in -omics research.

5. Meta-Research: The changing career paths of PhDs and postdocs trained at EMBL

   Junyan Lu, Britta Velten ... Rachel Coulthard-Graf

   An analysis of the career and publication outcomes for 2284 early career researchers trained at EMBL provides insights into the evolving career landscape for life scientists.
6. 1. Medicine

   Gender differences in submission behavior exacerbate publication disparities in elite journals

   Isabel Basson, Chaoqun Ni ... Vincent Larivière
7. 1. Neuroscience

   ChatGPT identifies gender disparities in scientific peer review

   Jeroen PH Verharen

   Generative artificial intelligence, such as OpenAI’s ChatGPT, can be used to analyze scientific texts with specialized constructions, including peer review reports.
8. 1. Computational and Systems Biology

   Analysis of NIH K99/R00 awards and the career progression of awardees

   Nicole C Woitowich, Sarah R Hengel ... Daniel J Tyrrell

   There is a significant disadvantage to receive a major grant (i.e., R01) for K99/R00 awardees that are women, those at lower funded institutions, and those with lower career mobility.
9. 1. Neuroscience

   Trends in Self-citation Rates in High-impact Neurology, Neuroscience, and Psychiatry Journals

   Matthew Rosenblatt, Saloni Mehta ... Dustin Scheinost
10. 1. Computational and Systems Biology

    Analysis of science journalism reveals gender and regional disparities in coverage

    Natalie R Davidson, Casey S Greene

    Those quoted and mentioned in stories in Nature News mostly have names associated with men and are much less likely to have names associated with East Asia.

11. Meta-Research: The effect of the COVID-19 pandemic on the gender gap in research productivity within academia

    Kiran GL Lee, Adele Mennerat ... Antica Culina

    A systematic review and meta-analysis of 55 articles about the impact of the COVID-19 pandemic on researchers confirms that the pandemic increased the gender gap in research productivity.

12. Meta-Research: Systemic racial disparities in funding rates at the National Science Foundation

    Christine Yifeng Chen, Sara S Kahanamoku ... Justin Hosbey

    White principal investigators applying to the National Science Foundation are consistently funded at higher rates than most non-white PIs.
13. 1. Computational and Systems Biology

    Meta-Research: How parenthood contributes to gender gaps in academia

    Xiang Zheng, Haimiao Yuan, Chaoqun Ni

    Gender gaps in academic achievement are in fact "parenthood gender gaps", with mothers experiencing higher levels of work-family conflict, and receiving lower levels of partner support, than fathers.
14. 1. Computational and Systems Biology
    2. Epidemiology and Global Health

    National Institutes of Health research project grant inflation 1998 to 2021

    Michael S Lauer, Joy Wang, Deepshikha Roychowdhury

    National Institutes of Health (NIH) grants have evolved over time, with a greater proportion of funds going to solicited projects, but inflation-adjusted costs have remained relatively stable following a brief increase during the NIH-doubling (1998-2003).
15. 1. Medicine

    Biomedical supervisors’ role modeling of open science practices

    Tamarinde L Haven, Susan Abunijela, Nicole Hildebrand

    Supervisors' engagement in open science practices such as data sharing is associated with a greater odds that Ph.D. students engage in these practices and share their data.
16. 1. Neuroscience

    Science Forum: How failure to falsify in high-volume science contributes to the replication crisis

    Sarah M Rajtmajer, Timothy M Errington, Frank G Hillary

    An increased emphasis on falsification – the direct testing of strong hypotheses – will lead to faster progress in science by allowing well-specified hypotheses to be eliminated.
17. 1. Medicine

    Tracing the path of 37,050 studies into practice across 18 specialties of the 2.4 million published between 2011 and 2020

    Moustafa Abdalla, Salwa Abdalla, Mohamed Abdalla

    Comprehensive decade-long analysis of point-of-care resources reveals insights into how clinical research makes its way into practice across different medical specialties and through time to enable us to identify potential translational bottlenecks in the pathway from research to practice.
18. 1. Medicine

    Insights from full-text analyses of the Journal of the American Medical Association and the New England Journal of Medicine

    Moustafa Abdalla, Mohamed Abdalla ... Scott H Podolsky

    Historical analysis of nearly half-a-million articles from the Journal of the American Medical Association and the New England Journal of Medicine since their inception frames the shifting scientific, material, ethical, and epistemic underpinnings of medicine over time, including today.
19. 1. Neuroscience

    Meta-Research: Lessons from a catalogue of 6674 brain recordings

    Alexis DJ Makin, John Tyson-Carr ... Marco Bertamini

    A complete public catalogue of brain recording data from a single laboratory has revealed new scientific findings, and has also shed light on aspects of the scientific process, such as statistical power and publication bias.
20. 1. Medicine

    Meta-research: justifying career disruption in funding applications, a survey of Australian researchers

    Adrian Barnett, Katie Page ... Susanna Cramb

    A survey of health and medical researchers revealed important diffrences in how career disruption is treated in funding decisions, with differences in what applicants with disruption are willing to divulge and the adjustments made by reviewers.
21. 1. Computational and Systems Biology
    2. Medicine

    Meta-Research: Author-level data confirm the widening gender gap in publishing rates during COVID-19

    Emil Bargmann Madsen, Mathias Wullum Nielsen ... Jens Peter Andersen

    The gender gap in publication productivity has increased during the COVID-19 pandemic, with potential implications for the careers of women working in research.

22. Meta-Research: Investigating disagreement in the scientific literature

    Wout S Lamers, Kevin Boyack ... Dakota Murray

    An approach based on cue phrases can be used to identify instances of disagreement in scientific articles and compare the level of disagreement in various disciplines.
23. 1. Cell Biology
    2. Medicine

    An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

    Richard K Nakamura, Lee S Mann ... Bruce Reed

    Redaction, intended to obscure personal and institutional identity, reduced the gap between Black vs. White applicants in simulated review of real NIH scientific grant applications, thus providing qualified support for double-blinded models of peer review.
24. 1. Computational and Systems Biology

    Inequalities in the distribution of National Institutes of Health research project grant funding

    Michael S Lauer, Deepshikha Roychowdhury

    Funding inequalities for NIH-supported research project grant investigators and organizations have increased over the past 25 years, but have modestly decreased over the past 3 years.
25. 1. Computational and Systems Biology

    Associations of topic-specific peer review outcomes and institute and center award rates with funding disparities at the National Institutes of Health

    Michael S Lauer, Jamie Doyle ... Deepshikha Roychowdhury

    An analysis of peer review and funding outcomes of NIH research applications shows that funding disparities of topics preferred by African American Black investigators are not due to peer review preferences or biases.

26. Meta-Research: Weak evidence of country- and institution-related status bias in the peer review of abstracts

    Mathias Wullum Nielsen, Christine Friis Baker ... Jens Peter Andersen

    A preregistered survey experiment spanning six disciplines has found weak evidence of bias in favour of authors from high-status countries and institutions.

27. Meta-Research: A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018

    David G Pina, Ivan Buljan ... Ana Marušić

    A study of more than 75,000 grant proposals to the European Union indicates that the outcomes of the peer review process remain stable in response to changes in the way that peer review is organized.
28. 1. Computational and Systems Biology
    2. Microbiology and Infectious Disease

    Meta-Research: COVID-19 research risks ignoring important host genes due to pre-established research patterns

    Thomas Stoeger, Luís A Nunes Amaral

    An analysis of 10,395 research publications about COVID-19 that mention at least one human gene reveals that many genes implicated in SARS-CoV-2 infection by genome-wide studies remain unstudied.

29. Meta-Research: Journal policies and editors’ opinions on peer review

    Daniel G Hamilton, Hannah Fraser ... Fiona Fidler

    A survey of journals and editors in five areas of research - ecology, economics, medicine, physics and psychology - reveals a range of differences in their approach to peer review.
30. 1. Biochemistry and Chemical Biology

    Meta-Research: Task specialization across research careers

    Nicolas Robinson-Garcia, Rodrigo Costas ... Gabriela F Nane

    An analysis of author contribution statements in a large number of published papers reveals biases in the career trajectories of scientists.

31. Meta-Research: Questionable research practices may have little effect on replicability

    Rolf Ulrich, Jeff Miller

    Although questionable research practices inflate false positive rates, they have little effect on replicability because they also increase power.

32. Meta-Research: Evaluating the impact of open access policies on research institutions

    Chun-Kai (Karl) Huang, Cameron Neylon ... Chloe Brookes-Kenworthy

    A longitudinal study of open access levels at 1,207 research institutions worldwide reveals high performing institutions in Latin America, Africa and Asia.
33. 1. Computational and Systems Biology

    Meta-Research: International authorship and collaboration across bioRxiv preprints

    Richard J Abdill, Elizabeth M Adamowicz, Ran Blekhman

    An analysis of 67,885 preprints on bioRxiv finds evidence for disparities in international participation that are similar to the disparities found in conventional journals.

34. Meta-Research: The growth of acronyms in the scientific literature

    Adrian Barnett, Zoe Doubleday

    A study of 24 million articles has revealed that scientists have created more than 1 million acronyms since 1950, most of which have been used fewer than 10 times.

35. Meta-Research: Large-scale language analysis of peer review reports

    Ivan Buljan, Daniel Garcia-Costa ... Ana Marušić

    The linguistic characteristics of peer review reports are not influenced by research area, type of review or reviewer gender, which is evidence for the robustness of peer review.
36. 1. Medicine

    Meta-Research: COVID-19 medical papers have fewer women first authors than expected

    Jens Peter Andersen, Mathias Wullum Nielsen ... Reshma Jagsi

    Lockdowns in the United States caused by the COVID-19 pandemic appear related to a decrease in the number of women publishing research papers, especially as first authors.

37. Meta-Research: A 10-year follow-up study of sex inclusion in the biological sciences

    Nicole C Woitowich, Annaliese Beery, Teresa Woodruff

    Sex-inclusive research practices have increased in many biological disciplines over the past decade, yet critical sex-based analyses and reporting have remained stagnant in most of these disciplines.
38. 1. Medicine

    Knowledge synthesis of 100 million biomedical documents augments the deep expression profiling of coronavirus receptors

    AJ Venkatakrishnan, Arjun Puranik ... Venky Soundararajan

    SARS-CoV-2 receptor ACE2 is expressed in nasal olfactory epithelia, tongue keratinocytes and small intestine enterocytes, connected with the COVID-19 patient phenotypes such as anosmia and diarrhea.
39. 1. Computational and Systems Biology
    2. Neuroscience

    Meta-Research: Dataset decay and the problem of sequential analyses on open datasets

    William Hedley Thompson, Jessey Wright ... Russell A Poldrack

    Open data provides an opportunity to perform new analyses on preexisting data, but trade-offs are required to limit an increase in false positives.

40. Meta-Research: Reader engagement with medical content on Wikipedia

    Lauren A Maggio, Ryan M Steinberg ... John M Willinsky

    Readers of health and medicine Wikipedia pages are more likely to hover over and view footnotes than other readers, but less likely to view the hyperlinked sources in these footnotes.
41. 1. Neuroscience

    NeuroQuery, comprehensive meta-analysis of human brain mapping

    Jérôme Dockès, Russell A Poldrack ... Gaël Varoquaux

    Full-text analysis of the human brain-imaging literature can predict the spatial distribution of observations in the brain given the description of a subject of study.
42. 1. Computational and Systems Biology

    

    Meta-Research: Releasing a preprint is associated with more attention and citations for the peer-reviewed article

    Darwin Y Fu, Jacob J Hughey

    An analysis of more than 70,000 journal articles, including 5405 that were first released as a preprint on bioRxiv, shows that articles with a preprint received 49% more attention and 36% more citations than articles without one.
43. 1. Computational and Systems Biology

    Meta-Research: Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations

    Erin C McKiernan, Lesley A Schimanski ... Juan P Alperin

    Almost a quarter of faculty evaluation documents from US and Canadian universities mention Journal Impact Factor and often imply that it measures research quality.

44. Meta-Research: Gender variations in citation distributions in medicine are very small and due to self-citation and journal prestige

    Jens Peter Andersen, Jesper Wiborg Schneider ... Mathias Wullum Nielsen

    In studies of gender disparities in academia, increased focus is required on within-group variability and between-group overlap of distributions when interpreting and reporting results.
45. 1. Computational and Systems Biology

    Meta-Research: Centralized scientific communities are less likely to generate replicable results

    Valentin Danchev, Andrey Rzhetsky, James A Evans

    Analysis of data on drug-gene interactions suggests that decentralized collaboration will increase the robustness of scientific findings in biomedical research.
46. 1. Medicine

    Meta-Research: A comprehensive review of randomized clinical trials in three medical journals reveals 396 medical reversals

    Diana Herrera-Perez, Alyson Haslam ... Vinay Prasad

    An analysis of more than 3000 randomized controlled trials published in JAMA, the Lancet and NEJM has identified 396 medical reversals.

47. Meta-Research: Tracking the popularity and outcomes of all bioRxiv preprints

    Richard J Abdill, Ran Blekhman

    Data are reported for the monthly number of uploads to and downloads from bioRxiv, and for the number of preprints that are later published in peer-reviewed journals.

48. Meta-Research: How significant are the public dimensions of faculty work in review, promotion and tenure documents?

    Juan P Alperin, Carol Muñoz Nieves ... Erin C McKiernan

    An analysis of review, promotion and tenure documents from 129 US and Canadian universities suggests institutions could better fulfill their public missions by changing how they incentivize the public dimensions of faculty work.
49. 1. Microbiology and Infectious Disease

    Meta-Research: Gender inequalities among authors who contributed equally

    Nichole A Broderick, Arturo Casadevall

    An analysis of papers in which two or more authors shared first-author position found that male authors were more likely than female authors to appear first in the author list.
50. 1. Computational and Systems Biology

    Meta-Research: Incidences of problematic cell lines are lower in papers that use RRIDs to identify cell lines

    Zeljana Babic, Amanda Capes-Davis ... Anita E Bandrowski

    The use of Research Resource Identifiers (RRIDs) improves the proper use of cell lines in the biomedical literature.
51. 1. Medicine

    Meta-Research: Why we need to report more than 'Data were Analyzed by t-tests or ANOVA'

    Tracey L Weissgerber, Oscar Garcia-Valencia ... Stacey J Winham

    Many papers in basic biomedical science do not contain the information that is needed to determine what statistical tests were used and to verify the results of these tests.
52. 1. Medicine

    Research: Adequate statistical power in clinical trials is associated with the combination of a male first author and a female last author

    Willem M Otte, Joeri K Tijdink ... Christiaan H Vinkers

    An analysis of 31,873 clinical trials shows that adequate statistical power was most often present in trials with a male first author and a female last author.

53. Point of View: The NIH must reduce disparities in funding to maximize its return on investments from taxpayers

    Wayne P Wahls

    A more balanced distribution of NIH grant funding among investigators would strengthen the diversity of the research enterprise, increase the likelihood of scientific breakthroughs, and lead to a greater return on taxpayers' investments.
54. 

    Research: Sci-Hub provides access to nearly all scholarly literature

    Daniel S Himmelstein, Ariel Rodriguez Romero ... Casey S Greene

    The availability of almost all articles from toll access journals in the Sci-Hub repository will disrupt scholarly publishing towards more open models.
55. 1. Epidemiology and Global Health

    Standardized mean differences cause funnel plot distortion in publication bias assessments

    Peter-Paul Zwetsloot, Mira Van Der Naald ... Kimberley E Wever

    Funnel plots of the Standardized Mean Difference versus the standard error are prone to distortion, leading to false-positive tests for funnel plot asymmetry, and therefore using the Normalised Mean Difference, or a sample size-based precision estimate, are more reliable alternatives.
56. 

    Research: The readability of scientific texts is decreasing over time

    Pontus Plavén-Sigray, Granville James Matheson ... William Hedley Thompson

    Scientific abstracts have become less readable over the past 130 years, in part because recent texts include more general scientific jargon than older texts.

57. Research: Gender bias in scholarly peer review

    Markus Helmer, Manuel Schottdorf ... Demian Battaglia

    Gender-bias in peer reviewing might persist even when gender-equity is reached because both male and female editors operate with a same-gender preference whose characteristics differ by editor-gender.

58. Research: Publication bias and the canonization of false facts

    Silas Boye Nissen, Tali Magidson ... Carl T Bergstrom

    Publication bias, in which positive results are preferentially reported by authors and published by journals, can restrict the visibility of evidence against false claims and allow such claims to be canonized inappropriately as facts.

59. Research: Decoupling of the minority PhD talent pool and assistant professor hiring in medical school basic science departments in the US

    Kenneth D Gibbs Jr, Jacob Basson ... David A Broniatowski

    A systems-level analysis of the biomedical workforce in the US shows that current strategies to enhance faculty diversity are unlikely to have a significant impact, and that there is a need to increase the number of PhDs from underrepresented minority backgrounds who move on to postdoctoral positions.
60. 1. Computational and Systems Biology

    Research: Bias in the reporting of sex and age in biomedical research on mouse models

    Oscar Flórez-Vargas, Andy Brass ... Goran Nenadic

    A text-mining study suggests that about half of the papers reporting the results of experiments on mice do not report the sex and age of the mice.

61. Research: NIH peer review percentile scores are poorly predictive of grant productivity

    Ferric C Fang, Anthony Bowen, Arturo Casadevall

    Peer review scores were poorly predictive of research project success in this large dataset, suggesting that reviewers cannot reliably predict which meritorious applications are most likely to be productive.
62. 1. Epidemiology and Global Health
    2. Medicine

    A meta-analysis of threats to valid clinical inference in preclinical research of sunitinib

    Valerie C Henderson, Nadine Demko ... Jonathan Kimmelman

    Preclinical efficacy experiments testing sunitinib in animal cancer models display a lack of methodological rigour, with trim-and-fill analysis suggesting prominent publication bias that leads to an overestimation of treatment effect.
63. 1. Developmental Biology
    2. Evolutionary Biology

    Research: Titles and abstracts of scientific reports ignore variation among species

    Barbara R Migeon

    Ignoring the probability of species variation when reporting observations about biological processes leads to misinterpretation of the experimental data.

64. Research: Financial costs and personal consequences of research misconduct resulting in retracted publications

    Andrew M Stern, Arturo Casadevall ... Ferric C Fang

    In the first study attempting to formally quantify the deleterious impact of research misconduct on funding sources and publication output, we found that misconduct accounts for a small but substantial portion of American biomedical science funding dollars and damages the productivity and rate of funding acquisition of those who commit misconduct.