Figures and data
Verification categories used to assess claims.
Proportion of claims in broad reproducibility assessment categories before and after experimental validation by the ReproSci project.
See description of each category in Material and Methods. In this article, we are using indirect reproducibility, also called conceptual replicability, to assess the validity of a claim using alternate methods to those used in the original study.
Flow of the study.
Claims were first classified in five categories based on literature. 45 unchallenged claims were selected to be experimentally tested an classified either as challenged or as verified. We further classify some unchallenged as consistent or inconsistent depending on their consistency with current knowledge.s
Percentage of claims by category according to journal-impact and institution ranking.
(A, B) The distribution of claims across the five assessed categories for articles published in low-impact, high-impact, and trophy journals. The number of claims is indicated at the top. Panel A presents raw counts, while Panel B shows normalized data. (C) The distribution of claims across the five assessed categories according to the ranking of the last author institution in Shanghai Ranking’s 2010 Academic Ranking of World Universities. The number of claims is indicated at the top, while the number of institutions in each category is indicated at the bottom. The Not Ranked categories encompass research institutes (ex. CNRS), hospitals and other institution that are not included in the Shanghai ranking. In this figure, and in the rest of the paper, the number nc indicates the number of claims in each category, while the number at the bottom highlight the count in each category (e.g there are 11 high-impact journals, and 73 Top-50 institutions).
Evolution of claim replicability over time.
(A, B) The distribution of claims by categories of assessment over time is shown in A, and the proportion normalized over the total number of claims per time period is shown in B.
Distribution of irreproducibility according to first authors.
A) Distribution of irreproducible claims among first author. We note that all irreproducible claims are produced by less than 20% of the first authors. B) Distribution of irreproducibility for all first authors. Each dot represents an author, ranked according to the proportion of challenged claims. The size of the circle indicates the number of claims and the color the percentage of verified claims.
Influence of parameters associated with first author on reproducibility.
A) Repartition of claims in each assessment category according to the gender of the first authors. The gender of four authors could not be inferred based on first name ambiguity explaining why the total number of authors is 285. B) Repartition of claims in each assessment category according to status of the first authors. The stages of career of 279 first authors were obtained by interviewing last authors, querying Linkedln public profiles or extrapolated according to the publication list. Senior staff refers to permanent position scientist working in a lab. ND stands for Not Determined. C) Proportion of challenged claims as a function of the number of published articles for first-authors. The color indicates the percentage of claims that are verified.
Distribution of irreproducibility according to last authors (PI).
A) Distribution of irreproducible claims among leading authors. B) Distribution of irreproducibility for all leading authors. Each dot represents a leading author, ranked according to the proportion of challenged claims. The size of the circle indicates the number of claims and the color the percentage of verified claims.
Influence of various parameters associated with the last author on reproducibility.
A) Distribution of claims by category according to the gender of the last authors. B) Distribution of claims by category according to the seniority of the last authors. Senior PI were defined as having published a last author article at least 5 years before the considered articles. The total of junior and senior authors is higher than the total of last authors as some PIs published both as junior and senior. C) Proportion of challenged claims by last author as a function of number of articles.
Patterns of irreproducibility by last author according to time of starting their laboratory and research style.
A. The proportion of challenged claims by last author is shown according to the time at which the author published their first paper as a first or last authors. The increase in irreproducibility is associated with PIs who started their lab after 1995. Circle size indicates the number of published articles by each author. B. Distribution of claims by last author according to whether they have published at least one first author article on Drosophila immunity in another laboratory. C. Distribution of claims according to research style. Proportion of challenged claims by PIs who have continuously work on the Drosophila immunity (‘continuity PIs’) or made a transient incursion into the field (‘exploratory PIs’). Only PIs, who published as last authors after 1995, were taken into consideration.
