Meta-Research: Investigating disagreement in the scientific literature
- Centre for Science and Technology Studies, Leiden University, Netherlands
- SciTech Strategies, Inc, United States
- École de bibliothéconomie et des sciences de l’information, Université de Montréal, Canada
- School of Public Policy, Georgia Institute of Technology, United States
- School of Informatics, Computing, and Engineering, Indiana University, United States
Figures
Figure 1 with 1 supplement
Agreement and validity of different combinations of signal term and filter term.
Measures calculated from 50 randomly-sampled citances for each combination of signal term (vertical axis) and filter term (horizontal axis), annotated as valid or invalid instances of disagreement by two independent coders. (a) Percentage agreement, or the proportion of citances for which coders independently agreed on the label. (b) Percentage validity, or the proportion of citances which both coders labeled as valid. Averages for the various signal terms are shown in the left-most column; averages for the various filter terms are shown in the bottom row. (c) Percentage agreement (blue circles) and validity (red diamonds) of each signal/filter term combination, ordered from highest percent validity (top) to lowest percent validity (bottom). Numbers on the right are the total number of citances returned by querying using the signal/filter term combination, and are colored according to their log-transformed value. (d) Log-transformed count of citances returned by each query combination, colored by the (log-transformed) number of citances. Citance counts are non-exclusive, meaning that citances of the form debat* + studies will also be counted towards debat* _standalone_.
Figure 1—figure supplement 1
Distribution of citances returned by signal/filter term queries.
Callouts (I, II, …, VIII) map to examples in Table S3. a. Distribution of all disagreement citances appearing in papers across five fields: Biomedical and Health Sciences, Life and Earth Sciences, Physical Sciences and Engineering, Social Sciences and Humanities, and Math and Computer Science. b–d. Percentage change between the actual number of citances per field and signal/filter term combination compared to the expected given the disciplinary distribution (from a). The red line corresponds to 0 percent increase between the actual and expected. White dots indicate that the citances for that signal/filter term are under-represented (lower than expected, ratio less than zero), whereas black dots indicate that citances are over-represented (more than expected). Shown aggregated across signal terms (b), filter terms (c), and for all signal/filter term combinations (d).
Figure 2 with 1 supplement
Disagreement reflects a hierarchy of fields.
(a) Percent of all citances in each field that contain signals of disagreement, meaning they were returned by one of the 23 queries with validity of 80% or higher. Fields marked by lower consensus, such as in Soc & Hum, had a greater proportion of disagreement. (b) Percent of disagreement by field and over time, showing little change overall, but some changes by field. Text indicates the average percentage-point change per-year by field.
Figure 2—figure supplement 1
Percent of all citances returned by each of the 23 queries with validity over 80%.
Each panel corresponds to the signal phrase, and lines within each panel to filter phrases.
Figure 3 with 4 supplements
Heterogeneity in disagreement across meso-fields.
Fine-grained view across 817 meso-level fields, each a cluster of publications grouped and positioned based on their citation links derived from the Web of Science database (see Materials and methods), 2000–2015. The area of each point is proportional to the number of disagreement citances in that field. Overlapping points are an artifact of their position and size, and bear no additional meaning. Color maps to the log ratio of the share of disagreement citances given the mean share across all fields, truncated at 4 x greater and 4 x lower than the mean. Soc & Hum tends to have a greater proportion of disagreement citances, and Math & Comp the least. Other panels show the same data, but highlight the meso-fields in each high-level field. Meso-fields of interest are highlighted, and labels show a selection of journals in which papers in each field are published. Journals listed in labels are representative of each meso-field in the Web of Science, and is not limited to those represented in the Elsevier ScienceDirect data. An interactive version of this visualization is available online at https://tinyurl.com/disagreement-meso-fields.
Figure 3—figure supplement 1
On average, older papers are less likely to receive a disagreement citance, though this trend does not hold for the “hard” sciences.
Percentage of disagreement citances by the relative age of the citing to the cited paper, in years, and high-level field, for papers published between 2000 and 2015. Intensity of color corresponds to the age category of the cited paper.
Figure 3—figure supplement 2
Distribution of citances by their position in the text of the manuscript, and by field.
Shown for all citances (solid line) and disagreement citances (dotted line). For example, about 15% of disagreement citances in Physical Sciences and Engineering appear in the first 0%–5% of the sentences in documents.
Figure 3—figure supplement 3
Little difference in disagreement between men and women.
Percentage of disagreement citances by gender of the citing-paper author, their authorship position (first or last), and the high-level field. Numbers above each bar corresponds to the ratio difference between the percentage of disagreement between women and men. The number below each bar corresponds to the number of disagreement citances. we infer a gender for the first and last authors of papers with a disagreement citance published after 2008, determined based on the author’s first name as in past work (Larivière et al., 2013).
Figure 3—figure supplement 4
Authors disagree less when citing their own work.
Percentage of disagreement citances among instances of non-self and self-citation, 2000–2015. A citance is defined as a self-citation when the citing and cited paper have at least one name in common. Results are shown by field. Numbers below each bar are the number of disagreement citances. Overall, disagreement is 2.4 times more common for non-self citation than for self-citation, with variance between major fields.
Figure 4
Full research articles with a disagreement citance are cited more.
The y-axis shows the difference in average citation counts for papers containing at least one disagreement citance, and for papers without. Positive values indicate that publications with disagreement received more citations than those without. Values are shown for the population of publications in each year following publication (x-axis). Shown here for only articles labeled in the Web of Science database as full research articles.
Tables
Table 1
Specific terms comprising each of the thirteen signal term sets and specific exceptions.
The “*” symbol (wildcard) captures possible variants.
| Signal term | Variants | Exclusions | Results |
|---|---|---|---|
| challenge* | 405,613 | ||
| conflict* | 212,246 | ||
| contradict* | 115,375 | ||
| contrary | 171,711 | ||
| contrast* | 1,257,866 | ||
| controvers* | 154,608 | ||
| debat* | “parliament* debat*”, “congress* debat*”, “senate* debat*”, “polic* debat*”, “politic* debat*”, “public* debat*”, “societ* debat*” | 150,617 | |
| differ* | “different*” | 2,003,677 | |
| disagree* | “not agree*”, “no agreement” | “range”, “scale”, “kappa”, “likert”, “agree*” and/or “disagree” within a ten-word range of each other. | 52,615 |
| disprov* | “prove*” and “disprove*” within a ten-word range | 2,938 | |
| no consensus | “lack of consensus” | “consensus sequence”, “consensus site” | 16,632 |
| questionable | 24,244 | ||
| refut* | “refutab*” | 10,322 | |
| total | 4,578,464 |
Table 2
Specific terms comprising each of the four filter term sets.
| studies | studies; study; previous work; earlier work; literature; analysis; analyses; report; reports |
| ideas | idea*; theory; theories; assumption*; hypothesis; hypotheses |
| methods | model*, method*, approach*; technique* |
| results | result*; finding*; outcome*; evidence; data; conclusion*; observation* |
Table 3
Being cited in the context of disagreement has little impact on citations in the year following.
For each field, shown are the number of cited papers, as well as for t + 1, t + 2 and t + 3 with t being the year in which a cited paper first featured in the context of disagreement, its average number of received citations, expected number of received citations, and d the ratio between these two values. When d is greater than one, papers cited in the context of disagreement receive more citations in the following year than expected. When d is less than one, they receive fewer citations than expected.
| Scientific field | Number of records | Avg. citations, t + 1 following disagreement | Expected citations, t + 1 following disagreement | Avg. citations, t + 2 | Expected citations, t + 2 | Avg. citations, t + 3 | Expected citations, t + 3 | |||
|---|---|---|---|---|---|---|---|---|---|---|
| All | 109,545 | 3.03 | 3.08 | 0.983 | 3.02 | 3.05 | 0.990 | 2.96 | 2.98 | 0.993 |
| Bio & Health | 60,707 | 2.73 | 2.81 | 0.969 | 2.68 | 2.75 | 0.974 | 2.56 | 2.65 | 0.966 |
| Life & Earth | 20,581 | 3.43 | 3.35 | 1.023 | 3.55 | 3.42 | 1.038 | 3.63 | 3.44 | 1.056 |
| Math & Comp | 770 | 3.36 | 3.34 | 1.005 | 3.54 | 3.28 | 1.080 | 3.29 | 2.97 | 1.109 |
| Phys & Engr | 18,011 | 3.55 | 3.52 | 1.006 | 3.48 | 3.44 | 1.010 | 3.43 | 3.34 | 1.027 |
| Soc & Hum | 9,476 | 3.04 | 3.11 | 0.979 | 3.20 | 3.28 | 0.975 | 3.30 | 3.40 | 0.971 |
Additional files
-
Transparent reporting form
- https://cdn.elifesciences.org/articles/72737/elife-72737-transrepform1-v1.pdf
-
Supplementary file 1
Tables S1 – S4.
- https://cdn.elifesciences.org/articles/72737/elife-72737-supp1-v1.docx
-
Supplementary file 2
Papers with most disagreement citances and papers most often cited in the context of disagreement.
- https://cdn.elifesciences.org/articles/72737/elife-72737-supp2-v1.docx
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Meta-Research: Investigating disagreement in the scientific literature
eLife 10:e72737.
https://doi.org/10.7554/eLife.72737
