Concerns over reliability (Ioannidis, 2005) and reproducibility (Prinz et al., 2011; Begley and Ellis, 2012) in biomedical science call into question the cumulative process of building on prior published results. In a publication environment that rewards novel findings over verifications (Nosek et al., 2015; Alberts et al., 2015), the replicability of research claims that biomedical scientists assemble into biological models, drug development trials, and treatment regimes remains unknown (Begley and Ellis, 2012; Yildirim et al., 2007). Exact replications of biomedical research (Errington et al., 2014) occur only on small scales due to prohibitive expense and limited professional incentive.

Replication failures are typically attributed to systemic bias in a publication system that favors positive results (Ioannidis, 2005). This incentivizes questionable research choices such as p-hacking (Head et al., 2015; Simonsohn et al., 2014), ‘flexible’ data analysis (Simmons et al., 2011), low statistical power (Dumas-Mallet et al., 2017), selective reporting (the ‘file drawer problem’) (Rosenthal, 1979), and confirmation bias (Nuzzo, 2015). These questionable choices, combined with incomplete reporting of statistical methods and data (Nosek et al., 2015), contribute to the publication of false results that are unlikely to replicate in future experiments (Simmons et al., 2011).

Here we investigate the community that coalesces around a drug-gene interaction claim. We hypothesize that a decentralized community of largely independent, non-overlapping teams, which draws from a diverse pool of prior publications, using distinct methods under varying experimental conditions, is more likely to produce robust results. Conversely, we expect that a centralized community involving repeated collaborations and using a narrow range of methods, knowledge from prior publications and experimental conditions is likely to produce less robust results. Unfortunately, repeated collaboration (Hicks and Katz, 1996; Guimerà et al., 2005), growing teams (Wuchty et al., 2007), star scientists (Merton, 1968; Azoulay et al., 2014), expensive shared equipment, and common citations (Evans, 2008; Simkin and Roychowdhury, 2005; White et al., 2004) are defining characteristics of the biomedical research enterprise today (Hicks and Katz, 1996; Hand, 2010).

Prior simulations have suggested that independent labs are less prone to peer pressure than a densely connected network of scientists, in which misleading early results can propagate more easily (Zollman, 2007; Payette, 2012). Related research on the ‘wisdom of crowds’ (Lorenz et al., 2011) and the exploration-exploitation trade-off (Lazer and Friedman, 2007) also found densely connected networks to be inefficient, and suggested that networks of semi-isolated subgroups would lead to an improvement in collective performance (Fang et al., 2010). A more recent experiment demonstrates that decentralized networks, rather than independence, may most improve collective performance. In estimation tasks completed by networks of individuals, it was found that the dominance of central individuals in networks tended to bias the collective estimation process and decrease the average accuracy of group estimates (Becker et al., 2017).

A separate body of literature attributes robustness of scientific findings to diverse methods (Kaelin, 2017; Wimsatt, 2012) used to corroborate them or distinct theories used to motivate them. A classic example is Jean Perrin’s use of multiple experimental techniques and theories to precisely determine Avogadro’s number (Salmon, 1984). Nevertheless, there has been no comprehensive evaluation of the relationship between the way scientific communities are networked and the robustness and replicability of published findings. Moreover, when empirical data on scientific collaboration have been used (Guimerà et al., 2005), the outcomes of collective performance have typically been measured indirectly (e.g., via article or journal citations). Similarly, literature on research reliability has focused on methodological features rather than the way scientific communities are networked. Moreover, there has been relatively little research of this nature in the field of biomedical science.

Here we demonstrate a strategy for evaluating the replication likelihood for tens of thousands of drug-gene interaction claims. This strategy builds on the synergy of two advances. First, databases of empirical claims curated from the scientific literature in certain subject areas such as molecular toxicology and biochemistry (Davis et al., 2017) can be linked to databases of scientific articles such as MEDLINE and the Web of Science to systematically analyze features that characterize the provenance of a scientific claim (Evans and Foster, 2011) such as authors, affiliations, and the number of experiments for and against. Second, data from high-throughput experiments (Subramanian et al., 2017) performed by robots allow researchers to estimate the replication likelihood for many published claims. Here we report the results of analyses performed on claims identified by comparing the Comparative Toxicogenomics Database (CTD; Davis et al., 2017) and the LINCS L1000 experiment (Subramanian et al., 2017; see Figure 1 and Materials and methods).

Figure 1 with 2 supplements see all

Download asset Open asset

The CTD had recorded over 1.26 million drug-gene interaction (DGI) claims published in journals as of June 7, 2016, with each claim being a ‘triple’ of a drug, a gene and an interaction effect. We selected interaction effects in which a drug either ‘increases expression’ or ‘decreases expression’ of an mRNA in human tissues. This resulted in a sample of 239,713 DGI claims curated from 11,754 scientific articles. The LINCS L1000 experiment generated 1.3 million gene expression profiles from 42,080 perturbagens across a range of different cell lines, time points and dosage levels. Each profile consisted of a drug, a gene (mRNA) and a z-score. The LINCS L1000 experiment consolidated multiple expression profiles to generate a moderated z-score for each experimental condition, and we combined these into a single combined z-score for each drug and gene (mRNA). We matched these triples from the LINCS L1000 experiments to triples in the CTD, and found 51,292 drug-gene interactions at the intersection, corresponding to 60,159 supportive findings and 4253 opposing findings from the literature, annotated from 3363 scientific articles (see Supplementary file 1). We use this sample to estimate how the probability of claim replication depends on support in the literature, social independence, methodological independence, knowledge independence, scientist centralization, journal prominence, and experimental variability (see Materials and methods).

Our high-throughput replication strategy evaluates the replicability of a broad scientific claim rather than the reproducibility of any single experiment contributing to that claim. (The evaluation of an individual experiment requires the original experimental protocol to be repeated in the new experiment [Errington et al., 2014].) Nevertheless, collective agreement across many published findings provides evidence for a robust claim – that is, a claim that withstands changes in the technique used (Wimsatt, 2012; Nosek and Errington, 2017; Kaelin, 2017), the scientists doing the research and the experimental setting. Such claims form a solid foundation for further research and, potentially clinical trials based on the claim.

This paper repurposes high-throughput experiments to evaluate the replicability of tens of thousands of DGI claims from the biomedical literature. It provides evidence that research replicability is associated with the way scientific collaborations are networked, after adjusting for variability in high-throughput experiments and the methodology of published studies. Centralized, overlapping communities with shared methodologies and exposure to the same prior knowledge are supported by extensive collaboration (Hicks and Katz, 1996; Hand, 2010), ubiquitous communication technologies, a reward system that places a premium on productivity, and cumulative advantage processes that create central, star biomedical scientists (Azoulay et al., 2014). Our results indicate that such communities are associated with scientific findings of lower replicability. Decentralized, sparsely connected communities explore diverse research methodologies and prior knowledge (Figure 3—figure supplement 1) and are more likely to generate replicable findings.

Our work addresses widely shared concerns about the robustness of research results in biomedicine. Recent work (Nissen et al., 2016; McElreath and Smaldino, 2015) submitted that reliability issues uniformly impact scientific findings, afflicting even widely accepted ‘facts’. Nissen et al. reason that even if many studies converge on the same finding, this may not increase our confidence because the same systematic biases – publication bias in particular – that made the original finding false are also likely to affect subsequent findings, resulting in a canonization of ‘false facts’ (Nissen et al., 2016). Our analysis identifies conditions under which such an argument holds, namely for research claims studied by centralized, overlapping collaborations. In the absence of social independence, replicability is low and the likelihood of a claim being revised or withdrawn is virtually non-existent as authors and methods reinforce agreement with themselves. Both disagreement and replication increase when teams from a decentralized or sparsely connected community provide separate confirming evidence for a claim. Our findings allay science policy concerns over a universal ‘replication crisis’ and identify conditions – decentralized and novel collaborations – that facilitate collective convergence on replicable findings in biomedical science.

Our findings highlight the importance of science policies that promote decentralized and non-repeated collaborations. We acknowledge that certain ‘big science’ initiatives (Hand, 2010), such as the human genome project (International Human Genome Sequencing Consortium, 2001), involve large international consortiums that require a degree of repeated collaboration and centralization by design. It is also the case that the current organization of science incentivizes repeated collaborations (Guimerà et al., 2005; Lungeanu and Contractor, 2015; Hilton and Cooke, 2015), including centralized communities revolving around star scientists (Azoulay et al., 2014) or prestigious and well-endowed institutes. By reducing information and coordination costs, repeated collaborations are more productive than new ones (Hilton and Cooke, 2015). Consequently, projects that involve prior collaborators are more likely to be funded (Lungeanu et al., 2014) and successfully completed (Cummings and Kiesler, 2008). Such positive feedback, however, can lead to the lock-in of rigid collaborative clusters, which produce voluminous scientific output with diminished value for the wider biomedical field. Science policies supporting biomedicine should account for the trade-off between increased productivity and diminished reliability.

Our choice of repurposing the LINCS L1000 data to estimate the replication likelihood of published claims places importance on unfiltered high-throughput experiments for our results. We note, however, that our approach does not rely on the LINCS L1000 data being completely free from error. Rather, we argue that the LINCS L1000 data are: i) produced with methods representative of DGI research; ii) free from social, prior knowledge and narrow methodological dependencies associated with the sequential publication of findings by communities of researchers. In this way, our manuscript attempts to elucidate biases introduced by the social, cultural and methodological structure of science to noisy experimental data. From this perspective, LINCS L1000 experiments must remain unfiltered by the publication system and the expectations of any particular scientific community. In the current state of the biomedical literature, where most reported results are confirmatory, scalable approaches for identifying uncertain claims are in short supply. Experimental data such as the LINCS L1000 provides an informed approach to evaluate published claims.

This paper demonstrates an approach to evaluate the replicability of potentially vast numbers of published biomedical claims simultaneously. With the proliferation of high-throughput experimental platforms and improvements in cross-platform reproducibility (Haibe-Kains et al., 2013), the approach we report here could be further refined and extended to a continuous replication system (Goodman et al., 2016) that revises scientific results in light of new experimental evidence, increasing the credibility and robustness of biomedical knowledge.

The datasets generated or analysed during this study are included in the manuscript and supporting files and have been made available at OSF (http://dx.doi.org/10.17605/OSF.IO/XMVDA).

1. 1. Valentin Danchev
   2. Andrey Rzhetsky
   3. James A Evans

   (2019) Open Science Framework

   Centralized scientific communities are less likely to generate replicable results.

   https://doi.org/10.17605/OSF.IO/XMVDA

1. 1. The Broad Institute

   (2015) NCBI Gene Expression Omnibus

   ID GSE70138. L1000 Connectivity Map perturbational profiles from Broad Institute LINCS Center for Transcriptomics LINCS PHASE *II* (n=354,123; updated March 30, 2017).

   https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE70138
2. 1. MDI Biological Laboratory
   2. NC State University

   (2016) Comparative Toxicogenomics Database

   Comparative Toxicogenomics Database.

   http://ctdbase.org/downloads/

1. 1. Alberts B
   2. Cicerone RJ
   3. Fienberg SE
   4. Kamb A
   5. McNutt M
   6. Nerem RM
   7. Schekman R
   8. Shiffrin R
   9. Stodden V
   10. Suresh S
   11. Zuber MT
   12. Pope BK
   13. Jamieson KH

   (2015) Self-correction in science at work

   Science 348:1420–1422.

   https://doi.org/10.1126/science.aab3847

   • Google Scholar
2. 1. Azoulay P
   2. Stuart T
   3. Wang Y

   (2014) Matthew: Effect or fable?

   Management Science 60:92–109.

   https://doi.org/10.1287/mnsc.2013.1755

   • Google Scholar
3. Website

   1. Azoulay P
   2. Fons-Rosen C
   3. Zivin JSG

   (2015) Does science advance one funeral at a time?

   Accessed June 14, 2019.

   https://www.nber.org/papers/w21788
4. 1. Badham JM

   (2013) Measuring the shape of degree distributions

   Network Science 1:213–225.

   https://doi.org/10.1017/nws.2013.10

   • Google Scholar
5. 1. Bandrowski A
   2. Brinkman R
   3. Brochhausen M
   4. Brush MH
   5. Bug B
   6. Chibucos MC
   7. Clancy K
   8. Courtot M
   9. Derom D
   10. Dumontier M
   11. Fan L
   12. Fostel J
   13. Fragoso G
   14. Gibson F
   15. Gonzalez-Beltran A
   16. Haendel MA
   17. He Y
   18. Heiskanen M
   19. Hernandez-Boussard T
   20. Jensen M
   21. Lin Y
   22. Lister AL
   23. Lord P
   24. Malone J
   25. Manduchi E
   26. McGee M
   27. Morrison N
   28. Overton JA
   29. Parkinson H
   30. Peters B
   31. Rocca-Serra P
   32. Ruttenberg A
   33. Sansone SA
   34. Scheuermann RH
   35. Schober D
   36. Smith B
   37. Soldatova LN
   38. Stoeckert CJ
   39. Taylor CF
   40. Torniai C
   41. Turner JA
   42. Vita R
   43. Whetzel PL
   44. Zheng J

   (2016) The ontology for biomedical investigations

   PLOS ONE 11:e0154556.

   https://doi.org/10.1371/journal.pone.0154556

   • PubMed
   • Google Scholar
6. 1. Becker J
   2. Brackbill D
   3. Centola D

   (2017)

   Network dynamics of social influence in the wisdom of crowds

   PNAS 114:E5070–E5076.

   • Google Scholar
7. 1. Begley CG
   2. Ellis LM

   (2012) Drug development: Raise standards for preclinical cancer research

   Nature 483:531–533.

   https://doi.org/10.1038/483531a

   • PubMed
   • Google Scholar
8. 1. Benjamini Y
   2. Hochberg Y

   (1995) Controlling the false discovery rate: A practical and powerful approach to multiple testing

   Journal of the Royal Statistical Society: Series B 57:289–300.

   https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

   • Google Scholar
9. 1. Bergstrom CT
   2. West JD
   3. Wiseman MA

   (2008) The eigenfactor metrics

   Journal of Neuroscience 28:11433–11434.

   https://doi.org/10.1523/JNEUROSCI.0003-08.2008

   • PubMed
   • Google Scholar
10. Conference

    1. Cummings JN
    2. Kiesler S

    (2008) Who collaborates successfully? Prior experience reduces collaboration barriers in distributed interdisciplinary research

    Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work.

    https://doi.org/10.1145/1460563.1460633

    • Google Scholar
11. Book

    1. Davidson-Pilon C

    (2015)

    Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference

    New York: Pearson Education.

    • Google Scholar
12. 1. Davis AP
    2. Grondin CJ
    3. Johnson RJ
    4. Sciaky D
    5. King BL
    6. McMorran R
    7. Wiegers J
    8. Wiegers TC
    9. Mattingly CJ

    (2017) The Comparative Toxicogenomics Database: update 2017

    Nucleic Acids Research 45:D972–D978.

    https://doi.org/10.1093/nar/gkw838

    • PubMed
    • Google Scholar
13. 1. Dumas-Mallet E
    2. Button KS
    3. Boraud T
    4. Gonon F
    5. Munafò MR

    (2017) Low statistical power in biomedical science: A review of three human research domains

    Royal Society Open Science 4:160254.

    https://doi.org/10.1098/rsos.160254

    • PubMed
    • Google Scholar
14. 1. Errington TM
    2. Iorns E
    3. Gunn W
    4. Tan FE
    5. Lomax J
    6. Nosek BA

    (2014) An open investigation of the reproducibility of cancer biology research

    eLife 3:e04333.

    https://doi.org/10.7554/eLife.04333

    • PubMed
    • Google Scholar
15. 1. Evans JA

    (2008) Electronic publication and the narrowing of science and scholarship

    Science 321:395–399.

    https://doi.org/10.1126/science.1150473

    • PubMed
    • Google Scholar
16. 1. Evans JA
    2. Foster JG

    (2011) Metaknowledge

    Science 331:721–725.

    https://doi.org/10.1126/science.1201765

    • PubMed
    • Google Scholar
17. 1. Fang C
    2. Lee J
    3. Schilling MA

    (2010) Balancing exploration and exploitation through structural design: The isolation of subgroups and organizational learning

    Organization Science 21:625–642.

    https://doi.org/10.1287/orsc.1090.0468

    • Google Scholar
18. 1. Fox J

    (2003) Effect displays in R for generalised Linear models

    Journal of Statistical Software 8:1–27.

    https://doi.org/10.18637/jss.v008.i15

    • Google Scholar
19. 1. Freeman LC

    (1978) Centrality in social networks conceptual clarification

    Social Networks 1:215–239.

    https://doi.org/10.1016/0378-8733(78)90021-7

    • Google Scholar
20. Book

    1. Gelman A
    2. Carlin JB
    3. Stern HS
    4. Dunson DB
    5. Vehtari A
    6. Rubin DB

    (2013)

    Bayesian Data Analysis (Third Edition)

    Chapman and Hall/CRC.

    • Google Scholar
21. Website

    1. Gonzalez-Beltran A
    2. Rocca-Serra P
    3. Burke O
    4. Sansone S-A
    5. Ontology S

    (2016) STATistics ontology

    Accessed November 7, 2017.

    http://bioportal.bioontology.org/ontologies/STATO
22. 1. Goodman SN
    2. Fanelli D
    3. Ioannidis JP

    (2016) What does research reproducibility mean?

    Science Translational Medicine 8:341ps12.

    https://doi.org/10.1126/scitranslmed.aaf5027

    • PubMed
    • Google Scholar
23. 1. Guimerà R
    2. Uzzi B
    3. Spiro J
    4. Amaral LA

    (2005) Team assembly mechanisms determine collaboration network structure and team performance

    Science 308:697–702.

    https://doi.org/10.1126/science.1106340

    • PubMed
    • Google Scholar
24. Conference

    1. Hagberg A
    2. Swart P
    3. Chult D S

    (2008) Exploring network structure, dynamics, and function using NetworkX

    Proceedings of the 7th Python in Science Conference (SciPy 2008).

    https://www.osti.gov/biblio/960616

    • Google Scholar
25. 1. Haibe-Kains B
    2. El-Hachem N
    3. Birkbak NJ
    4. Jin AC
    5. Beck AH
    6. Aerts HJWL
    7. Quackenbush J

    (2013) Inconsistency in large pharmacogenomic studies

    Nature 504:389–393.

    https://doi.org/10.1038/nature12831

    • Google Scholar
26. 1. Hand E

    (2010) 'Big science' spurs collaborative trend

    Nature 463:282–283.

    https://doi.org/10.1038/463282a

    • PubMed
    • Google Scholar
27. 1. Head ML
    2. Holman L
    3. Lanfear R
    4. Kahn AT
    5. Jennions MD

    (2015) The extent and consequences of p-hacking in science

    PLOS Biology 13:e1002106.

    https://doi.org/10.1371/journal.pbio.1002106

    • PubMed
    • Google Scholar
28. 1. Hicks D
    2. Katz JS

    (1996) Science policy for a highly collaborative science system

    Science and Public Policy 23:39–44.

    https://doi.org/10.1093/spp/23.1.39

    • Google Scholar
29. Book

    1. Hilton ML
    2. Cooke NJ

    (2015) Enhancing the Effectiveness of Team Science

    Washington: National Academies Press.

    https://doi.org/10.17226/19007

    • Google Scholar
30. 1. Himmelstein DS
    2. Lizee A
    3. Hessler C
    4. Brueggeman L
    5. Chen SL
    6. Hadley D
    7. Green A
    8. Khankhanian P
    9. Baranzini SE

    (2017) Systematic integration of biomedical knowledge prioritizes drugs for repurposing

    eLife 6:e26726.

    https://doi.org/10.7554/eLife.26726

    • PubMed
    • Google Scholar
31. 1. International Human Genome Sequencing Consortium

    (2001) Initial sequencing and analysis of the human genome

    Nature 409:860–921.

    https://doi.org/10.1038/35057062

    • PubMed
    • Google Scholar
32. 1. Ioannidis JP

    (2005) Why most published research findings are false

    PLOS Medicine 2:e124.

    https://doi.org/10.1371/journal.pmed.0020124

    • PubMed
    • Google Scholar
33. 1. Kaelin WG

    (2017) Common pitfalls in preclinical cancer target validation

    Nature Reviews Cancer 17:441–450.

    https://doi.org/10.1038/nrc.2017.32

    • Google Scholar
34. 1. Kessler MM

    (1963) Bibliographic coupling between scientific papers

    American Documentation 14:10–25.

    https://doi.org/10.1002/asi.5090140103

    • Google Scholar
35. 1. Kivela M
    2. Arenas A
    3. Barthelemy M
    4. Gleeson JP
    5. Moreno Y
    6. Porter MA

    (2014) Multilayer networks

    Journal of Complex Networks 2:203–271.

    https://doi.org/10.1093/comnet/cnu016

    • Google Scholar
36. Software

    1. Kivelä M

    (2017) Multilayer Networks Library for Python (Pymnet), version fdd96ee

    Bitbucket.

    http://bitbucket.org/bolozna/multilayer-networks-library
37. Book

    1. Kruschke J

    (2014)

    Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan

    Academic Press.

    • Google Scholar
38. Conference

    1. Kunegis J
    2. Preusse J

    (2012) Fairness on the web: Alternatives to the power law

    Proceedings of the 4th Annual ACM Web Science Conference. pp. 175–184.

    https://doi.org/10.1145/2380718.2380741

    • Google Scholar
39. 1. Lazer D
    2. Friedman A

    (2007) The network structure of exploration and exploitation

    Administrative Science Quarterly 52:667–694.

    https://doi.org/10.2189/asqu.52.4.667

    • Google Scholar
40. 1. Lorenz J
    2. Rauhut H
    3. Schweitzer F
    4. Helbing D

    (2011) How social influence can undermine the wisdom of crowd effect

    PNAS 108:9020–9025.

    https://doi.org/10.1073/pnas.1008636108

    • PubMed
    • Google Scholar
41. Software

    1. Lüdecke D

    (2019)

    sjPlot: Data Visualization for Statistics in Social Science, version 2.3.3

    R Package Version, https://CRAN.R-project.org/package=sjPlot.
42. 1. Lungeanu A
    2. Huang Y
    3. Contractor NS

    (2014) Understanding the assembly of interdisciplinary teams and its impact on performance

    Journal of Informetrics 8:59–70.

    https://doi.org/10.1016/j.joi.2013.10.006

    • PubMed
    • Google Scholar
43. 1. Lungeanu A
    2. Contractor NS

    (2015) The effects of diversity and network ties on innovations: The emergence of a new scientific field

    The American Behavioral Scientist 59:548–564.

    https://doi.org/10.1177/0002764214556804

    • PubMed
    • Google Scholar
44. 1. McElreath R
    2. Smaldino PE

    (2015) Replication, communication, and the population dynamics of scientific discovery

    PLOS ONE 10:e0136088.

    https://doi.org/10.1371/journal.pone.0136088

    • PubMed
    • Google Scholar
45. 1. Merton RK

    (1968) The Matthew Effect in Science

    Science 159:56–63.

    https://doi.org/10.1126/science.159.3810.56

    • Google Scholar
46. 1. Nissen SB
    2. Magidson T
    3. Gross K
    4. Bergstrom CT

    (2016) Publication bias and the canonization of false facts

    eLife 5:e21451.

    https://doi.org/10.7554/eLife.21451

    • PubMed
    • Google Scholar
47. 1. Nosek BA
    2. Alter G
    3. Banks GC
    4. Borsboom D
    5. Bowman SD
    6. Breckler SJ
    7. Buck S
    8. Chambers CD
    9. Chin G
    10. Christensen G
    11. Contestabile M
    12. Dafoe A
    13. Eich E
    14. Freese J
    15. Glennerster R
    16. Goroff D
    17. Green DP
    18. Hesse B
    19. Humphreys M
    20. Ishiyama J
    21. Karlan D
    22. Kraut A
    23. Lupia A
    24. Mabry P
    25. Madon T
    26. Malhotra N
    27. Mayo-Wilson E
    28. McNutt M
    29. Miguel E
    30. Paluck EL
    31. Simonsohn U
    32. Soderberg C
    33. Spellman BA
    34. Turitto J
    35. VandenBos G
    36. Vazire S
    37. Wagenmakers EJ
    38. Wilson R
    39. Yarkoni T

    (2015) Promoting an open research culture

    Science 348:1422–1425.

    https://doi.org/10.1126/science.aab2374

    • Google Scholar
48. 1. Nosek BA
    2. Errington TM

    (2017) Making sense of replications

    eLife 6:e23383.

    https://doi.org/10.7554/eLife.23383

    • PubMed
    • Google Scholar
49. 1. Nuzzo R

    (2015) Fooling ourselves

    Nature 526:182–185.

    https://doi.org/10.1038/526182a

    • Google Scholar
50. Book

    1. Payette N

    (2012) Agent-based models of science

    In: Scharnhorst A, Börner K, van den Besselaar P, editors. Models of Science Dynamics: Encounters Between Complexity Theory and Information Sciences. Heidelberg: Springer. pp. 127–157.

    https://doi.org/10.1007/978-3-642-23068-4

    • Google Scholar
51. Software

    1. Peirson B
    2. Erick R

    (2017) Tethne

    GitHub.

    http://diging.github.io/tethne
52. 1. Prinz F
    2. Schlange T
    3. Asadullah K

    (2011) Believe it or not: How much can we rely on published data on potential drug targets?

    Nature Reviews Drug Discovery 10:712.

    https://doi.org/10.1038/nrd3439-c1

    • PubMed
    • Google Scholar
53. 1. Rosenthal R

    (1979) The file drawer problem and tolerance for null results

    Psychological Bulletin 86:638–641.

    https://doi.org/10.1037/0033-2909.86.3.638

    • Google Scholar
54. Book

    1. Salmon W

    (1984)

    Scientific Explanation and the Causal Structure of the World

    Princeton: Princeton University Press.

    • Google Scholar
55. 1. Simkin MV
    2. Roychowdhury VP

    (2005) Stochastic modeling of citation slips

    Scientometrics 62:367–384.

    https://doi.org/10.1007/s11192-005-0028-2

    • Google Scholar
56. 1. Simmons JP
    2. Nelson LD
    3. Simonsohn U

    (2011) False-positive psychology

    Psychological Science 22:1359–1366.

    https://doi.org/10.1177/0956797611417632

    • Google Scholar
57. 1. Simonsohn U
    2. Nelson LD
    3. Simmons JP

    (2014) P-curve: a key to the file-drawer

    Journal of Experimental Psychology: General 143:534–547.

    https://doi.org/10.1037/a0033242

    • Google Scholar
58. 1. Subramanian A
    2. Narayan R
    3. Corsello SM
    4. Peck DD
    5. Natoli TE
    6. Lu X
    7. Gould J
    8. Davis JF
    9. Tubelli AA
    10. Asiedu JK
    11. Lahr DL
    12. Hirschman JE
    13. Liu Z
    14. Donahue M
    15. Julian B
    16. Khan M
    17. Wadden D
    18. Smith IC
    19. Lam D
    20. Liberzon A
    21. Toder C
    22. Bagul M
    23. Orzechowski M
    24. Enache OM
    25. Piccioni F
    26. Johnson SA
    27. Lyons NJ
    28. Berger AH
    29. Shamji AF
    30. Brooks AN
    31. Vrcic A
    32. Flynn C
    33. Rosains J
    34. Takeda DY
    35. Hu R
    36. Davison D
    37. Lamb J
    38. Ardlie K
    39. Hogstrom L
    40. Greenside P
    41. Gray NS
    42. Clemons PA
    43. Silver S
    44. Wu X
    45. Zhao WN
    46. Read-Button W
    47. Wu X
    48. Haggarty SJ
    49. Ronco LV
    50. Boehm JS
    51. Schreiber SL
    52. Doench JG
    53. Bittker JA
    54. Root DE
    55. Wong B
    56. Golub TR

    (2017) A next generation connectivity map: L1000 platform and the first 1,000,000 profiles

    Cell 171:1437–1452.

    https://doi.org/10.1016/j.cell.2017.10.049

    • PubMed
    • Google Scholar
59. 1. Van Bavel JJ
    2. Mende-Siedlecki P
    3. Brady WJ
    4. Reinero DA

    (2016) Contextual sensitivity in scientific reproducibility

    PNAS 113:6454–6459.

    https://doi.org/10.1073/pnas.1521897113

    • PubMed
    • Google Scholar
60. Book

    1. Wasserman S
    2. Faust K

    (1994)

    Social Network Analysis: Methods and Applications

    Cambridge: Cambridge University Press.

    • Google Scholar
61. 1. White HD
    2. Wellman B
    3. Nazer N

    (2004) Does citation reflect social structure? Longitudinal evidence from the "Globenet" interdisciplinary research group

    Journal of the American Society for Information Science and Technology 55:111–126.

    https://doi.org/10.1002/asi.10369

    • Google Scholar
62. 1. Whitlock MC

    (2005) Combining probability from independent tests: the weighted Z-method is superior to Fisher's approach

    Journal of Evolutionary Biology 18:1368–1373.

    https://doi.org/10.1111/j.1420-9101.2005.00917.x

    • PubMed
    • Google Scholar
63. Book

    1. Wickham H

    (2016) Ggplot2: Elegant Graphics for Data Analysis

    New York: Springer.

    https://doi.org/10.1007/978-0-387-98141-3

    • Google Scholar
64. Book

    1. Wimsatt WC

    (2012) Reliability, and Overdetermination (1981)

    In: Soler L, Trizio E, Nickles T, Wimsatt W, editors. Characterizing the Robustness of Science: After the Practice Turn in Philosophy of Science. Dordrecht: Springer. pp. 89–104.

    https://doi.org/10.1007/978-94-007-2759-5

    • Google Scholar
65. 1. Wuchty S
    2. Jones BF
    3. Uzzi B

    (2007) The increasing dominance of teams in production of knowledge

    Science 316:1036–1039.

    https://doi.org/10.1126/science.1136099

    • PubMed
    • Google Scholar
66. 1. Yildirim MA
    2. Goh KI
    3. Cusick ME
    4. Barabási AL
    5. Vidal M

    (2007) Drug-target network

    Nature Biotechnology 25:1119–1126.

    https://doi.org/10.1038/nbt1338

    • PubMed
    • Google Scholar
67. 1. Zollman KJS

    (2007) The communication structure of epistemic communities

    Philosophy of Science 74:574–587.

    https://doi.org/10.1086/525605

    • Google Scholar

Thank you for submitting your article "Centralized Scientific Communities Less Likely to Generate Replicable Results" to eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by Peter Rodgers, the eLife Features Editor. The following individual involved in review of your submission have agreed to reveal his identity: Luis AN Amaral (Reviewer #1).

The reviewers have discussed the reviews with one another and the eLife Features Editor has drafted this decision to help you prepare a revised submission.

Summary:

Danchev, Rzhetshy, and Evans investigate the replicability and reliability of published findings by comparing published findings with the high throughput results from the NIH LINCS L1000 program. This is a clever approach. The paper is overall well written, and the analysis is thorough and complete. All in all, the authors provide important results that may help guide the development of more reliable science.

Essential revisions:

1) eLife papers use the Introduction/Results/Discussion/Materials and methods format, so you will need to reorder these sections in your manuscript. Also, eLife papers do not include supplementary information: rather, figures in the supplementary information need to become supplements to the figures in the main text (called Figure 1—figure supplement 1 etc.), and the text in the supplementary information needs to be moved to the main text (usually to the methods section).

2) The authors' analysis seems to rely on assuming that the LINCS L1000 data is true. A brief discussion about the importance of this assumption and perhaps what would happen were it violated would be helpful.

3) While this study is very thorough as it stands, I believe that the authors have the opportunity to answer another question that could increase the impact of the paper. The authors find that most DGI claims are only reported in a single study. Why is this? I can imagine two scenarios: i) Gene is not thought to be important, so there is no incentive for further study, or ii) Gene is important but claim is by star scientist and attempts at publishing contradictory claim are stymied. It would be interesting to investigate whether the data support either of these scenarios (or another scenario). If the authors feel that such an analysis is beyond the scope of the present paper, they should at least comment on this issue.

4) The reader needs more detail about bootstrapping (what is being bootstrapped? What is the typical sample size? What is the distribution of perturbations for each gene?). Please either add these details to the subsections “High-throughput Claim Replication Strategy”, last paragraph, “Experimental Generalizability and Variability” and “Measuring Relative Replication Increase”, or include references to the relevant parts of the methods section.

5) Please clarify how replication is defined. What if LINCS data is not consistent? Or if the other data/papers are not consistent (e.g. includes + and – results)?

6) In the first paragraph of the Discussion, the authors imply a causal relationship between centralized communities and reliability of scientific findings. I recommend that this (and any other similar statements) be modified so as to not imply causation. It could be that some third confounding factor leads to lower reliability. For instance, this sentence could be written: "Our results indicate that such communities are associated with lower reliability of scientific findings." If the authors believe this analysis can discern causality, please describe why.

7) Figure 1. Panel A would make more sense to me as a scatter plot of number of supporting claims versus number of opposing claims. Panel C would be easier to read if the survival function was plotted instead of a frequency plot. In their current forms Panels B and D containing relatively little information: please consider deleting these panels and including the relevant data in the figure legend. Also, regarding panel D: it would be more interesting to plot the fractions that are significant as the p-value threshold for significance is changed.

8) Figure 2. It is not clear to me what subset of data is being considered in Panel B. It appears to be different from the subset in Panel C since the number of contradictory claims is different for the two panels (43 vs. 77). I would name the different subsets of data, provide a table with their definition, and use the names in the text so as to help the reader. Also, please explain why there are two random groups in Panel C. Also, 'contradictory' seems to me not a good name since a potentially large majority of claims does not support the existence of the interaction: 'not supported' would appear to be a better designation.

9) In figures including p-values (e.g. Figure 2D and 2E), it would be helpful to have p-values rather than stating "n.s." Additionally, clarity on the p-values and how they are calculated in Figure 2D and 2E would be helpful

10) Figure 3. Please provide a y-axis label for Panel B. This panel is also difficult to read so please consider having the lines end at the middle of the bottom edge of the rectangle/square, rather than inside the rectangle/square.

11) Figure 4A claims to show the OR for replication in LINCS L1000 data but the figure legend claims that something else is being plotted. Moreover, several studies have showed that OR are usually misinterpreted and that use of relative risk (RR, which can be derived from the OR) is more informative of the actual impact of a factor. I recommend also plotting the RR.

12) Figure 4—figure supplement 3C: the 95% confidence intervals for the top 2 rows do not appear to contain any of the parameters in the table. More clarification on what they represent is needed.

Essential revisions:

1) eLife papers use the Introduction/Results/Discussion/Materials and methods format, so you will need to reorder these sections in your manuscript. Also, eLife papers do not include supplementary information: rather, figures in the supplementary information need to become supplements to the figures in the main text (called Figure 1—figure supplement 1 etc.), and the text in the supplementary information needs to be moved to the main text (usually to the Materials and methods section).

We have reordered the manuscript accordingly and incorporated supplementary figures and information in the main text and in Appendices.

2) The authors' analysis seems to rely on assuming that the LINCS L1000 data is true. A brief discussion about the importance of this assumption and perhaps what would happen were it violated would be helpful.

While we evaluate the reliability and biases in published DGI claims against the LINCS L1000 experiments, our approach does not rely on the LINCS L1000 data being completely correct or true. Rather, we argue that the LINCS L1000 data are (1) produced with methods representative of research in this area, and (2) free from the social, prior knowledge and selective technical dependencies associated with the sequential publication of findings by communities of researchers. In this way, our manuscript attempts to elucidate biases introduced by the social, cultural and technical structure of science to noisy experimental data. In this framework, LINCS L1000 experiments must remain unfiltered by the publication system and expectations of any particular scientific community. To clarify this, we have these details to the current Discussion section.

3) While this study is very thorough as it stands, I believe that the authors have the opportunity to answer another question that could increase the impact of the paper. The authors find that most DGI claims are only reported in a single study. Why is this? I can imagine two scenarios: i) Gene is not thought to be important, so there is no incentive for further study, or ii) Gene is important but claim is by star scientist and attempts at publishing contradictory claim are stymied. It would be interesting to investigate whether the data support either of these scenarios (or another scenario). If the authors feel that such an analysis is beyond the scope of the present paper, they should at least comment on this issue.

In response to this thoughtful suggestion, we have explored reasons behind the tendency for most DGI claims to be reported in a single paper. Because our research design is tailored towards the examination of network dependencies underlying biomedical claims reported in multiple published studies, our analysis of single-studied claims is limited and of an exploratory nature.

First, a major factor associated with single-studied claims in our corpus appears to be their relatively low replication likelihood, indicating that many could have been obtained by chance, minimizing the possibility of further reporting. Consider our finding (Figures 2D and 2E) that DGI claims studied in a single study (“Low” support) are less likely replicated in LINCS L1000 than DGI claims reported in multiple studies (“Very high” and “High” support). Further, focusing only on the first published study among multiple-studied claims with “Very high” and “High” support, we find that those also have higher observed replication rate of 0.7196 than single-studied claims in our corpus (“Low” support), which yield an observed replication rate of 0.6010. (Because opposing findings are rare, the replication rates for the first published finding and for all published findings about multiple-studied claims (“Very high” and “High” support) are very similar: 0.7196 and 0.7232, respectively.) In the event that a single-studied claim was tested again, researchers would be less likely to obtain positive results compared to a multiple-studied claim, placing the finding in the “file drawer”. We relate this to the expectation of novel results in the current publication system that drives the introduction of novel but not necessarily reliable claims, less likely to be confirmed in subsequent studies.

Second, among all claims in our corpus, we find no significant correlation between the number of findings published about a claim and the importance of the publication journal measured using the eigenfactor score [1] (Pearson corr. coef. = -0.0026, P = 0.5614, n = 50,346; journal eigenfactor scores are available for 50,346 of 51,292 claims).

Third, we found that DGI claims contributed from one or more researchers who published five or more articles in the whole corpus are much more likely to be reported in multiple studies (17.7%) than claims contributed only by researchers who published less than five articles in our corpus (4.1%). Recall in the manuscript that we found centralization of scientific communities correlates with overlapping methods, see Figure 3—figure supplement 1. Consistent with that finding, multiple studies from a related set of researchers tended to cluster on a few claims. Conversely, authors that publish a small number of articles in our corpus rarely add to the network of existing claims, but tend to introduce isolated, “singleton” claims.

Fourth, we examined whether centralized sources of a claim would discourage outside researchers from reporting findings about that claim, as suggested by the reviewer. We add the following to the manuscript: “Social dependencies could impact the composition of researchers studying a claim by discouraging researchers outside the dense communities of researchers who initially lay claim to it from pursuing or reporting their findings [2]. Focusing on “Very high” and “High” support claims with some degree of centralization (n = 295), we found that claims that were both originally reported and subsequently repeated and confirmed in by papers with overlapping authors (n = 117) ended up as much more centralized communities (mean C = 0.6164; 95CI, 0.5925 to 0.6402) compared to claims reported and subsequently repeated and confirmed by papers with independent authors (n = 178; mean C = 0.4004; 95CI, 0.3765 to 0.4242; two-way ANOVA test with unbalanced design.) This exploratory result suggests that if claims exhibit social dependencies very early in their development, they likely receive limited attention and reportage from researchers outside the overlapping communities that originated them.” This is broadly consistent with the finding by Azoulay and colleagues, cited within this section, that demonstrates how closed, high-status communities in biomedicine not only discourage, but also block participation from others.

We have added elements from each of these demonstrations into the manuscript.

4) The reader needs more detail about bootstrapping (what is being bootstrapped? What is the typical sample size? What is the distribution of perturbations for each gene?). Please either add these details to the subsections “High-throughput Claim Replication Strategy”, last paragraph, “Experimental Generalizability and Variability” and “Measuring Relative Replication Increase”, or include references to the relevant parts of the Materials and methods section.

We agree and have added details about the bootstrapping procedure and substantially revised text at the specified lines. We provide one example for reference:

“We bootstrapped (10,000 iterations) each sample of moderated z-scores per DGI to estimate confidence intervals. The bootstrapped samples vary across DGIs as a function of the number of cell lines, dosages, and durations under which the DGI was tested (the mean and median of moderated z-scores per DGI in our corpus are 143.8 and 49, respectively).”

5) Please clarify how replication is defined. What if LINCS data is not consistent? Or if the other data/papers are not consistent (e.g. includes + and – results)?

Our definition of claim replication triangulates the following components. First, we consider the amount of support a claim received from the literature and categorized claims into five classes on the basis of their respective amount of support. In the literature, if both positive and negative results are reported about a claim, the claim would be categorized as “Low” support or “Contradictory”, depending on the distribution of positive and negative findings. In LINCS L1000, inconsistent interactions would not, by definition, generalize across conditions, and, on this basis, will not be included in the central analysis. Then we consider if DGIs in the LINCS L1000 data match the direction of the effect in the literature. We define replication as a binary variableR∈{0,1}, where R = 1 indicates that LINCS L1000 matches the direction (increase or decrease) of the effect in literature, and R = 0 otherwise. In summary, to account for cases in which the LINCS L1000 and/or the literature are inconsistent, our claim replication strategy keeps track of (1) the amount of support in the literature and (2) the generalizability of interactions in LINCS L1000 when replicability of published DGI claims is evaluated against LINCS L1000 high-throughput experiments.

6) In the first paragraph of the Discussion, the authors imply a causal relationship between centralized communities and reliability of scientific findings. I recommend that this (and any other similar statements) be modified so as to not imply causation. It could be that some third confounding factor leads to lower reliability. For instance, this sentence could be written: "Our results indicate that such communities are associated with lower reliability of scientific findings." If the authors believe this analysis can discern causality, please describe why.

We agree with the reviewers’ concern and we rewrote this and similar statements to reflect the fact that our research design identifies associations rather than causal relationships. In addition to rewriting the above sentence as suggested by the reviewer, we rewrote the sentence which now reads: “Decentralized, sparsely connected communities explore diverse research methodologies and prior knowledge (Figure 3—figure supplement 1), and are more likely to generate replicable findings.” We use this association to propose a causal hypothesis, but we do not claim that we have causally identified a finding.

7) Figure 1. Panel A would make more sense to me as a scatter plot of number of supporting claims versus number of opposing claims. Panel C would be easier to read if the survival function was plotted instead of a frequency plot. In their current forms Panels B and D containing relatively little information: please consider deleting these panels and including the relevant data in the figure legend. Also, regarding panel D: it would be more interesting to plot the fractions that are significant as the p-value threshold for significance is changed.

We implemented most of these suggestions. As recommended, we plotted the survival probability in Panel C. In addition, we removed Panels B and D. Panel B is currently included as an inset figure in Panel A. The information conveyed in Panel D is currently plotted in Panel B.

We considered plotting the number of supporting findings versus the number of opposing findings but because the majority of findings in the literature are supporting, visualizing the two quantities poses a challenge.

8) Figure 2. It is not clear to me what subset of data is being considered in Panel B. It appears to be different from the subset in Panel C since the number of contradictory claims is different for the two panels (43 vs. 77). I would name the different subsets of data, provide a table with their definition, and use the names in the text so as to help the reader. Also, please explain why there are two random groups in Panel C. Also, 'contradictory' seems to me not a good name since a potentially large majority of claims does not support the existence of the interaction: 'not supported' would appear to be a better designation.

We regret that this was unclear. We have now clarified the two subsets of data we used in Figure 2B and C, as suggested. We have also included a table that clarifies the subsets of data as well as the number of claims pertaining to each subset. With regard to the two random groups in Panel C, they correspond to claims that are found to be significant and non-significant in LINCS L1000, respectively. We agree with the reviewer that the notion “contradictory” might not convey the actual distribution of support for those claims. We have adopted the suggested name “not supported” for those claims.

9) In figures including p-values (e.g. Figure 2D and 2E), it would be helpful to have p-values rather than stating "n.s." Additionally, clarity on the p-values and how they are calculated in Figure 2D and 2E would be helpful

We have added the actual p-values in the two figures and clarified the approach we used to calculate them.

“We bootstrapped 100,000 times the RRI distributions for each class of claims and calculated the p-value as the proportion of iterations in which the difference between two claim classes’ RRI distributions contains 0.”

10) Figure 3. Please provide a y-axis label for Panel B. This panel is also difficult to read so please consider having the lines end at the middle of the bottom edge of the rectangle/square, rather than inside the rectangle/square.

We thank for those helpful suggestions. We have added a y-axis and improved the readability for Panel B in Figure 3, as suggested.

11) Figure 4A claims to show the OR for replication in LINCS L1000 data but the figure legend claims that something else is being plotted. Moreover, several studies have showed that OR are usually misinterpreted and that use of relative risk (RR, which can be derived from the OR) is more informative of the actual impact of a factor. I recommend also plotting the RR.

We have clarified in the caption of Figure 4A that we show odds ratios derived from logistic regression models [with replication as a binary response variable and predictors modeled independently (dots) and simultaneously (triangles)]. Specifically, we fitted a set of generalized linear models using the glm() function in R, specifying binomial distribution and the canonical logit link function. We then took the exponent of the coefficients (log odds) to obtain the odds ratios we show in Figure 4A. As suggested by the reviewer, we have also derived the relative risk (RR) from our logistic models (see Figure 4A) using the following method [3] and the sjstats [4] package for R. (We also considered alternative specifications to directly estimate relative risk using poisson / quasipoisson regression, typically used for count outcomes, but a reliable poisson / quasipoisson regression for binary outcomes would involve additional assumptions and modifications [5].). We integrated in Figure 4—figure supplement 1 the plot of RR of claim replication with plots of predicted probabilities (PP) of claim replication derived from the independent logistic models in Figure 4A.

12) Figure 4—figure supplement 3C: the 95% confidence intervals for the top 2 rows do not appear to contain any of the parameters in the table. More clarification on what they represent is needed.

Thank you for pointing out this problem. We have identified the source of the misalignment. While the coefficients (and the 95% confidence intervals) were correct, the exponentiated values did not refer to the correct coefficients. We have exponentiated the correct coefficients and included the resulting values in the table.

References:

1) Bergstrom CT, West JD, Wiseman MA. The Eigenfactor Metrics. The Journal of Neuroscience 2008;28:11433.

82) Azoulay P, Fons-Rosen C, Zivin JSG. Does Science Advance One Funeral at a Time? National Bureau of Economic Research Working Paper Series 2015;No. 21788.

3) Zhang J, Yu KF. What's the Relative Risk?A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes. JAMA 1998;280:1690-1.

4) Lüdecke D. sjstats: Statistical Functions for Regression Models. 2019; R package version 0.17.5, doi: 10.5281/zenodo.1284472, https://CRAN.R-project.org/package= sjstats..

5) Zou G. A Modified Poisson Regression Approach to Prospective Studies with Binary Data. American Journal of Epidemiology 2004;159:702-6.

Author details

1. Valentin Danchev

   Valentin Danchev is in the Department of Sociology, University of Chicago, Chicago, and the Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, United States

   Contribution

   Conceptualization, Data curation, Investigation, Visualization, Methodology, Writing—original draft, Writing—review and editing

   For correspondence

   vdanchev@stanford.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-7563-0168

2. Andrey Rzhetsky

   Andrey Rzhetsky is in the Departments of Medicine and Human Genetics, and the Institute for Genomic and Systems Biology, University of Chicago, Chicago, United States

   Contribution

   Conceptualization, Funding acquisition, Project administration, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-6959-7405

3. James A Evans

   James A Evans is in the Department of Sociology, University of Chicago, Chicago, and the Santa Fe Institute, Santa Fe, United States

   Contribution

   Conceptualization, Supervision, Funding acquisition, Investigation, Writing—original draft, Writing—review and editing

   For correspondence

   jevans@uchicago.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-9838-0707

• 4,036

  views

• 396

  downloads

• 17

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.