Integrating data and knowledge is a formidable challenge in biomedical research. Although new scientific findings are being discovered at a rapid pace, a large proportion of that knowledge is either locked in data silos (where integration is hindered by differing nomenclature, data models, and licensing terms; Wilkinson et al., 2016) or locked away in free-text. The lack of an integrated and structured version of biomedical knowledge hinders efficient querying or mining of that information, thus preventing the full utilization of our accumulated scientific knowledge.

Recently, there has been a growing emphasis within the scientific community to ensure all scientific data are FAIR – Findable, Accessible, Interoperable, and Reusable – and there is a growing consensus around a concrete set of principles to ensure FAIRness (Wilkinson et al., 2019; Wilkinson et al., 2016). Widespread implementation of these principles would greatly advance efforts by the open-data community to build a rich and heterogeneous network of scientific knowledge. That knowledge network could, in turn, be the foundation for many computational tools, applications and analyses.

Most data- and knowledge-integration initiatives fall on either end of a spectrum. At one end, centralized efforts seek to bring multiple knowledge sources into a single database (see, for example, Mungall et al., 2017): this approach has the advantage of data alignment according to a common data model and of enabling high performance queries. However, centralized resources are difficult and expensive to maintain and expand (Chandras et al., 2009; Gabella et al., 2018), at least in part because of bottlenecks that are inherent in a centralized design.

At the other end of the spectrum, distributed approaches to data integration result in a broad landscape of individual resources, focusing on technical infrastructure to query and integrate across them for each query. These approaches lower the barriers to adding new data by enabling anyone to publish data by following community standards. However, performance is often an issue when each query must be sent to many individual databases, and the performance of the system as a whole is highly dependent on the stability and performance of each individual component. In addition, data integration requires harmonizing the differences in the data models and data formats between resources, a process that can often require significant skill and effort. Moreover, harmonizing differences in data licensing can sometimes be impossible.

Here we explore the use of Wikidata (www.wikidata.org; Vrandečić, 2012; Mora-Cantallops et al., 2019) as a platform for knowledge integration in the life sciences. Wikidata is an openly-accessible knowledge base that is editable by anyone. Like its sister project Wikipedia, the scope of Wikidata is nearly boundless, with items on topics as diverse as books, actors, historical events, and galaxies. Unlike Wikipedia, Wikidata focuses on representing knowledge in a structured format instead of primarily free text. As of September 2019, Wikidata's knowledge graph included over 750 million statements on 61 million items (tools.wmflabs.org/wikidata-todo/stats.php). Wikidata was also the first project run by the Wikimedia Foundation (which also runs Wikipedia) to have surpassed one billion edits, achieved by a community of 12,000 active users, including 100 active computational ‘bots’ (Figure 1—figure supplement 1).

As a knowledge integration platform, Wikidata combines several of the key strengths of the centralized and distributed approaches. A large portion of the Wikidata knowledge graph is based on the automated imports of large structured databases via Wikidata bots, thereby breaking down the walls of existing data silos. Since Wikidata is also based on a community-editing model, it harnesses the distributed efforts of a worldwide community of contributors, including both domain experts and bot developers. Anyone is empowered to add new statements, ranging from individual facts to large-scale data imports. Finally, all knowledge in Wikidata is queryable through a SPARQL query interface (query.wikidata.org/), which also enables distributed queries across other Linked Data resources.

In previous work, we seeded Wikidata with content from public and authoritative sources of structured knowledge on genes and proteins (Burgstaller-Muehlbacher et al., 2016) and chemical compounds (Willighagen et al., 2018). Here, we describe progress on expanding and enriching the biomedical knowledge graph within Wikidata, both by our team and by others in the community (Turki et al., 2019). We also describe several representative biomedical use cases on how Wikidata can enable new analyses and improve the efficiency of research. Finally, we discuss how researchers can contribute to this effort to build a continuously-updated and community-maintained knowledge graph that epitomizes the FAIR principles.

The original effort behind this work focused on creating and annotating Wikidata items for human and mouse genes and proteins (Burgstaller-Muehlbacher et al., 2016), and was subsequently expanded to include microbial reference genomes from NCBI RefSeq (Putman et al., 2017). Since then, the Wikidata community (including our team) has significantly expanded the depth and breadth of biological information within Wikidata, resulting in a rich, heterogeneous knowledge graph (Figure 1). Some of the key new data types and resources are described below.

Genes and proteins: Wikidata contains items for over 1.1 million genes and 940 thousand proteins from 201 unique taxa. Annotation data on genes and proteins come from several key databases including NCBI Gene (Agarwala et al., 2018), Ensembl (Zerbino et al., 2018), UniProt (UniProt Consortium, 2019), InterPro (Mitchell et al., 2019), and the Protein Data Bank (Burley et al., 2019). These annotations include information on protein families, gene functions, protein domains, genomic location, and orthologs, as well as links to related compounds, diseases, and variants.

Genetic variants: Annotations on genetic variants are primarily drawn from CIViC (http://www.civicdb.org), an open and community-curated database of cancer variants (Griffith et al., 2017). Variants are annotated with their relevance to disease predisposition, diagnosis, prognosis, and drug efficacy. Wikidata currently contains 1502 items corresponding to human genetic variants, focused on those with a clear clinical or therapeutic relevance.

Chemical compounds including drugs: Wikidata has items for over 150 thousand chemical compounds, including over 3500 items which are specifically designated as medications. Compound attributes are drawn from a diverse set of databases, including PubChem (Wang et al., 2009), RxNorm (Nelson et al., 2011), the IUPHAR Guide to Pharmacology (Harding et al., 2018; Pawson et al., 2014; Southan et al., 2016), NDF-RT (National Drug File – Reference Terminology), and LIPID MAPS (Sud et al., 2007). These items typically contain statements describing chemical structure and key physicochemical properties, and links to databases with experimental data, such as MassBank (Horai et al., 2010; Wohlgemuth et al., 2016) and PDB Ligand (Shin, 2004), and toxicological information, such as the EPA CompTox Dashboard (Williams et al., 2017). Additionally, these items contain links to compound classes, disease indications, pharmaceutical products, and protein targets.

Pathways: Wikidata has items for almost three thousand human biological pathways, primarily from two established public pathway repositories: Reactome (Fabregat et al., 2018) and WikiPathways (Slenter et al., 2018). The full details of the different pathways remain with the respective primary sources. Our bots enter data for Wikidata properties such as pathway name, identifier, organism, and the list of component genes, proteins, and chemical compounds. Properties for contributing authors (via ORCID properties; Sprague, 2017), descriptions and ontology annotations are also being added for Wikidata pathway entries.

Diseases: Wikidata has items for over 16 thousand diseases, the majority of which were created based on imports from the Human Disease Ontology (Schriml et al., 2019), with additional disease terms added from the Monarch Disease Ontology (Mungall et al., 2017). Disease attributes include medical classifications, symptoms, relevant drugs, as well as subclass relationships to higher-level disease categories. In instances where the Human Disease Ontology specifies a related anatomic region and/or a causative organism (for infectious diseases), corresponding statements are also added.

References: Whenever practical, the provenance of each statement added to Wikidata was also added in a structured format. References are part of the core data model for a Wikidata statement. References can either cite the primary resource from which the statement was retrieved (including details like version number of the resource), or they can link to a Wikidata item corresponding to a publication as provided by a primary resource (as an extension of the WikiCite project; Ayers et al., 2019), or both. Wikidata contains over 20 million items corresponding to publications across many domain areas, including a heavy emphasis on biomedical journal articles.

Figure 1 with 1 supplement see all

Download asset Open asset

Figure 1—source data 1

Most frequent data sources cited as references for the biomedical subset of the Wikidata knowledge graph shown in Figure 1.

https://cdn.elifesciences.org/articles/52614/elife-52614-fig1-data1-v1.csv

Download elife-52614-fig1-data1-v1.csv

Translating between identifiers from different databases is one of the most common operations in bioinformatics analyses. Unfortunately, these translations are most often done by bespoke scripts and based on entity-specific mapping tables. These translation scripts are repetitively and redundantly written across our community and are rarely kept up to date, nor integrated in a reusable fashion.

An identifier translation service is a simple and straightforward application of the biomedical content in Wikidata. Based on mapping tables that have been imported, Wikidata items can be mapped to databases that are both widely- and rarely-used in the life sciences community. Because all these mappings are stored in a centralized database and use a systematic data model, generic and reusable translation scripts can easily be written (Figure 2). These scripts can be used as a foundation for more complex Wikidata queries, or the results can be downloaded and used as part of larger scripts or analyses.

Figure 2

Download asset Open asset

There are a number of other tools that are also aimed at solving the identifier translation use case, including the BioThings APIs (Xin et al., 2018), BridgeDb (van Iersel et al., 2010), BioMart (Smedley et al., 2015), UMLS (Bodenreider, 2004), and NCI Thesaurus (de Coronado et al., 2009). Relative to these tools, Wikidata distinguishes itself with a unique combination of the following: an almost limitless scope including all entities in biology, chemistry, and medicine; a data model that can represent exact, broader, and narrow matches between items in different identifier namespaces (beyond semantically imprecise 'cross-references'); programmatic access through web services with a track record of high performance and high availability.

Moreover, Wikidata is also unique as it is the only tool that allows real-time community editing. So while Wikidata is certainly not complete with respect to identifier mappings, it can be continually improved independent of any centralized effort or curation authority. As a database of assertions and not of absolute truth, Wikidata is able to represent conflicting information (with provenance) when, for example, different curation authorities produce different mappings between entities. (However, as with any bioinformatics integration exercise, harmonization of cross-references between resources can include relationships other than ‘exact match’. These instances can lead to Wikidata statements that are not explicitly declared, but rather the result of transitive inference.)

Integrative Queries

Wikidata contains a much broader set of information than just identifier cross-references. Having biomedical data in one centralized data resource facilitates powerful integrative queries that span multiple domain areas and data sources. Performing these integrative queries through Wikidata obviates the need to perform many time-consuming and error-prone data integration steps.

As an example, consider a pulmonologist who is interested in identifying candidate chemical compounds for testing in disease models (schematically illustrated in Figure 3). They may start by identifying genes with a genetic association to any respiratory disease, with a particular interest in genes that encode membrane-bound proteins (for ease in cell sorting). They may then look for chemical compounds that either directly inhibit those proteins, or finding none, compounds that inhibit another protein in the same pathway. Because they have collaborators with relevant expertise, they may specifically filter for proteins containing a serine-threonine kinase domain.

Figure 3

Download asset Open asset

Almost any competent informatician can perform the query described above by integrating cell localization data from Gene Ontology annotations, genetic associations from GWAS Catalog, disease subclass relationships from the Human Disease Ontology, pathway data from WikiPathways and Reactome, compound targets from the IUPHAR Guide to Pharmacology, and protein domain information from InterPro. However, actually performing this data integration is a time-consuming and error-prone process. At the time of publication of this manuscript, this Wikidata query completed in less than 10 s and reported 31 unique compounds. Importantly, the results of that query will always be up-to-date with the latest information in Wikidata.

This query, and other example SPARQL queries that take advantage of the rich, heterogeneous knowledge network in Wikidata are available at https://www.wikidata.org/wiki/User:ProteinBoxBot/SPARQL_Examples. That page additionally demonstrates federated SPARQL queries that perform complex queries across other biomedical SPARQL endpoints. Federated queries are useful for accessing data that cannot be included in Wikidata directly due to limitations in size, scope, or licensing.

Phenotype based disease diagnosis

Phenomizer is a web application that suggests clinical diagnoses based on an array of patient phenotypes (Köhler et al., 2009). On the back end, the latest version of Phenomizer uses BOQA, an algorithm that uses ontological structure in a Bayesian network (Bauer et al., 2012). For phenotype-based disease diagnosis, BOQA takes as input a list of phenotypes (using the Human Phenotype Ontology [HPO; Köhler et al., 2017]) and an association file between phenotypes and diseases. BOQA then suggests disease diagnoses based on semantic similarity (Köhler et al., 2009). Here, we studied whether phenotype-disease associations from Wikidata could improve BOQA’s ability to make differential diagnoses for certain sets of phenotypes. We modified the BOQA codebase to accept arbitrary inputs and to be able to run from the command line (code available at https://github.com/SuLab/boqa; archived at Köhler and Stupp, 2020) and also wrote a script to extract and incorporate the phenotype-disease annotations in Wikidata (code available at https://github.com/SuLab/Wikidata-phenomizer; archived at Tu et al., 2020).

As of September 2019, there were 273 phenotype-disease associations in Wikidata that were not in the HPO's annotation file (which contained a total of 172,760 associations). Based on parallel biocuration work by our team, many of these new associations were related to the disease Congenital Disorder of Deglycosylation (CDDG; also known as NGLY-1 deficiency) based on two papers describing patient phenotypes (Enns et al., 2014; Lam et al., 2017). To see if the Wikidata-sourced annotations improved the ability of BOQA to diagnose CDDG, we ran our modified version using the phenotypes taken from a third publication describing two siblings with suspected cases of CDDG (Caglayan et al., 2015). Using these phenotypes and the annotation file supplemented with Wikidata-derived associations, BOQA returned a much stronger semantic similarity to CDDG relative to the HPO annotation file alone (Figure 4). Analyses with the combined annotation file reported CDDG as the top result for each of the past 14 releases of the HPO annotation file, whereas CDDG was never the top result when run without the Wikidata-derived annotations.

Figure 4

Download asset Open asset

This result demonstrated an example scenario in which Wikidata-derived annotations could be a useful complement to expert curation. This example was specifically chosen to illustrate a favorable case, and the benefit of Wikidata would likely not currently generalize to a random sampling of other diseases. Nevertheless, we believe that this proof-of-concept demonstrates the value of the crowd-based Wikidata model and may motivate further community contributions.

Drug repurposing

The mining of graphs for latent edges has been an area of interest in a variety of contexts from predicting friend relationships in social media platforms to suggesting movies based on past viewing history. A number of groups have explored the mining of knowledge graphs to reveal biomedical insights, with the open source Rephetio effort for drug repurposing as one example (Himmelstein et al., 2017). Rephetio uses logistic regression, with features based on graph metapaths, to predict drug repurposing candidates.

The knowledge graph that served as the foundation for Rephetio was manually assembled from many different resources into a heterogeneous knowledge network. Here, we explored whether the Rephetio algorithm could successfully predict drug indications on the Wikidata knowledge graph. Based on the class diagram in Figure 1, we extracted a biomedically-focused subgraph of Wikidata with 19 node types and 41 edge types. We performed five-fold cross validation on drug indications within Wikidata and found that Rephetio substantially enriched the true indications in the hold-out set. We then downloaded historical Wikidata versions from 2017 and 2018 and observed marked improvements in performance over time (Figure 5). We also performed this analysis using an external test set based on Drug Central, which showed a similar improvement in Rephetio results over time (Figure 5—figure supplement 1).

Figure 5 with 1 supplement see all

Download asset Open asset

This analysis demonstrates the value of a community-maintained, centralized knowledge base to which many researchers are contributing. It suggests that scientific analyses based on Wikidata may continually improve irrespective of any changes to the underlying algorithms, but simply based on progress in curating knowledge through the distributed, and largely uncoordinated efforts of the Wikidata community.

We believe that the design of Wikidata is well-aligned with the FAIR data principles. 

Findable: Wikidata items are assigned globally unique identifiers with direct cross-links into the massive online ecosystem of Wikipedias. Wikidata also has broad visibility within the Linked Data community and is listed in the life science registries FAIRsharing (https://fairsharing.org/; Sansone et al., 2019) and Identifiers.org (Wimalaratne et al., 2018). Wikidata has already attracted a robust, global community of contributors and consumers. 

Accessible: Wikidata provides access to its underlying knowledge graph via both an online graphical user interface and an API, and access includes both read- and write-privileges. Wikidata provides database dumps at least weekly (https://www.wikidata.org/wiki/Wikidata:Database_download), ensuring the long-term accessibility of the Wikidata knowledge graph independent of the organization and web application. Finally, Wikidata is also natively multilingual. 

Interoperable: Wikidata items are extensively cross-linked to other biomedical resources using Universal Resource Identifiers (URIs), which unambiguously anchor these concepts in the Linked Open Data cloud (Jacobsen et al., 2018). Wikidata is also available in many standard formats in computer programming and knowledge management, including JSON, XML, and RDF. 

Reusable: Data provenance is directly tracked in the reference section of the Wikidata statement model. The Wikidata knowledge graph is released under the Creative Commons Zero (CC0) Public Domain Declaration, which explicitly declares that there are no restrictions on downstream reuse and redistribution.

The open data licensing of Wikidata is particularly notable. The use of data licenses in biomedical research has rapidly proliferated, presumably in an effort to protect intellectual property and/or justify long-term grant funding (see, for example, Reiser et al., 2016). However, even seemingly innocuous license terms (like requirements for attribution) still impose legal requirements and therefore expose consumers to legal liability. This liability is especially problematic for data integration efforts, in which the license terms of all resources (dozens or hundreds or more) must be independently tracked and satisfied (a phenomenon referred to as 'license stacking'). Because it is released under CC0, Wikidata can be freely and openly used in any other resource without any restriction. This freedom greatly simplifies and encourages downstream use, albeit at the cost of not being able to incorporate ontologies or datasets with more restrictive licensing.

In addition to simplifying data licensing, Wikidata offers significant advantages in centralizing the data harmonization process. Consider the use case of trying to get a comprehensive list of disease indications for the drug bupropion. The National Drug File – Reference Terminology (NDF-RT) reported that bupropion may treat nicotine dependence and attention deficit hyperactivity disorder, the Inxight database listed major depressive disorder, and the FDA Adverse Event Reporting System (FAERS) listed anxiety and bipolar disorder. While no single database listed all these indications, Wikidata provided an integrated view that enabled seamless query and access across resources. Integrating drug indication data from these individual data resources was not a trivial process. Both Inxight and NDF-RT mint their own identifiers for both drugs and diseases. FAERS uses Medical Dictionary for Regulatory Activities (MedDRA) names for diseases and free-text names for drugs (Stupp and Su, 2018). By harmonizing and integrating all resources in the context of Wikidata, we ensure that those data are immediately usable by others without having to repeat the normalization process. Moreover, by harmonizing data at the time of data loading, consumers of that data do not need to perform the repetitive and redundant work at the point of querying and analysis.

As the biomedical data within Wikidata continues to grow, we believe that its unencumbered use will spur the development of many new innovative tools and analyses. These innovations will undoubtedly include the machine learning-based mining of the knowledge graph to predict new relationships (also referred to as knowledge graph reasoning; Das et al., 2017; Lin et al., 2018; Xiong et al., 2017).

For those who subscribe to this vision for cultivating a FAIR and open graph of biomedical knowledge, there are two simple ways to contribute to Wikidata. First, owners of data resources can release their data using the CC0 declaration. Because Wikidata is released under CC0, it also means that all data imported in Wikidata must also use CC0-compatible terms (e.g., be in the public domain). For resources that currently use a restrictive data license primarily for the purposes of enforcing attribution or citation, we encourage the transition to CC0 (+BY), a model that "move[s] the attribution from the legal realm into the social or ethical realm by pairing a permissive license with a strong moral entreaty’ (Cohen, 2013). For resources that must retain data license restrictions, consider releasing a subset of data or older versions of data using CC0. Many biomedical resources were created under or transitioned to CC0 (in part or in full) in recent years , including the Disease Ontology (Schriml et al., 2019), Pfam (El-Gebali et al., 2019), Bgee (Bastian et al., 2008), WikiPathways (Slenter et al., 2018), Reactome (Fabregat et al., 2018), ECO (Chibucos et al., 2014), and CIViC (Griffith et al., 2017).

Second, informaticians can contribute to Wikidata by adding the results of data parsing and integration efforts to Wikidata as, for example, new Wikidata items, statements, or references. Currently, the useful lifespan of data integration code typically does not extend beyond the immediate project-specific use. As a result, that same data integration process is likely performed repetitively and redundantly by other informaticians elsewhere. If every informatician contributed the output of their effort to Wikidata, the resulting knowledge graph would be far more useful than the stand-alone contribution of any single individual, and it would continually improve in both breadth and depth over time. Indeed, the growth of biomedical data in Wikidata is driven not by any centralized or coordinated process, but rather the aggregated effort and priorities of Wikidata contributors themselves.

FAIR and open access to the sum total of biomedical knowledge will improve the efficiency of biomedical research. Capturing that information in a centralized knowledge graph is useful for experimental researchers, informatics tool developers and biomedical data scientists. As a continuously-updated and collaboratively-maintained community resource, we believe that Wikidata has made significant strides toward achieving this ambitious goal.

Links to all data and code used in this manuscript have been provided.

1. 1. Agarwala R
   2. Barrett T
   3. Beck J
   4. Benson DA
   5. Bollin C
   6. Bolton E
   7. Bourexis D
   8. Brister JR
   9. Bryant SH
   10. Canese K
   11. Cavanaugh M
   12. Charowhas C
   13. Clark K
   14. Dondoshansky I
   15. Feolo M
   16. Fitzpatrick L
   17. Funk K
   18. Geer LY
   19. Gorelenkov V
   20. Graeff A
   21. Hlavina W
   22. Holmes B
   23. Johnson M
   24. Kattman B
   25. Khotomlianski V
   26. Kimchi A
   27. Kimelman M
   28. Kimura M
   29. Kitts P
   30. Klimke W
   31. Kotliarov A
   32. Krasnov S
   33. Kuznetsov A
   34. Landrum MJ
   35. Landsman D
   36. Lathrop S
   37. Lee JM
   38. Leubsdorf C
   39. Lu Z
   40. Madden TL
   41. Marchler-Bauer A
   42. Malheiro A
   43. Meric P
   44. Karsch-Mizrachi I
   45. Mnev A
   46. Murphy T
   47. Orris R
   48. Ostell J
   49. O'Sullivan C
   50. Palanigobu V
   51. Panchenko AR
   52. Phan L
   53. Pierov B
   54. Pruitt KD
   55. Rodarmer K
   56. Sayers EW
   57. Schneider V
   58. Schoch CL
   59. Schuler GD
   60. Sherry ST
   61. Siyan K
   62. Soboleva A
   63. Soussov V
   64. Starchenko G
   65. Tatusova TA
   66. Thibaud-Nissen F
   67. Todorov K
   68. Trawick BW
   69. Vakatov D
   70. Ward M
   71. Yaschenko E
   72. Zasypkin A
   73. Zbicz K
   74. Coordinators NR
   75. NCBI Resource Coordinators

   (2018) Database resources of the National Center for Biotechnology Information

   Nucleic Acids Research 46:D8–D13.

   https://doi.org/10.1093/nar/gkx1095

   • PubMed
   • Google Scholar
2. 1. Amberger JS
   2. Hamosh A

   (2017) Searching Online Mendelian Inheritance in Man (OMIM): A knowledgebase of human genes and genetic phenotypes

   Current Protocols in Bioinformatics 58:27.

   https://doi.org/10.1002/cpbi.27

   • Google Scholar
3. Report

   1. Ayers P
   2. Mietchen D
   3. Orlowitz J
   4. Proffitt M
   5. Rodlund S
   6. Seiver E
   7. Taraborelli D
   8. Vershbow B

   (2019)

   WikiCite 2018-2019: Citations for the Sum of All Human Knowledge

   Wikimedia Foundation.

   • Google Scholar
4. Book

   1. Bastian F
   2. Parmentier G
   3. Roux J
   4. Moretti S
   5. Laudet V
   6. Robinson-Rechavi M

   (2008) Bgee: Integrating and Comparing Heterogeneous Transcriptome Data Among Species

   In: Bairoch A, Cohen-Boulakia S, Froidevaux C, editors. Data Integration in the Life Sciences, Lecture Notes in Computer Science. Berlin Heidelberg: Springer. pp. 124–131.

   https://doi.org/10.1007/978-3-540-69828-9_12

   • Google Scholar
5. 1. Bauer S
   2. Köhler S
   3. Schulz MH
   4. Robinson PN

   (2012) Bayesian ontology querying for accurate and noise-tolerant semantic searches

   Bioinformatics 28:2502–2508.

   https://doi.org/10.1093/bioinformatics/bts471

   • PubMed
   • Google Scholar
6. 1. Bodenreider O

   (2004) The Unified Medical Language System (UMLS): integrating biomedical terminology

   Nucleic Acids Research 32:267D–270.

   https://doi.org/10.1093/nar/gkh061

   • Google Scholar
7. 1. Bunt SM
   2. Grumbling GB
   3. Field HI
   4. Marygold SJ
   5. Brown NH
   6. Millburn GH
   7. FlyBase Consortium

   (2012) Directly e-mailing authors of newly published papers encourages community curation

   Database 2012:bas024.

   https://doi.org/10.1093/database/bas024

   • PubMed
   • Google Scholar
8. 1. Burgstaller-Muehlbacher S
   2. Waagmeester A
   3. Mitraka E
   4. Turner J
   5. Putman T
   6. Leong J
   7. Naik C
   8. Pavlidis P
   9. Schriml L
   10. Good BM
   11. Su AI

   (2016) Wikidata as a semantic framework for the Gene Wiki initiative

   Database 2016:baw015.

   https://doi.org/10.1093/database/baw015

   • PubMed
   • Google Scholar
9. Software

   1. Burgstaller-Muehlbacher S
   2. Waagmeester A
   3. Stupp GS
   4. Arrow T
   5. konstin
   6. jleong-ndn
   7. Putman TE
   8. Mystou
   9. Mietchen D
   10. Su AI
   11. pdehaye
   12. Leturia I
   13. Vrandečić D
   14. Andrea

   (2020) SuLab/WikidataIntegrator 0.5.1

   Zenodo.

   http://doi.org/10.5281/zenodo.3621065
10. 1. Burley SK
    2. Berman HM
    3. Bhikadiya C
    4. Bi C
    5. Chen L
    6. Costanzo LD
    7. Christie C
    8. Duarte JM
    9. Dutta S
    10. Feng Z
    11. Ghosh S
    12. Goodsell DS
    13. Green RK
    14. Guranovic V
    15. Guzenko D
    16. Hudson BP
    17. Liang Y
    18. Lowe R
    19. Peisach E
    20. Periskova I
    21. Randle C
    22. Rose A
    23. Sekharan M
    24. Shao C
    25. Tao Y-P
    26. Valasatava Y
    27. Voigt M
    28. Westbrook J
    29. Young J
    30. Zardecki C
    31. Zhuravleva M
    32. Kurisu G
    33. Nakamura H
    34. Kengaku Y
    35. Cho H
    36. Sato J
    37. Kim JY
    38. Ikegawa Y
    39. Nakagawa A
    40. Yamashita R
    41. Kudou T
    42. Bekker G-J
    43. Suzuki H
    44. Iwata T
    45. Yokochi M
    46. Kobayashi N
    47. Fujiwara T
    48. Velankar S
    49. Kleywegt GJ
    50. Anyango S
    51. Armstrong DR
    52. Berrisford JM
    53. Conroy MJ
    54. Dana JM
    55. Deshpande M
    56. Gane P
    57. Gáborová R
    58. Gupta D
    59. Gutmanas A
    60. Koča J
    61. Mak L
    62. Mir S
    63. Mukhopadhyay A
    64. Nadzirin N
    65. Nair S
    66. Patwardhan A
    67. Paysan-Lafosse T
    68. Pravda L
    69. Salih O
    70. Sehnal D
    71. Varadi M
    72. Vařeková R
    73. Markley JL
    74. Hoch JC
    75. Romero PR
    76. Baskaran K
    77. Maziuk D
    78. Ulrich EL
    79. Wedell JR
    80. Yao H
    81. Livny M
    82. Ioannidis YE
    83. wwPDB consortium

    (2019) Protein Data Bank: the single global archive for 3D macromolecular structure data

    Nucleic Acids Research 47:D520–D528.

    https://doi.org/10.1093/nar/gky949

    • PubMed
    • Google Scholar
11. 1. Caglayan AO
    2. Comu S
    3. Baranoski JF
    4. Parman Y
    5. Kaymakçalan H
    6. Akgumus GT
    7. Caglar C
    8. Dolen D
    9. Erson-Omay EZ
    10. Harmanci AS
    11. Mishra-Gorur K
    12. Freeze HH
    13. Yasuno K
    14. Bilguvar K
    15. Gunel M

    (2015) NGLY1 mutation causes neuromotor impairment, intellectual disability, and neuropathy

    European Journal of Medical Genetics 58:39–43.

    https://doi.org/10.1016/j.ejmg.2014.08.008

    • PubMed
    • Google Scholar
12. 1. Chandras C
    2. Weaver T
    3. Zouberakis M
    4. Smedley D
    5. Schughart K
    6. Rosenthal N
    7. Hancock JM
    8. Kollias G
    9. Schofield PN
    10. Aidinis V

    (2009) Models for financial sustainability of biological databases and resources

    Database 2009:bap017.

    https://doi.org/10.1093/database/bap017

    • PubMed
    • Google Scholar
13. 1. Chibucos MC
    2. Mungall CJ
    3. Balakrishnan R
    4. Christie KR
    5. Huntley RP
    6. White O
    7. Blake JA
    8. Lewis SE
    9. Giglio M

    (2014) Standardized description of scientific evidence using the Evidence Ontology (ECO)

    Database 2014:bau075.

    https://doi.org/10.1093/database/bau075

    • PubMed
    • Google Scholar
14. 1. Cohen D

    (2013) CC0 (+BY)

    CC0 (+BY), https://dancohen.org/2013/11/26/cc0-by/.

    https://dancohen.org/2013/11/26/cc0-by/

    • Google Scholar
15. Preprint

    1. Das R
    2. Dhuliawala S
    3. Zaheer M
    4. Vilnis L
    5. Durugkar I
    6. Krishnamurthy A
    7. Smola A
    8. McCallum A

    (2017) Go for a walk and arrive at the answer: reasoning over paths in knowledge bases using reinforcement learning

    arXiv.

    https://arxiv.org/abs/1711.05851

    • Google Scholar
16. 1. de Coronado S
    2. Wright LW
    3. Fragoso G
    4. Haber MW
    5. Hahn-Dantona EA
    6. Hartel FW
    7. Quan SL
    8. Safran T
    9. Thomas N
    10. Whiteman L

    (2009) The NCI Thesaurus quality assurance life cycle

    Journal of Biomedical Informatics 42:530–539.

    https://doi.org/10.1016/j.jbi.2009.01.003

    • PubMed
    • Google Scholar
17. 1. El-Gebali S
    2. Mistry J
    3. Bateman A
    4. Eddy SR
    5. Luciani A
    6. Potter SC
    7. Qureshi M
    8. Richardson LJ
    9. Salazar GA
    10. Smart A
    11. Sonnhammer ELL
    12. Hirsh L
    13. Paladin L
    14. Piovesan D
    15. Tosatto SCE
    16. Finn RD

    (2019) The Pfam protein families database in 2019

    Nucleic Acids Research 47:D427–D432.

    https://doi.org/10.1093/nar/gky995

    • PubMed
    • Google Scholar
18. 1. Enns GM
    2. Shashi V
    3. Bainbridge M
    4. Gambello MJ
    5. Zahir FR
    6. Bast T
    7. Crimian R
    8. Schoch K
    9. Platt J
    10. Cox R
    11. Bernstein JA
    12. Scavina M
    13. Walter RS
    14. Bibb A
    15. Jones M
    16. Hegde M
    17. Graham BH
    18. Need AC
    19. Oviedo A
    20. Schaaf CP
    21. Boyle S
    22. Butte AJ
    23. Chen R
    24. Chen R
    25. Clark MJ
    26. Haraksingh R
    27. Cowan TM
    28. He P
    29. Langlois S
    30. Zoghbi HY
    31. Snyder M
    32. Gibbs RA
    33. Freeze HH
    34. Goldstein DB
    35. FORGE Canada Consortium

    (2014) Mutations in NGLY1 cause an inherited disorder of the endoplasmic reticulum-associated degradation pathway

    Genetics in Medicine 16:751–758.

    https://doi.org/10.1038/gim.2014.22

    • PubMed
    • Google Scholar
19. 1. Fabregat A
    2. Jupe S
    3. Matthews L
    4. Sidiropoulos K
    5. Gillespie M
    6. Garapati P
    7. Haw R
    8. Jassal B
    9. Korninger F
    10. May B
    11. Milacic M
    12. Roca CD
    13. Rothfels K
    14. Sevilla C
    15. Shamovsky V
    16. Shorser S
    17. Varusai T
    18. Viteri G
    19. Weiser J
    20. Wu G
    21. Stein L
    22. Hermjakob H
    23. D'Eustachio P

    (2018) The Reactome Pathway Knowledgebase

    Nucleic Acids Research 46:D649–D655.

    https://doi.org/10.1093/nar/gkx1132

    • PubMed
    • Google Scholar
20. 1. Gabella C
    2. Durinx C
    3. Appel R

    (2018) Funding knowledgebases: towards a sustainable funding model for the UniProt use case

    F1000Research 6:2051.

    https://doi.org/10.12688/f1000research.12989.2

    • Google Scholar
21. Book

    1. Gil Y
    2. Garijo D
    3. Ratnakar V
    4. Khider D
    5. Emile-Geay J
    6. McKay N

    (2017) A Controlled Crowdsourcing Approach for Practical Ontology Extensions and Metadata Annotations

    In: d’Amato C, Fernandez M, Tamma V, Lecue F, Cudré-Mauroux P, Sequeda J, Lange C, Heflin J, editors. The Semantic Web – ISWC 2017, Lecture Notes in Computer Science. Springer International Publishing. pp. 231–246.

    https://doi.org/10.1007/978-3-319-68204-4

    • Google Scholar
22. 1. Griffith M
    2. Spies NC
    3. Krysiak K
    4. McMichael JF
    5. Coffman AC
    6. Danos AM
    7. Ainscough BJ
    8. Ramirez CA
    9. Rieke DT
    10. Kujan L
    11. Barnell EK
    12. Wagner AH
    13. Skidmore ZL
    14. Wollam A
    15. Liu CJ
    16. Jones MR
    17. Bilski RL
    18. Lesurf R
    19. Feng YY
    20. Shah NM
    21. Bonakdar M
    22. Trani L
    23. Matlock M
    24. Ramu A
    25. Campbell KM
    26. Spies GC
    27. Graubert AP
    28. Gangavarapu K
    29. Eldred JM
    30. Larson DE
    31. Walker JR
    32. Good BM
    33. Wu C
    34. Su AI
    35. Dienstmann R
    36. Margolin AA
    37. Tamborero D
    38. Lopez-Bigas N
    39. Jones SJ
    40. Bose R
    41. Spencer DH
    42. Wartman LD
    43. Wilson RK
    44. Mardis ER
    45. Griffith OL

    (2017) CIViC is a community knowledgebase for expert crowdsourcing the clinical interpretation of variants in cancer

    Nature Genetics 49:170–174.

    https://doi.org/10.1038/ng.3774

    • PubMed
    • Google Scholar
23. 1. Harding SD
    2. Sharman JL
    3. Faccenda E
    4. Southan C
    5. Pawson AJ
    6. Ireland S
    7. Gray AJG
    8. Bruce L
    9. Alexander SPH
    10. Anderton S
    11. Bryant C
    12. Davenport AP
    13. Doerig C
    14. Fabbro D
    15. Levi-Schaffer F
    16. Spedding M
    17. Davies JA
    18. NC-IUPHAR

    (2018) The IUPHAR/BPS guide to PHARMACOLOGY in 2018: updates and expansion to encompass the new guide to IMMUNOPHARMACOLOGY

    Nucleic Acids Research 46:D1091–D1106.

    https://doi.org/10.1093/nar/gkx1121

    • PubMed
    • Google Scholar
24. 1. Himmelstein DS
    2. Lizee A
    3. Hessler C
    4. Brueggeman L
    5. Chen SL
    6. Hadley D
    7. Green A
    8. Khankhanian P
    9. Baranzini SE

    (2017) Systematic integration of biomedical knowledge prioritizes drugs for repurposing

    eLife 6:e26726.

    https://doi.org/10.7554/eLife.26726

    • PubMed
    • Google Scholar
25. 1. Horai H
    2. Arita M
    3. Kanaya S
    4. Nihei Y
    5. Ikeda T
    6. Suwa K
    7. Ojima Y
    8. Tanaka K
    9. Tanaka S
    10. Aoshima K
    11. Oda Y
    12. Kakazu Y
    13. Kusano M
    14. Tohge T
    15. Matsuda F
    16. Sawada Y
    17. Hirai MY
    18. Nakanishi H
    19. Ikeda K
    20. Akimoto N
    21. Maoka T
    22. Takahashi H
    23. Ara T
    24. Sakurai N
    25. Suzuki H
    26. Shibata D
    27. Neumann S
    28. Iida T
    29. Tanaka K
    30. Funatsu K
    31. Matsuura F
    32. Soga T
    33. Taguchi R
    34. Saito K
    35. Nishioka T

    (2010) MassBank: a public repository for sharing mass spectral data for life sciences

    Journal of Mass Spectrometry 45:703–714.

    https://doi.org/10.1002/jms.1777

    • PubMed
    • Google Scholar
26. 1. Jacobsen A
    2. Kaliyaperumal R
    3. Stupp GS
    4. Schriml LM
    5. Thompson M
    6. Su AI
    7. Roos M

    (2018)

    Proceedings of the 11th International Conference Semantic Web Applications and Tools for Life Sciences, {SWAT4LS} 2018, Antwerp, Belgium, December 3-6, 2018

    Wikidata as an intuitive resource towards semantic data modeling in data FAIRification, Proceedings of the 11th International Conference Semantic Web Applications and Tools for Life Sciences, {SWAT4LS} 2018, Antwerp, Belgium, December 3-6, 2018, 2275, CEUR-WS.org.

    • Google Scholar
27. 1. Köhler S
    2. Schulz MH
    3. Krawitz P
    4. Bauer S
    5. Dölken S
    6. Ott CE
    7. Mundlos C
    8. Horn D
    9. Mundlos S
    10. Robinson PN

    (2009) Clinical diagnostics in human genetics with semantic similarity searches in ontologies

    The American Journal of Human Genetics 85:457–464.

    https://doi.org/10.1016/j.ajhg.2009.09.003

    • PubMed
    • Google Scholar
28. 1. Köhler S
    2. Vasilevsky NA
    3. Engelstad M
    4. Foster E
    5. McMurry J
    6. Aymé S
    7. Baynam G
    8. Bello SM
    9. Boerkoel CF
    10. Boycott KM
    11. Brudno M
    12. Buske OJ
    13. Chinnery PF
    14. Cipriani V
    15. Connell LE
    16. Dawkins HJ
    17. DeMare LE
    18. Devereau AD
    19. de Vries BB
    20. Firth HV
    21. Freson K
    22. Greene D
    23. Hamosh A
    24. Helbig I
    25. Hum C
    26. Jähn JA
    27. James R
    28. Krause R
    29. F Laulederkind SJ
    30. Lochmüller H
    31. Lyon GJ
    32. Ogishima S
    33. Olry A
    34. Ouwehand WH
    35. Pontikos N
    36. Rath A
    37. Schaefer F
    38. Scott RH
    39. Segal M
    40. Sergouniotis PI
    41. Sever R
    42. Smith CL
    43. Straub V
    44. Thompson R
    45. Turner C
    46. Turro E
    47. Veltman MW
    48. Vulliamy T
    49. Yu J
    50. von Ziegenweidt J
    51. Zankl A
    52. Züchner S
    53. Zemojtel T
    54. Jacobsen JO
    55. Groza T
    56. Smedley D
    57. Mungall CJ
    58. Haendel M
    59. Robinson PN

    (2017) The Human Phenotype Ontology in 2017

    Nucleic Acids Research 45:D865–D876.

    https://doi.org/10.1093/nar/gkw1039

    • PubMed
    • Google Scholar
29. Software

    1. Köhler S
    2. Stupp GS

    (2020) SuLab/boqa: release v1.0 2020-01-21

    Zenodo.

    http://doi.org/10.5281/zenodo.3620979
30. 1. Lam C
    2. Ferreira C
    3. Krasnewich D
    4. Toro C
    5. Latham L
    6. Zein WM
    7. Lehky T
    8. Brewer C
    9. Baker EH
    10. Thurm A
    11. Farmer CA
    12. Rosenzweig SD
    13. Lyons JJ
    14. Schreiber JM
    15. Gropman A
    16. Lingala S
    17. Ghany MG
    18. Solomon B
    19. Macnamara E
    20. Davids M
    21. Stratakis CA
    22. Kimonis V
    23. Gahl WA
    24. Wolfe L

    (2017) Prospective phenotyping of NGLY1-CDDG, the first congenital disorder of deglycosylation

    Genetics in Medicine 19:160–168.

    https://doi.org/10.1038/gim.2016.75

    • PubMed
    • Google Scholar
31. 1. Lewis J
    2. Snyder M
    3. Hyatt-Knorr H

    (2017) Marking 15 years of the Genetic and Rare Diseases Information Center

    Translational Science of Rare Diseases 2:77–88.

    https://doi.org/10.3233/TRD-170011

    • PubMed
    • Google Scholar
32. Preprint

    1. Lin XV
    2. Socher R
    3. Xiong C

    (2018) Multi-hop knowledge graph reasoning with reward shaping

    arXiv.

    https://arxiv.org/abs/1808.10568

    • Google Scholar
33. 1. Maiella S
    2. Olry A
    3. Hanauer M
    4. Lanneau V
    5. Lourghi H
    6. Donadille B
    7. Rodwell C
    8. Köhler S
    9. Seelow D
    10. Jupp S
    11. Parkinson H
    12. Groza T
    13. Brudno M
    14. Robinson PN
    15. Rath A

    (2018) Harmonising phenomics information for a better interoperability in the rare disease field

    European Journal of Medical Genetics 61:706–714.

    https://doi.org/10.1016/j.ejmg.2018.01.013

    • PubMed
    • Google Scholar
34. 1. Manske M

    (2020) Mix’n’match

    Mix’n’match, https://tools.wmflabs.org/mix-n-match/#/.

    https://tools.wmflabs.org/mix-n-match/#/

    • Google Scholar
35. Software

    1. Mayers MD
    2. Su AI

    (2020) SuLab/WD-rephetio-analysis: v1.1 release on 2020-01-21

    Zenodo.

    http://doi.org/10.5281/zenodo.3620811
36. Software

    1. Mayers MD
    2. Stupp GS
    3. Su AI

    (2020) SuLab/genewikiworld: v1.1 release on 2020-01-21

    Zenodo.

    http://doi.org/10.5281/zenodo.3620812
37. 1. McKusick VA

    (2007) Mendelian Inheritance In Man and its online version, OMIM

    The American Journal of Human Genetics 80:588–604.

    https://doi.org/10.1086/514346

    • PubMed
    • Google Scholar
38. 1. Mitchell AL
    2. Attwood TK
    3. Babbitt PC
    4. Blum M
    5. Bork P
    6. Bridge A
    7. Brown SD
    8. Chang HY
    9. El-Gebali S
    10. Fraser MI
    11. Gough J
    12. Haft DR
    13. Huang H
    14. Letunic I
    15. Lopez R
    16. Luciani A
    17. Madeira F
    18. Marchler-Bauer A
    19. Mi H
    20. Natale DA
    21. Necci M
    22. Nuka G
    23. Orengo C
    24. Pandurangan AP
    25. Paysan-Lafosse T
    26. Pesseat S
    27. Potter SC
    28. Qureshi MA
    29. Rawlings ND
    30. Redaschi N
    31. Richardson LJ
    32. Rivoire C
    33. Salazar GA
    34. Sangrador-Vegas A
    35. Sigrist CJA
    36. Sillitoe I
    37. Sutton GG
    38. Thanki N
    39. Thomas PD
    40. Tosatto SCE
    41. Yong SY
    42. Finn RD

    (2019) InterPro in 2019: improving coverage, classification and access to protein sequence annotations

    Nucleic Acids Research 47:D351–D360.

    https://doi.org/10.1093/nar/gky1100

    • PubMed
    • Google Scholar
39. 1. Mora-Cantallops M
    2. Sánchez-Alonso S
    3. García-Barriocanal E

    (2019) A systematic literature review on Wikidata

    Data Technologies and Applications 53:250–268.

    https://doi.org/10.1108/DTA-12-2018-0110

    • Google Scholar
40. 1. Mungall CJ
    2. McMurry JA
    3. Köhler S
    4. Balhoff JP
    5. Borromeo C
    6. Brush M
    7. Carbon S
    8. Conlin T
    9. Dunn N
    10. Engelstad M
    11. Foster E
    12. Gourdine JP
    13. Jacobsen JO
    14. Keith D
    15. Laraway B
    16. Lewis SE
    17. NguyenXuan J
    18. Shefchek K
    19. Vasilevsky N
    20. Yuan Z
    21. Washington N
    22. Hochheiser H
    23. Groza T
    24. Smedley D
    25. Robinson PN
    26. Haendel MA

    (2017) The Monarch Initiative: an integrative data and analytic platform connecting phenotypes to genotypes across species

    Nucleic Acids Research 45:D712–D722.

    https://doi.org/10.1093/nar/gkw1128

    • PubMed
    • Google Scholar
41. 1. Nelson SJ
    2. Zeng K
    3. Kilbourne J
    4. Powell T
    5. Moore R

    (2011) Normalized names for clinical drugs: RxNorm at 6 years

    Journal of the American Medical Informatics Association 18:441–448.

    https://doi.org/10.1136/amiajnl-2011-000116

    • PubMed
    • Google Scholar
42. Book

    1. Nielsen FA
    2. Mietchen D
    3. Willighagen E

    (2017) Scholia, Scientometrics and Wikidata

    In: Blomqvist E, Hose K, Paulheim H, Ławrynowicz A, Ciravegna F, Hartig O, editors. The Semantic Web: ESWC 2017 Satellite Events, Lecture Notes in Computer Science. Springer International Publishing. pp. 237–259.

    https://doi.org/10.1007/978-3-319-70407-4_36

    • Google Scholar
43. 1. Pawson AJ
    2. Sharman JL
    3. Benson HE
    4. Faccenda E
    5. Alexander SP
    6. Buneman OP
    7. Davenport AP
    8. McGrath JC
    9. Peters JA
    10. Southan C
    11. Spedding M
    12. Yu W
    13. Harmar AJ
    14. NC-IUPHAR

    (2014) The IUPHAR/BPS guide to PHARMACOLOGY: an expert-driven knowledgebase of drug targets and their ligands

    Nucleic Acids Research 42:D1098–D1106.

    https://doi.org/10.1093/nar/gkt1143

    • PubMed
    • Google Scholar
44. 1. Putman TE
    2. Lelong S
    3. Burgstaller-Muehlbacher S
    4. Waagmeester A
    5. Diesh C
    6. Dunn N
    7. Munoz-Torres M
    8. Stupp GS
    9. Wu C
    10. Su AI
    11. Good BM

    (2017) WikiGenomes: an open web application for community consumption and curation of gene annotation data in Wikidata

    Database 2017:bax025.

    https://doi.org/10.1093/database/bax025

    • Google Scholar
45. 1. Putman T
    2. Hybiske K
    3. Jow D
    4. Afrasiabi C
    5. Lelong S
    6. Cano MA
    7. Wu C
    8. Su AI

    (2019) ChlamBase: a curated model organism database for the Chlamydia research community

    Database 2019:baz041.

    https://doi.org/10.1093/database/baz041

    • PubMed
    • Google Scholar
46. 1. Reiser L
    2. Berardini TZ
    3. Li D
    4. Muller R
    5. Strait EM
    6. Li Q
    7. Mezheritsky Y
    8. Vetushko A
    9. Huala E

    (2016) Sustainable funding for biocuration: The Arabidopsis Information Resource (TAIR) as a case study of a subscription-based funding model

    Database 2016:baw018.

    https://doi.org/10.1093/database/baw018

    • PubMed
    • Google Scholar
47. 1. Sansone SA
    2. McQuilton P
    3. Rocca-Serra P
    4. Gonzalez-Beltran A
    5. Izzo M
    6. Lister AL
    7. Thurston M
    8. FAIRsharing Community

    (2019) FAIRsharing as a community approach to standards, repositories and policies

    Nature Biotechnology 37:358–367.

    https://doi.org/10.1038/s41587-019-0080-8

    • PubMed
    • Google Scholar
48. 1. Scholia

    (2019) Scholia, ACE Inhibitor Pathway

    Scholia, ACE Inhibitor Pathway, https://tools.wmflabs.org/scholia/pathway/Q29892242.

    https://tools.wmflabs.org/scholia/pathway/Q29892242

    • Google Scholar
49. 1. Schriml LM
    2. Mitraka E
    3. Munro J
    4. Tauber B
    5. Schor M
    6. Nickle L
    7. Felix V
    8. Jeng L
    9. Bearer C
    10. Lichenstein R
    11. Bisordi K
    12. Campion N
    13. Hyman B
    14. Kurland D
    15. Oates CP
    16. Kibbey S
    17. Sreekumar P
    18. Le C
    19. Giglio M
    20. Greene C

    (2019) Human Disease Ontology 2018 update: classification, content and workflow expansion

    Nucleic Acids Research 47:D955–D962.

    https://doi.org/10.1093/nar/gky1032

    • PubMed
    • Google Scholar
50. 1. Shin J-M

    (2004) PDB-Ligand: a ligand database based on PDB for the automated and customized classification of ligand-binding structures

    Nucleic Acids Research 33:D238–D241.

    https://doi.org/10.1093/nar/gki059

    • Google Scholar
51. 1. Slenter DN
    2. Kutmon M
    3. Hanspers K
    4. Riutta A
    5. Windsor J
    6. Nunes N
    7. Mélius J
    8. Cirillo E
    9. Coort SL
    10. Digles D
    11. Ehrhart F
    12. Giesbertz P
    13. Kalafati M
    14. Martens M
    15. Miller R
    16. Nishida K
    17. Rieswijk L
    18. Waagmeester A
    19. Eijssen LMT
    20. Evelo CT
    21. Pico AR
    22. Willighagen EL

    (2018) WikiPathways: a multifaceted pathway database bridging metabolomics to other omics research

    Nucleic Acids Research 46:D661–D667.

    https://doi.org/10.1093/nar/gkx1064

    • PubMed
    • Google Scholar
52. 1. Smedley D
    2. Haider S
    3. Durinck S
    4. Pandini L
    5. Provero P
    6. Allen J
    7. Arnaiz O
    8. Awedh MH
    9. Baldock R
    10. Barbiera G
    11. Bardou P
    12. Beck T
    13. Blake A
    14. Bonierbale M
    15. Brookes AJ
    16. Bucci G
    17. Buetti I
    18. Burge S
    19. Cabau C
    20. Carlson JW
    21. Chelala C
    22. Chrysostomou C
    23. Cittaro D
    24. Collin O
    25. Cordova R
    26. Cutts RJ
    27. Dassi E
    28. Di Genova A
    29. Djari A
    30. Esposito A
    31. Estrella H
    32. Eyras E
    33. Fernandez-Banet J
    34. Forbes S
    35. Free RC
    36. Fujisawa T
    37. Gadaleta E
    38. Garcia-Manteiga JM
    39. Goodstein D
    40. Gray K
    41. Guerra-Assunção JA
    42. Haggarty B
    43. Han DJ
    44. Han BW
    45. Harris T
    46. Harshbarger J
    47. Hastings RK
    48. Hayes RD
    49. Hoede C
    50. Hu S
    51. Hu ZL
    52. Hutchins L
    53. Kan Z
    54. Kawaji H
    55. Keliet A
    56. Kerhornou A
    57. Kim S
    58. Kinsella R
    59. Klopp C
    60. Kong L
    61. Lawson D
    62. Lazarevic D
    63. Lee JH
    64. Letellier T
    65. Li CY
    66. Lio P
    67. Liu CJ
    68. Luo J
    69. Maass A
    70. Mariette J
    71. Maurel T
    72. Merella S
    73. Mohamed AM
    74. Moreews F
    75. Nabihoudine I
    76. Ndegwa N
    77. Noirot C
    78. Perez-Llamas C
    79. Primig M
    80. Quattrone A
    81. Quesneville H
    82. Rambaldi D
    83. Reecy J
    84. Riba M
    85. Rosanoff S
    86. Saddiq AA
    87. Salas E
    88. Sallou O
    89. Shepherd R
    90. Simon R
    91. Sperling L
    92. Spooner W
    93. Staines DM
    94. Steinbach D
    95. Stone K
    96. Stupka E
    97. Teague JW
    98. Dayem Ullah AZ
    99. Wang J
    100. Ware D
    101. Wong-Erasmus M
    102. Youens-Clark K
    103. Zadissa A
    104. Zhang SJ
    105. Kasprzyk A

    (2015) The BioMart community portal: an innovative alternative to large, centralized data repositories

    Nucleic Acids Research 43:W589–W598.

    https://doi.org/10.1093/nar/gkv350

    • PubMed
    • Google Scholar
53. 1. Southan C
    2. Sharman JL
    3. Benson HE
    4. Faccenda E
    5. Pawson AJ
    6. Alexander SP
    7. Buneman OP
    8. Davenport AP
    9. McGrath JC
    10. Peters JA
    11. Spedding M
    12. Catterall WA
    13. Fabbro D
    14. Davies JA
    15. NC-IUPHAR

    (2016) The IUPHAR/BPS guide to PHARMACOLOGY in 2016: towards curated quantitative interactions between 1300 protein targets and 6000 ligands

    Nucleic Acids Research 44:D1054–D1068.

    https://doi.org/10.1093/nar/gkv1037

    • PubMed
    • Google Scholar
54. 1. Sprague ER

    (2017) ORCID

    Journal of the Medical Library Association 105:207–208.

    https://doi.org/10.5195/jmla.2017.89

    • Google Scholar
55. Software

    1. Stupp GS
    2. Su AI

    (2018) Drug indications extracted from FAERS

    Zenodo.

    http://doi.org/10.5281/zenodo.3620977
56. Software

    1. Stupp GS
    2. Waagmeester A
    3. Tsueng G
    4. Pico AR
    5. Tu R
    6. Ul-Hasan S
    7. Burgstaller-Muehlbacher S
    8. Riutta A
    9. Jacobson M
    10. Su AI

    (2020) SuLab/scheduled-bots: release v1.0 2020-01-21

    Zenodo.

    http://doi.org/10.5281/zenodo.1436000
57. 1. Sud M
    2. Fahy E
    3. Cotter D
    4. Brown A
    5. Dennis EA
    6. Glass CK
    7. Merrill AH
    8. Murphy RC
    9. Raetz CR
    10. Russell DW
    11. Subramaniam S

    (2007) LMSD: LIPID MAPS Structure Database

    Nucleic Acids Research 35:D527–D532.

    https://doi.org/10.1093/nar/gkl838

    • PubMed
    • Google Scholar
58. Software

    1. Tu R
    2. Stupp GS
    3. Su AI

    (2020) SuLab/Wikidata-phenomizer: release v1.0 on 2020-01-15

    Zenodo.

    http://doi.org/10.5281/zenodo.3609142
59. 1. Turki H
    2. Shafee T
    3. Hadj Taieb MA
    4. Ben Aouicha M
    5. Vrandečić D
    6. Das D
    7. Hamdi H

    (2019) Wikidata: a large-scale collaborative ontological medical database

    Journal of Biomedical Informatics 99:103292.

    https://doi.org/10.1016/j.jbi.2019.103292

    • PubMed
    • Google Scholar
60. 1. UniProt Consortium

    (2019) UniProt: a worldwide hub of protein knowledge

    Nucleic Acids Research 47:D506–D515.

    https://doi.org/10.1093/nar/gky1049

    • PubMed
    • Google Scholar
61. 1. Ursu O
    2. Holmes J
    3. Knockel J
    4. Bologa CG
    5. Yang JJ
    6. Mathias SL
    7. Nelson SJ
    8. Oprea TI

    (2017) DrugCentral: online drug compendium

    Nucleic Acids Research 45:D932–D939.

    https://doi.org/10.1093/nar/gkw993

    • PubMed
    • Google Scholar
62. 1. van Iersel MP
    2. Pico AR
    3. Kelder T
    4. Gao J
    5. Ho I
    6. Hanspers K
    7. Conklin BR
    8. Evelo CT

    (2010) The BridgeDb framework: standardized access to gene, protein and metabolite identifier mapping services

    BMC Bioinformatics 11:5.

    https://doi.org/10.1186/1471-2105-11-5

    • PubMed
    • Google Scholar
63. Conference

    1. Vrandečić D

    (2012) Wikidata: a new platform for collaborative data collection

    Proceedings of the 21st International Conference on World Wide Web.

    https://doi.org/10.1145/2187980.2188242

    • Google Scholar
64. 1. Wang Y
    2. Xiao J
    3. Suzek TO
    4. Zhang J
    5. Wang J
    6. Bryant SH

    (2009) PubChem: a public information system for analyzing bioactivities of small molecules

    Nucleic Acids Research 37:W623–W633.

    https://doi.org/10.1093/nar/gkp456

    • PubMed
    • Google Scholar
65. 1. Wang M
    2. Carver JJ
    3. Phelan VV
    4. Sanchez LM
    5. Garg N
    6. Peng Y
    7. Nguyen DD
    8. Watrous J
    9. Kapono CA
    10. Luzzatto-Knaan T
    11. Porto C
    12. Bouslimani A
    13. Melnik AV
    14. Meehan MJ
    15. Liu WT
    16. Crüsemann M
    17. Boudreau PD
    18. Esquenazi E
    19. Sandoval-Calderón M
    20. Kersten RD
    21. Pace LA
    22. Quinn RA
    23. Duncan KR
    24. Hsu CC
    25. Floros DJ
    26. Gavilan RG
    27. Kleigrewe K
    28. Northen T
    29. Dutton RJ
    30. Parrot D
    31. Carlson EE
    32. Aigle B
    33. Michelsen CF
    34. Jelsbak L
    35. Sohlenkamp C
    36. Pevzner P
    37. Edlund A
    38. McLean J
    39. Piel J
    40. Murphy BT
    41. Gerwick L
    42. Liaw CC
    43. Yang YL
    44. Humpf HU
    45. Maansson M
    46. Keyzers RA
    47. Sims AC
    48. Johnson AR
    49. Sidebottom AM
    50. Sedio BE
    51. Klitgaard A
    52. Larson CB
    53. P CAB
    54. Torres-Mendoza D
    55. Gonzalez DJ
    56. Silva DB
    57. Marques LM
    58. Demarque DP
    59. Pociute E
    60. O'Neill EC
    61. Briand E
    62. Helfrich EJN
    63. Granatosky EA
    64. Glukhov E
    65. Ryffel F
    66. Houson H
    67. Mohimani H
    68. Kharbush JJ
    69. Zeng Y
    70. Vorholt JA
    71. Kurita KL
    72. Charusanti P
    73. McPhail KL
    74. Nielsen KF
    75. Vuong L
    76. Elfeki M
    77. Traxler MF
    78. Engene N
    79. Koyama N
    80. Vining OB
    81. Baric R
    82. Silva RR
    83. Mascuch SJ
    84. Tomasi S
    85. Jenkins S
    86. Macherla V
    87. Hoffman T
    88. Agarwal V
    89. Williams PG
    90. Dai J
    91. Neupane R
    92. Gurr J
    93. Rodríguez AMC
    94. Lamsa A
    95. Zhang C
    96. Dorrestein K
    97. Duggan BM
    98. Almaliti J
    99. Allard PM
    100. Phapale P
    101. Nothias LF
    102. Alexandrov T
    103. Litaudon M
    104. Wolfender JL
    105. Kyle JE
    106. Metz TO
    107. Peryea T
    108. Nguyen DT
    109. VanLeer D
    110. Shinn P
    111. Jadhav A
    112. Müller R
    113. Waters KM
    114. Shi W
    115. Liu X
    116. Zhang L
    117. Knight R
    118. Jensen PR
    119. Palsson BO
    120. Pogliano K
    121. Linington RG
    122. Gutiérrez M
    123. Lopes NP
    124. Gerwick WH
    125. Moore BS
    126. Dorrestein PC
    127. Bandeira N

    (2016) Sharing and community curation of mass spectrometry data with global natural products social molecular networking

    Nature Biotechnology 34:828–837.

    https://doi.org/10.1038/nbt.3597

    • PubMed
    • Google Scholar
66. 1. Wilkinson MD
    2. Dumontier M
    3. Aalbersberg IJ
    4. Appleton G
    5. Axton M
    6. Baak A
    7. Blomberg N
    8. Boiten JW
    9. da Silva Santos LB
    10. Bourne PE
    11. Bouwman J
    12. Brookes AJ
    13. Clark T
    14. Crosas M
    15. Dillo I
    16. Dumon O
    17. Edmunds S
    18. Evelo CT
    19. Finkers R
    20. Gonzalez-Beltran A
    21. Gray AJ
    22. Groth P
    23. Goble C
    24. Grethe JS
    25. Heringa J
    26. 't Hoen PA
    27. Hooft R
    28. Kuhn T
    29. Kok R
    30. Kok J
    31. Lusher SJ
    32. Martone ME
    33. Mons A
    34. Packer AL
    35. Persson B
    36. Rocca-Serra P
    37. Roos M
    38. van Schaik R
    39. Sansone SA
    40. Schultes E
    41. Sengstag T
    42. Slater T
    43. Strawn G
    44. Swertz MA
    45. Thompson M
    46. van der Lei J
    47. van Mulligen E
    48. Velterop J
    49. Waagmeester A
    50. Wittenburg P
    51. Wolstencroft K
    52. Zhao J
    53. Mons B

    (2016) The FAIR guiding principles for scientific data management and stewardship

    Scientific Data 3:160018.

    https://doi.org/10.1038/sdata.2016.18

    • PubMed
    • Google Scholar
67. 1. Wilkinson MD
    2. Dumontier M
    3. Sansone SA
    4. Bonino da Silva Santos LO
    5. Prieto M
    6. Batista D
    7. McQuilton P
    8. Kuhn T
    9. Rocca-Serra P
    10. Crosas M
    11. Schultes E

    (2019) Evaluating FAIR maturity through a scalable, automated, community-governed framework

    Scientific Data 6:1–12.

    https://doi.org/10.1038/s41597-019-0184-5

    • PubMed
    • Google Scholar
68. 1. Williams AJ
    2. Grulke CM
    3. Edwards J
    4. McEachran AD
    5. Mansouri K
    6. Baker NC
    7. Patlewicz G
    8. Shah I
    9. Wambaugh JF
    10. Judson RS
    11. Richard AM

    (2017) The CompTox chemistry dashboard: a community data resource for environmental chemistry

    Journal of Cheminformatics 9:61.

    https://doi.org/10.1186/s13321-017-0247-6

    • PubMed
    • Google Scholar
69. Data

    1. Willighagen E
    2. Slenter D
    3. Mietchen D
    4. Evelo C
    5. Nielsen F

    (authors) (2018) Wikidata and scholia as a hub linking chemical knowledge

    Figshare.

    https://figshare.com/articles/Wikidata_and_Scholia_as_a_hub_linking_chemical_knowledge/6356027
70. 1. Wimalaratne SM
    2. Juty N
    3. Kunze J
    4. Janée G
    5. McMurry JA
    6. Beard N
    7. Jimenez R
    8. Grethe JS
    9. Hermjakob H
    10. Martone ME
    11. Clark T

    (2018) Uniform resolution of compact identifiers for biomedical data

    Scientific Data 5:180029.

    https://doi.org/10.1038/sdata.2018.29

    • PubMed
    • Google Scholar
71. 1. Wohlgemuth G
    2. Mehta SS
    3. Mejia RF
    4. Neumann S
    5. Pedrosa D
    6. Pluskal T
    7. Schymanski EL
    8. Willighagen EL
    9. Wilson M
    10. Wishart DS
    11. Arita M
    12. Dorrestein PC
    13. Bandeira N
    14. Wang M
    15. Schulze T
    16. Salek RM
    17. Steinbeck C
    18. Nainala VC
    19. Mistrik R
    20. Nishioka T
    21. Fiehn O

    (2016) SPLASH, a hashed identifier for mass spectra

    Nature Biotechnology 34:1099–1101.

    https://doi.org/10.1038/nbt.3689

    • PubMed
    • Google Scholar
72. 1. Xin J
    2. Afrasiabi C
    3. Lelong S
    4. Adesara J
    5. Tsueng G
    6. Su AI
    7. Wu C

    (2018) Cross-linking BioThings APIs through JSON-LD to facilitate knowledge exploration

    BMC Bioinformatics 19:30.

    https://doi.org/10.1186/s12859-018-2041-5

    • PubMed
    • Google Scholar
73. Preprint

    1. Xiong W
    2. Hoang T
    3. Wang WY

    (2017) DeepPath: a reinforcement learning method for knowledge graph reasoning

    arXiv.

    https://arxiv.org/abs/1707.06690

    • Google Scholar
74. 1. Zerbino DR
    2. Achuthan P
    3. Akanni W
    4. Amode MR
    5. Barrell D
    6. Bhai J
    7. Billis K
    8. Cummins C
    9. Gall A
    10. Girón CG
    11. Gil L
    12. Gordon L
    13. Haggerty L
    14. Haskell E
    15. Hourlier T
    16. Izuogu OG
    17. Janacek SH
    18. Juettemann T
    19. To JK
    20. Laird MR
    21. Lavidas I
    22. Liu Z
    23. Loveland JE
    24. Maurel T
    25. McLaren W
    26. Moore B
    27. Mudge J
    28. Murphy DN
    29. Newman V
    30. Nuhn M
    31. Ogeh D
    32. Ong CK
    33. Parker A
    34. Patricio M
    35. Riat HS
    36. Schuilenburg H
    37. Sheppard D
    38. Sparrow H
    39. Taylor K
    40. Thormann A
    41. Vullo A
    42. Walts B
    43. Zadissa A
    44. Frankish A
    45. Hunt SE
    46. Kostadima M
    47. Langridge N
    48. Martin FJ
    49. Muffato M
    50. Perry E
    51. Ruffier M
    52. Staines DM
    53. Trevanion SJ
    54. Aken BL
    55. Cunningham F
    56. Yates A
    57. Flicek P

    (2018) Ensembl 2018

    Nucleic Acids Research 46:D754–D761.

    https://doi.org/10.1093/nar/gkx1098

    • PubMed
    • Google Scholar

Thank you for submitting your article "Wikidata as a FAIR knowledge graph for the life sciences" for consideration by eLife. Your article has been reviewed by two peer reviewers, and the evaluation has been overseen by the eLife Features Editor (Peter Rodgers). The following individuals involved in review of your submission have agreed to reveal their identity: Chris Mungall (Reviewer #3).

The reviewers have discussed the reviews with one another and the Features Editor has drafted this decision to help you prepare a revised submission. We hope you will be able to submit the revised version within two months.

SUMMARY

The manuscript describes the life sciences component of the Wikidata knowledge base, which combines multiple disaggregated knowledge/databases into a single source enabling integrated querying. The authors describe the process for keeping Wikidata up to date, and they describe compelling use cases showing the power. Although there are many efforts in the life sciences to create integrated knowledge bases / knowledge graphs, Wikidata is unique in the breadth, scope, and in the community/crowdsourcing aspects. It is an increasingly important resource in the biomedical landscape, and the manuscript provides a clear description of the authors' significant efforts in building this component of the resource. However, the manuscript would benefit from addressing a number of issues in greater detail - see below.

ESSENTIAL REVISIONS:

1. An argument is made that Wikidata combines centralized and distributed approaches. This is an interesting concept. It would be important to know the relative importance of those two approaches. In other words, what proportion of edits is centralized (bots) and what proportion is distributed (contributors). Authors repeatedly refer to the fact that anyone is empowered to add new content, but how much of this potential is realized, and how this compares to other Wiki projects, e.g., Wikipedia?

2. Performance metrics and measurable assessment are often missing from the description of the applications. For example, when the process of translation of identifiers from various databases is described, no information is provided if the translation is done deterministically or probabilistically, how it is done algorithmically, how the performance can be evaluated (AUC ROC?), how Wikidata compares to other systems (e.g., UMLS). It is admirable that the authors admit that "Wikidata is certainly not complete with respect to identifier mappings...", but readers would benefit from knowing more about the proportion of mappings available and those not, and how this differs across databases integrated within Wikidata.

3. In such a broad overview, including multiple use cases, some selectivity in the presentation of examples is probably inevitable. This is understandable. Yet one can make those choices more informed, and offer, in addition, examples less favorable to the system (Wikidata), and in this way provide a balanced assessment. For example, on pages 10-11, for the purpose of disease diagnosis, annotations from the Human Phenotype Ontology (HPO) were compared to combined HPO + Wikidata annotations. However, comparison to only one set of annotations with respect to only one disease could be uninformative. At least, performance should be evaluated against a set of different randomly selected diseases.

4. An argument is made that the "Wikidata is among the most FAIR biomedical resources available". What are the other resources, and how Wikidata compares to those resources with respect to the four FAIR criteria?

5. The FAIR principles do not address the issue of completeness, as they were designed for datasets which are inherently self-contained and presumably complete with respect to the experiment/study. For biomedical knowledge bases (KBs), especially warehouse-style, a key question is: how complete is the KB? How often will my queries have false negatives? (Of course, KBs are always incomplete due to both knowledge and curation gaps. However, it is still important to provide users a sense of the areas in which there are major gaps, for example due to inability to integrate a key resource due to licensing).

An example is on figure 1: there are only 636 disease to symptom/phenotype associations. This is a very small amount, when compared to reference resources like the Human Phenotype Ontology. This may have negative consequences for applications built around Wikidata, causing people to lose trust. Another example is in the subset of genes imported - from 200 species. What determines the set of species included? Are there performance implications in including all genes in all species?

Of course, this is a difficult challenge for any KB, and completeness is not expected. I would still recommend a section that acknowledges and addresses this, addressing questions such as:

- how is a user best able to determine whether a portion of Wikidata is complete enough for their use case

- what is the process for deciding what is included vs excluded?

6. Identifier merging details: The identifier mapping use case is well articulated, but some details are omitted. Some identifier mapping resources store pairwise mappings without privileging any one resource. Wikidata is more like UMLS, in which they mint a new concept ID for the unified concept, and map everything to this. The challenge with this scheme is deciding on the criteria for lumping and splitting. Many source mappings are not 1:1, which can lead to excessive merging when these are traversed transitively

- For bot-derived entities, what is the algorithm for doing this?

- Is one source (e.g. NCBI gene) taken as canonical?

Additionally, it would be useful to see a stricter comparison with sources like bridgedb, in terms of completeness and consistency.

7. Trustworthiness of curation: The strength of Wikidata is in the crowdsourcing of knowledge. While this can be scaled up more easily due less of a need for funded curators, the downside is that the information may be less accurate and reliable. The paper provides good evidence of reliability via the disease cross-referencing experiment, in which 99% of 2030 crowd-sourced mappings were reviewed and accepted.

I am a little skeptical of the generalizability of these results. Mappings are generally quite easy and can be done with reasonable reliability by an automated string-matching process (with some caveats). It would be useful to know more about how many different people contributed to the 2030 (was it one person running a script)?

Refinements to the ontology structure are harder, and it's not clear how often these were incorporated. The mapping results are still a nice example, but they need to be qualified more, and there should be more discussion on reliability.

8. Phenomizer analysis: It's not clear if there is circularity in the Phenomizer analysis:

- were there publications that incorporated the case reports that were then annotated? Was this controlled for?

- What happened on 2018-09 when the HPO-only semsim score jumps? Were the WD annotations incorporated into HPO? Or was this independent annotation? The subsequent drop is curious.

9. Some lines read as if written for a grant proposal, e.g., "Wikidata has a proven track record for leveraging...", "Wikimedia Foundation... has a long track record of developing and maintaining..." see Introduction. In many instances, Wikidata is presented as "unique". Please reword such sentences/passages.

CODE/DATA

The Phenomizer analysis can't be reproduced at the moment, as the curated case reports with HPO IDs are not made available. Additionally, the settings for Phenomizer should be made available, also Phenomizer provides p-values, it would be useful to see this in the analysis results.

We thank you and the reviewers for the helpful and detailed evaluation of our manuscript. We have made many changes in response to these comments, and believe the manuscript is substantially strengthened as a result. A point-by-point description of these changes is included below. We repeat the reviewers’ points here in italic, followed by our response in plain text.

SUMMARY

The manuscript describes the biomedical life sciences component of the Wikidata knowledge base, which combines multiple disaggregated knowledge/databases into a single source enabling integrated querying. The authors describe the process for keeping Wikidata up to date, and they describe compelling use cases showing the power. Although there are many efforts in the life sciences to create integrated knowledge bases / knowledge graphs, Wikidata is unique in the breadth, scope, and in the community/crowdsourcing aspects. It is an increasingly important resource in the biomedical landscape, and the manuscript provides a clear description of the authors' significant efforts in building this component of the resource. However, the manuscript would benefit from addressing a number of issues in greater detail - see below.

ESSENTIAL REVISIONS:

1. An argument is made that Wikidata combines centralized and distributed approaches. This is an interesting concept. It would be important to know the relative importance of those two approaches. In other words, what proportion of edits is centralized (bots) and what proportion is distributed (contributors). Authors repeatedly refer to the fact that anyone is empowered to add new content, but how much of this potential is realized, and how this compares to other Wiki projects, e.g., Wikipedia?

The Wikimedia Statistics tracker shows that over the last two years, 46% of all edits on Wikidata were attributed to normal "user" accounts while ~54% are attributed to accounts that are registered as bots, with the trend over the last year showing an increasing trend toward non-bot user edits ((Wikimedia Foundation, 2020); Figure 1-figure supplement 1 in revised manuscript). For comparison, that same site reports that English Wikipedia had 69% user edits and 14% bot edits (and an additional 17% anonymous edits).

In addition to the statistics above that demonstrate a balance between human and bot edits, the absolute number of editors is noteworthy. The number of "active editors" (five or more edits per month) has steadily grown from 10k to 12k over the last two years, with the number of bot accounts staying relatively stable over that time.

Finally, we also note that while Wikidata bots tend to focus on importing large-scale 'centralized' resources, the bot community itself is made up of the decentralized and distributed efforts of many bot developers.

These points have been clarified in the manuscript (including the addition of a new Figure 1-figure supplement 1).

2. Performance metrics and measurable assessment are often missing from the description of the applications. For example, when the process of translation of identifiers from various databases is described, no information is provided if the translation is done deterministically or probabilistically, how it is done algorithmically, how the performance can be evaluated (AUC ROC?), how Wikidata compares to other systems (e.g., UMLS). It is admirable that the authors admit that "Wikidata is certainly not complete with respect to identifier mappings...", but readers would benefit from knowing more about the proportion of mappings available and those not, and how this differs across databases integrated within Wikidata.

It is important to note that Wikidata is designed to be a database of assertions (with provenance) and not a database that attempts to resolve disagreements in search of absolute truth. Therefore, all identifier mappings in Wikidata are added deterministically – each Wikidata statement is an assertion from a database or curation authority. If different mapping resources disagree, then Wikidata can and should reflect that disagreement. Each consumer of Wikidata content is then free to add filters to prioritize or ignore specific mappings according to their own rules and biases. However, we also recognize that identifiers are a special class of statements because they establish the entities that are the subjects/objects of other statements. We provide more discussion of these topics below in response to point #6.

3. In such a broad overview, including multiple use cases, some selectivity in the presentation of examples is probably inevitable. This is understandable. Yet one can make those choices more informed, and offer, in addition, examples less favorable to the system (Wikidata), and in this way provide a balanced assessment. For example, on pages 10-11, for the purpose of disease diagnosis, annotations from the Human Phenotype Ontology (HPO) were compared to combined HPO + Wikidata annotations. However, comparison to only one set of annotations with respect to only one disease could be uninformative. At least, performance should be evaluated against a set of different randomly selected diseases.

Our goal was not to conclusively state that HPO + Wikidata is uniformly better for Phenomizer analyses. We acknowledge that the NGLY1 deficiency example was specifically selected to demonstrate a favorable case, and that the benefit of Wikidata would likely not currently generalize to a random sampling of other diseases. In this manuscript, we have attempted to balance examples in which Wikidata already has a rigorously-proven benefit to researchers, and proof-of-concept examples that demonstrate the value of the crowd-based Wikidata model and that motivate community contributions. Phenotype-based disease classification falls into the latter category. We have clarified this point in the manuscript.

4. An argument is made that the "Wikidata is among the most FAIR biomedical resources available". What are the other resources, and how Wikidata compares to those resources with respect to the four FAIR criteria?

Our intent is primarily to describe how Wikidata aligns with the FAIR data principles, and not to perform a comparative analysis of FAIRness relative to other biomedical resources. Therefore, we have reworded the text accordingly.

5. The FAIR principles do not address the issue of completeness, as they were designed for datasets which are inherently self-contained and presumably complete with respect to the experiment/study. For biomedical knowledge bases (KBs), especially warehouse-style, a key question is: how complete is the KB? How often will my queries have false negatives? (Of course, KBs are always incomplete due to both knowledge and curation gaps. However, it is still important to provide users a sense of the areas in which there are major gaps, for example due to inability to integrate a key resource due to licensing).

An example is on figure 1: there are only 636 disease to symptom/phenotype associations. This is a very small amount, when compared to reference resources like the Human Phenotype Ontology. This may have negative consequences for applications built around Wikidata, causing people to lose trust. Another example is in the subset of genes imported - from 200 species. What determines the set of species included? Are there performance implications in including all genes in all species?

Of course, this is a difficult challenge for any KB, and completeness is not expected. I would still recommend a section that acknowledges and addresses this, addressing questions such as:

- how is a user best able to determine whether a portion of Wikidata is complete enough for their use case

- what is the process for deciding what is included vs excluded?

There are many important issues raised in this point. Our overarching strategy for conveying the completeness is the class diagram in Figure 1. We believe that that Figure contains key information on the number of each entity type and the numbers of relationships between entity types. This information is relevant for each user to assess whether the appropriate data exist within Wikidata for their particular use case.

What is not included in Figure 1 are denominators – estimates of the total number of nodes/edges of a given type. Those denominators are extremely difficult to provide, since there are many reasonable ways to define them. For example, they could be based on resources that have been identified and screened but not yet loaded, based on resources that have been identified but are not suitable for Wikidata (e.g., for licensing reasons), based on an estimate of all structured knowledge (and/or unstructured knowledge) that is currently known, or based on an estimate of all knowledge that will eventually be known and discovered. We believe that the wide differences in how those denominators would be interpreted (and significant challenges in computing them) would counter any benefit to potential users. Instead, we have added a new supplemental table that shows the data sources that are cited as references for the most common properties. While this is not exactly the information that was requested by the reviewers, we believe this information will be more clearly interpretable.

A related question is raised here about how we prioritize resources for inclusion in Wikidata. In principle, we could write bots to load nearly any suitable data resources we come across. However, the bot writing process is actually a small proportion of the overall effort necessary to load a new resource. Each additional data resource requires a time-consuming data modeling process (in coordination with the broader Wikidata community and often the maintainers of the resource to be imported), adds overhead to our bot maintenance requirements, and introduces complexity in our data synchronization routines. Therefore, we avoid indiscriminate "stamp collecting" efforts in favor of targeted data loading driven by our own data-mining priorities. Accordingly, readers should feel empowered to import new data of interest to them, as that has been the primary mechanism for Wikidata's growth.

These points have been clarified in the manuscript.

6. Identifier merging details: The identifier mapping use case is well articulated, but some details are omitted. Some identifier mapping resources store pairwise mappings without privileging any one resource. Wikidata is more like UMLS, in which they mint a new concept ID for the unified concept, and map everything to this. The challenge with this scheme is deciding on the criteria for lumping and splitting. Many source mappings are not 1:1, which can lead to excessive merging when these are traversed transitively.

- For bot-derived entities, what is the algorithm for doing this?

- Is one source (e.g. NCBI gene) taken as canonical?

Wikidata bots generally do take one identifier as canonical (e.g., NCBI gene for genes, UniProt for proteins), but there is no requirement that other bots nor human editors use that same canonical identifier. Wikidata is based on a community process for harmonizing disconnected data resources. That means that any differing views on identifier mappings, like all statements on Wikidata items, are resolved via community discussion and consensus. Community input is solicited at many levels, starting at the approval process for bots to import large resources, all the way to discussions over individual statements on individual Wikidata items.

So while the process for resolving identifier mappings is the same as for any other Wikidata statements, we recognize that choices in identifier mappings have greater downstream implications (related to, for example, transitivity). However, the community-focused design of Wikidata means that there is no systematic scheme for making consistent lumping and splitting decisions. We recognize (and the original Wikidata creators recognized) that this design choice is not without its flaws, but also that it does have significant advantages with respect to usability at query time.

Finally, it is important to recognize that the issue of lumping and splitting is not specific to Wikidata, but one that exists across the biomedical knowledge management community. This issue has been exacerbated by the widespread use of semantically-imprecise cross references ('hasDbXref') in biomedical ontologies (as explained in this detailed blog post from Chris Mungall (Mungall, 2019)). So while we agree that this issue is not completely solved within Wikidata, we believe that the core issue lies further upstream in the knowledge management ecosystem. (If, for example, source databases and ontologies differentiated exact cross-references from non-exact cross references, Wikidata would be able to more precisely model these relationships.)

We have clarified these points in the manuscript.

Additionally, it would be useful to see a stricter comparison with sources like bridgedb, in terms of completeness and consistency.

We believe that a quantitative comparison to other mapping resources like BridgeDb would not be meaningful for a variety of reasons. First, both Wikidata and BridgeDb aggregate mappings from other 'authoritative' community resources (e.g., Ensembl for genes and proteins, ChEBI for chemicals, etc.). So a comparison between Wikidata and BridgeDb would really boil down to a comparison of the resources that were imported into each system. Second, for chemical compounds, BridgeDb actually imports mappings from Wikidata because it has a clear and transparent data model, and it allows people to fix inconsistencies and add missing content. Third, the scope of these resources is drastically different. BridgeDb currently focuses on genes, proteins, metabolic reactions, and metabolites, while Wikidata also includes many additional closely-related entity types (e.g., variants, diseases, organisms) as well as many more distantly-related types (e.g. clinical trials, people, countries).

7. Trustworthiness of curation: The strength of Wikidata is in the crowdsourcing of knowledge. While this can be scaled up more easily due less of a need for funded curators, the downside is that the information may be less accurate and reliable. The paper provides good evidence of reliability via the disease cross-referencing experiment, in which 99% of 2030 crowd-sourced mappings were reviewed and accepted.

I am a little skeptical of the generalizability of these results. Mappings are generally quite easy and can be done with reasonable reliability by an automated string-matching process (with some caveats). It would be useful to know more about how many different people contributed to the 2030 (was it one person running a script)?

Refinements to the ontology structure are harder, and it's not clear how often these were incorporated. The mapping results are still a nice example, but they need to be qualified more, and there should be more discussion on reliability.

We agree that identifier mapping represents the most common community contribution, and likely the most accessible crowdsourcing task. And the majority of the disease identifier mappings were performed by a handful of editors through the automated Mix'n'Match ID mapping interface (currently seven users with > 20 mappings). So while disease mapping through Mix'n'Match is skewed to a relatively small number of editors, the broader identifier mapping activities across all of Mix'n'Match is distributed across a much larger set of users, each with expertise and/or interest in a different area.

Regarding automated mapping via string matching, we agree that in many instances, automated methods are effective. However, in the case of the 2030 proposed MeSH and GARD mappings we reported, 771 (38%) were based on something other than simple string matching. We believe this supports the idea that crowdsourcing human mappings from Wikidata is a useful bridge between automated methods and expert curation.

We chose to focus on identifier mapping because the evaluation of accuracy by expert curators was relatively straightforward. We agree the refinements to ontology structure and the addition of other statements are likely to be more difficult and error-prone, and as we allude to in the manuscript, much more difficult to quantitatively evaluate for accuracy.

We have clarified all of these points in the manuscript.

8. Phenomizer analysis: It's not clear if there is circularity in the Phenomizer analysis:

- were there publications that incorporated the case reports that were then annotated? Was this controlled for?

- What happened on 2018-09 when the HPO-only semsim score jumps? Were the WD annotations incorporated into HPO? Or was this independent annotation? The subsequent drop is curious.

As part of a separate manuscript focused on NGLY1 deficiency (in revision), we curated disease-related phenotypes from two papers by Enns et al. (Enns et al., 2014) and Lam et al. (Lam et al., 2017, p. 1). These phenotypes were added as statements in Wikidata. In our analysis for this paper, we extracted phenotypes from suspected cases of CDDG / NGLY1-deficiency described in a third paper by Caglayan et al. (Caglayan et al., 2015) and used those as input to the Phenomizer analysis. So while our team was responsible for the majority of curation of phenotypes associated with NGLY1 deficiency, there was no "circularity" in the sense that different phenotype sets were used as the Phenomizer input and Wikidata additions. We have revised the manuscript to clarify this point.

After curation of the Enns et al. and Lam et al. papers, we also submitted those annotations for review and curation by curators with the Human Phenotype Ontology (https://github.com/obophenotype/human-phenotype-ontology/issues/3287). Based on annotations contributed by us and others, new phenotypes associated with CDDG / NGLY1-deficiency were added over the period shown in Figure 4. The sharp increase in the probability score around 2018-09 was primarily due to the addition of corneal ulceration (HP:0012804), which is a phenotype that is relatively specific for CDDG / NGLY1-deficiency and one of the phenotypes from Caglayan et al. that served as an input to the analysis. While there is a slight decline in the HPO-only probability score after 2018-09, we do not believe that reflects substantial changes to the underlying phenotypes annotated to CDDG / NGLY1-deficiency.

9. Some lines read as if written for a grant proposal, e.g., "Wikidata has a proven track record for leveraging...", "Wikimedia Foundation... has a long track record of developing and maintaining..." see Introduction. In many instances, Wikidata is presented as "unique". Please reword such sentences/passages.

We believe that the substance of those sentences is both correct and important to emphasize to readers for several reasons. First, many initiatives claim to be based on "crowdsourcing", but the vast majority fail to build and/or sustain a critical mass of users. It is noteworthy that the Wikimedia Foundation (WMF) has been successful at this on multiple occasions, and that Wikidata is one such example of a mature community-based resource.Second, the bioinformatics community continually generates innovative new tools, but the scalability of these tools are rarely tested with large numbers of users and large datasets. The WMF runs several online information resources (including Wikipedia and Wikidata) that run at a high level of performance and availability, which are important features for information management infrastructure. Third, Wikidata is sustained by funding streams that are different from the vast majority of biomedical resources (which are mostly funded by the NIH). Insulation from the 4-5 year funding cycles that are typical of NIH-funded biomedical resources does make Wikidata quite unique.

We have reworded several sections to hopefully strike a better balance between factual observation and advocacy. However, the intent of these sentences was not (and is not) to gratuitously praise Wikidata, but to convey to the reader that Wikidata is different than other online biomedical resources in several fundamental ways. We believe these points are important to communicate to readers as they decide how much effort to devote to using and/or contributing to Wikidata.

CODE/DATA

The Phenomizer analysis can't be reproduced at the moment, as the curated case reports with HPO IDs are not made available. Additionally, the settings for Phenomizer should be made available, also Phenomizer provides p-values, it would be useful to see this in the analysis results.

On more careful examination, we realized that we were imprecise with how we described our analysis. The published version of Phenomizer (Köhler et al., 2009) and the original Phenomizer web tool (http://compbio.charite.de/phenomizer/) are based on an "ontological similarity search", which includes the computation of a p-value. The most recent version of Phenomizer ("Orphamizer"; http://compbio.charite.de/phenomizer_orphanet/) is based on a newer algorithm for Bayesian network inference (Bauer et al., 2012) and a software package called BOQA (Köhler, 2020), which outputs probability scores. Our results are based on a BOQA analysis (with and without phenotype annotations from Wikidata). We have updated the manuscript text accordingly.

With that clarification, a Jupyter notebook that demonstrates the entire BOQA analysis from raw files to published figure has been added to our previously-cited Github repository: https://github.com/SuLab/Wikidata-phenomizer/ (Tu et al., 2019) (now also archived at (Tu et al., 2020)).

References

Bauer S, Köhler S, Schulz MH, Robinson PN. 2012. Bayesian ontology querying for accurate and noise-tolerant semantic searches. Bioinforma Oxf Engl 28:2502–2508. doi:10.1093/bioinformatics/bts471

Caglayan AO, Comu S, Baranoski JF, Parman Y, Kaymakçalan H, Akgumus GT, Caglar C, Dolen D, Erson-Omay EZ, Harmanci AS, Mishra-Gorur K, Freeze HH, Yasuno K, Bilguvar K, Gunel M. 2015. NGLY1 mutation causes neuromotor impairment, intellectual disability, and neuropathy. Eur J Med Genet 58:39–43. doi:10.1016/j.ejmg.2014.08.008

Enns GM, Shashi V, Bainbridge M, Gambello MJ, Zahir FR, Bast T, Crimian R, Schoch K, Platt J, Cox R, Bernstein JA, Scavina M, Walter RS, Bibb A, Jones M, Hegde M, Graham BH, Need AC, Oviedo A, Schaaf CP, Boyle S, Butte AJ, Chen Rui, Chen Rong, Clark MJ, Haraksingh R, FORGE Canada Consortium, Cowan TM, He P, Langlois S, Zoghbi HY, Snyder M, Gibbs RA, Freeze HH, Goldstein DB. 2014. Mutations in NGLY1 cause an inherited disorder of the endoplasmic reticulum-associated degradation pathway. Genet Med Off J Am Coll Med Genet 16:751–758. doi:10.1038/gim.2014.22

Köhler S. 2020. BOQA | Phenomics and Machine Learning @ Berlin. https://phenomics.github.io/software-boqa.html

Köhler S, Schulz MH, Krawitz P, Bauer S, Dölken S, Ott CE, Mundlos C, Horn D, Mundlos S, Robinson PN. 2009. Clinical diagnostics in human genetics with semantic similarity searches in ontologies. Am J Hum Genet 85:457–464. doi:10.1016/j.ajhg.2009.09.003

Lam C, Ferreira C, Krasnewich D, Toro C, Latham L, Zein WM, Lehky T, Brewer C, Baker EH, Thurm A, Farmer CA, Rosenzweig SD, Lyons JJ, Schreiber JM, Gropman A, Lingala S, Ghany MG, Solomon B, Macnamara E, Davids M, Stratakis CA, Kimonis V, Gahl WA, Wolfe L. 2017. Prospective phenotyping of NGLY1-CDDG, the first congenital disorder of deglycosylation. Genet Med Off J Am Coll Med Genet 19:160–168. doi:10.1038/gim.2016.75

Mungall CJ. 2019. Never mind the logix: taming the semantic anarchy of mappings in ontologies. Monkeying OWL. https://douroucouli.wordpress.com/2019/05/27/never-mind-the-logix-taming-the-semantic-anarchy-of-mappings-in-ontologie/

Tu R, Stupp GS, Su AI. 2020. SuLab/Wikidata-phenomizer: Release v1.0 on 2020-01-15. Zenodo. doi:10.5281/zenodo.3609142

Tu R, Stupp GS, Su AI. 2019. SuLab/Wikidata-phenomizer 7b25781. https://github.com/SuLab/Wikidata-phenomizer

Wikimedia Foundation. 2020. Wikimedia Statistics - Wikidata - Edits. https://stats.wikimedia.org/v2/#/wikidata.org/contributing/edits/normal|line|2-year|editor_type~anonymous*group-bot*name-bot*user|monthly

Author details

1. Andra Waagmeester

   Andra Waagmeester is at Micelio, Antwerp, Belgium

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Visualization, Writing - review and editing

   Contributed equally with

   Gregory Stupp

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-9773-4008

2. Gregory Stupp

   Gregory Stupp is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Visualization, Writing - original draft

   Contributed equally with

   Andra Waagmeester

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0644-7212

3. Sebastian Burgstaller-Muehlbacher

   Sebastian Burgstaller-Muehlbacher is in the Center for Integrative Bioinformatics Vienna, Max Perutz Laboratories, University of Vienna and Medical University of Vienna, Vienna, Austria

   Contribution

   Conceptualization, Data curation, Software

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4640-3510

4. Benjamin M Good

   Benjamin M Good is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

   Contribution

   Conceptualization, Data curation, Software, Supervision, Project administration

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-7334-7852

5. Malachi Griffith

   Malachi Griffith is in the McDonnell Genome Institute, Washington University School of Medicine, St. Louis, MO, United States

   Contribution

   Data curation, Funding acquisition

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-6388-446X

6. Obi L Griffith

   Obi L Griffith is in the McDonnell Genome Institute, Washington University School of Medicine, St. Louis, MO, United States

   Contribution

   Data curation, Funding acquisition, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0843-4271

7. Kristina Hanspers

   Kristina Hanspers is in the Institute of Data Science and Biotechnology, Gladstone Institutes, San Francisco, CA, United States

   Contribution

   Data curation, Software

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5410-599X

8. Henning Hermjakob

   Henning Hermjakob is at the European Bioinformatics Institute (EMBL-EBI), Hinxton, United Kingdom

   Contribution

   Data curation, Supervision, Funding acquisition

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8479-0262

9. Toby S Hudson

   Toby S Hudson is in the School of Chemistry, University of Sydney, Sydney, Australia

   Contribution

   Data curation, Validation

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-3348-3622

10. Kevin Hybiske

    Kevin Hybiske is in the Division of Allergy and Infectious Diseases, Department of Medicine, University of Washington, Seattle, WA, United States

    Contribution

    Data curation, Funding acquisition

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-2967-3079

11. Sarah M Keating

    Sarah M Keating is at the European Bioinformatics Institute (EMBL-EBI), Hinxton, United Kingdom

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-3356-3542

12. Magnus Manske

    Magnus Manske is at the Wellcome Trust Sanger Institute, Hinxton, United Kingdom

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-5916-0947

13. Michael Mayers

    Michael Mayers is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Data curation, Software, Formal analysis, Visualization

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-7792-0150

14. Daniel Mietchen

    Daniel Mietchen is in the School of Data Science, University of Virginia, Charlottesville, VA, United States

    Contribution

    Data curation, Validation, Writing - review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-9488-1870

15. Elvira Mitraka

    Elvira Mitraka is in the University of Maryland School of Medicine, Baltimore, MD, United States

    Contribution

    Data curation, Validation

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-0719-3485

16. Alexander R Pico

    Alexander R Pico is in the Institute of Data Science and Biotechnology, Gladstone Institutes, San Francisco, CA, United States

    Contribution

    Data curation, Supervision, Writing - original draft, Writing - review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-5706-2163

17. Timothy Putman

    Timothy Putman is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-4291-0737

18. Anders Riutta

    Anders Riutta is in the Institute of Data Science and Biotechnology, Gladstone Institutes, San Francisco, CA, United States

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-4693-0591

19. Nuria Queralt-Rosinach

    Núria Queralt-Rosinach is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-0169-8159

20. Lynn M Schriml

    Lynn M Schriml is in the University of Maryland School of Medicine, Baltimore, MD, United States

    Contribution

    Data curation, Supervision, Funding acquisition

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-8910-9851

21. Thomas Shafee

    Thomas Shafee is in the Department of Animal Plant and Soil Sciences, La Trobe University, Melbourne, Australia

    Contribution

    Data curation, Visualization, Writing - review and editing

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-2298-7593

22. Denise Slenter

    Denise Slenter is in the Department of Bioinformatics-BiGCaT, NUTRIM, Maastricht University, Maastricht, Netherlands

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-8449-1318

23. Ralf Stephan

    Ralf Stephan is a retired researcher based in Berlin, Germany

    Contribution

    Data curation, Validation

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-4650-631X

24. Katherine Thornton

    Katherine Thornton is at Yale University Library, Yale University, New Haven, CT, United States

    Contribution

    Software, Validation

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-4499-0451

25. Ginger Tsueng

    Ginger Tsueng is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-9536-9115

26. Roger Tu

    Roger Tu is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Software, Formal analysis, Visualization

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-7899-1604

27. Sabah Ul-Hasan

    Sabah Ul-Hasan is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-6334-452X

28. Egon Willighagen

    Egon Willighagen is in the Department of Bioinformatics-BiGCaT, NUTRIM, Maastricht University, Maastricht, Netherlands

    Contribution

    Data curation, Software

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-7542-0286

29. Chunlei Wu

    Chunlei Wu is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Data curation, Software, Supervision, Funding acquisition

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-2629-6124

30. Andrew I Su

    Andrew I Su is in the Department of Integrative Structural and Computational Biology, The Scripps Research Institute, La Jolla, CA, United States

    Contribution

    Conceptualization, Formal analysis, Supervision, Funding acquisition, Validation, Writing - original draft, Project administration, Writing - review and editing

    For correspondence

    asu@scripps.edu

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-9859-4104

• 9,506

  views

• 558

  downloads

• 90

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.