Recent estimates suggest paywalls on the web limit access to three-quarters of scholarly literature (Piwowar et al., 2018; Khabsa et al., 2014; Bosman and Kramer, 2018). The open access movement strives to remedy this situation (Tennant et al., 2016). After decades of effort by the open access community (Royster, 2016), nearly 50% of newly published articles are available without paywalls (Piwowar et al., 2018; Archambault et al., 2014; Van Noorden, 2013a).

Despite these gains, access to scholarly literature remains a pressing global issue. Foremost, widespread subscription access remains restricted to institutions, such as universities or medical centers. Smaller institutions or those in the developing world often have poor access to scholarly literature (Meadows, 2015; Kirsop and Chan, 2005; Bendezú-Quispe et al., 2016). As a result, only a tiny percentage of the world’s population has been able to access much of the scholarly literature, despite the fact that the underlying research was often publicly or philanthropically funded. Compounding the problem is that publications have historically been the primary, if not sole, output of scholarship. Although copyright does not apply to ideas, journals leverage the copyright covering an article’s prose, figures, and typesetting to effectively paywall its knowledge.

Since each article is unique, libraries cannot substitute one journal subscription for another without depriving their users of potentially crucial access. As a result, the price of journal subscriptions has grown at a faster rate than inflation for several decades (Association of Research Libraries, 2017), leading to an ever-present “serials crisis” that has pushed library budgets to their brink while diverting funds from other services (Roth, 1990). Meanwhile, publishing has trended towards oligopoly (Larivière et al., 2015), with nondisclosure clauses obfuscating price information among subscribers (Bergstrom et al., 2014) while publishers profit immensely (Morrison, 2012; Buranyi, 2017; Van Noorden, 2013b). Price increases have persisted over the last decade (Bosch and Henderson, 2017; Lawson et al., 2015; Lawson, 2017a). For example, EBSCO estimates that per-journal subscription costs increased by 25% from 2013–2017, with annual subscription to a journal for research libraries now averaging $1,396 (EBSCO, 2017).

In this study, we use the term “toll access” (also known as “closed access”) to refer to paywalled literature (Suber, 2017). On the other hand, we refer to literature that is free to read as “open access”. Furthermore, we discuss two variants of open access: “libre” and “gratis” (Suber, 2017; Suber, 2008). Libre open access refers to literature that is openly licensed to allow reuse. Gratis open access refers to literature that is accessible free of charge, although permission barriers may remain (usually due to copyright) (Himmelstein, 2016).

The website Sci-Hub, now in its sixth year of existence, provides gratis access to scholarly literature, despite the continued presence of paywalls. Sci-Hub brands itself as “the first pirate website in the world to provide mass and public access to tens of millions of research papers.” The website, started in 2011, is run by Alexandra Elbakyan, a graduate student and native of Kazakhstan who now resides in Russia (Bohannon, 2016a; Schiermeier, 2015). Elbakyan describes herself as motivated to provide universal access to knowledge (Elbakyan, 2016a; Elbakyan, 2015; Milova, 2017).

Sci-Hub does not restrict itself to only openly licensed content. Instead, it retrieves and distributes scholarly literature without regard to copyright. Readers should note that, in many jurisdictions, use of Sci-Hub may constitute copyright infringement. Users of Sci-Hub do so at their own risk. This study is not an endorsement of using Sci-Hub, and its authors and publishers accept no responsibility on behalf of readers. There is a possibility that Sci-Hub users — especially those not using privacy-enhancing services such as Tor — could have their usage history unmasked and face legal or reputational consequences.

Sci-Hub is currently served at domains including https://sci-hub.hk, https://sci-hub.la, https://sci-hub.mn, https://sci-hub.name, https://sci-hub.tv, and https://sci-hub.tw, as well as at scihub22266oqcxt.onion — a Tor Hidden Service (Dingledine et al., 2004). Elbakyan described the project’s technical scope in July 2017 (Elbakyan, 2017): “Sci-Hub technically is by itself a repository, or a library if you like, and not a search engine for some other repository. But of course, the most important part in Sci-Hub is not a repository, but the script that can download papers closed behind paywalls.”

One method Sci-Hub uses to bypass paywalls is by obtaining leaked authentication credentials for educational institutions (Elbakyan, 2017). These credentials enable Sci-Hub to use institutional networks as proxies and gain subscription journal access. While the open access movement has progressed slowly (Björk, 2017), Sci-Hub represents a seismic shift in access to scholarly literature. Since its inception, Sci-Hub has experienced sustained growth, with spikes in interest and awareness driven by legal proceedings, service outages, news coverage, and social media (Figure 1 and Figure 1—figure supplement 1). Here we investigate the extent to which Sci-Hub provides access to scholarly literature. If Sci-Hub’s coverage is sufficiently broad, then a radical shift may be underway in how individuals access scholarly literature.

Figure 1 with 1 supplement see all

Download asset Open asset

In Figure 1, The letters A, B, C… refer to the following events:

• A Created by Alexandra Elbakyan, the Sci-Hub website goes live on September 5, 2011.

• B Several LibGen domains go down when their registration expires, allegedly due to a longtime site administrator passing away from cancer.

• C Elsevier files a civil suit against Sci-Hub and LibGen — at the respective domains sci-hub.org and libgen.org — in the U.S. District Court for the Southern District of New York (Van der Sar, 2015a; DeMarco et al., 2015a). The complaint seeks a “prayer for relief” that includes domain name seizure, damages, and “an order disgorging Defendants’ profits”.

• D Elsevier is granted a preliminary injunction to suspend domain names and restrain the site operators from distributing Elsevier’s copyrighted works (Van der Sar, 2015b; Sweet, 2015). Shortly after, Sci-Hub and LibGen resurface at alternative domains outside of U.S. court jurisdiction, including on the dark web (Schiermeier, 2015; Van der Sar, 2015c).

• E The article “Meet the Robin Hood of Science” by Simon Oxenham spurs a wave of attention and news coverage on Sci-Hub and Alexandra Elbakyan (Oxenham, 2016), culminating in The New York Times asking “Should all research papers be free?” (Murphy, 2016).

• F The article “Who’s downloading pirated papers? Everyone” by John Bohannon shows Sci-Hub is used worldwide, including in developed countries (Bohannon, 2016b). These findings spark debate among scholars, with a large contingent of scientists supporting Sci-Hub’s mission (Woolston, 2016; Travis, 2016).

• G Alexandra Elbakyan is named one of “Nature’s 10”, which featured “ten people who mattered” in 2016 (Van Noorden, 2016). This article profiles Alexandra and includes an estimate that Sci-Hub serves “3% of all downloads from science publishers worldwide.”

• H The court finds that Alexandra Elbakyan, Sci-Hub, and LibGen are “liable for willful copyright infringement” in a default judgment, since none of the defendants answered Elsevier’s complaint (Schiermeier, 2017a; Van der Sar, 2017a; Sweet, 2017). The court issues a permanent injunction and orders the defendants to pay Elsevier $15 million, or $150,000 for each of 100 copyrighted works. The statutory damages, which the defendants do not intend to pay, now bear interest.

• I The American Chemical Society files suit against Sci-Hub in the U.S. District Court for the Eastern District of Virginia. Their “prayer for relief” requests that Internet search engines and Internet service providers “cease facilitating access” to Sci-Hub (Van der Sar, 2017b; Barnes A et al., 2017).

• J The version 1 preprint of this study is published (Himmelstein et al., 2017a), generating headlines such as Science’s “subscription journals are doomed” (McKenzie, 2017) and Inside Higher Ed’s “Inevitably Open” (Fister, 2017).

• K Sci-Hub blocks access to Russian IP addresses due to disputes with the Russian Scientific establishment and the naming of a newly discovered parasitoid wasp species, Idiogramma elbakyanae, after Alexandra Elbakyan (Standish, 2017; Khalaim and Ruíz-Cancino, 2017). Four days later, Sci-Hub restores access after receiving “many letters of support from Russian researchers”.

• L The court rules on the American Chemical Society suit, ordering Sci-Hub to pay $4.8 million in damages and that “any person or entity in active concert or participation” with Sci-Hub “including any Internet search engines, web hosting and Internet service providers, domain name registrars, and domain name registries, cease facilitating access” (Schiermeier, 2017b; Brinkema L, 2017). Within five weeks, the domains sci-hub.io, sci-hub.ac, sci-hub.cc, and sci-hub.bz were suspended by their respective domain name registries (Silver, 2017), leaving only the Tor hidden service and several newly-registered/revealed domains in operation.

Past research sheds some light on Sci-Hub’s reach. From the Spring of 2013 until the end of 2014, Sci-Hub relied on the Library Genesis (LibGen) scimag repository to store articles (Elbakyan, 2017). Whenever a user requested an article, Sci-Hub would check LibGen for a copy. If the article was not in LibGen, Sci-Hub would fetch the article for the user and then upload it to LibGen. Cabanac compared the number of articles in the LibGen scimag database at the start of 2014 to the total number of Crossref DOIs, estimating that LibGen contained 36% of all published scholarly articles (Cabanac, 2016). Coverage was higher for several prominent publishers: 77% for Elsevier, 73% for Wiley, and 53% for Springer (prior to its merger with Macmillan/Nature; Van Noorden, 2015).

Later, Bohannon analyzed six months of Sci-Hub’s server access logs, starting in September 2015 (Bohannon, 2016b). He found a global pattern of usage. Based on these logs, Gardner, McLaughlin, and Asher estimated the ratio of publisher downloads to Sci-Hub downloads within the U.S. for several publishers (Gardner et al., 2017). They estimated this ratio at 20:1 for the Royal Society of Chemistry and 48:1 for Elsevier. They also noted that 25% of Sci-Hub downloads in the U.S. were for articles related to clinical medicine. Greshake also analyzed the logs to identify per capita Sci-Hub usage (Greshake, 2016). Portugal, Iran, Tunisia, and Greece had the highest usage, suggesting Sci-Hub is preferentially used in countries with poor institutional access to scholarly literature. In a subsequent study, he found especially high Sci-Hub usage in chemistry, with 12 of the top 20 requested journals specializing in chemistry (Greshake, 2017a; Greshake, 2017b).

Since 2015, Sci-Hub has operated its own repository, distinct from LibGen. On March 19, 2017, Sci-Hub released the list of DOIs for articles in its database. Greshake retrieved metadata for 77% of Sci-Hub DOIs (Greshake, 2017a; Greshake, 2017b). He found that 95% of articles in Sci-Hub were published after 1950. Sci-Hub requests were even more skewed towards recent articles, with only 5% targeting articles published before 1983. Greshake’s study did not incorporate a catalog of all scholarly literature. This study analyzes Sci-Hub’s catalog in the context of all scholarly literature and thus assesses coverage. In other words, what percentage of articles in a given domain does Sci-Hub have in its repository?

To define the extent of the scholarly literature, we relied on DOIs from the Crossref database, as downloaded on March 21, 2017. We define the “scholarly literature” as 81,609,016 texts identified by their DOIs. We refer to these texts as “articles”, although Sci-Hub encompasses a range of text types, including book chapters, conference papers, and journal front matter. To assess the articles available from Sci-Hub, we relied on a list of DOIs released by Sci-Hub on March 19, 2017. All DOIs were lowercased to be congruent across datasets (see Methods). Sci-Hub’s offerings included 56,246,220 articles from the corpus of scholarly literature, equating to 68.9% of all articles.

Coverage by article type

Each article in Crossref’s database is assigned a type. Figure 2 shows coverage by article type. The scholarly literature consisted primarily of journal articles, for which Sci-Hub had 77.8% coverage. Sci-Hub’s coverage was also strong for the 5 million proceedings articles at 79.7%. Overall coverage suffered from the 10 million book chapters, where coverage was poor (14.2%). The remaining Crossref types were uncommon, and hence contributed little to overall coverage.

Figure 2

Download asset Open asset

Coverage by journal

We defined a comprehensive set of scholarly publishing venues, referred to as “journals”, based on the Scopus database. In reality, these include conferences with proceedings as well as book series. For inclusion in this analysis, each required an ISSN and at least one article as part of the Crossref-derived catalog of scholarly literature. Accordingly, our catalog consisted of 23,037 journals encompassing 56,755,671 articles. Of these journals, 4,598 (20.0%) were inactive (i.e. no longer publishing articles), and 2,933 were open access (12.7%). Only 70 journals were inactive and also open access.

We calculated Sci-Hub’s coverage for each of the 23,037 journals (examples in Table 1). A complete journal coverage table is available in our Sci-Hub Stats Browser: https://greenelab.github.io/scihub/#/journals. The Browser also provides views for each journal and publisher with detailed coverage and access-log information.

Table 1

Journal	Sci-Hub	Crossref	Coverage
The Lancet	457,650	458,580	99.8%
Nature	385,619	399,273	96.6%
British Medical Journal (Clinical Research Edition)	17,141	392,277	4.4%
Lecture Notes in Computer Science	103,675	356,323	29.1%
Science	230,649	251,083	91.9%
Journal of the American Medical Association	191,950	248,369	77.3%
Journal of the American Chemical Society	189,142	189,567	99.8%
Scientific American	22,600	186,473	12.1%
New England Journal of Medicine	180,321	180,467	99.9%
PLOS ONE	4,731	177,260	2.7%

In general, a journal’s coverage was either nearly complete or near zero (Figure 3). As a result, relatively few journals had coverage between 5–75%. At the extremes, 2,574 journals had zero coverage in Sci-Hub, whereas 2,095 journals had perfect coverage. Of zero-coverage journals, 22.2% were inactive, and 27.9% were open access. Of perfect-coverage journals, 81.6% were inactive, and 2.0% were open access. Hence, inactive, toll access journals make up the bulk of perfect-coverage journals.

Figure 3

Download asset Open asset

Next, we explored article coverage according to journal attributes (Figure 4). Sci-Hub covered 83.1% of the 56,755,671 articles that were attributable to a journal. Articles from inactive journals had slightly lower coverage than active journals (77.3% versus 84.1%). Strikingly, coverage was substantially higher for articles from toll rather than open access journals (85.1% versus 48.3%). Coverage did vary by subject area, with the highest coverage in chemistry at 93.0% and the lowest coverage in computer science at 76.3%. Accordingly, no discipline had coverage below 75%. See Figure 4—figure supplement 1 for coverage according to a journal’s country of publication.

Figure 4 with 1 supplement see all

Download asset Open asset

We also evaluated whether journal coverage varied by journal impact. We assessed journal impact using the 2015 CiteScore, which measures the average number of citations that articles published in 2012–2014 received during 2015. Highly cited journals tended to have higher coverage in Sci-Hub (Figure 9A). The 1,734 least cited journals (lowest decile) had 40.9% coverage on average, whereas the 1,733 most cited journals (top decile) averaged 90.0% coverage.

Coverage by publisher

Next, we evaluated coverage by publisher (Figure 5; full table available at https://greenelab.github.io/scihub/#/publishers). The largest publisher was Elsevier, with 13,115,639 articles from 3,410 journals. Sci-Hub covered 96.9% of Elsevier articles. For the eight publishers with more than one million articles, the following coverage was observed: 96.9% of Elsevier, 89.7% of Springer Nature, 94.7% of Wiley-Blackwell, 92.6% of Taylor & Francis, 79.4% of Wolters Kluwer, 88.3% of Oxford University Press, 90.9% of SAGE, and 98.8% of American Chemical Society articles. In total, 3,832 publishers were represented in the journal catalog. The coverage distribution among publishers resembled the journal coverage distribution, with most publishers occupying the extremities (Figure 3). Sci-Hub had zero coverage for 1,249 publishers, and complete coverage for 341 publishers.

Figure 5

Download asset Open asset

Coverage by year

Next, we investigated coverage based on the year an article was published (Figure 6). For most years since 1850, annual coverage is between 60–80%. However, there is a dropoff in coverage, starting in 2010, for recently published articles. For example, 2016 coverage was 56.0% and 2017 coverage (for part of the year) was 45.3%. One factor is that it can take some time for Sci-Hub to retrieve articles following their publication, as many articles are not downloaded until requested by a user. Another possible factor is that some publishers are now deploying more aggressive measures to deter unauthorized article downloads (Rovner, 2014; Becker, 2016), making recent articles less accessible.

Figure 6 with 1 supplement see all

Download asset Open asset

In addition, the prevalence of open access has been increasing, while Sci-Hub preferentially covers articles in toll access journals. Figure 6—figure supplement 1 tracks yearly coverage separately for articles in toll and open access journals. Toll access coverage exceeded 80% every year since 1950 except for 2016 and 2017. For both toll and open articles, the recent dropoff in coverage appears to begin in 2014 (Figure 6—figure supplement 1) compared to 2010 when calculated across all articles (Figure 6). We speculate this discrepancy results from the proliferation of obscure, low-quality journals over the last decade (Shen and Björk, 2015), as these journals generally issue DOIs but are not indexed in Scopus, and therefore would be included in Figure 6 but not in Figure 6—figure supplement 1. In addition to having limited readership demand, these journals are generally open access, and thus less targeted by Sci-Hub.

Sci-Hub’s coverage of 2016 articles in open access journals was just 32.7% compared to 78.8% for articles in toll access journals (Figure 6—figure supplement 1). Upon further investigation, we discovered that in June 2015, Sci-Hub ceased archiving articles in PeerJ, eLife, and PLOS journals, although they continued archiving articles in other open access journals such as Scientific Reports, Nature Communications, and BMC-series journals. Sci-Hub currently redirects requests for these delisted journals to the publisher’s site, unless it already possesses the article, in which case it serves the PDF. These findings suggest Sci-Hub prioritizes circumventing access barriers rather than creating a single repository containing every scholarly article.

Coverage by category of access status

In the previous analyses, open access status was determined at the journal level according to Scopus. This category of access is frequently referred to as “gold” open access, meaning that all articles from the journal are available gratis. However, articles in toll access journals may also be available without charge. Adopting the terminology from the recent “State of OA” study (Piwowar et al., 2018), articles in toll access journals may be available gratis from the publisher under a license that permits use (termed “hybrid”) or with all rights reserved (termed “bronze”). Alternatively, “green” articles are paywalled on the publisher’s site, but available gratis from an open access repository (e.g. a pre- or post-print server, excluding Sci-Hub and academic social networks).

The State of OA study determined the access status of 290,120 articles using the oaDOI utility (see Methods). Figure 7 shows Sci-Hub’s coverage for each category of access status. In line with our findings on the entire Crossref article catalog where Sci-Hub covered 49.1% of articles in open access journals, Sci-Hub’s coverage of gold articles in the State of OA dataset was 49.2%. Coverage of the 165,340 closed articles was 90.4%.

Figure 7 with 1 supplement see all

Download asset Open asset

Sci-Hub’s coverage was higher for closed and green articles than for hybrid or bronze articles. Furthermore, Sci-Hub’s coverage of closed articles was similar to its coverage of green articles (Figure 7). These findings suggest a historical pattern where users resort to Sci-Hub after encountering a paywall but before checking oaDOI or a search engine for green access. As such, Sci-Hub receives requests for green articles, triggering it to retrieve green articles at a similar rate to closed articles. However, hybrid and bronze articles, which are available gratis from their publishers, are requested and thus retrieved at a lower rate.

Coverage combining access methods

In practice, readers of the scholarly literature likely use a variety of methods for access. Figure 8 compares several of these methods, as well as their combinations. Users without institutional access may simply attempt to view an article on its publisher’s site. Based on our manual evaluation of 500 articles, we found 34.8% [30.7%–39.1%] of articles were accessible this way. The remaining 326 articles that were not accessible from their publisher’s site are considered toll access. oaDOI — a utility that redirects paywalled DOIs to gratis, licit versions, when possible (Piwowar et al., 2018) — was able to access 15.3% [11.7%–19.5%] of these toll access articles, indicating that green open access is still limited in its reach. This remained true on the full set of 208,786 toll access articles from the State of OA dataset, where oaDOI only provided access to 12.4% [12.3%–12.6%]. Although oaDOI’s overall access rate was 37.0% [36.8%–37.2%], this access consisted largely of gold, hybrid, and bronze articles, whereby gratis access is provided by the publisher.

Figure 8

Download asset Open asset

Sci-Hub and Penn had similar coverage on all articles: 85.2% [81.9%–88.1%] versus 87.4% [84.3%–90.1%] on the manual article set and 84.8% [84.7%–84.9%] versus 84.4% [84.3%–84.5%] on the larger but automated set. However, when considering only toll access articles, Sci-Hub’s coverage exceeds Penn’s: 94.2% [91.2%–96.3%] versus 80.7% [76.1%–84.7%] on the manual set and 90.7% [90.5%–90.8%] versus 83.5% [83.4%–83.7%] on the automated set. This reflects Sci-Hub’s focus on paywalled articles. In addition, Sci-Hub’s coverage is a lower bound for its access rate, since it can retrieve articles on demand, so in practice Sci-Hub’s access to toll access articles could exceed Penn’s by a higher margin. Remarkably, Sci-Hub provided greater access to paywalled articles than a leading research university spending millions of US dollars per year on subscriptions. However, since Sci-Hub is able to retrieve articles through many university networks, it is perhaps unsurprising that its coverage would exceed that of any single university.

Combining access methods can also be synergistic. Specifically when including open access articles, combining Sci-Hub’s repository with oaDOI’s or Penn’s access increased coverage from around 85% to 95%. The benefits of oaDOI were reduced when only considering toll access articles, where oaDOI only improved Sci-Hub’s or Penn’s coverage by approximately 1%. On toll access articles, Penn’s access appeared to complement Sci-Hub’s. Together, Sci-Hub’s repository and Penn’s access covered approximately 96% of toll access articles [95.0%–98.6% (manual set), 95.9%–96.1% (automated set)]. Our findings suggest that users with institutional subscriptions comparable to those at Penn as well as knowledge of oaDOI and Sci-Hub are able to access over 97% of all articles [96.7%–99.1% (manual set), 97.3%–97.5% (automated set)], online and without payment.

Sci-Hub access logs

Sci-Hub released article access records from its server logs, covering 165 days from September 2015 through February 2016 (Elbakyan and Bohannon, 2016; Bohannon, 2016b). After processing, the logs contained 26,984,851 access events. Hence, Sci-Hub provided access to an average of 164,000 valid requests per day in late 2015–early 2016.

In the first version of this study (Himmelstein et al., 2017a), we mistakenly treated the log events as requests rather than downloads. Fortunately, Sci-Hub reviewed the preprint in a series of tweets, and pointed out the error, stating “in Sci-Hub access logs released previous year, all requests are resolved requests, i.e. user successfully downloaded PDF with that DOI … unresolved requests are not saved”. Interestingly however, 198,600 access events from the logs pointed to DOIs that were not in Sci-Hub’s subsequent DOI catalog. 99.1% of these events — corresponding to DOIs logged as accessed despite later being absent from Sci-Hub — were for book chapters. Upon further investigation, we identified several DOIs in this category that Sci-Hub redirected to LibGen book records as of September 2017. The LibGen landing pages were for the entire books, which contained the queried chapters, and were part of LibGen’s book (not scimag) collection. The explanation that Sci-Hub outsources some book access to LibGen (and logged such requests as accessed) is corroborated by Elbakyan’s statement that (Elbakyan, 2017): “Currently, the Sci-Hub does not store books, for books users are redirected to LibGen, but not for research papers. In future, I also want to expand the Sci-Hub repository and add books too.” Nonetheless, Sci-Hub’s catalog contains 72.4% of the 510,760 distinct book chapters that were accessed according to the logs. Therefore, on a chapter-by-chapter basis, Sci-Hub does already possess many of the requested scholarly books available from LibGen.

We computed journal-level metrics based on average article downloads. The “visitors” metric assesses the average number of IP addresses that accessed each article published by a journal during the 20 months preceding September 2015 (the start date of the Sci-Hub logs). In aggregate, articles from toll access journals averaged 1.30 visitors, whereas articles from open access journals averaged 0.25 visitors. Figure 9B shows that articles from highly cited journals were visited much more frequently on average. Articles in the least cited toll access journals averaged almost zero visitors, compared to approximately 15 visitors for the most cited journals. In addition, Figure 9B shows that articles in toll access journals received many times more visitors than those in open access journals, even after accounting for journal impact. One limitation of using this analysis to judge Sci-Hub’s usage patterns is that we do not know to what extent certain categories of articles were resolved (and thus logged) at different rates.

Figure 9

Download asset Open asset

Sci-Hub’s repository contained 69% of all scholarly articles with DOIs. Coverage for the 54.5 million articles attributed to toll access journals — which many users would not otherwise be able to access — was 85.1%. Since Sci-Hub can retrieve, in real time, requested articles that are not in its database, our coverage figures are a lower bound. Furthermore, Sci-Hub preferentially covered popular, paywalled articles. We find that 91.0% of citations since 2015 were present in Sci-Hub’s repository, which increased to 96.2% when excluding citations to articles in open access journals. Journals with very low (including zero) coverage tended to be obscure, less cited venues, while average coverage of the most cited journals exceeded 90%.

We find strong evidence that Sci-Hub is primarily used to circumvent paywalls. In particular, users accessed articles from toll access journals much more frequently than open access journals. Additionally, within toll access journals, Sci-Hub provided higher coverage of articles in the closed and green categories (paywalled by the publisher) as opposed to the hybrid and bronze categories (available gratis from the publisher). Accordingly, many users likely only resort to Sci-Hub when access through a commercial database is cumbersome or costly. Finally, we observed evidence that Sci-Hub’s primary operational focus is circumventing paywalls rather than compiling all literature, as archiving was deactivated in 2015 for several journals that exemplify openness. Attesting to its success in this mission, Sci-Hub’s database already contains more toll access articles than are immediately accessible via the University of Pennsylvania, a leading research university.

Judging from donations, many users appear to value Sci-Hub’s service. In the past, Sci-Hub accepted donations through centralized and regulated payment processors such as PayPal, Yandex, WebMoney, and QiQi (DeMarco et al., 2015a; Woltermann, 2015). Now however, Sci-Hub only advertises donation via Bitcoin, presumably to avoid banking blockades or government seizure of funds. Since the ledger of Bitcoin transactions is public, we can evaluate the donation activity to known Sci-Hub addresses (1K4t2vSBSS2xFjZ6PofYnbgZewjeqbG1TM, 14ghuGKDAPdEcUQN4zuzGwBUrhQgACwAyA, 1EVkHpdQ8VJQRpQ15hSRoohCztTvDMEepm). We find that, prior to 2018, these addresses have received 1,232 donations, totaling 94.494 (Figure 10). Using the US dollar value at the time of transaction confirmation, Sci-Hub has received an equivalent of $69,224 in bitcoins. 85.467 bitcoins have been withdrawn from the Sci-Hub addresses via 174 transactions. Since the price of bitcoins has risen, the combined US dollar value at time of withdrawal was $421,272. At the conclusion of 2017, the Sci-Hub accounts had an outstanding balance of 9.027 bitcoins, valued at roughly $120,000. In response to this study’s preprint (Himmelstein et al., 2017a), Sci-Hub tweeted: “the information on donations … is not very accurate, but I cannot correct it: that is confidential.” Therefore, presumably, Sci-Hub has received considerable donations via alternative payment systems or to unrevealed Bitcoin addresses, which our audit did not capture. Since we do not know the identity of the depositors, another possibility would be that Sci-Hub transfered bitcoins from other addresses it controlled to the identified donation addresses.

Figure 10 with 1 supplement see all

Download asset Open asset

The largest, most prominent academic publishers are thoroughly covered by Sci-Hub, and these publishers have taken note. Elsevier (whose 13.5 million works are 96.9% covered by Sci-Hub) and the American Chemical Society (whose 1.4 million works are 98.8% covered) both filed suit against Sci-Hub, despite the limited enforcement options of United States courts. The widespread gratis access that Sci-Hub provides to previously paywalled articles calls into question the sustainability of the subscription publishing model (McKenzie, 2017; Lawson, 2017b). Avoiding biblioleaks and retaining exclusive possession of digital media may prove an insurmountable challenge for publishers (Dunn et al., 2014). As distributed and censorship-resistant file storage protocols mature (Benet, 2014; ConsenSys, 2016), successors to Sci-Hub may emerge that no longer rely on a centralized service. Indeed, Alexandra Elbakyan is only one individual in the larger “guerilla access” movement (Bodó, 2016; Bodó, 2015; Laskow, 2016), which will persist regardless of Sci-Hub’s fate. As such, Sci-Hub’s corpus of gratis scholarly literature may be extremely difficult to suppress.

Surveys from 2016 suggest awareness and usage of Sci-Hub was not yet commonplace (Travis, 2016; Mejia et al., 2017). However, adoption appears to be growing. According to Elbakyan, the number of Sci-Hub downloads increased from 42 million in 2015 to 75 million in 2016, equating to a 79% gain (Van Noorden, 2016). Comparing the search interest peaks following D and L in Figure 1, which both correspond to domain outages and hence existing users searching how to access Sci-Hub, we estimate annual growth of 88%. As per Figure 1—figure supplement 1, Sci-Hub averaged 185,243 downloads per day in January–February 2016, whereas in 2017 daily downloads averaged 458,589. Accordingly, the ratio of Sci-Hub to Penn Libraries downloads in 2017 was 20:1. In addition, adoption of Sci-Hub or similar sites could accelerate due to new technical burdens on authorized access (the flip side of anti-piracy measures) (Davis, 2016; Esposito, 2016), crackdowns on article sharing via academic social networks (Singh Chawla, 2017a; Singh Chawla, 2017b), or large-scale subscription cancellations by libraries (Esposito, 2017).

Historically, libraries have often canceled individual journal subscriptions or switched from bundled to à-la-carte selections (Roth, 1990; Fernandez et al., 2014; Rogers, 2012). More recently, library consortia have threatened wholesale cancellation of specific publishers. In 2010, Research Libraries of the UK threatened to let Elsevier contracts expire (Bergstrom et al., 2014; Prosser, 2011), while the University of California raised the possibility of boycotting Nature Publishing Group. But these disputes were ultimately resolved before major cancellations transpired. But in 2017, researchers began losing access to entire publishers. Universities in the Netherlands canceled all Oxford University Press subscriptions in May 2017 (Else, 2017). University of Montreal reduced its subscriptions to Taylor & Francis periodicals by 93%, axing 2,231 journals (Gagnon, 2017). Negotiations with Elsevier reached impasses in Germany, Peru, and Taiwan. As a result, hundreds of universities have cancelled all Elsevier subscriptions (Schiermeier and Mega, 2016; Schiermeier, 2018). These developments echo the predictions of Elsevier’s attorneys in 2015 (DeMarco et al., 2015b): “Defendants’ actions also threaten imminent irreparable harm to Elsevier because it appears that the Library Genesis Project repository may be approaching (or will eventually approach) a level of ‘completeness’ where it can serve as a functionally equivalent, although patently illegal, replacement for ScienceDirect.”

In the worst case for toll access publishers, growing Sci-Hub usage will become both the cause and the effect of dwindling subscriptions. Librarians rely on usage metrics and user feedback to evaluate subscriptions (Roth, 1990). Sci-Hub could decrease the use of library subscriptions as many users find it more convenient than authorized access (Travis, 2016). Furthermore, librarians may receive fewer complaints after canceling subscriptions, as users become more aware of alternatives. Green open access also provides an access route outside of institutional subscription. The posting of preprints and postprints has been growing rapidly (Piwowar et al., 2018; Kaiser, 2017), with new search tools to help locate them (Singh Chawla, 2017c). The trend of increasing green availability is poised to continue as funders mandate postprints (Van Noorden, 2014) and preprints help researchers sidestep the slow pace of scholarly publishing (Powell, 2016). In essence, scholarly publishers may have already lost the access battle. Publishers will be forced to adapt quickly to open access publishing models. In the words of Alexandra Elbakyan (Elbakyan, 2016b): “The effect of long-term operation of Sci-Hub will be that publishers change their publishing models to support Open Access, because closed access will make no sense anymore.”

Sci-Hub is poised to fundamentally disrupt scholarly publishing. The transition to gratis availability of scholarly articles is currently underway, and such a model may be inevitable in the long term (Lewis, 2012; Sutton, 2011; Jha, 2012). However, we urge the community to take this opportunity to fully liberate scholarly articles, as well as explore more constructive business models for publishing (Paul et al., 2017; Vogel, 2017; Logan, 2017). Only libre access, enabled by open licensing, allows building applications on top of scholarly literature without fear of legal consequences (Himmelstein, 2016). For example, fulltext mining of scholarly literature is an area of great potential (Westergaard et al., 2017), but is currently impractical due to the lack of a large-scale preprocessed corpus of articles. The barriers here are legal, not technological (Brook et al., 2014; Van Noorden, 2012). In closing, were all articles libre, there would be no such thing as a “pirate website” for accessing scholarly literature.

This project was performed entirely in the open, via the GitHub repository greenelab/scihub. Several authors of this study became involved after we mentioned their usernames in GitHub discussions. This project’s fully transparent and online model enabled us to assemble an international team of individuals with complementary expertise and knowledge.

We managed our computational environment using Conda, allowing us to specify and install dependencies for both Python and R. We performed our analyses using a series of Jupyter notebooks. In general, data integration and manipulation were performed in Python 3, relying heavily on Pandas, while plotting was performed with ggplot2 in R. Tabular data were saved in TSV (tab-separated values) format, and large datasets were compressed using XZ. We used Git Large File Storage (Git LFS) to track large files, enabling us to make nearly all of the datasets generated and consumed by the analyses available to the public. The Sci-Hub Stats Browser is a single-page application built using React and hosted via GitHub Pages. Frontend visualizations use Vega-Lite (Satyanarayan et al., 2017). Certain datasets for the browser are hosted in the greenelab/scihub-browser-data repository.

The manuscript source for this study is located at greenelab/scihub-manuscript. We used the Manubot to automatically generate the manuscript from Markdown files. This system — originally developed for the Deep Review to enable collaborative writing on GitHub (Ching et al., 2017) — uses continuous analysis to fetch reference metadata and rebuild the manuscript upon changes (Beaulieu-Jones and Greene, 2017).

LibGen scimag’s catalog of articles

Request a detailed protocol

Library Genesis (LibGen) is a shadow library primarily comprising illicit copies of academic books and articles. Compared to Sci-Hub, the operations of LibGen are more opaque, as the contributors maintain a low profile and do not contact journalists (Elbakyan, 2017). LibGen hosts several collections, including distinct repositories for scientific books and textbooks, fiction books, and comics (Cabanac, 2016). In 2012, LibGen added the “scimag” database for scholarly literature. Since the spring of 2013, Sci-Hub has uploaded articles that it obtains to LibGen scimag (Elbakyan, 2017). At the end of 2014, Sci-Hub forked LibGen scimag and began managing its own distinct article repository.

We downloaded the LibGen scimag metadata database on April 7, 2017 as a SQL dump. We imported the SQL dump into MySQL, and then exported the scimag table to a TSV file (Himmelstein and McLaughlin, 2017). Each row of this table corresponds to an article in LibGen, identified by its DOI. The TimeAdded field apparently indicates when the publication was uploaded to LibGen. After removing records missing TimeAdded, 64,195,940 DOIs remained. 56,205,763 (87.6%) of the DOIs were in our Crossref-derived catalog of scholarly literature. The 12.4% of LibGen scimag DOIs missing from our Crossref catalog likely comprise incorrect DOIs, DOIs whose metadata availability postdates our Crossref export, DOIs from other Registration Agencies, and DOIs for excluded publication types.

Next, we explored the cumulative size of LibGen scimag over time according to the TimeAdded field (Figure 11). However, when we compared our plot to one generated from the LibGen scimag database SQL dump on January 1, 2014 (Cabanac, 2016; Cabanac, 2017), we noticed a major discrepancy. The earlier analysis identified a total of 22,829,088 DOIs, whereas we found only 233,707 DOIs as of January 1, 2014. We hypothesize that the discrepancy arose because TimeAdded indicates the date modified rather than created. Specifically, when an article in the database is changed, the database record for that DOI is entirely replaced. Hence, the TimeAdded value is effectively overwritten upon every update to a record. Unfortunately, many research questions require the date first added. For example, lag-time analyses (the time from study publication to LibGen upload) may be unreliable. Therefore, we do not report on these findings in this manuscript. Instead, we provide Figure 11—figure supplement 1 as an example analysis that would be highly informative were reliable creation dates available. In addition, findings from some previous studies may require additional scrutiny. For example, Cabanac writes (Cabanac, 2016): “The growth of LibGen suggests that it has benefited from a few isolated, but massive, additions of scientific articles to its cache. For instance, 71% of the article collection was uploaded in 13 days at a rate of 100,000+ articles a day. It is likely that such massive collections of articles result from biblioleaks (Dunn et al., 2014), but one can only speculate about this because of the undocumented source of each file cached at LibGen.” While we agree this is most likely the case, confirmation is needed that the bulk addition of articles does not simply correspond to bulk updates rather than bulk initial uploads.

Figure 11 with 1 supplement see all

Download asset Open asset

The source code data analysis and interactive browser associated with this study are available at the following GitHub repositories: https://github.com/greenelab/crossref (copy archived at https://github.com/elifesciences-publications/crossref)https://github.com/greenelab/scihub (copy archived at https://github.com/elifesciences-publications/scihub)https://github.com/greenelab/scihub-manuscript (copy archived at https://github.com/elifesciences-publications/scihub-manuscript)https://github.com/greenelab/scihub-browser-data (copy archived at https://github.com/elifesciences-publications/sciub-brower-data)https://github.com/dhimmel/scopus (copy archived at https://github.com/elifesciences-publications/scopus)https://github.com/greenelab/library-access (copy archived at https://github.com/elifesciences-publications/library-access)

The MongoDB database export of DOI metadata from the Crossref API are available on Figshare (https://doi.org/10.6084/m9.figshare.4816720.v1).

1. Website

   1. Archambault É
   2. Amyot D
   3. Deschamps P
   4. Nicol A
   5. Provencher F
   6. Rebout L
   7. Roberge G

   (2014) Proportion of Open Access Papers Published in Peer-Reviewed Journals at the European and World Levels: 1996–2013

   University of Nebraska-Lincoln. Accessed February 7, 2018.

   http://digitalcommons.unl.edu/scholcom/8
2. Website

   1. Association of Research Libraries

   (2017) Graph 4: Expenditure Trends in ARL Libraries, 1986-2015

   ARL Statistics 2014–2015. Accessed February 7, 2018.

   http://www.arl.org/storage/documents/expenditure-trends.pdf
3. Website

   1. Barnes A L
   2. Weslow D E
   3. Gardner M J

   (2017) Complaint – Document #1 of American Chemical Society v. Does – Case 1:17-cv-00726-LMB-JFA

   Eastern District Court of Virginia. Accessed February 7, 2018.

   https://www.courtlistener.com/docket/6146630/1/american-chemical-society-v-does-1-99/
4. 1. Beaulieu-Jones BK
   2. Greene CS

   (2017) Reproducibility of computational workflows is automated using continuous analysis

   Nature Biotechnology 35:342–346.

   https://doi.org/10.1038/nbt.3780

   • PubMed
   • Google Scholar
5. 1. Becker R

   (2016) Publisher under fire for fake article webpages

   Nature 535:11.

   https://doi.org/10.1038/535011f

   • Google Scholar
6. 1. Bendezú-Quispe G
   2. Nieto-Gutiérrez W
   3. Pacheco-Mendoza J
   4. Taype-Rondan A

   (2016) Sci-Hub and medical practice: an ethical dilemma in Peru

   The Lancet Global Health 4:e608.

   https://doi.org/10.1016/S2214-109X(16)30188-7

   • PubMed
   • Google Scholar
7. Website

   1. Benet J

   (2014) IPFS - Content Addressed, Versioned, P2P File System

   arXiv. Accessed February 7, 2018.

   https://arxiv.org/abs/1407.3561v1
8. 1. Bergstrom TC
   2. Courant PN
   3. McAfee RP
   4. Williams MA

   (2014) Evaluating big deal journal bundles

   PNAS 111:9425–9430.

   https://doi.org/10.1073/pnas.1403006111

   • PubMed
   • Google Scholar
9. 1. Björk B-C

   (2017) Scholarly journal publishing in transition- from restricted to open access

   Electronic Markets 27:101–109.

   https://doi.org/10.1007/s12525-017-0249-2

   • Google Scholar
10. 1. Bodó B

    (2015)

    Copyrighting Creativity

    Libraries in the Post-Scarcity Era, Copyrighting Creativity, Routledge, 10.2139/ssrn.2616636.

    • Google Scholar
11. Conference

    1. Bodó B

    (2016) Pirates in the Library An Inquiry into the Guerilla Open Access Movement

    8th Annual Workshop of the International Society for the History and Theory of Intellectual Property.

    https://doi.org/10.2139/ssrn.2816925

    • Google Scholar
12. 1. Bohannon J

    (2016a) The frustrated science student behind Sci-Hub

    Science 352:511.

    https://doi.org/10.1126/science.352.6285.511

    • PubMed
    • Google Scholar
13. 1. Bohannon J

    (2016b) Who's downloading pirated papers? Everyone

    Science 352:508–512.

    https://doi.org/10.1126/science.352.6285.508

    • Google Scholar
14. Website

    1. Bosch S
    2. Henderson K

    (2017) New World, Same Model | Periodicals Price Survey 2017

    Library Journal. Accessed February 7, 2018.

    http://lj.libraryjournal.com/2017/04/publishing/new-world-same-model-periodicals-price-survey-2017/
15. Preprint

    1. Bosman J
    2. Kramer B

    (2018) Open access levels: a quantitative exploration using Web of Science and oaDOI data

    PeerJ Preprints.

    https://doi.org/10.7287/peerj.preprints.3520v1

    • Google Scholar
16. 1. Boudry C
    2. Chartron G

    (2017) Availability of digital object identifiers in publications archived by PubMed

    Scientometrics 110:1453–1469.

    https://doi.org/10.1007/s11192-016-2225-6

    • Google Scholar
17. Website

    1. Brinkema L M

    (2017) Order – Document #37 of American Chemical Society v. Sci-Hub – Case 1:17-cv-00726-LMB-JFA

    Eastern District Court of Virginia. Accessed February 7, 2018.

    https://regmedia.co.uk/2017/11/07/sci_hub_block_order_short.pdf
18. 1. British Standards Institute

    (2012) BS ISO 26324:2012 Information and Documentation. Digital Object Identifier System

    BS ISO 26324:2012 Information and Documentation. Digital Object Identifier System, 10.3403/30177056.

    https://doi.org/10.3403/30177056

    • Google Scholar
19. 1. Brook M
    2. Murray-Rust P
    3. Oppenheim C

    (2014) The social, political and legal aspects of text and data mining (TDM)

    D-Lib Magazine 20:.

    https://doi.org/10.1045/november14-brook

    • Google Scholar
20. Website

    1. Buranyi S

    (2017) Is the staggeringly profitable business of scientific publishing bad for science?

    The Guardian. Accessed February 7, 2018.

    https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science
21. 1. Cabanac G

    (2016) Bibliogifts in LibGen? A study of a text-sharing platform driven by biblioleaks and crowdsourcing

    Journal of the Association for Information Science and Technology 67:874–884.

    https://doi.org/10.1002/asi.23445

    • Google Scholar
22. Data

    1. Cabanac G

    (authors) (2017) Scimag catalogue of LibGen as of January 1st, 2014

    Figshare.

    https://doi.org/10.6084/m9.figshare.4906367.v1
23. Preprint

    1. Ching T
    2. Himmelstein DS
    3. Beaulieu-Jones BK
    4. Kalinin AA
    5. Do BT
    6. Way GP
    7. Ferrero E
    8. Agapow P-M
    9. Xie W
    10. Rosen GL

    (2017) Opportunities and obstacles for deep learning in biology and medicine

    bioRxiv.

    https://doi.org/10.1101/142760

    • Google Scholar
24. Website

    1. ConsenSys

    (2016) Decentralized Storage: The Backbone of the Third Web

    ConsenSys Media. Accessed February 7, 2018.

    https://media.consensys.net/decentralized-storage-the-backbone-of-the-third-web-d4bc54e79700
25. Website

    1. Davis P

    (2016) Two-step authentication: Finally coming to a university near you

    The Scholarly Kitchen. Accessed February 7, 2018.

    https://scholarlykitchen.sspnet.org/2016/06/21/two-step-authentication-finally-coming-to-a-university-near-you/
26. Website

    1. DeMarco JV
    2. Hirschberg DM
    3. Sen U

    (2015a) Complaint – Document #1 of Elsevier Inc. v. Sci-Hub – Case 1:15-cv-04282-RWS

    Southern District Court of New York. Accessed February 7, 2018.

    https://www.courtlistener.com/docket/4355308/1/elsevier-inc-v-sci-hub/
27. Website

    1. DeMarco JV
    2. Hirschberg DM
    3. Sen U

    (2015b) Memorandum of Law in Support of Motion – Document #6 of Elsevier Inc. v. Sci-Hub – Case 1:15-cv-04282-RWS

    Southern District Court of New York. Accessed February 7, 2018.

    https://www.courtlistener.com/docket/4355308/6/elsevier-inc-v-sci-hub/
28. Website

    1. Dingledine R
    2. Mathewson N
    3. Syverson P

    (2004) Tor: The Second-Generation Onion Router

    Accessed February 7, 2018.

    http://www.dtic.mil/docs/citations/ADA465464
29. 1. Dunn AG
    2. Coiera E
    3. Mandl KD

    (2014) Is Biblioleaks inevitable?

    Journal of Medical Internet Research 16:e112.

    https://doi.org/10.2196/jmir.3331

    • PubMed
    • Google Scholar
30. Website

    1. EBSCO

    (2017) Five Year Journal Price Increase History (2013-2017)

    EBSCO. Accessed February 7, 2018.

    https://www.ebscohost.com/promoMaterials/Five_Year_Journal_Price_Increase_History_EBSCO_2013-2017.pdf
31. Data

    1. Elbakyan A
    2. Bohannon J

    (authors) (2016) Sci-Hub download data

    Dryad Digital Repository.

    https://doi.org/10.5061/dryad.q447c/1
32. Website

    1. Elbakyan A

    (2015) Letter – Document #50 of Elsevier Inc. v. Sci-Hub – Case 1:15-cv-04282-RWS

    Southern District Court of New York. Accessed February 7, 2018.

    https://www.courtlistener.com/docket/4355308/50/elsevier-inc-v-sci-hub/
33. Website

    1. Elbakyan A

    (2016a) Sci-Hub is a goal, changing the system is a method

    Engineuring. Accessed February 7, 2018.

    https://engineuring.wordpress.com/2016/03/11/sci-hub-is-a-goal-changing-the-system-is-a-method/
34. Website

    1. Elbakyan A

    (2016b) Why Sci-Hub is the true solution for Open Access: reply to criticism

    Engineuring. Accessed February 7, 2018.

    https://engineuring.wordpress.com/2016/02/24/why-sci-hub-is-the-true-solution-for-open-access-reply-to-criticism/
35. Website

    1. Elbakyan A

    (2017) Some facts on Sci-Hub that Wikipedia gets wrong

    Engineuring. Accessed February 7, 2018.

    https://engineuring.wordpress.com/2017/07/02/some-facts-on-sci-hub-that-wikipedia-gets-wrong/
36. Website

    1. Else H

    (2017) Dutch lose access to OUP journals in subscription standoff

    Times Higher Education. Accessed February 7, 2018.

    https://www.timeshighereducation.com/news/dutch-lose-access-oup-journals-subscription-standoff
37. Website

    1. Esposito JJ

    (2016) Sci-Hub and the four horsemen of the internet

    The Scholarly Kitchen. Accessed February 7, 2018.

    https://scholarlykitchen.sspnet.org/2016/03/02/sci-hub-and-the-four-horsemen-of-the-internet/
38. Website

    1. Esposito JJ

    (2017) Sci-Hub moves to the center of the ecosystem

    The Scholarly Kitchen. Accessed February 7, 2018.

    https://scholarlykitchen.sspnet.org/2017/09/05/sci-hub-moves-center-ecosystem/
39. 1. Fernandez P
    2. Williamson J
    3. Dixon L

    (2014) Factors in Science Journal Cancellation Projects: The Roles of Faculty Consultations and Data

    Issues in Science and Technology Librarianship.

    https://doi.org/10.5062/F4G73BP3

    • Google Scholar
40. Website

    1. Fister B

    (2017) Inevitably open

    Inside Higher Ed. Accessed February 7, 2018.

    https://www.insidehighered.com/blogs/library-babel-fish/inevitably-open
41. Website

    1. Gagnon S

    (2017) UdeM Libraries cancel Big Deal subscription to 2,231 periodical titles published by Taylor & Francis Group

    Communiqués, Bibliothèques, Université de Montréal. Accessed February 7, 2018.

    http://www.bib.umontreal.ca/communiques/20170504-DC-annulation-taylor-francis-va.htm
42. Website

    1. Gardner GJ
    2. McLaughlin SR
    3. Asher AD

    (2017) Shadow Libraries and You: Sci-Hub Usage and the Future of ILL

    ACRL 2017. Accessed February 7, 2018.

    https://hdl.handle.net/10760/30981
43. 1. Gorraiz J
    2. Melero-Fuentes D
    3. Gumpenberger C
    4. Valderrama-Zurián J-C

    (2016) Availability of digital object identifiers (DOIs) in Web of Science and Scopus

    Journal of Informetrics 10:98–109.

    https://doi.org/10.1016/j.joi.2015.11.008

    • Google Scholar
44. 1. Greshake B

    (2016) Correlating the Sci-Hub data with World Bank Indicators and Identifying Academic Use

    The Winnower.

    https://doi.org/10.15200/winn.146485.57797

    • Google Scholar
45. 1. Greshake B

    (2017a) Looking into Pandora's Box: The Content of Sci-Hub and its Usage

    F1000Research 6:541.

    https://doi.org/10.12688/f1000research.11366.1

    • PubMed
    • Google Scholar
46. Data

    1. Greshake B

    (authors) (2017b) Data and Scripts for Looking into Pandora's Box: The Content of Sci-Hub and its Usage

    Zenodo.

    https://doi.org/10.5281/zenodo.472493
47. Data

    1. Himmelstein D
    2. McLaughlin S

    (authors) (2017) A user-friendly extract of the LibGen scimag metadata SQL dump on 2017-04-07

    Figshare.

    https://doi.org/10.6084/m9.figshare.5231245.v1
48. Data

    1. Himmelstein D
    2. Wheeler K
    3. Greene C

    (authors) (2017b) Metadata for all DOIs in Crossref: JSON MongoDB exports of all works from the Crossref API

    Figshare.

    https://doi.org/10.6084/m9.figshare.4816720
49. Website

    1. Himmelstein D

    (2016) The licensing of bioRxiv preprints

    Satoshi Village. Accessed February 7, 2018.

    http://blog.dhimmel.com/biorxiv-licenses/
50. Preprint

    1. Himmelstein DS
    2. Romero AR
    3. McLaughlin SR
    4. Tzovaras BG
    5. Greene CS

    (2017a) Sci-Hub provides access to nearly all scholarly literature

    PeerJ Preprints.

    https://doi.org/10.7287/peerj.preprints.3100v1

    • Google Scholar
51. 1. International DOI Foundation

    (2017)

    DOI Handbook

    DOI Handbook, 10.1000/182.

    • Google Scholar
52. Website

    1. Jha A

    (2012) Open access to research is inevitable, says Nature editor-in-chief

    The Guardian. Accessed February 7, 2018.

    https://www.theguardian.com/science/2012/jun/08/open-access-research-inevitable-nature-editor
53. 1. Kaiser J

    (2017) Are preprints the future of biology? A survival guide for scientists

    Science.

    https://doi.org/10.1126/science.aaq0747

    • Google Scholar
54. 1. Khabsa M
    2. Giles CL
    3. Lee Giles C

    (2014) The number of scholarly documents on the public web

    PLoS One 9:e93949.

    https://doi.org/10.1371/journal.pone.0093949

    • PubMed
    • Google Scholar
55. 1. Khalaim AI
    2. Ruíz-Cancino E

    (2017) Ichneumonidae (Hymenoptera) associated with xyelid sawflies (Hymenoptera, Xyelidae) in Mexico

    Journal of Hymenoptera Research 58:17–27.

    https://doi.org/10.3897/jhr.58.12919

    • Google Scholar
56. 1. Kikkawa J
    2. Takaku M
    3. Yoshikane F

    (2016) DOI Links on Wikipedia

    Digital Libraries: Knowledge, Information, and Data in an Open Access Society.

    https://doi.org/10.1007/978-3-319-49304-6_40

    • Google Scholar
57. 1. Kirsop B
    2. Chan L

    (2005) Transforming access to research literature for developing countries

    Serials Review 31:246–255.

    https://doi.org/10.1080/00987913.2005.10764998

    • Google Scholar
58. 1. Lammey R

    (2014) CrossRef developments and initiatives: an update on services for the scholarly publishing community from CrossRef

    Science Editing 1:13–18.

    https://doi.org/10.6087/kcse.2014.1.13

    • Google Scholar
59. 1. Larivière V
    2. Haustein S
    3. Mongeon P

    (2015) The oligopoly of academic publishers in the digital era

    PLoS One 10:e0127502.

    https://doi.org/10.1371/journal.pone.0127502

    • PubMed
    • Google Scholar
60. Website

    1. Laskow S

    (2016) The Rise of Pirate Libraries

    Atlas Obscura. Accessed February 7, 2018.

    http://www.atlasobscura.com/articles/the-rise-of-illegal-pirate-libraries
61. Data

    1. Lawson S
    2. Meghreblian B
    3. Brook M

    (authors) (2015) Journal subscription costs - FOIs to UK universities

    Figshare.

    https://doi.org/10.6084/m9.figshare.1186832.v23
62. Data

    1. Lawson S

    (authors) (2017a) Journal subscription expenditure in the UK 2015-16

    Figshare.

    https://doi.org/10.6084/m9.figshare.4542433.v6
63. 1. Lawson S

    (2017b) Access, ethics and piracy

    Insights the UKSG Journal 30:25–30.

    https://doi.org/10.1629/uksg.333

    • Google Scholar
64. 1. Lewis DW

    (2012) The Inevitability of Open Access

    College & Research Libraries 73:493–506.

    https://doi.org/10.5860/crl-299

    • Google Scholar
65. 1. Logan CJ

    (2017) We can shift academic culture through publishing choices

    F1000Research 6:518.

    https://doi.org/10.12688/f1000research.11415.1

    • PubMed
    • Google Scholar
66. 1. McKenzie L

    (2017) Sci-Hub’s cache of pirated papers is so big, subscription journals are doomed, data analyst suggests

    Science.

    https://doi.org/10.1126/science.aan7164

    • Google Scholar
67. 1. Meadows A

    (2015) Beyond Open: Expanding Access to Scholarly Content

    The Journal of Electronic Publishing 18:.

    https://doi.org/10.3998/3336451.0018.301

    • Google Scholar
68. 1. Mejia CR
    2. Valladares-Garrido MJ
    3. Miñan-Tapia A
    4. Serrano FT
    5. Tobler-Gómez LE
    6. Pereda-Castro W
    7. Mendoza-Flores CR
    8. Mundaca-Manay MY
    9. Valladares-Garrido D

    (2017) Use, knowledge, and perception of the scientific contribution of Sci-Hub in medical students: Study in six countries in Latin America

    PLoS One 12:e0185673.

    https://doi.org/10.1371/journal.pone.0185673

    • PubMed
    • Google Scholar
69. Website

    1. Milova E

    (2017) Alexandra Elbakyan – Science Should be Open to all Not Behind Paywalls

    Life Extension Advocacy Foundation. Accessed February 7, 2018.

    http://www.leafscience.org/alexandra-elbakyan/
70. 1. Mongeon P
    2. Paul-Hus A

    (2015) The journal coverage of Web of Science and Scopus: a comparative analysis

    Scientometrics.

    https://doi.org/10.1007/s11192-015-1765-5

    • Google Scholar
71. Website

    1. Morrison HG

    (2012) Freedom for scholarship in the internet age

    Simon Fraser University. Accessed February 7, 2018.

    http://summit.sfu.ca/item/12537
72. Website

    1. Murphy K

    (2016) Should All Research Papers Be Free?

    New York Times. Accessed February 7, 2018.

    https://www.nytimes.com/2016/03/13/opinion/sunday/should-all-research-papers-be-free.html
73. Data

    1. OpenCitations

    (authors) (2017a) OCC dataset of all the identifiers, made on 2017-07-25

    Figshare.

    https://doi.org/10.6084/m9.figshare.5255368.v1
74. Data

    1. OpenCitations

    (authors) (2017b) OCC dataset of all the bibliographic resources, made on 2017-07-25

    Figshare.

    https://doi.org/10.6084/m9.figshare.5255365.v1
75. Website

    1. Oxenham S

    (2016) Meet the Robin Hood of Science

    Big Think. Accessed February 7, 2018.

    http://bigthink.com/neurobonkers/a-pirate-bay-for-science
76. 1. Paul EM
    2. de Vries Saskia CJ
    3. Johan R

    (2017) The transition to open access: The state of the market, offsetting deals, and a demonstrated model for fair open access with the open library of humanities

    Stand Alone.

    https://doi.org/10.3233/978-1-61499-769-6-118

    • Google Scholar
77. 1. Peroni S
    2. Dutton A
    3. Gray T
    4. Shotton D

    (2015) Setting our bibliographic references free: towards open citation data

    Journal of Documentation 71:253–277.

    https://doi.org/10.1108/JD-12-2013-0166

    • Google Scholar
78. Data

    1. Peroni S
    2. Shotton D

    (authors) (2016) Metadata for the OpenCitations Corpus

    Figshare.

    https://doi.org/10.6084/m9.figshare.3443876.v3
79. 1. Piwowar H
    2. Priem J
    3. Larivière V
    4. Alperin JP
    5. Matthias L
    6. Norlander B
    7. Farley A
    8. West J
    9. Haustein S

    (2018) The State of OA: A large-scale analysis of the prevalence and impact of Open Access articles

    PeerJ 6:e4375.

    https://doi.org/10.7717/peerj.4375

    • Google Scholar
80. Data

    1. Piwowar H
    2. Priem J
    3. Larivière V
    4. Alperin JP
    5. Matthias L
    6. Norlander B
    7. Farley A
    8. West J
    9. Haustein S

    (authors) (2017) Data from: The State of OA: A large-scale analysis of the prevalence and impact of Open Access articles

    Zenodo.

    https://doi.org/10.5281/zenodo.837902
81. Website

    1. Piwowar H

    (2016) Introducing oaDOI: resolve a DOI straight to OA

    Impactstory Blog. Accessed February 7, 2018.

    http://blog.impactstory.org/introducting-oadoi/
82. 1. Powell K

    (2016) Does it take too long to publish research?

    Nature 530:148–151.

    https://doi.org/10.1038/530148a

    • Google Scholar
83. 1. Prosser DC

    (2011) Reassessing the value proposition: first steps towards a fair(er) price for scholarly journals

    Serials: The Journal for the Serials Community 24:60–63.

    https://doi.org/10.1629/2460

    • Google Scholar
84. Website

    1. Rogers J

    (2012) Walking away from the American Chemical Society

    Attempting Elegance. Accessed February 7, 2018.

    http://www.attemptingelegance.com/?p=1765
85. 1. Roth DL

    (1990) The Serials Crisis Revisited

    The Serials Librarian 18:123–129.

    https://doi.org/10.1300/J123v18n01_09

    • Google Scholar
86. Website

    1. Rovner SL

    (2014) Online Access To ACS Publications Is Restored After Some Customers Were Unintentionally Blocked

    Chemical & Engineering News. Accessed February 7, 2018.

    http://cen.acs.org/articles/92/web/2014/04/Online-Access-ACS-Publications-Restored.html
87. Website

    1. Royster P

    (2016) A Brief History of Open Access

    Library Conference Presentations and Speeches. Accessed February 7, 2018.

    http://digitalcommons.unl.edu/library_talks/123
88. 1. Rubin DB
    2. Schenker N

    (1987) Logit-Based Interval Estimation for Binomial Data Using the Jeffreys Prior

    Sociological Methodology 17:131.

    https://doi.org/10.2307/271031

    • Google Scholar
89. 1. Satyanarayan A
    2. Moritz D
    3. Wongsuphasawat K
    4. Heer J

    (2017) Vega-Lite: A Grammar of Interactive Graphics

    IEEE Transactions on Visualization and Computer Graphics 23:341–350.

    https://doi.org/10.1109/TVCG.2016.2599030

    • PubMed
    • Google Scholar
90. 1. Schiermeier Q
    2. Mega ER

    (2016) Scientists in Germany, Peru and Taiwan to lose access to Elsevier journals

    Nature 541:13.

    https://doi.org/10.1038/nature.2016.21223

    • PubMed
    • Google Scholar
91. 1. Schiermeier Q

    (2015) Pirate research-paper sites play hide-and-seek with publishers

    Nature.

    https://doi.org/10.1038/nature.2015.18876

    • Google Scholar
92. 1. Schiermeier Q

    (2017a) US court grants Elsevier millions in damages from Sci-Hub

    Nature.

    https://doi.org/10.1038/nature.2017.22196

    • Google Scholar
93. 1. Schiermeier Q

    (2017b) Pirate paper website Sci-Hub dealt another blow by US courts

    Nature.

    https://doi.org/10.1038/nature.2017.22971

    • Google Scholar
94. 1. Schiermeier Q

    (2018) Germany vs Elsevier: universities win temporary journal access after refusing to pay fees

    Nature 553:137.

    https://doi.org/10.1038/d41586-018-00093-7

    • PubMed
    • Google Scholar
95. 1. Shen C
    2. Björk BC

    (2015) 'Predatory' open access: a longitudinal study of article volumes and market characteristics

    BMC Medicine 13:230.

    https://doi.org/10.1186/s12916-015-0469-2

    • PubMed
    • Google Scholar
96. Website

    1. Silver A

    (2017) Sci-Hub domains inactive following court order

    The Register. Accessed February 7, 2018.

    http://www.theregister.co.uk/2017/11/23/scihubs_become_inactive_following_court_order/
97. 1. Singh Chawla D

    (2017a) Publishers go after networking site for illicit sharing of journal papers

    Science.

    https://doi.org/10.1126/science.aaq0132

    • Google Scholar
98. 1. Singh Chawla D

    (2017b) Publishers take ResearchGate to court, alleging massive copyright infringement

    Science.

    https://doi.org/10.1126/science.aaq1560

    • Google Scholar
99. 1. Singh Chawla D

    (2017c) Need a paper? Get a plug-in

    Nature 551:399–400.

    https://doi.org/10.1038/d41586-017-05922-9

    • PubMed
    • Google Scholar
100. 1. Singh Chawla D

     (2017d) Unpaywall finds free versions of paywalled papers

     Nature.

     https://doi.org/10.1038/nature.2017.21765

     • Google Scholar
101. Website

     1. Standish R

     (2017) The World’s Largest Free Scientific Resource Is Now Blocked in Russia

     Foreign Policy. Accessed February 7, 2018.

     http://foreignpolicy.com/2017/09/06/the-worlds-largest-free-scientific-resource-is-now-blocked-in-russia/
102. Website

     1. Suber P

     (2008) Gratis and libre open access

     SPARC Open Access Newsletter. Accessed February 7, 2018.

     https://dash.harvard.edu/handle/1/4322580
103. Software

     1. Suber P

     (2017) Open Access

     MIT Press.

     https://mitpress.mit.edu/books/open-access
104. 1. Sutton C

     (2011) Is free inevitable in scholarly communication?: The economics of open access

     College & Research Libraries News 72:642–645.

     https://doi.org/10.5860/crln.72.11.8671

     • Google Scholar
105. Website

     1. Sweet RW

     (2015) Memorandum & Opinion – Document #53 of Elsevier Inc. v. Sci-Hub – Case 1:15-cv-04282-RWS

     Southern District Court of New York. Accessed February 7, 2018.

     https://www.courtlistener.com/docket/4355308/53/elsevier-inc-v-sci-hub/
106. Website

     1. Sweet RW

     (2017) Judgement – Document #87 of Elsevier Inc. v. Sci-Hub – Case 1:15-cv-04282-RWS

     Southern District Court of New York. Accessed February 7, 2018.

     https://www.documentcloud.org/documents/3878258-2017-06-21-Elsevier-Sci-Hub-Final-Judgement.html
107. 1. Tennant JP
     2. Waldner F
     3. Jacques DC
     4. Masuzzo P
     5. Collister LB
     6. Hartgerink CH

     (2016) The academic, economic and societal impacts of Open Access: an evidence-based review

     F1000Research 5:632.

     https://doi.org/10.12688/f1000research.8460.3

     • PubMed
     • Google Scholar
108. 1. Travis J

     (2016) In survey, most give thumbs-up to pirated papers

     Science.

     https://doi.org/10.1126/science.aaf5704

     • Google Scholar
109. Data

     1. Tzovaras BG

     (authors) (2018) Sci-Hub download log of 2017

     Zenodo.

     https://doi.org/10.5281/zenodo.1158301
110. Website

     1. Van der Sar E

     (2015a) Elsevier Cracks Down on Pirated Scientific Articles

     TorrentFreak. Accessed February 7, 2018.

     https://torrentfreak.com/elsevier-cracks-down-on-pirated-scientific-articles-150609/
111. Website

     1. Van der Sar E

     (2015b) Court Orders Shutdown of Libgen, Bookfi and Sci-Hub

     TorrentFreak. Accessed February 7, 2018.

     https://torrentfreak.com/court-orders-shutdown-of-libgen-bookfi-and-sci-hub-151102/
112. Website

     1. Van der Sar E

     (2015c) Sci-Hub, BookFi and LibGen Resurface After Being Shut Down

     TorrentFreak. Accessed February 7, 2018.

     https://torrentfreak.com/sci-hub-and-libgen-resurface-after-being-shut-down-151121/
113. Website

     1. Van der Sar E

     (2017a) Sci-Hub Ordered to Pay $15 Million in Piracy Damages

     TorrentFreak. Accessed February 7, 2018.

     https://torrentfreak.com/sci-hub-ordered-to-pay-15-million-in-piracy-damages-170623/
114. Website

     1. Van der Sar E

     (2017b) New Lawsuit Demands ISP Blockades Against “Pirate” Site Sci-Hub

     TorrentFreak. Accessed February 7, 2018.

     https://torrentfreak.com/new-lawsuit-demands-isp-blockades-against-pirate-site-sci-hub-170629/
115. 1. van Leewenhoeck A

     (1677) Observations, communicated to the publisher by Mr. Antony van Leewenhoeck, in a dutch Letter of the 9th Octob. 1676. here English'd: concerning little animals by him observed in rain-well-sea- and snow water; as also in water wherein pepper had lain infused

     Philosophical Transactions of the Royal Society of London 12:821–831.

     https://doi.org/10.1098/rstl.1677.0003

     • Google Scholar
116. 1. Van Noorden R

     (2012) Trouble at the text mine

     Nature 483:134–135.

     https://doi.org/10.1038/483134a

     • PubMed
     • Google Scholar
117. 1. Van Noorden R

     (2013a) Half of 2011 papers now free to read

     Nature 500:386–387.

     https://doi.org/10.1038/500386a

     • Google Scholar
118. 1. Van Noorden R

     (2013b) Open access: The true cost of science publishing

     Nature 495:426–429.

     https://doi.org/10.1038/495426a

     • PubMed
     • Google Scholar
119. 1. Van Noorden R

     (2014) Funders punish open-access dodgers

     Nature 508:161.

     https://doi.org/10.1038/508161a

     • PubMed
     • Google Scholar
120. 1. Van Noorden R

     (2015) Nature owner merges with publishing giant

     Nature.

     https://doi.org/10.1038/nature.2015.16731

     • Google Scholar
121. 1. Van Noorden R

     (2016) Alexandra Elbakyan: Paper pirate

     Nature.

     https://doi.org/10.1038/540507a

     • Google Scholar
122. 1. Vogel G

     (2017) A bold open-access push in Germany could change the future of academic publishing

     Science.

     https://doi.org/10.1126/science.aap7562

     • Google Scholar
123. 1. Wang J

     (2007) Digital Object Identifiers and Their Use in Libraries

     Serials Review 33:161–164.

     https://doi.org/10.1080/00987913.2007.10765116

     • Google Scholar
124. 1. Westergaard D
     2. Stærfeldt H-H
     3. Tønsberg C
     4. Jensen LJ
     5. Brunak S

     (2017) A comprehensive and quantitative comparison of text-mining in 15 million full-text articles versus their corresponding abstracts

     PLoS Computational Biology 14:e1005962.

     https://doi.org/10.1371/journal.pcbi.1005962

     • Google Scholar
125. Website

     1. Woltermann A

     (2015) Declaration in Support of Motion – Document #8 Attachment #23 of Elsevier Inc. v. Sci-Hub – Case 1:15-cv-04282-RWS

     Southern District Court of New York. Accessed February 7, 2018.

     https://www.courtlistener.com/docket/4355308/8/23/elsevier-inc-v-sci-hub/
126. 1. Woolston C

     (2016) Paper piracy sparks online debate

     Nature.

     https://doi.org/10.1038/nature.2016.19841

     • Google Scholar

Thank you for submitting your article "Sci-Hub provides access to nearly all scholarly literature" to eLife for consideration as a Feature Article. Your article has been reviewed by three peer reviewers and the evaluation has been overseen by Peter Rogers as the eLife Features Editor.

The following individual involved in review of your submission has agreed to reveal their identity: David Prosser (peer reviewer).

Overall the reviewers were positive about the article, but they had a small number of concerns. We therefore invite you to prepare a revised submission that addresses these concerns.

Summary:

This is an interesting and original piece of work that focusses on one of the most potentially disruptive forces in scholarly communication in the 21st Century. It outlines, for I think the first time, the shear size and scope of Sci-Hub and the amount of material that is now available through it. It is perhaps a little early to say with any certainly what the effect of Sci-Hub will be on subscriptions and I am pleased that the authors have highlighted the issue, but not moved too far into the space of prediction. They have allowed the analysis to speak for itself.

Essential revisions:

1) The authors make one claim that seems to me not supported by the evidence. They claim that their paper shows that toll-based publishing is becoming unsustainable. But they also point to a recent study that estimates that the ratio of the number of times papers are downloaded from the publisher to downloaded from Sci-Hub is 48:1 for Elsevier and 20:1 for Royal Society of Chemistry. This suggests that Sci-Hub so far has very little influence on the subscription demand for journal articles.

2) To calculate coverage, DOIs from Crossref and Sci-Hub for over 56M records are compared. Please give details of the algorithm used to compare the DOIs.

A straight comparison of 56M DOI's with another similarly large number of DOIs would require at least 2.5 X 10¹⁴ comparisons. Was it done this way? If so, how long did it take? Was another clustering method used?

And if such comparisons were made, why are the percentages of coverage only given to third order? Is this just for convenience? (Giving the percentages of coverage to third order seems to imply a sample and not a complete comparison).

3) I believe if the authors compare Figure 4 with Table 2 in Piwowar et al., 2017, a strong negative correlation can be observed whereby the more likely a paper is to be found on the web, the less likely it is in Sci-Hub.

We'd like to thank the reviewers and editors for their feedback, which we have aimed to address in our revised manuscript, corresponding to version 3 of the study's preprint. Sci-Hub and scholarly publishing are both rapidly developing topics. As such, the revised manuscript contains additions resulting from new developments since version 2. These include the judgment in the ACS suit and the subsequent suspension of four Sci-Hub domains; the release of Sci-Hub download logs for 2017, which we visualize in Figure 1—figure supplement 1; and additional withdrawals from Sci-Hub's bitcoin addresses. As such, we extended the Google trends and Bitcoin analyses through the end of 2017. The revised version contains many enhancements, of which the major ones are summarized here.

We updated our journal information to the October 2017 release of Scopus. In addition, we created patches to standardize publisher names in Scopus. As a result, many of the numbers reported in the manuscript have changed slightly. Thomas Munro assisted with the Scopus enhancements, as well as providing feedback in other areas, and has been added as an author.

We investigated the level of access provided by the University of Pennsylvania Libraries. This is a major addition to the study, since it allows us to compare Sci-Hub's database with the access of a major research university. Furthermore, the data for this analysis are openly available, making it the most comprehensive public analysis of a university's coverage of the scholarly literature (that we're aware of). Jacob Levernier, who helped perform these analyses, has been added as an author.

[…] Essential revisions:

1) The authors make one claim that seems to me not supported by the evidence. They claim that their paper shows that toll-based publishing is becoming unsustainable. But they also point to a recent study that estimates that the ratio of the number of times papers are downloaded from the publisher to downloaded from Sci-Hub is 48:1 for Elsevier and 20:1 for Royal Society of Chemistry. This suggests that Sci-Hub so far has very little influence on the subscription demand for journal articles.

In light of this reviewer feedback, we recently discussed the effect of Sci-Hub on the toll access business model in greater depth. While there is disagreement on the matter, we have added additional evidence and reasoning to the manuscript regarding these claims.

The 48:1 and 20:1 estimates from Gardner et al. assess Sci-Hub's usage at the start of 2016. These numbers are, of course, small. However, awareness of Sci-Hub was also low at that time. For example, a survey conducted in the first half of 2016, found that only 19% of Latin American medical students were aware of Sci-Hub. An online survey by Science in May 2016 found 59% of 10,874 respondents had used Sci-Hub, although it noted "the survey sample is likely biased heavily toward fans of the site". Incidentally, the 62% of respondents thought "Sci-Hub will disrupt the traditional science publishing industry".

An important factor here is the rate at which Sci-Hub adoption will grow. Our manuscript now includes two distinct metrics for assessing Sci-Hub's annual growth: 79% according to download statistics provided by Sci-Hub and 88% according to peak Google search interest following service outage. In addition, Sci-Hub released access logs for 2017, which we analyze in Figure 1—figure supplement 1 to show how Sci-Hub downloads per days have increased over time. We observe rapid growth in 2017, peaking in July 2017 with an average of 593,880 downloads per day. Combining the new Sci-Hub log data with information from the Penn Libraries, we calculate that Sci-Hub usage for electronic access to scholarly articles exceeds Penn's usage by a ratio of 20:1. We now discuss how Sci-Hub usage could lead to subscription cancellations by affecting the usage metrics and feedback librarians use to evaluate subscriptions. Finally, we discuss the growth of green open access, composed of preprints and postprints, which may also diminish the need for subscription access.

These developments are happening in the context of library budgets that are increasingly burdened by subscription costs, as mentioned in our Introduction. Since our manuscript submission, SPARC released a Big Deal Cancellation Tracker, which supports our observation that large-scale subscription cancellations are becoming more prevalent. We now reference this resource as well as the latest coverage of the Project DEAL negations with Elsevier in Germany. We also cover the domain name suspensions following the ACS suit judgment and Sci-Hub's re-emergence at several new domains. We modified a few sentences to make it clear that our assessment of Sci-Hub's disruptive influence is a prediction and not a certainty.

2) To calculate coverage, DOIs from Crossref and Sci-Hub for over 56M records are compared. Please give details of the algorithm used to compare the DOIs.

A straight comparison of 56M DOI's with another similarly large number of DOIs would require at least 2.5 X 10¹⁴ comparisons. Was it done this way? If so, how long did it take? Was another clustering method used?

And if such comparisons were made, why are the percentages of coverage only given to third order? Is this just for convenience? (Giving the percentages of coverage to third order seems to imply a sample and not a complete comparison).

We thank the reviewers for their interest in the computational methods used for our coverage calculations. Python includes a set data type to store unordered collections with no duplicate elements. Set operations enable efficient membership testing and intersection with other sets. As a result, we do not need to apply a list intersection algorithm, as described by the reviewers, that would scale quadratically due to nested iteration.

We evaluate each Crossref DOI for membership in the set of Sci-Hub DOIs in 01.catalog-dois.ipynb. The specific implementation we apply passes a set of Sci-Hub DOIs to the pandas.Series.isin function. This function uses a hash table, the formative data structure behind sets, to enable an efficient comparison of DOIs. The 01.catalog-dois.ipynb notebook does take considerable time (perhaps up to an hour) to execute. However, much of this time is spend on file input/output, which requires reading/writing to disk and (de)compressing streams, both of which can be runtime intensive.

We create a tabular file doi.tsv.xz that contains a binary indicator (coded as 0/1) of whether each Crossref DOI is in Sci-Hub's repository. The general workflow we followed to compute coverage was to subset the DOI dataset for the relevant DOIs (e.g. all DOIs published in 2016) and take the sum/mean of the binary membership column. These operations made heavy use of pandas.DataFrame.join to incorporate additional information about DOIs and pandas.DataFrame.groupby to compute coverage for all possible subsets at once.

3) I believe if the authors compare Figure 4 with Table 2 in Piwowar et al., 2017, a strong negative correlation can be observed whereby the more likely a paper is to be found on the web, the less likely it is in Sci-Hub.

We believe this comment is comparing Figure 4 of the OA study (titled "Percentage of different access types […] per NSF discipline", also see Figure A5) with Figure 4 of our study (titled "Coverage by journal attributes"). For example, among plotted disciplines, Sci-Hub coverage is highest for chemistry, whereas oaDOI's coverage is lowest for chemistry. On the other hand, oaDOI exceeds 50% coverage of mathematics, whereas mathematics coverage is relatively low in Sci-Hub.

Regarding the hypothesis, "the more likely a paper is to be found on the web, the less likely it is in Sci-Hub", we evaluate this directly in the "Coverage by category of access status" section. This hypothesis appears true for most types of gratis web access. Compared to closed articles, bronze, hybrid, and gold OA articles are less likely to be in Sci-Hub. However, green articles (available without charge, but not from the publisher) do not seem have lower coverage than closed articles in Sci-Hub. The conclusion from the reviewers' comparison could therefore be rephrased as "the less gratis availability of a discipline's articles on the web (according to oaDOI), the greater the coverage on Sci-Hub." One possibility is that researchers from disciplines with poor oaDOI coverage more frequently encounter access problems, leading to greater awareness and usage of Sci-Hub.

Author details

1. Daniel S Himmelstein

   Daniel S Himmelstein is in the Department of Systems Pharmacology and Translational Therapeutics, University of Pennsylvania, Philadelphia, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Supervision, Validation, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   daniel.himmelstein@gmail.com

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-3012-7446

2. Ariel Rodriguez Romero

   Ariel Rodriguez Romero is at Bidwise, Inc, Miami, United States

   Contribution

   Software, Investigation, Visualization, Methodology

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2290-4927

3. Jacob G Levernier

   Jacob G Levernier is in Library Technology Services and Strategic Initiatives, University of Pennsylvania, Philadelphia, United States

   Contribution

   Data curation, Software, Formal analysis, Validation, Investigation, Methodology, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-1563-7314

4. Thomas Anthony Munro

   Thomas Anthony Munro is in the School of Life and Environmental Sciences, Deakin University, Melbourne, Australia

   Contribution

   Data curation, Methodology, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-3366-7149

5. Stephen Reid McLaughlin

   Stephen Reid McLaughlin is in the School of Information, University of Texas at Austin, Austin, United States

   Contribution

   Conceptualization, Methodology, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-9888-3168

6. Bastian Greshake Tzovaras

   Bastian Greshake Tzovaras is in the Department of Applied Bioinformatics, Institute of Cell Biology and Neuroscience, Goethe University, Frankfurt, Germany

   Contribution

   Data curation, Formal analysis, Investigation, Methodology, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-9925-9623

7. Casey S Greene

   Casey S Greene is in the Department of Systems Pharmacology & Translational Therapeutics, University of Pennsylvania, Philadelphia, United States

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Methodology, Project administration, Writing—review and editing

   For correspondence

   greenescientist@gmail.com

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8713-9213

• 227,097

  views

• 4,645

  downloads

• 90

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.