While there is substantial disagreement on the best route to achieve open access, the idea that research outputs should be freely available is broadly shared. Over the past decade, there has been a large increase in the volume of publications available open access (Piwowar et al., 2018), and this looks set to continue: indeed, a recent projection estimates that by 2025, 44% of all journal articles will be available as open access (OA) and that 70% of article views will be to OA articles (Piwowar et al., 2019).

This massive increase has largely been driven by policy initiatives. Medical research funders in the UK, such as the Wellcome Trust and Medical Research Council, and the National Institutes of Health in the US have led a wide range of funder policy interventions. Universities such as Harvard University, University of Liège, University of Southampton and others developed local polices and infrastructures that became more widely adopted. In 2018, a coalition of funders set out an initiative called Plan S that requires all scholarly publications funded by public grants to be made immediately open access. This is the most ambitious, and therefore the most controversial, policy initiative to date with questions raised about the approach (Rabesandratana, 2019; Haug, 2019; Barbour and Nicholls, 2019), implementation details (McNutt, 2019; Gómez-Fernández, 2019; Brainard, 2019; Agustini and Berk, 2019), and unintended side effects for existing programs outside North America and North-western Europe (Debat and Babini, 2019; Aguado-López and Becerril-García, 2019).

A recent report showed a link between the monitoring of policy and its effectiveness, demonstrating that research outputs supported by funders that implemented monitoring and compliance checks for their policies were more likely to be published open access (Larivière and Sugimoto, 2018). By comparison, open access for works funded by Canadian funders, which did not monitor compliance, were shown to lag substantially even when disciplinary effects were taken into account.

There is a need for critical and inclusive evaluation of open access performance that can address regional and political differences. For example, the SciELO project has successfully implemented an electronic publishing model for journals resulting in a surge of publisher-mediated open access (Packer, 2009; Wang et al., 2018). Recent work showed that, for biomedical research, there was a greater level of open access for articles published from countries with a lower GDP, particularly for those in sub-Saharan Africa (Iyandemye and Thomas, 2019). This provides evidence of national or regional effects on publication cultures that lead to open access. Meanwhile, another study showed that, for the field of Global Health, lower-ranked institutions are more likely to publish in closed outlets (Siler et al., 2018). They suggest this is due to the cost of article processing charges showing the importance of considering institutional context when examining open access performance.

Despite the scale and success (at least in some areas) of policy interventions, there is limited comparative and quantitative research about which policy interventions have been the most successful. In part this is due to a historical lack of high-quality data on open access, the heterogeneous nature of the global scholarly publishing endeavour, and the consequent lack of any baseline against which to make comparisons.

To map the rate and degree of progress to open access, we developed a reproducible workflow capable of quantifying a wide range of open access characteristics at the institutional level. The overall workflow is summarised diagrammatically in Figure 1. This includes mapping open access definitions and the Unpaywall information we used to construct them. Briefly, we gather output metadata from searches in Microsoft Academic (Sinha et al., 2015; Wang et al., 2019), Web of Science and Scopus, for each university. From this full set we gather the corresponding Crossref DOIs from the metadata of each output focusing on this set. Unpaywall is consulted to determine open access status. Detailed discussions of the data sources, precise data snapshots used, and technical details of the data infrastructure can be found in Supplementary file 1. The code used in the workflow is available via Zenodo at the following link.

Figure 1

Download asset Open asset

We have decided to focus mainly on total open access, publisher-mediated open access (i.e., gold), and repository-mediated open access (i.e., green) due to the ease of comprehension, data quality, and ability to show potential effects of selected policies (Table 1). Levels of hybrid open access and green in home repository for selected universities (ones for which we have more confidence in the data) are also included to further support the analysis of policy effects.

As we have noted previously (Huang et al., 2020a), there is a sensitivity associated to the choices in bibliographic data sources when they are used to create a ranking. For this analysis we therefore chose to combine all three datasets: Microsoft Academic, Web of Science and Scopus. In the companion white paper (Huang et al., 2020b) we provide a comprehensive sensitivity analysis on the use of these different datasets, the use of different versions of Unpaywall, and the relations between confidence levels and sample size.

Briefly, it is our view that to provide a robust assessment of open access performance the following set of essential requirements must be met:

1. The set of outputs included in each category and a traceable description of how they were collected must be transparently described. Provided here by a description of the data sources and the procedures used to collect DOIs for each institution (Supplementary file 1). In this article, institutions define the categories of outputs but they could also be categorised by individuals, disciplines, or countries etc.

2. A clearly defined, open and auditable data source on open access status. Provided here by a defined and identified Unpaywall snapshot (Supplementary file 1).

3. A clearly defined and implementable description of how open access status data is interpreted in the form of the SQL query used to establish open access status categories for each DOI (Figure 1 and Supplementary file 1). We decided to include the checkmark 'is_oa = TRUE’ in the description of Bronze open access as this makes a slight difference to the numbers obtained from Unpaywall.

4. Provision of derived data and analysis in auditable form. Provided here the derived data as open data (Huang et al., 2020c), code for the analysis of derived data as Jupyter notebooks (Huang et al., 2020d), and upstream data analysis in the form of SQL queries used (Huang et al., 2020c).

We have limited our data sharing in two ways. Firstly, we do not provide the full list of DOIs obtained from each source, due to Terms of Service restrictions. Secondly, we have not identified institutions individually except for those that fall within the top 100 globally for total open access, publisher-mediated open access, or repository-mediated open access. Both derived datasets of the de-anonymised top 100 and the full dataset containing all institutions in anonymised form are made available (Huang et al., 2020c).

Our results have significant implications for the details of policy interventions. Firstly, we have demonstrated the ability to detect signals of policy interventions in the behaviour of institutions. We see potential effects and results arising from the efforts of national funders and policy makers, particularly in the UK. The combined policy change and funding provided by the UK Research Councils in 2012 is associated with an increase in the level of publisher-mediated open access, and the level of increase appears to be associated with the level of funding provided. Similarly, the requirement for outputs to be deposited in a repository for eligibility for the 2021 Research Excellence Framework is associated with substantial increase in repository-mediated open access around 2015.

These findings may also have implications for deciding on the effectiveness of directly funding open access publishing. It is perhaps surprising to some readers that the overall levels of publisher-mediated open access in the UK are not higher. Specific funders, most notably the Wellcome Trust, have achieved very high levels of open access for articles from research they support through the provision of funding for open access publication. In addition, the UK Research Councils invested significant resources in supporting gold open access. However, these have not translated to high levels of publisher-mediated open access across the full diversity of outputs of UK institutions. The majority gains over the past five years have come from repository-mediated open access.

In the animated version of Figure 3 (Figure 3—animation 1), there is a clear signal of saturation with respect to publisher-mediated open access (gold open access) for European and North American universities. With few exceptions, institutions do not achieve levels of gold open access greater than 40% and this level is stable from 2014 to 2018. Similarly, in Figure 4 we see evidence of shifts in response to stimuli (funding and policy interventions) which then stabilise. Even those UK universities with very high levels of repository-mediated open access (green) see a slowing down of the rise in levels a few years after the Research Excellence Framework policy intervention.

These signals suggest that achieving 100% open access may be very difficult, and possibly expensive to achieve. There will always be areas and cases where open access is challenging. Current challenges include disciplinary areas such as the humanities where suitable business models and venues are still developing, as well as types of content, particularly books, where there is greater overlap between scholarly and general publishing. Both of these could be addressed rapidly through direct funding, but this may not be the most efficient approach. Achieving ‘100% open access’ may therefore require a tighter definition of exactly which outputs are in scope. For those areas where we see signals of saturation much lower than 100% these are likely signals of the complexity of the system, and of large categories of outputs where open access is harder to achieve, or the motivation of institutions (including authors, libraries, and other support staff) to achieve it is lower.

In comparison to results presented by Robinson-Garcia et al., 2019, Robinson-Garcia et al., 2020, our analysis shows significantly improved performance of Latin American universities. This is driven by our inclusion of data sources beyond Web of Science, which has resulted in increased publisher-mediated open access for Latin America (Huang et al., 2020b). Similarly, our results also depict more clear observations of some high performing African and Asian universities. These produce a slightly different view in contrast to the one where UK universities dominate the top-ranking positions for open access.

For Latin America, this result indicates how effective infrastructures such as SciELO can be at supporting the uptake of open access practices. In the case of Africa, there may be effects of funder requirements (with funders such as the Bill and Melinda Gates Foundation and Wellcome Trust that have strong open access requirements playing a significant role) as well as disciplinary spread. In both cases we are likely to have a limited view of the full diversity of research outputs due to their poor capture in information systems from the North Atlantic.

The continued leadership of Latin American institutions on publisher-mediated open access levels is the continuation of a trend set more than a decade ago through the provision of publishing infrastructures. Taken alongside the clear response in the Netherlands for hybrid open access in response to publish and read agreements, this suggests that increasing levels of open access publishing through article processing charges is potentially expensive compared to the costs of providing infrastructure.

Another interesting natural experiment is how the strength of funder actions is associated with overall change in levels of open access. In the Netherlands and the UK in particular, but also in the US, where funder policies have moved from encouragement, to mandates, to monitoring with sanctions for non-compliance, there are substantial shifts in overall levels of open access. By contrast, in countries where policy remains effectively at the level of a recommendation, such as Australia, levels of open access lag significantly. Recent increases in reporting requirements by Australian funders might therefore be expected to lead to a detectable signal over the next 12–24 months.

The value of analysis at the level of universities is that we gain a picture of open access performance across a diverse research ecosystem. We see differences across countries and regions, and differences between universities within countries. Overall, we see that there are multiple different paths towards improving access, and that different paths may be more or less appropriate in different contexts. Most importantly, while further research is needed to unpick the details of the differences in open access provision, we hope this work provides a framework that enables this longitudinal analysis to be taken forward and used wherever it is needed.

Our analysis process includes automated approaches for collecting the outputs related to specific universities, and the analysis of those outputs. Currently the addition of new universities, and the updating of large data sources is partly manual (Supplementary file 1), but we also expect to automate this in the near future. As a result, our analysis has limitations in its capacity to provide comparable estimates of open access status across all universities, but does provide a reproducible and transparent view of overall global performance.

There are challenges to be addressed with respect to small universities and research organisations, and we have taken a necessarily subjective view of which institutions to include (see Supplementary file 1). Our approach systematically leaves out universities with very small numbers of outputs (i.e., less than 100 outputs), and universities with very extreme open access proportions as these are the universities for which we have less statistical confidence in the results. This is also in-line with our intended focus on research-intensive universities. These small institutions are of significant interest but will require a different analysis approach. We include a country-level summary result for the full set of institutions (including small institutions left out from the main article) in our data set in Figure 2—figure supplement 3. We recognise that the inclusion of universities beyond our initial selection can also potentially change the results.

We have used multiple sources of bibliographic information with the goal of gaining a more inclusive view of research outputs. Despite this, there are still limitations in the coverage of these data sources, and a likely bias towards science, technology, engineering and maths (STEM) disciplines. In addition, the focus of Unpaywall on analysis of outputs with Crossref DOIs means that we are missing outputs for disciplines (humanities) and output types (books) where the use of DOIs is lower. While we have performed additional sensitivity analysis on the use of different data sources (Huang et al., 2020b), further work is required to understand the relationships between disciplinary biases in data sources and the influence of external policies (such as those of the National Institutes of Health and the Wellcome Trust) on repositories. In addition, due to the nature of this work and to limitations on the use of Web of Science and Scopus APIs, we have collected data from these two sources over a period of time. Although we expect such changes to be small, those effects are not clearly represented in our data. For other data sources we are able to precisely define the data dump used for our analysis, supporting reproducibility as well as modified analyses. We also recognise that our longitudinal analysis is done post hoc from a fixed time point (i.e., looking back). Our results for a given year of publication will not be the same as the levels of open access that would have been observed in that year, due to embargoes and moving paywalls.

In this article, we have drawn comparisons between publisher-mediated open access and repository-mediated open access to identify different institutional paths to open access. The overall trends are retained if we separate out outputs that are only available through repositories (i.e., not free to access via publishers) and compare these to publisher-mediated open access instead. However, we recognise that at the individual university level, this can be dependent on institutional policy and whether the corresponding data can be properly captured by current systems that determines open access status.

There is significant opportunity for improving the data sources on sets of outputs and how they can be grouped (e.g. by people, discipline, organisation, country, etc.). Improvements to institutional identifier systems such as the Research Organisation Registry, increased completeness of metadata records, particularly that provided by publishers via Crossref on affiliation, ORCIDs and funders, and enhancing the coverage of open access status data (for instance by incorporating data from CORE and BASE), will all enhance coverage. There are also opportunities to expand the coverage by incorporating a wider range of bibliographic data sources.

We have sought to make our methodology and approach as reproducible and reusable as is practicable. The main challenges lie in the level of accessibility to the closed data sources (i.e., Web of Science and Scopus) and more generally with data sources that are not available in the form of identifiable snapshots (see Supplementary file 1). In addition, semi-automated processing and manual searches are required to link institutional identifiers across multiple data sources (as discussed in detail in Supplementary file 1). For the rest of the framework, we have provided transparent descriptions and source code that allow reprocessing, as described in the next section. Further work is required to provide a complete and deployable framework for replication but we are making progress in this area (see https://github.com/The-Academic-Observatory/observatory-platform for an example).

Our technical infrastructure is constructed based on the aim to make openly available both the data and analysis code as much as possible. The derived datasets for analysis, visualisation codes, and the SQL queries used to generate the derived datasets are all made available online as described in Supplementary file 1. Raw data is not provided to preserve the anonymity of institutions and respect the terms of service of data providers.

1. 1. Huang C-K
   2. Neylon C
   3. Hosking R
   4. Montgomery L
   5. Wilson K
   6. Ozaygen A
   7. Brookes-Kenworthy C

   (2020) Zenodo

   Data and Intermediate Queries for: Evaluating institutional open access performance: Methodology, challenges and assessment.

   https://doi.org/10.5281/zenodo.3693221
2. 1. Huang C-K
   2. Neylon C
   3. Hosking R
   4. Montgomery L
   5. Wilson K
   6. Ozaygen A
   7. Brookes-Kenworthy C

   (2020) Zenodo

   Jupyter Notebooks for the article: Evaluating institutional open access performance: Methodology, challenges and assessment.

   https://doi.org/10.5281/zenodo.3716063

1. Website

   1. Aguado-López E
   2. Becerril-García A

   (2019) Latin America’s longstanding open access ecosystem could be undermined by proposals from the Global North [LSE Latin America and Caribbean]

   Accessed November 6, 2019.

   https://blogs.lse.ac.uk/latamcaribbean/2019/11/06/latin-americas-longstanding-open-access-ecosystem-could-be-undermined-by-proposals-from-the-global-north/
2. 1. Agustini B
   2. Berk M

   (2019) The open access mandate: be careful what you wish for

   Australian & New Zealand Journal of Psychiatry 53:1044–1046.

   https://doi.org/10.1177/0004867419864436

   • PubMed
   • Google Scholar
3. 1. Barbour G
   2. Nicholls S

   (2019)

   Open access: should one model ever fit all?

   Australian Quarterly 90:3–9.

   • Google Scholar
4. 1. Brainard J

   (2019) Scientific societies worry about threat from plan S

   Science 363:332–333.

   https://doi.org/10.1126/science.363.6425.332

   • Google Scholar
5. 1. Debat H
   2. Babini D

   (2019) Plan S in Latin America: a precautionary note

   PeerJ 7:e27834.

   https://doi.org/10.7287/peerj.preprints.27834v1

   • Google Scholar
6. 1. Gómez-Fernández JC

   (2019) Plan S for publishing science in an open access way: not everyone is likely to be happy

   Biophysical Reviews 11:841–842.

   https://doi.org/10.1007/s12551-019-00604-4

   • Google Scholar
7. 1. Haug CJ

   (2019) No free lunch - What price Plan S for scientific publishing?

   New England Journal of Medicine 380:1181–1185.

   https://doi.org/10.1056/NEJMms1900864

   • PubMed
   • Google Scholar
8. 1. Huang CK
   2. Neylon C
   3. Brookes-Kenworthy C
   4. Hosking R
   5. Montgomery L
   6. Wilson K
   7. Ozaygen A

   (2020a) Comparison of bibliographic data sources: implications for the robustness of university rankings

   Quantitative Science Studies 13:1–34.

   https://doi.org/10.1162/qss_a_00031

   • Google Scholar
9. Software

   1. Huang C-K
   2. Neylon C
   3. Hosking R
   4. Montgomery L
   5. Wilson K
   6. Ozaygen A
   7. Brookes-Kenworthy C

   (2020b) Evaluating institutional open access performance: sensitivity analysis, version 3

   Zenodo.

   https://doi.org/10.5281/zenodo.3696857
10. Software

    1. Huang C-K
    2. Neylon C
    3. Hosking R
    4. Brookes-Kenworthy C
    5. Wilson K
    6. Montgomery L

    (2020c) Data and intermediate queries for: evaluating institutional open access performance: methodology, challenges and assessment, version 4.0

    Zenodo.

    https://doi.org/10.5281/zenodo.3693221
11. Software

    1. Huang C-K
    2. Neylon C
    3. Hosking R
    4. Brookes-Kenworthy C
    5. Montgomery L
    6. Wilson K
    7. Ozaygen A

    (2020d) Jupyter notebooks for the article: evaluating institutional open access performance: methodology, challenges and assessment, version 4.0

    Zenodo.

    https://doi.org/10.5281/zenodo.3716063
12. 1. Iyandemye J
    2. Thomas MP

    (2019) Low income countries have the highest percentages of open access publication: a systematic computational analysis of the biomedical literature

    PLOS ONE 14:e0220229.

    https://doi.org/10.1371/journal.pone.0220229

    • PubMed
    • Google Scholar
13. 1. Larivière V
    2. Sugimoto CR

    (2018) Do authors comply when funders enforce open access to research?

    Nature 562:483–486.

    https://doi.org/10.1038/d41586-018-07101-w

    • PubMed
    • Google Scholar
14. Website

    1. Lawson S

    (2018) RCUK open access block grant allocation 2013-18

    Figshare. Accessed March 1, 2020.

    https://figshare.com/articles/RCUK_open_access_block_grant_allocation_201317/4047315
15. 1. McNutt M

    (2019) Opinion: "Plan S" falls short for society publishers-and for the researchers they serve

    PNAS 116:2400–2403.

    https://doi.org/10.1073/pnas.1900359116

    • PubMed
    • Google Scholar
16. 1. Mongeon P
    2. Paul-Hus A

    (2016) The journal coverage of Web of Science and Scopus: a comparative analysis

    Scientometrics 106:213–228.

    https://doi.org/10.1007/s11192-015-1765-5

    • Google Scholar
17. Book

    1. Montgomery L
    2. Hartley J
    3. Neylon C
    4. Gillies M
    5. Gray E
    6. Herrmann-Pillath C
    7. Huang C-K
    8. Leach J
    9. Potts J
    10. Ren X
    11. Skinner K
    12. Sugimoto CR
    13. Wilson K

    (2018)

    Open Knowledge Institutions

    Reinventing Universities. MIT Press Work in Progress.

    • Google Scholar
18. 1. Packer AL

    (2009)

    The SciELO open access: a gold way from the south

    Canadian Journal of Higher Education 39:111–126.

    • Google Scholar
19. 1. Piwowar H
    2. Priem J
    3. Larivière V
    4. Alperin JP
    5. Matthias L
    6. Norlander B
    7. Farley A
    8. West J
    9. Haustein S

    (2018) The state of OA: a large-scale analysis of the prevalence and impact of open access articles

    PeerJ 6:e4375.

    https://doi.org/10.7717/peerj.4375

    • PubMed
    • Google Scholar
20. Preprint

    1. Piwowar H
    2. Priem J
    3. Orr R

    (2019) The future of OA: a large-scale analysis projecting open access publication and readership

    bioRxiv.

    https://doi.org/10.1101/795310

    • Google Scholar
21. 1. Rabesandratana T

    (2019) The world debates open-access mandates

    Science 363:11–12.

    https://doi.org/10.1126/science.363.6422.11

    • PubMed
    • Google Scholar
22. Preprint

    1. Robinson-Garcia N
    2. Costas R
    3. van Leeuwen TN

    (2019) Indicators of open access for universities

    arXiv.

    https://arxiv.org/abs/1906.03840

    • Google Scholar
23. Preprint

    1. Robinson-Garcia N
    2. Costas R
    3. van Leeuwen TN

    (2020) State of open access penetration in universities worldwide

    arXiv.

    https://arxiv.org/abs/2003.12273

    • Google Scholar
24. 1. Siler K
    2. Haustein S
    3. Smith E
    4. Larivière V
    5. Alperin JP

    (2018) Authorial and institutional stratification in open access publishing: the case of global health research

    PeerJ 6:e4269.

    https://doi.org/10.7717/peerj.4269

    • PubMed
    • Google Scholar
25. Conference

    1. Sinha A
    2. Shen Z
    3. Song Y
    4. Ma H
    5. Eide D
    6. Hsu B-J
    7. Wang K

    (2015)

    An overview of microsoft academic service (MAS) and applications

    Proceedings of the 24th International Conference on World Wide Web (WWW’ 15 Companion). ACM. pp. 243–246.

    • Google Scholar
26. 1. Van Noorden R

    (2019) Indonesia tops open access publishing charts

    Nature 1:5.

    https://doi.org/10.1038/d41586-019-01536-5

    • Google Scholar
27. 1. Wang X
    2. Cui Y
    3. Xu S
    4. Hu Z

    (2018) The state and evolution of gold open access: a country and discipline level analysis

    Aslib Journal of Information Management 70:573–584.

    https://doi.org/10.1108/AJIM-02-2018-0023

    • Google Scholar
28. 1. Wang K
    2. Shen Z
    3. Huang C
    4. Wu C-H
    5. Eide D
    6. Dong Y
    7. Qian J
    8. Kanakia A
    9. Chen A
    10. Rogahn R

    (2019) A review of Microsoft Academic Services for science of science studies

    Frontiers in Big Data 2:45.

    https://doi.org/10.3389/fdata.2019.00045

    • Google Scholar

Thank you for submitting your article "Evaluating institutional open access performance: Methodology, challenges and assessment" for consideration by eLife. Your article has been reviewed by 2 peer reviewers, and the evaluation has been overseen by two members of the eLife Features Team. The following individuals involved in review of your submission have agreed to reveal their identity: Bianca Kramer (Reviewer #1).

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

Summary:

This work provides a re-usable framework for tracking the open access performance of institutions and identifies instances where funding and policy interventions have changed the level of open access happening at the institutional, country or regional level.

Major comments to address:

1) As Figure 2 will be difficult to see and interpret in the web and PDF versions of the article, please can you replace this figure with Supplementary Figure 6. This will also be more in line with the examples given in the paper highlighting potential uses of the framework to analyse policy effects which are based on either comparing institutional results at country or region level for the full dataset (Figure 3), or on comparing specific institutions based on a common or differing policy context (Figure 4,5). Furthermore, including the ranking of top 100 institutions in terms of the proportion of different types of OA as a main result could invite undue focus on comparing individual institutions in a competitive way, which seems not in line with the aims of the work.

2) The paper introduces several analytical categories that it states should be used as the 'minimum reporting standard'. However, why these categories should be the 'minimum reporting standard' requires a much stronger justification. For example, the authors have included data sources with restricted access (Scopus, WoS) along with the Microsoft Academic Graph, to increase recall. Additionally, there is no mention as to why research disciplines as a possible determinant of OA publication are not included as a "minimum reporting standard". It is also unclear who the minimum reporting standard is for - Universities? Ranking producers? Researchers?

Overall, this is a pretty good paper with good data and methods, but I do not think that it is a good proposal of a "minimum reporting standard" and suggest this claim is dropped from the paper. The sentence "this study proposes a minimum reporting standard for institutional open access evaluation" should be removed from Section 1.1 and other places where this is mentioned in the paper.

3) The following points should be added to the Discussion or Limitations section:

- the effects and limitations of comparing only 'all green OA' (and not 'green only')

to 'gold OA', as 'all green' reflects both the extent to which gold OA papers are also included in repositories, as the extent to which otherwise closed papers are made available in repositories, and this may differ for different universities, depending on institutional policies.

- the limitation of longitudinal analysis done from a fixed point in time (i.e. looking back), as for earlier years, the effect of embargoes (in proportion of green OA) and moving walls (for bronze OA) will influence the results - i.e. results of the previous year are not just a reflection of proportion of OA in that year.

- the effect of disciplinary differences, regarding a) the extent to which the research outputs included (present in WoS/Scopus/MA, with DOI) can be expected to represent the total research output of an institution, depending on b) the extent to which observed levels of green OA are influenced by external policies such as those of NIH and Wellcome for biomedical research, resulting in green OA through PubMedCentral and EuropePMC, rather than through home repositories.

- the potential of extension of the approach to universities beyond the initial selection (which favours research-intensive universities), and expected differences in results.

4) It is stated in Section 1.1 that the authors intend to favour recall over precision. From looking at the details provided in the supplementary methods, it seems that what the authors mean here is that by using a variety of data sources they aimed to increase recall (but not at the expense of precision). If this is the case, please can the authors revise this sentence. If this is not a mistake, a justification as to why it makes more sense to focus on more data rather than better data needs to be included, especially when the next point on the list focuses on size-independent indicators, for which precision is more important than recall.

5) To give the table displaying the different OA categories more context, please can you add a caption similar to the information given in the Supplementary Methodology. This should include, which OA types are reported in this paper (Total OA, Gold, Hyrbid, Green, and Green in Homo Repo) and an explanation as to why you chose to include these categories in the main analysis and not the other types (Hybrid, Bronze, Gold DOAJ and Green only).

6) I think that shifts in trends that coincide with policy changes should be included in whatever body of evidence used to determine whether policies have been effective. That said, I think that the empirical setup simply doesn't allow to make any "effect" claims, and I would advise the authors to use the term sparingly and to speak perhaps of "possible effect" or something like that. For example, I am not convinced by the interpretation of Figure 4A: the trends could very well be a somewhat random fluctuation in the 2%-6% range, and I think it is a bit exaggerated to claim that this figure shows any kind of clear effects.

7) In the accompanying sensitivity analysis, some interesting patterns are detected (higher levels of total OA and gold OA, but lower levels of green OA for Latin America when Microsoft Academic (MA) is included, and lower levels of total OA and green OA for the UK when MA is included (cf. WoS and/or Scopus only). It would be good to provide some context/explanations for these observed effects, either in the white paper or in the main paper itself.

8) Consider adding a paragraph on the possibilities and limitations of re-using this methodology, particularly in regards to the closed data sources (with ramifications for what data can be shared). It may also be worth including more detail on the semi-automatic steps of the methodology so that people would be able to reproduce them if they have access to the data sources. In a broader context, there might be an opportunity to discuss the potential of making the framework more widely available to interested parties to create an open infrastructure that can assess open access levels at various aggregation levels.

Data/Stats to be included:

1) Include an explanation of how the confidence intervals used are derived (esp. since they are an important addition of this analysis compared to previous analyses of OA levels).

2) Either provide anonymized or de-anonymized data for all universities in the sample, not just for a subset.

3) Please can you make the examples of code in the supplementary methodology available to download.

Major comments to address:

1) As Figure 2 will be difficult to see and interpret in the web and PDF versions of the article, please can you replace this figure with Supplementary Figure 6. This will also be more in line with the examples given in the paper highlighting potential uses of the framework to analyse policy effects which are based on either comparing institutional results at country or region level for the full dataset (Figure 3), or on comparing specific institutions based on a common or differing policy context (Figure 4,5). Furthermore, including the ranking of top 100 institutions in terms of the proportion of different types of OA as a main result could invite undue focus on comparing individual institutions in a competitive way, which seems not in line with the aims of the work.

As suggested in the review report, the Figure showing the top 100 is now relocated to Figure 2 – figure supplement 2. Figure 2 is replaced with the figure showing groupings by regions for 2017, with figures for 2016 and 2018 included in Figure 2 – figure supplement 1.

2) The paper introduces several analytical categories that it states should be used as the 'minimum reporting standard'. However, why these categories should be the 'minimum reporting standard' requires a much stronger justification. For example, the authors have included data sources with restricted access (Scopus, WoS) along with the Microsoft Academic Graph, to increase recall. Additionally, there is no mention as to why research disciplines as a possible determinant of OA publication are not included as a "minimum reporting standard". It is also unclear who the minimum reporting standard is for - Universities? Ranking producers? Researchers?

Overall, this is a pretty good paper with good data and methods, but I do not think that it is a good proposal of a "minimum reporting standard" and suggest this claim is dropped from the paper. The sentence "this study proposes a minimum reporting standard for institutional open access evaluation" should be removed from Section 1.1 and other places where this is mentioned in the paper.

We have now removed the phrase “minimum reporting standard”. We have also updated the discussion to include ways to which our methodology can be adapted to perform analysis for other categories, such as disciplines, individuals, countries, etc, instead of institutions.

3) The following points should be added to the Discussion or Limitations section:

i) the effects and limitations of comparing only 'all green OA' (and not 'green only')to 'gold OA', as 'all green' reflects both the extent to which gold OA papers are also included in repositories, as the extent to which otherwise closed papers are made available in repositories, and this may differ for different universities, depending on institutional policies.

We have included an extra paragraph in the Limitations section to discuss this issue. The main message is that the general trends are not affected, but we recognise that at the individual university level this can be dependent on institutional policy and data quality.

ii) the limitation of longitudinal analysis done from a fixed point in time (i.e. looking back), as for earlier years, the effect of embargoes (in proportion of green OA) and moving walls (for bronze OA) will influence the results - i.e. results of the previous year are not just a reflection of proportion of OA in that year.

This is now included in the Limitations section.

iii) The effect of disciplinary differences, regarding a) the extent to which the research outputs included (present in WoS/Scopus/MA, with DOI) can be expected to represent the total research output of an institution, depending on b) the extent to which observed levels of green OA are influenced by external policies such as those of NIH and Wellcome for biomedical research, resulting in green OA through PubMedCentral and EuropePMC, rather than through home repositories.

Additional discussion on this matter is now included in the Limitations section.

iv) The potential of extension of the approach to universities beyond the initial selection (which favours research-intensive universities), and expected differences in results.

This is now mentioned in the Limitations section.

4) It is stated in Section 1.1 that the authors intend to favour recall over precision. From looking at the details provided in the supplementary methods, it seems that what the authors mean here is that by using a variety of data sources they aimed to increase recall (but not at the expense of precision). If this is the case, please can the authors revise this sentence. If this is not a mistake, a justification as to why it makes more sense to focus on more data rather than better data needs to be included, especially when the next point on the list focuses on size-independent indicators, for which precision is more important than recall.

This has been corrected to say “increase recall but not at the expense of precision”.

5) To give the table displaying the different OA categories more context, please can you add a caption similar to the information given in the Supplementary Methodology. This should include, which OA types are reported in this paper (Total OA, Gold, Hyrbid, Green, and Green in Homo Repo) and an explanation as to why you chose to include these categories in the main analysis and not the other types (Hybrid, Bronze, Gold DOAJ and Green only).

This is now included under the Methods section, below the table of OA definitions.

6) I think that shifts in trends that coincide with policy changes should be included in whatever body of evidence used to determine whether policies have been effective. That said, I think that the empirical setup simply doesn't allow to make any "effect" claims, and I would advise the authors to use the term sparingly and to speak perhaps of "possible effect" or something like that. For example, I am not convinced by the interpretation of Figure 4A: the trends could very well be a somewhat random fluctuation in the 2%-6% range, and I think it is a bit exaggerated to claim that this figure shows any kind of clear effects.

We have now rephrased the wording used in response to this comment.

7) In the accompanying sensitivity analysis, some interesting patterns are detected (higher levels of total OA and gold OA, but lower levels of green OA for Latin America when Microsoft Academic (MA) is included, and lower levels of total OA and green OA for the UK when MA is included (cf. WoS and/or Scopus only). It would be good to provide some context/explanations for these observed effects, either in the white paper or in the main paper itself.

Additional discussions have been added to the white paper on sensitivity analysis.

8) Consider adding a paragraph on the possibilities and limitations of re-using this methodology, particularly in regards to the closed data sources (with ramifications for what data can be shared). It may also be worth including more detail on the semi-automatic steps of the methodology so that people would be able to reproduce them if they have access to the data sources. In a broader context, there might be an opportunity to discuss the potential of making the framework more widely available to interested parties to create an open infrastructure that can assess open access levels at various aggregation levels.

A short description is now included at the end of the “Limitations” section to discuss limitations of reuse and further work in this direction.

Data/stats to be included:

1) Include an explanation of how the confidence intervals used are derived (esp. since they are an important addition of this analysis compared to previous analyses of OA levels).

Additional explanations are included in Supplementary file 1, under the section “Description of data workflow and selection criteria”.

2) Either provide anonymized or de-anonymized data for all universities in the sample, not just for a subset.

We would like to keep universities anonymised except for the top 100, which highlight important messages about universities not traditionally ranked the highest leading the way for OA. As such we have included two different sets of the processed data. The first includes all universities in anonymised form, and the second with only the top 100 universities included in de-anonymised form. This is made clearer at the end of the “Methods” section.

3) Please can you make the examples of code in the supplementary methodology available to download.

These are available via Zenodo, with the link provided in the article and Supplementary file 1.

Author details

1. Chun-Kai (Karl) Huang

   Chun-Kai (Karl) Huang is in the Centre for Culture and Technology, School of Media, Creative Arts and Social Inquiry, Curtin University, Perth, Australia

   Contribution

   Conceptualization, Resources, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   karl.huang@curtin.edu.au

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-9656-5932

2. Cameron Neylon

   Cameron Neylon is in the Centre for Culture and Technology, School of Media, Creative Arts and Social Inquiry, and Curtin Institute for Computation, Curtin University, Perth, Australia

   Contribution

   Conceptualization, Resources, Data curation, Software, Formal analysis, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0068-716X

3. Richard Hosking

   Richard Hosking is in the Curtin Institute for Computation, Curtin University, Perth, Australia

   Contribution

   Resources, Data curation, Software, Writing - review and editing

   Competing interests

   No competing interests declared

4. Lucy Montgomery

   Lucy Montgomery is in the Centre for Culture and Technology, School of Media, Creative Arts and Social Inquiry, and Curtin Institute for Computation, Curtin University, Perth, Australia

   Contribution

   Conceptualization, Funding acquisition, Validation, Project administration, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-6551-8140

5. Katie S Wilson

   Katie S Wilson is in the Centre for Culture and Technology, School of Media, Creative Arts and Social Inquiry, Curtin University, Perth, Australia

   Contribution

   Validation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8705-1027

6. Alkim Ozaygen

   Alkim Ozaygen is in the Centre for Culture and Technology, School of Media, Creative Arts and Social Inquiry, Curtin University, Perth, Australia

   Contribution

   Validation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-6813-8362

7. Chloe Brookes-Kenworthy

   Chloe Brookes-Kenworthy is in the Centre for Culture and Technology, School of Media, Creative Arts and Social Inquiry, Curtin University, Perth, Australia

   Contribution

   Validation, Writing - review and editing

   Competing interests

   No competing interests declared

• 6,293

  views

• 565

  downloads

• 52

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.