There is a pressing need to increase the rigor of research in the life and biomedical sciences. To address this issue, we propose that communities of 'rigor champions' be established to campaign for reforms of the research culture that has led to shortcomings in rigor. These communities of rigor champions would also assist in the development and adoption of a comprehensive educational platform that would teach the principles of rigorous science to researchers at all career stages.
The scientific enterprise relies on mentors teaching their students and trainees how to design and conduct studies that produce reliable scientific knowledge. A crucial part of this is teaching students and trainees how to minimize the risks that chance observations, subconscious biases, or other factors might lead to incorrect or inflated claims. However, as the demands on mentors increase, some of them unintentionally overlook this crucial aspect of scientific investigation, meaning that students and trainees are not taught how to distinguish between high- and low-quality evidence when working on their own studies and when reading about other studies (Ioannidis et al., 2014; Bosch and Casadevall, 2017; Landis et al., 2012).
Additional complications stem from the welcome rise in team-based science and a greater sophistication and range of experimental techniques (National Research Council, 2015), which may, in part, be driven by a feeling that only exciting and complete stories will appeal to journals and funders (Nosek et al., 2012; Casadevall et al., 2016). Increasingly, an individual scientist cannot be an expert in all the techniques used in a research project.
Taken together, these developments suggest that enhanced training in the fundamental principles of rigorous research common to most, if not all, experimental practices is needed to ensure that the outputs of scientific research remain reliable and robust. Such principles include strong reasoning and inference based on valid assertions, which requires the proper interpretation of uncertainty and a motivation to identify inconsistencies (Bosch and Casadevall, 2017; Casadevall and Fang, 2016; Munafò and Davey Smith, 2018; Wasserstein et al., 2019). For studies that test hypotheses, researchers should: clearly define interventions; identify and disclose possible confounding factors; transparently report project workflows, experimental plans, methods, data analyses, and any divergence from pre-planned procedures; and fully report their competing interests (see https://www.equator-network.org/ for reporting guidelines). The requirements for studies intended to generate hypotheses will be different but should be equally described (Dirnagl, 2019).
Before formulating solutions to these issues, we assessed current training practices at the graduate and postdoctoral levels by surveying all 41 institutions in the United States that held at least one training grant from the National Institute of Neurological Disorders and Stroke (NINDS) in May 2018. Only 5 of the 37 institutions that responded to the survey reported providing a course predominantly dedicated to principles of rigorous research, with others using a range of approaches – such as seminars, lectures within other coursework, workshops, and informal mentoring – to teach good research practices. However, few if any of the institutions covered the full range of principles that need to be learned and understood. Although the sample in our survey was small, the responses reinforced the common belief that formal training in rigorous research needs to be enhanced (Ioannidis et al., 2014; Munafò et al., 2017).
While numerous training materials related to rigorous research are available online, finding suitable materials and assembling them into a cohesive course is challenging. Having access to a free, organized suite of educational resources could greatly reduce the energy barrier for institutions and scientists to implement enhanced training at all levels, from undergraduate education to faculty professional development.
Towards this end NINDS convened a workshop attended by a range of stakeholders: basic, translational, and clinical neuroscientists; scholars of education and science communication; educational platform developers; and trainees. Although neuroscience served as a focal point, the four outcomes of the discussions apply widely across the biomedical sciences: i) there is a clear need for a platform that teaches the principles of rigorous research and covers the needs of scientists at all career stages; ii) effective educational interventions should lead to measurable behavioral change; iii) academic institutions need to play a proactive role in promoting rigorous research practices; iv) progress in this area will require cultural change at academic institutions, funders, and publishers (Casadevall et al., 2016; Munafò et al., 2017; Collins and Tabak, 2014; Begley et al., 2015; Casadevall and Fang, 2012).
To unleash the motivation for a cultural change evident in discussions between the authors and early-career researchers and others, and to provide momentum for change across different sectors, we propose the establishment of inter- and intra-institutional communities of 'rigor champions' who are committed to promoting rigor and transparency in research. We know there are many such individuals working at different levels of seniority in different types of organizations (such as universities, funders, publishers, and scientific societies), but they often feel isolated and under-resourced. To seed this effort and to help like-minded individuals in different organizations to find each other and join forces, NINDS has created a website for researchers, educators, trainees, organizational leaders and others who are passionate about the issues discussed here. This website includes currently available resources for making science more rigorous and transparently reporting results, as well as instructions for identifying yourself as a rigor champion.
More information about the different activities that these communities could undertake are given in Table 1. Researchers, educators and trainees are best placed to collaborate on new tools, share best practices, and promote rigorous research in their local scientific communities. Societies are in a position to advocate for widespread policy changes, while funders and journals have important gatekeeping roles (Collins and Tabak, 2014; McNutt, 2014; Cressey, 2015; PLOS Biology, 2018). The recently established UK Reproducibility Network (Munafò et al., 2020) and the PREMIER project (Dirnagl et al., 2018), both of which aim to improve scientific practices, may serve as models for these communities.
NINDS, for example, has proactively sought effective approaches to support greater transparency in reporting. An NINDS meeting with publishers led to changes in journal policies regarding transparency of reporting at various journals (Nature, 2013; Kelner, 2013). Recommendations for greater transparency at scientific meetings stemmed from an NINDS roundtable with conference organizing bodies (Silberberg et al., 2017) and are being piloted by the Federation of American Societies for Experimental Biology (FASEB). To recognize outstanding mentors, NINDS established the Landis Mentoring Award, and by providing greater stability to meritorious scientists though the NINDS R35 Program, it is anticipated that the pressures to rush studies to publication will be mitigated.
In particular we hope that leaders at academic institutions – such as department chairs, deans, and vice-presidents of research – will become involved because they are uniquely placed to shape the culture and social norms of institutions (Begley et al., 2015). For example, faculty evaluation criteria should be modified to place greater emphasis on data sharing, methods transparency, demonstrated rigor, collaboration, and mentoring, with less emphasis on the number of publications and journal impact factors (Casadevall and Fang, 2012; Moher et al., 2018; Bertuzzi and Jamaleddine, 2016; Lundwall, 2019; Strech et al., 2020; Casci and Adams, 2020; see also https://sfdora.org/read). When publications are being evaluated, rigorously obtained null results should be valued as highly as positive findings. Institutional leaders are also uniquely placed to ensure that scientific rigor is properly taught to trainees and incorporated into day-to-day lab work (Casadevall et al., 2016; Begley et al., 2015; Bosch, 2018; Button et al., 2020). Moreover, evaluations of trainees should emphasize experimental and analytic skills rather than where papers are published.
The establishment of communities of rigor champions will set the stage for the creation of an educational platform designed by the scientific community to communicate the principles of rigorous research. Given the rapid evolution of technologies and learning practices, it is difficult to predict what resource formats will be most effective in the future, so the platform will need to be open and freely available, easily discoverable, engaging, modular, adaptable, and upgradable. It will also need to be available during coursework and beyond so that scientists can use it to answer questions when they are doing research or as part of life-long learning (Figure 1). This means that the platform will have to embody a number of principles of effective teaching and mentoring (see Table 2).
We envision the platform being developed via a hub-and-spoke approach as discussed at a recent National Advisory Neurological Disorders and Stroke Council meeting. A centralized mechanism (the 'hub') will provide financial and infrastructural support and guidance (possibly via a steering committee) and facilitate sharing and coordination between groups, while rigor champions will come together to design specific modules (spokes) for the platform by using existing resources or designing new ones from scratch as needed. We envision worldwide teams of experts collaborating on building and testing the resource. Rigor champions with experience in defining clear learning objectives, building curricula, and evaluating success, for example, will collaborate with content experts to design topics needed in the resource. Importantly, potential users will be involved from the beginning of the development stage, and onwards through the design and implementation stages, to provide feedback about effectiveness and usability.
Given the importance of being able to measure the effectiveness (or otherwise) of the platform (Table 2), individual components should be released publicly as they are completed to allow educators and users to iteratively test and improve the resource as it unfolds. As with science itself, the developers will need to experiment with content and delivery. If the resource does not improve the comprehension and research practice of individuals, or add value to the research community, rigorous approaches should be applied to improve it.
Once a functioning and effective resource has been built, it will be essential to promote its use and adoption. One approach would be to host 'train-the-trainer' programs (Spencer et al., 2018; Pfund et al., 2006): those involved in building the resource share it with small groups of mentors, who are then better equipped to use the resource with their own mentees and to encourage their colleagues to use it. This form of dissemination also creates buy-in from mentors who need to model the behaviors they are teaching. Rigor champions, meanwhile, can encourage their institutions and colleagues to adopt and use the resource.
Setting up and supporting communities of rigor champions and developing educational resources on rigorous research will be complex and likely require multiple sources of support. However, with the participation of all sectors of the scientific enterprise, the actions proposed herein should, within a decade, lead to improvements in the culture of science as well as improvements in the design, conduct, analysis, and reporting of biomedical research. The result will be a healthier and more effective scientific community.
The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the US Government.
Self-regulated learning: beliefs, techniques, and illusionsAnnual Review of Psychology 64:417–444.https://doi.org/10.1146/annurev-psych-113011-143823
Minds on fire: open education, the long tail, and learning 2.0EDUCAUSE Review 43:16–32.
Grassroots training for reproducible science: a consortium-based approach to the empirical dissertationPsychology Learning & Teaching 19:77–90.https://doi.org/10.1177/1475725719857659
Reforming science: methodological and cultural reformsInfection and Immunity 80:891–896.https://doi.org/10.1128/IAI.06183-11
Science writing: Too good to be true?New York Times. Accessed February 29, 2020.
Handbook of Experimental PharmacologyResolving the ttension between exploration and confirmation in preclinical biomedical research, Handbook of Experimental Pharmacology, Berlin, Heidelberg, Springer, 10.1007/164_2019_278.
Confusion can be beneficial for learningLearning and Instruction 29:153–170.https://doi.org/10.1016/j.learninstruc.2012.05.003
Changing institutional incentives to foster sound scientific practices: one departmentInfant Behavior and Development 55:69–76.https://doi.org/10.1016/j.infbeh.2019.03.006
Inquiry-based science instruction-what is it and does it matter? results from a research synthesis years 1984 to 2002Journal of Research in Science Teaching 47:474–496.https://doi.org/10.1002/tea.20347
Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishabilityPerspectives on Psychological Science 7:615–631.https://doi.org/10.1177/1745691612459058
Building a sustainable national infrastructure to expand research mentor trainingCBE—Life Sciences Education 17:ar48.https://doi.org/10.1187/cbe.18-03-0034
Personalization of instruction: design dimensions and implications for cognitionThe Journal of Experimental Education 86:50–68.https://doi.org/10.1080/00220973.2017.1380590
Boring but important: a self-transcendent purpose for learning fosters academic self-regulationJournal of Personality and Social Psychology 107:559–580.https://doi.org/10.1037/a0037637
Funded by the National Institute of Neurological Disorders and Stroke (NINDS).
- Received: February 10, 2020
- Accepted: February 21, 2020
- Version of Record published: March 4, 2020 (version 1)
This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.