Science Forum: RIPOSTE: a framework for improving the design and analysis of laboratory-based research

Laboratory-based studies play a central role in preclinical biomedical research, encompassing a diverse range of techniques and spanning a broad range of fields across the biomedical sciences. For example, investigations of the biological pathways underpinning drug response or microbial pathogenesis, the assessment of safety and efficacy of interventions, and the discovery of biomarkers all rely on laboratory-based methods for at least some stages of observation, measurement and/or processing. Despite this key role, approaches to the design, analysis and reporting of laboratory studies can be highly varied. Moreover, the frequently dynamic nature of laboratory based research can mean that studies are often complex and may consist of various exploratory components, which may not be fully documented when results are published. This can lead to a lack of transparency about the research methodology, and may prevent any results and findings from being successfully reproduced.

Lack of reproducibility (or ‘irreproducibility’) is an acknowledged problem within biomedicine that has recently been gaining increased attention (Begley and Ellis, 2012; Begley, 2013; The Economist, 2013; Macleod et al., 2014). Attempts to independently confirm or follow-up on spurious research findings waste time, money (which may have public or charitable origins) and resources, and also raises ethical concerns. Initiatives aiming to address irreproducibility in the biomedical sciences are therefore underway (Errington et al., 2014; Morrison, 2014). Initial efforts have largely focussed on improving reporting standards from research publications. For example, both Science and Nature have recently introduced new reporting guidelines that aim to improve the transparency of research disseminated in their journals (Nature, 2013; McNutt, 2014). Attempts to harmonise and improve reporting standards across particular types of study have also been made. The Minimum Information About a Microarray Experiment (MIAME) (Brazma, 2009) initiative targets experimental protocols in microarray experiments and other ‘omics’ studies, while the REMARK guidelines (Altman et al., 2012) focus on the appropriate and transparent use of statistical methods in tumour marker prognostic studies. A recent report from the Institute of Medicine in the US also focuses on ‘omics’ studies (Institute of Medicine, 2012). Several other relevant guidelines for reporting health related research can be found through the EQUATOR network (equator-network.org). A common theme in these guidelines is the appropriate and transparent reporting of statistical methods.

Whilst the above initiatives aim to improve transparency in published laboratory based research, a focus only on the reporting of studies does not address other key factors that may also contribute to irreproducibility. For instance, a recent retraction of a MIAME compliant study (Sebastiani et al., 2011) demonstrates that targeting reporting standards alone cannot prevent irreproducibility. A large number of other retractions have also been highlighted (www.retractionwatch.com), increasing the focus on what contributes to the problem and how to tackle it (Irizarry et al., 2005; Baggerly and Coombes, 2009; Prinz et al., 2011; Sebastiani et al., 2011; Begley and Ellis, 2012; Lambert and Black, 2012; Parker and Leek, 2012; Freedman and Inglese, 2014).

Several factors may lead to irreproducibility in laboratory studies. As highlighted above, poor reporting can limit the ability to accurately reproduce results. Although general methodology and procedures are usually reported, key details needed to guarantee that an entire analysis pipeline can be reproduced are often missing, such as information about a study's methods and/or analysis. This may include information about the modality of data handling and manipulation, version of software and/or libraries used, and implementation of the statistical methods.

Issues relating to the generation of data, including study design and methods to minimise the introduction of bias, may also contribute to irreproducibility. Any bias introduced into a study often cannot be removed and may impact on the results in ways that may be difficult to quantify (Bogardus et al., 1999). These issues may stem from practices within the laboratory itself; for example, unwanted variation posed by batch effects or other confounding variables can systematically and irreversibly distort the measurements taken within a study unless appropriately accounted for at the design stage. Technical issues relating to the analysis may also lead to errors; for instance, incorrectly distinguishing between repeated and independent measurements can increase the likelihood of obtaining false positive or false negative results.

A lack of formal guidance on the process of laboratory study design may also give rise to irreproducibility. Although in some respects laboratory-based research is highly regulated, such regulation largely relates to materials, processes and ethics rather than focussing on aspects of study design or improving methodological rigor. For example, procedures and protocols must be approved by the Control of Substances Hazardous to Health Regulations (COSHH), while pre-clinical pharmaceutical and medical device research is governed very strictly by Good Laboratory Practice (GLP) regulations. Clinical studies using human samples are subject to ethical review, research governance and the International Committee on Harmonisation of Good Clinical Practice (ICH GCP) and may also be subject to the Human Tissue Act (2004). Certain laboratory work is also conducted under accreditation from UKAS and CPA or to ISO/BSI standards. In contrast to other methods of experimental research such as clinical trials, however, none of these regulations specifically addresses study design and there is often no formal requirement to produce a study-specific protocol or analysis plan in advance of data collection.

The existing culture where novel research is rewarded over and above attempts to replicate findings may also contribute to irreproducibility. Those who attempt to replicate results currently face expenses in terms of time and resources, and can find it hard to publish their findings whether they confirm or not (Ioannidis, 2005, 2006). This may contribute to the well-known phenomenon of ‘publication bias’, where positive but potentially one-off or chance novel results disproportionately enter the literature at the expense of negative findings. Failing to adequately document negative findings can also lead to publication bias, and may lead to others unnecessarily repeating the work in future. Bias towards publication of statistically significant results has been shown to be substantively greater for observational and laboratory-based studies than for randomised clinical trials (Easterbrook et al., 1991).

It is now generally accepted that poor study design is a major problem in laboratory based research (Collins and Tabak, 2014). While most scientists will have received training in experimental design in an abstract form, it may be difficult to put it into practice, especially when some experiments can be conducted by a single researcher. Currently this poor design is being picked up at the reporting stage as was the case in clinical research some 30 years ago. For instance, in the 1980s weaknesses in the reporting of clinical studies led to a number of initiatives to improve statistical awareness and understanding. As a result, reporting guidelines were developed (Moher et al., 2001; Schulz et al., 2010) to promote the reporting of key methodological components and results that enable study bias to be assessed and to support evidence synthesis. This recognition of the key role of statistical principles in study design and analysis has resulted in integrated and critical involvement by statisticians in all aspects of clinical trials. Ethical concerns have led regulatory bodies to impose strict standards concerning all aspects of the design, analysis and reporting of clinical trials, which ensure that they are properly planned and implemented.

A clinical trial can be considered the equivalent of a single experiment to test a specific hypothesis. These single experiment trials require their own funding and generally result in at least two publications; the protocol and the results on completion. By contrast in basic science it is very unusual for the results of a single experiment to be published in isolation. It is more common to find a series of experiments presented, linked with inductive and deductive reasoning. This tendency to present a broad range of linked experiments and results in a single publication has been a barrier to the development of appropriate reporting guidelines. Some journals are now actively promoting the submission of short follow-on reports (the Research Advance in eLife) or breakthrough articles where simple but important questions are addressed (Corey et al., 2014). Although the methods employed in laboratory studies are diverse and experiments can be completed within very short time frames, much can be learnt from the standards upheld in trials. Trials are designed and managed by regular consultation within full, multidisciplinary teams. Such teams can involve health-economists, computer scientists and statisticians, as well as clinicians, scientists and/or qualitative researchers. Input from the full interdisciplinary team at all stages of a study helps to ensure that trials are optimally designed, making efficient use of resources and avoiding potential difficulties at the analysis stage. Trials are long term projects where protocols are first established, participants are recruited and then endpoints are measured. By contrast, in laboratory research many experiments may be conducted in parallel at many levels within a research group, and the rigid clinical trial design structure would not allow the flexibility required for new research to emerge. We assert that greater consideration of the principles of good experimental design coupled with early and regular discussion amongst all the members of the research team will help improve the design, analysis and reporting of laboratory based studies. This, in turn, should lead to higher quality data and reproducible research. Such improvements will require a gear change from all involved in the field especially from research funders.

To support the implementation of such an integrated approach we have developed the RIPOSTE framework, which draws together key elements of laboratory study design and analysis that may contribute to reproducibility. The framework is accompanied by three hypothetical case-studies to demonstrate the discussion that may follow the consideration of each prompt point. The overall aim of the framework is to support discussion within a multidisciplinary research team (including the statistician), to ensure that potential sources of bias and/or variability have been considered and, where possible, eliminated at the design stage. We are aware that scientific advances can be made through a mix of inductive and deductive reasoning. This framework is focussed to support more discussion in the deductive stages when hypotheses and specific questions are proposed.

The framework was developed in two stages. The NIHR Statistics Group held a laboratory research studies day, during which the initial project was conceived and major elements for the framework identified. A prompt-list using items commonly encountered in reporting guidelines was then constructed and revised to be relevant for laboratory experiments at the design stage. For the second stage we invited 12 statisticians and 12 laboratory scientists to a one-day workshop where the framework was piloted as a means to facilitate discussion on aspects of study design and analysis. The framework was trialled in small groups: two scientists and two statisticians worked in each group and the framework was tested using examples supplied by the scientists. At the end of the workshop feedback was obtained and suggested modifications to the framework were collated. Modifications were made and further feedback was obtained from the RIPOSTE consortium via an online survey. In the survey delegates were asked to score the inclusion of items on a 0–10 scale (high score to retain item). Items receiving a median score less than 8 were removed, and any which had been scored 0 by at least one respondent, irrespective of the median score, were revised if necessary in line with the respondents' comments. Suggestions on the structure and presentation were also incorporated.

We present here a framework to support early discussions within a multidisciplinary research team, which should consist of both scientists and statisticians. The framework contains a comprehensive list of the details that facilitate reproduction of research and is intended to promote discussion about key aspects of the design, conduct and analysis of a planned laboratory study. The framework offers a series of prompts that raise pertinent questions to facilitate shared understanding of the research and the environment in which it is being undertaken.

The catch-all term ‘laboratory studies’ covers a wide range of study types (Box 4), and some aspects of the framework will not always be applicable in all studies. Similarly, some aspects of the framework will not always be relevant for discussion with statisticians, but nevertheless concern issues that still require careful consideration within the research team. We see this framework as a useful toolbox in the hands of the scientist, which takes and builds upon many points raised in recent journal and topic specific publication guidelines. Our workshop confirmed that it can take a long time for a statistician to fully understand the basic designs of a series of experiments when first presented. This is often due to lack of familiarity with the field of application. We felt that using some carefully selected case-studies to demonstrate how the prompts in the framework can be used would help both statisticians and scientists in its implementation. We have, therefore, included three hypothetical case studies as examples which have been selected to cover a broad spectrum of biomedical laboratory settings. The first (Box 1) is a study of combinations of components of automated medical dosing equipment, where the motivation is to look for equivalent performance. The second study (Box 2) examines macrophage activity when cells are infected with bacteria and treated with a drug. This experiment illustrates replicate measurement, treatment and infection control contrasts and plate or batch effects. The third (Box 3) is a gene expression study in patients with hypertension (cases) and without hypertension (controls), where the aim is to identify genes that are differentially expressed. This example allows us to illustrate multiple hypothesis testing and a variety of sources of batch effects from tissue processing through to RNA analysis. The framework is presented in Table 1; this sets out the major prompts for topics to be considered and gives some brief notes for each. The following sections follow the headings in Table 1 and provide a more detailed breakdown and discussion of items from the framework, clarifying our recommendations.

Box 1

Example: Elastomer pump study

A study is planned to assess a new type of disposable elastomer pump and catheter for use in delivering anaesthetic directly to wounds following major surgery. The study aims to assess whether the new pump and catheter—or combinations of the new pump and catheter with an existing pump and catheter—achieve an acceptable flow rate over time (i.e., where an ‘acceptable’ flow rate is defined as within 15% of the set rate). The researchers also wish to assess whether the performance of the equipment declines after with reuse.

Methods and materials: The experimental set-up is presented in Box 1 Figure 1. In order to mimic clinical practice, the flow rate will be set to 4 ml/hr, and each experiment will run over a period of 48 hr. Automated weight measurements of the pump will be taken every 2 hr via a laptop, and concurrent measurements of the room temperature will also be made as temperature may impact upon the flow. Each type of pump (P = existing pump; p = new pump) is to be tested with each type of catheter (C = existing catheter; c = new catheter). Four experiments will be run simultaneously over four units, with each experiment repeated three times before changing equipment (i.e., each experiment will be run in triplicate). Due to limited study resources, only four pumps and four catheters of each type are available for use.

Box 1 Figure 1

Download asset Open asset

Design: Box 1 Table 1 illustrates two possible arrangements of the pump/catheter combinations over the 4 units. Design A runs a particular combination over all 4 units at the same time before switching to the next combination, while Design B tests the four different combinations of pumps and catheters simultaneously before alternating the order of the combinations over the units after each set of triplicate experiments.

Box 1 Table 1

Arrangement	Duration	Bench 1	Bench 2	Bench 3	Bench 4
Suboptimal design with potential for confounding
1	48 hrs × 3	P C	P C	P C	P C
2	48 hrs × 3	P c	P c	P c	P c
3	48 hrs × 3	p C	p C	p C	p C
4	48 hrs × 3	p c	p c	p c	p c
Optimal, balanced design
1	48 hrs × 3	P C	P c	p C	p c
2	48 hrs × 3	P c	P C	p c	p C
3	48 hrs × 3	p c	p C	P c	P C
4	48 hrs × 3	p C	p c	P C	P c

1. There are four benches of equipment being tested, each with one of each type of pump and one of each type of catheter (P = existing pump; p = new pump, C = existing catheter; c = new catheter).

https://doi.org/10.7554/eLife.05519.003

Box 2

Example: Macrophage Study

A series of experiments are planned to characterise macrophage activity (cytokine production and apoptosis) when cells which are infected with bacteria are treated with a drug. Blood will be taken from multiple volunteer donors to obtain peripheral blood mononuclear cells from which differentiated macrophages are produced. The macrophages will be infected with a specific dose of bacteria and treated with a drug. The cytokine production and apoptosis will be measured at intervals over 24 hr. The panel of 10 cytokines will be measured by a multiplex bead system. Each donor will be processed with internal controls so the four combinations of infection status (infected/mock infected) and treatment (drug treatment/control) will be measured.

Research Question: Does treatment with a specific drug to cells infected with bacteria affect macrophage immune function measured by cytokine production and apoptosis?

The basic experimental design will include:

1. The assessment of baseline cytokine production in infected and mock infected macrophages.

2. The time course of cytokine production following the infection point, captured by measuring levels every 2 hr.

3. The matched design ensures that cells from each donor can be studied for response to both infection and treatment. Exactly half of the infected and half of the mock infected macrophages are treated with the drug and this is balanced over all donors.

4. The four combinations of treatment and infection will be processed in parallel on the samples.

Box 2 Figure 2 illustrates two possible ways that macrophages from just two donors might be arranged, for incubation in a single experiment on two eight well sections of a plate. Each subject has macrophages grown in eight wells, four of these will be infected with the same bacteria, and four will be mock infection controls. Two of the infected and two of the mock infected will be treated with the drug. Hence for each donor the measurement of variables under each condition is done twice (i.e., in duplicate). Arrangements A and B show a total of four possible plate arrangements. Some arrangements have conditions or donors clustered or organised into rows or columns. The two plates for ‘A’ make it easy for the infectious agent to be dosed out in one block, whereas ‘B’ has all the wells to be treated with the drug in a single column. In three of the plates, wells from different donors are never direct neighbours; however, the infection is done in blocks or pairs of neighbours. The diagram shown here shows only a part of a larger plate. Plate sizes of 24 or 96 well plates are available for use here; therefore multiple plates need to be used. The bead system for measuring cytokine levels uses assays which are automated, however, the assessment of apoptosis involves visual inspection and counting of cells. The colour of the medium indicates exactly which samples are infected and which are treated, which means the measurements cannot be taken ‘blind’ to the treatment.

Box 2 Figure 1

Download asset Open asset

Box 2 Figure 2

Download asset Open asset

https://doi.org/10.7554/eLife.05519.006

Once a well-designed laboratory study has been completed, it will need to be reported to a high standard to enable future reproduction of the results. There are a wide range of publications available which give detailed instructions on how best to report the results of different types of studies. Many journals now require authors to refer to specific guidelines for certain research designs.

It is not our intention to give an exhaustive list here, as new or updated guidelines are released with regularity. However, authors should search for relevant guidelines when preparing for publication; the Enhancing QUAlity and Transparency Of health Research (EQUATOR) network website is an excellent place to start, featuring a searchable library which aims to include all reporting guidelines published since 1996 (www.equator-network.org).

Example(s): If the elastomer pump study in Box 1 was treated as an equivalence study then many of the recommendations in the CONSORT extension for equivalence clinical trials would be relevant.

The RIPOSTE framework aims to reduce irreproducibility in laboratory based research by encouraging early discussion of study design and analysis within a multidisciplinary team including statisticians. We seek to steer discussions within research teams towards addressing key aspects of experimental design and analysis at the earliest stages of a study, and believe that this increased focus on planning will lead to more rigorous research and ultimately reduced wastage in preclinical research.

Lack of reproducibility is not the sole reason for wastage within laboratory studies. In January 2014 the Lancet printed a special issue focussing on how to increase value and reduce waste in medical research (Macleod et al., 2014). It has been claimed that much of the waste is due to incomplete and unusable results (Chalmers and Glasziou, 2009). The problem of poor research practice and documentation is widespread and entrenched in the scientific culture (Collins and Tabak, 2014). Currently scientific rewards are disproportionately high for being the first to publish, and this pressure has played a major part in generating the problems with reproducibility that are now being highlighted (Begley, 2013; Ioannidis, 2014).

A number of recent initiatives have drawn further attention to these critical issues and proposed strategies to address and change the scientific culture. The common themes emerging from these initiatives are to improve training on experimental design and analysis, to involve experienced statisticians at all stages of design and analysis, to raise awareness at grant review stage of aspects of design such as randomisation and blinding, and to reward good quality, well designed research. Ioannidis et al. (2014) make three broad recommendations to improve study design, conduct and analysis. The first of these is to make study protocols publicly available including the raw data and analytical algorithms. The second promotes raising the profile of defensible research proposals within well-trained research teams. The third is to reward reproducible practices through funding and academic recognition. In the same special issue of the Lancet, Glasziou et al. (2014) recommend that funders should support and encourage their research institutions to share research protocols and study materials and ultimately to promote high quality complete reporting. At publication the emphasis must move towards reporting results in which they have confidence (these will often be negative) in detail, rather than selectively reporting the details of the positive results which, if spurious, will serve to misguide the research community. Several recent incentives to promote direct replication research are beginning to make an impact with the publication of registered reports (Nosek and Lakens, 2014). In this framework journals agree to accept a future publication based on acceptance of pre-registered proposals and prior to any data generation.

In addition to the above initiatives, there has also been a recent push for greater publication of raw data. The PLoS journals, for example, implemented a new data policy earlier this year stipulating that authors must, wherever legally and ethically possible, share all data, metadata and methods that underlie any research findings offered for publication (Bloom et al., 2014). Data must be deposited in a public repository, uploaded online in supporting files to accompany a manuscript, or made available upon request; any failure to ensure that sufficient provisions to share have been made can be grounds for rejection. In the US, the NIH also intends to promote greater access to raw data, requesting that funding applications include a Data Discovery Index to enable any unpublished data to be more easily located, accessed and referenced by other researchers in any future work (Collins and Tabak, 2014). This NIH initiative has been supported by recent calls to prospectively register laboratory studies (Hooft and Bossuyt, 2011; Altman, 2014). Registering studies upfront would necessitate that any deviations from protocols are both documented and justified, and would ensure that protocols are well thought out at an early stage (i.e., prior to registration). It would likely also significantly improve the transparency of research, as was seen following the implementation of a similar initiative in 2005, which sought to introduce a requirement to prospectively register certain types of trials (Hooft and Bossuyt, 2011). Registering studies and reducing bias against publication of negative results will also help to ensure that replication studies with negative findings receive the appropriate attention amongst the scientific community.

These initiatives suggest the need for a major culture change within preclinical research; tackling these issues will require effort on multiple levels. The shift to making statisticians an integral part of the research team rather than to be consulted in isolation will be challenging. Statisticians' knowledge and experience of experimental data and the laboratory environment are highly variable. Scientists may be reluctant to work with statisticians in this way due to variable experiences in the past. The RIPOSTE framework has been designed to support this shift and help scientists and statisticians alike form a deeper understanding of the issues surrounding the reproducibility of laboratory research. This should ensure that the considerations relevant to a particular study can be addressed efficiently with greater confidence on both sides. To allow for a greater involvement of statisticians in the study design process, additional funds will be needed and this will require commitment from funding bodies. We recommend that statisticians be considered an integral part of the research team wherever possible, and that they should be involved at the planning stages of studies. We encourage use of our framework for all laboratory research studies and not just those seeking funding. In conjunction with other initiatives (Collins and Tabak, 2014) the RIPOSTE framework can be a useful tool in combating irreproducibility of preclinical study results, offering a powerful riposte to the criticisms regarding wastage in laboratory research.

1. 1. Altman DG

   (2014) The time has come to register diagnostic and prognostic research

   Clinical Chemistry 60:580–582.

   https://doi.org/10.1373/clinchem.2013.220335

   • Google Scholar
2. 1. Altman DG
   2. McShane LM
   3. Sauerbrei W
   4. Taube SE

   (2012) Reporting recommendations for tumor marker prognostic studies (REMARK): explanation and elaboration

   BMC Medicine 10:51.

   https://doi.org/10.1186/1741-7015-10-51

   • Google Scholar
3. 1. Baggerly KA
   2. Coombes KR

   (2009) Deriving chemosensitivity from cell lines: forensic bioinformatics and reproducible research in high-throughput biology

   Annals of Applied Statistics 3:1309–1334.

   https://doi.org/10.1214/09-AOAS291

   • Google Scholar
4. 1. Barnhart HX
   2. Kosinski AS
   3. Haber MJ

   (2007) Assessing individual agreement

   Journal of Biopharmaceutical Statistics 17:697–719.

   https://doi.org/10.1080/10543400701329489

   • Google Scholar
5. 1. Begley CG

   (2013) Six red flags for suspect work

   Nature 497:433–434.

   https://doi.org/10.1038/497433a

   • Google Scholar
6. 1. Begley CG
   2. Ellis LM

   (2012) Raise standards for preclinical cancer research

   Nature 483:531–533.

   https://doi.org/10.1038/483531a

   • Google Scholar
7. 1. Bloom T
   2. Ganley E
   3. Winker M

   (2014) Data access for the open access literature: PLOS's data policy

   PLOS Biology 12:e1001797.

   https://doi.org/10.1371/journal.pbio.1001797

   • Google Scholar
8. 1. Bogardus ST Jr
   2. Concato J
   3. Feinstein AR

   (1999) Clinical epidemiological quality in molecular genetic research - the need for methodological standards

   Journal of the American Medical Association 281:1919–1926.

   https://doi.org/10.1001/jama.281.20.1919

   • Google Scholar
9. 1. Brazma A

   (2009) Minimum information about a microarray experiment (MIAME) - successes, failures, challenges

   The Scientific World Journal 9:420–423.

   https://doi.org/10.1100/tsw.2009.57

   • Google Scholar
10. 1. Chalmers I
    2. Glasziou P

    (2009) Avoidable waste in the production and reporting of research evidence

    Lancet 374:86–89.

    https://doi.org/10.1016/S0140-6736(09)60329-9

    • Google Scholar
11. 1. Collins FS
    2. Tabak LA

    (2014) NIH plans to enhance reproducibility

    Nature 505:612–613.

    https://doi.org/10.1038/505612a

    • Google Scholar
12. 1. Corey DR
    2. Wise JA
    3. Fox KR
    4. Stoddard BL

    (2014) Breakthrough articles: putting science first

    Nucleic Acids Research 42:11273–11274.

    https://doi.org/10.1093/nar/gku853

    • Google Scholar
13. 1. Easterbrook PJ
    2. Berlin JA
    3. Gopalan R
    4. Matthews DR

    (1991) Publication bias in clinical research

    Lancet 337:867–872.

    https://doi.org/10.1016/0140-6736(91)90201-Y

    • Google Scholar
14. 1. Errington TM
    2. Iorns E
    3. Gunn W
    4. Tan FE
    5. Lomax J
    6. Nosek BA

    (2014) An open investigation of the reproducibility of cancer biology research

    eLife 3:e04333.

    https://doi.org/10.7554/eLife.04333

    • Google Scholar
15. 1. Freedman LP
    2. Inglese J

    (2014) The increasing urgency for standards in basic biologic research

    Cancer Research 74:4024–4029.

    https://doi.org/10.1158/0008-5472.CAN-14-0925

    • Google Scholar
16. 1. Glasziou P
    2. Altman DG
    3. Bossuyt P
    4. Boutron I
    5. Clarke M
    6. Julious S
    7. Michie S
    8. Moher D
    9. Wager E

    (2014) Reducing waste from incomplete or unusable reports of biomedical research

    Lancet 383:267–276.

    https://doi.org/10.1016/S0140-6736(13)62228-X

    • Google Scholar
17. 1. Hooft L
    2. Bossuyt PM

    (2011) Prospective Registration of marker evaluation studies: time to act

    Clinical Chemistry 57:1684–1686.

    https://doi.org/10.1373/clinchem.2011.176230

    • Google Scholar
18. 1. Hurlbert SH

    (1984) Pseudoreplication and the design of ecological field experiments

    Ecological Monographs 54:187–211.

    https://doi.org/10.2307/1942661

    • Google Scholar
19. Book

    1. Institute of Medicine

    (2012)

    Evolution of Translational Omics: Lessons Learned and the Path Forward

    Washington, DC: National Academies Press.

    • Google Scholar
20. 1. Ioannidis JP
    2. Greenland S
    3. Hlatky MA
    4. Khoury MJ
    5. Macleod MR
    6. Moher D
    7. Schulz KF
    8. Tibshirani R

    (2014) Increasing value and reducing waste in research design, conduct, and analysis

    Lancet 383:166–175.

    https://doi.org/10.1016/S0140-6736(13)62227-8

    • Google Scholar
21. 1. Ioannidis JP

    (2005)

    Why most published research findings are false

    PLOS Medicine 2:696–701.

    • Google Scholar
22. 1. Ioannidis JP

    (2006) Journals should publish all ‘null’ results and should sparingly publish ‘positive’ results

    Cancer Epidemiology Biomarkers & Prevention 15:186.

    https://doi.org/10.1158/1055-9965.EPI-05-0921

    • Google Scholar
23. 1. Ioannidis JP

    (2014) How to make more published research true

    PLOS Medicine 11:e1001747.

    https://doi.org/10.1371/journal.pmed.1001747

    • Google Scholar
24. 1. Irizarry RA
    2. Warren D
    3. Spencer F
    4. Kim IF
    5. Biswal S
    6. Frank BC
    7. Gabrielson E
    8. Garcia JG
    9. Geoghegan J
    10. Germino G
    11. Griffin C
    12. Hilmer SC
    13. Hoffman E
    14. Jedlicka AE
    15. Kawasaki E
    16. Martínez-Murillo F
    17. Morsberger L
    18. Lee H
    19. Petersen D
    20. Quackenbush J
    21. Scott A
    22. Wilson M
    23. Yang Y
    24. Ye SQ
    25. Yu W

    (2005) Multiple-laboratory comparison of microarray platforms

    Nature Methods 2:477.

    https://doi.org/10.1038/nmeth0605-477b

    • Google Scholar
25. 1. Lambert CG
    2. Black LJ

    (2012) Learning from our GWAS mistakes: from experimental design to scientific method

    Biostatistics 13:195–203.

    https://doi.org/10.1093/biostatistics/kxr055

    • Google Scholar
26. 1. Macleod MR
    2. Michie S
    3. Roberts I
    4. Dirnagl U
    5. Chalmers I
    6. Ioannidis JP
    7. Al-Shahi Salman R
    8. Chan AW
    9. Glasziou P

    (2014) Biomedical research: increasing value, reducing waste

    Lancet 383:101–104.

    https://doi.org/10.1016/S0140-6736(13)62329-6

    • Google Scholar
27. 1. Maecker HT
    2. McCoy JP Jr
    3. Immunophenotyping FH

    (2011) A model for harmonizing flow cytometry in clinical trials

    Nature Immunology 12:271.

    https://doi.org/10.1038/ni0311-271d

    • Google Scholar
28. 1. McNutt M

    (2014) Reproducibility

    Science 343:231.

    https://doi.org/10.1126/science.1250475

    • Google Scholar
29. 1. Moher D
    2. Schulz KF
    3. Altman DG

    (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials

    Lancet 357:1191–1194.

    https://doi.org/10.1016/S0140-6736(00)04337-3

    • Google Scholar
30. 1. Morrison SJ

    (2014) Time to do something about reproducibility

    eLife 3:e03981.

    https://doi.org/10.7554/eLife.03981

    • Google Scholar
31. 1. Nature

    (2013) Reducing our irreproducibility

    Nature 496:398.

    https://doi.org/10.1038/496398a

    • Google Scholar
32. 1. Nosek BA
    2. Lakens D

    (2014) Registered reports

    Social Psychology 45:137–141.

    https://doi.org/10.1027/1864-9335/a000192

    • Google Scholar
33. Book

    1. Piantadosi S

    (2005)

    Clinical Trials: A Methodological Perspective

    Wiley.

    • Google Scholar
34. 1. Parker HS
    2. Leek JT

    (2012)

    The practical effect of batch on genomic prediction

    Statistical Applications in Genetics and Molecular Biology, 11, Article 10.

    • Google Scholar
35. 1. Prinz F
    2. Schlange T
    3. Asadullah K

    (2011) Believe it or not: how much can we rely on published data on potential drug targets?

    Nature Reviews Drug Discovery 10:712.

    https://doi.org/10.1038/nrd3439-c1

    • Google Scholar
36. 1. Schulz KF
    2. Altman DG
    3. Moher D
    4. CONSORT Group

    (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials

    Journal of Clinical Epidemiology 63:834–840.

    https://doi.org/10.1016/j.jclinepi.2010.02.005

    • Google Scholar
37. 1. Sebastiani P
    2. Solovieff N
    3. Puca A
    4. Hartley SW
    5. Melista E
    6. Andersen S
    7. Dworkis DA
    8. Wilk JB
    9. Myers RH
    10. Steinberg MH
    11. Montano M
    12. Baldwin CT
    13. Perls TT

    (2011) Retraction

    Science 333:404.

    https://doi.org/10.1126/science.333.6041.404-a

    • Google Scholar
38. 1. Stegle O
    2. Parts L
    3. Piipari M
    4. Winn J
    5. Durbin R

    (2012) Using probabilistic estimation of expression residuals (PEER) to obtain increased power and interpretability of gene expression analyses

    Nature Protocols 7:500–507.

    https://doi.org/10.1038/nprot.2011.457

    • Google Scholar
39. 1. The Economist

    (2013) Unreliable research: trouble at the lab

    The Economist (19 October 2013), Available at: http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble.

    http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

    • Google Scholar
40. 1. Westgard JO
    2. Barry PL
    3. Hunt MR
    4. Groth T

    (1981)

    A multi-rule shewhart chart for quality-control in clinical-chemistry

    Clinical Chemistry 27:493–501.

    • Google Scholar

Author details

1. Nicholas GD Masca

   Cardiovascular Biomedical Research Unit, University of Leicester, Leicester, United Kingdom

   Contribution

   NGDM, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   Contributed equally with

   Elizabeth MA Hensor and Victoria R Cornelius

   Competing interests

   The authors declare that no competing interests exist.

2. Elizabeth MA Hensor

   Leeds Institute of Rheumatic and Musculoskeletal Medicine, University of Leeds, Leeds, United Kingdom

   Contribution

   EMAH, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   Contributed equally with

   Nicholas GD Masca and Victoria R Cornelius

   Competing interests

   The authors declare that no competing interests exist.

3. Victoria R Cornelius

   Department of Primary Care and Public Health Sciences, King's College London, London, United Kingdom

   Contribution

   VRC, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   Contributed equally with

   Nicholas GD Masca and Elizabeth MA Hensor

   Competing interests

   The authors declare that no competing interests exist.

4. Francesca M Buffa

   Applied Computational Genomics, University of Oxford, Oxford, United Kingdom

   Contribution

   FMB, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

5. Helen M Marriott

   Department of Infection and Immunity, University of Sheffield, Sheffield, United Kingdom

   Contribution

   HMM, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

6. James M Eales

   Department of Cardiovascular Sciences, University of Leicester, Leicester, United Kingdom

   Contribution

   JME, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

7. Michael P Messenger

   NIHR Diagnostic Evidence Co-Operative Leeds, Leeds Teaching Hospitals NHS Trust, Leeds, United Kingdom

   Contribution

   MPM, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

8. Amy E Anderson

   Musculoskeletal Research Group, Institute of Cellular Medicine, University of Newcastle, Newcastle, United Kingdom

   Contribution

   AEA, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

9. Chris Boot

   Newcastle Hospitals NHS Trust, Newcastle, United Kingdom

   Contribution

   CB, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   The authors declare that no competing interests exist.

10. Catey Bunce

    NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, United Kingdom

    Contribution

    CB, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

11. Robert D Goldin

    Centre for Pathology, Imperial College, London, United Kingdom

    Contribution

    RDG, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

12. Jessica Harris

    Clinical Trials and Evaluation Unit, School of Clinical Sciences, University of Bristol, Bristol, United Kingdom

    Contribution

    JH, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

13. Rod F Hinchliffe

    Department of Paediatric Haematology, Sheffield Children's NHS Foundation Trust, Sheffield, United Kingdom

    Contribution

    RFH, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

14. Hiba Junaid

    Royal London Hospital, London, United Kingdom

    Contribution

    HJ, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

15. Shaun Kingston

    Respiratory Biomedical Research Unit, Royal Brompton and Harefield NHS Trust, London, United Kingdom

    Contribution

    SK, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

16. Carmen Martin-Ruiz

    Institute for Ageing and Health, Newcastle University, Newcastle, United Kingdom

    Contribution

    CM-R, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

17. Christopher P Nelson

    Department of Cardiovascular Sciences, NIHR Leicester Cardiovascular Biomedical Research Unit, University of Leicester, Leicester, United Kingdom

    Contribution

    CPN, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

18. Janet Peacock

    Division of Health and Social Care Research, Kings College London, London, United Kingdom

    Contribution

    JP, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

19. Paul T Seed

    Division of Women's Health, King's College London, London, United Kingdom

    Contribution

    PTS, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

    "This ORCID iD identifies the author of this article:" 0000-0001-7904-7933

20. Bethany Shinkins

    Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom

    Contribution

    BS, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

21. Karl J Staples

    Clinical and Experimental Sciences, University of Southampton and NIHR Southampton Respiratory Biomedical Research Unit, Southampton General Hospital, Southampton, United Kingdom

    Contribution

    KJS, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

22. Jamie Toombs

    Department of Molecular Neuroscience, Institute of Neurology, University College London, London, United Kingdom

    Contribution

    JT, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

23. Adam KA Wright

    Institute of Lung Health, Respiratory Biomedical Unit, University Hospitals of Leicester NHS Trust, Leicester, United Kingdom

    Contribution

    AKAW, Conception and design, Analysis and interpretation of data, Drafting or revising the article

    Competing interests

    The authors declare that no competing interests exist.

24. M Dawn Teare

    Sheffield School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom

    Contribution

    MDT, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

    For correspondence

    m.d.teare@sheffield.ac.uk

    Competing interests

    The authors declare that no competing interests exist.