The impact of preclinical research using animal models is conditional on its scientific validity, reproducibility, and representation of human physiology and condition (Landis et al., 2012; Chalmers et al., 2014; van der Worp et al., 2010). Improving the quality of the design, conduct, and reporting of preclinical studies may lead to a reduction in research waste (Chalmers et al., 2014; Ioannidis et al., 2014), as well as increase their utility in informing the development of novel therapies (Begley and Ellis, 2012; Langley et al., 2017). One method to do so may be the application of multicenter experimentation in preclinical studies. This is analogous to what is done in clinical trials where positive findings from a single center study are usually evaluated and confirmed in a multicenter study (Chamuleau et al., 2018; Bath et al., 2009; Bellomo et al., 2009). Multicenter studies allow for the comparison of effects between centers, which provides insight into the generalizability of effects across institutions (Cheng et al., 2017). Thus, they inherently test reproducibility while also increasing efficiency in attaining sufficient sample sizes (Bath et al., 2009). In addition, rigorously designed and reported multicenter studies may enhance the confidence in study findings and increase transparency (Maertens et al., 2017). This approach has been adopted in other fields such as social and developmental psychology (Visser et al., 2022; Baumeister et al., 2022).

Multiple calls from the biomedical science community have been made to adopt and apply multicenter study design to preclinical laboratory-based research (Langley et al., 2017; Chamuleau et al., 2018; Maertens et al., 2017; O’Brien et al., 2013; Boltze et al., 2016; Dirnagl and Fisher, 2012). Some recent examples have been published that exemplify the successful implementation of this approach (Llovera et al., 2015; Jones et al., 2015; Maysami et al., 2016). Indeed, multilaboratory studies may offer a method to test issues of reproducibility that have been highlighted by several studies (Begley and Ellis, 2012; Errington et al., 2021b). As interest in preclinical multilaboratory studies grows, and major funders begin to invest in this approach (Federal Ministry of Education and Research, 2022), a systematic evaluation of this method is needed. This will inform and optimize future multicenter preclinical studies by producing a synthesis of current practices and outcomes, while also identifying knowledge gaps and areas for improvement (Dirnagl and Fisher, 2012; Llovera and Liesz, 2016; Fernández-Jiménez and Ibanez, 2015). Moreover, it is currently unknown how the results obtained from a preclinical multilaboratory study compare to a preclinical study conducted in a single laboratory. This comparison is of interest as multiple clinical meta-epidemiological studies have shown that single center clinical trials have a higher risk of bias and overestimate treatment effects compared to multicenter trials (i.e. smaller clinical trials at single sites have a higher probability of methodological shortcomings, lower inferential strength, and may provide inaccurately high estimates of treatment effects) (Unverzagt et al., 2013; Dechartres et al., 2011; Bafeta et al., 2012). For this reason, results from single center studies are generally used cautiously for clinical decision-making (Bellomo et al., 2009). Currently, there has been no empirical investigation into whether this trend occurs in the preclinical domain. This knowledge would provide greater insight into the potential value of the multilaboratory design in preclinical research.

The first objective of this systematic review was to identify, assess, and synthesize the current preclinical multilaboratory study literature. The second objective was to empirically determine if differences exist in the methodological rigor and effect sizes between single lab and multilaboratory studies.

Multiple calls for the use of multilaboratory study design in preclinical research have been published (O’Brien et al., 2013; Dirnagl and Fisher, 2012; Llovera and Liesz, 2016; Fernández-Jiménez and Ibanez, 2015; Mondello et al., 2016). Here, we have synthesized characteristics and outcomes from all interventional preclinical multilaboratory studies published in our search period. Our results suggest that this is an emerging, novel, and promising area of research. The sixteen identified multilaboratory studies had investigated a broad range of diseases, promoted collaboration, adopted many methods to reduce bias, and demonstrated high completeness of reporting. In addition, we found that multilaboratory studies had higher methodological rigor than single lab studies, demonstrated by the greater level of implementation of several key practices known to reduce bias. We observed that multilaboratory studies showed significantly smaller intervention effect sizes than matched single lab studies. This approach addresses pressing issues that have been recently highlighted such as reproducibility, transparent reporting, and collaboration with data and resource sharing (Errington et al., 2021b; Kane and Kimmelman, 2021; Errington et al., 2021a).

The differences between single and multilaboratory preclinical studies observed here have also been noted in clinical studies. Several comparisons between clinical single-and multilaboratory RCTs have been performed, all finding that single center RCTs demonstrate larger effect sizes (Bellomo et al., 2009; Unverzagt et al., 2013; Dechartres et al., 2011; Bafeta et al., 2012). These studies also found that multicenter clinical RCTs had larger sample sizes and greater adherence to practices such as allocation concealment, randomization, and blinding. This difference in sample size and methodological quality, which we have also observed, may explain the discrepancy in effect sizes. It has been shown that smaller studies included in meta-analyses provide larger intervention effects than larger studies (Zhang et al., 2013; Fraley and Vazire, 2014). Furthermore, preclinical studies with a high risk of bias (i.e. low methodological quality) may produce inflated estimates of intervention effects (Landis et al., 2012; Collins and Tabak, 2014). Interestingly, the discrepancy in methodological quality between single and multilaboratory studies was larger in our preclinical comparison than in previous clinical comparisons. This could be explained by the fact that practices such as blinding and randomization are better established in clinical research, thus, even small single center clinical trials are more likely to adhere to them.

A second, somewhat less intuitive issue that may have contributed to larger effect sizes in single lab studies is the smaller sample sizes that may lead to skewed results. It has been suggested that, even in the absence of other biases, under-powered studies have a greater likelihood of effect inflation and generate more false positives than high-powered studies; a lower probability that any observed differences reach the minimum threshold for asserting the findings reflect a true effect; and have a greater likelihood of effect inflation (Button et al., 2013). Perhaps the low power of these single lab studies, combined with well-recognized issues of publication bias (Sena et al., 2010), contributed to the larger effect sizes we observed.

Completeness of reporting in the multilaboratory studies was also noted to be high across many domains. This is in stark contrast to previous preclinical systematic reviews of single lab studies by our group and others that have found significant deficiencies in reporting (Landis et al., 2012; Fergusson et al., 2019; Avey et al., 2016). Within our sample of multilaboratory studies, replicates, statistics, and blinding were overall transparently reported in the majority of studies. Items specific to multilaboratory designs, such as indicating the number of participating centers in the abstract and identifying as a multilaboratory study in the title were less frequent. One potential explanation for this finding is that guidelines and standards for multilaboratory preclinical studies are just emerging, and there have yet to be any reporting recommendations specific to a preclinical multilaboratory design.

The difference in observed methodological quality and high completeness of reporting in preclinical multilaboratory studies could be explained by the routine oversight and quality control that were employed by some of the multilaboratory studies included in our sample. Though not all multilaboratory studies reported routine oversight, we expect that this is inherent in collaborative studies between multiple independent research groups. As reported by several studies, the coordination of a successful preclinical multilaboratory study requires greater training, standardization of protocols, and study-level management when compared to a preclinical study within a single laboratory. Another barrier was the issue of obtaining adequate funding for a multilaboratory study. As a consequence of limited funding, we would speculate that these studies may have undergone more scrutiny and refinement by multiple investigators, funders, and other stakeholders. Indeed, comparison of single lab studies that had been conducted by authors of multilaboratory studies suggested differences in the conduct and outcomes of these studies (despite having the some of the same researchers involved in both). However, this post hoc analysis was qualitative with a limited sample; thus, future studies will need to explore these issues further.

Due to the greater methodological rigor and transparent reporting, the inferred routine oversight, and larger sample sizes, we speculate that preclinical multilaboratory studies may provide a more precise evaluation of the intervention’s effects than do single lab studies. As research groups globally consider adopting this approach, the biomedical community may benefit by emulating successful existing networks of multicenter studies in social psychology (https://psysciacc.org/), developmental psychology (https://manybabies.github.io), and special education research (https://edresearchaccelerator.org/). Moreover, identified barriers and enablers to these studies should be further explored from a variety of stakeholder perspectives (e.g. researchers, animal ethics committees, institutes, and funders) in order to maximize future chances of success.

The protocol for the effect size comparison was developed a priori and posted on Open Science Framework (https://osf.io/awvs9/).Supplementary documents contains the search strategies, risk of bias assessments, reporting checklists, quality scores, effect sizes, effect size ratios, and standardized mean differences to generate the figures and tables.

1. 1. Lalu M
   2. Hunniford V

   (2020) Open Science Framework

   ID awvs9. Study Protocol: Quantitative comparison between the effect sizes of preclinical multicenter studies and single center studies.

   https://osf.io/awvs9/

1. 1. Alam HB
   2. Bice LM
   3. Butt MU
   4. Cho SD
   5. Dubick MA
   6. Duggan M
   7. Englehart MS
   8. Holcomb JB
   9. Morris MS
   10. Prince MD
   11. Schreiber MA
   12. Shults C
   13. Sondeen JL
   14. Tabbara M
   15. Tieu BH
   16. Underwood SA
   17. Hemostatic Resuscitation Research Group

   (2009) Testing of blood products in a polytrauma model: results of a multi-institutional randomized preclinical trial

   The Journal of Trauma 67:856–864.

   https://doi.org/10.1097/TA.0b013e3181b5ae75

   • PubMed
   • Google Scholar
2. 1. Arroyo-Araujo M
   2. Graf R
   3. Maco M
   4. van Dam E
   5. Schenker E
   6. Drinkenburg W
   7. Koopmans B
   8. de Boer SF
   9. Cullum-Doyle M
   10. Noldus LPJJ
   11. Loos M
   12. van Dommelen W
   13. Spooren W
   14. Biemans B
   15. Buhl DL
   16. Kas MJ

   (2019) Reproducibility via coordinated standardization: a multi-center study in a SHANK2 genetic rat model for autism spectrum disorders

   Scientific Reports 9:11602.

   https://doi.org/10.1038/s41598-019-47981-0

   • PubMed
   • Google Scholar
3. 1. Arroyo-Araujo M
   2. Voelkl B
   3. Laloux C
   4. Novak J
   5. Koopmans B
   6. Waldron A-M
   7. Seiffert I
   8. Stirling H
   9. Aulehner K
   10. Janhunen SK
   11. Ramboz S
   12. Potschka H
   13. Holappa J
   14. Fine T
   15. Loos M
   16. Boulanger B
   17. Würbel H
   18. Kas MJ
   19. Lino de Oliveira C

   (2022) Systematic assessment of the replicability and generalizability of preclinical findings: impact of protocol harmonization across laboratory sites

   PLOS Biology 20:e3001886.

   https://doi.org/10.1371/journal.pbio.3001886

   • Google Scholar
4. 1. Avey MT
   2. Moher D
   3. Sullivan KJ
   4. Fergusson D
   5. Griffin G
   6. Grimshaw JM
   7. Hutton B
   8. Lalu MM
   9. Macleod M
   10. Marshall J
   11. Mei SHJ
   12. Rudnicki M
   13. Stewart DJ
   14. Turgeon AF
   15. McIntyre L
   16. Canadian Critical Care Translational Biology Group

   (2016) The devil is in the details: incomplete reporting in preclinical animal research

   PLOS ONE 11:e0166733.

   https://doi.org/10.1371/journal.pone.0166733

   • PubMed
   • Google Scholar
5. 1. Bafeta A
   2. Dechartres A
   3. Trinquart L
   4. Yavchitz A
   5. Boutron I
   6. Ravaud P

   (2012) Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study

   BMJ 344:e813.

   https://doi.org/10.1136/bmj.e813

   • PubMed
   • Google Scholar
6. 1. Bailey KR

   (1991) Detecting fabrication of data in a multicenter collaborative animal study

   Controlled Clinical Trials 12:741–752.

   https://doi.org/10.1016/0197-2456(91)90037-m

   • PubMed
   • Google Scholar
7. 1. Bath PMW
   2. Macleod MR
   3. Green AR

   (2009) Emulating multicentre clinical stroke trials: a new paradigm for studying novel interventions in experimental models of stroke

   International Journal of Stroke 4:471–479.

   https://doi.org/10.1111/j.1747-4949.2009.00386.x

   • PubMed
   • Google Scholar
8. 1. Baumeister RF
   2. Tice DM
   3. Bushman BJ

   (2022) A review of multisite replication projects in social psychology: is it viable to sustain any confidence in social psychology’s knowledge base?

   Perspectives on Psychological Science 2022:17456916221121816.

   https://doi.org/10.1177/17456916221121815

   • PubMed
   • Google Scholar
9. 1. Begley CG
   2. Ellis LM

   (2012) Drug development: raise standards for preclinical cancer research

   Nature 483:531–533.

   https://doi.org/10.1038/483531a

   • PubMed
   • Google Scholar
10. 1. Bello S
    2. Krogsbøll LT
    3. Gruber J
    4. Zhao ZJ
    5. Fischer D
    6. Hróbjartsson A

    (2014) Lack of blinding of outcome assessors in animal model experiments implies risk of observer bias

    Journal of Clinical Epidemiology 67:973–983.

    https://doi.org/10.1016/j.jclinepi.2014.04.008

    • PubMed
    • Google Scholar
11. 1. Bellomo R
    2. Warrillow SJ
    3. Reade MC

    (2009) Why we should be wary of single-center trials

    Critical Care Medicine 37:3114–3119.

    https://doi.org/10.1097/CCM.0b013e3181bc7bd5

    • PubMed
    • Google Scholar
12. 1. Boltze J
    2. Wagner DC
    3. Henninger N
    4. Plesnila N
    5. Ayata C

    (2016) Phase III preclinical trials in translational stroke research: community response on framework and guidelines

    Translational Stroke Research 7:241–247.

    https://doi.org/10.1007/s12975-016-0474-6

    • PubMed
    • Google Scholar
13. 1. Bramlett HM
    2. Dietrich WD
    3. Dixon CE
    4. Shear DA
    5. Schmid KE
    6. Mondello S
    7. Wang KKW
    8. Hayes RL
    9. Povlishock JT
    10. Tortella FC
    11. Kochanek PM

    (2016) Erythropoietin treatment in traumatic brain injury: operation brain trauma therapy

    Journal of Neurotrauma 33:538–552.

    https://doi.org/10.1089/neu.2015.4116

    • PubMed
    • Google Scholar
14. 1. Browning M
    2. Shear DA
    3. Bramlett HM
    4. Dixon CE
    5. Mondello S
    6. Schmid KE
    7. Poloyac SM
    8. Dietrich WD
    9. Hayes RL
    10. Wang KKW
    11. Povlishock JT
    12. Tortella FC
    13. Kochanek PM

    (2016) Levetiracetam treatment in traumatic brain injury: operation brain trauma therapy

    Journal of Neurotrauma 33:581–594.

    https://doi.org/10.1089/neu.2015.4131

    • PubMed
    • Google Scholar
15. 1. Button KS
    2. Ioannidis JPA
    3. Mokrysz C
    4. Nosek BA
    5. Flint J
    6. Robinson ESJ
    7. Munafò MR

    (2013) Power failure: why small sample size undermines the reliability of neuroscience

    Nature Reviews. Neuroscience 14:365–376.

    https://doi.org/10.1038/nrn3475

    • PubMed
    • Google Scholar
16. 1. Chalmers I
    2. Bracken MB
    3. Djulbegovic B
    4. Garattini S
    5. Grant J
    6. Gülmezoglu AM
    7. Howells DW
    8. Ioannidis JPA
    9. Oliver S

    (2014) How to increase value and reduce waste when research priorities are set

    Lancet 383:156–165.

    https://doi.org/10.1016/S0140-6736(13)62229-1

    • PubMed
    • Google Scholar
17. 1. Chamuleau SAJ
    2. van der Naald M
    3. Climent AM
    4. Kraaijeveld AO
    5. Wever KE
    6. Duncker DJ
    7. Fernández-Avilés F
    8. Bolli R
    9. Transnational Alliance for Regenerative Therapies in Cardiovascular Syndromes Group

    (2018) Translational research in cardiovascular repair: a call for a paradigm shift

    Circulation Research 122:310–318.

    https://doi.org/10.1161/CIRCRESAHA.117.311565

    • PubMed
    • Google Scholar
18. 1. Cheng A
    2. Kessler D
    3. Mackinnon R
    4. Chang TP
    5. Nadkarni VM
    6. Hunt EA
    7. Duval-Arnould J
    8. Lin Y
    9. Pusic M
    10. Auerbach M

    (2017) Conducting multicenter research in healthcare simulation: lessons learned from the INSPIRE network

    Advances in Simulation 2:6.

    https://doi.org/10.1186/s41077-017-0039-0

    • PubMed
    • Google Scholar
19. 1. Collins FS
    2. Tabak LA

    (2014) Policy: NIH plans to enhance reproducibility

    Nature 505:612–613.

    https://doi.org/10.1038/505612a

    • PubMed
    • Google Scholar
20. 1. Crabbe JC
    2. Wahlsten D
    3. Dudek BC

    (1999) Genetics of mouse behavior: interactions with laboratory environment

    Science 284:1670–1672.

    https://doi.org/10.1126/science.284.5420.1670

    • PubMed
    • Google Scholar
21. 1. Dechartres A
    2. Boutron I
    3. Trinquart L
    4. Charles P
    5. Ravaud P

    (2011) Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study

    Annals of Internal Medicine 155:39–51.

    https://doi.org/10.7326/0003-4819-155-1-201107050-00006

    • PubMed
    • Google Scholar
22. 1. Dirnagl U
    2. Fisher M

    (2012) Reprint: international, multicenter randomized preclinical trials in translational stroke research: it is time to act

    Stroke 43:1453–1454.

    https://doi.org/10.1161/STROKEAHA.112.653709

    • PubMed
    • Google Scholar
23. 1. Dixon CE
    2. Bramlett HM
    3. Dietrich WD
    4. Shear DA
    5. Yan HQ
    6. Deng-Bryant Y
    7. Mondello S
    8. Wang KKW
    9. Hayes RL
    10. Empey PE
    11. Povlishock JT
    12. Tortella FC
    13. Kochanek PM

    (2016) Cyclosporine treatment in traumatic brain injury: operation brain trauma therapy

    Journal of Neurotrauma 33:553–566.

    https://doi.org/10.1089/neu.2015.4122

    • PubMed
    • Google Scholar
24. 1. Errington TM
    2. Denis A
    3. Perfito N
    4. Iorns E
    5. Nosek BA

    (2021a) Challenges for assessing replicability in preclinical cancer biology

    eLife 10:e67995.

    https://doi.org/10.7554/eLife.67995

    • PubMed
    • Google Scholar
25. 1. Errington TM
    2. Mathur M
    3. Soderberg CK
    4. Denis A
    5. Perfito N
    6. Iorns E
    7. Nosek BA

    (2021b) Investigating the replicability of preclinical cancer biology

    eLife 10:e71601.

    https://doi.org/10.7554/eLife.71601

    • PubMed
    • Google Scholar
26. Website

    1. Federal Ministry of Education and Research

    (2022) Call for proposals for preclinical confirmatory studies and systematic reviews

    Accessed April 5, 2023.

    https://www.gesundheitsforschung-bmbf.de/de/14868.php
27. 1. Fergusson DA
    2. Wesch NL
    3. Leung GJ
    4. MacNeil JL
    5. Conic I
    6. Presseau J
    7. Cobey KD
    8. Diallo J-S
    9. Auer R
    10. Kimmelman J
    11. Kekre N
    12. El-Sayes N
    13. Krishnan R
    14. Keller BA
    15. Ilkow C
    16. Lalu MM

    (2019) Assessing the completeness of reporting in preclinical oncolytic virus therapy studies

    Molecular Therapy Oncolytics 14:179–187.

    https://doi.org/10.1016/j.omto.2019.05.004

    • PubMed
    • Google Scholar
28. 1. Fernández-Jiménez R
    2. Ibanez B

    (2015) CAESAR: one step beyond in the construction of a translational bridge for cardioprotection

    Circulation Research 116:554–556.

    https://doi.org/10.1161/CIRCRESAHA.115.305841

    • PubMed
    • Google Scholar
29. 1. Fraley RC
    2. Vazire S

    (2014) The N-pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power

    PLOS ONE 9:e109019.

    https://doi.org/10.1371/journal.pone.0109019

    • PubMed
    • Google Scholar
30. 1. Gill RG
    2. Pagni PP
    3. Kupfer T
    4. Wasserfall CH
    5. Deng S
    6. Posgai A
    7. Manenkova Y
    8. Bel Hani A
    9. Straub L
    10. Bernstein P
    11. Atkinson MA
    12. Herold KC
    13. von Herrath M
    14. Staeva T
    15. Ehlers MR
    16. Nepom GT

    (2016) A preclinical Consortium approach for assessing the efficacy of combined anti-CD3 plus IL-1 blockade in reversing new-onset autoimmune diabetes in NOD mice

    Diabetes 65:1310–1316.

    https://doi.org/10.2337/db15-0492

    • PubMed
    • Google Scholar
31. 1. Haby MM
    2. Chapman E
    3. Clark R
    4. Barreto J
    5. Reveiz L
    6. Lavis JN

    (2016) What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review

    Health Research Policy and Systems 14:83.

    https://doi.org/10.1186/s12961-016-0155-7

    • PubMed
    • Google Scholar
32. Report

    1. Health Canada

    (2016)

    E6(R2) Good Clinial Practice: Integrated Addendum to ICH E6(R1) - Guidance for Industry

    US Food and Drug Administration.

    • Google Scholar
33. 1. Holman C
    2. Piper SK
    3. Grittner U
    4. Diamantaras AA
    5. Kimmelman J
    6. Siegerink B
    7. Dirnagl U

    (2016) Where have all the rodents gone? the effects of attrition in experimental research on cancer and stroke

    PLOS Biology 14:e1002331.

    https://doi.org/10.1371/journal.pbio.1002331

    • PubMed
    • Google Scholar
34. 1. Hooijmans CR
    2. Tillema A
    3. Leenaars M
    4. Ritskes-Hoitinga M

    (2010) Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed

    Laboratory Animals 44:170–175.

    https://doi.org/10.1258/la.2010.009117

    • PubMed
    • Google Scholar
35. 1. Hooijmans CR
    2. Rovers MM
    3. de Vries RBM
    4. Leenaars M
    5. Ritskes-Hoitinga M
    6. Langendam MW

    (2014) SYRCLE’s risk of bias tool for animal studies

    BMC Medical Research Methodology 14:43.

    https://doi.org/10.1186/1471-2288-14-43

    • PubMed
    • Google Scholar
36. 1. Ioannidis JPA
    2. Greenland S
    3. Hlatky MA
    4. Khoury MJ
    5. Macleod MR
    6. Moher D
    7. Schulz KF
    8. Tibshirani R

    (2014) Increasing value and reducing waste in research design, conduct, and analysis

    Lancet 383:166–175.

    https://doi.org/10.1016/S0140-6736(13)62227-8

    • PubMed
    • Google Scholar
37. 1. Jadad AR
    2. Moore RA
    3. Carroll D
    4. Jenkinson C
    5. Reynolds DJ
    6. Gavaghan DJ
    7. McQuay HJ

    (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary?

    Controlled Clinical Trials 17:1–12.

    https://doi.org/10.1016/0197-2456(95)00134-4

    • PubMed
    • Google Scholar
38. 1. Jha RM
    2. Mondello S
    3. Bramlett HM
    4. Dixon CE
    5. Shear DA
    6. Dietrich WD
    7. Wang KKW
    8. Yang Z
    9. Hayes RL
    10. Poloyac SM
    11. Empey PE
    12. Lafrenaye AD
    13. Yan HQ
    14. Carlson SW
    15. Povlishock JT
    16. Gilsdorf JS
    17. Kochanek PM

    (2020) Glibenclamide treatment in traumatic brain injury: operation brain trauma therapy

    Journal of Neurotrauma 38:628–645.

    https://doi.org/10.1089/neu.2020.7421

    • PubMed
    • Google Scholar
39. 1. Jones SP
    2. Tang X-L
    3. Guo Y
    4. Steenbergen C
    5. Lefer DJ
    6. Kukreja RC
    7. Kong M
    8. Li Q
    9. Bhushan S
    10. Zhu X
    11. Du J
    12. Nong Y
    13. Stowers HL
    14. Kondo K
    15. Hunt GN
    16. Goodchild TT
    17. Orr A
    18. Chang CC
    19. Ockaili R
    20. Salloum FN
    21. Bolli R

    (2015) The NHLBI-sponsored Consortium for preclinical assessment of cardioprotective therapies (Caesar): a new paradigm for rigorous, accurate, and reproducible evaluation of putative infarct-sparing interventions in mice, rabbits, and pigs

    Circulation Research 116:572–586.

    https://doi.org/10.1161/CIRCRESAHA.116.305462

    • PubMed
    • Google Scholar
40. 1. Kane PB
    2. Kimmelman J

    (2021) Is preclinical research in cancer biology reproducible enough?

    eLife 10:e67527.

    https://doi.org/10.7554/eLife.67527

    • PubMed
    • Google Scholar
41. 1. Khangura S
    2. Konnyu K
    3. Cushman R
    4. Grimshaw J
    5. Moher D

    (2012) Evidence summaries: the evolution of a rapid review approach

    Systematic Reviews 1:10.

    https://doi.org/10.1186/2046-4053-1-10

    • PubMed
    • Google Scholar
42. 1. Kliewer A
    2. Gillis A
    3. Hill R
    4. Schmiedel F
    5. Bailey C
    6. Kelly E
    7. Henderson G
    8. Christie MJ
    9. Schulz S

    (2020) Morphine-Induced respiratory depression is independent of β-arrestin2 signalling

    British Journal of Pharmacology 177:2923–2931.

    https://doi.org/10.1111/bph.15004

    • PubMed
    • Google Scholar
43. 1. Kochanek PM
    2. Bramlett HM
    3. Dixon CE
    4. Shear DA
    5. Dietrich WD
    6. Schmid KE
    7. Mondello S
    8. Wang KKW
    9. Hayes RL
    10. Povlishock JT
    11. Tortella FC

    (2016) Approach to modeling, therapy evaluation, drug selection, and biomarker assessments for a multicenter pre-clinical drug screening Consortium for acute therapies in severe traumatic brain injury: operation brain trauma therapy

    Journal of Neurotrauma 33:513–522.

    https://doi.org/10.1089/neu.2015.4113

    • PubMed
    • Google Scholar
44. 1. Landis SC
    2. Amara SG
    3. Asadullah K
    4. Austin CP
    5. Blumenstein R
    6. Bradley EW
    7. Crystal RG
    8. Darnell RB
    9. Ferrante RJ
    10. Fillit H
    11. Finkelstein R
    12. Fisher M
    13. Gendelman HE
    14. Golub RM
    15. Goudreau JL
    16. Gross RA
    17. Gubitz AK
    18. Hesterlee SE
    19. Howells DW
    20. Huguenard J
    21. Kelner K
    22. Koroshetz W
    23. Krainc D
    24. Lazic SE
    25. Levine MS
    26. Macleod MR
    27. McCall JM
    28. Moxley RT
    29. Narasimhan K
    30. Noble LJ
    31. Perrin S
    32. Porter JD
    33. Steward O
    34. Unger E
    35. Utz U
    36. Silberberg SD

    (2012) A call for transparent reporting to optimize the predictive value of preclinical research

    Nature 490:187–191.

    https://doi.org/10.1038/nature11556

    • PubMed
    • Google Scholar
45. 1. Langley GR
    2. Adcock IM
    3. Busquet F
    4. Crofton KM
    5. Csernok E
    6. Giese C
    7. Heinonen T
    8. Herrmann K
    9. Hofmann-Apitius M
    10. Landesmann B
    11. Marshall LJ
    12. McIvor E
    13. Muotri AR
    14. Noor F
    15. Schutte K
    16. Seidle T
    17. van de Stolpe A
    18. Van Esch H
    19. Willett C
    20. Woszczek G

    (2017) Towards a 21st-century roadmap for biomedical research and drug discovery: consensus report and recommendations

    Drug Discovery Today 22:327–339.

    https://doi.org/10.1016/j.drudis.2016.10.011

    • PubMed
    • Google Scholar
46. 1. Llovera G
    2. Hofmann K
    3. Roth S
    4. Salas-Pérdomo A
    5. Ferrer-Ferrer M
    6. Perego C
    7. Zanier ER
    8. Mamrak U
    9. Rex A
    10. Party H
    11. Agin V
    12. Fauchon C
    13. Orset C
    14. Haelewyn B
    15. De Simoni MG
    16. Dirnagl U
    17. Grittner U
    18. Planas AM
    19. Plesnila N
    20. Vivien D
    21. Liesz A

    (2015) Results of a preclinical randomized controlled multicenter trial (prct): anti-cd49d treatment for acute brain ischemia

    Science Translational Medicine 7:299.

    https://doi.org/10.1126/scitranslmed.aaa9853

    • PubMed
    • Google Scholar
47. 1. Llovera G
    2. Liesz A

    (2016) The next step in translational research: lessons learned from the first preclinical randomized controlled trial

    Journal of Neurochemistry 139 Suppl 2:271–279.

    https://doi.org/10.1111/jnc.13516

    • PubMed
    • Google Scholar
48. 1. Macleod MR
    2. van der Worp HB
    3. Sena ES
    4. Howells DW
    5. Dirnagl U
    6. Donnan GA

    (2008) Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality

    Stroke 39:2824–2829.

    https://doi.org/10.1161/STROKEAHA.108.515957

    • PubMed
    • Google Scholar
49. 1. Maertens O
    2. McCurrach ME
    3. Braun BS
    4. De Raedt T
    5. Epstein I
    6. Huang TQ
    7. Lauchle JO
    8. Lee H
    9. Wu J
    10. Cripe TP
    11. Clapp DW
    12. Ratner N
    13. Shannon K
    14. Cichowski K

    (2017) A collaborative model for accelerating the discovery and translation of cancer therapies

    Cancer Research 77:5706–5711.

    https://doi.org/10.1158/0008-5472.CAN-17-1789

    • PubMed
    • Google Scholar
50. 1. Maysami S
    2. Wong R
    3. Pradillo JM
    4. Denes A
    5. Dhungana H
    6. Malm T
    7. Koistinaho J
    8. Orset C
    9. Rahman M
    10. Rubio M
    11. Schwaninger M
    12. Vivien D
    13. Bath PM
    14. Rothwell NJ
    15. Allan SM

    (2016) A cross-laboratory preclinical study on the effectiveness of interleukin-1 receptor antagonist in stroke

    Journal of Cerebral Blood Flow and Metabolism 36:596–605.

    https://doi.org/10.1177/0271678X15606714

    • PubMed
    • Google Scholar
51. 1. McGowan J
    2. Sampson M
    3. Salzwedel DM
    4. Cogo E
    5. Foerster V
    6. Lefebvre C

    (2016) Press peer review of electronic search strategies: 2015 guideline statement

    Journal of Clinical Epidemiology 75:40–46.

    https://doi.org/10.1016/j.jclinepi.2016.01.021

    • PubMed
    • Google Scholar
52. Software

    1. Mitchell M
    2. Muftakhidinov B
    3. Winchen TE

    (2020) Digitizer software, version 12.1

    Github.

    http://markummitchell.github.io/engauge-digitizer
53. 1. Moher D
    2. Liberati A
    3. Tetzlaff J
    4. Altman DG
    5. Group P

    (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

    PLOS Medicine 6:e1000097.

    https://doi.org/10.1371/journal.pmed.1000097

    • PubMed
    • Google Scholar
54. 1. Moher D
    2. Hopewell S
    3. Schulz KF
    4. Montori V
    5. Gøtzsche PC
    6. Devereaux PJ
    7. Elbourne D
    8. Egger M
    9. Altman DG
    10. Consolidated Standards of Reporting Trials Group

    (2010) Consort 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials

    Journal of Clinical Epidemiology 63:e1–e37.

    https://doi.org/10.1016/j.jclinepi.2010.03.004

    • PubMed
    • Google Scholar
55. 1. Mondello S
    2. Shear DA
    3. Bramlett HM
    4. Dixon CE
    5. Schmid KE
    6. Dietrich WD
    7. Wang KKW
    8. Hayes RL
    9. Glushakova O
    10. Catania M
    11. Richieri SP
    12. Povlishock JT
    13. Tortella FC
    14. Kochanek PM

    (2016) Insight into pre-clinical models of traumatic brain injury using circulating brain damage biomarkers: operation brain trauma therapy

    Journal of Neurotrauma 33:595–605.

    https://doi.org/10.1089/neu.2015.4132

    • PubMed
    • Google Scholar
56. 1. Mountney A
    2. Bramlett HM
    3. Dixon CE
    4. Mondello S
    5. Dietrich WD
    6. Wang KKW
    7. Caudle K
    8. Empey PE
    9. Poloyac SM
    10. Hayes RL
    11. Povlishock JT
    12. Tortella FC
    13. Kochanek PM
    14. Shear DA

    (2016) Simvastatin treatment in traumatic brain injury: operation brain trauma therapy

    Journal of Neurotrauma 33:567–580.

    https://doi.org/10.1089/neu.2015.4130

    • PubMed
    • Google Scholar
57. 1. O’Brien TJ
    2. Ben-Menachem E
    3. Bertram EH
    4. Collins SD
    5. Kokaia M
    6. Lerche H
    7. Klitgaard H
    8. Staley KJ
    9. Vaudano E
    10. Walker MC
    11. Simonato M

    (2013) Proposal for a “phase II” multicenter trial model for preclinical new antiepilepsy therapy development

    Epilepsia 54 Suppl 4:70–74.

    https://doi.org/10.1111/epi.12300

    • PubMed
    • Google Scholar
58. 1. O’Connor AM
    2. Sargeant JM

    (2014) Critical appraisal of studies using laboratory animal models

    ILAR Journal 55:405–417.

    https://doi.org/10.1093/ilar/ilu038

    • PubMed
    • Google Scholar
59. Preprint

    1. Page MJ
    2. McKenzie J
    3. Bossuyt P
    4. Boutron I
    5. Hoffmann T
    6. Mulrow C
    7. Shamseer L
    8. Tetzlaff J
    9. Akl E
    10. Brennan SE
    11. Chou R
    12. Glanville J
    13. Grimshaw J
    14. Hróbjartsson A
    15. Lalu MM
    16. Li T
    17. Loder E
    18. Mayo-Wilson E
    19. McDonald S
    20. McGuinness LA
    21. Stewart L
    22. Thomas J
    23. Tricco A
    24. Welch VA
    25. Whiting P
    26. Moher D

    (2020) The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews

    MetaArXiv.

    https://doi.org/10.31222/osf.io/v7gm2

    • Google Scholar
60. 1. Peng W
    2. Xing Z
    3. Yang J
    4. Wang Y
    5. Wang W
    6. Huang W

    (2014) The efficacy of erythropoietin in treating experimental traumatic brain injury: a systematic review of controlled trials in animal models

    Journal of Neurosurgery 121:653–664.

    https://doi.org/10.3171/2014.6.JNS132577

    • PubMed
    • Google Scholar
61. 1. Percie du Sert N
    2. Ahluwalia A
    3. Alam S
    4. Avey MT
    5. Baker M
    6. Browne WJ
    7. Clark A
    8. Cuthill IC
    9. Dirnagl U
    10. Emerson M
    11. Garner P
    12. Holgate ST
    13. Howells DW
    14. Hurst V
    15. Karp NA
    16. Lazic SE
    17. Lidster K
    18. MacCallum CJ
    19. Macleod M
    20. Pearl EJ
    21. Petersen OH
    22. Rawle F
    23. Reynolds P
    24. Rooney K
    25. Sena ES
    26. Silberberg SD
    27. Steckler T
    28. Würbel H

    (2020) Reporting animal research: explanation and elaboration for the ARRIVE guidelines 2.0

    PLOS Biology 18:e3000411.

    https://doi.org/10.1371/journal.pbio.3000411

    • PubMed
    • Google Scholar
62. Book

    1. Popay J
    2. Roberts H
    3. Sowden A
    4. Petticrew M
    5. Arai L
    6. Rodgers M

    (2006) Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. A product of the ESRC Methods Programme

    Lancaster University.

    https://doi.org/10.13140/2.1.1018.4643

    • Google Scholar
63. 1. Ramirez FD
    2. Jung RG
    3. Motazedian P
    4. Perry-Nguyen D
    5. Di Santo P
    6. MacDonald Z
    7. Clancy AA
    8. Labinaz A
    9. Promislow S
    10. Simard T
    11. Provencher S
    12. Bonnet S
    13. Graham ID
    14. Wells GA
    15. Hibbert B

    (2020) Journal initiatives to enhance preclinical research: analyses of stroke, nature medicine, science translational medicine

    Stroke 51:291–299.

    https://doi.org/10.1161/STROKEAHA.119.026564

    • PubMed
    • Google Scholar
64. 1. Reimer KA
    2. Jennings RB
    3. Cobb FR
    4. Murdock RH
    5. Greenfield JC
    6. Becker LC
    7. Bulkley BH
    8. Hutchins GM
    9. Schwartz RP
    10. Bailey KR

    (1985) Animal models for protecting ischemic myocardium: results of the NHLBI cooperative study: comparison of unconscious and conscious dog models

    Circulation Research 56:651–665.

    https://doi.org/10.1161/01.res.56.5.651

    • PubMed
    • Google Scholar
65. 1. Reynolds PS
    2. Garvan CW

    (2021) Preclinical research reporting in shock: room for improvement

    Shock 55:573–580.

    https://doi.org/10.1097/SHK.0000000000001544

    • PubMed
    • Google Scholar
66. 1. Sena ES
    2. van der Worp HB
    3. Bath PMW
    4. Howells DW
    5. Macleod MR

    (2010) Publication bias in reports of animal stroke studies leads to major overstatement of efficacy

    PLOS Biology 8:e1000344.

    https://doi.org/10.1371/journal.pbio.1000344

    • PubMed
    • Google Scholar
67. 1. Shear DA
    2. Dixon CE
    3. Bramlett HM
    4. Mondello S
    5. Dietrich WD
    6. Deng-Bryant Y
    7. Schmid KE
    8. Wang KKW
    9. Hayes RL
    10. Povlishock JT
    11. Kochanek PM
    12. Tortella FC

    (2016) Nicotinamide treatment in traumatic brain injury: operation brain trauma therapy

    Journal of Neurotrauma 33:523–537.

    https://doi.org/10.1089/neu.2015.4115

    • PubMed
    • Google Scholar
68. 1. Spoerke N
    2. Zink K
    3. Cho SD
    4. Differding J
    5. Muller P
    6. Karahan A
    7. Sondeen J
    8. Holcomb JB
    9. Schreiber M

    (2009) Lyophilized plasma for resuscitation in a swine model of severe injury

    Archives of Surgery 144:829–834.

    https://doi.org/10.1001/archsurg.2009.154

    • PubMed
    • Google Scholar
69. 1. Unverzagt S
    2. Prondzinsky R
    3. Peinemann F

    (2013) Single-center trials tend to provide larger treatment effects than multicenter trials: a systematic review

    Journal of Clinical Epidemiology 66:1271–1280.

    https://doi.org/10.1016/j.jclinepi.2013.05.016

    • PubMed
    • Google Scholar
70. 1. van der Worp HB
    2. Howells DW
    3. Sena ES
    4. Porritt MJ
    5. Rewell S
    6. O’Collins V
    7. Macleod MR

    (2010) Can animal models of disease reliably inform human studies?

    PLOS Medicine 7:e1000245.

    https://doi.org/10.1371/journal.pmed.1000245

    • PubMed
    • Google Scholar
71. 1. Varker T
    2. Forbes D
    3. Dell L
    4. Weston A
    5. Merlin T
    6. Hodson S
    7. O’Donnell M

    (2015) Rapid evidence assessment: increasing the transparency of an emerging methodology

    Journal of Evaluation in Clinical Practice 21:1199–1204.

    https://doi.org/10.1111/jep.12405

    • PubMed
    • Google Scholar
72. 1. Visser I
    2. Bergmann C
    3. Byers-Heinlein K
    4. Dal Ben R
    5. Duch W
    6. Forbes S
    7. Franchin L
    8. Frank MC
    9. Geraci A
    10. Hamlin JK
    11. Kaldy Z
    12. Kulke L
    13. Laverty C
    14. Lew-Williams C
    15. Mateu V
    16. Mayor J
    17. Moreau D
    18. Nomikou I
    19. Schuwerk T
    20. Simpson EA
    21. Singh L
    22. Soderstrom M
    23. Sullivan J
    24. van den Heuvel MI
    25. Westermann G
    26. Yamada Y
    27. Zaadnoordijk L
    28. Zettersten M

    (2022) Improving the generalizability of infant psychological research: the manybabies model

    The Behavioral and Brain Sciences 45:e35.

    https://doi.org/10.1017/S0140525X21000455

    • PubMed
    • Google Scholar
73. 1. Wahlsten D
    2. Metten P
    3. Phillips TJ
    4. Boehm SL
    5. Burkhart-Kasch S
    6. Dorow J
    7. Doerksen S
    8. Downing C
    9. Fogarty J
    10. Rodd-Henricks K
    11. Hen R
    12. McKinnon CS
    13. Merrill CM
    14. Nolte C
    15. Schalomon M
    16. Schlumbohm JP
    17. Sibert JR
    18. Wenger CD
    19. Dudek BC
    20. Crabbe JC

    (2003) Different data from different labs: lessons from studies of gene-environment interaction

    Journal of Neurobiology 54:283–311.

    https://doi.org/10.1002/neu.10173

    • PubMed
    • Google Scholar
74. 1. Wever KE
    2. Hooijmans CR
    3. Riksen NP
    4. Sterenborg TB
    5. Sena ES
    6. Ritskes-Hoitinga M
    7. Warlé MC

    (2015) Determinants of the efficacy of cardiac ischemic preconditioning: a systematic review and meta-analysis of animal studies

    PLOS ONE 10:e0142021.

    https://doi.org/10.1371/journal.pone.0142021

    • PubMed
    • Google Scholar
75. 1. Zhang Z
    2. Xu X
    3. Ni H

    (2013) Small studies may overestimate the effect sizes in critical care meta-analyses: a meta-epidemiological study

    Critical Care 17:R2.

    https://doi.org/10.1186/cc11919

    • PubMed
    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "A systematic assessment of preclinical multicenter studies and a comparison to single laboratory studies" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Mone Zaidi as the Senior Editor. The following individuals involved in the review of your submission have agreed to reveal their identity: David Mellor (Reviewer #1); Andrew SC Rice (Reviewer #3).

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

1) Introduction: I believe that your audience reach would be expanded, and your intended audience would be even more interested in your work, by noting how this method of multi-lab collaborations is appropriate beyond preclinical research. I will not suggest changing your inclusion criteria for the study already conducted, but it would be worth noting how efforts in social psychology* and human development** have seen similar results as those you describe. Likewise, existing networks in psychology (https://psysciacc.org/), development biology (https://manybabies.github.io/), and special education research (https://edresearchaccelerator.org/) would be worth mentioning as models for emulation when discussing how these practices could be scaled to more communities within preclinical research (perhaps mention this in the Conclusion).

2) Line 123: Can you please define "interventional" and clarify your definition of "Preclinical" so as to make it clear to readers outside of your domain of focus why psychological or developmental biology was not included (as these do have human health implications). (Just to be clear, I am not being sarcastic or suggesting that these should have been included, I just know that these terms get blurry when going from the preclinical to the basic sciences).

3) I appreciate that the calculations for the effect size ratio and the difference of standardized mean difference were specified a priori. Even though they were straightforward calculations, there is room for divergence or selective reporting and it appears that those analyses were reported as pre-specified.

4) The sample size is rather limited. While I do not doubt that the authors invested a lot of work into finding multi-centre studies fulfilling their inclusion criteria, they ended up with a list of only 16 studies. Multi-centre studies are tedious to perform and more expensive, so it is not surprising that the authors could not find more. However, given an N=16 for a rather diverse collection of studies, spanning 35 years, different model species (mice, rats, dogs, pigs, and rabbits) and investigating rather different interventions, this might be better regarded as a preliminary analysis.

5) There is one question that cannot be answered but which is nevertheless rather important. Is the difference in effect size estimates (with the multi-lab studies presumably providing more accurate effect size estimates) due to the multi-lab setting itself or due to the fact that researchers willing to go the extra length of conducting a multi-lab study are stricter and more rigorous (and more aware) than the average researcher? If the latter is the case, then this would be a confounding factor and one could not conclude that the observed difference is due to the multi-lab approach itself.

6) As the authors noted themselves, sample sizes of single-lab and multi-lab differed substantially (median 19 vs. 111) and this difference in sample size might itself lead to more accurate effect size estimates (closer to the meta-analytic estimate). I am, therefore, missing an analysis, of whether this effect alone could explain the discrepancy. That is – would single-lab studies using the same number of animals be equally accurate as a multi-lab study?

7) I have some minor methodological issues regarding the evaluation of the risk of bias and specifically its interpretation as a measure for the quality of scientific rigor. This is more of a general critique of the use of 'risk of bias' in systematic reviews and is not specific to this study, but it nevertheless means that the results of this part of the study should be taken with a grain of salt and be interpreted with care. The risk of bias questionnaire is focusing on what is reported in the publication. As it is visible in table 3, even for the multi-lab studies-where risk of bias was considerably lower than for the single-lab studies-a substantial proportion of questions was answered with unclear-i.e., it was, for example, not reported whether allocation concealment was done or not. This could mean that it was not done or it can mean that it was done, but researchers didn't find it noteworthy to report it. Reporting practice slightly differs between fields, it differs between journals (due to space restriction, editorial guidelines, and also the rigor of the editors), and it very strongly changes over time – due to the current discussion about replicability editors and reviewers are now asking for more details than they did a decade ago. So, any analysis of the risk of bias using those criteria should consider journal and year of publication as co-variates. In the present case the authors mentioned, that they tried to match single-lab studies with multi-lab studies also according to age, though a quick look at publication years suggests that this was not always achieved.

8) There seem to be some inconsistencies between Table 3 and Table S11. For example, in the column "Blinding of outcome assessment" in Table S3 there are 4 studies with "unknown" (Crabbe, Alam, Spoerke, Gill), while in Table S11 in the column "Blinding of outcome assessment" there are 4 study scoring 0, but here Arroyo-Araujo scores also 0, while Gill scores 1. How does this fit together?

9) Methods: maybe I have missed that: have you described, whether for the reporting scores you checked only the main article or also the electronic supplementary (supporting information)?

10) Language limits: In the methods description you wrote "No language limits were imposed" (line 147). In the Results section, I could, however, not find any information, on if any non-English papers were found and whether they were included in the analysis. There is also no mention of what you would have done if you encountered a paper in a language you were not capable of. When would this have been an exclusion criterion? In any case, in the methods section where you wrote "no language limits were imposed", please add the information that all search terms were in English. (Or were they also translated into other languages?)

11) Line 166: Primary and secondary research outcomes. According to my own experience, one sees many papers, where the authors report outcomes of several analyses and do not clearly define what the primary and the secondary outcomes are. In fact, I only rarely read that an author writes "this is our primary outcome" and "this was a secondary outcome". Did you encounter such cases and which criteria did you use to decide what counted as a primary and secondary outcome?

12) Line 196: Rapid review method. Please describe in one or two sentences what that is.

13) Line 259: were the effect sizes just pooled or were they weighted by study size or variance?

14) Line 267: I think giving a ratio (without multiplication by 100) might be better

15) Line 345: potentially industry-related influences are considered a risk of bias. Is there actually any study clearly showing that papers with authors from industry are more biased than papers from authors from academia? I know that this is an item on the SYRCLE list of bias tools, but is this prejudice against industry justified – i.e., based on evidence? Just recently there have been several cases of spectacular fraud (data fabrication) by authors in academia (behavioural ecology). Specifically for early career researchers in academia publication pressure for securing tenure or a faculty position is often described as extremely high, creating incentives for "beautifying" results which are then easier to publish. If you are, however, aware of any publication or data showing that bias in industry is more of an issue than in academia then it would be good to cite it here.

16) Line 388: As mentioned in the public review: I would be very careful with equating low reporting scores with "quality of research"

17) Analysis (Meta-analysis, Figure S12): It would be really interesting to add publication year as a covariate in the models. I would assume that estimates get closer to zero the younger the publication.

18) The list of references seems incomplete or incorrect with errors in text citation numbers. There are far more citations in the text (up to 82, whereas only 17 are on the list).

19) It would be good to know more about the training/qualification/validation criteria for the interviewers.

20) page 3 the sentence starting "we found that the…..rigorous manner" did not quite make sense to me.

21) I was curious as to why the group considered that testing of efficacy was the major use of such studies (rather than testing a new mechanism or fundamental finding) and that evaluation of a new method, model, or outcome measure was not even mentioned. That would not have been my view had I been interviewed. It would be nice to see a critical discussion about this.

22) I wonder if a more forensic discussion of what the authors see as the probable barriers to funders, peer scientists, and other stakeholders would be informative. Perhaps even a more extensive discussion of potential solutions may have been informative.

23) The main area of disagreement between interviewees was the issue of harmonisation of protocols- a strong contrast to the clinical multi-centre practice where harmonisation is the norm. This interesting response could perhaps have been discussed more?

Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., Brown, E. R., Budiman, N. I., Cairo, A. H., Capaldi, C. A., Chartier, C. R., Chung, J. M., Cicero, D. C., Coleman, J. A., Conway, J. G., … Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68-82. https://doi.org/10.1016/j.jesp.2015.10.012

Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., … Nosek, B. A. (2014). Investigating Variation in Replicability: A "Many Labs" Replication Project. Social Psychology, 45(3), 142-152. https://doi.org/10.1027/1864-9335/a000178

Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M. J., Berry, D. R., Bialobrzeska, O., Binan, E. D., Bocian, K., Brandt, M. J., Busching, R., … Nosek, B. A. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443-490. https://doi.org/10.1177/2515245918810225

Byers-Heinlein, Tsui, A. S. M., K., Bergmann, C., …, Wermelinger, S. (2021). A multi-lab study of bilingual infants: Exploring the preference for infant-directed speech. Advances in Methods and Practices in Psychological Science. PsyArXiv PrePrint

Byers-Heinlein, K., Tsui, R. K. Y., van Renswoude, D., Black, A. K., Barr, R., Brown, A., … Singh, L. (2020). The development of gaze following in monolingual and bilingual infants: A multi-lab study Infancy, Advance online publication. PsyArXiv Preprint

ManyBabies Consortium (2020). Quantifying sources of variability in infancy research using infant-directed speech preference. Advances in Methods and Practices in Psychological Science, 3, 24-52. PsyArXiv Preprint

Frank, M. C., Bergelson, E., Bergmann, C., Cristia, A., Floccia, C., Gervain, J., Hamlin, J. K., Hannon, E. E., Kline, M., Levelt, C., Lew-Williams, C., Nazzi, T., Panneton, R., Rabagliati, H., Soderstrom, M., Sullivan, J., Waxman, S., Yurovsky, D. (2017). A collaborative approach to infant research: Promoting reproducibility, best practices, and theory-building. Infancy, 22, 421-435. PsyArXiv Preprint

Reviewer #1:

This paper provides two important benefits to the field. First is the clear data showing the risk of bias assessments and effect size differences between single and multi-lab studies. This provides more clear evidence about the problematic process of single study science. Second, and I believe more important, is the qualitative information provided through the interviews with participating labs that describe the barriers and recommendations from participating labs (eg timelines, pilot periods, budget limitations set by the least well-resourced lab, ethics approval processes, centralized data processing, and even interpersonal lab coordination suggestions). Summarizing these qualitative findings in specific recommendations or a set of proposed guidelines would expand the relevance of this paper.

Below are my recommendations for improvement, which are not substantial but will hopefully help the authors widen their audience and convince their peers that these methods are not limited to their own community.

Introduction: I believe that your audience reach would be expanded, and your intended audience would be even more interested in your work, by noting how this method of multi-lab collaborations is appropriate beyond preclinical research. I will not suggest changing your inclusion criteria for the study already conducted, but it would be worth noting how efforts in social psychology and human development have seen similar results as those you describe. Likewise, existing networks in psychology (https://psysciacc.org/), development biology (https://manybabies.github.io/), and special education research (https://edresearchaccelerator.org/) would be worth mentioning as models for emulation when discussing how these practices could be scaled to more communities within preclinical research (perhaps mention this in the Conclusion).

Line 123: Can you please define "interventional" and clarify your definition of "Preclinical" so as to make it clear to readers outside of your domain of focus why psychological or developmental biology was not included (as these do have human health implications). (Just to be clear, I am not being sarcastic or suggesting that these should have been included, I just know that these terms get blurry when going from the preclinical to the basic sciences).

I appreciate that the calculations for the effect size ratio and the difference of standardized mean difference were specified a priori. Even though they were straightforward calculations, there is room for divergence or selective reporting and it appears that those analyses were reported as pre-specified.

Reviewer #2:

This paper reports results of a systematic review comparing single-lab with multi-lab studies with regard to reported effect size and risk of bias. The main outcome was that risk of bias was lower in multi-lab studies and so was the reported effect size. The authors argue, that the higher effect sizes observed in the single lab studies are presumably over-estimates and that, hence, the multi-lab studies provide more accurate effect size estimates. They, thus, conclude that this meta-analysis demonstrates the potential value of multi-centre studies in pre-clinical research. While I do agree that multi-centre studies are valuable, I have certain reservations as to what extent this is actually shown in the current study. Here are my concerns.

1) The sample size is rather limited. While I do not doubt that the authors invested a lot of work into finding multi-centre studies fulfilling their inclusion criteria, they ended up with a list of only 16 studies. Multi-centre studies are tedious to perform and more expensive, so it is not surprising that the authors could not find more. However, given an N=16 for a rather diverse collection of studies, spanning 35 years, different model species (mice, rats, dogs, pigs, and rabbits) and investigating rather different interventions, this might be better regarded as a preliminary analysis.

2) There is one question that cannot be answered but which is nevertheless rather important. Is the difference in effect size estimates (with the multi-lab studies presumably providing more accurate effect size estimates) due to the multi-lab setting itself or due to the fact that researchers willing to go the extra length of conducting a multi-lab study are stricter and more rigorous (and more aware) than the average researcher? If the latter is the case, then this would be a confounding factor and one could not conclude that the observed difference is due to the multi-lab approach itself.

3) As the authors noted themselves, sample sizes of single-lab and multi-lab differed substantially (median 19 vs. 111) and this difference in sample size might itself lead to more accurate effect size estimates (closer to the meta-analytic estimate). I am, therefore, missing an analysis, of whether this effect alone could explain the discrepancy. That is-would single-lab studies using the same number of animals be equally accurate as a multi-lab study?

4) I have some minor methodological issues regarding the evaluation of the risk of bias and specifically its interpretation as a measure for the quality of scientific rigor. This is more of a general critique of the use of 'risk of bias' in systematic reviews and is not specific to this study, but it nevertheless means that the results of this part of the study should be taken with a grain of salt and be interpreted with care. The risk of bias questionnaire is focusing on what is reported in the publication. As it is visible in table 3, even for the multi-lab studies – where the risk of bias was considerably lower than for the single-lab studies-a substantial proportion of questions was answered with unclear – i.e., it was, for example, not reported whether allocation concealment was done or not. This could mean that it was not done or it can mean that it was done, but researchers didn't find it noteworthy to report it. Reporting practice slightly differs between fields, it differs between journals (due to space restriction, editorial guidelines, and also the rigor of the editors), and it very strongly changes over time – due to the current discussion about replicability editors and reviewers are now asking for more details than they did a decade ago. So, any analysis of the risk of bias using those criteria should consider journal and year of publication as co-variates. In the present case the authors mentioned, that they tried to match single-lab studies with multi-lab studies also according to age, though a quick look at publication years suggests that this was not always achieved.

Overall, I think this study provides evidence that multi-centric preclinical studies do report effect size estimates that might be more trustworthy than those provided by single-lab studies. I can therefore fully agree with the authors' conclusion about the value of such studies. Yet, why exactly multi-center studies fare better (whether it is because of the larger sample sizes, the different mindset of the experimenters, or the necessity to coordinate protocols), remains an open question.

Reviewer #3:

This is a well-written and clear manuscript covering an important and emerging area, albeit one that has not yet significantly penetrated into scientific practice. The authors were attempting to ascertain professional attitudes to multi-centre studies in pre-clinical biomedical research by using a semi-structured interview method. I am not an expert in the methods used and defer to expert reviewers in this regard; my expertise is in multi-centre animal studies.

A potential limitation is that only 16 participants were included in the interviews, although this is a sample size justified by reference to norms in the literature. Furthermore, in the results, the authors suggest that data saturation had been reached by the 13th interview, so perhaps this is sufficient. Ten out of 16 interviewees were US-based, the remainder European. A further potential bias may be that the interviewees had all participated in pre-clinical multicentre research, so were perhaps likely to be enthusiasts of the approach.

I was curious as to why the group considered that testing of efficacy was the major use of such studies (rather than testing a new mechanism or fundamental finding) and that evaluation of a new method, model, or outcome measure was not even mentioned. That would not have been my view had I been interviewed. It would be nice to see a critical discussion about this.

I wonder if a more forensic discussion of what the authors see as the probable barriers to funders, peer scientists, and other stakeholders would be informative. Perhaps even a more extensive discussion of potential solutions may have been informative.

The main area of disagreement between interviewees was the issue of harmonisation of protocols – a strong contrast to the clinical multi-centre practice where harmonisation is the norm. This interesting response could perhaps have been discussed more?

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "A systematic assessment of preclinical multicenter studies and a comparison to single-laboratory studies" for further consideration by eLife. Your revised article has been evaluated by Mone Zaidi (Senior Editor) and a Reviewing Editor.

The manuscript has been improved but there are some remaining issues that need to be addressed, as outlined below:

Reviewer #1 (Recommendations for the authors):

The authors have sufficiently addressed the comments I made in the first submission.

However, one of their additions in response to another reviewer's comments may not be sufficient. The question is "Is the difference in effect size estimates (with the multi-lab studies presumably providing more accurate effect size estimates) due to the multi-lab setting itself or due to the fact that researchers are willing to go the extra length of conducting a multi-lab study are stricter and more rigorous (and more aware) than the average researcher?" I agree that the cause and effect cannot be addressed by this review and that only an experimental manipulation could start to truly answer that question (which would be a very tough experiment). However, the authors' response is also a bit limited and does add some additional questions from me. They state: "For ten multilaboratory studies, researchers had authored at least one of the matched single lab studies." I am now curious as to whether or not those ~10 single lab studies were in some way different than the ~90 other single lab studies (which would suggest that it's the person instead of the process that makes the difference). HOWEVER- I do not suggest that the authors go down that rabbit hole. No matter what is found from such digging, the root question ("is it the multi-lab process or the better scientist that makes the better studies?") will not be well addressed through this process simply because it will be an exercise in chasing smaller and smaller sub-samples with too much noise to make any reasonable conclusion. Instead, I have to disagree with the authors' first response to this comment and rather just recommend that they note these two possible explanations and state that this observational study cannot determine if it is the method or the authors that explain the differences reported.

All other changes and comments seem appropriate to me.

Reviewer #2 (Recommendations for the authors):

Overall the authors answered the questions coming up during the first review and changed the text accordingly.

There is, however, one point, where I would love to see a bit more. This regards the rather central question of whether multi-lab (ML) and single-lab (SL) differ because of the procedure or because they were done by a different 'population' of authors or with a different objective in mind.

The authors responded:

"We found that for the majority of the multilaboratory studies (i.e. ten), researchers had previously authored single lab studies on the same topic. " and then " This has now been addressed in the manuscript. Results: "For ten multilaboratory studies, researchers had authored at least one of the matched single lab studies." "

I would ask the authors, not only to mention in the results that for 10 ML studies, but researchers had also authored SL studies but also to describe how those 10 pairs compared. I don't advocate a statistical analysis, however, a descriptive summary (in how many of those 10 cases were the ML closer to the meta-analytic outcome than the SL?) and maybe these 10 SL studies could also be highlighted in figure 3 (if they were included there, though I presume they were).

Essential revisions:

1) Introduction: I believe that your audience reach would be expanded, and your intended audience would be even more interested in your work, by noting how this method of multi-lab collaborations is appropriate beyond preclinical research. I will not suggest changing your inclusion criteria for the study already conducted, but it would be worth noting how efforts in social psychology* and human development** have seen similar results as those you describe. Likewise, existing networks in psychology (https://psysciacc.org/), development biology (https://manybabies.github.io/), and special education research (https://edresearchaccelerator.org/) would be worth mentioning as models for emulation when discussing how these practices could be scaled to more communities within preclinical research (perhaps mention this in the Conclusion).

These are excellent points we have now addressed in our introduction:

“…This approach has been adopted in other fields such social and developmental psychology (12, 13).”

In addition, we have mentioned the influential networks in our Discussion. We hope this will allow interested readers to seek out additional information easily.

“…. As research groups globally consider adopting this approach, the biomedical community may benefit by emulating successful existing networks of multicenter studies in social psychology (https://psysciacc.org/), developmental psychology (https://manybabies.github.io) and special education research (https://edresearchaccelerator.org/).”

2) Line 123: Can you please define "interventional" and clarify your definition of "Preclinical" so as to make it clear to readers outside of your domain of focus why psychological or developmental biology was not included (as these do have human health implications). (Just to be clear, I am not being sarcastic or suggesting that these should have been included, I just know that these terms get blurry when going from the preclinical to the basic sciences).

We agree these terms have various meanings depending on field of study. They are often a source of debate even with our closest collaborators as there are no consensus definitions. We have now clarified our intentions with these terms in the first paragraph of the methods:

Preclinical multilaboratory eligibility criteria

Population – The population of interest was preclinical, interventional, multilaboratory, controlled comparison studies. Preclinical was defined as research conducted using nonhuman models that involves the evaluation of potential therapeutic interventions of relevance to human health…

Intervention, comparators, outcomes – Interventions were restricted to agents with potential effects when considering human health.

3) I appreciate that the calculations for the effect size ratio and the difference of standardized mean difference were specified a priori. Even though they were straightforward calculations, there is room for divergence or selective reporting and it appears that those analyses were reported as pre-specified.

Thank-you for providing this supportive comment.

4) The sample size is rather limited. While I do not doubt that the authors invested a lot of work into finding multi-centre studies fulfilling their inclusion criteria, they ended up with a list of only 16 studies. Multi-centre studies are tedious to perform and more expensive, so it is not surprising that the authors could not find more. However, given an N=16 for a rather diverse collection of studies, spanning 35 years, different model species (mice, rats, dogs, pigs, and rabbits) and investigating rather different interventions, this might be better regarded as a preliminary analysis.

We would point out that, although the publication dates of included studies spanned 35 years, the majority have been published since 2015. In addition, despite the diversity of studies noted by reviewers, the overall trends are similar across studies (more complete reporting, lower risk of bias, and much smaller effect sizes than those seen in single-laboratory studies). Finally, we would note that our comparative approach emulates similar landmark clinical meta-research studies; the first highly cited landmark study on this issue in clinical research investigated only five topics (Ref #9, Bellomo 2009). Nonetheless, we agree with the reviewers that we have much to learn still and that our study is an important starting point. In our discussion we have now explicitly stated this:

“Strengths and limitations

… In addition, our quantitative analysis included only sixteen studies, and thus our results might be better regarded as a preliminary analysis that will require future confirmation when more multilaboratory studies have been conducted. We would note, however, that despite the diversity of included multilaboratory studies, overall trends were quite similar across these sixteen studies (e.g. more complete reporting, lower risk of bias, and smaller effect size than comparable single-laboratory studies).”

5) There is one question that cannot be answered but which is nevertheless rather important. Is the difference in effect size estimates (with the multi-lab studies presumably providing more accurate effect size estimates) due to the multi-lab setting itself or due to the fact that researchers willing to go the extra length of conducting a multi-lab study are stricter and more rigorous (and more aware) than the average researcher? If the latter is the case, then this would be a confounding factor and one could not conclude that the observed difference is due to the multi-lab approach itself.

In order to address this interesting question raised by the reviewers we compared authorship of the multilaboratory studies to matched single lab studies. We found that for the majority of the multilaboratory studies (i.e. ten), researchers had previously authored single lab studies on the same topic. This suggests that the multilaboratory approach itself is perhaps more rigorous.

Of interest, we would speculate that may in part due to the shared mental models that may develop in the multilaboratory approach. Of note, this is actually the topic of an ongoing funded study by our group (published protocol: DOI: 10.1371/journal.pone.0273077).

This has now been addressed in the manuscript.

Results:

“For ten multilaboratory studies, researchers had authored at least one of the matched single lab studies.”

6) As the authors noted themselves, sample sizes of single-lab and multi-lab differed substantially (median 19 vs. 111) and this difference in sample size might itself lead to more accurate effect size estimates (closer to the meta-analytic estimate). I am, therefore, missing an analysis, of whether this effect alone could explain the discrepancy. That is – would single-lab studies using the same number of animals be equally accurate as a multi-lab study?

We agree that larger sample size would lead to a more accurate effect size estimate, with smaller confidence intervals. However, as shown in Figure S12, the majority of the multi-laboratory studies effect sizes fall outside the upper limits of the 95% confidence interval of the overall single-lab studies effect size. Simply increasing the sample size would only increase precision of the estimates, but we would still anticipate the effects would fall into the confidence interval 95% of the time.

This indicates that the true value of the overall single-lab effect size does not overlap with the true effect size of multi-laboratory studies in a majority of cases. We have now mentioned this issue in the Results:

“…A scatterplot of the study effect sizes for each comparison is presented in Figure 3; and the forest plots of each of the 14 comparisons can be found in the supplemental information (S12 Figure). Of note, 8 of the of the 14 multilaboraotry studies had 95% confidence intervals that fell outside of the pooled single-laboratory 95% confidence intervals.”

7) I have some minor methodological issues regarding the evaluation of the risk of bias and specifically its interpretation as a measure for the quality of scientific rigor. This is more of a general critique of the use of 'risk of bias' in systematic reviews and is not specific to this study, but it nevertheless means that the results of this part of the study should be taken with a grain of salt and be interpreted with care. The risk of bias questionnaire is focusing on what is reported in the publication. As it is visible in table 3, even for the multi-lab studies-where risk of bias was considerably lower than for the single-lab studies-a substantial proportion of questions was answered with unclear-i.e. it was, for example, not reported whether allocation concealment was done or not. This could mean that it was not done or it can mean that it was done, but researchers didn't find it noteworthy to report it. Reporting practice slightly differs between fields, it differs between journals (due to space restriction, editorial guidelines, and also the rigor of the editors), and it very strongly changes over time – due to the current discussion about replicability editors and reviewers are now asking for more details than they did a decade ago. So, any analysis of the risk of bias using those criteria should consider journal and year of publication as co-variates. In the present case the authors mentioned, that they tried to match single-lab studies with multi-lab studies also according to age, though a quick look at publication years suggests that this was not always achieved.

This is a valid point raised by the reviewers. We have now acknowledged this in the Discussion:

“…Another limitation is that our assessment of risk of bias relies on complete reporting; reporting, however, can be influenced by space restrictions in some journals, temporal trends (e.g. better reporting in more recent studies), as well as accepted norms in certain fields of basic science. However, with the increasing prevalence of reporting checklists and standards in preclinical research, (62) future assessments will be less susceptible to this information bias (63).”

8) There seem to be some inconsistencies between Table 3 and Table S11. For example, in the column "Blinding of outcome assessment" in Table S3 there are 4 studies with "unknown" (Crabbe, Alam, Spoerke, Gill), while in Table S11 in the column "Blinding of outcome assessment" there are 4 study scoring 0, but here Arroyo-Araujo scores also 0, while Gill scores 1. How does this fit together?

Thank you for pointing this out – the inconsistencies were noted and have been corrected. Studies Gill and Arroyo-Araujo both blinded the outcome assessment, thus, in Table 3 the “unknown” was changed to “low” for Gill, 2016 under "Blinding of outcome assessment"; and in Table S10 the “0” was changed to “1” for Arroyo-Araujo, 2019 under "Blinding of outcome assessment". Additionally, the Results section has been adjusted accordingly – changed from “Twelve studies” to “Thirteen studies”.

9) Methods: maybe I have missed that: have you described, whether for the reporting scores you checked only the main article or also the electronic supplementary (supporting information)?

Indeed, we did consult the supporting information in addition to the main article for evaluating the reporting scores. This has now been clarified in the methods:

“For both assessments, the main articles along with the supporting information (when provided) were consulted.”

10) Language limits: In the methods description you wrote "No language limits were imposed" (line 147). In the Results section, I could, however, not find any information, on if any non-English papers were found and whether they were included in the analysis. There is also no mention of what you would have done if you encountered a paper in a language you were not capable of. When would this have been an exclusion criterion? In any case, in the methods section where you wrote "no language limits were imposed", please add the information that all search terms were in English. (Or were they also translated into other languages?)

We did not impose any language limits in the search and no non-English articles were found. For other meta-research projects we have typically had the article translated by members in our research institute familiar with the language.

We have added the following to clarify in the Results:

“There were no non-English articles identified in the search.”

11) Line 166: Primary and secondary research outcomes. According to my own experience, one sees many papers, where the authors report outcomes of several analyses and do not clearly define what the primary and the secondary outcomes are. In fact, I only rarely read that an author writes "this is our primary outcome" and "this was a secondary outcome". Did you encounter such cases and which criteria did you use to decide what counted as a primary and secondary outcome?

This is an important point and reflects differences in approach and language used by lab researchers (versus clinical trialists, for instance). Thus, we have now further clarified in the Methods:

“Qualitative data included… all study outcomes (primary, secondary, or undefined) …” and “Quantitative data included… sample sizes for the outcome used in the meta-analysis and for the control group.”

The heading in Table 2 was changed from “Primary Outcome” to “Study Outcomes”.

12) Line 196: Rapid review method. Please describe in one or two sentences what that is.

Thank you for this suggestion. We added a more detailed description to the Methods:

“…, which consisted of the search of a single database, and by having a single reviewer screen, extract, and appraise studies while an additional reviewer auditing all information.”

13) Line 259: were the effect sizes just pooled or were they weighted by study size or variance?

Pooled effect sizes (i.e. for single lab studies) were weighted (inverse variance).

For the effect size ratio calculation, there was no weighting of the two data points used (i.e. multilab effect size, pooled single lab effect size). We recognized a priori the potential drawbacks of this, and this is why several measures to compare effect sizes of multilaboratory vs. single lab studies were prespecified and reported.

14) Line 267: I think giving a ratio (without multiplication by 100) might be better

Point taken. We have modified the effect size ratio calculation to be a simple ratio without multiplying by 100%. The has been adjusted in the methods, results, and supporting information Table S11.

15) Line 345: potentially industry-related influences are considered a risk of bias. Is there actually any study clearly showing that papers with authors from industry are more biased than papers from authors from academia? I know that this is an item on the SYRCLE list of bias tools, but is this prejudice against industry justified – i.e., based on evidence? Just recently there have been several cases of spectacular fraud (data fabrication) by authors in academia (behavioural ecology). Specifically for early career researchers in academia publication pressure for securing tenure or a faculty position is often described as extremely high, creating incentives for "beautifying" results which are then easier to publish. If you are, however, aware of any publication or data showing that bias in industry is more of an issue than in academia then it would be good to cite it here.

This is an important point. Dr. Lisa Bero and colleagues have investigated this issue in several studies and demonstrated that the relationship is complicated. For studies investigating statins on atherosclerosis and bone outcomes they found that non-industry studies were more likely to demonstrate efficacy than industry sponsored studies (i.e. larger effect size in non-industry studies; PMID 25880564 and 24465178). However, in animal studies assessing the effect of atrazine exposure, industry sponsored studies were less likely to demonstrate harm than non-industry sponsored studies (i.e. smaller effect size in non-industry studies; PMID 26694022). In our group’s work (not-published yet) we found that industry sponsored animal studies of hydroxyethyl starch intravenous fluids were more likely to demonstrate efficacy than non-industry sponsored studies (i.e. smaller effect size in non-industry studies). This is notable since these (very expensive) fluids were subsequently pulled from markets with a black-box warning from the FDA.

These data demonstrate that there is no straightforward/easy approach to understanding the relationship between industry/non-industry sponsorship and risk of bias in preclinical studies.

We agree that perhaps the SYRCLE risk of bias tool should reevaluate this issue in the future and we have brought it to their attention through an email communication with Dr. Kimberly Wever, who is a member of their leadership. Ultimately, this remains a question that needs to be investigated further and addressed by future iterations of the SYRCLE risk of bias tool.

16) Line 388: As mentioned in the public review: I would be very careful with equating low reporting scores with "quality of research"

We agree and have adjusted our terminology throughout (i.e. “completeness of reporting”).

17) Analysis (Meta-analysis, Figure S12): It would be really interesting to add publication year as a covariate in the models. I would assume that estimates get closer to zero the younger the publication.

In order to address this question we have explored the relationship between absolute effect size vs. year of publication (all single lab studies plotted in Author response image 1). There is no clear relationship between the two variables (r² = 0.002).

Author response image 1

Download asset Open asset

18) The list of references seems incomplete or incorrect with errors in text citation numbers. There are far more citations in the text (up to 82, whereas only 17 are on the list).

Thanking for pointing this out. There was an error in the continuation of the references/citations for the interview study that was appended. This has now been corrected.

19) It would be good to know more about the training/qualification/validation criteria for the interviewers.

This is an important issue in qualitative research and reflects on reflexivity. As such, it is a mandatory aspect of the COREQ reporting guideline we used (found in the appendix for that paper).

20) Page 3 the sentence starting "we found that the…..rigorous manner" did not quite make sense to me.

Thank you for pointing this out in the appended interview study. We have adjusted the sentence to, “We found that the multicenter studies undertaken to date have been conducted using more rigorous methodology and quality control”.

21) I was curious as to why the group considered that testing of efficacy was the major use of such studies (rather than testing a new mechanism or fundamental finding) and that evaluation of a new method, model, or outcome measure was not even mentioned. That would not have been my view had I been interviewed. It would be nice to see a critical discussion about this.

Indeed, we agree that multilaboratory studies could be used for other purposes (e.g. mechanistic/fundamental). However, the majority of studies using this method have been interventional/therapeutic. We have now discussed this in the Discussion:

Discussion:

“…The application of rigorous inclusion criteria limited the eligible studies to interventional, controlled-comparison studies, which could omit valuable information that may have come from the excluded studies of non-controlled and/or observational designs, or studies focused on mechanistic insights.”

22) I wonder if a more forensic discussion of what the authors see as the probable barriers to funders, peer scientists, and other stakeholders would be informative. Perhaps even a more extensive discussion of potential solutions may have been informative.

As we had only interviewed the scientists involved, and not other stakeholders, we are hesitant to speculate on solutions from their viewpoints. Although we are hesitant to further lengthen our discussion (which we feel is already quite long), we have now included a statement that further studies will need to investigate this important issue. We believe a stakeholder response should be:

Discussion:

“…Moreover, identified barriers and enablers to these studies should be further explored from a variety of stakeholder perspectives (e.g. researchers, animal ethics committees, institutes and funders) in order to maximize future chances of success.”

23) The main area of disagreement between interviewees was the issue of harmonisation of protocols- a strong contrast to the clinical multi-centre practice where harmonisation is the norm. This interesting response could perhaps have been discussed more?

This is an excellent point and we have now elaborated on this issue in the Results:

“If the protocol was to be harmonized, then it had to be adapted to fit each lab’s budget accordingly (i.e. the lab with the smallest budget set the spending limit (19)). Alternatively, labs developed a general protocol but adapted it to fit their own respective budget with what resources they had. Of note, recent work has suggested that harmonization reduces between-lab variability, however, systematic heterogenization did not reduce variability further (52); this may suggest that, even in fully harmonized protocols, enough uncontrolled heterogeneity exists that further purposeful heterogenization has little effect.”

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Reviewer #2 (Recommendations for the authors):

Overall the authors answered the questions coming up during the first review and changed the text accordingly.

There is, however, one point, where I would love to see a bit more. This regards the rather central question of whether multi-lab (ML) and single-lab (SL) differ because of the procedure or because they were done by a different 'population' of authors or with a different objective in mind.

The authors responded:

"We found that for the majority of the multilaboratory studies (i.e. ten), researchers had previously authored single lab studies on the same topic. " and then " This has now been addressed in the manuscript. Results: "For ten multilaboratory studies, researchers had authored at least one of the matched single lab studies." "

I would ask the authors, not only to mention in the results that for 10 ML studies, but researchers had also authored SL studies but also to describe how those 10 pairs compared. I don't advocate a statistical analysis, however, a descriptive summary (in how many of those 10 cases were the ML closer to the meta-analytic outcome than the SL?) and maybe these 10 SL studies could also be highlighted in figure 3 (if they were included there, though I presume they were).

The following text has been added:

Methods:

“Peer reviewers requested additional analysis to qualitatively assess identified single-laboratory studies that were conducted by the same authors of matched multilaboratory studies.”

Results:

“The ESR was greater than 1 (i.e. the single lab studies produced a larger summary effect size) in 10 of the 14 comparisons. The median effect size ratio between multilaboratory and single lab studies across all 14 comparisons was 4.18 (range: 0.57 to 17.14). The ESRs for each comparison along with the mean effect sizes and ratio of the mean effect sizes is found in the supplemental information (Appendix 11 – table 7).

For 10 multilaboratory studies, researchers also had authored 11 matched single lab studies. Median effect size ratio of this smaller matched cohort was 2.50 (mean 4.10, range 0.35-17.63). Median quality assessment score for the 10 multilaboratory studies was 3 (range 1-5); median quality score of the 11 single-laboratory studies matched by authors was 1 (range 1-4).”

Discussion:

“The difference in observed methodological quality and high completeness of reporting in preclinical multilaboratory studies could be explained by the routine oversight and quality control that were employed by some of the multilaboratory studies included in our sample. Though not all multilaboratory studies reported routine oversight, we expect that this is inherent of collaborative studies between multiple independent research groups. As reported by several studies, the coordination of a successful preclinical multilaboratory study requires greater training, standardization of protocols, and study-level management when compared to a preclinical study within a single-laboratory. Another barrier was the issue of obtaining adequate funding for a multilaboratory study. As consequence of limited funding, we would speculate that these studies may have undergone more scrutiny and refinement by multiple investigators, funders, and other stakeholders Indeed, comparison single-laboratory studies that had been conducted by authors of multilaboratory studies suggested differences in the conduct and outcomes of these studies (despite having the some of the same researchers involved in both). However, this post hoc analysis was qualitative with a limited sample; thus, future studies will need to explore these issues further.”

We chose not to highlight the studies in Figure 3 to avoid distracting from our a priori planned analysis.

Author details

1. Victoria T Hunniford

   1. Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada
   2. Telfer School of Management, University of Ottawa, Ottawa, Canada

   Contribution

   Conceptualization, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

2. Agnes Grudniewicz

   Telfer School of Management, University of Ottawa, Ottawa, Canada

   Contribution

   Supervision, Writing – review and editing

   Competing interests

   No competing interests declared

3. Dean A Fergusson

   1. Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada
   2. Faculty of Medicine, University of Ottawa, Ottawa, Canada
   3. Department of Surgery, University of Ottawa, Ottawa, Canada
   4. School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada

   Contribution

   Conceptualization, Supervision, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

4. Joshua Montroy

   Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada

   Contribution

   Formal analysis, Writing – review and editing

   Competing interests

   No competing interests declared

5. Emma Grigor

   1. Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada
   2. Faculty of Medicine, University of Ottawa, Ottawa, Canada

   Contribution

   Investigation, Writing – review and editing

   Competing interests

   No competing interests declared

6. Casey Lansdell

   Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada

   Contribution

   Investigation, Writing – review and editing

   Competing interests

   No competing interests declared

7. Manoj M Lalu

   1. Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Ottawa, Canada
   2. School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada
   3. Department of Anesthesiology and Pain Medicine, The Ottawa Hospital, University of Ottawa, Ottawa, Canada
   4. Regenerative Medicine Program, The Ottawa Hospital Research Institute, Ottawa, Canada
   5. Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, Canada

   Contribution

   Conceptualization, Resources, Supervision, Methodology, Writing – review and editing

   For correspondence

   mlalu@toh.ca

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0322-382X

• 330

  Page views

• 49

  Downloads

• 0

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.