Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies

ELTE Eotvos Lorand University, Hungary
Karolinska Institutet, Sweden
Stockholm University, Sweden
Tilburg University, Netherlands
University of Groningen, Netherlands
Utrecht University, Netherlands
University Medical Center Groningen, University of Groningen, Netherlands
Friesland Mental Health Care Services, Netherlands
University of California Los Angeles, United States
National Bureau of Economic Research, United States
University of Amsterdam, Netherlands
Dartmouth College, United States
University of Münster, Germany
University of Rennes, CNRS, Inria and Inserm, France
McLean Hospital, United States
Harvard Medical School, United States
University of Missouri, United States
National University of Singapore, Singapore
University of New South Wales, Australia
Stockholm School of Economics, Sweden
University of Innsbruck, Austria
University Hospital Basel, Switzerland
Monash University, Australia
University of Wisconsin-Madison, United States
Ludwig-Maximilians-University, Germany
Maastricht University, Netherlands
Princeton University, United States
University of Victoria, Canada
Université Paris-Saclay, France
Neurospin, CEA, France
Amsterdam University, Netherlands
University of Bristol, United Kingdom
Center for Open Science, United States
University of Virginia, United States
Stanford University, United States
University of Basel, Switzerland
Tel Aviv University, Israel
ESMT Berlin, Germany
University College London, United Kingdom
University of Sussex, United Kingdom
University of Illinois, United States
University of Alberta, Canada
Lund University, United States
University of Massachusetts Amherst, United States
INSEAD, Singapore

Nov 9, 2021

https://doi.org/10.7554/eLife.72185

Open access
Copyright information

Download
Cite
CommentOpen annotations (there are currently 0 annotations on this page).
Share

1 figure, 2 tables and 3 additional files

Figures

Figure 1

Download asset Open asset

Analysis choices and alternative plausible paths.

The analysis of a large dataset can involve a sequence of analysis choices, as depicted in these schematic diagrams. The analyst first must decide between two options at the start of the analysis (top), and must make three additional decisions during the analysis: this leads to 16 possible paths for the analysis (grey lines). The left panel shows an example in which all possible paths lead to the same conclusion; the right panel shows an example in which some paths lead to conclusion A and other paths lead to conclusion B. Unless we can test alternative paths, we cannot know if the results obtained by following one particular path (thick black line) are robust, or if other plausible paths would lead to different results.

Tables

Table 1

Recommended practices for the main stages of the multi-analyst method.

Stage	Recommended practices
Recruiting co-analysts	1. Determine a minimum target number of co-analysts and outline clear eligibility criteria before recruiting co-analysts. We recommend that the final report justifies why these choices are adequate to achieve the study goals. 2. When recruiting co-analysts, inform them about (a) their tasks and responsibilities; (b) the project code of conduct (e.g., confidentiality/ non-disclosure agreements); (c) the plans for publishing the research report and presenting the data, analyses, and conclusion; (d) the conditions for an analysis to be included or excluded from the study; (e) whether their names will be publicly linked to the analyses; (f) the co-analysts’ rights to update or revise their analyses; (g) the project time schedule; and (h) the nature and criteria of compensation (e.g., authorship).
Providing datasets, research questions, and research tasks	3. Provide the datasets accompanied with a codebook that contains a comprehensive explanation of the variables and the datafile structure. 4. Ensure that co-analysts understand any restrictions on the use of the data, including issues of ethics, privacy, confidentiality, or ownership. 5. Provide the research questions (and potential theoretically derived hypotheses that should be tested) without communicating the lead team’s preferred analysis choices or expectations about the conclusions.
Conducting the independent analyses	6. To ensure independence, we recommend that co-analysts should not communicate with each other about their analyses until after all initial reports have been submitted. In general, it should be clearly explained why and at what stage co-analysts are allowed to communicate about the analyses (e.g., to detect errors or call attention to outlying data points).
Processing the results	7. Require co-analysts to share with the lead team their results, the analysis code with explanatory comments (or a detailed description of their point-and-click analyses), their conclusions, and an explanation of how their conclusions follow from their results. 8. The lead team makes the commented code, results, and conclusions of all non-withdrawn analyses publicly available before or at the same time as submitting the research report.
Reporting the methods and results	9. The lead team should report the multi-analyst process of the study, including (a) the justification for the number of co-analysts; (b) the eligibility criteria and recruitment of co-analysts; (c) how co-analysts were given the data sets and research questions; (d) how the independence of analyses was ensured; (e) the numbers of and reasons for withdrawals and omissions of analyses; (f) whether the lead team conducted an independent analysis; (g) how the results were processed; (h) the summary of the results of co-analysts; (i) and the limitations and potential biases of the study. 10. Data management should follow the FAIR principles (Wilkinson et al., 2016), and the research report should be transparent about access to the data and code for all analyses (Aczel et al., 2020).