1. Medicine
  2. Neuroscience
Download icon

Science Forum: Consensus-based guidance for conducting and reporting multi-analyst studies

  1. Balazs Aczel  Is a corresponding author
  2. Barnabas Szaszi  Is a corresponding author
  3. Gustav Nilsonne
  4. Olmo R van den Akker
  5. Casper J Albers
  6. Marcel ALM van Assen
  7. Jojanneke A Bastiaansen
  8. Daniel Benjamin
  9. Udo Boehm
  10. Rotem Botvinik-Nezer
  11. Laura F Bringmann
  12. Niko A Busch
  13. Emmanuel Caruyer
  14. Andrea M Cataldo
  15. Nelson Cowan
  16. Andrew Delios
  17. Noah NN van Dongen
  18. Chris Donkin
  19. Johnny B van Doorn
  20. Anna Dreber
  21. Gilles Dutilh
  22. Gary F Egan
  23. Morton Ann Gernsbacher
  24. Rink Hoekstra
  25. Sabine Hoffmann
  26. Felix Holzmeister
  27. Juergen Huber
  28. Magnus Johannesson
  29. Kai J Jonas
  30. Alexander T Kindel
  31. Michael Kirchler
  32. Yoram K Kunkels
  33. D Stephen Lindsay
  34. Jean-Francois Mangin
  35. Dora Matzke
  36. Marcus R Munafò
  37. Ben R Newell
  38. Brian A Nosek
  39. Russell A Poldrack
  40. Don van Ravenzwaaij
  41. Jörg Rieskamp
  42. Matthew J Salganik
  43. Alexandra Sarafoglou
  44. Tom Schonberg
  45. Martin Schweinsberg
  46. David Shanks
  47. Raphael Silberzahn
  48. Daniel J Simons
  49. Barbara A Spellman
  50. Samuel St-Jean
  51. Jeffrey J Starns
  52. Eric Luis Uhlmann
  53. Jelte Wicherts
  54. Eric-Jan Wagenmakers
  1. ELTE Eotvos Lorand University, Hungary
  2. Karolinska Institutet, Sweden
  3. Stockholm University, Sweden
  4. Tilburg University, Netherlands
  5. University of Groningen, Netherlands
  6. Utrecht University, Netherlands
  7. University Medical Center Groningen, University of Groningen, Netherlands
  8. Friesland Mental Health Care Services, Netherlands
  9. University of California Los Angeles, United States
  10. National Bureau of Economic Research, United States
  11. University of Amsterdam, Netherlands
  12. Dartmouth College, United States
  13. University of Münster, Germany
  14. University of Rennes, CNRS, Inria and Inserm, France
  15. McLean Hospital, United States
  16. Harvard Medical School, United States
  17. University of Missouri, United States
  18. National University of Singapore, Singapore
  19. University of New South Wales, Australia
  20. Stockholm School of Economics, Sweden
  21. University of Innsbruck, Austria
  22. University Hospital Basel, Switzerland
  23. Monash University, Australia
  24. University of Wisconsin-Madison, United States
  25. Ludwig-Maximilians-University, Germany
  26. Maastricht University, Netherlands
  27. Princeton University, United States
  28. University of Victoria, Canada
  29. Université Paris-Saclay, France
  30. Neurospin, CEA, France
  31. Amsterdam University, Netherlands
  32. University of Bristol, United Kingdom
  33. Center for Open Science, United States
  34. University of Virginia, United States
  35. Stanford University, United States
  36. University of Basel, Switzerland
  37. Tel Aviv University, Israel
  38. ESMT Berlin, Germany
  39. University College London, United Kingdom
  40. University of Sussex, United Kingdom
  41. University of Illinois, United States
  42. University of Alberta, Canada
  43. Lund University, United States
  44. University of Massachusetts Amherst, United States
  45. INSEAD, Singapore
Feature Article
Cite this article as: eLife 2021;10:e72185 doi: 10.7554/eLife.72185
1 figure, 2 tables and 3 additional files

Figures

Analysis choices and alternative plausible paths.

The analysis of a large dataset can involve a sequence of analysis choices, as depicted in these schematic diagrams. The analyst first must decide between two options at the start of the analysis (top), and must make three additional decisions during the analysis: this leads to 16 possible paths for the analysis (grey lines). The left panel shows an example in which all possible paths lead to the same conclusion; the right panel shows an example in which some paths lead to conclusion A and other paths lead to conclusion B. Unless we can test alternative paths, we cannot know if the results obtained by following one particular path (thick black line) are robust, or if other plausible paths would lead to different results.

Tables

Table 1
Recommended practices for the main stages of the multi-analyst method.
StageRecommended practices
Recruiting co-analysts1. Determine a minimum target number of co-analysts and outline clear eligibility criteria before recruiting co-analysts. We recommend that the final report justifies why these choices are adequate to achieve the study goals.
2. When recruiting co-analysts, inform them about (a) their tasks and responsibilities; (b) the project code of conduct (e.g., confidentiality/ non-disclosure agreements); (c) the plans for publishing the research report and presenting the data, analyses, and conclusion; (d) the conditions for an analysis to be included or excluded from the study; (e) whether their names will be publicly linked to the analyses; (f) the co-analysts’ rights to update or revise their analyses; (g) the project time schedule; and (h) the nature and criteria of compensation (e.g., authorship).
Providing datasets, research questions, and research tasks3. Provide the datasets accompanied with a codebook that contains a comprehensive explanation of the variables and the datafile structure.
4. Ensure that co-analysts understand any restrictions on the use of the data, including issues of ethics, privacy, confidentiality, or ownership.
5. Provide the research questions (and potential theoretically derived hypotheses that should be tested) without communicating the lead team’s preferred analysis choices or expectations about the conclusions.
Conducting the independent analyses6. To ensure independence, we recommend that co-analysts should not communicate with each other about their analyses until after all initial reports have been submitted. In general, it should be clearly explained why and at what stage co-analysts are allowed to communicate about the analyses (e.g., to detect errors or call attention to outlying data points).
Processing the results7. Require co-analysts to share with the lead team their results, the analysis code with explanatory comments (or a detailed description of their point-and-click analyses), their conclusions, and an explanation of how their conclusions follow from their results.
8. The lead team makes the commented code, results, and conclusions of all non-withdrawn analyses publicly available before or at the same time as submitting the research report.
Reporting the methods and results9. The lead team should report the multi-analyst process of the study, including (a) the justification for the number of co-analysts; (b) the eligibility criteria and recruitment of co-analysts; (c) how co-analysts were given the data sets and research questions; (d) how the independence of analyses was ensured; (e) the numbers of and reasons for withdrawals and omissions of analyses; (f) whether the lead team conducted an independent analysis; (g) how the results were processed; (h) the summary of the results of co-analysts; (i) and the limitations and potential biases of the study.
10. Data management should follow the FAIR principles (Wilkinson et al., 2016), and the research report should be transparent about access to the data and code for all analyses (Aczel et al., 2020).
Author response table 1
Item1Item2Item3Item4Item5Item6Item7Item8Item9Item10
Median ratings8999998.5999
Interquar-tile range21101.251.251.2511

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)