Computational modeling of behavioral phenomena is currently experiencing rapid growth, in particular with respect to methodological improvements. For instance, seminal work by Wilson and Collins, 2019 has been crucial in raising methodological standards and bringing attention to how computational analyses can add value to the study of human (and animal) behavior. Meanwhile, in most instances, computational analyses are applied to data collected from experiments that were designed based on intuition and convention. That is, while experimental designs are usually motivated by scientific questions in mind, they are often chosen without explicitly and quantitatively considering how informative these data might be for the computational analyses and, finally, the scientific questions being studied. This can, in the worst case, completely undermine the research effort, especially as particularly valuable experimental designs can be counter-intuitive. Today, advancements in machine learning (ML) open up the possibility of applying computational methods when deciding how experiments should be designed in the first place, to yield data that are maximally informative with respect to the scientific question at hand.

In this work, we provide an introduction to modern BOED and a step-by-step tutorial on how these advancements can be combined and leveraged to find optimal experimental designs for any computational model that we can simulate data from, and show how by-products of this procedure allow for quick and straightforward evaluation of models and their parameters against real experimental data. Formalizing the scientific goal of an experiment such that our design aligns with our intention can be difficult. BOED forces the researcher to make explicit, and engage with, various assumptions and constraints that would otherwise remain implicit, which helps in making experimental research more rigorous but can also lead to unexpected optimal designs. We discuss several issues related to experimental design that practitioners may encounter no matter how they design their experiments. Importantly, these issues are made more salient by formalizing the process.

Experiments are the bedrock of scientific data collection, making it possible to discriminate between different theories or models, and discovering what specific commitments a model must make to align with reality by means of parameter estimation. The process of designing experiments involves making a set of complex decisions, where the space of possible experimental designs is typically navigated based on a combination of researchers’ prior experience, intuitions about what designs may be informative, and convention.

This approach has been successful for centuries across various scientific fields, but it has important limitations that are amplified as theories become richer, more nuanced, and more complex. Often, a theory does not make a concrete prediction but rather has free parameters that license a variety of possible predictions of varying plausibility. For example, if we want to model human behavior, it is important to recognize that different participants in a task might have different strategies or priorities, which can be captured by parameters in a model. As we expand models to accommodate more complex effects, it becomes ever more time-consuming and difficult to design experiments that distinguish between models, or allow for effective parameter estimation. As an example from cognitive science, there are instances where models turned out to be empirically indistinguishable under certain conditions, a fact that was only recognized after a large number of experiments had already been done (Jones and Dzhafarov, 2014). Such problems also play an important role in the larger context of the replication crisis (e.g. Pashler and Wagenmakers, 2012).

Collecting experimental data is typically a costly and time-consuming process, especially in the case of studies involving resource-intensive recording techniques, such as neuroimaging or eye-tracking (e.g. Dale, 1999; Lorenz et al., 2016). There is a strong economic as well as ethical case for conducting experiments that are expected to lead to maximally informative data with respect to the scientific question at hand. For instance, in computational psychiatry, researchers are often interested in estimating parameters describing people’s traits and how they relate, often doing so under constrained resources in terms of the number of participants and their time. Additionally, researchers and practitioners alike require that experiments are sufficiently informative to provide confidence that they would reveal effects if they are there and also that negative results are true negatives (e.g. Huys et al., 2013; Karvelis et al., 2018). Consequently, as theories of natural phenomena become more realistic and complex, researchers must expand their methodological toolkit for designing scientific experiments.

As a potential solution, Bayesian optimal experimental design (BOED; see Ryan et al., 2016; Rainforth et al., 2023, for reviews) provides a principled framework for optimizing the design of experiments. Broadly speaking, BOED rephrases the task of finding optimal experimental designs to that of solving an optimization problem. That is, the researcher specifies all controllable parameters of an experiment, also known as experimental designs, for which they wish to find optimal settings, which are then determined by maximizing a utility function. Exactly what we might want to optimize depends on our experiment, but can include any aspect of the design that we can specify, such as which stimuli to present, when or where measurements should be taken in a quasi-experiment, or, as an example, how to reward risky choices in a behavioral experiment. The utility function is selected to measure the quality of a given experimental design with respect to the scientific goal at hand, such as model discrimination, parameter estimation or prediction of future observations, with typical choices including expected information gain, uncertainty reduction, and many others (Ryan et al., 2016), as explained below. Importantly, BOED approaches require that theories are formalized via computational models of the underlying natural phenomena.

Scientific theories are increasingly formalized via computational models (e.g. Guest and Martin, 2021). These models take many forms, but one desideratum for a model is that, given a set of experimental data, we can compare it to other models based on how well it predicts that data. To that end, many computational models are constructed in a way that allows for the analytic evaluation of the likelihood of the model given data, by means of a likelihood function. However, this imposes strong constraints on the space of models we can consider. Models that are rich enough to accurately describe complex phenomena—like human behavior—often have intractable likelihood functions (e.g. Lintusaari et al., 2017; Cranmer et al., 2020). In other words, we cannot compute likelihoods due to the prohibitive computational cost of doing so, or the inability to express the likelihood function mathematically. This has at least two implications (1) When we develop rich models, we often lack the tools to evaluate and compare them directly, and are left with coarse, qualitative proxy measures–like the “the model curves look like the human curves for some parameter settings!”; (2) when we are intent on conducting a systematic comparison of models, we are often forced to use simplified models that have tractable likelihoods, even when the simplifications lead to consequential departures from the theories we might want our models to capture.

In this tutorial, we present a flexible workflow that combines recent advances in ML and BOED in order to optimize experimental designs for any computational model from which we can simulate data. A high-level overview of the underlying ML-based BOED method is shown in Figure 1, including researchers’ inputs and resulting outputs. The presented approach is applicable to any model from which we can simulate data and scales well to realistic numbers of design variables, experimental trials and blocks. Furthermore, as well as optimized designs, the presented approach also results in tractable amortized posterior inference—that is, it allows researchers to use their actual experimental data to easily compute posterior distributions, which may otherwise be computationally expensive or intractable to compute. Moreover, we note that, to our knowledge, there has been no study that performs BOED for simulator models without likelihoods and then uses the optimal designs to run a real experiment. Instead, most practical applications of BOED have been limited to simple scientific theories of nature (e.g. Liepe et al., 2013). This paucity of real-world applications is partly because the situations where BOED would be most useful are those where it has previously been computationally intractable to use. The ML-based approach presented in this work aims to overcome these limitations.

Figure 1

Download asset Open asset

As with many new computational tools, however, the techniques for using BOED are not trivial to implement from scratch. In an effort to help other scientists understand these methods and use them to unlock the full potential of BOED in their own research, we next present a case study in using BOED to better understand human decision-making. Specifically, we (1) describe new simulator models that generalize earlier decision-making models that were constrained by the need to have tractable likelihoods; (2) walk through the steps to use BOED to design experiments comparing these simulator models and estimating the psychologically interpretable parameters they contain; and (3) contrast our new, optimized experiments to previous designs and illustrate their advantages and the new lessons they offer. All our steps and analyses are accompanied by code that we have commented and structured to be easily adapted by other researchers.

As a case study application, we consider the question of how people balance their pursuit of short-term reward (‘exploitation’) against learning how to maximize reward in the longer term (‘exploration’). This question has been studied at length in psychology, neuroscience, and computer science, and one of the key frameworks for investigating it is the multi-armed bandit decision-making task (e.g. Steyvers et al., 2009; Dezfouli et al., 2019; Schulz et al., 2020; Gershman, 2018). Multi-armed bandits have a long history in statistics and machine learning (e.g. Robbins, 1952; Sutton and Barto, 2018) and formalize a general class of sequential decision problems—repeatedly choosing between a set of options under uncertainty with the goal of maximizing cumulative reward (for more background, see Box 1).

Box 1

Bandit tasks.

Bandit tasks constitute a minimal reinforcement learning task, in which the agent is faced with the problem of balancing exploration and exploitation. Various algorithms have been proposed for maximizing the reward in bandit tasks, and some of these algorithms have been used directly or in modified form as models of human behavior. A selection of such algorithms indeed captures important psychological mechanisms in solving bandit tasks (Steyvers et al., 2009). However, a richer account of human behavior in bandit tasks would seem to be required to accommodate flexibility and nuance of human behavior (e.g. see Dezfouli et al., 2019; Lee et al., 2011), leading to simulator models that do not necessarily permit familiar likelihood-based inference, due to, for example unobserved latent states, and which complicate the task of intuiting informative experiments. See Box 1—figure 1 for a schematic of the data-generating process of multi-armed bandit tasks.

Box 1—figure 1

Download asset Open asset

We demonstrate the applicability of our method by optimizing the design of a multi-armed bandit decision-making task, considering three flexible extensions of previously proposed models of human behavior. In our experiments, we consider two general scientific tasks: model discrimination and parameter estimation. We then evaluate the optimal designs found using our method empirically with simulations and real human behavioral data collected from online experiments, and compare them to designs commonly used in the literature. We find that, as compared to commonly used designs, our optimal experimental designs yield significantly better model recovery, more informative posterior distributions and improved model parameter disentanglement.

In the following sections, we describe and walk through our workflow step-by-step, using the aforementioned multi-armed bandit task setting as a concrete case study to further illustrate ML-based BOED. At a high level, our workflow comprises the following steps:

1. Defining a scientific goal, for example, which model, among a set of models, best describes a natural phenomenon.

2. Formalizing a theory, or theories, as a computational model(s) that can be sampled from.

3. Setting up the design optimization problem and deciding which aspects of the experimental design need to be optimized.

4. Constructing the required ML models and training them with simulated data.

5. Validating the obtained optimal designs in silico and, potentially, re-evaluating design and modeling choices.

6. After confirming and validating the optimal design, the real experiment can be performed. The trained ML models used for BOED can be used afterwards to easily compute posterior distributions.

We provide a more detailed description of the above workflow in the following sections. The outputs of the ML-based BOED method are optimal experimental designs, an estimate of the (maximum) expected information gain and an amortized posterior distribution, which allows for straightforward computation of posterior distributions from collections of real-world data and saves us a potentially computationally expensive (likelihood-free) inference step. Readers may find that many of these steps should be part of the design of experiments no matter whether using BOED or not. However, BOED here provides the additional advantage of making all of these steps, which are sometimes carried out heuristically, explicit and formal.

The first step in setting up the experimental design optimization is to define our scientific goal. Although there may be many different goals for an experiment, they often fall into the broad categories of model discrimination or parameter estimation.

After defining our scientific goal, we need to ensure that our scientific theories are, or can be, formalized as computational models. This either requires us to formalize them from the ground up, or leverage existing computational models from literature that support our theories. In the former case, this entails making all aspects of the theory explicit, which is therefore a highly useful and established practice; we refer the reader to Wilson and Collins, 2019 for a more general guidance on building computational models. For the presented ML-based BOED approach, we require that our theories are cast as computational models that allow us to simulate data, that is simulator models. Further we need to decide on a general experimental paradigm that needs to be able to interact with our computational models.

Having formalized our scientific theories as simulator models and decided on our scientific goals, we need to think about which parts of the experiment we want to optimize. In principle, any controllable aspect of the experimental apparatus could be tuned as part of the experimental design optimization. Concretely, in order to solve an experimental design problem by means of BOED, our experimental designs ${\displaystyle \mathbf{d}}$ need to be formalized as part of the simulator model ${\displaystyle p(\mathbf{y}|\mathit{\theta },\mathbf{d})}$. Instead of optimizing all controllable aspects of an experiment, it may then help to leverage domain knowledge and prior work to choose a sufficient experimental design set, in order to reduce the dimensionality of the design problem.

In our case study, for instance, we have chosen the reward probabilities of different bandit arms to be the experimental designs to optimize over. This choice posits that certain reward environments yield more informative data than others, in the context of model discrimination and parameter estimation. Our task is then to find the optimal reward environment that yields maximally informative data.

There may be other experimental choices that we will have to make, in particular to limit variability in the observed data, to control for confounding effects, and to reduce non-stationarity, among many other reasons. In our multi-armed bandit setting, for example, we may choose to limit the number of trials in the experiment to avoid participant fatigue, which is usually not accounted for in computational models of behavioral phenomena. Additionally, we may limit the number of available choices in a discrete choice task or impose constraints on any other aspects suggested by domain knowledge, or where we want to ensure comparability with prior research. For example, we may have reason to believe that certain designs are more ecologically valid than others—we discuss such considerations in the Caveats Section.

As previously explained, MI shown in Equation 1 is an effective means of measuring the value (i.e. the utility) of an experimental design setting ${\displaystyle {\displaystyle \mathbf{d}}}$ in BOED (Paninski, 2005; Ryan et al., 2016). Unfortunately, it is generally an expensive, or intractable, quantity to compute, especially for simulator models. The reason for this is twofold: Firstly, the MI is defined via a high-dimensional integral. Secondly, the MI requires density evaluations of either the posterior distribution or the marginal likelihood, both of which are expensive to compute for general statistical models, and intractable for simulator models. Recent advances in BOED for simulator models thus advocate the use of cheaper MI lower bounds instead (e.g. Foster et al., 2019; Kleinegesse et al., 2020, Kleinegesse and Gutmann, 2021; Ivanova et al., 2021).

Many recent innovations in BOED rely on ML to accurately and efficiently estimate functions. ML is currently revolutionizing large parts of the natural sciences, with applications ranging from understanding the spread of viruses (Currie et al., 2020) to discovering new molecules (Jumper et al., 2021), forecasting climate change (Runge et al., 2019), and developing new theories of human decision-making (Peterson et al., 2021). Due to its versatility, ML can be integrated effectively into a wide range of scientific workflows, facilitating theory building, data modeling and analysis. In particular, ML allows us to automatically discover patterns in high-dimensional data sets. Many recent state-of-the-art methods in natural science, therefore, deeply integrate ML into their data analysis pipelines (e.g. Blei and Smyth, 2017). However, there has been little focus on applying ML to improve the data collection process, which ultimately determines the quality of data, and thus the efficacy of downstream analyses and inferences. In this work, we use ML methods as part of our optimization scheme, where they serve as flexible function approximators.

The step of training a good ML model currently requires some previous expertise with (or willingness to learn about) applying ML methods. Fortunately, this step is less experiment-specific than it may first appear. Many different experiments generate similar data, and the architecture we provide can deal with multiple blocks and automatically learns summary statistics, as we explain above. Many excellent resources on the basics of ML exist, and a detailed treatment is beyond the scope of this tutorial. However, we provide pointers to several aspects in the code repository: https://github.com/simonvalentin/boed-tutorial (copy archived at Valentin, 2023).

Approximating mutual information with machine learning

In this work, we leverage the MINEBED method (Kleinegesse et al., 2020), which allows us to efficiently estimate the MI using ML. Specifically, for a particular design ${\displaystyle {\displaystyle \mathbf{d}}}$, we first simulate data from the computational models under consideration, (shown in Box Box 2). The approach works by constructing a lower bound on the MI that is parameterized via an ML model. In our case study, we use a neural network with an architecture customized for behavioral experiments (shown in Box 4), but any other ML model would work in place. We then train the ML model by using the simulated data as input and the lower bound as an objective function, which gradually tightens the lower bound.

Box 4

Bespoke neural network architecture for behavioral experiments.

Box 4—figure 1

Download asset Open asset

More concretely, this method works by training a neural network ${\displaystyle T_{\mathit{\psi }}(\mathit{v},\mathbf{y})}$, or any other ML model, where ${\displaystyle \mathit{\psi }}$ are the neural network parameters and the data ${\displaystyle {\displaystyle \mathbf{y}}}$ is simulated at design ${\displaystyle {\displaystyle \mathbf{d}}}$ with samples from the prior $p(\mathit{v})$. We train this neural network by maximizing the following objective function, which is a lower bound of the MI,

(2) $U(\mathbf{d};\mathit{\psi })={\mathbb{E}}_{p(\mathbf{y}|\mathit{v},\mathbf{d})p(\mathit{v})}[T_{\mathit{\psi }}(\mathit{v},\mathbf{y})]-{\mathbb{E}}_{p(\mathbf{y}|\mathbf{d})p(\mathit{v})}\left[e^{T_{\mathit{\psi }}(\mathit{v},\mathbf{y})-1}\right]$

The above lower bound is also known as the NWJ lower bound and has several appealing bias-variance properties (Poole et al., 2019; we note that alternative bounds on MI can be used in the place of the NWJ bound as our loss function, as discussed by Kleinegesse and Gutmann, 2021). The expectations in Equation 2 are usually approximated using (Monte-Carlo) sample-averages.

Once we have trained the neural network ${\displaystyle T_{\mathit{\psi }}(\mathit{v},\mathbf{y})}$ on ${\displaystyle \mathit{\psi }}$, we estimate the MI at ${\displaystyle {\displaystyle \mathbf{d}}}$ by computing Equation 2 using a held-out test set of data simulated from the computational model(s).

Once we have converged in our search over experimental designs, we can simulate the behavior of our theories under our optimal experimental design(s). Irrespective of the final experiment that will be run, model simulations can already provide useful information for theory building, for example by noticing that the simulator models generate implausible data for certain experimental designs. This means that we can use simulations to assess whether we can recover our models, or individual model parameters, given the experimental design, as we did in our simulation study. We highlight this again as a critical step, since issues at this stage warrant revising the models under consideration and, importantly, any corresponding real-world experiment would be expected to yield uninformative data and waste resources (Wilson and Collins, 2019).

As we discuss in the Caveats Section, BOED may sometimes lead to unusual yet effective experimental designs, and we suggest that surprising optimal designs may deserve careful consideration, as they may reveal subtle differences in predicted behaviors or model misspecifications. One appeal of optimal designs is that some of the ways of obtaining information about which models is correct correspond to finding situations in which one of them makes extreme predictions. BOED can thereby be seen as facilitating good prior predictive checking. We have illustrated this step using our case study in Box 5 and Box 6, where we present simulation study results for model discrimination and parameter estimation, respectively.

Box 5

Model discrimination.

Simulation study Our method reveals that the optimal reward probabilities for the model discrimination task are, approximately, ${\displaystyle [0,0,0.6]}$ for one block of trials and ${\displaystyle [1,1,0]}$ for the other block. These optimal designs stand in stark contrast to the usual reward probabilities used in such behavioral experiments, which characteristically use less extreme values (Steyvers et al., 2009). Contrary to our initial intuitions, and, presumably those of previous experiment designers, the extreme probabilities in these optimal designs are effective at distinguishing between particular theoretical commitments of our different models. For instance, the AEG model (see Box 2) can break ties between options with equivalent (observed) reward rates via the ‘stickiness’ parameter. To illustrate this effect, consider the bandit with ${\displaystyle [1,1,0]}$ reward probabilities, as found above. Here, switching between the two options that always produce a reward is different from switching to the option that never produces a reward. Specifically, switching between the winning options can be seen as ‘anti-stickiness’, motivated for example, out of boredom or epistemic curiosity about subtle differences in the true reward rates of the winning options. This type of strategy is less apparent when the bandit arms are stochastic. As such, these mechanisms can be investigated most effectively by not introducing additional variability due to stochastic rewards and instead isolating their effects. Hence, we can interpret the, perhaps counter-intuitive, optimal experimental designs as effective choices for isolating distinctive mechanisms of our behavioral models or their parameters under the assumed priors.

Using the neural network that was trained at that optimal design, we can cheaply compute approximate posterior distributions of the model indicator (as described in Materials and methods). This yields the confusion matrices in Box 5—figure 1, which show that our optimal designs result in considerably better model recovery than the baseline designs.

Box 5—figure 1

Download asset Open asset

Box 6

Parameter estimation: Simulation study.

We next discuss the parameter estimation results, focusing on the WSLTS model; the results for the AEG and GLS model can be found in Appendix 5. We find that the optimal reward probabilities for the WSLTS model are [0, 1, 0], [0, 1, 1] and [1, 0, 1] for the three blocks. Similar to the model discrimination task, these optimal designs take extreme values, unlike commonly used reward probabilities in the literature. In Box 6—figure 1 we show posterior distributions of the WSLTS model parameters for optimal and baseline designs. We find that our optimal designs yield data that result in considerably improved parameter recovery, as compared to baseline designs.

Box 6—figure 1

Download asset Open asset

Once we are satisfied with the found optimal experimental design, we can run the actual experiment. Here, the ML models we used to approximate MI allow us to cheaply perform posterior inference. After running the real-world experiment, we can use our trained ML models to cheaply compute approximate posterior distributions over models and their parameters. This procedure provides a valuable addition to a Bayesian modeling workflow Schad et al., 2021; Wilson and Collins, 2019, as computing posterior distributions is a computationally expensive step, which is especially difficult for models with intractable likelihoods. We explain theoretical details below, and showcase an analysis of a real experiment with our case study in Box 7, Box 8, Box 9.

Box 8

Human participant study: Results.

We here briefly present the results of our human participant study. For the model discrimination tasks, we find that our optimal designs yield posterior distributions that tend to have lower Shannon entropy than the baseline designs, as shown in Box 8—figure 1. Our ML-based approach thus allows us to obtain more informative beliefs about which model matches real individual human behavior best, as compared to baselines.

Having determined the best model to explain a given participant’s behavior, we turn to the parameter estimation task, focusing our efforts on estimating the model parameters that explain their specific strategy. Similar to the model discrimination task, we here assess the quality of joint posterior distributions using entropy, where smaller values directly correspond to smaller uncertainty. Box 8—figure 2 shows the distribution of differential posterior entropies across participants that were assigned to the AEG simulator model; we show corresponding plots for the WSLTS and GLS models in Appendix 5. Similar to the model discrimination task, we find that our optimal designs provide more informative data, that is they result in joint posterior distributions that tend to have lower differential entropy, as compared to the ones resulting from baseline designs, as shown in the figure.

Moreover, the proportions of participants allocated to the WSLTS, AEG, and GLS models for the optimal design were 62(37%), 75(45%) and 29(18%), respectively. For the baseline designs, the proportions were 57(36%), 22(14%) and 81(51%), for the WSLTS, AEG, and GLS models, respectively. The finding that the largest proportion of participants for the optimal designs were best described by the AEG model differs from prior work, where most participants were best described by Win-Stay Lose-Shift (a special case of our WSLTS model) Steyvers et al., 2009. This suggests that a reinterpretation of previous results is required, where the human tendency to stick with previous choices is less about myopically repeating the last successful choice, and more about balancing reward maximization, exploration, and a drive to be consistent over recent runs of choices. Interestingly, we find different distributions over models for the optimal as compared to the baseline designs (A chi-square test revealed a significant difference between the distributions of WSLTS, AEG, and GLS in the Optimal and Baseline groups, ${\displaystyle {\chi }^2(2)=53.66}$, p < 0.0001.). In particular, while, descriptively, the proportions of participants allocated to the WSLTS model are very close (37%, and 36% for the optimal and baseline designs, respectively), the proportions allocated to the AEG and GLS model differ more substantially. This may suggest that people’s strategy depends on the experimental design, potentially with some participants favoring strategies similar to the AEG model with more reliable reward environments, and the GLS in more unreliable (baseline) environments. The models we have considered in our case study do not account for the possibility of transferring knowledge across experimental blocks. However, learning about the reward probabilities in one (or several) experimental blocks may influence people’s expectations about the reward distribution of future block(s). For instance, having inferred (correctly or not), that at least one arm bandit arm always pays a reward, a learner would do well to quickly identify this arm in a future block (see, e.g. Wang et al., 2018, for a related line of research). Our space of computational models does not include such strategy switching mechanisms, and we discuss the impact of such potential misspecifications below in more detail. While exploring these findings in detail from a cognitive science perspective is beyond the scope of the present work, we view this as a fruitful avenue for future work.

The extreme designs here may be surprising, considering the typically used stochastic rewards in bandit problems. However, these designs allow the experiment to focus on particular aspects of people’s behavior, which turns out to be most informative for distinguishing between the computational models we consider. For instance, while our AEG model performs uniformly random exploration, the WSLTS model performs uncertainty-guided exploration. Meanwhile, our optimal designs raise the question of ecological validity, as it is unclear which kinds of ‘reward environments’ are more reflective of real-world situations. On the one hand, research in causal reasoning suggests that people expect relationships to be deterministic and reliable (e.g. Schulz and Sommerville, 2006). On the other hand, some environments are known to be random and more akin to a classic, stochastic bandit problem, such as the stock market. We view the question of the ecological validity of stochastic rewards as an important direction for future work, and discuss this more below in the Caveats Section.

Box 8—figure 1

Download asset Open asset

Box 8—figure 2

Download asset Open asset

Summary statistics

Additionally, as another component of our approach, we also obtain automatically-learned (approximate) sufficient summary statistics for data observed with the optimal experimental designs. Summary statistics are often crafted manually by domain experts, which tends to be a time-consuming and difficult task. In fact, hand-crafted summary statistics are becoming less effective and informative as our models naturally become more complex as science advances (Chen et al., 2021). Our learned summary statistics could be used for various downstream tasks, for example as an input for approximate inference techniques (such as ABC) or as means to interpret and analyze the data more effectively, for example as a lower dimensional representation used to visualize the data.

Box 9

Exploring model parameter disentanglement.

In addition to measuring the efficacy of our optimal experiments via the differential entropy of posterior distributions, we can ask how well they isolate, or disentangle, the effects of individual model parameters. Specifically, we measure this by means of the (Pearson) correlation between model parameters in the joint posterior distribution, which we present in Box 9—figure 1 for all simulator models and for both optimal designs and baseline designs. Here, a correlation of ${\displaystyle r=0}$ means that there is no (linear) relationship between the respective pair of inferred model parameters across individuals, thereby providing evidence that the data under this design allow us to separate the natural mechanisms corresponding to those parameters. On the other hand, correlations closer to ${\displaystyle r=-1}$ or ${\displaystyle r=1}$ would imply that the model parameters essentially encode the same mechanism. As shown descriptively in Box 9—figure 1, our optimal designs are indeed able to isolate the mechanisms of individual model parameters more effectively than baseline designs. Note that these are not confusion matrices, as displayed in Box 5—figure 1, but rather average correlation matrices of the model parameter posterior distributions.

To assess this observation quantitatively, we first ${\displaystyle z}$-transform (Fisher, 1915) the correlation coefficients (i.e. all entries below the main diagonal in the correlation matrices) and compute the average absolute ${\displaystyle z}$-score over the model parameters within each participant. The resulting average of absolute ${\displaystyle z}$-scores is an expression for the average magnitude of linear dependency between the posterior model parameters for each participant. Next, we assess whether the average absolute ${\displaystyle z}$-scores are significantly different between those participants allocated to our optimal designs and those allocated to the baseline designs. We compute this statistical significance by conducting two-sided Welch’s ${\displaystyle t}$-tests (Welch, 1947). We find that our optimal designs significantly outperform baseline designs for all three models, that is for the WSLTS model we find a p-value of p < 0.001 (${\displaystyle t(60.12)=13.89}$), for the AEG model we have p = 0.004 (${\displaystyle t(27.30)=3.12}$) and for the GLS model we have p < 0.001 (${\displaystyle t(91.01)=8.44}$). This provides additional evidence that our optimal designs are better able to disentangle the effects of individual model parameters than baseline designs. As explained earlier, we argue that this isolation of effects is facilitated by the extreme values of our optimal designs.

Box 9—figure 1

Download asset Open asset

We now turn to more high-level considerations and discuss several categories of potential issues and caveats related to BOED. We first discuss how the optimality of an experimental design is always defined with respect to a specific goal. Second, we consider how posterior inferences always depend on the experimental design and how this impacts BOED considerations. Third, we bring attention to the important issue of model misspecification in the context of BOED. Fourth, we discuss the topic of ecological validity and generalizability of our inferences with BOED. Finally, we cover the interpretability of our optimal designs. While we discuss these issues in light of BOED, they are just as important with hand-crafted experiments. Importantly, however, BOED makes these issues more salient, by turning the design of experiments into a more explicit process. Therefore, these points should be relevant to any researcher collecting or analyzing experimental data using computational models.

This tutorial provides a flexible step-by-step workflow to finding optimal experimental designs, by leveraging recent advances in ML and BOED. As a first step, the proposed workflow suggests to define a scientific goal, such as model discrimination or parameter estimation, thereby expressing the value of a particular design as a measurable quantity using a utility function. Secondly, the scientific theories under investigation need to be cast as a computational model from which we can sample synthetic data. The methodologies discussed in this tutorial make no assumption, however, on whether or not the likelihood function of the computational model needs be tractable, which opens up the space of scientific theories that can be tested considerably. In the third step, the design optimization problem needs to be set up, which includes deciding which experimental design variables need to be optimized and whether there are any additional constraints that should be included. The fourth step of our proposed workflow then involves using ML to estimate and optimize the utility function. In doing so, we discussed a recent approach that leverages neural networks to learn a mapping between experimental designs and expected information gain, that is MI, and provided a novel extension that efficiently deals with behavioral data. In the fifth step, the discovered optimal designs are then validated in silico using synthetic data. This ensures that the optimal designs found are sound and that no experimental resources are wasted, possibly also if it is found that the computational models need to be revisited after this step. Lastly, in the sixth step, the real experiment can be performed using the discovered optimal design. Using the discussed methodologies, the trained neural networks can then conveniently be used to compute posterior distributions and summary statistics, immediately after the data collection.

This tutorial and the accompanying case study demonstrate the usefulness of modern ML and BOED methods for improving the way in which we design experiments and thus collect empirical data. Furthermore, the methodologies discussed in this tutorial optimize experimental designs for models of cognition without requiring computable likelihoods, or marginal likelihoods. This is critical to provide the methodological support required for scientific theories and models of human behavior that become increasingly realistic and complex. As part of our case study, using simulations and real-world data, we showed that the proposed workflow and the discussed methodologies yield optimal designs that outperform human-crafted designs found in the literature, offering more informative data for model discrimination as well as parameter estimation. We showed that adopting more powerful methodological tools allows us to study more realistic theories of human behavior, and our results provide empirical support for the expressiveness of the simulator models we studied in our case study. BOED is currently an active area of research and future work will likely make some steps easier and more computationally efficient, such as automatic tuning of the ML methods used to estimate MI, variations of the loss function, improved ways of searching over the design space, new ways of dealing with model misspecification or computationally efficient sequential BOED (Rainforth et al., 2023). What will remain despite such technical advancements, however, is the need to formulate sound scientific questions and actively engaging with tools that help to automate designs. We thus believe that the steps and considerations outlined in the workflow presented in this work will stay relevant. More broadly, ML has seen success in behavioral research regarding the analysis of large datasets and discovering new theories with promising results (e.g. Peterson et al., 2021) but its potential for data collection and in particular experimental design has been explored remarkably little. We view the present work as an important step in this direction.

1. 1. Baker M

   (2016) 1,500 scientists lift the lid on reproducibility

   Nature 533:452–454.

   https://doi.org/10.1038/533452a

   • Google Scholar
2. 1. Bitzer S
   2. Park H
   3. Blankenburg F
   4. Kiebel SJ

   (2014) Perceptual decision making: drift-diffusion model is equivalent to a Bayesian model

   Frontiers in Human Neuroscience 8:102.

   https://doi.org/10.3389/fnhum.2014.00102

   • PubMed
   • Google Scholar
3. 1. Blei DM
   2. Smyth P

   (2017) Science and data science

   PNAS 114:8689–8692.

   https://doi.org/10.1073/pnas.1702076114

   • Google Scholar
4. 1. Bramley NR
   2. Gerstenberg T
   3. Tenenbaum JB
   4. Gureckis TM

   (2018) Intuitive experimentation in the physical world

   Cognitive Psychology 105:9–38.

   https://doi.org/10.1016/j.cogpsych.2018.05.001

   • PubMed
   • Google Scholar
5. 1. Brehmer J
   2. Louppe G
   3. Pavez J
   4. Cranmer K

   (2020) Mining gold from implicit models to improve likelihood-free inference

   PNAS 117:5242–5249.

   https://doi.org/10.1073/pnas.1915980117

   • Google Scholar
6. 1. Breiman L

   (2001) Random forests

   Machine Learning 45:5–32.

   https://doi.org/10.1023/A:1010933404324

   • Google Scholar
7. Conference

   1. Chen Y
   2. Zhang D
   3. Gutmann MU
   4. Courville A
   5. Zhu Z

   (2021)

   Neural approximate sufficient statistics for implicit models

   In International Conference on Learning Representations (ICLR.

   • Google Scholar
8. 1. Clark HH

   (1973) The language-as-fixed-effect fallacy: A critique of language statistics in psychological research

   Journal of Verbal Learning and Verbal Behavior 12:335–359.

   https://doi.org/10.1016/S0022-5371(73)80014-3

   • Google Scholar
9. 1. Corander J
   2. Fraser C
   3. Gutmann MU
   4. Arnold B
   5. Hanage WP
   6. Bentley SD
   7. Lipsitch M
   8. Croucher NJ

   (2017) Frequency-dependent selection in vaccine-associated pneumococcal population dynamics

   Nature Ecology & Evolution 1:1950–1960.

   https://doi.org/10.1038/s41559-017-0337-x

   • PubMed
   • Google Scholar
10. 1. Cranmer K
    2. Brehmer J
    3. Louppe G

    (2020) The frontier of simulation-based inference

    PNAS 117:30055–30062.

    https://doi.org/10.1073/pnas.1912789117

    • PubMed
    • Google Scholar
11. 1. Currie CSM
    2. Fowler JW
    3. Kotiadis K
    4. Monks T
    5. Onggo BS
    6. Robertson DA
    7. Tako AA

    (2020) How simulation modelling can help reduce the impact of COVID-19

    Journal of Simulation 14:83–97.

    https://doi.org/10.1080/17477778.2020.1751570

    • Google Scholar
12. 1. Dale AM

    (1999) Optimal experimental design for event-related fMRI

    Human Brain Mapping 8:109–114.

    https://doi.org/10.1002/(SICI)1097-0193(1999)8:2/3<109::AID-HBM7>3.0.CO;2-W

    • PubMed
    • Google Scholar
13. 1. Dezfouli A
    2. Griffiths K
    3. Ramos F
    4. Dayan P
    5. Balleine BW

    (2019) Models that learn how humans learn: The case of decision-making and its disorders

    PLOS Computational Biology 15:e1006903.

    https://doi.org/10.1371/journal.pcbi.1006903

    • PubMed
    • Google Scholar
14. 1. Dziak JJ
    2. Coffman DL
    3. Lanza ST
    4. Li R
    5. Jermiin LS

    (2020) Sensitivity and specificity of information criteria

    Briefings in Bioinformatics 21:553–565.

    https://doi.org/10.1093/bib/bbz016

    • Google Scholar
15. 1. Eduardo A
    2. Gutmann MU

    (2023)

    Bayesian optimization with informative covariance

    Transactions on Machine Learning Research pp. 1–37.

    • Google Scholar
16. 1. Elsken T
    2. Metzen JH
    3. Hutter F

    (2019) Neural architecture search: A survey

    Journal of Machine Learning Research 20:1–21.

    https://doi.org/10.1007/978-3-030-05318-5

    • Google Scholar
17. 1. Fisher RA

    (1915) Frequency distribution of the values of the correlation coeffients in samples from an indefinitely large popu;ation

    Biometrika 10:507–521.

    https://doi.org/10.1093/biomet/10.4.507

    • Google Scholar
18. Conference

    1. Foster A
    2. Jankowiak M
    3. Bingham E
    4. Horsfall P
    5. Teh YW
    6. Rainforth T
    7. Goodman N

    (2019)

    Variational Bayesian optimal experimental design

    Advances in Neural Information Processing Systems.

    • Google Scholar
19. 1. Friedman JH

    (2001) Greedy function approximation: A gradient boosting machine

    The Annals of Statistics 29:1232.

    https://doi.org/10.1214/aos/1013203451

    • Google Scholar
20. 1. Gebhardt C
    2. Oulasvirta A
    3. Hilliges O

    (2021) Hierarchical reinforcement learning explains task interleaving behavior

    Computational Brain & Behavior 4:284–304.

    https://doi.org/10.1007/s42113-020-00093-9

    • Google Scholar
21. 1. Gershman SJ

    (2018) Deconstructing the human algorithms for exploration

    Cognition 173:34–42.

    https://doi.org/10.1016/j.cognition.2017.12.014

    • PubMed
    • Google Scholar
22. 1. Gershman SJ

    (2020) Origin of perseveration in the trade-off between reward and complexity

    Cognition 204:104394.

    https://doi.org/10.1016/j.cognition.2020.104394

    • PubMed
    • Google Scholar
23. 1. Gouriéroux C
    2. Phillips PCB
    3. Yu J

    (2010) Indirect inference for dynamic panel models

    Journal of Econometrics 157:68–77.

    https://doi.org/10.1016/j.jeconom.2009.10.024

    • Google Scholar
24. 1. Griffiths TL

    (2015) Manifesto for a new (computational) cognitive revolution

    Cognition 135:21–23.

    https://doi.org/10.1016/j.cognition.2014.11.026

    • PubMed
    • Google Scholar
25. 1. Guest O
    2. Martin AE

    (2021) How computational modeling can force theory building in psychological science

    Perspectives on Psychological Science 16:789–802.

    https://doi.org/10.1177/1745691620970585

    • PubMed
    • Google Scholar
26. 1. Huys QJ
    2. Pizzagalli DA
    3. Bogdan R
    4. Dayan P

    (2013) Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis

    Biology of Mood & Anxiety Disorders 3:12.

    https://doi.org/10.1186/2045-5380-3-12

    • PubMed
    • Google Scholar
27. 1. Ioannidis JPA

    (2005) Why most published research findings are false

    PLOS Medicine 2:e124.

    https://doi.org/10.1371/journal.pmed.0020124

    • PubMed
    • Google Scholar
28. Conference

    1. Ivanova DR
    2. Foster A
    3. Kleinegesse S
    4. Gutmann MU
    5. Rainforth T

    (2021)

    Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods

    35th Conference on Neural Information Processing Systems.

    • Google Scholar
29. 1. Jones M
    2. Dzhafarov EN

    (2014) Unfalsifiability and mutual translatability of major modeling schemes for choice reaction time

    Psychological Review 121:1–32.

    https://doi.org/10.1037/a0034190

    • PubMed
    • Google Scholar
30. 1. Jumper J
    2. Evans R
    3. Pritzel A
    4. Green T
    5. Figurnov M
    6. Ronneberger O
    7. Tunyasuvunakool K
    8. Bates R
    9. Žídek A
    10. Potapenko A
    11. Bridgland A
    12. Meyer C
    13. Kohl SAA
    14. Ballard AJ
    15. Cowie A
    16. Romera-Paredes B
    17. Nikolov S
    18. Jain R
    19. Adler J
    20. Back T
    21. Petersen S
    22. Reiman D
    23. Clancy E
    24. Zielinski M
    25. Steinegger M
    26. Pacholska M
    27. Berghammer T
    28. Bodenstein S
    29. Silver D
    30. Vinyals O
    31. Senior AW
    32. Kavukcuoglu K
    33. Kohli P
    34. Hassabis D

    (2021) Highly accurate protein structure prediction with AlphaFold

    Nature 596:583–589.

    https://doi.org/10.1038/s41586-021-03819-2

    • PubMed
    • Google Scholar
31. 1. Kangasrääsiö A
    2. Jokinen JPP
    3. Oulasvirta A
    4. Howes A
    5. Kaski S

    (2019) Parameter inference for computational cognitive models with approximate bayesian computation

    Cognitive Science 43:e12738.

    https://doi.org/10.1111/cogs.12738

    • PubMed
    • Google Scholar
32. 1. Karvelis P
    2. Seitz AR
    3. Lawrie SM
    4. Seriès P

    (2018) Autistic traits, but not schizotypy, predict increased weighting of sensory information in Bayesian visual integration

    eLife 7:e34115.

    https://doi.org/10.7554/eLife.34115

    • PubMed
    • Google Scholar
33. Conference

    1. Kleinegesse S
    2. Gutmann MU

    (2019)

    Efficient Bayesian experimental design for implicit models

    In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics.

    • Google Scholar
34. Preprint

    1. Kleinegesse S
    2. Drovandi C
    3. Gutmann MU

    (2020) Sequential bayesian experimental design for implicit models via mutual information

    arXiv.

    https://arxiv.org/abs/2003.09379

    • Google Scholar
35. Preprint

    1. Kleinegesse S
    2. Gutmann MU

    (2021) Gradient-based Bayesian experimental design for implicit models using mutual information lower bounds

    arXiv.

    https://arxiv.org/abs/2105.04379

    • Google Scholar
36. 1. LeCun Y
    2. Bengio Y
    3. Hinton G

    (2015) Deep learning

    Nature 521:436–444.

    https://doi.org/10.1038/nature14539

    • PubMed
    • Google Scholar
37. 1. Lee MD
    2. Zhang S
    3. Munro M
    4. Steyvers M

    (2011) Psychological models of human and optimal performance in bandit problems

    Cognitive Systems Research 12:164–174.

    https://doi.org/10.1016/j.cogsys.2010.07.007

    • Google Scholar
38. 1. Lieder F
    2. Griffiths TL

    (2017) Strategy selection as rational metareasoning

    Psychological Review 124:762–794.

    https://doi.org/10.1037/rev0000075

    • PubMed
    • Google Scholar
39. 1. Liepe J
    2. Filippi S
    3. Komorowski M
    4. Stumpf MPH

    (2013) Maximizing the information content of experiments in systems biology

    PLOS Computational Biology 9:e1002888.

    https://doi.org/10.1371/journal.pcbi.1002888

    • PubMed
    • Google Scholar
40. 1. Lindley DV

    (1956) On a Measure of the Information Provided by an Experiment

    The Annals of Mathematical Statistics 27:986–1005.

    https://doi.org/10.1214/aoms/1177728069

    • Google Scholar
41. 1. Lintusaari J
    2. Gutmann MU
    3. Dutta R
    4. Kaski S
    5. Corander J

    (2017) Fundamentals and recent developments in approximate bayesian computation

    SYSTEMATIC BIOLOGY 66:e66–e82.

    https://doi.org/10.1093/sysbio/syw077

    • PubMed
    • Google Scholar
42. 1. Lorenz R
    2. Monti RP
    3. Violante IR
    4. Anagnostopoulos C
    5. Faisal AA
    6. Montana G
    7. Leech R

    (2016) The Automatic Neuroscientist: A framework for optimizing experimental design with closed-loop real-time fMRI

    NeuroImage 129:320–334.

    https://doi.org/10.1016/j.neuroimage.2016.01.032

    • Google Scholar
43. 1. Marjoram P
    2. Molitor J
    3. Plagnol V
    4. Tavaré S

    (2003) Markov chain Monte Carlo without likelihoods

    PNAS 100:15324–15328.

    https://doi.org/10.1073/pnas.0306899100

    • Google Scholar
44. 1. Martinez-Cantin R

    (2014)

    Bayesopt: a bayesian optimization library for nonlinear optimization, experimental design and bandits

    Journal of Machine Learning Research: JMLR 15:3735–3739.

    • Google Scholar
45. Preprint

    1. Mikkola P
    2. Martin OA
    3. Chandramouli S
    4. Hartmann M
    5. Pla OA
    6. Thomas O
    7. Pesonen H
    8. Corander J
    9. Vehtari A
    10. Kaski S

    (2021) Prior Knowledge Elicitation: The Past, Present, and Future

    arXiv.

    https://arxiv.org/abs/2112.01380

    • Google Scholar
46. 1. Müller P

    (1999) Simulation-based optimal design

    Bayesian Statistics 6:459–474.

    https://doi.org/10.1093/oso/9780198504856.001.0001

    • Google Scholar
47. 1. Myung JI
    2. Pitt MA

    (2009) Optimal experimental design for model discrimination

    Psychological Review 116:499–518.

    https://doi.org/10.1037/a0016104

    • Google Scholar
48. 1. Myung JI
    2. Cavagnaro DR
    3. Pitt MA

    (2013) A tutorial on adaptive design optimization

    Journal of Mathematical Psychology 57:53–67.

    https://doi.org/10.1016/j.jmp.2013.05.005

    • PubMed
    • Google Scholar
49. Conference

    1. Oh C
    2. Gavves E
    3. Welling M

    (2018)

    BOCK: Bayesian optimization with cylindrical kernels

    In Proceedings of the 35th International Conference on Machine Learning.

    • Google Scholar
50. Conference

    1. Ouyang L
    2. Tessler MH
    3. Ly D
    4. Goodman ND

    (2018)

    webppl-oed: A practical optimal experiment design system

    In Proceedings of the annual meeting of the cognitive science society.

    • Google Scholar
51. 1. Overstall AM
    2. Woods DC

    (2017) Bayesian design of experiments using approximate coordinate exchange

    Technometrics 59:458–470.

    https://doi.org/10.1080/00401706.2016.1251495

    • Google Scholar
52. Book

    1. Palestro JJ
    2. Sederberg PB
    3. Osth AF
    4. Van Zandt T
    5. Turner BM

    (2018) Likelihood-Free Methods for Cognitive Science

    Springer.

    https://doi.org/10.1007/978-3-319-72425-6

    • Google Scholar
53. 1. Palminteri S
    2. Wyart V
    3. Koechlin E

    (2017) The Importance of falsification in computational cognitive modeling

    Trends in Cognitive Sciences 21:425–433.

    https://doi.org/10.1016/j.tics.2017.03.011

    • Google Scholar
54. 1. Paninski L

    (2005) Asymptotic theory of information-theoretic experimental design

    Neural Computation 17:1480–1507.

    https://doi.org/10.1162/0899766053723032

    • PubMed
    • Google Scholar
55. 1. Pashler H
    2. Wagenmakers E-J

    (2012) Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence?

    Perspectives on Psychological Science 7:528–530.

    https://doi.org/10.1177/1745691612465253

    • PubMed
    • Google Scholar
56. 1. Peterson JC
    2. Bourgin DD
    3. Agrawal M
    4. Reichman D
    5. Griffiths TL

    (2021) Using large-scale experiments and machine learning to discover theories of human decision-making

    Science 372:1209–1214.

    https://doi.org/10.1126/science.abe2629

    • PubMed
    • Google Scholar
57. 1. Pitt MA
    2. Myung IJ

    (2002) When a good fit can be bad

    Trends in Cognitive Sciences 6:421–425.

    https://doi.org/10.1016/S1364-6613(02)01964-2

    • Google Scholar
58. Conference

    1. Poole B
    2. Ozair S
    3. Den Oord A
    4. Alemi A
    5. Tucker G

    (2019)

    On variational bounds of mutual information

    In Proceedings of the 36th International Conference on Machine Learning.

    • Google Scholar
59. Preprint

    1. Rainforth T
    2. Foster A
    3. Ivanova DR
    4. Smith FB

    (2023) Modern bayesian experimental design

    arXiv.

    https://arxiv.org/abs/2302.14545

    • Google Scholar
60. 1. Ren P
    2. Xiao Y
    3. Chang X
    4. Huang P
    5. Li Z
    6. Chen X
    7. Wang X

    (2021) A comprehensive survey of neural architecture search: Challenges and solutions

    ACM Computing Surveys 54:3447582.

    https://doi.org/10.1145/3447582

    • Google Scholar
61. 1. Robbins H

    (1952) Some aspects of the sequential design of experiments

    Bulletin of the American Mathematical Society 58:527–535.

    https://doi.org/10.1090/S0002-9904-1952-09620-8

    • Google Scholar
62. 1. Ross RJH
    2. Baker RE
    3. Parker A
    4. Ford MJ
    5. Mort RL
    6. Yates CA

    (2017) Using approximate Bayesian computation to quantify cell-cell adhesion parameters in a cell migratory process

    NPJ Systems Biology and Applications 3:9.

    https://doi.org/10.1038/s41540-017-0010-7

    • PubMed
    • Google Scholar
63. 1. Rumelhart DE
    2. Hinton GE
    3. Williams RJ

    (1986) Learning representations by back-propagating errors

    Nature 323:533–536.

    https://doi.org/10.1038/323533a0

    • Google Scholar
64. 1. Runge J
    2. Bathiany S
    3. Bollt E
    4. Camps-Valls G
    5. Coumou D
    6. Deyle E
    7. Glymour C
    8. Kretschmer M
    9. Mahecha MD
    10. Muñoz-Marí J
    11. van Nes EH
    12. Peters J
    13. Quax R
    14. Reichstein M
    15. Scheffer M
    16. Schölkopf B
    17. Spirtes P
    18. Sugihara G
    19. Sun J
    20. Zhang K
    21. Zscheischler J

    (2019) Inferring causation from time series in Earth system sciences

    Nature Communications 10:2553.

    https://doi.org/10.1038/s41467-019-10105-3

    • PubMed
    • Google Scholar
65. 1. Ryan EG
    2. Drovandi CC
    3. McGree JM
    4. Pettitt AN

    (2016) A review of modern computational algorithms for bayesian optimal design

    International Statistical Review 84:128–154.

    https://doi.org/10.1111/insr.12107

    • Google Scholar
66. 1. Schad DJ
    2. Betancourt M
    3. Vasishth S

    (2021) Toward a principled Bayesian workflow in cognitive science

    Psychological Methods 26:103–126.

    https://doi.org/10.1037/met0000275

    • PubMed
    • Google Scholar
67. Book

    1. Schafer CM
    2. Freeman PE

    (2012) Likelihood-free inference in cosmology: potential for the estimation of luminosity functions

    In: Feigelson E, Babu G, editors. Statistical Challenges in Modern Astronomy V. New York: Springer. pp. 3–19.

    https://doi.org/10.1007/978-1-4614-3520-4

    • Google Scholar
68. 1. Schulz LE
    2. Sommerville J

    (2006) God does not play dice: causal determinism and preschoolers’ causal inferences

    Child Development 77:427–442.

    https://doi.org/10.1111/j.1467-8624.2006.00880.x

    • PubMed
    • Google Scholar
69. 1. Schulz E
    2. Franklin NT
    3. Gershman SJ

    (2020) Finding structure in multi-armed bandits

    Cognitive Psychology 119:101261.

    https://doi.org/10.1016/j.cogpsych.2019.101261

    • PubMed
    • Google Scholar
70. 1. Shahriari B
    2. Swersky K
    3. Wang Z
    4. Adams RP
    5. de Freitas N

    (2015) Taking the human out of the loop: a review of bayesian optimization

    Proceedings of the IEEE 104:148–175.

    https://doi.org/10.1109/JPROC.2015.2494218

    • Google Scholar
71. 1. Shannon CE

    (1948) A mathematical theory of communication

    Bell System Technical Journal 27:379–423.

    https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

    • Google Scholar
72. 1. Sisson SA
    2. Fan Y
    3. Tanaka MM

    (2007) Sequential monte carlo without likelihoods

    PNAS 104:1760–1765.

    https://doi.org/10.1073/pnas.0607208104

    • PubMed
    • Google Scholar
73. 1. Steyvers M
    2. Lee MD
    3. Wagenmakers EJ

    (2009) A Bayesian analysis of human decision-making on bandit problems

    Journal of Mathematical Psychology 53:168–179.

    https://doi.org/10.1016/j.jmp.2008.11.002

    • Google Scholar
74. Book

    1. Sutton RS
    2. Barto AG

    (2018)

    Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series

    The MIT Press.

    • Google Scholar
75. 1. Thompson WR

    (1933) On the likelihood that one unknown probability exceeds another in view of the evidence of two samples

    Biometrika 25:285–294.

    https://doi.org/10.1093/biomet/25.3-4.285

    • Google Scholar
76. 1. Turner BM
    2. Van Zandt T

    (2012) A tutorial on approximate Bayesian computation

    Journal of Mathematical Psychology 56:69–85.

    https://doi.org/10.1016/j.jmp.2012.02.005

    • Google Scholar
77. 1. Turner BM
    2. Sederberg PB

    (2014) A generalized, likelihood-free method for posterior estimation

    Psychonomic Bulletin & Review 21:227–250.

    https://doi.org/10.3758/s13423-013-0530-0

    • PubMed
    • Google Scholar
78. 1. Turner BM
    2. Forstmann BU
    3. Love BC
    4. Palmeri TJ
    5. Van Maanen L

    (2017) Approaches to analysis in model-based cognitive neuroscience

    Journal of Mathematical Psychology 76:65–79.

    https://doi.org/10.1016/j.jmp.2016.01.001

    • PubMed
    • Google Scholar
79. 1. Turner BM
    2. Schley DR
    3. Muller C
    4. Tsetsos K

    (2018) Competing theories of multialternative, multiattribute preferential choice

    Psychological Review 125:329–362.

    https://doi.org/10.1037/rev0000089

    • PubMed
    • Google Scholar
80. 1. Ullman TD
    2. Stuhlmüller A
    3. Goodman ND
    4. Tenenbaum JB

    (2018) Learning physical parameters from dynamic scenes

    Cognitive Psychology 104:57–82.

    https://doi.org/10.1016/j.cogpsych.2017.05.006

    • PubMed
    • Google Scholar
81. Preprint

    1. Valentin S
    2. Kleinegesse S
    3. Bramley NR
    4. Gutmann MU
    5. Lucas CG

    (2021) Bayesian optimal experimental design for simulator models of cognition

    arXiv.

    https://arxiv.org/abs/2110.15632

    • Google Scholar
82. Software

    1. Valentin S

    (2023) Boed-Tutorial, version swh:1:rev:94f32a04693763ec6af5d31247caf129bfe99142

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:18d09063fd478a0d6b69251334b52ffe9c7f66d8;origin=https://github.com/simonvalentin/boed-tutorial;visit=swh:1:snp:f53dfc8f8da5a38fe0e90465366f408bb1489f01;anchor=swh:1:rev:94f32a04693763ec6af5d31247caf129bfe99142
83. 1. Van Bavel JJ
    2. Mende-Siedlecki P
    3. Brady WJ
    4. Reinero DA

    (2016) Contextual sensitivity in scientific reproducibility

    PNAS 113:6454–6459.

    https://doi.org/10.1073/pnas.1521897113

    • Google Scholar
84. Conference

    1. Vaswani A
    2. Shazeer N
    3. Parmar N
    4. Uszkoreit J
    5. Jones L
    6. Gomez AN
    7. Kaiser Ł
    8. Polosukhin I

    (2017)

    Attention is all you need

    Advances in Neural Information Processing Systems.

    • Google Scholar
85. 1. Wang JX
    2. Kurth-Nelson Z
    3. Kumaran D
    4. Tirumala D
    5. Soyer H
    6. Leibo JZ
    7. Hassabis D
    8. Botvinick M

    (2018) Prefrontal cortex as a meta-reinforcement learning system

    Nature Neuroscience 21:860–868.

    https://doi.org/10.1038/s41593-018-0147-8

    • PubMed
    • Google Scholar
86. 1. Weiss DJ
    2. Kingsbury GG

    (1984) Application of computerized adaptive testing to educational problems

    Journal of Educational Measurement 21:361–375.

    https://doi.org/10.1111/j.1745-3984.1984.tb01040.x

    • Google Scholar
87. 1. Welch BL

    (1947) The generalisation of student’s problems when several different population variances are involved

    Biometrika 34:28–35.

    https://doi.org/10.1093/biomet/34.1-2.28

    • PubMed
    • Google Scholar
88. 1. Wilson RC
    2. Collins AG

    (2019) Ten simple rules for the computational modeling of behavioral data

    eLife 8:e49547.

    https://doi.org/10.7554/eLife.49547

    • PubMed
    • Google Scholar
89. 1. Wolpert DH

    (1996) The lack of a priori distinctions between learning algorithms

    Neural Computation 8:1341–1390.

    https://doi.org/10.1162/neco.1996.8.7.1341

    • Google Scholar
90. 1. Yarkoni T

    (2020) The generalizability crisis

    The Behavioral and Brain Sciences 45:e1.

    https://doi.org/10.1017/S0140525X20001685

    • PubMed
    • Google Scholar
91. 1. Zhang S
    2. Lee MD
    3. Munro M

    (2009)

    Human and optimal exploration and exploitation in bandit problems

    Ratio 13:14.

    • Google Scholar
92. 1. Zhang S
    2. Lee MD

    (2010) Optimal experimental design for a class of bandit problems

    Journal of Mathematical Psychology 54:499–508.

    https://doi.org/10.1016/j.jmp.2010.08.002

    • Google Scholar
93. 1. Zhou J
    2. Cui G
    3. Hu S
    4. Zhang Z
    5. Yang C
    6. Liu Z
    7. Wang L
    8. Li C
    9. Sun M

    (2020) Graph neural networks: A review of methods and applications

    AI Open 1:57–81.

    https://doi.org/10.1016/j.aiopen.2021.01.001

    • Google Scholar

Author details

1. Simon Valentin

   School of Informatics, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing

   Contributed equally with

   Steven Kleinegesse

   For correspondence

   simonvalentin@me.com

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2039-8928

2. Steven Kleinegesse

   School of Informatics, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing

   Contributed equally with

   Simon Valentin

   Competing interests

   No competing interests declared

3. Neil R Bramley

   Department of Psychology, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Conceptualization, Formal analysis, Supervision, Methodology, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

4. Peggy Seriès

   School of Informatics, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Supervision, Writing – review and editing

   Competing interests

   No competing interests declared

5. Michael U Gutmann

   School of Informatics, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Conceptualization, Formal analysis, Supervision, Methodology, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

6. Christopher G Lucas

   School of Informatics, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Conceptualization, Formal analysis, Supervision, Methodology, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

• 790

  views

• 84

  downloads

• 0

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.