The goal of computational modeling in behavioral science is to use precise mathematical models to make better sense of behavioral data. The behavioral data most often come in the form of choices, but can also be reaction times, eye movements, or other easily observable behaviors, and even neural data. The models come in the form of mathematical equations that link the experimentally observable variables (e.g. stimuli, outcomes, past experiences) to behavior in the immediate future. In this sense, computational models instantiate different ‘algorithmic hypotheses’ about how behavior is generated.

Exactly what it means to ‘make sense’ of behavioral data is, to some extent, a matter of taste that will vary according to the researcher’s goals (Kording et al., 2018). In some cases, a simple model that can explain broad qualitative features of the data is enough. In other cases, more detailed models that make quantitative predictions are required (Breiman, 2001). The exact form of the models, and exactly what we do with them, is limited only by our imaginations, but four uses dominate the literature: simulation, parameter estimation, model comparison, and latent variable inference.

• Simulation involves running the model with particular parameter settings to generate ‘fake’ behavioral data. These simulated data can then be analyzed in much the same way as one would analyze real data, to make precise, falsifiable predictions about qualitative and quantitative patterns in the data. Simulation is a way to make theoretical predictions more precise and testable. (Some examples include Cohen et al., 1990; Collins and Frank, 2014; Rescorla and Wagner, 1972; Farashahi et al., 2017; Montague et al., 1996; Abbott et al., 2015; Lee and Webb, 2005).

• Parameter estimation involves finding the set of parameter values that best account for real behavioral data for a given model. These parameters can be used as a succinct summary of a given data set (Ratcliff, 1978; Wilson et al., 2013; Daw et al., 2011; Donkin et al., 2016), for investigating individual differences (Frank et al., 2007; Starns and Ratcliff, 2010; Collins and Frank, 2012; Gillan et al., 2016; Somerville et al., 2017; Nilsson et al., 2011) and for quantifying the effects of interventions such as drugs, lesions, illness, or experimental conditions (Frank et al., 2004; Lorains et al., 2014; Dowd et al., 2016; Zajkowski et al., 2017; Warren et al., 2017; Wimmer et al., 2018; van Ravenzwaaij et al., 2011).

• Model comparison involves trying to compute which of a set of possible models best describes the behavioral data, as a way to understand which mechanisms are more likely to underlie behavior. This is especially useful when the different models make similar qualitative predictions but differ quantitatively (Wilson and Niv, 2011; Daw et al., 2011; Collins and Frank, 2012; Collins and Frank, 2012; Fischer and Ullsperger, 2013; Steyvers et al., 2009; Haaf and Rouder, 2017; Donkin et al., 2014).

• Latent variable inference involves using the model to compute the values of hidden variables (for example values of different choices) that are not immediately observable in the behavioral data, but which the theory assumes are important for the computations occurring in the brain. Latent variable inference is especially useful in neuroimaging where it is used to help search for the neural correlates of the model (O'Doherty et al., 2007; Wilson and Niv, 2015; Donoso et al., 2014; Cohen et al., 2017), but also for electroencephalogram (EEG), electrocorticography (ECOG), electrophysiology and pupillometry among many other data sources (O'Reilly et al., 2013; Collins and Frank, 2018; Samejima et al., 2005; Cavanagh et al., 2014; Nassar et al., 2012).

Each of these uses has its strengths and weaknesses, and each of them can be mishandled in a number of ways, causing us to draw wrong and misleading conclusions (Nassar and Frank, 2016; Palminteri et al., 2017). Here we present a beginner-friendly, pragmatic, practical and details-oriented introduction (complete with example code available at [code]) on how to relate models to data and how to avoid many potential modeling mistakes. Our goal for this paper is to go beyond the mere mechanics of implementing models — as important as those mechanics are — and instead focus on the harder question of how to figure out what, exactly, a model is telling us about the mind. For this reason, we focus primarily on the simplest modeling techniques most accessible to beginning modelers, but almost all of our points apply more generally and readers interested in more advanced modeling techniques should consult the many excellent tutorials, didactic examples, and books on the topic (Busemeyer and Diederich, 2010; Daw, 2011; Daw and Tobler, 2014; Heathcote et al., 2015; Huys, 2017; Turner et al., 2013; Vandekerckhove et al., 2015; Wagenmakers and Farrell, 2004; Rigoux et al., 2014; Nilsson et al., 2011; Farrell and Lewandowsky, 2018; Lee et al., 2019).

For clarity of exposure, we chose to make all of the examples in this paper reflect a single narrow domain - reinforcement learning models applied to choice data (Sutton and Barto, 2018). We chose this domain for a few reasons. (1) Modeling is particularly popular in the field of learning. Indeed, this field benefits from modeling particularly because of the nature of the behavioral data: trials are dependent on all past history and thus unique, making classic data analysis with aggregation across conditions less successful. (2) The sequential dependency of trials in learning contexts can lead to technical challenges when fitting models that are absent in non-learning contexts. However, the same techniques are widely and successfully applied to other observable behavior, such as reaction times (Ratcliff and Rouder, 1998; Viejo et al., 2015; Ballard and McClure, 2019; Wiecki et al., 2013), and to other domains, including but not limited to perception (Sims, 2018), perceptual decision-making (Ratcliff and Rouder, 1998; Drugowitsch et al., 2016; Findling et al., 2018), economic decision-making (van Ravenzwaaij et al., 2011; Nilsson et al., 2011), visual short-term memory (Donkin et al., 2016; Donkin et al., 2014; Nassar et al., 2018), long-term memory (Batchelder and Riefer, 1990), category learning (Lee and Webb, 2005), executive functions (Haaf and Rouder, 2017; Jahfari et al., 2019), and so on. Thus, our hope is that, regardless of the techniques you use or the domain you model, by following these 10 simple steps (Figure 1), you will be able to minimize your modeling mishaps and unleash the power of computational modeling on your own behavioral data!

Figure 1

Download asset Open asset

Computational modeling is a powerful technique, but it can never replace good experimental design. Modeling attempts to capture how information is manipulated behind the scenes to produce the behavior; thus it is fundamentally limited by the behavioral data, which is itself fundamentally limited by the experimental protocol. A researcher studying face perception would not attempt to fit Prospect Theory to a face perception task; and a researcher studying the differential effects of gain and loss would not do it in a gambling task with only gains. Although obvious in these simple cases, the question becomes more difficult as the complexity of the model increases: is a given learning protocol rich enough to allow the identification of dynamic changes in learning rate, of working memory or episodic memory contributions to learning, or of reward range adaptation? Often, the answer to these questions will be ‘no’ unless the protocol has been deliberately designed to provide this power.

So, how should you go about designing a good experiment with computational modeling in mind? While this process will always be something of an art form, we suggest that you ask yourself the following questions in order to optimize your experimental design:

The ten rules in this paper are quite general, but we will illustrate many of our points using simple examples from our own field of reinforcement learning. Code for implementing all of these examples is available on GitHub (https://github.com/AnneCollins/TenSimpleRulesModeling) (Collins and Wilson, 2019; copy archived at https://github.com/elifesciences-publications/TenSimpleRulesModeling). The goal of these example studies is to understand how people learn to maximize their rewards in a case where the most rewarding choice is initially unknown.

More specifically, we consider the case in which a participant makes a series of $T$ choices between $K$ slot machines, or ‘one-armed bandits’, to try to maximize their earnings. If played on trial $t$, each slot machine, $k$, pays out a reward, $r_t$, which is one with reward probability, ${\mu }_t^k$, and otherwise 0. The reward probabilities are different for each slot machine and are initially unknown to the subject. In the simplest version of the task, the reward probabilities are fixed over time.

The three experimental parameters of this task are: the number of trials, $T$, the number of slot machines, $K$, and the reward probabilities of the different options, ${\mu }_t^k$, which may or may not change over time. The settings of these parameters will be important for determining exactly what information we can extract from the experiment. In this example, we will assume that $T=1000$, $K=2$, and that the reward probabilities are ${\mu }_t^1=0.2$ for slot machine 1 and ${\mu }_t^2=0.8$ for slot machine 2.

Just as bad experiments can limit our ability to test different hypotheses, bad models – quite literally the mathematical embodiment of our hypotheses – can further limit the conclusions we can draw (Donkin et al., 2014). This point is especially important if we are designing new models, but even well-established computational models can be problematic in some cases (Broomell and Bhatia, 2014; Nilsson et al., 2011).

Critical to the design of the model is a clear understanding of your reason for modeling. Are you interested in a descriptive model that succinctly summarizes, but perhaps does not explain, behavioral data? A mechanistic model to tie behavior to the brain? Or an elegant mathematical model to illustrate a concept? As shown in an excellent article by Kording and colleagues (Kording et al., 2018), computational modelers have a wide variety of goals for their models, and understanding your own motivations is a great place to start.

More pragmatically, there are a number of different approaches for designing models that have been successfully used in the literature. Perhaps the simplest approach is to use heuristics to find a ‘reasonable’ way to handle information to produce the target behavior. This approach was how the delta rule (see Model three below) was first invented (Rescorla and Wagner, 1972). Another approach is to scour the artificial intelligence, computer science, and applied mathematics literature for algorithms that have been used to solve similar problems for artificial agents. This approach has been fruitfully applied in the field of reinforcement learning (Sutton and Barto, 2018), where algorithms such as Q-learning and temporal difference learning have been related to human and animal behavior and brain function (Watkins and Dayan, 1992; Montague et al., 1996). Another approach is to take a Bayes-optimal perspective, to design algorithms that perform optimally given a model of the environment and the task. Ideal observer models in vision are one example in which this approach has been applied successfully (Geisler, 2011). More generally, Bayes-optimal models can be further pursued by investigating simpler algorithms that approximate the ideal strategy, or by imposing bounded rationality constraints, such as limited computational resources, on ideal observer agents (Courville and Daw, 2008; Nassar et al., 2010; Collins and Frank, 2012; Daw and Courville, 2007; Lieder et al., 2018).

Regardless of the approach (or, better yet, approaches) that you take to design your models, it is important to keep the following points in mind:

Once you have an experimental design and a set of computational models, a really important step is to create fake, or surrogate data (Palminteri et al., 2017). That is, you should use the models to simulate the behavior of participants in the experiment, and to observe how behavior changes with different models, different model parameters, and different variants of the experiment. This step will allow you to refine the first two steps: confirming that the experimental design elicits the behaviors assumed to be captured by the computational model. To do this, here are some important steps.

Visualize the simulated behavior of different models

This will allow you to verify that behavior is qualitatively different for different models, making their predictions in the experimental setup different (Box 2—figure 1A). If the behavior of different models is not qualitatively different, this is a sign that you should try to design a better experiment. Although not always possible, distinguishing between models on the basis of qualitative patterns in the data is always preferable to quantitative model comparison (Navarro, 2019; Palminteri et al., 2017).

More generally, the goal of the simulation process is to clarify how the models and experimental design satisfy your goal of identifying a cognitive process in behavior. If the answer is positive — i.e. the experiment is rich enough to capture the expected behavior, the model’s parameters are interpretable, and competing models make dissociable predictions — you can move on to the next step. Otherwise, you should loop back through these first three sections to make sure that your experimental design and models work well together, and that the model parameters have identifiable effects on the behavior, which is a prerequisite for the fourth step, fitting the parameters (c.f. Figure 1).

Box 2.

Example: simulating behavior in the bandit task.

To simulate behavior, we first need to define the parameters of the task. These include the total number of trials, $T$ (=1000 in the example), as well as the number of bandits, $K(=2)$, and the reward probability for each bandit, ${\mu }^k$ (0.2 and 0.8 for bandits 1 and 2, respectively). The experiment parameters, as used in the simulation, should match the actual parameters used in the experiment.

Next we define the parameters of the model. One way to do this is to sample these parameters randomly from prior distributions over each parameter, the exact form of which will vary from model to model. These prior distributions should generally be as broad as possible, but if something is known about the distribution of possible parameter values for a particular model, this is one place to include it.

With the free parameters set, we then proceed with the simulation. First, we simulate the choice on the first trial, $a_1$, by assuming that the model chooses option $k$ with probability, $p_1^k$. Next we simulate the outcome, $r_1$, of this choice. In Models 2–5, we use the action and/or outcome to update the choice probabilities for the next trial. Repeating this process for all trials up to $t=T$ completes one simulation. The simulations can then be analyzed in the same way as participants’ data is, ideally with the same code taking different inputs. This process should be repeated several times, with different parameter settings, to get a handle on how the model behaves as a function of its parameters.

To illustrate how one might visualize the simulated results, we look at two model-independent measures that should capture fundamental aspects of learning: the probability of repeating an action, $p(\text{stay})$ (should I change my behavior in response to feedback?), and the probability of choosing the correct option, $p(\text{correct})$ (have I learned?). In Box 2—figure 1A below, we plot $p(\text{stay})$ as a function of the reward on the last trial for each of the models with a particular set of parameters (M1: $b=0.5$, M2: $ϵ=0.05$, M3: $\alpha =0.1$, $\beta =5$, M4: ${\alpha }_c=0.1$, ${\beta }_c=3$, M5: $\alpha =0.1$, $\beta =5$, ${\alpha }_c=0.1$, ${\beta }_c=1$). For some models (in particular the win-stay-lose-shift model (Model 2), we expect a strong dependence on past reward, but for others, such as the random responder (Model 1), we expect no dependence. Of course, the exact behavior of each model depends crucially on the parameters used in the simulations and care should be taken to ensure that these simulation parameters are reasonable, perhaps by matching to typical parameter values used in the literature or by constraining to human-like overall performance. Better yet is to simulate behavior across a range of parameter settings to determine how the model-independent measures change with different parameters.

A more thorough exploration of the parameter space for Model 3 is shown in Box 2—figure 1B, where we plot the $p(\text{correct})$ in the first and last 10 trials as a function of the learning rate, $\alpha$, and softmax parameter, $\beta$. Note that the ‘optimal’ learning rate, i.e. the value of $\alpha$ that maximizes ${\displaystyle p(\text{correct})}$, varies between early and late trials and as a function of the softmax parameter $\beta$, where for early trials higher $\beta$ implies a lower optimal $\alpha$ (Daw et al., 2011).

The question of how to choose the model-independent measures of behavior has no easy answer and calls to the domain knowledge of the specific scientific question that the modeler is attempting to answer. As a rule of thumb, the measures should capture global characteristics (e.g. overall performance) and diagnostic measures that relate to the question of interest, and may visualize different qualitative predictions of different models.

Box 2—figure 1

Download asset Open asset

A key component of computational modeling is estimating the values of the parameters that best describe your behavioral data. There are a number of different ways of estimating parameters, but here we focus on the maximum-likelihood approach, although almost all of our points apply to other methods such as Markov Chain Monte Carlo approaches (Lee and Wagenmakers, 2014). Mathematical details, as well as additional discussion of other approaches to model fitting can be found in Appendix 1.

In the maximum likelihood approach to model fitting, our goal is to find the parameter values of model $m$, ${\widehat{\mathit{𝜽}}}_m^{MLE}$, that maximize the likelihood of the data, $d_{1:T}$, given the parameters, $p(d_{1:T}|{\mathit{𝜽}}_m,m)$. Maximizing the likelihood is equivalent to maximizing the log of the likelihood, $LL=\log p(d_{1:T}|{\mathit{𝜽}}_m,m)$, which is numerically more tractable. (The likelihood is a product of many numbers smaller than 1, which can be rounded to 0 with limited precision computing. By contrast, the log-likelihood is a sum of negative numbers, which is usually tractable and will not be rounded to 0.) A simple mathematical derivation shows that this log-likelihood can be written in terms of the choice probabilities of the individual model as

where $p(c_t|d_{1:t-1},s_t,{\mathit{𝜽}}_m,m)$ is the probability of each individual choice given the parameters of the model and the information available up to that choice, which is at the heart of the definition of each model (for example in Equations 1-7).

In principle, finding the maximum likelihood parameters is as ‘simple’ as maximizing $LL$. In practice, of course, finding the maximum of a function is not a trivial process. The simplest approach, a brute force search of the entire parameter space, is occasionally useful, and may help you to understand how different parameters interact (see Box 3—figure 1). However, this approach is unfeasible outside of the simplest cases (e.g. one or two parameters with tight bounds) because of the high computational costs of evaluating the likelihood function at a large number of points.

Fortunately, a number of tools exist for finding local maxima (and minima) of functions quickly using variations on gradient ascent (or descent). For example, Matlab’s fmincon function can use a variety of sophisticated optimization algorithms (e.g. Moré and Sorensen, 1983; Byrd et al., 2000) to find the minimum of a function (and other factors such as the Hessian that can be useful in some situations [Daw, 2011]). So long as one remembers to feed fmincon the negative log-likelihood (whose minimum is at the same parameter values as the maximum of the positive log-likelihood), using tools such as fmincon can greatly speed up model fitting. Even here, though, a number of problems can arise when trying to maximize $LL$ that can be reduced by using the tips and tricks described below. Most of the tips come from understanding that optimization algorithms are not foolproof and in particular are subject to numerical constraints. They generalize to other black box optimization functions in other languages, for example the Python scipy.optimize package or the optim function in R.

Beware local minima!

Finally, a key limitation of optimization algorithms is that they are only guaranteed to find local minima, which are not guaranteed to be the global minima corresponding to the best fitting parameters. One way to mitigate this issue is to run the fitting procedure multiple times with random initial conditions, recording the best fitting log-likelihood for each run. The best fitting parameters are then the parameters corresponding to the run with the highest log-likelihood. There is no hard-and-fast rule for knowing how many starting points to use in a given situation, besides the fact that more complex models will require more starting points. Thus, this number must be determined empirically in each case. One way to validate the number of starting points is by plotting the best likelihood score as a function of the number of starting points. As the number of initial conditions increases, the best-fitting likelihood (and corresponding the parameters) will improve up to an asymptote close to the true maximum of the function (e.g. Box 3—figure 1).

Box 3.

Example: contending with multiple local maxima.

As a real example with local maxima, we consider the case of a simplified version of the mixed reinforcement learning and working memory model from Collins and Frank, 2012. For simplicity, we relegate the details of this model to Appendix 4. To appreciate the example, all one really needs to know is that in its simplest version, this model has two parameters: $\rho$, which captures the effect of working memory, and $\alpha$, which captures the learning rate of reinforcement learning. As is seen in Box 3—figure 1 below, this model (combined with an appropriate experiment) gives rise to a log-likelihood surface with multiple local maxima. Depending on the starting point, the optimization procedure can converge to any one of these local maxima, meaning that the ‘maximum’ likelihood fits may not reflect the global maximum likelihood.

To mitigate this concern, a simple and effective approach is to repeat the optimization procedure many times, keeping track of the best fitting log-likelihood and parameters in each case. An approximation to the global maximum is to take the best log-likelihood from this list of fits. The results of this multiple iteration procedure can be summarized by plotting the best log-likelihood as a function of the number of starting points, or similarly, by plotting the distance from the so-far best parameters to the final best parameters as a function of the number of starting points (Box 3—figure 1B). As the number of starting points increases, the best-fitting log-likelihood and parameters will converge to the global maximum. This plot also allows us to judge when we have used enough starting points. Specifically, if the best fitting parameters appear to have reached asymptote, that gives us a good indication that the fit is the best we can do.

Box 3—figure 1

Download asset Open asset

Before reading too much into the best-fitting parameter values, ${\mathit{𝜽}}_m^{MLE}$, it is important to check whether the fitting procedure gives meaningful parameter values in the best case scenario, -that is, when fitting fake data where the ‘true’ parameter values are known (Nilsson et al., 2011). Such a procedure is known as ‘Parameter Recovery’, and is a crucial part of any model-based analysis.

In principle, the recipe for parameter recovery is quite simple. First, simulate fake data with known parameter values. Next, fit the model to these fake data to try to ‘recover’ the parameters. Finally, compare the recovered parameters to their true values. In a perfect world, the simulated and recovered parameters will be tightly correlated, with no bias. If there is only a weak correlation between the simulated and recovered parameters and/or a significant bias, then this is an indication that there is either a bug in your code (which from our own experience is fairly likely) or the experiment is underpowered to assess this model.

To make the most of your parameter recovery analysis, we suggest the following tips:

Remember that even successful parameter recovery represents a best-case scenario!

What does successful parameter recovery tell you? That data generated by a known model with given parameters can be fit to recover those parameters. This is the best case you could possibly hope for in the model-based analysis and it is unlikely to ever occur as the ‘true’ generative process for behavior — that is, the inner workings of the mind and brain — is likely much more complex than any model you could conceive. There’s no easy answer to this problem. We only advise that you remember to be humble when you present your results!

Box 4.

Example: parameter recovery in the reinforcement learning model.

We performed parameter recovery with Model 3, the Rescorla Wagner model, on the two-armed bandit task. As before, we set the means of each bandit at ${\mu }_1=0.2$ and ${\mu }_2=0.8$ and the number of trials at $T=1000$. We then simulated the actions of the model according to Equations 3 and 4, with learning rate, $\alpha$, and softmax temperature, $\beta$, set according to

(9) $\alpha \sim U(0,1)\mathit{\hspace{1em}}\text{and}\mathit{\hspace{1em}}\beta \sim \text{Exp}(10)$

After simulating the model, we fit the parameters using a maximum likelihood approach to get fit values of learning rate, $\alpha$, and softmax parameter, $\beta$. We then repeated this process 1000 times using new values of $\alpha$ and $\beta$ each time. The results are plotted in Box 4—figure 1 below. As is clear from this plot, there is fairly good agreement between the simulated and fit parameter values. In addition, we can see that the fit for $\beta$ is best with a range, $1<\beta <10$, and that outside this range, the correspondence between simulation and fit is not as good. If we further select points where parameter recovery for $\alpha$ is bad (i.e., when $|{\alpha }_{sim}-{\alpha }_{fit}|>0.25$, grey dots in Box 4—figure 1), we find that parameter recovery for $\alpha$ is worse when $\beta$ is outside of the range. Depending on the values of $\beta$ that we obtain by fitting human behavior, this worse correspondence at small and large $\beta$ values may or may not be problematic. It may be a good idea to use the range of parameters obtained from fitting the real data to test the quality of recovery within the range that matters.

Box 4—figure 1

Download asset Open asset

In model comparison, our goal is to determine which model, out of a set of possible models, is most likely to have generated the data. There are a number of different ways to make this comparison (summarized in more detail in Appendix 2) that involve different approximations to the Bayesian evidence for each model (e.g., Daw, 2011; Rigoux et al., 2014). Here, we focus on the most common method which is related to the log-likelihood computed in 'Fit the parameters'.

A simplistic approach to model comparison would be to compare the log-likelihoods of each model at the best fitting parameter settings, $p(d_{1:T}|{\widehat{\mathit{𝜽}}}_m,m)$. However, if the data, $d_{1:T}$, used to evaluate the log-likelihood are the same as those used to fit the parameters, then this approach will lead to overfitting, as the model with the most free parameters will almost always fit this ‘training’ data best. As an extreme example, consider the case of a model with one ‘parameter’ per choice, which is the identity of the choice the person actually made. Such a ‘model’ would fit the data perfectly, but would of course tell us nothing about how the choices were actually determined and would make no predictions about what choices would be made in a different setting. Overfitting is a problem in that it decreases the generalizability of the model: it makes it less likely that the conclusions drawn would apply to a different sample.

One way to avoid overfitting is to perform cross-validation: by measuring fit on held-out data, we directly test generalizability. However, this is not always possible for practical reasons (number of samples) or more fundamental ones (dependence between data points). Thus, other methods mitigate the risk of overfitting by approximately accounting for the degrees of freedom in the model. There are several methods for doing this (including penalties for free parameters), which are discussed in more detail in the Appendices. There is a rich theoretical literature debating which method is best (Wagenmakers and Farrell, 2004; Vandekerckhove et al., 2015). Here, we do not position ourselves in this theoretical debate, and instead focus on one of the simplest methods, the Bayes Information Criterion, $BIC$, which has an explicit penalty for free parameters.

where $\widehat{LL}$ is the log-likelihood value at the best fitting parameter settings, and $k_m$ is the number of parameters in model $m$. The model with the smallest $BIC$ score is the model that best fits the data. Thus, the positive effect of $k_m$ in the last term corresponds to a penalty for models with large numbers of parameters.

While Equation 10 is simple enough to apply in order to find the model that, apparently, best fits your data, it is important to check that your model comparison process gives sensible results for simulated data. Just as parameter fitting should be validated by parameter recovery on simulated data, so model comparison should be validated by model recovery on simulated data.

More specifically, model recovery involves simulating data from all models (with a range of parameter values carefully selected as in the case of parameter recovery) and then fitting that data with all models to determine the extent to which fake data generated from model $A$ is best fit by model $A$ as opposed to model $B$. This process can be summarized in a confusion matrix (see Box 5—figure 1 below for an example) that quantifies the probability that each model is the best fit to data generated from the other models, that is, $p(\text{fit model}=B|\text{simulated model}=A)$. In a perfect world, the confusion matrix will be the identity matrix, but in practice, this is not always the case (e.g., Wilson and Niv, 2011).

When computing and interpreting a confusion matrix it is important to keep the following points in mind:

The elephant in the room with model comparison

As wonderful as it is to find that your model ‘best’ fits the behavioral data, the elephant in the room (or perhaps more correctly not in the room) with all model comparison is that it only tells you which of the models you considered fits the data best. In and of itself, this is rather limited information as there are infinitely many other models that you did not consider. This makes it imperative to start with a good set of models that rigorously capture the competing hypotheses (that is, think hard in Step 2). In addition, it will be essential to validate (at least) your winning model (see Step 9) to show how simulating its behavior can generate the patterns seen in the data that you did not explicitly fit, and thus obtain an absolute measure of how well your model relates to your data.

Box 5.

Example: confusion matrices in the bandit task.

To illustrate model recovery, we simulated the behavior of the five models on the two-armed bandit task. As before, the means were set at ${\mu }_1=0.2$ and ${\mu }_2=0.8$, and the number of trials was set at $T=1000$. For each simulation, model parameters were sampled randomly for each model. Each simulated data set was then fit to each of the given models to determine which model fit best (according to BIC). This process was repeated 100 times to compute the confusion matrices which are plotted below in Box 5—figure 1A and B.

The difference between these two confusion matrices is in the priors from which the simulation parameters were sampled. In panel A, parameters were sampled from the following priors:

Model	Priors
Model 1	$b\sim U(0,1)$
Model 2	$ϵ\sim U(0,1)$
Model 3	$\alpha \sim U(0,1)$, $\beta \sim \text{Exp}(1)$
Model 4	${\alpha }_c\sim U(0,1)$, ${\beta }_c\sim \text{Exp}(1)$
Model 5	$\alpha \sim U(0,1)$, $\beta \sim \text{Exp}(1)$, ${\alpha }_c\sim U(0,1)$, ${\beta }_c\sim \text{Exp}(1)$

In panel B, all of the softmax parameters $\beta$ and ${\beta }_c$ were increased by 1. This has the effect of reducing the amount of noise in the behavior, which makes the models more easily identifiable and the corresponding confusion matrix more diagonal. The fact that the confusion matrix can be so dependent on the simulating parameter values means that it is crucial to match the simulation parameters to the actual fit parameters as best as possible. Models that are identifiable in one parameter regime may be impossible to distinguish in another!

In addition to the confusion matrices, we also plot the inversion matrices in Box 5—figure 1C and D. These are computed from the confusion matrices using Bayes rule assuming a uniform prior on models (see Appendix 3). These matrices more directly address the question of how to interpret a model comparison result where one model fits a particular subject best.

Box 5—figure 1

Download asset Open asset

Once all the previous steps have been completed, you can finally move on to modeling your empirical data. The first step to complete is of course to analyze the data without the model, in the same way that we recommended for model simulations in section 'Simulate, simulate, simulate!' This model-independent analysis is extremely important: you designed the experiment to test specific hypotheses, and constructed models to reflect them. Simulations showed expected patterns of behaviors given those hypotheses. If the model-independent analyses do not show evidence of the expected results, there is almost no point in fitting the model. Instead, you should go back to the beginning, either re-thinking the computational models if the analyses show interesting patterns of behavior, or re-thinking the experimental design or even the scientific question you are trying to answer. In our experience, if there is no model-independent evidence that the processes of interest are engaged, then a model-based analysis is unlikely to uncover evidence for the processes either.

If, however, the behavioral results are promising, the next step is to fit the models developed previously and to perform model comparison. After this step, you should check that the parameter range obtained with the fitting is within a range where parameter and model recovery were good. If the range is outside what you explored with simulations, you should go back over the parameter and model recovery steps to match the empirical parameter range, and thus ensure that the model fitting and model comparison procedures lead to interpretable results.

An important point to remember is that human behavior is always messier than the model, and it is unlikely that the class of models you explored actually contains the ‘real’ model thatgenerated human behavior. At this point, you should consider looping back to Steps 2–5 to improve the models, guided by in depth model-independent analysis of the data.

For example, you may consider modeling ‘unimportant parameters’, representing mechanisms that are of no interest to your scientific question but that might still affect your measures. Modeling these unimportant parameters usually captures variance in the behavior that would otherwise be attributed to noise, and as such, makes for a better estimation of ‘important’ parameters. For example, capturing pre-existing biases (e.g. a preference for left/right choices) in a decision or learning task provides better estimation of the inverse temperature, by avoiding attributing systematic biases to noise, which then affords better estimation of other parameters like the learning rate (this is evident in Box 6—figure 1).

Box 6.

Example: improving parameter recovery by modeling unimportant parameters.

To illustrate the effect that ‘unimportant’ parameters (i.e., parameters that represent mechanisms that are of no interest to your scientific question, but may still affect your measures) can have on fitting results, we model the effect of a side bias on parameter recovery in Model 3. In particular, we assume that, in addition to choosing based on learned value, the model also had a side bias, $B$, that effectively changes the value of the left bandit. That is, in the two-bandit case, the choice probabilities are given by

(11) $p_t^{left}=\frac{1}{1+\exp \left(\beta (Q_t^{right}-Q_t^{left}-B)\right)}$

We then simulated behavior with this model for a range of parameter values and fit the model with the original version of model 3, without the bias, and the modified version of model 3, with the bias. In this simulation, agents learn for 10 independent two-armed bandits in successive 50-trial blocks, with $\mu =\{0.2,0.8\}$ or $\mu =\{0.8,0.2\}$ in different blocks. For simplicity, we assumed that the agent treats each block as independent, and started from the same initial values of $Q_1^{right}=Q_1^{left}=0.5$.

As can be seen below, including the ‘unimportant’ bias in the fit greatly improves the extent to which both the learning rate, $\alpha$, and softmax parameter, $\beta$, can be recovered.

Box 6—figure 1

Download asset Open asset

All the previous steps measure a relative goodness of fit. Does model A fit better than model B? However, before interpreting any results from a model, it is essential to ensure that the model actually usefully captures the data in an absolute sense. This step is called model validation, and should never be skipped: it is possible to fit a model, get high fit measures, and nevertheless completely miss the essence of the behavior.

One method for model validation is computing the average trial likelihood as an absolute measure of fit. Although this measure has some nice properties — for example, the best possible value is one when the model predicts behavior perfectly — it offers limited value when choices are actually stochastic (which may be the case in many situations; Drugowitsch et al., 2016) or the environment is complex. In these cases, the best possible likelihood per trial is less than 1, but it is not known what the best possible likelihood per trial could be. For this reason, although the likelihood per trial can be a useful tool for model validation (Leong et al., 2017), interpreting it as an absolute measure of model fit is of limited value.

A better method to validate a model is to simulate it with the fit parameter values (Palminteri et al., 2017; Nassar and Frank, 2016; Navarro, 2019), a procedure long performed by statisticians as part of the ‘posterior predictive check’ (Roecker, 1991; Gelman et al., 1996). You should then analyze the simulated data in the same way that you analyzed the empirical data, to verify that all important behavioral effects are qualitatively and quantitatively captured by the simulations with the fit parameters. For example, if you observe a qualitative difference between two conditions empirically, the model should reproduce it. Likewise, if a learning curve reaches a quantitative asymptote of 0.7, simulations shouldn’t reach a vastly different one.

Some researchers analyze the posterior prediction of the model conditioned on the past history, instead of simulated data. In our previous notation, they evaluate the likelihood of choice $c_t$ given past data, $d_{1:t-1}$, where the past data includes choices made by the subject, not choices made by the model, $p(c_t|d_{1:t-1},s_t,{\mathit{𝜽}}_m,m)$. In some cases, this approach leads to very similar results to simulations, because simulations sample choices on the basis of a very similar probability, where the past data, $d_{1:t-1}$, include choices made by the model. However, it can also be dramatically different if the path of actions sampled by the participant is widely different from the paths likely to be selected by the model (leading to very different past histories).

Palminteri and colleagues (Palminteri et al., 2017) offer a striking example of this effect, where Model A fits better than Model B by any quantitative measure of model comparison, but is completely unable to capture the essence of the behavior. In their example, data are generated with a reinforcement learning agent (which takes the place of the subject) on a reversal learning task (where a choice that was previously good becomes bad, and reciprocally). These data are then fit with either a win-stay lose-shift model (model B), or a simplistic choice kernel model, which assumes that previous choices tend to be repeated (model A). Because of the autocorrelation in the choices made by the reinforcement learning agent, model A, which tends to repeat previous actions, fits better than model B, whose win-stay-lose-shift choices only depend on the action and outcome from the last trial. However, model A is completely insensitive to reward, and thus is unable to generate a reversal behavior when it is simulated with the fit model parameters. Thus, in this case, model A should be discarded, despite a greater quantitative fit. Nevertheless, the fact that the best validating model B captures less variance than model A should serve as a warning that model B is missing crucial components of the data and that a better model probably exists. This should incite the researcher to go back to the drawing board to develop a better model, for example one that combines elements of both models or a different model entirely, and perhaps a better experiment to test it.

More generally, if your validation step fails, you should go back to the drawing board! This may involve looking for a better model, as well as redesigning the task. Be careful interpreting results from a model that is not well validated! Of course, exactly what it means for a model to ‘fail’ the validation step is not well defined: no model is perfect, and there is no rule of thumb to tell us when a model is good enough. The most important aspect of validation is for you (and your readers) to be aware of its limitations, and in which ways they may influence any downstream results.

Box 7.

Example: model validation where the fit model performs too well.

Most examples of model validation involve a case where a model that fits well performs poorly on the task in simulation. For example, in the Palminteri et al. (2017) example, the choice kernel model cannot perform the task at all because its behavior is completely independent of reward. Here, we offer a different example of failed model validation in which the model performs better in simulation than the predicted and observed artificial agent’s behavior. Moreover, this model appears to fit data generated from a different model better than it fits data generated from itself! In this example, we imagine a deterministic stimulus-action learning task in which agents are presented with one of three stimuli ($s_1$, $s_2$, and $s_3$), which instruct them which of three actions ($a_1$, $a_2$, and $a_3$) will be rewarded when chosen. $a_1$ is the correct choice for both stimuli $s_1$ and $s_2$, $a_3$ for $s_3$, and $a_2$ is incorrect for all stimuli.

The two models that we consider are both reinforcement learning agents. The first, a ‘blind’ agent does not see the stimulus at all and learns only about the value of the three different actions, that is $Q(a_i)$, regardless of the stimulus. The second, a ‘state-based’ agent, observes the stimulus and learns a value for each action that can be different for each stimulus, that is $Q(a_i,s_i)$. Parameters in the models are set such that the learning curves for the two agents are approximately equal (Box 7—figure 1A). See appendices for details of the models.

We then consider how both models fit behavior simulated by either of these models. In Box 7—figure 1B, we plot the average likelihood with which the state-based model predicts the actual choices of the blind and state-based agents, that is the average $p(c_t|d_{1:t-1},{\mathit{𝜽}}_m,m=\text{state-based})$. As is clear from this figure, the state-based model predicts choices from the blind agent with higher likelihood than choices from the state-based agent! While counter intuitive, this result does not imply that the state-based model is unable to fit its own behavior. Instead, this result reflects the difference in noise (softmax parameters) between the two agents. The blind RL agent has a low noise parameter, allowing the state-based model to fit it quite well. Conversely, the state-based RL agent has a high noise parameter, meaning that the behavior is harder to predict even when it is fit with the correct model.

That the state-based model captures state-based behavior better than it fits blind behavior is illustrated in Box 7—figure 1C. Here, we plot the simulated learning curves of the state-based model using the parameter values that were fit to either the state-based agent or the blind agent. Although the parameters of the state-based model obtained through the fit to the state-based agent generate a learning curve that is quite similar to that of the agent (compare blue lines in Box 7—figure 1A and C), the state-based fit to the blind agent performs too well (compare yellow lines in Box 7—figure 1A and C).

Thus the model validation step provides support for the state-based model when it is the correct model of behavior, but rules out the state-based model when the generating model was different. The take-away from this example should be that measures of model-fit and model comparison cannot replace a thorough validation step, which can contradict them.

Box 7—figure 1

Download asset Open asset

To minimize risks of p-hacking, model-dependent analyses should only be performed on the winning model, after researchers are satisfied that the model captures the behavior. One particularly powerful application of model-based analysis of behavior involves estimating the latent variables in the model. Latent variables are the hidden components of the algorithms underlying the behavior that are not directly observable from the behavior itself. These latent variables shed light on the internal workings of the model and, if we are to take the model seriously, should have some representation in the subjects’ mind and brain (Cohen et al., 2017; O'Doherty et al., 2007).

Extracting latent variables from the model is as simple as simulating the model and recording how the latent variables evolve over time. The parameters of the simulation should be the fit parameters for each subject. In most cases, it is useful to yoke the choices of the model to the choices the participants actually made, thus the latent variables evolve according to the experience participants actually had. This is especially true if the choices can influence what participants see in the future.

Once estimated, the latent variables can be used in much the same way as any other observable variable in the analysis of data. Perhaps the most powerful application comes when combined with physiological data such as pupil dilation, EEG, and fMRI. The simplest of these approaches uses linear regression to test whether physiological variables correlate with the latent variables of interest. Such an approach has led to a number of insights into the neural mechanisms underlying behavior (Nassar et al., 2012; Daw et al., 2011; Donoso et al., 2014; Collins and Frank, 2018; Fischer and Ullsperger, 2013), although, as with any modeling exercise, latent variable analysis should be done with care (Wilson and Niv, 2015).

Other model-dependent analyzes include studying individual differences as captured by fit parameters. Fit parameters can be treated as a dependent variable in continuous analyses (e.g. correlating with age, symptom scales, and so on [Gillan et al., 2016]) or group comparisons (e.g. patients vs. matched controls [Collins et al., 2014]).

Congratulations! You have developed, simulated, and fit your model (and maybe several other competing models) to your data. You have estimated parameters, computed model comparison scores, and validated whether your model can generate realistic-looking behavior. It’s time to start writing! But what exactly should you report in your paper? And how should you report it?

Our goal for this paper was to offer practical advice, for beginners as well as seasoned researchers, on the computational modeling of behavioral data. To this end, we offered guidance on how to generate models, simulate models, fit models, compare models, validate models, and extract latent variables from models to compare with physiological data. We have talked about how to avoid common pitfalls and misinterpretations that can arise with computational modeling, and lingered, quite deliberately, on the importance of good experimental design. Many of these lessons were lessons we learned the hard way, by actually making these mistakes for ourselves over a combined 20+ years in the field. By following these steps, we hope that you will avoid some of the errors that slowed our own research, and that the overall quality of computational modeling in behavioral science will improve.

1. 1. Abbott JT
   2. Austerweil JL
   3. Griffiths TL

   (2015) Random walks on semantic networks can resemble optimal foraging

   Psychological Review 122:558–569.

   https://doi.org/10.1037/a0038693

   • PubMed
   • Google Scholar
2. Conference

   1. Acerbi L

   (2018) Variational bayesian monte carlo

   Advances in Neural Information Processing Systems. pp. 8213–8223.

   https://papers.nips.cc/paper/8043-variational-bayesian-monte-carlo

   • Google Scholar
3. Conference

   1. Acerbi L
   2. Ji W

   (2017) Practical bayesian optimization for model fitting with bayesian adaptive direct search

   Advances in Neural Information Processing Systems. pp. 1836–1846.

   https://papers.nips.cc/paper/6780-practical-bayesian-optimization-for-model-fitting-with-bayesian-adaptive-direct-search

   • Google Scholar
4. 1. Akaike H

   (1974) A new look at the statistical model identification

   IEEE Transactions on Automatic Control 19:716–723.

   https://doi.org/10.1109/TAC.1974.1100705

   • Google Scholar
5. 1. Ballard IC
   2. McClure SM

   (2019) Joint modeling of reaction times and choice improves parameter identifiability in reinforcement learning models

   Journal of Neuroscience Methods 317:37–44.

   https://doi.org/10.1016/j.jneumeth.2019.01.006

   • PubMed
   • Google Scholar
6. 1. Batchelder WH
   2. Riefer DM

   (1990) Multinomial processing models of source monitoring

   Psychological Review 97:548–564.

   https://doi.org/10.1037/0033-295X.97.4.548

   • Google Scholar
7. 1. Box GE

   (1979)

   Robustness in Statistics

   201–236, Robustness in the strategy of scientific model building, Robustness in Statistics, Elsevier, 10.1016/b978-0-12-438150-6.50018-2.

   • Google Scholar
8. 1. Breiman L

   (2001) Statistical modeling: the two cultures (with comments and a rejoinder by the author)

   Statistical Science 16:199–231.

   https://doi.org/10.1214/ss/1009213726

   • Google Scholar
9. 1. Broomell SB
   2. Bhatia S

   (2014) Parameter recovery for decision modeling using choice data

   Decision 1:252–274.

   https://doi.org/10.1037/dec0000020

   • Google Scholar
10. Book

    1. Busemeyer JR
    2. Diederich A

    (2010)

    Cognitive Modeling

    Sage.

    • Google Scholar
11. 1. Byrd RH
    2. Gilbert JC
    3. Nocedal J

    (2000) A trust region method based on interior point techniques for nonlinear programming

    Mathematical Programming 89:149–185.

    https://doi.org/10.1007/PL00011391

    • Google Scholar
12. 1. Cavanagh JF
    2. Wiecki TV
    3. Kochar A
    4. Frank MJ

    (2014) Eye tracking and pupillometry are indicators of dissociable latent decision processes

    Journal of Experimental Psychology: General 143:1476–1488.

    https://doi.org/10.1037/a0035813

    • Google Scholar
13. 1. Cohen JD
    2. Dunbar K
    3. McClelland JL

    (1990) On the control of automatic processes: a parallel distributed processing account of the stroop effect

    Psychological Review 97:332–361.

    https://doi.org/10.1037/0033-295X.97.3.332

    • PubMed
    • Google Scholar
14. 1. Cohen JD
    2. Daw N
    3. Engelhardt B
    4. Hasson U
    5. Li K
    6. Niv Y
    7. Norman KA
    8. Pillow J
    9. Ramadge PJ
    10. Turk-Browne NB
    11. Willke TL

    (2017) Computational approaches to fMRI analysis

    Nature Neuroscience 20:304–313.

    https://doi.org/10.1038/nn.4499

    • PubMed
    • Google Scholar
15. 1. Collins AG
    2. Brown JK
    3. Gold JM
    4. Waltz JA
    5. Frank MJ

    (2014) Working memory contributions to reinforcement learning impairments in schizophrenia

    The Journal of Neuroscience 34:13747–13756.

    https://doi.org/10.1523/JNEUROSCI.0989-14.2014

    • PubMed
    • Google Scholar
16. 1. Collins AG
    2. Frank MJ

    (2012) How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis

    European Journal of Neuroscience 35:1024–1035.

    https://doi.org/10.1111/j.1460-9568.2011.07980.x

    • PubMed
    • Google Scholar
17. 1. Collins AG
    2. Frank MJ

    (2013) Cognitive control over learning: creating, clustering, and generalizing task-set structure

    Psychological Review 120:190–229.

    https://doi.org/10.1037/a0030852

    • PubMed
    • Google Scholar
18. 1. Collins AG
    2. Frank MJ

    (2014) Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive

    Psychological Review 121:337–366.

    https://doi.org/10.1037/a0037015

    • PubMed
    • Google Scholar
19. 1. Collins AG
    2. Frank MJ

    (2018)

    Within-and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory

    PNAS 201720963.

    • Google Scholar
20. Software

    1. Collins AG
    2. Wilson RC

    (2019) TenSimpleRulesModeling, version 3a01850

    Github.

    https://github.com/AnneCollins/TenSimpleRulesModeling
21. Conference

    1. Courville AC
    2. Daw ND

    (2008) The rat as particle filter

    Advances in Neural Information Processing Systems. pp. 369–376.

    https://papers.nips.cc/paper/3205-the-rat-as-particle-filter

    • Google Scholar
22. 1. Daw ND

    (2011) Trial-by-trial data analysis using computational models

    Decision Making, Affect, and Learning: Attention and Performance XXIII 23:3–38.

    https://doi.org/10.1093/acprof:oso/9780199600434.003.0001

    • Google Scholar
23. 1. Daw ND
    2. Gershman SJ
    3. Seymour B
    4. Dayan P
    5. Dolan RJ

    (2011) Model-based influences on humans' choices and striatal prediction errors

    Neuron 69:1204–1215.

    https://doi.org/10.1016/j.neuron.2011.02.027

    • PubMed
    • Google Scholar
24. Conference

    1. Daw ND
    2. Courville AC

    (2007)

    The pigeon as particle filter

    Advances in Neural Information Processing Systems.

    • Google Scholar
25. 1. Daw ND
    2. Tobler PN

    (2014)

    Neuroeconomics

    283–298, Value learning through reinforcement: the basics of dopamine and reinforcement learning, Neuroeconomics, Second Edition, Elsevier, 10.1016/b978-0-12-416008-8.00015-2.

    • Google Scholar
26. 1. Donkin C
    2. Tran SC
    3. Nosofsky R

    (2014) Landscaping analyses of the ROC predictions of discrete-slots and signal-detection models of visual working memory

    Attention, Perception, & Psychophysics 76:2103–2116.

    https://doi.org/10.3758/s13414-013-0561-7

    • Google Scholar
27. 1. Donkin C
    2. Kary A
    3. Tahir F
    4. Taylor R

    (2016) Resources masquerading as slots: Flexible allocation of visual working memory

    Cognitive Psychology 85:30–42.

    https://doi.org/10.1016/j.cogpsych.2016.01.002

    • Google Scholar
28. 1. Donoso M
    2. Collins AGE
    3. Koechlin E

    (2014) Foundations of human reasoning in the prefrontal cortex

    Science 344:1481–1486.

    https://doi.org/10.1126/science.1252254

    • Google Scholar
29. 1. Dowd EC
    2. Frank MJ
    3. Collins A
    4. Gold JM
    5. Barch DM

    (2016) Probabilistic reinforcement learning in patients with schizophrenia: Relationships to anhedonia and avolition

    Biological Psychiatry: Cognitive Neuroscience and Neuroimaging 1:460–473.

    https://doi.org/10.1016/j.bpsc.2016.05.005

    • Google Scholar
30. 1. Drugowitsch J
    2. Wyart V
    3. Devauchelle AD
    4. Koechlin E

    (2016) Computational precision of mental inference as critical source of human choice suboptimality

    Neuron 92:1398–1411.

    https://doi.org/10.1016/j.neuron.2016.11.005

    • PubMed
    • Google Scholar
31. 1. Farashahi S
    2. Rowe K
    3. Aslami Z
    4. Lee D
    5. Soltani A

    (2017) Feature-based learning improves adaptability without compromising precision

    Nature Communications 8:1768.

    https://doi.org/10.1038/s41467-017-01874-w

    • PubMed
    • Google Scholar
32. Book

    1. Farrell S
    2. Lewandowsky S

    (2018) Computational Modeling of Cognition and Behavior

    Cambridge University Press.

    https://doi.org/10.1017/CBO9781316272503

    • Google Scholar
33. Preprint

    1. Findling C
    2. Skvortsova V
    3. Dromnelle R
    4. Palminteri S
    5. Wyart V

    (2018) Computational noise in reward-guided learning drives behavioral variability in volatile environments

    bioRxiv.

    https://doi.org/10.1101/439885

    • Google Scholar
34. 1. Fischer AG
    2. Ullsperger M

    (2013) Real and fictive outcomes are processed differently but converge on a common adaptive mechanism

    Neuron 79:1243–1255.

    https://doi.org/10.1016/j.neuron.2013.07.006

    • PubMed
    • Google Scholar
35. 1. Frank MJ
    2. Seeberger LC
    3. O'reilly RC

    (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism

    Science 306:1940–1943.

    https://doi.org/10.1126/science.1102941

    • PubMed
    • Google Scholar
36. 1. Frank MJ
    2. Moustafa AA
    3. Haughey HM
    4. Curran T
    5. Hutchison KE

    (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning

    PNAS 104:16311–16316.

    https://doi.org/10.1073/pnas.0706111104

    • PubMed
    • Google Scholar
37. 1. Geisler WS

    (2011) Contributions of ideal observer theory to vision research

    Vision Research 51:771–781.

    https://doi.org/10.1016/j.visres.2010.09.027

    • PubMed
    • Google Scholar
38. 1. Gelman A
    2. Meng XL
    3. Stern H

    (1996)

    Posterior predictive assessment of model fitness via realized discrepancies

    Statistica Sinica 6:733–760.

    • Google Scholar
39. 1. Gershman SJ

    (2016) Empirical priors for reinforcement learning models

    Journal of Mathematical Psychology 71:1–6.

    https://doi.org/10.1016/j.jmp.2016.01.006

    • Google Scholar
40. 1. Gillan CM
    2. Kosinski M
    3. Whelan R
    4. Phelps EA
    5. Daw ND

    (2016) Characterizing a psychiatric symptom dimension related to deficits in goal-directed control

    eLife 5:e11305.

    https://doi.org/10.7554/eLife.11305

    • PubMed
    • Google Scholar
41. 1. Haaf JM
    2. Rouder JN

    (2017) Developing constraint in bayesian mixed models

    Psychological Methods 22:779–798.

    https://doi.org/10.1037/met0000156

    • PubMed
    • Google Scholar
42. 1. Heathcote A
    2. Brown SD
    3. Wagenmakers EJ

    (2015)

    An Introduction to Model-Based Cognitive Neuroscience

    25–48, An introduction to good practices in cognitive modeling, An Introduction to Model-Based Cognitive Neuroscience, Springer, 10.1007/978-1-4939-2236-9_2.

    • Google Scholar
43. 1. Huys QJM
    2. Cools R
    3. Gölzer M
    4. Friedel E
    5. Heinz A
    6. Dolan RJ
    7. Dayan P

    (2011) Disentangling the roles of approach, activation and Valence in instrumental and pavlovian responding

    PLOS Computational Biology 7:e1002028.

    https://doi.org/10.1371/journal.pcbi.1002028

    • Google Scholar
44. 1. Huys QJM

    (2017)

    Computational Psychiatry: Mathematical Modeling of Mental Illness 

    247–271, Bayesian Approaches to Learning and Decision-Making, Computational Psychiatry: Mathematical Modeling of Mental Illness , Academic Press, 10.1016/b978-0-12-809825-7.00010-9.

    • Google Scholar
45. 1. Jahfari S
    2. Ridderinkhof KR
    3. Collins AGE
    4. Knapen T
    5. Waldorp LJ
    6. Frank MJ

    (2019) Cross-Task contributions of frontobasal ganglia circuitry in response inhibition and Conflict-Induced slowing

    Cerebral Cortex 29:1969–1983.

    https://doi.org/10.1093/cercor/bhy076

    • PubMed
    • Google Scholar
46. 1. Kass RE
    2. Raftery AE

    (1995) Bayes factors

    Journal of the American Statistical Association 90:773–795.

    https://doi.org/10.1080/01621459.1995.10476572

    • Google Scholar
47. 1. Katahira K

    (2016) How hierarchical models improve point estimates of model parameters at the individual level

    Journal of Mathematical Psychology 73:37–58.

    https://doi.org/10.1016/j.jmp.2016.03.007

    • Google Scholar
48. Preprint

    1. Kording K
    2. Blohm G
    3. Schrater P
    4. Kay K

    (2018) Appreciating diversity of goals in computational neuroscience

    OSF Preprints.

    https://osf.io/3vy69/

    • Google Scholar
49. 1. Lee MD

    (2011) How cognitive modeling can benefit from hierarchical bayesian models

    Journal of Mathematical Psychology 55:1–7.

    https://doi.org/10.1016/j.jmp.2010.08.013

    • Google Scholar
50. Preprint

    1. Lee MD
    2. Criss AH
    3. Devezer B
    4. Donkin C
    5. Etz A
    6. Leite FP
    7. Matzke D
    8. Rouder JN
    9. Trueblood J
    10. White C

    (2019) Robust modeling in cognitive science

    PsyArXiv.

    https://psyarxiv.com/dmfhk/

    • Google Scholar
51. Book

    1. Lee MD
    2. Wagenmakers EJ

    (2014) Bayesian Cognitive Modeling: A Practical Course

    Cambridge university press.

    https://doi.org/10.1017/CBO9781139087759

    • Google Scholar
52. 1. Lee MD
    2. Webb MR

    (2005) Modeling individual differences in cognition

    Psychonomic Bulletin & Review 12:605–621.

    https://doi.org/10.3758/BF03196751

    • PubMed
    • Google Scholar
53. 1. Leong YC
    2. Radulescu A
    3. Daniel R
    4. DeWoskin V
    5. Niv Y

    (2017) Dynamic interaction between reinforcement learning and attention in multidimensional environments

    Neuron 93:451–463.

    https://doi.org/10.1016/j.neuron.2016.12.040

    • Google Scholar
54. 1. Lieder F
    2. Griffiths TL
    3. M Huys QJ
    4. Goodman ND

    (2018) Empirical evidence for resource-rational anchoring and adjustment

    Psychonomic Bulletin & Review 25:775–784.

    https://doi.org/10.3758/s13423-017-1288-6

    • PubMed
    • Google Scholar
55. 1. Lorains FK
    2. Dowling NA
    3. Enticott PG
    4. Bradshaw JL
    5. Trueblood JS
    6. Stout JC

    (2014) Strategic and non-strategic problem gamblers differ on decision-making under risk and ambiguity

    Addiction 109:1128–1137.

    https://doi.org/10.1111/add.12494

    • PubMed
    • Google Scholar
56. Book

    1. MacKay DJ

    (2003)

    Information Theory, Inference and Learning Algorithms

    Cambridge University Press.

    • Google Scholar
57. 1. Montague PR
    2. Dayan P
    3. Sejnowski TJ

    (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning

    The Journal of Neuroscience 16:1936–1947.

    https://doi.org/10.1523/JNEUROSCI.16-05-01936.1996

    • PubMed
    • Google Scholar
58. 1. Moré JJ
    2. Sorensen DC

    (1983) Computing a trust region step

    SIAM Journal on Scientific and Statistical Computing 4:553–572.

    https://doi.org/10.1137/0904038

    • Google Scholar
59. 1. Nassar MR
    2. Wilson RC
    3. Heasly B
    4. Gold JI

    (2010) An approximately bayesian delta-rule model explains the dynamics of belief updating in a changing environment

    Journal of Neuroscience 30:12366–12378.

    https://doi.org/10.1523/JNEUROSCI.0822-10.2010

    • PubMed
    • Google Scholar
60. 1. Nassar MR
    2. Rumsey KM
    3. Wilson RC
    4. Parikh K
    5. Heasly B
    6. Gold JI

    (2012) Rational regulation of learning dynamics by pupil-linked arousal systems

    Nature Neuroscience 15:1040–1046.

    https://doi.org/10.1038/nn.3130

    • PubMed
    • Google Scholar
61. 1. Nassar MR
    2. Helmers JC
    3. Frank MJ

    (2018) Chunking as a rational strategy for lossy data compression in visual working memory

    Psychological Review 125:486–511.

    https://doi.org/10.1037/rev0000101

    • PubMed
    • Google Scholar
62. 1. Nassar MR
    2. Frank MJ

    (2016) Taming the beast: extracting generalizable knowledge from computational models of cognition

    Current Opinion in Behavioral Sciences 11:49–54.

    https://doi.org/10.1016/j.cobeha.2016.04.003

    • PubMed
    • Google Scholar
63. 1. Navarro DJ

    (2019)

    Between the Devil and the deep blue sea: tensions between scientific judgement and statistical model selection

    Computational Brain & Behavior 2:28–34.

    • Google Scholar
64. 1. Nilsson H
    2. Rieskamp J
    3. Wagenmakers E-J

    (2011) Hierarchical Bayesian parameter estimation for cumulative prospect theory

    Journal of Mathematical Psychology 55:84–93.

    https://doi.org/10.1016/j.jmp.2010.08.006

    • Google Scholar
65. 1. O'Doherty JP
    2. Hampton A
    3. Kim H

    (2007) Model-based fMRI and its application to reward learning and decision making

    Annals of the New York Academy of Sciences 1104:35–53.

    https://doi.org/10.1196/annals.1390.022

    • PubMed
    • Google Scholar
66. 1. O'Reilly JX
    2. Schuffelgen U
    3. Cuell SF
    4. Behrens TEJ
    5. Mars RB
    6. Rushworth MFS

    (2013) Dissociable effects of surprise and model update in parietal and anterior cingulate cortex

    PNAS 110:E3660–E3669.

    https://doi.org/10.1073/pnas.1305373110

    • Google Scholar
67. 1. Otto AR
    2. Raio CM
    3. Chiang A
    4. Phelps EA
    5. Daw ND

    (2013) Working-memory capacity protects model-based learning from stress

    PNAS 110:20941–20946.

    https://doi.org/10.1073/pnas.1312011110

    • PubMed
    • Google Scholar
68. 1. Palminteri S
    2. Wyart V
    3. Koechlin E

    (2017) The importance of falsification in computational cognitive modeling

    Trends in Cognitive Sciences 21:425–433.

    https://doi.org/10.1016/j.tics.2017.03.011

    • PubMed
    • Google Scholar
69. Preprint

    1. Piray P
    2. Dezfouli A
    3. Heskes T
    4. Frank MJ
    5. Daw ND

    (2018) Hierarchical bayesian inference for concurrent model fitting and comparison for group studies

    bioRxiv.

    https://doi.org/10.1101/393561

    • Google Scholar
70. 1. Ratcliff R

    (1978) A theory of memory retrieval

    Psychological Review 85:59–108.

    https://doi.org/10.1037/0033-295X.85.2.59

    • Google Scholar
71. 1. Ratcliff R
    2. Rouder JN

    (1998) Modeling response times for Two-Choice decisions

    Psychological Science 9:347–356.

    https://doi.org/10.1111/1467-9280.00067

    • Google Scholar
72. 1. Rescorla RA
    2. Wagner AR

    (1972)

    A theory of pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement

    Classical Conditioning II: Current Research and Theory 2:64–99.

    • Google Scholar
73. 1. Rigoux L
    2. Stephan KE
    3. Friston KJ
    4. Daunizeau J

    (2014) Bayesian model selection for group studies - revisited

    NeuroImage 84:971–985.

    https://doi.org/10.1016/j.neuroimage.2013.08.065

    • PubMed
    • Google Scholar
74. 1. Roecker EB

    (1991) Prediction error and its estimation for Subset-Selected models

    Technometrics 33:459–468.

    https://doi.org/10.1080/00401706.1991.10484873

    • Google Scholar
75. 1. Samejima K
    2. Ueda Y
    3. Doya K
    4. Kimura M

    (2005) Representation of action-specific reward values in the striatum

    Science 310:1337–1340.

    https://doi.org/10.1126/science.1115270

    • PubMed
    • Google Scholar
76. 1. Schwarz G

    (1978) Estimating the dimension of a model

    The Annals of Statistics 6:461–464.

    https://doi.org/10.1214/aos/1176344136

    • Google Scholar
77. 1. Sims CR

    (2018) Efficient coding explains the universal law of generalization in human perception

    Science 360:652–656.

    https://doi.org/10.1126/science.aaq1118

    • PubMed
    • Google Scholar
78. 1. Somerville LH
    2. Sasse SF
    3. Garrad MC
    4. Drysdale AT
    5. Abi Akar N
    6. Insel C
    7. Wilson RC

    (2017) Charting the expansion of strategic exploratory behavior during adolescence

    Journal of Experimental Psychology: General 146:155–164.

    https://doi.org/10.1037/xge0000250

    • Google Scholar
79. 1. Starns JJ
    2. Ratcliff R

    (2010) The effects of aging on the speed-accuracy compromise: boundary optimality in the diffusion model

    Psychology and Aging 25:377–390.

    https://doi.org/10.1037/a0018022

    • PubMed
    • Google Scholar
80. 1. Steyvers M
    2. Lee MD
    3. Wagenmakers E-J

    (2009) A bayesian analysis of human decision-making on bandit problems

    Journal of Mathematical Psychology 53:168–179.

    https://doi.org/10.1016/j.jmp.2008.11.002

    • Google Scholar
81. 1. Sunnåker M
    2. Busetto AG
    3. Numminen E
    4. Corander J
    5. Foll M
    6. Dessimoz C

    (2013) Approximate bayesian computation

    PLOS Computational Biology 9:e1002803.

    https://doi.org/10.1371/journal.pcbi.1002803

    • PubMed
    • Google Scholar
82. Book

    1. Sutton RS
    2. Barto AG

    (2018)

    Reinforcement Learning: An Introduction

    MIT press.

    • Google Scholar
83. 1. Turner BM
    2. Forstmann BU
    3. Wagenmakers EJ
    4. Brown SD
    5. Sederberg PB
    6. Steyvers M

    (2013) A bayesian framework for simultaneously modeling neural and behavioral data

    NeuroImage 72:193–206.

    https://doi.org/10.1016/j.neuroimage.2013.01.048

    • PubMed
    • Google Scholar
84. 1. Turner BM
    2. Rodriguez CA
    3. Norcia TM
    4. McClure SM
    5. Steyvers M

    (2016) Why more is better: simultaneous modeling of EEG, fMRI, and behavioral data

    NeuroImage 128:96–115.

    https://doi.org/10.1016/j.neuroimage.2015.12.030

    • PubMed
    • Google Scholar
85. 1. Turner BM
    2. Sederberg PB

    (2012) Approximate Bayesian computation with differential evolution

    Journal of Mathematical Psychology 56:375–385.

    https://doi.org/10.1016/j.jmp.2012.06.004

    • Google Scholar
86. 1. van Ravenzwaaij D
    2. Dutilh G
    3. Wagenmakers E-J

    (2011) Cognitive model decomposition of the BART: assessment and application

    Journal of Mathematical Psychology 55:94–105.

    https://doi.org/10.1016/j.jmp.2010.08.010

    • Google Scholar
87. Book

    1. Vandekerckhove J
    2. Matzke D
    3. Wagenmakers EJ

    (2015) Model Comparison and the Principle of Parsimony

    In: Busemeyer J. R, Wang Z, Townsend J. T, Eidels A, editors. The Oxford Handbook of Computational and Mathematical Psychology, 300. Oxford University Press. 300319.

    https://doi.org/10.1093/oxfordhb/9780199957996.013.14

    • Google Scholar
88. 1. Viejo G
    2. Khamassi M
    3. Brovelli A
    4. Girard B

    (2015) Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning

    Frontiers in Behavioral Neuroscience 9:225.

    https://doi.org/10.3389/fnbeh.2015.00225

    • PubMed
    • Google Scholar
89. 1. Wagenmakers E-J
    2. Lodewyckx T
    3. Kuriyal H
    4. Grasman R

    (2010) Bayesian hypothesis testing for psychologists: a tutorial on the Savage–Dickey method

    Cognitive Psychology 60:158–189.

    https://doi.org/10.1016/j.cogpsych.2009.12.001

    • Google Scholar
90. 1. Wagenmakers EJ
    2. Farrell S

    (2004) AIC model selection using akaike weights

    Psychonomic Bulletin & Review 11:192–196.

    https://doi.org/10.3758/BF03206482

    • PubMed
    • Google Scholar
91. 1. Warren CM
    2. Wilson RC
    3. van der Wee NJ
    4. Giltay EJ
    5. van Noorden MS
    6. Cohen JD
    7. Nieuwenhuis S

    (2017) The effect of atomoxetine on random and directed exploration in humans

    PLOS ONE 12:e0176034.

    https://doi.org/10.1371/journal.pone.0176034

    • Google Scholar
92. 1. Watkins CJCH
    2. Dayan P

    (1992) Q-learning

    Machine Learning 8:279–292.

    https://doi.org/10.1007/BF00992698

    • Google Scholar
93. 1. Wiecki TV
    2. Sofer I
    3. Frank MJ

    (2013) HDDM: hierarchical bayesian estimation of the Drift-Diffusion model in Python

    Frontiers in Neuroinformatics 7:14.

    https://doi.org/10.3389/fninf.2013.00014

    • PubMed
    • Google Scholar
94. 1. Wilson RC
    2. Nassar MR
    3. Gold JI

    (2013) A mixture of delta-rules approximation to bayesian inference in change-point problems

    PLOS Computational Biology 9:e1003150.

    https://doi.org/10.1371/journal.pcbi.1003150

    • PubMed
    • Google Scholar
95. 1. Wilson RC
    2. Niv Y

    (2011) Inferring relevance in a changing world

    Frontiers in Human Neuroscience 5:189.

    https://doi.org/10.3389/fnhum.2011.00189

    • PubMed
    • Google Scholar
96. 1. Wilson RC
    2. Niv Y

    (2015) Is model fitting necessary for Model-Based fMRI?

    PLOS Computational Biology 11:e1004237.

    https://doi.org/10.1371/journal.pcbi.1004237

    • PubMed
    • Google Scholar
97. 1. Wimmer GE
    2. Li JK
    3. Gorgolewski KJ
    4. Poldrack RA

    (2018) Reward learning over weeks versus minutes increases the neural representation of value in the human brain

    The Journal of Neuroscience 38:7649–7666.

    https://doi.org/10.1523/JNEUROSCI.0075-18.2018

    • PubMed
    • Google Scholar
98. 1. Zajkowski WK
    2. Kossut M
    3. Wilson RC

    (2017) A causal role for right frontopolar cortex in directed, but not random, exploration

    eLife 6:e27430.

    https://doi.org/10.7554/eLife.27430

    • PubMed
    • Google Scholar

Author details

1. Robert C Wilson

   1. Department of Psychology, University of Arizona, Tucson, United States
   2. Cognitive Science Program, University of Arizona, Tucson, United States

   Contributed equally with

   Anne GE Collins

   For correspondence

   bob@email.arizona.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-2963-2971

2. Anne GE Collins

   1. Department of Psychology, University of California, Berkeley, Berkeley, United States
   2. Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, United States

   Contributed equally with

   Robert C Wilson

   For correspondence

   annecollins@berkeley.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3751-3662

• 56,359

  views

• 6,726

  downloads

• 397

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.