Experiments on decision making typically take the form of discrete trials in which a subject is required to make a choice between two and more alternatives (Packard and McGaugh, 1996; Yin and Knowlton, 2004; Behrens et al., 2007; Churchland et al., 2008; Buckley et al., 2009; Hanks and Summerfield, 2017; Izquierdo et al., 2017; Juavinett et al., 2018). The subject’s goal is to learn by trial and error the target rule leading to the correct choice (Mansouri et al., 2020), guided by feedback alone (reward and/or error signals; Ito et al., 2015; Amarante et al., 2017) or with additional predictive stimuli (such as texture or odour; van Wingerden et al., 2010; Wang et al., 2019; Banerjee et al., 2020). Such experimental designs can be equally applied whether the subject is primate (Rudebeck et al., 2008; Buckley et al., 2009; Leeson et al., 2009; Shiner et al., 2015), rodent (Raposo et al., 2012; Brady and Floresco, 2015; Tait et al., 2018; Campagner et al., 2019; Harris et al., 2021), or insect (Giurfa and Sandoz, 2012).

Whatever the species, neuroscience is increasingly looking to fine-grained analyses of decision-making behaviour (Krakauer et al., 2017; Pereira et al., 2020), seeking to characterise not just the variation between subjects but also a subject’s variability across trials (Smith et al., 2004; Roy et al., 2021; Ashwood et al., 2022). When analysing the behaviour of each subject in a choice task, we ideally want to know not only when the subject has learnt the correct rule but also what the subject tried while learning. Rules correspond to particular choice strategies, like ‘turn right’ or ‘press the cued lever’; exploratory decisions made while learning are also typically characterised as choice strategies, like ‘win-stay’. Fully characterising the variation within and between individuals on choice tasks requires that we can ask of any trial: what choice strategy are they using now?

Classic approaches to analysing performance on decision-making tasks, like psychometric curves, are computed post hoc and implicitly assume a fixed strategy throughout (Kim and Shadlen, 1999; Carandini and Churchland, 2013). Choice behaviour is though inherently non-stationary: learning, by definition, requires changes to behaviour, humans and other animals switch strategies as they explore tasks, and experimenters switch target rules, introducing new rules or switching previous reward contingencies (Birrell and Brown, 2000; Genovesio et al., 2005; Rich and Shapiro, 2007; Leeson et al., 2009; Donoso et al., 2014; Brady and Floresco, 2015; Jang et al., 2015; Powell and Redish, 2016; Bartolo and Averbeck, 2020; Russo et al., 2021). Some algorithms can detect changes in choice behaviour that correspond to learning (Smith et al., 2004; Suzuki and Brown, 2005), but they lack information on what strategies subjects use to learn or whether the subject’s learnt strategy matches the target rule.

We introduce a simple but effective Bayesian approach to inferring the probability of different choice strategies at trial resolution. This can be used both for inferring when subjects learn, by tracking the probability of the strategy matching the target rule, and for inferring subjects’ use of exploratory strategies during learning. We show that our inference algorithm successfully tracks strategies used by synthetic agents, and then show how our approach can infer both successful learning and exploratory strategies across decision-making tasks performed by rats, non-human primates, and humans. The analysis provides general insights about learning: that win-stay and lose-shift, commonly discussed as though yoked together (e.g. Izquierdo et al., 2017; Miller et al., 2017; Aguillon-Rodriguez et al., 2021; Roy et al., 2021), are dissociable strategies; and that subjects’ lose-shift choices show latent learning of new rules. Our approach is computationally efficient, could be used in real time for triggering changes to task parameters or neural stimulation, and is easily extended to more complex decision-making tasks: to aid this we provide an open source toolbox in MATLAB (GitHub, copy archived at Humphries, 2023a) and Python (GitHub, copy archived at Humphries and Powell, 2023b).

Given a subject performing a decision-making task with two or more choices, and the subject’s observed history of choices up to and including trial ${\displaystyle t}$, our goal is to compute the probability ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ that a particular strategy i has been used on current trial ${\displaystyle t}$.

With this probability, we ask two questions about choice strategies:

1. What have the subjects learnt? By defining strategy i as one of the available rules in the task, high values for ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ are evidence of successfully learning that rule. This approach can be equally applied to naive subjects during training or to well-trained subjects across rule switches (e.g. to see which other rules they switch between).

2. What strategies are being tried in order to learn the rule? Subjects use choice strategies to explore the structure of the environment. For example, animals often show some combination of win-stay or lose-shift behaviour at the start of learning (Jang et al., 2015; Akrami et al., 2018; Constantinople et al., 2019). Potential exploratory strategies include random guessing, history dependence, and information seeking. By choosing strategy i to be one of these exploratory strategies, we can track its occurrence.

Computing strategy probabilities at trial resolution

We compute ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ using Bayes theorem to provide an estimate of each probability and the uncertainty of that estimate, given the available evidence up to trial ${\displaystyle t}$:

(1) ${\displaystyle \underset{\text{posterior}}{\underbrace{P({\text{strategy}}_i(t)|\text{choices}(1:t))}}\propto \underset{\text{likelihood}}{\underbrace{P(\text{choices}(1:t)|{\text{strategy}}_i(t))}}\times \underset{\text{prior}}{\underbrace{P({\text{strategy}}_i(t))}},}$

where the posterior is the observer’s estimate of the probability of strategy i being executed, which we want to obtain; the likelihood is the consistency of the subject’s choices with the strategy we are assessing; and the prior is the initial estimate of the probability that the subject uses strategy i. We now define each of the likelihood, prior, and posterior distributions.

On each trial, there are two observable outcomes for a given strategy ${\displaystyle i}$: the subject’s choice is either consistent with that strategy (a success, ${\displaystyle x=1}$) or it is not (a failure, ${\displaystyle x=0}$). The sequence of successes or failures to execute the strategy is then the history of choices ${\displaystyle [\text{choices}(1:t)]}$. As each trial in the sequence has two outcomes the likelihood ${\displaystyle P(\text{choices}(1:t)|{\text{strategy}}_i(t))}$ is a binomial distribution.

Setting the prior as the Beta distribution ${\displaystyle B({\alpha }_i,{\beta }_i)}$, defined in the range [0, 1], allows us to treat the unknown probability of strategy i as a random variable. The Beta distribution is the so-called conjugate prior for the binomial distribution, meaning that the posterior is also a Beta distribution, dramatically simplifying the calculations in Equation 1.

Indeed, so simple that updating the posterior distribution ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ on trial ${\displaystyle t}$ becomes, in practice, elementary arithmetic (Methods): given the parameters of the Beta distribution ${\displaystyle ({\alpha }_i(t-1),{\beta }_i(t-1))}$ for strategy ${\displaystyle i}$ on the previous trial, and that the choice on trial ${\displaystyle t}$ is either consistent (${\displaystyle x=1}$) or not (${\displaystyle x=0}$) with strategy ${\displaystyle i}$, then update ${\displaystyle {\alpha }_i(t)\leftarrow {\alpha }_i(t-1)+x}$ and ${\displaystyle {\beta }_i(t)\leftarrow {\beta }_i(t-1)+(1-x)}$. The initial prior is defined by the values set for ${\displaystyle \alpha (0)}$ and ${\displaystyle \beta (0)}$. This, then, uses the accumulated evidence up to trial ${\displaystyle t}$ to provide a full trial-by-trial estimate of the posterior probability of strategy i, in the Beta distribution ${\displaystyle B({\alpha }_i(t),{\beta }_i(t))}$.

Using Bayes theorem in this way assumes that the parameter being estimated – here ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ – is stationary. But choice behaviour is often non-stationary, as when a subject switches exploratory strategies while learning, or when an experimenter changes the target rule after learning, forcing a subject to switch away from the learnt strategy. As we show further below, naive use of the Bayesian approach will fail to track these changes.

We solve the non-stationarity problem by weighting the evidence entered into the Bayesian update by its recency. We keep a running total of past successes to execute strategy i up to trial ${\displaystyle t}$, exponentially weighted by how far in the past each success occurred: ${\displaystyle s_i^{\ast }(t)=\sum _{\tau =1}^t{\gamma }^{t-\tau }x(\tau )}$, where ${\displaystyle \gamma \in (0,1]}$ is the rate of evidence decay. Similarly, we keep a running total of past failures to execute strategy i: ${\displaystyle f_i^{\ast }(t)=\sum _{\tau =1}^t{\gamma }^{t-\tau }(1-x(\tau ))}$. We then obtain the following algorithm for trial ${\displaystyle t}$ (Methods):

• Observe whether the choice on trial ${\displaystyle t}$ was consistent (${\displaystyle x(t)=1}$) or not (${\displaystyle x(t)=0}$) with the execution of strategy ${\displaystyle i}$.

• Update the running totals of evidence, decaying the prior evidence: ${\displaystyle s_i^{\ast }(t)=\gamma s_i^{\ast }(t-1)+x(t)}$; and ${\displaystyle f_i^{\ast }(t)=\gamma f_i^{\ast }(t-1)+(1-x(t))}$.

• Update the parameters for the posterior distribution: ${\displaystyle {\alpha }_i(t)\leftarrow {\alpha }_i(0)+s_i^{\ast }(t)}$ and ${\displaystyle {\beta }_i(t)\leftarrow {\beta }_i(0)+f_i^{\ast }(t)}$.

The posterior distribution for ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ on trial ${\displaystyle t}$ is then the Beta distribution ${\displaystyle B({\alpha }_i(t),{\beta }_i(t))}$. With ${\displaystyle \gamma =1}$ we recover standard Bayesian inference; the lower ${\displaystyle \gamma }$, the stronger recent history is weighted. Throughout we use ${\displaystyle \gamma =0.9}$ unless otherwise noted; we show below how the choice of ${\displaystyle \gamma }$ affects strategy inference.

Figure 1 demonstrates the algorithm using example data from a naive rat performing consecutive sessions on a Y-maze task. In every trial, the rat had to choose either the left or right arm (Figure 1a), and received reward based on four rules that were applied in sequence: right arm, cued arm, left arm, and uncued arm. These rules switched after 10 consecutive correct trials or 11 correct out of 12 trials. Figure 1b plots the variables needed to infer the rat’s strategy: its choices, rewards, and the cue location. We used these to compute ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ for the strategies corresponding to the first three rules, ‘go right’, ‘go cued’, and ‘go left’. Figure 1c plots the resulting posterior distributions for those three strategies across three sequential trials during the learning of the ‘go left’ rule. These show how the rat consistently choosing the left arm updates ${\displaystyle (\alpha ,\beta )}$ for ${\displaystyle P(\text{go left})}$ to shift its posterior distribution rightward, while the rat choosing not to go down the right arm when it is cued updates ${\displaystyle (\alpha ,\beta )}$ for ${\displaystyle P(\text{go cued})}$ to shift its posterior distribution leftward; as a consequence, ‘go left’ becomes the most probable strategy.

Figure 1 with 2 supplements see all

Download asset Open asset

To track changes in ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ across all trials we take the best estimate of the probability that strategy ${\displaystyle i}$ is being executed as the maximum a posteriori (MAP) value of ${\displaystyle B(\alpha (t),\beta (t))}$ (Figure 1c). Plotting the MAP probabilities for each trial in Figure 1d shows how they capture three key dynamics of choice behaviour: first, learning when the probability the subject is using the target-rule’s strategy becomes large and dominant; second, perseverance, as when learnt strategies persist after the target rule has been changed – for example, ‘go right’ persists as the dominant strategy for over a hundred trials after the reinforced rule has switched to ‘go cued’ around trial 25; and, third, switching, as when the rat switches from ‘go right’ to ‘go cued’ in the session beginning at trial 152, emphasising that choice strategy is non-stationary. Indeed, in these example data, a Bayesian approach without the decay of evidence cannot track rule learning or strategy switching (Figure 1—figure supplement 1). With the decay of evidence, the estimated MAP probabilities, and hence inferred changes in strategy, are robust to the choice of prior distribution (Figure 1—figure supplement 2).

We can also summarise the amount of evidence we have for our MAP estimate of ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ by computing the precision – the inverse of the variance – of the full probability distribution: the higher the precision, the more concentrated the Beta distribution is around the MAP estimate of the probability of strategy ${\displaystyle i}$ (Figure 1e).

The choice of scalar summaries (MAP and precision) here was made to facilitate ease of plotting (Figure 1 onwards), quantifying (Figure 2 onwards), and combining across subjects (Figure 3 onwards) the changes to the posteriors ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$. As we show below, these scalar summaries were sufficient to reveal striking behavioural changes during learning and exploration; but we emphasise that the full posterior is calculated for each trial, offering rich potential for further applications of this algorithm.

Figure 2 with 5 supplements see all

Download asset Open asset

Figure 3 with 1 supplement see all

Download asset Open asset

The above focused on strategies corresponding to the target rules to learn. We can equally define models for strategies that are explorative, to track the strategies the subject engages while learning the rule. For example, a subject’s strategy of win-stay based on their previous choice would be: count a success if trial ${\displaystyle t-1}$ was correct and the subject made the same choice on trial ${\displaystyle t}$; count a failure if trial ${\displaystyle t-1}$ was correct and the subject made a different choice on trial ${\displaystyle t}$. In Figure 1f, we plot MAP probabilities for three such explorative strategies, win-stay and lose-shift for the prior choice of left or right arm, and lose-shift based on the prior cue position. This demonstrates when exploratory strategies are used to guide behaviour; for example, of these strategies, lose-shift in response to the choice of arm begins to dominate from around trial 50–150 (see Figure 1f).

Throughout we distinguish rule strategies, which correspond to one of the reinforced rules available in the task, from exploratory strategies, which are all other possible choice strategies given task information. The choice of strategies is down to the user (Methods).

Flexibility in responses to rule changes depends on rule type

Learning a target rule is one of two key events in decision-making tasks. The other is responding to a change in the target rule. Changes in behaviour around rule switches are useful to understand the flexibility of behaviour, and the relative dominance of one mode of responding (e.g. spatial) over another (e.g. cue-driven) (Birrell and Brown, 2000; Floresco et al., 2008; Buckley et al., 2009; Wang et al., 2019). To show how our approach can quantify behavioural flexibility, we thus used our inference algorithm to examine trial-resolution changes in choice strategy around rule-switches in the same Y-maze and lever-press tasks.

In both tasks, rats which had been training on a spatial rule continued to have a high probability of executing a spatial strategy on each trial long after the rule had switched to a cue-based target rule (Figure 4a, b, left panels). Thus, rats showed little behavioural response during switches from spatial to cued rules.

Figure 4

Download asset Open asset

The switch from a cued to a spatial rule in both tasks evoked a considerably faster change in strategy. Over the same time window as the spatial → cued switch, while the probability of selecting the cued strategy declined slowly, the probability of selecting the now-correct spatial strategy increased above chance within 20 trials of the rule switch (Figure 4a, b, right panels).

To demonstrate the fine-grained analysis possible with our Bayesian algorithm, we also quantified the variability in rule-switch responses across rats by calculating for each rat the proportion of trials in which each rule strategy was above chance after the switch (Figure 4c, d). Individual rats sustained their use of the now-incorrect strategy after both spatial → cued and cued → spatial rule switches (Figure 4c, d). The reduction in use of a cued strategy after cued → spatial rule switches was also evidenced by that strategy reverting to chance for a few trials so that the proportion of trials above chance was less than one (Figure 4d, right). Rats in both tasks also showed considerable variation in how often a spatial strategy was above chance after the switch to the spatial rule (Figure 4c, d, right), suggesting varying rates of adapting to the rule switch.

Our trial-resolution algorithm has thus provided evidence from both learning (Figure 3f) and responding to rule switches (Figure 4) that the spatial → cued rule switch is harder for rats to detect than the cued → spatial switch, across two different tasks. Moreover, the behavioural response to the cued → spatial rule switch (Figure 4a, b, right panels) shows that, as both were above chance, the newly correct strategy (go spatial) can be acquired even while the now incorrect strategy (go cued) is still actively used.

Independent changes in lose-shift and win-stay around learning and rule switches

Having used the changes in rule strategies to characterise both learning and the response to rule switches, we then sought to understand the exploratory strategies that were driving these changes.

To probe this, we computed ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ for four exploratory strategies based on the task features pertinent to the target rules. For the spatial rules, these were win-stay and lose-shift based on the choice of arm or lever; for example, repeating the choice of left arm or lever after being rewarded for it on the previous trial would be a spatial win-stay. For the cued rules, these were win-stay and lose-shift based on the cue position; for example, repeating the choice of the cued arm or lever after being rewarded for it on the previous trial would be a cued win-stay.

We found strong evidence that rule learning was preceded by a high probability of animals using lose-shift in response to the corresponding feature of the target rule (Figure 5a, d, top pair of panels). This elevated use of lose-shift to the target-rule’s feature was consistent for both classical and strategy definitions of learning trials and across both the Y-maze (Figure 5a) and lever-press (Figure 5d) tasks. The sustained probability of using lose-shift was not due to the lack of opportunity to update that strategy: before the identified learning trial animals had not yet reached ceiling performance with the rule, so were still incurring losses (Figure 5a, d, bottom panels). This use of lose-shift is consistent with latent learning of the relevant feature of the target rule.

Figure 5

Download asset Open asset

Lose-shift exploration rapidly altered at rule changes too. Immediately after a rule switch, lose-shift in response to the old rule’s target feature declined rapidly as animals now persevered in applying the old rule, so frequently re-chose the same option after losing (Figure 5b, c, e, f). In the lever-press task, there was a rapid increase of lose-shift in response to the new rule’s feature for both switches from spatial to cued rules (Figure 5e, right) and vice versa (Figure 5f, left), again consistent with latent awareness that the original rule had changed.

For both learning and rule changes, win-stay exploration did not change with lose-shift. During learning, win-stay changed at or around the learning trial in both the Y-maze (Figure 5a) and lever-press (Figure 5d) tasks. At a rule change, win-stay for the old rule’s feature changed faster for the cued → spatial (Figure 5c, f, right panels) than the spatial → cued rule switches (Figure 5b, e, left panels), underpinning the faster response to the cued → spatial switch (Figure 4). We ruled out that this independence of win-stay from lose-shift was because one of those strategies could not be updated, as the rate of correct trials was neither at floor nor ceiling before learning and after rule switches (Figure 5: bottom panels in a, c, d, f). Lose-shift and win-stay were thus independently used and changed during exploration around learning and rule changes.

Latent exploratory strategies disguise the absence of learning in humans

We now turn to demonstrating insights from the inference algorithm on a wider range of tasks. Here, we analyse human performance on a simple stimulus-choice task, with no rule switches, to illustrate how the algorithm performs with only a few trials; in the next section we consider non-human primate performance on a stimulus–action task with stochastic reward.

Human participants performed a two-choice task, similar to that of Pessiglione et al., 2006, in which they had to choose between two stimuli that appeared above and below a central fixation point (Figure 6a). Which of the pair of stimuli appeared above or below fixation was randomised on each trial, and three pairs of stimuli corresponded to three interleaved trial types. In ‘gain’ trials, one stimulus increased total reward with a probability of 0.8, the other nothing; in ‘loss’ trials, one stimulus reduced total reward with a probability of 0.8, the other nothing; and in ‘look’ trials, participants randomly chose one of the two stimuli, with no associated outcome. Their implicit goal was thus to maximise reward by learning to select the gain stimulus and avoid the loss stimulus. As participants only experienced 30 of each trial type, these data could present strong challenges to classical approaches of identifying if and when participants learnt to correctly choose the gain stimulus or avoid the loss stimulus, and what strategies they adopted during learning. In contrast, our Bayesian approach can track changes in strategies within a few trials (Figure 2).

Figure 6 with 1 supplement see all

Download asset Open asset

To identify learning, we separately computed ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ for the strategies of selecting the gain stimulus and of avoiding the loss stimulus (Figure 6b). Despite the few trials involved, these estimates were not sensitive to the choice of prior (Figure 6b and Figure 6—figure supplement 1). Across all participants, the MAP estimates of ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ showed evident learning of choosing the gain stimulus, but not of avoiding the loss stimulus (Figure 6c). Applying the same learning criterion as above, of the MAP probability being above 0.5 until the end of the session, our inference algorithm suggests most participants learnt to choose the gain stimulus (13/20), but not to avoid the loss stimulus (5/20) (Figure 6d). Even then, the few who did putatively learn to avoid the loss stimulus did so only towards the end of the 30 trials (Figure 6e). Our trial-resolution probabilities have thus shown both strong evidence of which participants learnt the gain trials, and that there is weak evidence at best of any participant learning to avoid the loss stimulus.

To further examine this issue, we asked if the apparent learning to avoid loss could be explained by an exploratory strategy. For the participants who putatively learnt either the gain or loss rule strategies we computed ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ for a range of exploratory strategies that captured either responding to the previously chosen stimulus or to the previous location of the chosen stimulus. We found stimulus-driven win-stay and lose-shift were both used for learning gain trials, but not for loss trials (Figure 6f). In contrast, only location-driven exploration was consistently, albeit weakly, above chance for loss trials (Figure 6f). Because this suggested participants could be sensitive to the location of the stimuli, we then also examined location-dependent strategies in the ‘look’ trials, and found participants were strongly biased towards choosing one location over the other; however, this bias did not correlate with the participants’ use of the avoid loss strategy (Figure 6—figure supplement 1). Together, these results suggest that no participant learnt avoid loss, and those who appeared to were instead executing a location-driven exploratory strategy that happened to coincide with avoid loss for the final few trials.

Choice strategy driven by reward probability not magnitude in a stochastic decision task

More complex decision-making tasks use probabilistic and variable-size rewards for all choices. These present further challenges for the analysis of choice behaviour, not least because the variation of both probability and size of reward increases the number of features upon which subjects may base their choice strategies. To examine these challenges, we used our inference algorithm to analyse a session of data from a rhesus macaque performing a stimulus-primed two-choice task (Figure 7a).

Figure 7

Download asset Open asset

In this task, the monkey was presented with a stimulus that predicted the probability of obtaining the larger of two rewards if it subsequently chose either the left or right option on a touchscreen (Figure 7a, left). Six stimuli were associated with a different pair of probabilities for obtaining the larger reward for the left or right choice (Figure 7a, right); whatever the monkey chose, the small reward was delivered if the probability check for the large reward failed. The stimulus and associated pair probabilities switched once the monkey chose the highest probability option in 80% of trials. With our trial-resolved approach we can immediately ask two questions: did this criterion give sufficient evidence for learning, and was the monkey’s choice based on the probability of obtaining the large reward or something else?

To test the switching criterion, we computed ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ for the strategy of choosing the side with the highest probability of reward (Figure 7b). This confirmed the monkey showed good evidence of learning by the time the criterion was reached, as ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ for the highest probability option became dominant at least 10 trials before the switch of reward probabilities (orange line, Figure 7c). Our analysis also suggests that in at least one block (trials 25–160 in Figure 7b), the monkey had learnt to choose the appropriate option long before the block switch.

As the choice of either option in this task could result in the large reward, the monkey may have been basing its exploration not on the probability of the large reward, but simply on the value of the obtained reward. To examine this, we computed ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ for two sets of exploratory strategies that used different features of the monkey’s choice history: one set looked at whether repeating the same option choice was driven by the value or the probability of reward on the previous trial (Figure 7d); and the other looked at whether switching the option choice was driven by the value or the probability of reward on the previous trial (Figure 7f). The monkey’s use of exploratory strategies that repeated or switched choices evolved over the blocks (Figure 7d, f). Independent of this evolution, both repeating and switching strategies were dominated by the probability of reward rather than the reward obtained (Figure 7e, g), evidence that the monkey was basing its decisions more on reward probability than immediate value. Because each session had new arbitrary stimuli for the six pairs of probabilities, the monkey was here re-learning the mapping between stimuli and probabilities, and so was not trivially choosing from an already-learnt association. Rather, these results suggest that, through its prior training, the monkey had learnt the structure of the probabilities, and so was able to use that abstract knowledge to guide its exploratory choices.

Subjects may adopt a range of choice strategies during decision-making tasks, which change during training as they learn a target rule and as the rule is changed. Uncovering how subjects learn to make decisions is essential for understanding what information drives behaviour and for linking neural activity to behaviour. Classical approaches to characterising decision making, such as trials-to-criterion measures (Birrell and Brown, 2000; Peyrache et al., 2009; Brady and Floresco, 2015), psychometric functions (Britten et al., 1992; Palmer et al., 2005; Churchland et al., 2008), or reward curves (Gallistel et al., 2004), cannot track changes over time; algorithms for detecting learning points (Smith et al., 2004; Suzuki and Brown, 2005) and behavioural change points (Durstewitz et al., 2010; Jang et al., 2015) are useful, but do not show what strategies were used during learning or around rule changes. To address these issues, we have introduced an approach to decision-making tasks that fully characterises the probability of choice strategies at trial resolution. Our inference algorithm allows tracking both learning, by defining strategies matching target rules, and how subjects learn, by defining exploratory strategies that may or may not depend on features of the task. We have applied our algorithm to a variety of two-choice tasks, across a range of species, including Y-maze and lever-press tasks in rats, a stimulus–outcome mapping task in humans and a stochastic stimulus–action task in non-human primates.

Across the tasks we analysed, our algorithm showed that lose-shift and win-stay are independent exploratory strategies, because a change in the probability of one being executed is not mirrored by a change in the probability of the other. This challenges views that win-stay and lose-shift are complementary strategies, often yoked together (Miller et al., 2017; Izquierdo et al., 2017; Aguillon-Rodriguez et al., 2021; Roy et al., 2021).

Emphasising this independence, our analysis shows that changes to lose-shift are evidence of latent learning. Rats adopted a lose-shift strategy matching the target rule before the learning of that rule was identified by overt choice behaviour (Figure 5a, d). Conversely, after a switch in rewarded rules in the lever-press task, rats rapidly started using a lose-shift strategy that matched the new rule while still using a win-stay strategy based on the old rule (Figure 5b, c, e, f). These data are consistent with ideas that negative and positive feedback can independently drive changes in decision making. Indeed they support the idea that learning what not to do is easier than learning what to do.

We derived three potential criteria for learning from the algorithm’s estimate of a subject’s probability of using the correct strategy. The most sensitive criterion showed in both the Y-maze and lever-press tasks that learning occurred earlier and more often than detected in traditional approaches based on counting successful trials in some fixed window. All three criteria showed that switching from a spatial to a cue-based rule is harder than the reverse switch. This is consistent with prior work suggesting rats learn spatial rules more easily than visually cued rules (Mackintosh, 1969). Prior work on the same lever-press task had shown some evidence of the opposite, that re-learning is faster after switching from a spatial to a cued rule than switching from a cued to a spatial rule (Floresco et al., 2008). Our results suggest that the choice of learning criterion do not underpin this discrepancy.

There has been considerable recent advocacy for studying the variety of behaviour within and between individuals (Krakauer et al., 2017; Honegger and de Bivort, 2018; Pereira et al., 2020; Roy et al., 2021; Ashwood et al., 2022), both to more richly characterise the diversity of behavioural solutions adopted by humans or other animals facing a task, and as a basis for understanding the link between neural activity and behaviour. As it provides trial-resolution probabilities for each subject, our algorithm quantifies the behavioural diversity in a given task. We illustrated that potential here by defining learning points for each subject in the Y-maze (Figure 3c), lever-press (Figure 3f), and human gain/loss tasks (Figure 6d, e), and by showing the distribution of exploratory behaviours around rule switches (Figure 4c, d). Further work here could use the trial-resolution probabilities to characterise an individual’s repertoire of strategies and when they deployed each strategy.

Reflecting this urgent need for fine-grained behavioural insights into how subjects make decisions, recent work by Pillow and colleagues introduced two innovative solutions to the problem of tracking a subject’s behaviour (Roy et al., 2021; Ashwood et al., 2022). One solution (PsyTrack: Roy et al., 2021) continuously infers the parameters of a psychometric function to track the relative influence of different task features on choice in two-alternative forced-choice tasks; the other (Ashwood et al., 2022) infers the hidden states underlying decision-making behaviour and the transitions between them, by modelling psychometric functions in each state. Their elegant work applies to well-learnt behaviour, using computationally intensive models fit to all available data post hoc, but lacks the ability to infer when learning occurs. In contrast, our approach focuses on behavioural strategies adopted during learning and rule switches, accumulates evidence up to trial ${\displaystyle t}$ so can be used on-line, and is able to infer strategies in a few tens of trials (Figures 2 and 6), but lacks a model of the transitions between strategies or the parameters which influence the current strategy. An interesting future avenue for our work would be to use the changing probability of different strategies to infer the transition probabilities between them. Our approach and that of Pillow and colleagues are thus complementary, collectively providing a richer toolbox for understanding a subject’s behaviour on choice tasks.

The probabilities computed by our algorithm are a description of the observer: what we can tell about the behaviour on trial ${\displaystyle t}$ from the evidence so far. This Bayesian analysis of behaviour does not imply the animal is behaving in any optimal, Bayesian way, nor that its brain is computing or using probabilities (Knill and Pouget, 2004; Fiser et al., 2010; Pouget et al., 2013; Sohn et al., 2019). Nonetheless, using a Bayesian strategy analysis could let us then formulate those questions of optimality in behaviour and look for neural correlates of probabilistic representations. More generally, such a trial-by-trial analysis of a strategy could then let us look for neural correlates, more accurately tracking the changes in neural activity that precede or align with changes in behavioural strategy (Rich and Shapiro, 2009; Durstewitz et al., 2010; Karlsson et al., 2012; Powell and Redish, 2016; Guise and Shapiro, 2017).

One issue we faced is that Bayesian inference assumes the parameter being estimated is stationary, but choice strategies are not, so any estimate of their probability must also change. Here, we solve that problem by weighting the evidence entering the Bayesian update by its recency. This allows tracking the probabilities of strategies as they change, and is robust to the exact weighting used (Figure 2; Figure 2—figure supplement 2 to Figure 2—figure supplement 4).

The choice of how much to decay the influence of past evidence can determine how quickly a change in strategy is detected (Figure 2; Figure 2—figure supplement 2 to Figure 2—figure supplement 4), suggesting other solutions to this non-stationary problem are worth pursuing. For example, having a principled way of varying the weighting ${\displaystyle \gamma }$ of evidence over time could be useful, potentially so that evidence is rapidly decayed around a switch in strategies and less rapidly during stable execution of a strategy. This would require defining ${\displaystyle \gamma }$ as a function of some observable parameter of behaviour. It could also potentially find different evidence decay rates ${\displaystyle \gamma }$ between different strategies.

Introducing ${\displaystyle \gamma }$ may seem to create a free parameter for the analysis. However, all frequentist approaches to analysing the strategies of subjects also have at least one free parameter, namely the length of the time window over which to compute the probability they are using. This is true both for computing the probability of a particular strategy being used as the proportion of trials it occurs on Singh et al., 2019; Harris et al., 2021; Trepka et al., 2021 and for classic learning criterion of subjects performing sequential, or a proportion of, successful trials in some time window (e.g. Birrell and Brown, 2000; Boulougouris et al., 2007; Floresco et al., 2008; Leeson et al., 2009; Peyrache et al., 2009; Brady and Floresco, 2015). Sometimes this time window is explicitly chosen (Singh et al., 2019); sometimes the choice is implicit, as the probabilities are computed over trial blocks (Trepka et al., 2021) or phases of the experiment (Harris et al., 2021). Either way, rarely is the influence of that time window considered. Thus, rather than ${\displaystyle \gamma }$ introducing a further free parameter, it instead makes the inevitable free parameter of time explicit and bounded: we have placed strong bounds on the values of ${\displaystyle \gamma }$ by showing the trade-off between the speed of detecting a new strategy and the stability of detecting it.

Our algorithm provides rich information about choice behaviour by computing a full posterior distribution for each modelled strategy, for every trial. How that information is then used to further our understanding of choice behaviour offers a wealth of potential. Here, we tackled the question of when subjects learnt by computing three criteria from the posterior distribution of the target-rule’s strategy and the question of how they explored by tracking a point estimate of the probability of each exploratory strategy from their posterior distribution. A third question, which we did not address here, is: which strategy is the subject most likely to be using now? In our simulations, the simple decision rule of choosing the strategy with the highest MAP estimate and highest precision worked well; further work could explore solutions that make fuller use of the posterior distribution, for example computing the probability (Cinotti and Humphries, 2022) that a strategy is dominant over the others. Whether this is a good question to ask depends upon our model for how a subject is using strategies: seeking the dominant strategy implies a model where subjects switch between strategies discretely; by contrast, a model where subjects sample from a repertoire of strategies on each trial means our estimated ${\displaystyle P({\text{strategy}}_i(t)|\text{choices}(1:t))}$ is tracking the probability of sampling each strategy. Disentangling these models would give deeper insight into how subjects explore while learning.

The richness of the information depends in part on the choice of strategies examined, which is down to the end-user and what questions they wish to ask of their data. We here distinguished rule strategies and exploratory strategies: while the former should be examined to understand learning, the latter can include a wider range of options than we examined here. We focused predominantly on variants of win-stay and lose-shift strategies as these are widely studied (Miller et al., 2017; Constantinople et al., 2019; Aguillon-Rodriguez et al., 2021; Roy et al., 2021; Ashwood et al., 2022), contrasting the different features of a task – subject’s spatial choice or the task cues – that define the stay or shift response. But the range of strategy models usable with our inference algorithm is broad enough to cover many potential questions about decision making: all the algorithm needs is the success or failure to execute a given strategy on trial ${\displaystyle t}$. As such, our approach is highly extensible. It can be used with more complex models of choice-history strategies than those considered here: for example, where classic win-stay and lose-shift consider only the previous trial, n-back history strategies (Lau and Glimcher, 2005; Akrami et al., 2018) could capture the effect of choices or outcomes ${\displaystyle t-n}$ trials ago on the decision at trial ${\displaystyle t}$. It could also be used with economic games, tracking the probability of each player using specific response strategies to others’ decisions, like Tit-for-Tat (Axelrod and Hamilton, 1981; Axelrod and Dion, 1988; Nowak and Sigmund, 1993; Ohtsuki et al., 2006). Pilot or bias tests could be used to estimate the prior distribution’s parameters for each subject. And while all tasks here had two choices, the algorithm can just as easily compute probabilities of choice strategies in tasks with three or more choices. Despite this extensibility, the computational cost per trial is extremely low, as elementary arithmetic alone is needed for computing the update to the posterior distribution. Consequently, the inference algorithm we introduce here can assess the probabilities of an extensive, flexible range of choice strategies, and could do so in real time, allowing detailed interrogation and potentially closed-loop control of animal and human behaviour during decision-making tasks.

Source data from the rat Y-maze task data are available from https://crcns.org/ at http://dx.doi.org/10.6080/K0KH0KH5. Source data from the rat lever-press task (32 rats), the human gain/loss task (20 participants), and the primate stimulus-to-action task (one session) are available from the Nottingham Research Data Management Repository at http://doi.org/10.17639/nott.7274. Processed data and analysis code to replicate all figures are available in our GitHub repository https://github.com/Humphries-Lab/Bayesian_strategy_analysis_Paper (copy archived at Humphries-Lab, 2024). Our copies of the source data for the Y-maze task, lever-press task, gain/loss task, and stimulus-to-action task are also freely available from the same repository.

1. 1. Hock R
   2. Bast T
   3. Buckley M
   4. O'Neill M
   5. Moran P
   6. Maggi S
   7. Humphries M

   (2023) Nottingham Research Data Management Repository

   Experimental data for the trial-resolution strategy inference paper.

   https://doi.org/10.17639/nott.7274

1. 1. Peyrache A
   2. Khamassi M
   3. Benchenane K
   4. Wiener SI
   5. Battaglia F

   (2018) Collaborative Research in Computational Neuroscience

   Activity of neurons in rat medial prefrontal cortex during learning and sleep.

   https://doi.org/10.6080/K0KH0KH5

1. 1. Aguillon-Rodriguez V
   2. Angelaki D
   3. Bayer H
   4. Bonacchi N
   5. Carandini M
   6. Cazettes F
   7. Chapuis G
   8. Churchland AK
   9. Dan Y
   10. Dewitt E
   11. Faulkner M
   12. Forrest H
   13. Haetzel L
   14. Häusser M
   15. Hofer SB
   16. Hu F
   17. Khanal A
   18. Krasniak C
   19. Laranjeira I
   20. Mainen ZF
   21. Meijer G
   22. Miska NJ
   23. Mrsic-Flogel TD
   24. Murakami M
   25. Noel J-P
   26. Pan-Vazquez A
   27. Rossant C
   28. Sanders J
   29. Socha K
   30. Terry R
   31. Urai AE
   32. Vergara H
   33. Wells M
   34. Wilson CJ
   35. Witten IB
   36. Wool LE
   37. Zador AM
   38. The International Brain Laboratory

   (2021) Standardized and reproducible measurement of decision-making in mice

   eLife 10:63711.

   https://doi.org/10.7554/eLife.63711

   • PubMed
   • Google Scholar
2. 1. Akrami A
   2. Kopec CD
   3. Diamond ME
   4. Brody CD

   (2018) Posterior parietal cortex represents sensory history and mediates its effects on behaviour

   Nature 554:368–372.

   https://doi.org/10.1038/nature25510

   • PubMed
   • Google Scholar
3. 1. Amarante LM
   2. Caetano MS
   3. Laubach M

   (2017) Medial frontal theta is entrained to rewarded actions

   The Journal of Neuroscience 37:10757–10769.

   https://doi.org/10.1523/JNEUROSCI.1965-17.2017

   • PubMed
   • Google Scholar
4. 1. Ashwood ZC
   2. Roy NA
   3. Stone IR
   4. International Brain Laboratory
   5. Urai AE
   6. Churchland AK
   7. Pouget A
   8. Pillow JW

   (2022) Mice alternate between discrete strategies during perceptual decision-making

   Nature Neuroscience 25:201–212.

   https://doi.org/10.1038/s41593-021-01007-z

   • PubMed
   • Google Scholar
5. 1. Axelrod R
   2. Hamilton WD

   (1981) The evolution of cooperation

   Science 211:1390–1396.

   https://doi.org/10.1126/science.7466396

   • PubMed
   • Google Scholar
6. 1. Axelrod R
   2. Dion D

   (1988) The further evolution of cooperation

   Science 242:1385–1390.

   https://doi.org/10.1126/science.242.4884.1385

   • PubMed
   • Google Scholar
7. 1. Banerjee A
   2. Parente G
   3. Teutsch J
   4. Lewis C
   5. Voigt FF
   6. Helmchen F

   (2020) Value-guided remapping of sensory cortex by lateral orbitofrontal cortex

   Nature 585:245–250.

   https://doi.org/10.1038/s41586-020-2704-z

   • PubMed
   • Google Scholar
8. 1. Bartolo R
   2. Averbeck BB

   (2020) Prefrontal cortex predicts state switches during reversal learning

   Neuron 106:1044–1054.

   https://doi.org/10.1016/j.neuron.2020.03.024

   • PubMed
   • Google Scholar
9. 1. Behrens TEJ
   2. Woolrich MW
   3. Walton ME
   4. Rushworth MFS

   (2007) Learning the value of information in an uncertain world

   Nature Neuroscience 10:1214–1221.

   https://doi.org/10.1038/nn1954

   • PubMed
   • Google Scholar
10. 1. Birrell JM
    2. Brown VJ

    (2000) Medial frontal cortex mediates perceptual attentional set shifting in the rat

    The Journal of Neuroscience 20:4320–4324.

    https://doi.org/10.1523/JNEUROSCI.20-11-04320.2000

    • PubMed
    • Google Scholar
11. 1. Boulougouris V
    2. Dalley JW
    3. Robbins TW

    (2007) Effects of orbitofrontal, infralimbic and prelimbic cortical lesions on serial spatial reversal learning in the rat

    Behavioural Brain Research 179:219–228.

    https://doi.org/10.1016/j.bbr.2007.02.005

    • PubMed
    • Google Scholar
12. 1. Brady AM
    2. Floresco SB

    (2015) Operant procedures for assessing behavioral flexibility in rats

    Journal of Visualized Experiments 1:e52387.

    https://doi.org/10.3791/52387

    • PubMed
    • Google Scholar
13. 1. Britten KH
    2. Shadlen MN
    3. Newsome WT
    4. Movshon JA

    (1992) The analysis of visual motion: a comparison of neuronal and psychophysical performance

    The Journal of Neuroscience 12:4745–4765.

    https://doi.org/10.1523/JNEUROSCI.12-12-04745.1992

    • PubMed
    • Google Scholar
14. 1. Buckley MJ
    2. Mansouri FA
    3. Hoda H
    4. Mahboubi M
    5. Browning PGF
    6. Kwok SC
    7. Phillips A
    8. Tanaka K

    (2009) Dissociable components of rule-guided behavior depend on distinct medial and prefrontal regions

    Science 325:52–58.

    https://doi.org/10.1126/science.1172377

    • PubMed
    • Google Scholar
15. 1. Campagner D
    2. Evans MH
    3. Chlebikova K
    4. Colins-Rodriguez A
    5. Loft MSE
    6. Fox S
    7. Pettifer D
    8. Humphries MD
    9. Svoboda K
    10. Petersen RS

    (2019) Prediction of choice from competing mechanosensory and choice-memory cues during active tactile decision making

    The Journal of Neuroscience 39:3921–3933.

    https://doi.org/10.1523/JNEUROSCI.2217-18.2019

    • PubMed
    • Google Scholar
16. 1. Carandini M
    2. Churchland AK

    (2013) Probing perceptual decisions in rodents

    Nature Neuroscience 16:824–831.

    https://doi.org/10.1038/nn.3410

    • PubMed
    • Google Scholar
17. 1. Churchland AK
    2. Kiani R
    3. Shadlen MN

    (2008) Decision-making with multiple alternatives

    Nature Neuroscience 11:693–702.

    https://doi.org/10.1038/nn.2123

    • PubMed
    • Google Scholar
18. 1. Cinotti F
    2. Humphries MD

    (2022) Bayesian mapping of the striatal microcircuit reveals robust asymmetries in the probabilities and distances of connections

    The Journal of Neuroscience 42:1417–1435.

    https://doi.org/10.1523/JNEUROSCI.1487-21.2021

    • PubMed
    • Google Scholar
19. 1. Constantinople CM
    2. Piet AT
    3. Bibawi P
    4. Akrami A
    5. Kopec C
    6. Brody CD

    (2019) Lateral orbitofrontal cortex promotes trial-by-trial learning of risky, but not spatial, biases

    eLife 8:e49744.

    https://doi.org/10.7554/eLife.49744

    • PubMed
    • Google Scholar
20. 1. Donoso M
    2. Collins AGE
    3. Koechlin E

    (2014) Human cognition. Foundations of human reasoning in the prefrontal cortex

    Science 344:1481–1486.

    https://doi.org/10.1126/science.1252254

    • PubMed
    • Google Scholar
21. 1. Durstewitz D
    2. Vittoz NM
    3. Floresco SB
    4. Seamans JK

    (2010) Abrupt transitions between prefrontal neural ensemble states accompany behavioral transitions during rule learning

    Neuron 66:438–448.

    https://doi.org/10.1016/j.neuron.2010.03.029

    • PubMed
    • Google Scholar
22. 1. Fiser J
    2. Berkes P
    3. Orbán G
    4. Lengyel M

    (2010) Statistically optimal perception and learning: from behavior to neural representations

    Trends in Cognitive Sciences 14:119–130.

    https://doi.org/10.1016/j.tics.2010.01.003

    • PubMed
    • Google Scholar
23. 1. Floresco SB
    2. Block AE
    3. Tse MTL

    (2008) Inactivation of the medial prefrontal cortex of the rat impairs strategy set-shifting, but not reversal learning, using a novel, automated procedure

    Behavioural Brain Research 190:85–96.

    https://doi.org/10.1016/j.bbr.2008.02.008

    • PubMed
    • Google Scholar
24. 1. Gallistel CR
    2. Fairhurst S
    3. Balsam P

    (2004) The learning curve: implications of a quantitative analysis

    PNAS 101:13124–13131.

    https://doi.org/10.1073/pnas.0404965101

    • PubMed
    • Google Scholar
25. 1. Genovesio A
    2. Brasted PJ
    3. Mitz AR
    4. Wise SP

    (2005) Prefrontal cortex activity related to abstract response strategies

    Neuron 47:307–320.

    https://doi.org/10.1016/j.neuron.2005.06.006

    • PubMed
    • Google Scholar
26. 1. Giurfa M
    2. Sandoz JC

    (2012) Invertebrate learning and memory: Fifty years of olfactory conditioning of the proboscis extension response in honeybees

    Learning & Memory 19:54–66.

    https://doi.org/10.1101/lm.024711.111

    • Google Scholar
27. 1. Guise KG
    2. Shapiro ML

    (2017) Medial prefrontal cortex reduces memory interference by modifying hippocampal encoding

    Neuron 94:183–192.

    https://doi.org/10.1016/j.neuron.2017.03.011

    • PubMed
    • Google Scholar
28. 1. Hanks TD
    2. Summerfield C

    (2017) Perceptual decision making in rodents, monkeys, and humans

    Neuron 93:15–31.

    https://doi.org/10.1016/j.neuron.2016.12.003

    • PubMed
    • Google Scholar
29. 1. Harris C
    2. Aguirre C
    3. Kolli S
    4. Das K
    5. Izquierdo A
    6. Soltani A

    (2021) Unique features of stimulus-based probabilistic reversal learning

    Behavioral Neuroscience 135:550–570.

    https://doi.org/10.1037/bne0000474

    • PubMed
    • Google Scholar
30. 1. Honegger K
    2. de Bivort B

    (2018) Stochasticity, individuality and behavior

    Current Biology 28:R8–R12.

    https://doi.org/10.1016/j.cub.2017.11.058

    • PubMed
    • Google Scholar
31. Software

    1. Humphries MD

    (2023a) Bayesian_Strategy_Analysis_MATLAB, version swh:1:rev:7014e7ee7c727cc354dd6bcb470b7773a02c9f0b

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:35ea132370bd1f7a96d8f8757e5345f6ad9071fa;origin=https://github.com/Humphries-Lab/Bayesian_Strategy_Analysis_MATLAB;visit=swh:1:snp:12792188063ed225b10500a0c7fd4714535b9afc;anchor=swh:1:rev:7014e7ee7c727cc354dd6bcb470b7773a02c9f0b
32. Software

    1. Humphries MD
    2. Powell L

    (2023b) Bayesian_Strategy_Analysis_PythonPublic Watch 3 Fork 0 Star 3, version swh:1:rev:fa978c04230a22b2c4ae687de328d6e81e1bac4e

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:b2ee23fda4ea226045702a723c0aac9d3642ba9d;origin=https://github.com/Humphries-Lab/Bayesian_Strategy_Analysis_Python;visit=swh:1:snp:c1ceff2d9985fb6650a51f061575fbbca7ec324c;anchor=swh:1:rev:fa978c04230a22b2c4ae687de328d6e81e1bac4e
33. Software

    1. Humphries-Lab

    (2024) Bayesian_Strategy_Analysis_Paper, version swh:1:rev:8e32969b6ab427cca27f0baf05330357f9266cf1

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:e73262548d7356fc1b64c7e4b2414696200d620f;origin=https://github.com/Humphries-Lab/Bayesian_strategy_analysis_Paper;visit=swh:1:snp:969d42be98d6520fa0687d9cba499112243bb46d;anchor=swh:1:rev:8e32969b6ab427cca27f0baf05330357f9266cf1
34. 1. Ito HT
    2. Zhang SJ
    3. Witter MP
    4. Moser EI
    5. Moser MB

    (2015) A prefrontal-thalamo-hippocampal circuit for goal-directed spatial navigation

    Nature 522:50–55.

    https://doi.org/10.1038/nature14396

    • PubMed
    • Google Scholar
35. 1. Izquierdo A
    2. Brigman JL
    3. Radke AK
    4. Rudebeck PH
    5. Holmes A

    (2017) The neural basis of reversal learning: An updated perspective

    Neuroscience 345:12–26.

    https://doi.org/10.1016/j.neuroscience.2016.03.021

    • PubMed
    • Google Scholar
36. 1. Jang AI
    2. Costa VD
    3. Rudebeck PH
    4. Chudasama Y
    5. Murray EA
    6. Averbeck BB

    (2015) The role of frontal cortical and medial-temporal lobe brain areas in learning a bayesian prior belief on reversals

    The Journal of Neuroscience 35:11751–11760.

    https://doi.org/10.1523/JNEUROSCI.1594-15.2015

    • PubMed
    • Google Scholar
37. 1. Juavinett AL
    2. Erlich JC
    3. Churchland AK

    (2018) Decision-making behaviors: weighing ethology, complexity, and sensorimotor compatibility

    Current Opinion in Neurobiology 49:42–50.

    https://doi.org/10.1016/j.conb.2017.11.001

    • PubMed
    • Google Scholar
38. 1. Karlsson MP
    2. Tervo DGR
    3. Karpova AY

    (2012) Network resets in medial prefrontal cortex mark the onset of behavioral uncertainty

    Science 338:135–139.

    https://doi.org/10.1126/science.1226518

    • PubMed
    • Google Scholar
39. 1. Kim JN
    2. Shadlen MN

    (1999) Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque

    Nature Neuroscience 2:176–185.

    https://doi.org/10.1038/5739

    • PubMed
    • Google Scholar
40. 1. Knill DC
    2. Pouget A

    (2004) The Bayesian brain: the role of uncertainty in neural coding and computation

    Trends in Neurosciences 27:712–719.

    https://doi.org/10.1016/j.tins.2004.10.007

    • PubMed
    • Google Scholar
41. 1. Krakauer JW
    2. Ghazanfar AA
    3. Gomez-Marin A
    4. MacIver MA
    5. Poeppel D

    (2017) Neuroscience needs behavior: Correcting a reductionist bias

    Neuron 93:480–490.

    https://doi.org/10.1016/j.neuron.2016.12.041

    • PubMed
    • Google Scholar
42. 1. Lau B
    2. Glimcher PW

    (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys

    Journal of the Experimental Analysis of Behavior 84:555–579.

    https://doi.org/10.1901/jeab.2005.110-04

    • PubMed
    • Google Scholar
43. 1. Leeson VC
    2. Robbins TW
    3. Matheson E
    4. Hutton SB
    5. Ron MA
    6. Barnes TRE
    7. Joyce EM

    (2009) Discrimination learning, reversal, and set-shifting in first-episode schizophrenia: stability over six years and specific associations with medication type and disorganization syndrome

    Biological Psychiatry 66:586–593.

    https://doi.org/10.1016/j.biopsych.2009.05.016

    • PubMed
    • Google Scholar
44. 1. Mackintosh NJ

    (1969) Further analysis of the overtraining reversal effect

    Journal of Comparative and Physiological Psychology 67:Suppl.

    https://doi.org/10.1037/h0026784

    • PubMed
    • Google Scholar
45. 1. Mansouri FA
    2. Freedman DJ
    3. Buckley MJ

    (2020) Emergence of abstract rules in the primate brain

    Nature Reviews. Neuroscience 21:595–610.

    https://doi.org/10.1038/s41583-020-0364-5

    • PubMed
    • Google Scholar
46. 1. Miller KJ
    2. Botvinick MM
    3. Brody CD

    (2017) Dorsal hippocampus contributes to model-based planning

    Nature Neuroscience 20:1269–1276.

    https://doi.org/10.1038/nn.4613

    • PubMed
    • Google Scholar
47. 1. Nowak M
    2. Sigmund K

    (1993) A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner’s Dilemma game

    Nature 364:56–58.

    https://doi.org/10.1038/364056a0

    • PubMed
    • Google Scholar
48. 1. Ohtsuki H
    2. Hauert C
    3. Lieberman E
    4. Nowak MA

    (2006) A simple rule for the evolution of cooperation on graphs and social networks

    Nature 441:502–505.

    https://doi.org/10.1038/nature04605

    • PubMed
    • Google Scholar
49. 1. Packard MG
    2. McGaugh JL

    (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning

    Neurobiology of Learning and Memory 65:65–72.

    https://doi.org/10.1006/nlme.1996.0007

    • PubMed
    • Google Scholar
50. 1. Palmer J
    2. Huk AC
    3. Shadlen MN

    (2005) The effect of stimulus strength on the speed and accuracy of a perceptual decision

    Journal of Vision 5:376–404.

    https://doi.org/10.1167/5.5.1

    • PubMed
    • Google Scholar
51. 1. Pereira TD
    2. Shaevitz JW
    3. Murthy M

    (2020) Quantifying behavior to understand the brain

    Nature Neuroscience 23:1537–1549.

    https://doi.org/10.1038/s41593-020-00734-z

    • PubMed
    • Google Scholar
52. 1. Pessiglione M
    2. Seymour B
    3. Flandin G
    4. Dolan RJ
    5. Frith CD

    (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans

    Nature 442:1042–1045.

    https://doi.org/10.1038/nature05051

    • PubMed
    • Google Scholar
53. 1. Peyrache A
    2. Khamassi M
    3. Benchenane K
    4. Wiener SI
    5. Battaglia FP

    (2009) Replay of rule-learning related neural patterns in the prefrontal cortex during sleep

    Nature Neuroscience 12:919–926.

    https://doi.org/10.1038/nn.2337

    • PubMed
    • Google Scholar
54. 1. Pezze M
    2. McGarrity S
    3. Mason R
    4. Fone KC
    5. Bast T

    (2014) Too little and too much: hypoactivation and disinhibition of medial prefrontal cortex cause attentional deficits

    The Journal of Neuroscience 34:7931–7946.

    https://doi.org/10.1523/JNEUROSCI.3450-13.2014

    • PubMed
    • Google Scholar
55. 1. Pouget A
    2. Beck JM
    3. Ma WJ
    4. Latham PE

    (2013) Probabilistic brains: knowns and unknowns

    Nature Neuroscience 16:1170–1178.

    https://doi.org/10.1038/nn.3495

    • PubMed
    • Google Scholar
56. 1. Powell NJ
    2. Redish AD

    (2016) Representational changes of latent strategies in rat medial prefrontal cortex precede changes in behaviour

    Nature Communications 7:12830.

    https://doi.org/10.1038/ncomms12830

    • PubMed
    • Google Scholar
57. 1. Raposo D
    2. Sheppard JP
    3. Schrater PR
    4. Churchland AK

    (2012) Multisensory decision-making in rats and humans

    The Journal of Neuroscience 32:3726–3735.

    https://doi.org/10.1523/JNEUROSCI.4998-11.2012

    • PubMed
    • Google Scholar
58. 1. Rich EL
    2. Shapiro ML

    (2007) Prelimbic/infralimbic inactivation impairs memory for multiple task switches, but not flexible selection of familiar tasks

    The Journal of Neuroscience 27:4747–4755.

    https://doi.org/10.1523/JNEUROSCI.0369-07.2007

    • PubMed
    • Google Scholar
59. 1. Rich EL
    2. Shapiro M

    (2009) Rat prefrontal cortical neurons selectively code strategy switches

    The Journal of Neuroscience 29:7208–7219.

    https://doi.org/10.1523/JNEUROSCI.6068-08.2009

    • PubMed
    • Google Scholar
60. 1. Roy NA
    2. Bak JH
    3. Laboratory IB
    4. Akrami A
    5. Brody CD
    6. Pillow JW

    (2021) Extracting the dynamics of behavior in sensory decision-making experiments

    Neuron 109:597–610.

    https://doi.org/10.1016/j.neuron.2020.12.004

    • PubMed
    • Google Scholar
61. 1. Rudebeck PH
    2. Behrens TE
    3. Kennerley SW
    4. Baxter MG
    5. Buckley MJ
    6. Walton ME
    7. Rushworth MFS

    (2008) Frontal cortex subregions play distinct roles in choices between actions and stimuli

    The Journal of Neuroscience 28:13775–13785.

    https://doi.org/10.1523/JNEUROSCI.3541-08.2008

    • PubMed
    • Google Scholar
62. 1. Russo E
    2. Ma T
    3. Spanagel R
    4. Durstewitz D
    5. Toutounji H
    6. Köhr G

    (2021) Coordinated prefrontal state transition leads extinction of reward-seeking behaviors

    The Journal of Neuroscience 41:2406–2419.

    https://doi.org/10.1523/JNEUROSCI.2588-20.2021

    • PubMed
    • Google Scholar
63. 1. Shiner T
    2. Symmonds M
    3. Guitart-Masip M
    4. Fleming SM
    5. Friston KJ
    6. Dolan RJ

    (2015) Dopamine, salience, and response set shifting in prefrontal cortex

    Cerebral Cortex 25:3629–3639.

    https://doi.org/10.1093/cercor/bhu210

    • PubMed
    • Google Scholar
64. 1. Singh A
    2. Peyrache A
    3. Humphries MD

    (2019) Medial prefrontal cortex population activity is plastic irrespective of learning

    The Journal of Neuroscience 39:3470–3483.

    https://doi.org/10.1523/JNEUROSCI.1370-17.2019

    • PubMed
    • Google Scholar
65. 1. Smith AC
    2. Frank LM
    3. Wirth S
    4. Yanike M
    5. Hu D
    6. Kubota Y
    7. Graybiel AM
    8. Suzuki WA
    9. Brown EN

    (2004) Dynamic analysis of learning in behavioral experiments

    The Journal of Neuroscience 24:447–461.

    https://doi.org/10.1523/JNEUROSCI.2908-03.2004

    • PubMed
    • Google Scholar
66. 1. Sohn H
    2. Narain D
    3. Meirhaeghe N
    4. Jazayeri M

    (2019) Bayesian computation through cortical latent dynamics

    Neuron 103:934–947.

    https://doi.org/10.1016/j.neuron.2019.06.012

    • PubMed
    • Google Scholar
67. 1. Suzuki WA
    2. Brown EN

    (2005) Behavioral and neurophysiological analyses of dynamic learning processes

    Behavioral and Cognitive Neuroscience Reviews 4:67–95.

    https://doi.org/10.1177/1534582305280030

    • PubMed
    • Google Scholar
68. 1. Tait DS
    2. Bowman EM
    3. Neuwirth LS
    4. Brown VJ

    (2018) Assessment of intradimensional/extradimensional attentional set-shifting in rats

    Neuroscience and Biobehavioral Reviews 89:72–84.

    https://doi.org/10.1016/j.neubiorev.2018.02.013

    • PubMed
    • Google Scholar
69. 1. Trepka E
    2. Spitmaan M
    3. Bari BA
    4. Costa VD
    5. Cohen JY
    6. Soltani A

    (2021) Entropy-based metrics for predicting choice behavior based on local response to reward

    Nature Communications 12:6567.

    https://doi.org/10.1038/s41467-021-26784-w

    • PubMed
    • Google Scholar
70. 1. van Wingerden M
    2. Vinck M
    3. Lankelma J
    4. Pennartz CMA

    (2010) Theta-band phase locking of orbitofrontal neurons during reward expectancy

    The Journal of Neuroscience 30:7078–7087.

    https://doi.org/10.1523/JNEUROSCI.3860-09.2010

    • PubMed
    • Google Scholar
71. 1. Wang J
    2. Tait DS
    3. Brown VJ
    4. Bowman EM

    (2019) Exacerbation of the credit assignment problem in rats with lesions of the medial prefrontal cortex is revealed by Bayesian analysis of behavior in the pre-solution period of learning

    Behavioural Brain Research 372:112037.

    https://doi.org/10.1016/j.bbr.2019.112037

    • PubMed
    • Google Scholar
72. 1. Wu Z
    2. Kavanova M
    3. Hickman L
    4. Boschin EA
    5. Galeazzi JM
    6. Verhagen L
    7. Ainsworth M
    8. Pedreira C
    9. Buckley MJ

    (2021) Low-beta repetitive transcranial magnetic stimulation to human dorsolateral prefrontal cortex during object recognition memory sample presentation, at a task-related frequency observed in local field potentials in homologous macaque cortex, impairs subsequent recollection but not familiarity

    The European Journal of Neuroscience 54:7918–7945.

    https://doi.org/10.1111/ejn.15535

    • PubMed
    • Google Scholar
73. 1. Yin HH
    2. Knowlton BJ

    (2004) Contributions of striatal subregions to place and response learning

    Learning & Memory 11:459–463.

    https://doi.org/10.1101/lm.81004

    • PubMed
    • Google Scholar

Author details

1. Silvia Maggi

   School of Psychology, University of Nottingham, Nottingham, United Kingdom

   Contribution

   Data curation, Software, Formal analysis, Validation, Visualization, Writing - original draft

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-6533-3509

2. Rebecca M Hock

   School of Psychology, University of Nottingham, Nottingham, United Kingdom

   Contribution

   Investigation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0917-570X

3. Martin O'Neill

   1. School of Psychology, University of Nottingham, Nottingham, United Kingdom
   2. Department of Health & Nutritional Sciences, Atlantic Technological University, Sligo, Ireland

   Contribution

   Investigation, Writing - review and editing

   Competing interests

   No competing interests declared

4. Mark Buckley

   Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom

   Contribution

   Supervision, Funding acquisition, Investigation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-7455-8486

5. Paula M Moran

   1. School of Psychology, University of Nottingham, Nottingham, United Kingdom
   2. Department of Neuroscience, University of Nottingham, Nottingham, United Kingdom

   Contribution

   Supervision, Writing - review and editing

   Competing interests

   No competing interests declared

6. Tobias Bast

   1. School of Psychology, University of Nottingham, Nottingham, United Kingdom
   2. Department of Neuroscience, University of Nottingham, Nottingham, United Kingdom

   Contribution

   Supervision, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-6163-3229

7. Musa Sami

   Institute of Mental Health, University of Nottingham, Nottingham, United Kingdom

   Contribution

   Investigation, Writing - review and editing

   Competing interests

   No competing interests declared

8. Mark D Humphries

   School of Psychology, University of Nottingham, Nottingham, United Kingdom

   Contribution

   Conceptualization, Software, Formal analysis, Supervision, Funding acquisition, Visualization, Methodology, Writing - original draft, Project administration

   For correspondence

   mark.humphries@nottingham.ac.uk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-1906-2581

• 1,079

  views

• 169

  downloads

• 1

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.