‘Should I stay or should I go?‘ refers not only to an iconic 1980s punk anthem but also the fundamental dilemma all animals face in uncertain or unstable environments. Should someone buy coffee from the cafe that serves their favorite roast or try the new cafe that opened down the street? If their favorite drink is bitter one day, is that a sign to switch to a new blend or is one subpar experience inadequate to prompt a switch? Ultimately, these decisions converge to a single predicament: whether we choose an action that we believe is likely to produce desirable results (i.e. exploit) or risk choosing another action that is less certain, on the chance that it will produce a more positive outcome (i.e. explore) (O’Reilly, 2013). Ultimately, this is the problem of knowing when to change your mind.

The shift of a decision policy from exploratory to exploitative states is driven by environmental context. To illustrate this, Figure 1A shows what happens when a simple reinforcement learning (RL) agent tries to maximize reward in a dynamic variant of the two-armed bandit task (Sutton and Barto, 1998; see Materials and methods). Here, the relative difference in reward probability for the two actions (conflict) and the frequency of a change in the optimal action (volatility) were independently manipulated. For each level of conflict and volatility, a set of tabular Q-learning (Sutton and Barto, 1998) agents played the task with learning rate held constant while the degree of randomness of the selection policy ($\beta$ in a Softmax function) varied. The agent that returned the most rewards was identified as the agent with the best exploration-exploitation balance. Increasing either form of uncertainty led to selecting agents with more random or exploratory selection policies (Figure 1A). As the value of the optimal choice decreases relative to the value of a suboptimal choice (conflict increases), the learner exploits what she already knows. Action values grow unstable (volatility increases) when the clarity of the optimal choice is constant (constant conflict), and the learner is biased toward exploration (Bland and Schaefer, 2012). As these two forms of uncertainty change together, the gradient of action selection strategy also changes.

Figure 1

Download asset Open asset

Knowing how decision policies shift in the face of dynamic environments requires looking at the algorithmic properties of the policy itself. One popular set of algorithms for describing the dynamics of decision making are accumulation-to-bound processes like the drift-diffusion model (DDM; Ratcliff, 1978). The normative form of the DDM proposes that a decision between two choices is described by a noisy accumulation process that drifts toward one of two decision boundaries at a specific rate (Figure 1B). Two parameters of this model are critical in determining the degree of randomness of a selection policy: the rate of evidence accumulation (drift rate; $v$) and the amount of information required to make a decision (boundary height; $a$). For example, decreasing the drift rate and increasing the boundary height leads to more random decisions (Figure 1C), with the speed of these decisions depending on the ratio of the two parameters (Figure 1D). Thus, exploratory policies can result in either fast or slow decisions, depending on the relative configuration of drift rate and boundary height.

Are the parameters that govern accumulation of evidence for decision making modifiable? Previous modeling work has shown that the parameters of a DDM process can be modulated by feedback signals and choice history (Pedersen et al., 2017; Ratcliff and Frank, 2012; Dunovan and Verstynen, 2019; Dunovan et al., 2019; Mendonça et al., 2020; Urai et al., 2018) with different mechanisms for adapting the drift rate and the boundary height. In value-based decision-making tasks where the statistics of sensory signals are equivalent for all actions, drift rate fluctuations appear to track the relative value of an action or the value difference between actions (Dunovan et al., 2019; Mikhael and Bogacz, 2016; Bariselli et al., 2019; Rubin et al., 2021). In contrast to value estimation, selection errors in this context have been linked to changes in the boundary height (Forstmann et al., 2008; Forstmann et al., 2010; Bogacz et al., 2010; Herz et al., 2016; Herz et al., 2017; Dunovan et al., 2019; Dunovan and Verstynen, 2019) and internal estimates of environmental change (Nassar et al., 2010; Wilson and Niv, 2011; Nassar et al., 2012; Behrens et al., 2007).

Given the adaptive sensitivity of the drift rate and the boundary height to value estimation and selection, respectively, these decision parameters define unique states on a surface of fast or slow and exploratory or exploitative decision policies. These policies, in turn, adaptively reconfigure based on current environmental feedback signals by modulating value estimation and the rate of selection errors (Figure 1E). Agents can move along the surface of decision policies, from exploitative states (bright colors, Figure 1E) to different types of exploratory states (darker colors, Figure 1E), as they commit a greater number of selection errors prompted by change in action-outcome contingencies. As the system relearns properties of the environment, the decision policy migrates along the surface to return to an exploitative state until a change occurs again.

One plausible neural mechanism for this migration along the surface of selection policies is the locus coereleus norepinephrine (LC-NE) system, which has been linked to adaptive behavioral variability in response to uncertainty (Urai et al., 2017; Dayan and Yu, 2006; Bouret and Sara, 2005). The LC-NE system has two distinct modes (Aston-Jones and Bloom, 1981) that map onto distinct decision states (Aston-Jones and Cohen, 2005). In the phasic mode, a burst of LC activity results in a global, temporally precise release of NE. This increases the gain on cortical processing and encourages exploitation. In the tonic mode, NE is released without the temporal precision of the phasic mode, increasing baseline NE (Aston-Jones and Bloom, 1981). This encourages disengagement from the current task and facilitates exploration. The dynamic fluctuation of these two modes is thought to optimize the trade-off between the exploitation of stable sources of reward and the exploration of potentially better options (Aston-Jones and Cohen, 2005). Thus the LC-NE system, which can be indirectly measured by fluctuations in pupil diameter (Aston-Jones and Cohen, 2005; Jepma and Nieuwenhuis, 2011), may be a central mechanism for modulating selection policies.

We investigated the malleability of decision policies as the environment necessitates a change of mind as to what constitutes the ‘best’ decision. To control environmental uncertainty, we manipulated the volatility of changes in action-outcome contingencies (i.e. which of two targets returns the most rewards), as well as ambiguity in optimal choice (conflict), while human participants performed a dynamic variant of the two-armed bandit task (Sutton and Barto, 2018). We predicted that, in response to suspected changes in action-outcome contingencies, humans would exhibit a stereotyped adjustment in the drift rate and boundary height that pushes decisions from certain, exploitative states to uncertain, exploratory states and back again (Figure 1E). In addition, using pupillary data, we explored whether the LC-NE system covaries with shifts of the boundary height in response to a change in action outcomes to facilitate exploration, consistent with prior studies (Keung et al., 2019; Murphy et al., 2014; Cavanagh et al., 2014).

Across two experiments, we used a dynamic two-armed bandit task with equivalent sensory reliability across arms to independently manipulate the reward conflict and the volatility of action outcomes in order to measure how underlying decision processes respond to changes in action-outcome contingencies (see Stimuli and Procedure). Both of these experiments shared a common feedback structure. Participants were asked to select either the left or right target presented on the screen using the corresponding key on a response box. Rewards were probabilistically determined for each target and, if a reward was delivered, it was sampled from a Gaussian distribution. The optimally rewarding target delivered reward with a predetermined probability ($P(optimal)$) and the suboptimal target gave reward with the inverse probability ($1-P(optimal)$). After a delay determined by the rate parameter of a Poisson distribution ($\lambda$), the reward probabilities for the optimal and suboptimal targets would switch.

In Experiment 1, 24 participants completed four sessions (high and low conflict; high and low volatility) each composed of 600 trials. During each session, they were asked to select one of two coin boxes (Exp. 1: Figure 2A). The levels of conflict and volatility for all four conditions in Experiment 1 are shown as gray dots in Figure 2C. Experiment 2 was a replication of Experiment 1 with more extensive within-subject sampling of conflict and volatility, as well as the inclusion of pupilometry as a proxy for measuring LC-NE dynamics. In Experiment 2, participants were asked to choose between one of two Greebles (one male, one female). Each Greeble probabilistically delivered a monetary reward (Exp. 2: Figure 2B). Participants were trained to discriminate between male and female Greebles prior to testing to prevent errors in perceptual discrimination from interfering with selection on the basis of value estimation. Four participants completed nine sessions composed of 400 trials each, generating 3600 trials in total per subject. The levels of conflict and volatility for all nine conditions in Experiment 2 are shown as black dots in Figure 2C. Importantly, Experiment 2 manipulated the same forms of uncertainty as Experiment 1, but had different perceptual features and more expansively sampled the space of conflict and volatility. Given the similarity in design, the behavioral results for both of these experiments are presented together below.

Figure 2

Download asset Open asset

The influence of ambiguity and instability on speed and accuracy

We first looked at overall speed and accuracy effects in both Experiments 1 and 2. In Experiment 1, accuracy (i.e. optimal choice selection) suffered as the optimal choice grew more ambiguous, with accuracy in the low conflict condition being 1.2 times higher than what is observed in the high conflict condition (Figure 3A; $\beta =1.213$, 95% CI: 1.192, 1.235, z = 21.36, p < 2e-16). In contrast, increasing conflict had no observable impact on overall reaction times (Figure 3A; $\beta =-6.902e^{-}5$, 95% CI: –0.002, 0.002, t = −0.06, p = 0.951). As expected, participants also became less accurate as the instability of action outcomes (i.e. volatility) grew (Figure 3B; $\beta =0.092$, 95% CI: 0.077, 0.111, z = 10.36, p < 2e-16). Under volatile conditions, participants also took slightly longer to make a decision ($\beta =-0.012$, 95% CI: –0.015,–.010, t = −10.80, p < 2e-16); however, while this effect on reaction times was statistically reliable, the impact of volatility on reaction times was weak (increasing volatility increased reaction time by ∼13 ms on average; Figure 3B).

Figure 3

Download asset Open asset

Experiment 2 served as a high powered test of whether the effects we observed in Experiment 1 were replicable at the within-subject level. Because Experiment 2 independently manipulated conflict and volatility, we were able to test whether conflict and volatility interacted to affect behavior. We found similar effects of conflict and volatility on accuracy as we observed in Experiment 1 (Figure 3C). Accuracy increased as conflict decreased (i.e. as the probability of reward increased; $\beta$=0.223, 95% CI = 0.189,0.256, z = 12.757, p<2e-16). As the environment grew less volatile, accuracy increased ($\beta$ = 0.101, 95% CI = 0.066,0.14, z = 5.828, p = 5.6e-09). We did not observe an interaction of conflict and volatility on accuracy ($\beta$ = 0.024, 95% CI = −0.013, 0.058, z = 1.364, p = 0.173).

However, we did find that conflict and volatility interacted to affect reaction time (RT; $\beta$=−0.002, 95% CI = −0.004, –0.001, t = −3.084, p = 0.002), with a linear increase in reaction time as the environment grew less volatile and conflict was highest (when $p(r)=0.65\overline{RT}=0.472,0.480,0.493$ as a function of $\lambda$; see Figure 3D for RT distributions). When conflict was moderate (${\displaystyle p(r)=0.75}$) or low ($p(r)=0.85$), volatility had a nonlinear effect on RTs. Here, reaction times decreased when volatility was moderate ($\overline{RT}=0.483$ when $\lambda$=20 and ${\displaystyle p(r)=0.75}$ when $\lambda$=20 and $p(r)=0.85$). Reaction times increased to approximately the same extent within moderate or low conflict conditions when volatility was high ($\overline{RT}=0.499$ when $\lambda$=10 and ${\displaystyle p(r)=75}$ when $\lambda$=10 and $p(r)=0.85$) and when volatility was low ($\overline{RT}=0.506$ when $\lambda =30$ and ${\displaystyle p(r)=0.75}$ when $\lambda =30$ and ${\displaystyle p(r)=0.85}$), with an increase in baseline reaction times when conflict was low relative to moderate ($\overline{RT}=0.527$ when ${\displaystyle p(r)=0.85}$ when $p(r)=0.75$; see Appendix 1—figure 1 for interaction visualization).

At the gross level, over all trials within an experimental condition, increasing the ambiguity of the optimal choice (conflict) and increasing the instability of action outcomes (volatility) decreases the probability of selecting the optimal choice. Reaction time effects were inconsistent, with a negligible effect of volatility in Experiment 1. Experiment two revealed that volatility and conflict interact to influence reaction times in complex ways. However, because trials where action-outcome contingencies change are so infrequent, even under high volatility conditions, these overall effects on speed and accuracy may be masking more subtle behavioral dynamics in response to feedback changes. We adopt a more focal, model-based analysis in the next section to clarify these peri-change point dynamics.

Tracking estimates of action value and environmental volatility

We calculated trial-by-trial estimates of two ideal observer parameters of environmental states (see Cognitive model for calculation details; Nassar et al., 2010; Vaghi et al., 2017). Belief in the value difference ($\mathrm{\Delta }B$) reflects the difference between the learned values of the optimal and suboptimal targets. For ease of interpretation, we refer to the converse of belief as doubt, such that when belief decreases doubt increases. $\mathrm{\Delta }B$ thus reflects a local estimate of uncertainty regarding the choices themselves. To capture the estimated probability of fundamental shifts in action values, we calculated how often the same action gave a different reward (change point probability; $\mathrm{\Omega }$). Here, $\mathrm{\Omega }$ reflects a global estimate of uncertainty in the environment, specifically the uncertainty in response contingencies. We used the data from Experiment 1 to assess how well these learning estimates captured our imposed manipulations, and observed similar results in Experiment 2 (Appendix 1—figure 4).

In Experiment 1, we observed a sharp decrease in $\mathrm{\Delta }B$ after a switch in action outcomes and a gradual return to asymptotic values (Figure 4A) with a decreased difference in reward probability resulting in increased doubt (Figure 4B; $\beta =0.216$, 95% CI:0.206, 0.224, t = 46.24, p < 2e-16). As expected, less volatile conditions allowed the learner to more fully update her belief in the value of the optimal choice over all trials ($\beta =0.058$, 95% CI:0.050, 0.068, t = 12.32, p < 2e-16), though to a smaller degree than low conflict conditions allowed (see Figure 4B). Increasing volatility resulted in a sharp increase in the estimate of $\mathrm{\Omega }$ at the onset of a change point with a quick return to a baseline estimate of change (Figure 4C). Notably, this estimate of $\mathrm{\Omega }$ was more sensitive to change points when conditions were relatively volatile, with a more pronounced peak in response to a change under high volatility conditions than under low volatility conditions (Figure 4C). Correspondingly, over all trials, $\mathrm{\Omega }$ was higher under more volatile conditions (Figure 4D, $\beta =-0.022$, 95% CI:−0.023,–0.020, t = −30.74, p < 2e-16) indicating sensitivity to the increased frequency of action outcome switches in the reward schedule.

Figure 4

Download asset Open asset

When the identity of the optimal choice was clear (i.e. when conflict was low), the estimate of $\mathrm{\Omega }$ was more sensitive to the presence of a true change point than when the optimal choice was ambiguous (i.e. when conflict was high) (Figure 4C and D). This observation is consistent with the idea that increasing the difficulty of value estimation and, thereby, the assignment of value to a given choice also impairs change point sensitivity. Interestingly, increasing conflict nevertheless resulted in a net increase in $\mathrm{\Omega }$ calculated over all trials (Figure 4D; $\beta$=−0.006, 95 %CI:−0.007,–0.004, t = −8.64, p < 2e-16), likely because higher conflict conditions increased the baseline estimate of change instead of enhancing sensitivity to true change points (see change point response and relative baseline values for the high conflict condition in Figure 4C). Here, the system conservatively over-estimates the volatility of action outcomes, assuming a slightly greater frequency of changes in the probability of reward for the optimal choice than we imposed (actual proportion of change points for high conflict condition: ${\displaystyle 0.041\pm 0.004}$; estimated ${\displaystyle \mathrm{\Omega }}$).

Reassuringly, net change point probability was much greater when change points were more frequent (see increased $\mathrm{\Omega }$ estimates for high volatility conditions over high conflict conditions in Figure 4D). These results suggest that our formulation of these ideal observer estimates adequately captures our manipulation of volatility and conflict at a continuous level.

Thus, these ideal observer parameters show a reliable response to a change in action-outcome contingencies. The difference in value belief decreases, or doubt increases, when a change point occurs and slowly recovers over the course of six to eight trials as participants learn new action-outcome contingencies. The initial drop in belief difference is deeper and the recovery time after a change point is slower in conditions with greater overall uncertainty (i.e. under high conflict and high volatility). In contrast, internal estimates that a change has occurred briefly spike at a change point, indicating that participants can reliably detect that something has changed, and quickly settle after a few trials. Interestingly, net change point probability estimates are higher in the conditions with higher uncertainty (high conflict, high volatility), likely reflecting increased vigilance for changes in those conditions. In the next section, we explore how the underlying parameters of the decision process itself respond to local changes in action-outcome contingencies.

Different forms of uncertainty impact distinct decision processes

Our next goal was to test which decision parameters were sensitive to a change point. To this end, we estimated the change point evoked response of the boundary height $a$, drift rate $v$, non-decision time $t$, starting bias $z$, and drift criterion $dc$ for each trial surrounding the change point. To detect changes in the change-point-evoked distributions for each decision parameter, we evaluated whether the sequential distributions evoked by each trial were significantly different, beginning with the trial preceding the change point and ending three trials after the change point. For example, if the 95% CI of the $z$ distribution evoked on the trial prior to the change point overlapped with the 95% CI of the distribution evoked on the change point and so on for all successive trials considered, then we would conclude that $z$ failed to show change point sensitivity (see Hierarchical drift diffusion modeling for details). To select the model that best accounted for the data, we compared the deviance information criterion (DIC) scores (Spiegelhalter et al., 2002) for these models. DIC scores provide a measure of model fit adjusted for model complexity and quantify information loss. A lower DIC score indicates a model that loses less information. Here, a difference of ≤ two points from the lowest-scoring model cannot rule out the higher scoring model; a difference of 3–7 points suggests that the higher scoring model has considerably less support; and a difference of 10 points suggests essentially no support for the higher scoring model (Spiegelhalter et al., 2002; Burnham and Anderson, 1998).

Under this analysis, we found that only the boundary height and drift rate showed change point sensitivity as defined above. The drift rate showed a clear, persistent separation between trial-specific distributions, with a rapid decrease at the onset of the change point ($t_{-1}$ 95% CI = 1.021, 1.218; $t_0$=−0.972, –0.779) and a return to baseline values thereafter ($t_1$ = −0.656,,–0.46; $t_2$=0.039, 0.241; $t_3$=0.411, 0.616; Figure 5A). The boundary height showed a transient response to the change point, spiking ($t_{-1}$ 95% CI = 0.792, 0.819; $t_0$=0.820, 0.847) and then dropping to baseline levels ($t_1$ = 0.789, 0.815; $t_2$=0.783, 0.811; $t_3$=0.780, 0.808; Figure 5B).

Figure 5

Download asset Open asset

The remainder of the decision parameters showed no change point sensitivity. Non-decision time showed no clear response ($t_{-1}$ 95% CI = 0.183, 0.188; $t_0$=0.183, 0.186; $t_1$=0.184, 0.191; $t_2$=0.181, 0.186; $t_3$=0.183, 0.186; Figure 5C) along with the starting bias ($t_{-1}$ 95% CI = −0.112,,–0.01; $t_0$=−0.098, 0.002; $t_1$=−0.090, 0.008; $t_2$=−0.069, 0.032; $t_3$=−0.055, 0.045; Figure 5D) and the drift criterion ($t_{-1}$ 95% CI = 0.229, 0.439; $t_0$=0.175, 0.374; $t_1$=0.223, 0.435; $t_2$=0.245, 0.458; $t_3$=0.244, 0.464; Figure 5E).

Further, models fitting drift rate and boundary height lost the least null-model-adjusted information relative to models of the change-point-evoked response for the other parameters, showing that a change-point-evoked decrease in drift rate and spike in the boundary height best accounted for our observational data in comparison to all alternatives ($\mathrm{\Delta }DI{C}_{null}$ for $v$ = –978 and $\mathrm{\Delta }DI{C}_{null}$ for $a$ = –13.7; see Figure 5F).

Given that only the drift rate and boundary height showed change point sensitivity, we next focused on how those two parameters related to internal estimates of change and conflict in both experiments. Recall that we used the ideal observer parameters $\mathrm{\Delta }B$ and $\mathrm{\Omega }$ as proxies for internal estimates of belief in the difference in learned target values and change point probability, respectively. This provided a continuous quantification of our manipulation of conflict and volatility (see Tracking estimates of action value and environmental volatility). Experiment 2 provided an intensively sampled within-subject test of the change-point-evoked mapping between decision processes and these ideal observer estimates.

In order to determine the nature of the mapping between the ideal observer parameters and the change-point sensitive decision parameters, we estimated single and dual-parameter models mapping $\mathrm{\Delta }B$ and $\mathrm{\Omega }$ and the change-point-sensitive decision parameters, drift rate and boundary height, and examined the fit of these models to our data. We found that the model mapping $\mathrm{\Delta }B$ to drift rate and $\mathrm{\Omega }$ to boundary height provided the best fit in Experiment 1 ($\mathrm{\Delta }DI{C}_{null}=-2698.0$; left panel of Table 1).

Table 1

Experiment 1
	${\displaystyle \mathrm{\Delta }B}$	${\displaystyle \mathrm{\Omega }}$	DIC	${\displaystyle \mathrm{\Delta }{\text{DIC}}_{\text{null}}}$	${\displaystyle \mathrm{\Delta }DI{C}_{best}}$
*I	$v$	$a$	–18643.9	–2698.0	0.0
II	$a$	$v$	–16265.6	–319.7	2378.3
III	–	$v$	–16180.5	–234.7	2463.3
IV	$v$	–	–18630.8	–2684.9	13.1
V	–	$a$	–15949.20	–3.4	2694.7
VI	$a$	–	–16032.8	–87.0	2611.1
VII	–	–	–15945.8	0.0	2698.0
Experiment 2
	$\mathrm{\Delta }B$	$\mathrm{\Omega }$	$\mathrm{\Delta }{\text{DIC}}_{\text{null}}$	$\mathrm{\Delta }{\text{DIC}}_{\text{best}}$
*∼I	$v$	$a$	–90.3 ± 71.7	1.0 ± 0.8
II	$a$	$v$	–7.6 ± 13.1	83.8 ± 60.5
III	–	$v$	–8.5 ± 13.1	82.9 ± 61.4
*∼IV	$v$	–	–90.8 ± 71.0	0.5 ± 1.1
V	–	$a$	0.3 ± 2.5	91.6 ± 70.6
VI	$a$	–	0.95 ± 1.4	92.3 ± 70.9
VII	–	–	0 ± 0	91.3 ± 71.5

To test whether this mapping was preserved in an independent data set, we performed the same model comparison procedure for Experiment 2. Because Experiment 2 followed a replication-based design, we fit a separate model to each subject to assess the replicability of the best fitting model from Experiment 1. While we found support for the model mapping $\mathrm{\Delta }B$ to drift rate and $\mathrm{\Omega }$ to boundary height, we also found that the DIC scores for the single-parameter model mapping $\mathrm{\Delta }B$ to $v$ alone fit the data equally well (see bottom panel of Table 1 for summary statistics and Appendix 3—table 1). Altogether, this suggests that we have strong evidentiary support for a mapping between value-driven belief and drift rate (Figure 6A, blue). However, the support for a mapping between change point probability and boundary height (Figure 6A, red), while robustly present in Experiment 1, fails to appear when tested in an independent data set.

Figure 6

Download asset Open asset

For a more granular assessment of how drift rate and boundary height respond to a change point, we quantified the change-point-evoked effect of $\mathrm{\Delta }B$ and $\mathrm{\Omega }$ on drift rate and boundary height, respectively, for both experiments (see Hierarchical drift diffusion modeling for details). In Experiment 1, we found that the rate of evidence accumulation, $v$, increased with the belief in the value of the optimal choice relative to a change point (${\beta }_{v\sim \mathrm{\Delta }B}=0.576$, 95% CI: 0.544, 0.609, empirical $p=0.000$; Figure 6B, left panel). The boundary height increased with change point probability (${\beta }_{a\sim \mathrm{\Omega }}=0.046$, 95% CI: 0.005, 0.088, empirical $p=0.001$; Figure 6B, right panel).

Experiment two showed similar, but attenuated, results, with drift rate increasing with $\mathrm{\Delta }B$ (${\beta }_{v\sim \mathrm{\Delta }B}=0.112$, 95% CI: 0.016, 0.227, empirical $p=0.004$; Figure 6B, inset panel on left) and an unreliable effect of $\mathrm{\Omega }$ on boundary height (${\beta }_{a\sim \mathrm{\Omega }}=-0.036$, 95% CI: –0.155, 0.097, empirical $p=0.282$; Figure 6B, inset panel on right). Therefore, as the belief in the value of the optimal choice approaches the reward value for the optimal choice, the rate of information accumulation increases. An internal estimate of change point probability weakly increases the amount of information required to make a decision, although this latter effect is less reliable.

Altogether, these results suggest a drift rate mechanism for adaptation to change that may also combine with boundary height dynamics (Figure 6A). However, the strength of the drift rate response weakened and the boundary height response was statistically unreliable in Experiment 2 (Figure 6B inset panels). When a change point is detected and the threshold for committing to a choice (a) responds, it shows a weak, transient increase. At the same time, the drift rate approaches zero, allowing time for the decision process to diffuse and encouraging a random selection. As the learner accrues information about the new optimal choice, the rate of information accumulation slowly recovers to asymptotic levels, with the decision process assuming a more directed path toward the choice that has accrued evidence for reward. Together, the changes in these underlying decision processes, largely driven by drift rate dynamics, point to a mechanism for gathering information in a relatively slow, unbiased manner shortly after the learner suspects she should update her valuation. We now explore these dynamics in more detail in the next section.

Environmental instability prompts a stereotyped decision trajectory

So far, we have established that both the drift rate and the boundary height can be independently manipulated by two different estimates of environmental uncertainty with different temporal dynamics, although this effect reduces to drift rate dynamics in Experiment 2. This suggests that a change in action-outcome contingencies prompts a unique trajectory through the space of possible decision policies (Figure 1E).

To visualize this trajectory, we plot the temporal relationship between drift rate and boundary height beginning with the trial prior to the change point and ending three trials after the change point (Figure 7A). To clearly visualize the distribution of the change-point driven response in the relationship between drift rate and boundary height over time, we also represent the trialwise shift in these two decision variables as vectors. The trial-by-trial estimates of drift rate and boundary height were taken from the best model of the fitted change-point-evoked response and z-scored (see Different forms of uncertainty impact distinct decision processes for model selection). Then the difference between each sequential set of boundary height and drift rate coordinates, $(a,v)$, was calculated to produce a vector length. The arc tangent between these differenced values was computed to yield an angle in radians between sequential decision vectors, concisely representing the overall decision dynamics ($\theta$, Figure 7B; see Decision vector representation for methodological details).

Figure 7

Download asset Open asset

For Experiment 1, following a shift in response contingencies, the navigation of this decision surface follows a stereotyped pattern. The boundary height spikes and drift rate decreases rapidly, gradually recovering and stabilizing over time (see the trial prior to the change point in Figure 7A). This decision trajectory is robust in Experiment 1 (Figure 7B, top panel).

Here, we find that the distribution of $\theta$ prior to a change point averages to ∼300°, sharply changes in response to the observation of a change point (∼165°) and steadily returns to values prior to the onset of a change (main panels in Figure 7B). One trial after the change point, drift rate sharply decreases and boundary height spikes, after which boundary height quickly recovers and drift rate steadily progresses toward its baseline value.

However, this trajectory is substantially more variable in Experiment 2, with most of the response restricted to the drift rate dimension and inconsistent trajectories along the boundary height dimension (Figure 7B, lower panel). Here, the distribution of $\theta$ prior to a change point averages to ∼270° and shifts to ∼90° with the observation of a change. In both experiments, we find that the decision trajectory quickly responds to a shift in action outcomes and also quickly recovers and stabilizes.

Having characterized the change-point-evoked trajectory through the range of decision policies, we next asked whether conditions of increased volatility and increased conflict might modify its path. To this end, we conducted a comparison of a null model with models specifying the change-point evoked response alone and this evoked response as a function of conflict and volatility. To estimate this relationship between drift rate and boundary height, we used Bayesian circular regression (Mulder and Klugkist, 2017). First, we tested the null hypothesis that the decision dynamics (the relationship between drift rate and boundary height; $\theta$) were solely a function of the intercept, or the average of the decision dynamics $\theta$:

$\theta ={\beta }_0$

We call this the null model.

To test the hypothesis that decision dynamics varied solely as a function of time after a switch in action-outcome contingencies, we estimated the change in $(a,v)$ coordinates ($\theta$) relative to a change point, with the time scale of consideration determined by the results of a stability analysis from Experiment 1 (see Model proposals and evaluation; Appendix 1—figure 5):

$\theta ={\beta }_0+{\beta }_{\mathrm{\Delta }{t}_{i:3}}$

We call this the evoked response model.

Our model comparison logic was as follows. We first evaluated whether the posterior probability of the evoked response model was greater than that for the null model. This would suggest that time relative to a change point alone is a better predictor of decision dynamics than the average response. If the posterior probability of the evoked response model reliably exceeded the posterior probability of the null model, we then quantified the evidence for alternative models relative to the evoked response model. The sole effect of time relative to a change point was then framed as the new null hypothesis.

We used Bayes Factors to quantify the ratio of evidence for two competing hypotheses. If the ratio is close to 1, then the evidence is equivocal. As the ratio grows more positive, there is greater evidence for the model specified in the numerator, and if the ratio is less than 1, then there is evidence for the model specified in the denominator (Jeffreys, 1998). Evidence for the null hypothesis is denoted $B{F}_{01}$ and evidence for the alternative hypothesis is denoted $B{F}_{10}$. Because Experiment 2 took a within-subject approach, a separate model was fit for each participant for all proposed models.

To determine whether volatility and conflict affected these peri-change decision dynamics, we modeled changes in decision policy on the drift rate and boundary height surface as a function of $\lambda$ and $p$, where $\lambda$ corresponds to the average period of stability and $p$ corresponds to the mean probability of reward for the optimal choice (see Figure 8 for the full set of models considered). We explored the potential influence of volatility and conflict on the relationship between drift rate and boundary height by examining the posterior probability for each hypothesized model given the set of alternative hypotheses (Model proposals and evaluation; Figure 8A). We found that the evoked response model describing the relationship between shifts in decision parameters and time relative to a change point was more probable than the null model (see Figure 8A).

Figure 8

Download asset Open asset

We also present the evidence for the null model against each alternative model as a Bayes Factor ($B{F}_{01}$) (Figure 8B). The 95% confidence interval for the $BF01$ comparing the ratio of evidence for the null model and the evoked response model specifying time-dependent effects of volatility included 1, suggesting inconclusive evidence for either of these models. Likewise, the 95% confidence interval for the $B{F}_{01}$ comparing the evidence for the null model against the model specifying change-point-evoked effects of conflict included 1, suggesting no substantive difference between them. Given the equivocal evidence for these two models we excluded them from further comparison with the evoked response model.

The remainder of the models had substantially negative $B{F}_{01}$ values (Figure 8B), suggesting that they better fit the data than the null model and allowing them to survive to the next stage of analysis. To evaluate the hypothesis that time alone best accounted for the data, we computed the $B{F}_{01}$ for the evoked response model against the surviving models from the null model analysis. We find that, for all the remaining models, the $B{F}_{01}$ is substantially positive (Figure 8C), indicating that the evoked response model best accounted for the data (posterior probability of evoked response model given the set of models considered: $0.76\pm 0.473$; posterior prob. for 3/4 participants > 0.99).

These analyses suggest that the relationship between the rate of evidence accumulation and the boundary height is only related to the change point itself. We find no evidence to suggest that changing the degree of volatility or changing the degree of conflict changes the path of the decision policy following a change point. Thus, the stereotyped response of the decision policy is solely dependent on the presence of a change point rather than either the history of change point frequency or the history of optimal choice ambiguity. Note that while the ideal observer estimates respond to our conditional manipulations of volatility and conflict, the decision dynamics $\theta$ we observe do not reflect these effects. This is due to the noisy, imperfect correspondence between the ideal observer signals and $a$ and $v$. This suggests that adaptation to environmental changes in action-outcome contingencies involves a rapid, coordinated increase in the relationship between the amount of information needed to make a decision and a decrease in the rate of information accumulation, with a stereotyped return to a stable baseline soon thereafter until another change occurs.

No evidence for locus-coeruleus norepinephrine (LC-NE) system contribution to the decision trajectory

The LC-NE system is known to modulate exploration states under uncertainty and pupil diameter shows a tight correspondence with LC neuron firing rate (Aston-Jones and Cohen, 2005; Rajkowski et al., 1994), with changes in pupil diameter indexing the explore-exploit decision state (Jepma and Nieuwenhuis, 2011). Similar to the classic Yerkes-Dodson curve relating arousal to performance (Yerkes and Dodson, 1908), performance is optimal when tonic LC activity is moderate and phasic LC activity increases following a goal-related stimulus (Aston-Jones et al., 1999, but see Joshi et al., 2016 for an exception). Because of this link between LC-NE and the regulation of behavioral variability in response to uncertainty, we expected that LC-NE system responses, as recorded by pupil diameter, would associate with environmental uncertainty and the trajectory through decision policy space following a change in action-contingencies. Specifically, if the LC-NE system were sensitive to a change in the optimal choice then we should observe a moderate spike in phasic activity following a change in action-outcome contingencies. Note that we do not observe previously established links between exploratory choice behavior and the pupillary response (Jepma and Nieuwenhuis, 2011; Murphy et al., 2011; van Kempen et al., 2019). We ask the reader to titrate their interpretation of these pupillary data accordingly.

We characterized the evoked pupillary response on each trial in Experiment 2 using seven metrics: the mean of the pupil data over each trial interval, the latency to the peak onset and offset, the latency to peak amplitude, the peak amplitude, and the area under the curve of the pupillary response (see Pupil data preprocessing; Figure 9A). From a computational perspective, reducing the dimensionality of this set of pupillary response metrics expands the set of models we can consider without taxing computational resources in a reasonable amount of time. Further, dimensionality reduction of the pupillary response allows us to capture separable sources of variance relating to timing and amplitude effects without restricting the data to a smaller set of metrics and possibly discarding information (e.g. timing effects may not be constrained to peak latency or onset latency; amplitude effects may not be constrained to peak dilation amplitude). Therefore, we submitted these metrics to principal component analysis to reduce their dimensionality while capturing maximum variance.

Figure 9

Download asset Open asset

Evoked response characterization and principal component analyses were conducted for each session and for each subject in Experiment 2. The 95% CI for the number of principal components needed to explain 95% of the variance in the data was calculated over subjects and sessions to determine the number of principal components to keep for further analysis. To aid in interpreting subsequent analysis using the selected principal components, the feature importance of each pupil metric was calculated for each principal component and aggregated across subjects as a mean and bootstrapped 95% CI (Figure 9). We found that the first two principal components explained 95% of the variance in the pupillary data. Peak onset, peak offset, and latency to peak amplitude had the greatest feature importance for the first principal component (Figure 9B, upper panel). Mean pupil diameter and peak amplitude had the greatest feature importance for the second principal component (Figure 9B, lower panel). Thus, for interpretability, we refer to the first and second principal components as timing and magnitude components, respectively (Figure 9B). Note that we also conduct this analysis using more conventional methods of pupillary analysis and continue to observe a null effect (see Pupil data preprocessing for details).

To test for the possibility that fluctuations in norepinephrine covaried with changes in the drift-rate and the boundary height, we evaluated a set of models exploring the relationship between the timing and magnitude components of the change-point-evoked pupillary response and shifts in $\theta$. As in our previous model comparison (Figure 8; see Environmental instability prompts a stereotyped decision trajectory), we found that the model describing the relationship between decision policy shift and time relative to a change point had the highest posterior probability given the set of models considered (Figure 10A). To further evaluate the extent of the evidence for the evoked response hypothesis, we present the evidence for the evoked response model against the original model set as $B{F}_{01}$ (Figure 10B). We find unambiguous evidence in favor of the evoked response model relative to the models specifying the modulation of $\theta$ via the timing and magnitude features of the change-point-evoked pupillary response (posterior probability of time-null model given the set of models considered: $0.997\pm 0.002$), with substantially positive $B{F}_{01}$ values. We find no evidence that the pupillary response associates with the dynamics of the decision policy changes in response to a change in action-outcome contingencies.

Figure 10

Download asset Open asset

We investigated how decision policies change when the rules of the environment change. In two separate experiments, we characterized how decision processes adapted in response to a change in action-outcome contingencies as a trajectory through the space of possible types of exploratory and exploitative decision policies. Our findings highlight how, in the context of two choice paradigms, when faced with a possible change in outcomes, humans rapidly shift to a slow exploratory strategy by reducing the drift rate and, sometimes, increasing the boundary height in a stereotyped manner. Using pupillary data, we were unable to detect a relationship between the LC-NE system and the dynamics of adaptive decision policies in unstable environments. Our findings show how the underlying decision algorithm adapts to different forms of uncertainty.

Exploration and exploitation states are not discrete, but exist along a continuum (Addicott et al., 2017). Instead of switching between binary states, humans manage environmental instability by adjusting the greediness of their decision policies (Sadeghiyeh et al., 2020; Prat-Carrabin et al., 2020; Feng et al., 2020; Wilson et al., 2014; Payzan-LeNestour and Bossaerts, 2011; Payzan-Lenestour and Bossaerts, 2012; Wilson et al., 2021). Depending on the relative configuration of parameters in the accumulation to bound process, this adjustment can manifest as either speeded or slowed decisions (Figure 1E; Alexandrowicz, 2020; Ratcliff, 1978). Our results suggest that, in the context of volatile two-choice decisions, humans adopt a mechanism that simultaneously changes the rate of evidence accumulation and, sometimes, the threshold of evidence needed to trigger a decision, so as to adapt to an environmental change (Figure 6A). As soon as a shift in action outcomes is suspected, an internal estimate of change point probability increases and an estimate of the belief in the value of the optimal target plummets (Figure 7A). The rapid increase in change point probability causes a rapid rise in the boundary height on the subsequent trial, thereby increasing the criterion for selecting a new action and allowing variability in the accumulation process to have a greater influence on choice (Figure 7B), although this latter effect is inconsistent across experiments. These changes lead to slow exploratory decisions that facilitate discovery of the new optimal action and result in a quick recovery of the original threshold value over the course of a few trials. In parallel, the rate of evidence accumulation for the optimal choice decreases, with an immediate drop that gradually returns to its asymptotic value as the belief in the value of the optimal choice stabilizes. These results show that when a learner confronts a change point, the decision policy becomes more exploratory by simultaneously increasing the amount of evidence needed to make a decision and slowing the integration of evidence over time. Together, these decision dynamics form a mechanism for gathering information in an unbiased manner that slows the decision at the decision process level but responds quickly relative to a suspected change in trial time.

Critically, our finding that underlying decision policies can reconfigure multiple underlying decision parameters closely parallels recent work in the domain of information-seeking. Information seeking has been decomposed into random and directed components (Wilson et al., 2014). Random exploration refers to inherent behavioral variability that leads us to explore other options, while directed exploration refers to the volitional pursuit of new information. Feng and colleagues recently found that random exploration is driven by changes in the drift rate and the boundary height, with drift rate changes dominating the policy shift (Feng et al., 2020). When environmental conditions encouraged exploration, the drift rate slowed, reducing the signal-to-noise ratio of the reward representation. This finding clearly aligns with our current observations showing that the drift rate sharply decreases in response to a change point and that this change in drift rate dominates the reconfiguration of decision processes, although our experiments were not designed to isolate the directed and random elements of exploration.

Our results are also broadly consistent with a growing body of research converging on the idea that decision policies are not static, but sensitive to changes in environmental dynamics (Dunovan and Verstynen, 2019; Urai et al., 2018). Previous work by our lab (Dunovan and Verstynen, 2019) has shown how, during a modified reactive inhibitory control task, different feedback signals target different parts of the accumulation-to-bound process. Specifically, errors in response timing drove rapid changes in the drift rate on subsequent trials, while selection errors (i.e. making a response on trials where the response should be inhibited) changed the boundary height. Further, there is new evidence that the drift rate adapts on the basis of previous choices, independent of the feedback given for those choices. Urai and colleagues have convincingly demonstrated that choice history signals sculpt the dynamics of the accumulation process by biasing the rate of evidence accumulation (Urai et al., 2018). Our current findings and these previous observations (Pedersen et al., 2017; Ratcliff and Frank, 2012) all highlight how sensitive the parameters of accumulation-to-bound processes are to immediate experience.

Previous literature has shown a conflict-induced spike in reaction time (e.g. Jahfari et al., 2019). However, our complex reaction time results depart from this. One reason for this departure may relate to the demands of the task we are asking participants to perform. While increased cognitive demand should increase reaction times across conditions, we observe a linear decrease in reaction time as a function of volatility when conflict is highest, and we also see that a net increase in conflict decreases reaction times (Figure 3D). We suspect that the presence of both conflict and volatility blurs the distinction between these two sources of uncertainty, especially under high volatility and high conflict conditions. We also see this effect in our formulation of change point probability (CPP), with a bias to overestimate CPP when conflict is high (Figure 4D). It is possible that participants also exhibit this bias to overestimate volatility when conflict is high, which could muddle the effect of conflict on reaction times. Future research should explore the interaction of change point and conflict estimation on the speed-accuracy trade-off.

We hypothesized that any shift in decision policy in response to a change in action-outcome contingencies would be linked to changes in phasic responses of the LC-NE pathways (Aston-Jones and Cohen, 2005). However, we failed to find any evidence of this link using pupillary responses as a proxy of LC-NE dynamics. It should be noted, however, that our experimental design cannot distinguish between pupillary dynamics driven by other catecholamines, such as dopamine, and those dynamics driven by the LC-NE system (Spiers and Calne, 1969; McClure et al., 2005; Gershman and Tzovaras, 2018; Gershman and Uchida, 2019), Thus it is possible that the LC-NE system may still be playing a role in shift of decision policies, and the pupil responses we collected were insensitive to the underlying dynamics. Nonetheless, this null association suggests that an alternative neural mechanism drives the adaptive changes that we observed behaviorally.

One possible alternative mechanism for resetting decision policies is is dopaminergic changes to the cortico-basal ganglia-thalamic (CBGT) pathways, or ‘loops’. Both recent experimental (Yartsev et al., 2018; Dunovan et al., 2015) and theoretical (Bogacz and Larsen, 2011; Caballero et al., 2018; Wei et al., 2015) studies have pointed to the CBGT loops as being a crucial pathway for accumulating evidence during decision making, with the wiring architecture of these pathways ideal for implementing the sequential probability ratio test (Bogacz and Gurney, 2007; Bogacz, 2007), the statistically optimal algorithm for evidence accumulation decisions and the basis for the DDM itself (Ratcliff, 1978). Further, multiple lines of theoretical work have suggested that, within the CBGT pathways, the difference in direct pathway activity between action channels covaries with the rate of evidence accumulation for individual decisions (Mikhael and Bogacz, 2016; Bariselli et al., 2019; Dunovan et al., 2019; Rubin et al., 2021), while the indirect pathways are linked to control of the boundary height (Wei et al., 2015; Herz et al., 2016; Bogacz, 2007; Ratcliff and Frank, 2012). This suggests that changes in the direct and indirect pathways, both within and between representations of different actions, may regulate shifts in decision policies.

Critically, the CBGT pathways are a target of the dopaminergic signaling that drives reinforcement learning (Schultz et al., 1992), suggesting that changes in relative action-value should drive trial-by-trial changes in the drift rate. Indeed, previous work relating dopaminergic circuitry to decision policy adaptation suggests that dopamine may play a critical role in modulating decision policies. Dopamine has substantial links to exploration (Kakade and Dayan, 2002) and recent pharmacological evidence suggests a role for dopaminergic regulation of exploration in humans (Chakroun et al., 2020). More explicitly, both directed and random exploration have been linked to variations in genes that affect dopamine levels in prefrontal cortex and striatum, respectively (Gershman and Tzovaras, 2018). Physiologically, previous work has found that a dopamine-controlled spike-timing-dependent plasticity rule alters the ratio of direct to indirect pathway efficacy in a simulated corticostriatal network (Vich et al., 2020), with overall indirect pathway activity (i.e. pre-decision firing rates) linked to the modulation of the boundary height in a DDM and the difference in direct pathway activation across action channels associating with changes in the drift rate (Dunovan et al., 2019; Rubin et al., 2021). Moreover, recent optogenetic work in mice suggests that activating the subthalamic nucleus, a key node in the indirect pathway, not only halts the motoric response but also interrupts cognitive processes related to action selection (Heston et al., 2020). Our current observations, combined with this previous work, suggests that the decision policy reconfiguration that we observe may associate with similar underlying corticostriatal dynamics, with belief-driven changes to drift rate varying with the difference in direct pathway firing rates across action channels (Dunovan et al., 2019), and change-point-probability-driven changes to the boundary height varying with overall indirect pathway activity (Dunovan et al., 2019; Vich et al., 2020). Future physiological studies should focus on validating this predicted relationship between decision policy reconfiguration and CBGT pathways.

The current study raises many more questions about the dynamics of adaptive decision policies than it answers. For example, we only sparsely sampled the space of possible states of value conflict and volatility. Future work would benefit from a more complete sampling of the conflict and volatility space. A psychophysical characterization of how decision states shift in response to varying forms of uncertainty will expose potential non-linear relationships between the decision policy and feedback uncertainty. Moreover, the decisions that we have modeled here are simple two choice decisions, constrained mostly by the normative form of the traditional DDM framework (Ratcliff, 1978). Scaling the complexity of the task will allow for a more complete assessment of how these relationships change with more complex decisions that better approximate the choices that we make outside the lab. This could be done by moving the cognitive model to frameworks that can fit processes for decisions involving more than two alternatives (e.g. Tajima et al., 2019). Finally, because our estimate of the relationship between our ideal observer estimates of uncertainty and human estimates of uncertainty were indirect, this work would benefit from online approximations of ideal observer estimates, as has been done previously (Wilson et al., 2010). Indeed, there can be substantive individual differences in the detection of of change points (Wilson et al., 2010). Thus, an approximation of how well the estimates of change point probability from our ideal observer correspond to estimates that human observers hold is needed. This approximation would validate the fidelity of the relationship between the ideal observer estimates of uncertainty and the decision parameters that we observed.

Together, our results suggest that when humans are forced to change their mind about the best action to take, the underlying decision policy adapts in a specific way. When a change in action-outcome contingency is suspected, the rate of evidence accumulation decreases and more evidence may briefly be required to commit to a response, allowing variability inherent to the decision process to play a greater role in response selection and resulting in a slow exploratory state. As the environment becomes stable, the system gradually adapts to an exploitative state. Importantly, we find no evidence that norepinephrine pathways associate with this response. This suggests that other pathways may be engaged in this adaptive reconfiguration of decision policies. These results reveal the multifaceted underlying decision processes that can adapt action selection policy under multiple forms of environmental uncertainty.

Behavioral data and their computational derivatives are available at https://github.com/kmbond/dynamic_decision_policy_reconfiguration (copy archived at swh:1:rev:0486705db0f004a5e1365759f5f5a391790771f8). Code used to generate figures can be found here. Raw pupillometry data (DOI: 10.1184/R1/13543133), the features of the task-evoked pupillometry response (DOI: 10.1184/R1/13543067.v1), and the principal components calculated from those features (DOI: 10.1184/R1/13543160.v1) are available here.

1. 1. Bond K

   (2021) KiltHub

   Raw pupillometry data.

   https://doi.org/10.1184/R1/13543133
2. 1. Bond K

   (2021) KiltHub

   Features of task-evoked pupillary response.

   https://doi.org/10.1184/R1/13543067.v1
3. 1. Bond K

   (2021) KiltHub

   Principal components of task-evoked pupillary response.

   https://doi.org/10.1184/R1/13543160.v1

1. 1. Addicott MA
   2. Pearson JM
   3. Sweitzer MM
   4. Barack DL
   5. Platt ML

   (2017) A primer on foraging and the explore/exploit trade-off for psychiatry research

   Neuropsychopharmacology: Official Publication of the American College of Neuropsychopharmacology 42:1931–1939.

   https://doi.org/10.1038/npp.2017.108

   • PubMed
   • Google Scholar
2. 1. Alexandrowicz RW

   (2020) The diffusion model visualizer: an interactive tool to understand the diffusion model parameters

   Psychological Research 84:1157–1165.

   https://doi.org/10.1007/s00426-018-1112-6

   • PubMed
   • Google Scholar
3. 1. Aston-Jones G
   2. Bloom FE

   (1981) Activity of norepinephrine-containing locus coeruleus neurons in behaving rats anticipates fluctuations in the sleep-waking cycle

   The Journal of Neuroscience 1:876–886.

   https://doi.org/10.1523/JNEUROSCI.01-08-00876.1981

   • PubMed
   • Google Scholar
4. 1. Aston-Jones G
   2. Rajkowski J
   3. Cohen J

   (1999) Role of locus coeruleus in attention and behavioral flexibility

   Biological Psychiatry 46:1309–1320.

   https://doi.org/10.1016/s0006-3223(99)00140-7

   • PubMed
   • Google Scholar
5. 1. Aston-Jones G
   2. Cohen JD

   (2005) An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance

   Annual Review of Neuroscience 28:403–450.

   https://doi.org/10.1146/annurev.neuro.28.061604.135709

   • PubMed
   • Google Scholar
6. 1. Bariselli S
   2. Fobbs WC
   3. Creed MC
   4. Kravitz AV

   (2019) A competitive model for striatal action selection

   Brain Research 1713:70–79.

   https://doi.org/10.1016/j.brainres.2018.10.009

   • PubMed
   • Google Scholar
7. 1. Beatty J

   (1982) Task-evoked pupillary responses, processing load, and the structure of processing resources

   Psychological Bulletin 91:276–292.

   https://doi.org/10.1037/0033-2909.91.2.276

   • PubMed
   • Google Scholar
8. 1. Behrens TEJ
   2. Woolrich MW
   3. Walton ME
   4. Rushworth MFS

   (2007) Learning the value of information in an uncertain world

   Nature Neuroscience 10:1214–1221.

   https://doi.org/10.1038/nn1954

   • PubMed
   • Google Scholar
9. 1. Bland AR
   2. Schaefer A

   (2012) Different varieties of uncertainty in human decision-making

   Frontiers in Neuroscience 6:85.

   https://doi.org/10.3389/fnins.2012.00085

   • PubMed
   • Google Scholar
10. 1. Bogacz R

    (2007) Optimal decision-making theories: linking neurobiology with behaviour

    Trends in Cognitive Sciences 11:118–125.

    https://doi.org/10.1016/j.tics.2006.12.006

    • PubMed
    • Google Scholar
11. 1. Bogacz R
    2. Gurney K

    (2007) The basal ganglia and cortex implement optimal decision making between alternative actions

    Neural Computation 19:442–477.

    https://doi.org/10.1162/neco.2007.19.2.442

    • PubMed
    • Google Scholar
12. 1. Bogacz R
    2. Wagenmakers EJ
    3. Forstmann BU
    4. Nieuwenhuis S

    (2010) The neural basis of the speed-accuracy tradeoff

    Trends in Neurosciences 33:10–16.

    https://doi.org/10.1016/j.tins.2009.09.002

    • PubMed
    • Google Scholar
13. 1. Bogacz R
    2. Larsen T

    (2011) Integration of reinforcement learning and optimal decision-making theories of the basal ganglia

    Neural Computation 23:817–851.

    https://doi.org/10.1162/NECO_a_00103

    • PubMed
    • Google Scholar
14. Book

    1. Bond K
    2. Dunovan K
    3. Verstynen TD

    (2018)

    The Influence of Volatility and Conflict on Adaptive Decision Making

    OSF.

    • Google Scholar
15. Software

    1. Bond K

    (2021) Dynamic decision policy reconfiguration under outcome uncertainty, version swh:1:rev:0486705db0f004a5e1365759f5f5a391790771f8

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:de31a380724a5954d16e67a0a6eda17431e11607;origin=https://github.com/kmbond/dynamic_decision_policy_reconfiguration;visit=swh:1:snp:561cc2fa2a2a7aa9aba3d43fe505a2d36b992b7d;anchor=swh:1:rev:0486705db0f004a5e1365759f5f5a391790771f8
16. 1. Bouret S
    2. Sara SJ

    (2005) Network reset: a simplified overarching theory of locus coeruleus noradrenaline function

    Trends in Neurosciences 28:574–582.

    https://doi.org/10.1016/j.tins.2005.09.002

    • PubMed
    • Google Scholar
17. Book

    1. Burnham KP
    2. Anderson DR

    (1998) Practical use of the information-theoretic approach

    In: Burnham KP, editors. Model Selection and Inference. New York, NY: Springer. pp. 75–117.

    https://doi.org/10.1007/978-1-4757-2917-7

    • Google Scholar
18. 1. Byrne JEM
    2. Hughes ME
    3. Rossell SL
    4. Johnson SL
    5. Murray G

    (2017) Time of day differences in neural reward functioning in healthy young men

    The Journal of Neuroscience 37:8895–8900.

    https://doi.org/10.1523/JNEUROSCI.0918-17.2017

    • PubMed
    • Google Scholar
19. 1. Caballero JA
    2. Humphries MD
    3. Gurney KN

    (2018) A probabilistic, distributed, recursive mechanism for decision-making in the brain

    PLOS Computational Biology 14:e1006033.

    https://doi.org/10.1371/journal.pcbi.1006033

    • PubMed
    • Google Scholar
20. 1. Cavanagh JF
    2. Wiecki TV
    3. Kochar A
    4. Frank MJ

    (2014) Eye tracking and pupillometry are indicators of dissociable latent decision processes

    Journal of Experimental Psychology. General 143:1476–1488.

    https://doi.org/10.1037/a0035813

    • PubMed
    • Google Scholar
21. 1. Chakroun K
    2. Mathar D
    3. Wiehler A
    4. Ganzer F
    5. Peters J

    (2020) Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

    eLife 9:e51260.

    https://doi.org/10.7554/eLife.51260

    • PubMed
    • Google Scholar
22. 1. Dayan P
    2. Yu AJ

    (2006) Phasic norepinephrine: a neural interrupt signal for unexpected events

    Network 17:335–350.

    https://doi.org/10.1080/09548980601004024

    • PubMed
    • Google Scholar
23. 1. Dunovan K
    2. Lynch B
    3. Molesworth T
    4. Verstynen T

    (2015) Competing basal ganglia pathways determine the difference between stopping and deciding not to go

    eLife 4:e08723.

    https://doi.org/10.7554/eLife.08723

    • PubMed
    • Google Scholar
24. 1. Dunovan K
    2. Verstynen T

    (2019) Errors in action timing and inhibition facilitate learning by tuning distinct mechanisms in the underlying decision process

    The Journal of Neuroscience 39:2251–2264.

    https://doi.org/10.1523/JNEUROSCI.1924-18.2019

    • PubMed
    • Google Scholar
25. 1. Dunovan K
    2. Vich C
    3. Clapp M
    4. Verstynen T
    5. Rubin J

    (2019) Reward-driven changes in striatal pathway competition shape evidence evaluation in decision-making

    PLOS Computational Biology 15:e1006998.

    https://doi.org/10.1371/journal.pcbi.1006998

    • PubMed
    • Google Scholar
26. Preprint

    1. Feng SF
    2. Wang S
    3. Zarnescu S
    4. Wilson RC

    (2020) The Dynamics of Explore-Exploit Decisions Reveal a Signal-to-Noise Mechanism for Random Exploration

    PsyArXiv.

    https://doi.org/10.31234/osf.io/uepr7

    • Google Scholar
27. 1. Forstmann BU
    2. Dutilh G
    3. Brown S
    4. Neumann J
    5. von Cramon DY
    6. Ridderinkhof KR
    7. Wagenmakers EJ

    (2008) Striatum and pre-SMA facilitate decision-making under time pressure

    PNAS 105:17538–17542.

    https://doi.org/10.1073/pnas.0805903105

    • PubMed
    • Google Scholar
28. 1. Forstmann BU
    2. Anwander A
    3. Schäfer A
    4. Neumann J
    5. Brown S
    6. Wagenmakers EJ
    7. Bogacz R
    8. Turner R

    (2010) Cortico-striatal connections predict control over speed and accuracy in perceptual decision making

    PNAS 107:15916–15920.

    https://doi.org/10.1073/pnas.1004932107

    • PubMed
    • Google Scholar
29. 1. Foster, MSLS ED
    2. Deardorff, MLIS A

    (2017) Open Science Framework (OSF)

    Journal of the Medical Library Association 105:88.

    https://doi.org/10.5195/JMLA.2017.88

    • Google Scholar
30. 1. Gauthier I
    2. Tarr MJ

    (1997) Becoming a “Greeble” expert: exploring mechanisms for face recognition

    Vision Research 37:1673–1682.

    https://doi.org/10.1016/s0042-6989(96)00286-6

    • PubMed
    • Google Scholar
31. 1. Gershman SJ
    2. Tzovaras BG

    (2018) Dopaminergic genes are associated with both directed and random exploration

    Neuropsychologia 120:97–104.

    https://doi.org/10.1016/j.neuropsychologia.2018.10.009

    • PubMed
    • Google Scholar
32. 1. Gershman SJ
    2. Uchida N

    (2019) Believing in dopamine

    Nature Reviews. Neuroscience 20:703–714.

    https://doi.org/10.1038/s41583-019-0220-7

    • PubMed
    • Google Scholar
33. 1. Gilzenrat MS
    2. Nieuwenhuis S
    3. Jepma M
    4. Cohen JD

    (2010) Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function

    Cognitive, Affective & Behavioral Neuroscience 10:252–269.

    https://doi.org/10.3758/CABN.10.2.252

    • PubMed
    • Google Scholar
34. 1. Herz DM
    2. Zavala BA
    3. Bogacz R
    4. Brown P

    (2016) Neural correlates of decision thresholds in the human subthalamic nucleus

    Current Biology 26:916–920.

    https://doi.org/10.1016/j.cub.2016.01.051

    • PubMed
    • Google Scholar
35. 1. Herz DM
    2. Tan H
    3. Brittain JS
    4. Fischer P
    5. Cheeran B
    6. Green AL
    7. FitzGerald J
    8. Aziz TZ
    9. Ashkan K
    10. Little S
    11. Foltynie T
    12. Limousin P
    13. Zrinzo L
    14. Bogacz R
    15. Brown P

    (2017) Distinct mechanisms mediate speed-accuracy adjustments in cortico-subthalamic networks

    eLife 6:e21481.

    https://doi.org/10.7554/eLife.21481

    • PubMed
    • Google Scholar
36. 1. Heston J
    2. Friedman A
    3. Baqai M
    4. Bavafa N
    5. Aron AR
    6. Hnasko TS

    (2020) Activation of subthalamic nucleus stop circuit disrupts cognitive performance

    ENeuro 7:ENEURO.0159-20.2020.

    https://doi.org/10.1523/ENEURO.0159-20.2020

    • PubMed
    • Google Scholar
37. 1. Hurwicz L

    (1972)

    On informationally decentralized systems

    Decision and Organization 1:320.

    • Google Scholar
38. 1. Jahfari S
    2. Ridderinkhof KR
    3. Collins AGE
    4. Knapen T
    5. Waldorp LJ
    6. Frank MJ

    (2019) Cross-task contributions of frontobasal ganglia circuitry in response inhibition and conflict-induced slowing

    Cerebral Cortex 29:1969–1983.

    https://doi.org/10.1093/cercor/bhy076

    • PubMed
    • Google Scholar
39. Book

    1. Jeffreys H

    (1998)

    The Theory of Probability

    OUP Oxford Press.

    • Google Scholar
40. 1. Jepma M
    2. Nieuwenhuis S

    (2011) Pupil diameter predicts changes in the exploration-exploitation trade-off: evidence for the adaptive gain theory

    Journal of Cognitive Neuroscience 23:1587–1596.

    https://doi.org/10.1162/jocn.2010.21548

    • PubMed
    • Google Scholar
41. 1. Joshi S
    2. Li Y
    3. Kalwani RM
    4. Gold JI

    (2016) Relationships between Pupil Diameter and Neuronal Activity in the Locus Coeruleus, Colliculi, and Cingulate Cortex

    Neuron 89:221–234.

    https://doi.org/10.1016/j.neuron.2015.11.028

    • PubMed
    • Google Scholar
42. 1. Kakade S
    2. Dayan P

    (2002) Dopamine: generalization and bonuses

    Neural Networks 15:549–559.

    https://doi.org/10.1016/s0893-6080(02)00048-5

    • PubMed
    • Google Scholar
43. 1. Keung W
    2. Hagen TA
    3. Wilson RC

    (2019) Regulation of evidence accumulation by pupil-linked arousal processes

    Nature Human Behaviour 3:636–645.

    https://doi.org/10.1038/s41562-019-0551-4

    • PubMed
    • Google Scholar
44. Book

    1. Kruschke JK
    2. Vanpaemel W

    (2015) Bayesian estimation in hierarchical models

    In: Busemeyer JR, Wang Z, Townsend JT, Eidels A, editors. The Oxford Handbook of Computational and Mathematical Psychology. Oxford University Press. pp. 279–299.

    https://doi.org/10.1093/oxfordhb/9780199957996.013.13

    • Google Scholar
45. Book

    1. Ledyard JO

    (1989) Incentive compatibility

    In: Eatwell J, Milgate M, Newman P, editors. Allocation, Information and Markets. Springer. pp. 1–2.

    https://doi.org/10.1007/978-1-349-20215-7

    • Google Scholar
46. 1. Luke SG

    (2017) Evaluating significance in linear mixed-effects models in R

    Behavior Research Methods 49:1494–1502.

    https://doi.org/10.3758/s13428-016-0809-y

    • PubMed
    • Google Scholar
47. Conference

    1. McClure SM
    2. Gilzenrat MS
    3. Cohen JD

    (2005)

    Advances in Neural Information Processing Systems

    An exploration-exploitation model based on norepinepherine and dopamine activity. pp. 867–874.

    • Google Scholar
48. 1. Mendonça AG
    2. Drugowitsch J
    3. Vicente MI
    4. DeWitt EEJ
    5. Pouget A
    6. Mainen ZF

    (2020) The impact of learning on perceptual decisions and its implication for speed-accuracy tradeoffs

    Nature Communications 11:1–15.

    https://doi.org/10.1038/s41467-020-16196-7

    • PubMed
    • Google Scholar
49. 1. Mikhael JG
    2. Bogacz R

    (2016) Learning Reward Uncertainty in the Basal Ganglia

    PLOS Computational Biology 12:e1005062.

    https://doi.org/10.1371/journal.pcbi.1005062

    • PubMed
    • Google Scholar
50. 1. Mulder K
    2. Klugkist I

    (2017) Bayesian estimation and hypothesis tests for a circular Generalized Linear Model

    Journal of Mathematical Psychology 80:4–14.

    https://doi.org/10.1016/j.jmp.2017.07.001

    • Google Scholar
51. 1. Murphy PR
    2. Robertson IH
    3. Balsters JH
    4. O’connell RG

    (2011) Pupillometry and P3 index the locus coeruleus-noradrenergic arousal function in humans

    Psychophysiology 48:1532–1543.

    https://doi.org/10.1111/j.1469-8986.2011.01226.x

    • PubMed
    • Google Scholar
52. 1. Murphy PR
    2. Vandekerckhove J
    3. Nieuwenhuis S

    (2014) Pupil-linked arousal determines variability in perceptual decision making

    PLOS Computational Biology 10:e1003854.

    https://doi.org/10.1371/journal.pcbi.1003854

    • PubMed
    • Google Scholar
53. 1. Murray G
    2. Nicholas CL
    3. Kleiman J
    4. Dwyer R
    5. Carrington MJ
    6. Allen NB
    7. Trinder J

    (2009) Nature’s clocks and human mood: the circadian system modulates reward motivation

    Emotion 9:705–716.

    https://doi.org/10.1037/a0017080

    • PubMed
    • Google Scholar
54. 1. Nassar MR
    2. Wilson RC
    3. Heasly B
    4. Gold JI

    (2010) An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment

    The Journal of Neuroscience 30:12366–12378.

    https://doi.org/10.1523/JNEUROSCI.0822-10.2010

    • PubMed
    • Google Scholar
55. 1. Nassar MR
    2. Rumsey KM
    3. Wilson RC
    4. Parikh K
    5. Heasly B
    6. Gold JI

    (2012) Rational regulation of learning dynamics by pupil-linked arousal systems

    Nature Neuroscience 15:1040–1046.

    https://doi.org/10.1038/nn.3130

    • PubMed
    • Google Scholar
56. 1. O’Reilly JX

    (2013) Making predictions in a changing world-inference, uncertainty, and learning

    Frontiers in Neuroscience 7:105.

    https://doi.org/10.3389/fnins.2013.00105

    • PubMed
    • Google Scholar
57. 1. Payzan-LeNestour E
    2. Bossaerts P

    (2011) Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings

    PLOS Computational Biology 7:e1001048.

    https://doi.org/10.1371/journal.pcbi.1001048

    • PubMed
    • Google Scholar
58. 1. Payzan-Lenestour E
    2. Bossaerts P

    (2012) Do not bet on the unknown versus try to find out more: Estimation uncertainty and “unexpected uncertainty” both modulate exploration

    Frontiers in Neuroscience 6:150.

    https://doi.org/10.3389/fnins.2012.00150

    • PubMed
    • Google Scholar
59. 1. Pedersen ML
    2. Frank MJ
    3. Biele G

    (2017) The drift diffusion model as the choice rule in reinforcement learning

    Psychonomic Bulletin & Review 24:1234–1251.

    https://doi.org/10.3758/s13423-016-1199-y

    • PubMed
    • Google Scholar
60. Preprint

    1. Prat-Carrabin A
    2. Wilson RC
    3. Cohen JD
    4. Da Silveira RA

    (2020) Human Inference in Changing Environments with Temporal Structure

    bioRxiv.

    https://doi.org/10.1101/720516

    • Google Scholar
61. 1. Rajkowski J
    2. Kubiak P
    3. Aston-Jones G

    (1994) Locus coeruleus activity in monkey: phasic and tonic changes are associated with altered vigilance

    Brain Research Bulletin 35:607–616.

    https://doi.org/10.1016/0361-9230(94)90175-9

    • PubMed
    • Google Scholar
62. 1. Ratcliff R

    (1978) A theory of memory retrieval

    Psychological Review 85:59–108.

    https://doi.org/10.1037/0033-295X.85.2.59

    • Google Scholar
63. 1. Ratcliff R
    2. Frank MJ

    (2012) Reinforcement-based decision making in corticostriatal circuits: mutual constraints by neurocomputational and diffusion models

    Neural Computation 24:1186–1229.

    https://doi.org/10.1162/NECO_a_00270

    • PubMed
    • Google Scholar
64. 1. Reimer J
    2. McGinley MJ
    3. Liu Y
    4. Rodenkirch C
    5. Wang Q
    6. McCormick DA
    7. Tolias AS

    (2016) Pupil fluctuations track rapid changes in adrenergic and cholinergic activity in cortex

    Nature Communications 7:1–7.

    https://doi.org/10.1038/ncomms13289

    • PubMed
    • Google Scholar
65. 1. Rubin JE
    2. Vich C
    3. Clapp M
    4. Noneman K
    5. Verstynen T

    (2021) The credit assignment problem in cortico-basal ganglia-thalamic networks: A review, a problem and a possible solution

    The European Journal of Neuroscience 53:2234–2253.

    https://doi.org/10.1111/ejn.14745

    • PubMed
    • Google Scholar
66. 1. Sadeghiyeh H
    2. Wang S
    3. Alberhasky MR
    4. Kyllo HM
    5. Shenhav A
    6. Wilson RC

    (2020) Temporal discounting correlates with directed exploration but not with random exploration

    Scientific Reports 10:1–10.

    https://doi.org/10.1038/s41598-020-60576-4

    • PubMed
    • Google Scholar
67. 1. Satterthwaite FE

    (1946) An approximate distribution of estimates of variance components

    Biometrics 2:110–114.

    https://doi.org/10.2307/3002019

    • PubMed
    • Google Scholar
68. 1. Schultz W
    2. Apicella P
    3. Scarnati E
    4. Ljungberg T

    (1992) Neuronal activity in monkey ventral striatum related to the expectation of reward

    The Journal of Neuroscience 12:4595–4610.

    https://doi.org/10.1523/JNEUROSCI.12-12-04595.1992

    • PubMed
    • Google Scholar
69. 1. Sirois S
    2. Brisson J

    (2014) Pupillometry

    Wiley Interdisciplinary Reviews. Cognitive Science 5:679–692.

    https://doi.org/10.1002/wcs.1323

    • PubMed
    • Google Scholar
70. 1. Spiegelhalter DJ
    2. Best NG
    3. Carlin BP
    4. van der Linde A

    (2002) Bayesian measures of model complexity and fit

    Journal of the Royal Statistical Society 64:583–639.

    https://doi.org/10.1111/1467-9868.00353

    • Google Scholar
71. 1. Spiers AS
    2. Calne DB

    (1969) Action of dopamine on the human iris

    British Medical Journal 4:333–335.

    https://doi.org/10.1136/bmj.4.5679.333

    • PubMed
    • Google Scholar
72. 1. Sutton RS
    2. Barto AG

    (1998) Reinforcement Learning: An Introduction

    IEEE Transactions on Neural Networks 9:1054.

    https://doi.org/10.1109/TNN.1998.712192

    • Google Scholar
73. Book

    1. Sutton RS
    2. Barto AG

    (2018)

    Reinforcement Learning: An Introduction

    MIT press.

    • Google Scholar
74. 1. Tajima S
    2. Drugowitsch J
    3. Patel N
    4. Pouget A

    (2019) Optimal policy for multi-alternative decisions

    Nature Neuroscience 22:1503–1511.

    https://doi.org/10.1038/s41593-019-0453-9

    • PubMed
    • Google Scholar
75. 1. Urai AE
    2. Braun A
    3. Donner TH

    (2017) Pupil-linked arousal is driven by decision uncertainty and alters serial choice bias

    Nature Communications 8:1–11.

    https://doi.org/10.1038/ncomms14637

    • PubMed
    • Google Scholar
76. Conference

    1. Urai A
    2. de Gee JW
    3. Tsetsos K
    4. Donner T

    (2018) 2018 Conference on Cognitive Computational Neuroscience

    Choice History Biases Subsequent Evidence Accumulation.

    https://doi.org/10.32470/CCN.2018.1192-0

    • Google Scholar
77. 1. Vaghi MM
    2. Luyckx F
    3. Sule A
    4. Fineberg NA
    5. Robbins TW
    6. De Martino B

    (2017) Compulsivity Reveals a Novel Dissociation between Action and Confidence

    Neuron 96:348–354.

    https://doi.org/10.1016/j.neuron.2017.09.006

    • PubMed
    • Google Scholar
78. 1. van Kempen J
    2. Loughnane GM
    3. Newman DP
    4. Kelly SP
    5. Thiele A
    6. O’Connell RG
    7. Bellgrove MA

    (2019) Behavioural and neural signatures of perceptual decision-making are modulated by pupil-linked arousal

    eLife 8:e42541.

    https://doi.org/10.7554/eLife.42541

    • PubMed
    • Google Scholar
79. 1. Vich C
    2. Dunovan K
    3. Verstynen T
    4. Rubin J

    (2020) Corticostriatal synaptic weight evolution in a two-alternative forced choice task: a computational study

    Communications in Nonlinear Science and Numerical Simulation 82:105048.

    https://doi.org/10.1016/j.cnsns.2019.105048

    • Google Scholar
80. 1. Wagenmakers EJ

    (2007) A practical solution to the pervasive problems of p values

    Psychonomic Bulletin & Review 14:779–804.

    https://doi.org/10.3758/bf03194105

    • PubMed
    • Google Scholar
81. 1. Wei W
    2. Rubin JE
    3. Wang XJ

    (2015) Role of the indirect pathway of the basal ganglia in perceptual decision making

    The Journal of Neuroscience 35:4052–4064.

    https://doi.org/10.1523/JNEUROSCI.3611-14.2015

    • PubMed
    • Google Scholar
82. 1. Wiecki TV
    2. Sofer I
    3. Frank MJ

    (2013) HDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python

    Frontiers in Neuroinformatics 7:14.

    https://doi.org/10.3389/fninf.2013.00014

    • PubMed
    • Google Scholar
83. 1. Wilson RC
    2. Nassar MR
    3. Gold JI

    (2010) Bayesian online learning of the hazard rate in change-point problems

    Neural Computation 22:2452–2476.

    https://doi.org/10.1162/NECO_a_00007

    • PubMed
    • Google Scholar
84. 1. Wilson RC
    2. Niv Y

    (2011) Inferring relevance in a changing world

    Frontiers in Human Neuroscience 5:189.

    https://doi.org/10.3389/fnhum.2011.00189

    • PubMed
    • Google Scholar
85. 1. Wilson RC
    2. Geana A
    3. White JM
    4. Ludvig EA
    5. Cohen JD

    (2014) Humans use directed and random exploration to solve the explore-exploit dilemma

    Journal of Experimental Psychology. General 143:2074–2081.

    https://doi.org/10.1037/a0038199

    • PubMed
    • Google Scholar
86. 1. Wilson RC
    2. Bonawitz E
    3. Costa VD
    4. Ebitz RB

    (2021) Balancing exploration and exploitation with information and randomization

    Current Opinion in Behavioral Sciences 38:49–56.

    https://doi.org/10.1016/j.cobeha.2020.10.001

    • PubMed
    • Google Scholar
87. 1. Yartsev MM
    2. Hanks TD
    3. Yoon AM
    4. Brody CD

    (2018) Causal contribution and dynamical encoding in the striatum during evidence accumulation

    eLife 7:e34929.

    https://doi.org/10.7554/eLife.34929

    • PubMed
    • Google Scholar
88. 1. Yerkes RM
    2. Dodson JD

    (1908) The relation of strength of stimulus to rapidity of habit-formation

    Journal of Comparative Neurology and Psychology 18:459–482.

    https://doi.org/10.1002/cne.920180503

    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "Dynamic decision policy reconfiguration under outcome uncertainty" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by Floris de Lange as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Redmond O'Connell (Reviewer #1), Niels A Kloosterman (Reviewer #3).

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

1) The authors should rewrite sections of the manuscript to more clearly articulate for the reader the core theoretical questions that this study sets out to address. Beyond linking elements of choice uncertainty to parameters of an influential decision model, what were the study goals and why are they important/interesting? Similarly, the authors should clarify the broader implications of their findings for current theory and future research. In addition, the Results section is very long and at times it is hard to follow the rationale for each analysis step or how it relates to the study goals. The authors might consider moving certain details that are not essential to the Methods or Supplemental Materials.

2) The reviewers agreed that the DDM analyses seem overly restricted, being limited to only two parameters when there are other parameters of this model that can plausibly mediate the observed uncertainty effects. The authors make strong claims about the specific nature of the decision policy adjustments observed here but this interpretation would be greatly strengthened if the authors examined model variants that leave other parameters (e.g. non-decision time, drift bias). Please also provide some illustration of the fits to individual data.

3) Several concerns were raised regarding the pupillometry analyses.

a) First, there is a concern that any underlying relationships with choice strategy may have been obscured if the task stimuli evoked strong pupil light reflexes (constrictions) within the measurement window. This cannot be properly assessed without the authors providing plots of the average pupil diameter timecourses along with details regarding the stimuli (e.g. brightness).

b) Second, the authors appear to overlook several relevant metrics. This includes unbaselined prestimulus pupil diameter which has been linked to variations in choice performance in a number of investigations and also to exploration/exploitation switches in at least one report (Jepma and Nieuwenhuis 2011). Previous work has also shown that the LC-NE system is in fact better tracked by using the first time derivative of the pupil signal (Reimer et al., Nat Comm, 2016). The authors should consider looking at the pupil derivative to see if this reveals a link to their experimental manipulations. Importantly, using the derivative instead of the actual pupil time series attenuates the pupil light reflect since only the slope is taken.

c) Third, the authors should check whether they can replicate the relationships between uncertainty and pupil diameter that have been reported in the previous literature. This would go a long way toward confirming the validity both of their pupil data and the null result in the relationship with exploration/exploitation

The authors should also take note of the individual comments of the reviewers provided below as many additional suggestions are provided that should help further improve the manuscript.

Reviewer #1:

In this paper the authors seek to uncover the decision policy adjustments that underpin participants choices on a two-armed bandit task in which the relative reward and probability of reward associated with each alternative varied unpredictably over time. In an extensive modelling analysis the authors examined whether decision dynamics on this task could be understood in terms of adjustments to the bound and/or drift rate parameters of the drift diffusion model. The results indicate that when participants detect a change in which of two choice alternatives is most rewarding they switch from an exploitative to an exploratory decision strategy by lowering both the bound and drift rate of the decision process. My sense is that these findings are supported by extensive analyses reported in the paper.

1. I note that the authors allowed only one decision model parameter to map on to volatility and conflict. It would be interesting to know whether or not allowing either bound or drift rate to account for both volatility and conflict would improve the model fits.

2. The paper also reports a failure to detect any relationship between pupillary responses (here used as a proxy for noradrenergic arousal) during decision making and the aforementioned decision policy adjustments. Here the authors should cite the work of Joshi et al. (2016, Neuron) which, to my knowledge, was the first peer-reviewed paper to report a relationship between locus coeruleus activity and pupil diameter. This paper is also important to consider because they report a failure to observe the tonic/phasic firing modes originally reported by Aston-Jones and colleagues that form the basis for the present hypotheses.

3. The prior literature suggests that there are important functional distinctions between average absolute (i.e. unbaselined) pupil diameter measured in a given time window and the pupil dilation responses that are elicited during decision making. The authors do not consider the former which has been linked to exploration/exploitation strategies at least once in the prior literature (Jepma and Nieuwenhuis, 2011, J Cog Neuro) and which has been linked to variations in choice performance on several occasions (e.g. Murphy et al. 2011, Psychophysiology; Van Kempen et al. 2019, eLife). The authors then conducted a principal component analysis on 7 different metrics extracted from the stimulus-evoked pupil dilation response.

4. Aside from the desire to reduce dimensionality a clear rationale for this approach is not provided. This limits the ability to compare the present results to those in the related literature since most previous studies have investigated relationships between choice behaviour and metrics like pupil dilation amplitude, peak latency and onset latency individually.

5. The manuscript would benefit from more clearly articulating the theoretical advances/insights that it provides. It could be argued that the modelling work results in a situation where one set of cognitive constructs (change point detection and value estimation) are swapped for another set (bound and drift rate) but it is not clear what new understanding is gained from this.

Reviewer #2:

In the present study the authors investigated decision-policy adjustments as a function of two distinct forms of uncertainty, conflict and volatility. They extend previous studies on explore-exploit dynamics by investigating specifically how choice parameters in an evidence accumulation framework vary with uncertainty.

To that aim they experimentally manipulated conflict and volatility in a two armed bandit task and combined bayesian models of learning with evidence accumulation models of decision-making. Then they quantify the degree to which the relationship between estimated choice parameters – indexing the choice policy – varies as a function of the context in which the choice is made. They further recorded pupil diameters to index LC-NE activity and test whether policy adjustments are related to this activity

Choice conflict modulated the rate of evidence accumulation and change points – while also increasing conflict – decreased the boundary height, leading to fast exploratory choices. The authors show that choice dynamics are linked to uncertainty dynamics by driving systematic changes in the decision-policy characterized by the combination of drift rate and threshold. Fast increases in uncertainty following change points drove fast exploratory choices (low drift rate, low threshold) that gradually recovered to exploitatory choices as the new reward contingencies were learned (high drift rate, high threshold).

They further find no evidence that these changes relate to fluctuations in the LC-NE system as indexed with pupil diameter.

This work is methodologically very impressive and theoretically very interesting. It expands the space of decision-policies beyond the explore-exploit dichotomy and characterizes elegantly how decision-policies should dynamically vary along a two-dimensional policy space as the environment is found to change.

Overall, I really liked the study. I do however have some questions and suggestions for additional analyses and edits.

Questions:

1. Typically more similar options/conflict leads to longer RTs. In Exp 1 the authors find no effect and in Exp 2 the opposite. That's usually a pretty robust effect. Any idea why that's not the case here?

2. Also how does this square with the positive effect of ΔB on drift rate? I think that might be worth picking up and unpacking a bit, if only to show the superiority of model-based analyses to raw behavior given that the behavioral findings are super confusing and the parameter results make perfect sense.

3. Could that be some power issue (re number of subjects, not observations)? Is maybe one subject weird in exp. 2 and can't detect change points so well or track the values?

4. What is meant by the similar time course of belief updating for high and low volatility? (p 5 line 178) Shouldn't people update more under higher volatility? Is that what's captured in the change point probability parameter? Maybe that sentence could be clarified so it doesn't confuse readers familiar with work linking volatility and the α parameter in RL (as in more learning/updating under high volatility).

5. If I understand correctly, when change point probability goes up, ΔB always goes down. What's the correlation between the two and if there is a correlation, what does that mean for the impact of those learning parameters on decision parameters? Can you assess conflict effects independent of change point effects (are there period where values are stable and super similar)?

Suggestions:

6. Provide a clearer rationale for the model comparisons to test hypotheses about policy adjustments.

I honestly found it a bit difficult to follow the model comparison for theta. I also think that when the pupil is added, the rationale could be explained a bit more clearly to allow the reader to follow. Specifically I got confused about the intercept and time null models (is that just time or time relative to change point if the latter, why is that a null model?).

7. Replicate relationship between uncertainty parameters and pupil measures before linking it to policy adjustments.

As I understand, you find that the adjustments in the decision-policy are unrelated to pupil diameter. As a sanity check, have the authors looked at pupil diameter as a function of the uncertainty parameters? It would be good to show that earlier effects of uncertainty and their temporal dynamics are replicated (have a plot with the betas over time pre-choice and post feedback). I think the conclusion can be stronger if the authors show that pupil dilation tracks both forms of uncertainty and the anticipation and response dynamics associate with that, but that the subsequent adjustments are not mediated by this system.

Right now something could be wrong with the pupil data and the reader has no way to know. It would also be important just to see that these earlier findings replicate.

8. Reconsider causal language/interpretation of drift rate effects in the discussion.

You say in the discussion that people reduce the drift rate. Isn't the drift rate here determined by the consistency of the evidence? Sure, people can focus more (i.e. in cognitive control tasks, where the response rules are well known and errors are primarily driven by early incorrect activations of prepotent responses or attentional lapses), but I can focus all I want when there's no evidence (when I just don't know what the relative values are because I currently have zero [or little] valid experience to draw from after I detected a change point ) and my drift rate will still be low. No? Couldn't it be that participants have a sense that their drift rate is low (because they have no idea what's going on) and because taking time to sample would be useless (because uncertainty is not reducible other than through action), dropping the threshold is the right thing to do? In that sense the (expected) drift rate would dictate the optimal boundary height. I'm thinking of work by Tajima et al.

9. Reconsider reinterpretation of previous findings in the discussion – add nuance where nuance is due.

I have a bit of a problem with the authors’ assertion that previous findings relating boundary height and conflict could be a misattribution of volatility effects (Frank and colleagues). These previous studies did not have change points. So that is an unlikely explanation of that finding. What is more likely is that the choice dynamics were different because the choices were not temporally dependent, i.e. participants made choices between different options on each trial, meaning that the conflict and thus the optimal decision-strategy differed on every trial (in addition to any learning related uncertainty, but importantly, the true values associated with stimuli never changed). That is not the same as a change point/volatility. Further in the present study, conflict is anticipated, except in the case of change points. So that could equally be the difference between expected and unexpected uncertainty that leads to dissociable effects on decision strategies. In both cases, what drives the threshold adjustment is probably some form of surprise (unexpected conflict). As it stands, the statement in the discussion is inaccurate/misleading. That’s an easy fix though.

Reviewer #3:

Shifting between more explorative and more exploitative modes of decision making is crucial for adaptive human behavior. Therefore, the authors' attempt to investigate the internal processes that allow these modes is important to begin to understand this remarkable ability. In addition, investigating the proposed link to the LC-NE system is sensible and establishing its role in these processes would help the field forward. The authors present a thorough, modelling-heavy set of analysis on two interesting datasets aimed at revealing the underlying mechanisms.

1. Despite these strong points, the manuscript in its current version falls somewhat short of answering the questions that it poses. For one, the DDM analyses are restricted to only two parameters, which begs the question whether other established parameters might be better able to explain the results and thereby shed more light on the underlying mechanisms. Also, regarding the role of the pupil-linked LC-NE system, no strong conclusions can be drawn from the data, since the visual stimulus design likely resulted in strong pupil light reflexes, which might well have overshadowed subtler, more interesting modulations of the pupil. Despite the manuscripts innovative and clever use of Bayesian modelling and PCA, these two shortcomings might limit the impact of the manuscript in its current form on the field.

Shifting between more explorative and more exploitative modes of decision making is crucial for adaptive human behavior. Therefore, the authors' attempt to investigate the internal processes that allow these modes is important to begin to understand this remarkable faculty. In addition, investigating the proposed link to the LC-NE system is sensible and establishing its role in these processes would help the field forward. However, although in general the presented analyses seem thorough, I have two main concerns that in my opinion should be addressed before conclusions can be drawn from the data.

First, the DDM modelling is too restrictive, only focusing on the bound and drift parameters. Besides these two main parameters, another main parameter of the standard DDM is non-decision time, which to my surprise is not mentioned at all in the manuscript. Moreover, recent work has shown that two further parameters can capture internal processes possibly related to explore/exploit policies: starting point (z) and drift bias (called drift criterion or dc by Ratcliff and McKoon (2008)). Including these latter two parameters possibly can explain the RTs better than drift only and shed more light on the components underlying conflict and volatility. In addition, non-decision time might also be affected by the experimental manipulations, and should at least be reported in the manuscript (I assume that the authors did include it in their currently reported DDMs). In my mind, investigating all these further parameters is crucial before the conclusion that bound and drift rate best capture conflict and volatility is warranted.

My second point concerns the pupil analysis.

a) Although I could not find information about visual stimulus size and brightness in the methods, Figure 2AB suggests that there were strong visual transients at trial onset (black screen → stimulus), which presumably resulted in strong pupil constrictions due to the pupil light reflex (PLR).

b) I would have liked to see pupil time courses in this manuscript. The first components of the PCA, as employed by the authors (which in principle I think is a great idea) is likely to capture exactly these PLR dynamics given the large variance due to PLR.

c) Now, previous work has shown that the LC-NE system is in fact better tracked by using the first time derivative of the pupil signal (Reimer et al., Nat Comm, 2016). The authors should consider looking at the pupil derivative to see if this reveals a link to their experimental manipulations. Importantly, using the derivative instead of the actual pupil time series attenuates the PLR since only the slope is taken. Hence, when using the derivative, the PCA might pick up more interesting, cognitive drivers of pupil dynamics, since the PLR dynamics are suppressed. It would be interesting to see if this would reveal a link to the experimental manipulations.

d) Further, please note that the pupil likely not only is linked to the LC-NE system, but generally to catecholamines, which includes dopamine (Joshi et al. Neuron 2015). Therefore, I would recommend to not exclusively link pupil to LC-NE in the manuscript while interpreting the pupil results.

e) In any case, the author should show raw pupil as well as pupil derivative time courses for the different conditions to give insight in their data.

Essential revisions:

1) The authors should rewrite sections of the manuscript to more clearly articulate for the reader the core theoretical questions that this study sets out to address. Beyond linking elements of choice uncertainty to parameters of an influential decision model, what were the study goals and why are they important/interesting? Similarly, the authors should clarify the broader implications of their findings for current theory and future research. In addition, the Results section is very long and at times it is hard to follow the rationale for each analysis step or how it relates to the study goals. The authors might consider moving certain details that are not essential to the Methods or Supplemental Materials.

We have made substantial changes to the frontend of manuscript (Abstract and Introduction) to more explicitly clarify the core theoretical questions that our study set out to address. We have also edited the Results and Discussion in targeted ways to better align the framing of our findings to the context of our theoretical questions. We hope that this better articulates the goals of our work.

2) The reviewers agreed that the DDM analyses seem overly restricted, being limited to only two parameters when there are other parameters of this model that can plausibly mediate the observed uncertainty effects. The authors make strong claims about the specific nature of the decision policy adjustments observed here but this interpretation would be greatly strengthened if the authors examined model variants that leave other parameters (e.g. non-decision time, drift bias). Please also provide some illustration of the fits to individual data.

The reviewers raise crucial points regarding the restricted set of model comparisons. Our original focus on these parameters was driven by prior work showing adaptation of the drift rate and boundary height terms under uncertainty, including findings linking cortico-basal ganglia dynamics with these drift-diffusion parameters under uncertain conditions (Dunovan et al. 2019; Dunovan and Verstynen 2019; Rubin et al. 2021). As we detail below, we have now expanded the set of models considered to include other plausible decision parameters (non-decision time (tr), drift criterion / drift bias (dc), and the starting point (z), in addition to the drift rate (v) and boundary height (a)).

These more rigorous analyses have resulted in an update to the change-point-evoked effect on boundary height (a). Instead of a decreasing in response to a suspected change in reward contingencies, a increases in response to a suspected change. Ironically, this is consistent with our preregistered hypothesis as well as with prior experimental results in our lab (and others). We have updated both our key figures and the text of the Results and Discussion to reflect this change. (See response below).

3) Several concerns were raised regarding the pupillometry analyses.

a) First, there is a concern that any underlying relationships with choice strategy may have been obscured if the task stimuli evoked strong pupil light reflexes (constrictions) within the measurement window. This cannot be properly assessed without the authors providing plots of the average pupil diameter timecourses along with details regarding the stimuli (e.g. brightness).

We were careful to control the effect of light on the pupillary response during data collection. This included the construction of a specific rig around the testing computer so as to reduce ambient reflection of light from walls and ceiling.

To control luminance we also used a Derrington-Krauskopf-Lennie (DKL) color space that allows for direct luminance control. As a result, the stimulus presentation display was rendered isoluminant throughout the task. In addition, the lead author built a booth to isolate the participant from ambient sources of light during data collection.

We now mention these details in the Methods:

“Throughout the task, the head-stabilized diameter and gaze position of the left pupil were measured with an Eyelink 1000 desktop mount at 1000 Hz. […] During the reward-learning task, we used this method to isolate the task-evoked pupillary response.”

Additionally, we include a reminder as part of the caption for Figure 2B to aid the reader in interpreting the results as they progress through the paper:

“(B) In Experiment 2, participants were asked to choose between one of two Greebles (one male, one female). The total number of points earned was displayed at the center of the screen. The stimulus display was rendered isoluminant throughout the task.”

(b) Second, the authors appear to overlook several relevant metrics. This includes unbaselined prestimulus pupil diameter which has been linked to variations in choice performance in a number of investigations and also to exploration/exploitation switches in at least one report (Jepma and Nieuwenhuis 2011). Previous work has also shown that the LC-NE system is in fact better tracked by using the first time derivative of the pupil signal (Reimer et al., Nat Comm, 2016). The authors should consider looking at the pupil derivative to see if this reveals a link to their experimental manipulations. Importantly, using the derivative instead of the actual pupil time series attenuates the pupil light reflect since only the slope is taken.

We have reanalyzed the data using un-baselined prestimulus pupil diameter and the first time derivative of the pupillary response. We again observe a null result. These analyses are detailed as part of our point-by-point reply to reviewers below. Figures visualizing these time courses have been added to the supplementary section of the manuscript (Supplementary Figures 10 and 11).

c) Third, the authors should check whether they can replicate the relationships between uncertainty and pupil diameter that have been reported in the previous literature. This would go a long way toward confirming the validity both of their pupil data and the null result in the relationship with exploration/exploitation

We have now conducted analyses to check for the previously reported links between exploratory choice behavior and the pupillary response. However, we observe no clear evidence for these pre-established links. Therefore, we qualify the use of the pupillary data. The Results section now states this lack of replication to titrate the reader’s confidence in the pupillary results and the corresponding inferences relating to LC-NE system influence on the progression through the decision manifold:

“Specifically, if the LC-NE system were sensitive to a change in the optimal choice, then we should observe a moderate spike in phasic activity following a change in action-outcome contingencies. […] We ask the reader to titrate their interpretation of these pupillary data accordingly.”

We include a similar addition to the pupillometry subsection of the Methods:

“Note that we also conducted a similar analysis using more conventional methods to assess the task-evoked pupillary response and observed another null effect. […] As such, we caution the reader to view our pupillary results in light of this lack of replication of pre-established exploration-driven pupillary responses.”

Reviewer #1:

In this paper the authors seek to uncover the decision policy adjustments that underpin participants choices on a two-armed bandit task in which the relative reward and probability of reward associated with each alternative varied unpredictably over time. In an extensive modelling analysis the authors examined whether decision dynamics on this task could be understood in terms of adjustments to the bound and/or drift rate parameters of the drift diffusion model. The results indicate that when participants detect a change in which of two choice alternatives is most rewarding they switch from an exploitative to an exploratory decision strategy by lowering both the bound and drift rate of the decision process. My sense is that these findings are supported by extensive analyses reported in the paper.

1. I note that the authors allowed only one decision model parameter to map on to volatility and conflict. It would be interesting to know whether or not allowing either bound or drift rate to account for both volatility and conflict would improve the model fits.

The choice to map distinct ideal observer estimates to distinct decision parameters reflects the theoretical motivation underlying the development of our hypotheses tested here. We now expand on this in the introduction to make clear the reasoning for our narrow focus, as below:

“Are the parameters that govern accumulation of evidence for decision making modifiable? […] These policies, in turn, adaptively reconfigure based on current environmental feedback signals by modulating value estimation and the rate of selection errors (Figure 1E).”

A complete, exhaustive sweep of decision parameters as proposed would be computationally inefficient. This would require evaluation of all possible single parameter, dual-parameter (n-parameter) pairings, with both many-to-one ideal observer to DDM mappings and many-to-one DDM parameter mappings to ideal observer mappings. As model complexity increases in these hierarchical DDM fits, the convergence or stability of the fits is more difficult to achieve. So we opted for parsimony in the set of model fits to avoid a data mining expedition that would very likely lead to a set of inconclusive model fits on overly complex models. This sort of restricted set test is quite common when using hierarchical DDM (and, indeed, many hierarchical models in general). Further, estimating pairwise ideal observer to DDM parameter mappings alone keeps model complexity constant, allowing us to make clear comparisons in information loss scores between candidate models.

2. The paper also reports a failure to detect any relationship between pupillary responses (here used as a proxy for noradrenergic arousal) during decision making and the aforementioned decision policy adjustments. Here the authors should cite the work of Joshi et al. (2016, Neuron) which, to my knowledge, was the first peer-reviewed paper to report a relationship between locus coeruleus activity and pupil diameter. This paper is also important to consider because they report a failure to observe the tonic/phasic firing modes originally reported by Aston-Jones and colleagues that form the basis for the present hypotheses.

We thank the reviewer for this omitted reference. Indeed, Joshi et al. 2016 provides clear evidence of a link between locus coeruleus activity and pupil diameter. To our knowledge, this observation extends back to the work of Rajkowski, Kubiak, and Aston-Jones 1994, showing the phasic and tonic modes of the locus-coeruleus system in relation to exploratory behavior. All of this prior work is clearly important to consider, and we thank the reviewer for bringing this to our attention.

We now cite both studies in our revised manuscript:

“The LC-NE system is known to modulate exploration states under uncertainty and pupil diam- eter shows a tight correspondence with LC neuron firing rate (Aston-Jones and Cohen, 2005; Rajkowski et al., 1994), with changes in pupil diameter indexing the explore-exploit decision state (Jepma and Nieuwenhuis, 2011). Similar to the classic Yerkes-Dodson curve relating arousal to performance (Yerkes et al., 1908), performance is optimal when tonic LC activity is moderate and phasic LC activity increases following a goal-related stimulus (Aston-Jones et al. (1999), but see Joshi et al. (2016) for an exception).”

3. The prior literature suggests that there are important functional distinctions between average absolute (i.e. unbaselined) pupil diameter measured in a given time window and the pupil dilation responses that are elicited during decision making. The authors do not consider the former which has been linked to exploration/exploitation strategies at least once in the prior literature (Jepma and Nieuwenhuis, 2011, J Cog Neuro) and which has been linked to variations in choice performance on several occasions (e.g. Murphy et al. 2011, Psychophysiology; Van Kempen et al. 2019, eLife).

Thank you for bringing this important functional distinction between un-baselined pupil diameter and the dilation response to our attention. In our data, if baseline pupil diameter were sensitive to shifts from exploitation to exploration, then we should observe a change in baseline pupil diameter proximal to a change point. However, we do not observe a change-point-evoked shift in baseline pupil diameter in our data, as we might expect given the previous links to exploratory behavior. We now mention our lack of support for these validation analyses in the Results section and visualize the pupillary time courses and results in the Supplementary section (Supp. Figures 10-13).

“Specifically, if the LC-NE system were sensitive to a change in the optimal choice, then we should observe a moderate spike in phasic activity following a change in action-outcome contingencies. Note that we do not observe previously established links between exploratory choice behavior and the pupillary response (Jepma and Nieuwenhuis, 2011; Murphy et al., 2011; van Kempen et al., 2019). We ask the reader to titrate their interpretation of these pupillary data accordingly.”

4. The authors then conducted a principal component analysis on 7 different metrics extracted from the stimulus-evoked pupil dilation response. Aside from the desire to reduce dimensionality a clear rationale for this approach is not provided. This limits the ability to compare the present results to those in the related literature since most previous studies have investigated relationships between choice behaviour and metrics like pupil dilation amplitude, peak latency and onset latency individually.

From a computational perspective, reducing the dimensionality of this set of pupillary response metrics expands the set of models we can consider without taxing computational resources in a reasonable amount of time.

Further, our original PCA method was intended to maximize the variability of the pupillary response linked to the decision manifold. This allowed us to capture separable sources of variance relating to timing and amplitude effects without restricting the data to a smaller set of metrics and possibly discarding information (e.g. timing effects may not be constrained to peak latency or onset latency; amplitude effects may not be constrained to peak dilation amplitude).

We have edited the Results section with the motivation for our PCA approach:

“We characterized the evoked pupillary response on each trial using seven metrics: the mean of the pupil data over each trial interval, the latency to the peak onset and offset, the latency to peak amplitude, the peak amplitude, and the area under the curve of the pupillary response. […] Therefore, we submitted these metrics to principal component analysis to reduce their dimensionality while capturing maximum variance.”

We have also reanalyzed these pupillary data using conventional analysis methods and continue to observe a null effect (see points 3C and 7 for Reviewer 3). We have edited the Results section to reflect this:

“Thus, for interpretability, we refer to the first and second principal components as timing and magnitude components, respectively (Figure 9B). Note that we also conduct this analysis using more conventional methods of pupillary analysis and continue to observe a null effect (see the Pupil data preprocessing for details).”

5. The manuscript would benefit from more clearly articulating the theoretical advances/insights that it provides. It could be argued that the modelling work results in a situation where one set of cognitive constructs (change point detection and value estimation) are swapped for another set (bound and drift rate) but it is not clear what new understanding is gained from this.

This is an excellent point and it reflects our somewhat opaque framing in the

Introduction, which we have now fixed (see our response to the first point raised by the Review Editor). Our primary goal was to understand how the evidence accumulation dynamics changed when the environment requires reassessing learned state action values. Rather than think of these accumulation dynamics, driven by drift rate, boundary height, etc., as a static process, we make the case of thinking of these as points on a continuum of possible states (see manifold in Figure 1e). So our primary focus is on the algorithms of information processing. However, if these are dynamic processes, e.g., drift rate fluctuates over time, then there has to be some learning signal that drives these fluctuations. We chose the ideal observer parameters as likely learning signals that drive plasticity in the decision policy state. We acknowledge that these may not be the only signals that drive adjustments in decision policy dynamics, but they are ideal in that they reflect two correlated, but separate estimates of environmental state.

In line with our response to point 1 from Reviewer 1, we now make this clearer in the Introduction:

“Knowing how decision policies shift in the face of dynamic environments requires looking at the algorithmic properties of the policy itself. […] We predicted that, in response to suspected changes in action-outcome contingencies, humans would exhibit a stereotyped adjustment in the drift rate and boundary height that pushes decisions from certain, exploitative states to uncertain, exploratory states and back again (Figure 1E).”

Reviewer #2:

[…] Overall, I really liked the study. I do however have some questions and suggestions for additional analyses and edits.

Questions:

1. Typically more similar options/conflict leads to longer RTs. In Exp 1 the authors find no effect and in Exp 2 the opposite. That's usually a pretty robust effect. Any idea why that's not the case here?

The reviewer is correct, we do observe different effects of conflict on reaction time in Experiments 1 and 2. The effect is absent in Experiment 1 and small enough in Experiment 2 that it can effectively be considered a null finding. One possible reason for attenuated effects of conflict on reaction time is that participants were overtrained

(Experiment 1 was 2hrs per participant, Experiment 2 was nine hours per participant). This may result in participants having developed an expectation of change point frequency and/or conflict manipulation. If this were the case, then we might expect to see an effect on reaction times in Experiment 1, where participants undergo four sessions of training, but not in Experiment 2, where participants undergo nine sessions of training each. However, we instead see negligible effects on reaction times in Experiment 2 and no effect of conflict on reaction times in Experiment 1.

A second possibility relates to the complexity of our manipulations. Here we impose a range of conflict levels that also vary with degrees of volatility, meaning that our observed reaction time effects are a mix of responses to different extents of conflict and volatility together. This contrasts with previous reports measuring a more restricted range of conflict and without the influence of volatility. Thus, participants are tracking two sources of uncertainty, which may attenuate the overall observed RT response to conflict alone.

We now acknowledge this discrepancy in the Discussion.

“Previous literature has shown a conflict-induced spike in reaction time (e.g. Jahfari et al. 2019). […] Future research should explore the interaction of change point and conflict estimation on the speed-accuracy tradeoff.”

Either way, this discrepancy between our results and other studies, as well as the lack of internal replication of the change point probability and boundary height association in Experiment 2 of our results, is an interesting avenue of exploratory research that we are currently following up on.

2. Also how does this square with the positive effect of ΔB on drift rate? I think that might be worth picking up and unpacking a bit, if only to show the superiority of model-based analyses to raw behavior given that the behavioral findings are super confusing and the parameter results make perfect sense.

The reviewer is absolutely correct. The lack of simple main effects on RT across experiments obscures meaningful behavioral patterns that can be detected with a model-based approach. We now reference the value of a model-based analysis after reviewing the ambiguous behavioral results:

“At the gross level, across all trials within an experimental condition, increasing the ambiguity of the optimal choice (conflict) and increasing the instability of action outcomes (volatility) decreases the probability of selecting the optimal choice. […] We adopt a more focal, model-based analysis in the next section to clarify these peri-change point dynamics.”

3. Could that be some power issue (re number of subjects, not observations)? Is maybe one subject weird in exp. 2 and can't detect change points so well or track the values?

We see fairly consistent sensitivity to change points across subjects in Experiment 2, with accuracy plummeting and recovering with similar time courses, suggesting that all participants track the value of the optimal choice in a consistent manner (see Supplementary Figures 8 and 9 for evoked response profiles by subject). In addition, we conducted a power analysis prior to data collection (preregistration) and the within-session, within-subject power for Experiment 2 is still high enough to detect our hypothesized effects.

As in our response to the previous comment, we think that the RT effects are masking compensatory changes in two different parameters in the accumulation process, rather than outlier participants or sessions.

That said, it may be the case that the reaction time effects simply require more power to detect than accuracy effects, and both of our experiments fail to detect them for that reason. We plan to conduct a replication experiment to recover the effects we observed using a high-powered experimental design both within and across subjects. We hope that the results of this replication address this question. However, given that this is a tangential focus to our original research question, it is more suitable to address this as a follow up paper.

4. What is meant by the similar time course of belief updating for high and low volatility? (p 5 line 178) Shouldn't people update more under higher volatility? Is that what's captured in the change point probability parameter? Maybe that sentence could be clarified so it doesn't confuse readers familiar with work linking volatility and the α parameter in RL (as in more learning/updating under high volatility).

This was an awkwardly worded sentence. We apologize. What we meant here is that the rate of change in relative reward value (ΔB) after a change point is qualitatively similar under both low and high volatility conditions. In other words, the slope of the lines for the two volatility manipulations in Figure 4A is approximately the same. However, the change-point-evoked response belies the main effect of volatility on overall estimates of relative reward value, as shown in Figure 4B. Therefore, we removed this sentence to maintain clarity.

5. If I understand correctly, when change point probability goes up, ΔB always goes down. What's the correlation between the two and if there is a correlation, what does that mean for the impact of those learning parameters on decision parameters? Can you assess conflict effects independent of change point effects (are there period where values are stable and super similar)?

These two parameters are indeed correlated. In fact, ΔB is included in the calculation of change point probability and vice versa. But this interdependence is expected: your certainty in value is going to decrease if you live in a chaotic and changing world. However, the real question is whether they are too correlated to impact our model interpretability. The correlation between change point probability and ΔB is small but reliable, with an increase in belief as change point probability decreases (Spearman’s rho = -0.234 +/- 0.029). However, the Variance Inflation Factor, a collinearity metric, is within an acceptable range for both experiments (Experiment 1: 1.100 +/- 0.013; Experiment 2: 1.058 +/- 0.017). This is generally considered to be an acceptable degree of collinearity (values greater than 10 cause concern; Chatterjee and Simonoff 2013, p. 28-29, notebook showing these results). Thus, the degree of correlation between change point probability and ΔB should have a minimal effect on the estimation of the decision parameters. This suggests that we can safely estimate independent effects of volatility and conflict using change point probability and belief.

Suggestions:

6. Provide a clearer rationale for the model comparisons to test hypotheses about policy adjustments.

I honestly found it a bit difficult to follow the model comparison for theta. I also think that when the pupil is added, the rationale could be explained a bit more clearly to allow the reader to follow. Specifically I got confused about the intercept and time null models (is that just time or time relative to change point if the latter, why is that a null model?).

We thank the reviewer for bringing this lack of clarity to our attention. The time-null model tested for an impact of time relative to a change point, separate from conditional influences of volatility and conflict and the influence of the pupillary response.

We have renamed these models for clarity and expanded on our selection logic for models specifying an impact on decision policy adjustment and updated our naming convention in the Methods section and in the Results section:

“First, we tested the null hypothesis that the decision dynamics was solely a function of the intercept, or the average of the decision dynamics. […] We call this the evoked response model.”

7. Replicate relationship between uncertainty parameters and pupil measures before linking it to policy adjustments.

As I understand, you find that the adjustments in the decision-policy are unrelated to pupil diameter. As a sanity check, have the authors looked at pupil diameter as a function of the uncertainty parameters? It would be good to show that earlier effects of uncertainty and their temporal dynamics are replicated (have a plot with the betas over time pre-choice and post feedback). I think the conclusion can be stronger if the authors show that pupil dilation tracks both forms of uncertainty and the anticipation and response dynamics associate with that, but that the subsequent adjustments are not mediated by this system.

Right now something could be wrong with the pupil data and the reader has no way to know. It would also be important just to see that these earlier findings replicate.

The reviewer raises an excellent point. We did look into this relationship and yet failed to observe evidence for a relationship between our uncertainty parameters – belief and change point probability – and the pupillary response, as measured by both the metrics we calculated and the principal components derived from those metrics.

We have now stated this lack of replication in the Results section to caution the reader:

“Specifically, if the LC-NE system were sensitive to a change in the optimal choice, then we should observe a moderate spike in phasic activity following a change in action-outcome contingencies. […] We ask the reader to titrate their interpretation of these pupillary data accordingly and to view the corresponding inferences relating noradrenergic and catecholaminergic systems to decision policy adjustment in this light.”

8. Reconsider causal language/interpretation of drift rate effects in the discussion.

You say in the discussion that people reduce the drift rate. Isn't the drift rate here determined by the consistency of the evidence? Sure, people can focus more (i.e. in cognitive control tasks, where the response rules are well known and errors are primarily driven by early incorrect activations of prepotent responses or attentional lapses), but I can focus all I want when there's no evidence (when I just don't know what the relative values are because I currently have zero [or little] valid experience to draw from after I detected a change point ) and my drift rate will still be low. No? Couldn't it be that participants have a sense that their drift rate is low (because they have no idea what's going on) and because taking time to sample would be useless (because uncertainty is not reducible other than through action), dropping the threshold is the right thing to do? In that sense the (expected) drift rate would dictate the optimal boundary height. I'm thinking of work by Tajima et al.

The reviewer brings up two points in this comment. We will address each separately.

First there appears to be a conflation of intention with causation. While we fully agree that we can reduce the causal certainty of the language used in the Discussion (something we now do in the revised text), the fact remains that in our data, drift rate reliably changes in response to a changepoint in a stereotypic fashion. Given the nature of the experimental design, we are careful to not make any assumptions as to whether this change is driven by explicit or intentional mechanisms (e.g., increased focus) versus implicit or automatic mechanisms.

The second point raised regards alignment with the work by Tajima and colleagues. It is very likely that the drift rate and boundary height are changing in a cooperative, adaptive manner, at least insofar as the trials immediately surrounding a change point are concerned. The temporal profile of the two parameter changes (at least in Exp. 1 given our new analysis) is quite different, with boundary height changes being brief and drift rate adaptation requiring more time. So, if a change in the boundary height is dictating drift rate changes it is only happening briefly. Therefore we think that a majority of the changes seen in response to a change point are occurring through independent means (consistent with our prior computational models of these pathways (Dunovan et al. 2019; Dunovan and Verstynen 2019; Rubin et al. 2021)).

9. Reconsider reinterpretation of previous findings in the discussion – add nuance where nuance is due.

I have a bit of a problem with the authors' assertion that previous findings relating boundary height and conflict could be a misattribution of volatility effects (Frank and colleagues). These previous studies did not have change points. So that is an unlikely explanation of that finding. What is more likely is that the choice dynamics were different because the choices were not temporally dependent, i.e. participants made choices between different options on each trial, meaning that the conflict and thus the optimal decision-strategy differed on every trial (in addition to any learning related uncertainty, but importantly, the true values associated with stimuli never changed). That is not the same as a change point/volatility. Further in the present study, conflict is anticipated, except in the case of change points. So that could equally be the difference between expected and unexpected uncertainty that leads to dissociable effects on decision strategies. In both cases, what drives the threshold adjustment is probably some form of surprise (unexpected conflict). As it stands, the statement in the discussion is inaccurate/misleading. That's an easy fix though.

Thank you for this careful reading of our critique. Given the update to our results after the more thorough set of model comparisons requested, we no longer include this point in the Discussion. Further, we have made sure to qualify our interpretation of how our findings integrate with the broader literature where necessary. We hope this reflects a more nuanced view of the prior literature.

Reviewer #3:

Shifting between more explorative and more exploitative modes of decision making is crucial for adaptive human behavior. Therefore, the authors' attempt to investigate the internal processes that allow these modes is important to begin to understand this remarkable ability. In addition, investigating the proposed link to the LC-NE system is sensible and establishing its role in these processes would help the field forward. The authors present a thorough, modelling-heavy set of analysis on two interesting datasets aimed at revealing the underlying mechanisms.

1. Despite these strong points, the manuscript in its current version falls somewhat short of answering the questions that it poses. For one, the DDM analyses are restricted to only two parameters, which begs the question whether other established parameters might be better able to explain the results and thereby shed more light on the underlying mechanisms. Also, regarding the role of the pupil-linked LC-NE system, no strong conclusions can be drawn from the data, since the visual stimulus design likely resulted in strong pupil light reflexes, which might well have overshadowed subtler, more interesting modulations of the pupil. Despite the manuscripts innovative and clever use of Bayesian modelling and PCA, these two shortcomings might limit the impact of the manuscript in its current form on the field.

Shifting between more explorative and more exploitative modes of decision making is crucial for adaptive human behavior. Therefore, the authors' attempt to investigate the internal processes that allow these modes is important to begin to understand this remarkable faculty. In addition, investigating the proposed link to the LC-NE system is sensible and establishing its role in these processes would help the field forward. However, although in general the presented analyses seem thorough, I have two main concerns that in my opinion should be addressed before conclusions can be drawn from the data.

First, the DDM modelling is too restrictive, only focusing on the bound and drift parameters. Besides these two main parameters, another main parameter of the standard DDM is non-decision time, which to my surprise is not mentioned at all in the manuscript. Moreover, recent work has shown that two further parameters can capture internal processes possibly related to explore/exploit policies: starting point (z) and drift bias (called drift criterion or dc by Ratcliff and McKoon (2008)). Including these latter two parameters possibly can explain the RTs better than drift only and shed more light on the components underlying conflict and volatility. In addition, non-decision time might also be affected by the experimental manipulations, and should at least be reported in the manuscript (I assume that the authors did include it in their currently reported DDMs). In my mind, investigating all these further parameters is crucial before the conclusion that bound and drift rate best capture conflict and volatility is warranted.

The reviewer is correct. Investigating the remaining DDM parameters is crucial to substantiate our claim that the drift rate and boundary height respond in a coordinated fashion to promote exploration in response to a suspected change. This was something that we did in our initial model evaluations but was left out for the sake of concision. We now include a more thorough test of the set of DDM parameters (a,v,t,z,dc) that could respond to a change point. This broader first-level test confirms our initial results showing that the t, z, and dc parameters do not reliably change in response to a change point (Figure 5).

My second point concerns the pupil analysis. Regarding the role of the pupil-linked LC-NE system, no strong conclusions can be drawn from the data, since the visual stimulus design likely resulted in strong pupil light reflexes, which might well have overshadowed subtler, more interesting modulations of the pupil.

a) Although I could not find information about visual stimulus size and brightness in the methods, Figure 2AB suggests that there were strong visual transients at trial onset (black screen → stimulus), which presumably resulted in strong pupil constrictions due to the pupil light reflex (PLR).

The representation of the display depicted in Figures 2A and B does not show the actual luminance of the stimulus display for Experiment 2. As we state in our general response to the Editor above and in point B of this response, we carefully controlled task-related luminance and ambient sources of light in order to maximize our capacity to detect subtle pupillary effects. See point 3A in our response to the Editor and the revised language included in that response.

b) I would have liked to see pupil time courses in this manuscript. The first components of the PCA, as employed by the authors (which in principle I think is a great idea) is likely to capture exactly these PLR dynamics given the large variance due to PLR.

It is unlikely that we are capturing pupillary light reflexes given our control of luminance in the experimental testing rig. However, we have now included average time courses of the pupillary response to the Supplementary section (Supp. Figure 9-13). We hope these are useful for readers who share this concern. See point 3C of our response to the editor for cautionary language added to the Results and Methods section.

c) Now, previous work has shown that the LC-NE system is in fact better tracked by using the first time derivative of the pupil signal (Reimer et al., Nat Comm, 2016). The authors should consider looking at the pupil derivative to see if this reveals a link to their experimental manipulations. Importantly, using the derivative instead of the actual pupil time series attenuates the PLR since only the slope is taken. Hence, when using the derivative, the PCA might pick up more interesting, cognitive drivers of pupil dynamics, since the PLR dynamics are suppressed. It would be interesting to see if this would reveal a link to the experimental manipulations.

We have now calculated the first time derivative of the pupil signal and reanalyzed our data on this measure. Using the first time derivative of the pupil signal, we recalculated the principal components of the pupillary response as with the first order signal. Using these recalculated principal components, we reassessed the relationship between the pupillary data and our conditional manipulations and retested the link between these principal components and theta, the relationship between a and v. We continue to observe null effects. See point 4 of our response to Reviewer 1.

d) Further, please note that the pupil likely not only is linked to the LC-NE system, but generally to catecholamines, which includes dopamine (Joshi et al. Neuron 2015). Therefore, I would recommend to not exclusively link pupil to LC-NE in the manuscript while interpreting the pupil results.

We appreciate the point that other catecholamines, such as dopamine, also contribute to the task-evoked pupillary response. We now acknowledge this lack of specificity in the Discussion:

“We hypothesized that these shifts in decision policies would be linked to changes in phasic responses of the LC-NE pathways, although we should note our experimental design does not distinguish between pupillary dynamics driven by other catecholamines, such as dopamine, and those dynamics driven by the LC-NE system.”

e) In any case, the author should show raw pupil as well as pupil derivative time courses for the different conditions to give insight in their data.

We now include subject-wise visualizations of the evoked pupillary response and the time derivative of that response for all combinations of conflict and volatility in the Supplementary section (Supp. Figures 11 and 13).

Author details

1. Krista Bond

   1. Department of Psychology, Carnegie Mellon University, Pittsburgh, United States
   2. Center for the Neural Basis of Cognition, Pittsburgh, United States
   3. Carnegie Mellon Neuroscience Institute, Pittsburgh, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Visualization, Writing - original draft, Writing - review and editing

   For correspondence

   kbond@andrew.cmu.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-1492-6798

2. Kyle Dunovan

   Department of Psychology, Carnegie Mellon University, Pittsburgh, United States

   Contribution

   Conceptualization, Methodology, Software, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-7857-5133

3. Alexis Porter

   Department of Psychology, Northwestern University, Evanston, United States

   Contribution

   Data curation, Investigation, Project administration

   Competing interests

   No competing interests declared

4. Jonathan E Rubin

   1. Center for the Neural Basis of Cognition, Pittsburgh, United States
   2. Department of Mathematics, University of Pittsburgh, Pittsburgh, United States

   Contribution

   Conceptualization, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-1513-1551

5. Timothy Verstynen

   1. Department of Psychology, Carnegie Mellon University, Pittsburgh, United States
   2. Center for the Neural Basis of Cognition, Pittsburgh, United States
   3. Carnegie Mellon Neuroscience Institute, Pittsburgh, United States
   4. Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, United States

   Contribution

   Conceptualization, Formal analysis, Funding acquisition, Investigation, Project administration, Resources, Supervision, Validation, Writing - review and editing

   For correspondence

   timothyv@andrew.cmu.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4720-0336

• 818

  Page views

• 152

  Downloads

• 2

  Citations

Article citation count generated by polling the highest count across the following sources: PubMed Central, Crossref, Scopus.