One of the keys to successful learning is being able to adjust behavior on the basis of experience. In simple terms, it pays to repeat behaviors with positive outcomes and avoid those with negative outcomes. A group of brain regions known collectively as the basal ganglia supports this process by guiding behavior based on trial and error.

However, circumstances can and do change: a behavior that routinely produces a positive outcome now will not necessarily always do so in the future. The outcome of a given behavior can also vary from time to time purely by chance: even the most appropriate action can sometimes lead to a negative outcome but should be repeated again. Exactly how the basal ganglia take into account this uncertainty over behavioral outcomes to appropriately guide learning is unclear.

Franklin and Frank have now come up with a possible explanation by building on an existing computational model of the basal ganglia. Whereas the old model assumes that the rate of learning given an unexpected outcome always remains constant, in the new model learning occurs more quickly when the outcome of a behavior is uncertain. This makes intuitive sense, in that rapid learning is especially useful during the initial stages of learning a new task or following a sudden change in circumstances.

The new model proposes that a group of cells called tonically active interneurons (TANs), which release the chemical acetylcholine, enable the basal ganglia to take account of uncertainty. TANs are found in a basal ganglia structure called the striatum and have a characteristic firing pattern during important outcomes, consisting of a burst of activity followed by a pause lasting several hundred milliseconds. The model suggests that when the outcome of a behavior is uncertain, the length of this pause is increased. This boosts the activity of another group of neurons in the striatum, known as spiny neurons, and this in turn increases the rate of learning.

Franklin and Frank found that by varying the length of the TAN pause, the basal ganglia can adjust learning rates based on the degree of uncertainty over behavioral outcomes. Comparisons show that the TAN computational model optimizes the accuracy and flexibility of learning across different environments, while also accounting for findings which show that TAN lesions induce insensitivity to changes in decision outcomes. The next step is to test some of the new predictions about uncertainty experimentally.

When tasked with taking an action in an unknown environment, there can be considerable uncertainty about which actions will lead to the best outcomes. A principled way to resolve this uncertainty is to use previous experience to guide behavior towards actions that have led to positive outcomes in the past and away from actions that have led to negative outcomes. Convergent evidence suggests that the basal ganglia can guide behavior by incorporating positive and negative feedback in a reinforcement learning process (O'Doherty et al., 2003; Barnes et al., 2005; Frank, 2005). However, learning can be complicated in a changing environment, as the validity of past experiences and the relationship between actions and outcomes become uncertain as well. Mathematical models suggest that it is optimal to take uncertainty into account in learning and decision making (Yu and Dayan, 2005; Behrens et al., 2007; Mathys et al., 2011), but it is unclear whether the basal ganglia can directly consider uncertainty in feedback-based learning.

Basal ganglia-dependent learning is often described within the normative framework of reinforcement learning (RL) following the observation that signaling from dopaminergic afferents matches the pattern of a reward prediction error (RPE) (Montague et al., 1996; Bayer et al., 2007). An RPE is the signed difference between the observed and expected outcomes and is often used in RL to generate point-estimates of action-values (Sutton and Barto, 1998). Phasic dopamine (DA) is thought to provide an RPE signal to striatal medium spiny neurons (MSNs) and induce learning through changes in corticostriatal plasticity (Montague et al., 1996; Reynolds and Wickens, 2002; Calabresi et al., 2007), with opponent learning signals in the direct and indirect pathways (Frank, 2005; Collins and Frank, 2014). Within these pathways, separate populations code for the (positive and negative) values of distinct action plans (Samejima et al., 2005). Multiple lines of evidence in humans and animals support this model, including optogenetic manipulations (Tsai et al., 2009; Kravitz et al., 2012), synaptic plasticity studies (Shen et al., 2008), functional imaging (McClure et al., 2003; O'Doherty et al., 2003), genetics and pharmacology in combination with imaging (Pessiglione et al., 2006; Frank et al., 2009; Jocham et al., 2011) and evidence from medication manipulations in Parkinson’s patients (Frank et al., 2004).

Despite the substantial empirical support for RPE signals conveyed by dopamine, the simple RL mechanisms used to model the basal ganglia are inflexible in the degree to which they learn in a changing environment. RL models typically adopt a fixed learning rate, such that every RPE of similar magnitude equally drives learning. However, a more adaptive strategy in a changing environment is to adjust learning rates as a function of uncertainty, so that unexpected outcomes have greater influence when one is more uncertain of which action to take (e.g., initially before contingencies are well known, or following a change-point), but less influence once the contingencies appear stable and the task is well known (Yu and Dayan, 2005; Behrens et al., 2007; Nassar et al., 2010; Mathys et al., 2011; Payzan-LeNestour et al., 2011). This Bayesian perspective presents the additional challenge for basal ganglia-dependent learning: in order to take advantage of its own uncertainty over action selection, the basal ganglia would need a mechanism to translate its uncertainty into a learning rate.

Cholinergic signaling within the striatum offers a potential solution to this challenge. With few exceptions (Tan and Bullock, 2008; Ashby and Crossley, 2011), models of the basal ganglia typically do not incorporate striatal acetylcholine. Within the striatum, cholinergic interneurons are the predominant source of acetylcholine (Woolf and Butcher, 1981). These interneurons, also known as tonically active neurons (TANs) due to their intrinsic 2–10-Hz firing pattern, are giant, spanning large areas of the striatum with dense axonal arborization and broad synaptic input (Goldberg and Reynolds, 2011). TANs appear to be necessary to learning only when flexibility is required (Ragozzino et al., 2009; Bradfield et al., 2013), suggesting that they might modulate the learning rate as a function of changes in outcome statistics (i.e., uncertainty). Similar to dopaminergic neurons, TANs show sensitivity to rewarding events and develop a learned phasic response to predictive cues (Aosaki et al., 1994; Morris et al., 2004). This response consists of a phasic burst in TAN activity followed by a pause that lasts a few hundred milliseconds (Aosaki et al., 1995). While the temporal pattern of the burst–pause response is temporally concomitant to the dopamine response (Morris et al., 2004), the unidirectional TAN response is not consistent with a bivalent RPE (Joshua et al., 2008) but instead is thought to provide a permissive signal for dopaminergic plasticity (Graybiel et al., 1994; Morris et al., 2004; Cragg, 2006).

But how would such a permissive signal be modulated by the network’s own uncertainty about which action to select? Because TANs receive broad inhibitory synaptic input from local sources, including MSNs and GABAergic interneurons (Bolam et al., 1986; Chuhma et al., 2011), we hypothesized that the pause would be modulated by a global measure of uncertainty across the population of spiny neurons. Given that MSN sub-populations code for distinct action values (Samejima et al., 2005; Frank, 2005; Lau and Glimcher, 2008), co-activation of multiple populations can signal enhanced uncertainty over action selection, which would translate into greater inhibition onto TANs. The synchrony in the TAN response suggests a global signal (Graybiel et al., 1994), which can then be modulated by inhibitory MSN collaterals across a large range of spiny inputs. The TAN pause response is consistent with a signal of uncertainty that adjusts learning. First, it increases with the unpredictability of a stochastic outcome (Apicella et al., 2009, 2011). Second, pharmacological blockade or lesioning excitatory input to TANs impairs learning, specifically after a change in outcome contingencies (Ragozzino et al., 2002; Bradfield et al., 2013). For an optimal learner, both increases in stochasticity and changes in outcome contingencies results in an increase in uncertainty (Yu and Dayan, 2005; Nassar et al., 2010).

Here, we augment a well-established computational model of the basal ganglia (BG) to include a mechanism by which the effective learning rate is modulated by cholinergic signaling, and where this signaling is, in turn, modulated by uncertainty in the MSN population code via reciprocal TAN–MSN interactions. In the model, cholinergic signals dynamically modulate the efficacy of reinforcement feedback, driving changes in the number of neurons available for synaptic plasticity during reinforcement, hence the effective learning rate of the network as a whole. Thus, TANs allow the basal ganglia to tailor its learning rate to its environment, balancing the tradeoff between learning flexibility and robustness to noise by adjusting learning as a function of policy uncertainty embedded in the MSN population code. We show that this behavior is consistent with a normative account using an approximately Bayesian model and that its main functionality can be simplified in algorithmic form using a modified RL model, thus spanning Marr’s levels of implementation, algorithm, and computation. The model is consistent with several existing lines of evidence and makes falsifiable predictions.

We extended a previously published neural network model of the basal ganglia to examine the potential role for TANs in learning. Figure 1 shows a graphical representation of the model, which has been adapted from Frank, 2006 to include TANs. The basic mechanics of the BG model without TANs have been reviewed elsewhere (e.g., Maia and Frank, 2011; Ratcliff and Frank, 2012) but are summarized here before articulating the TAN mechanisms. In the network, striatonigral and striatopallidal MSNs are simulated with separate populations of ‘Go’ and ‘NoGo’ units (rate-coded point neurons), which act to facilitate or suppress selection of specific actions. These MSN units receive excitatory input from a sensory input cortical layer and a motor cortical layer representing specific candidate responses. A given response is executed when its motor cortical firing rate exceeds a threshold.

Figure 1

Download asset Open asset

The basal ganglia participate in action selection by selectively disinhibiting thalamocortical activity for a given response while inhibiting thalamocortical activity for alternative responses, via MSN projections through different basal ganglia nuclei (Frank, 2006; Humphries et al., 2006). Activity in Go units contributes evidence toward the selection of an action in the thalamus and ultimately the cortex, while NoGo unit activity suppresses action selection and effectively provides evidence against particular responses. The difference in Go and NoGo unit activity for each candidate action effectively provides a decision variable, each represented in distinct columns of cortico-BG-thalamic circuits. The greater the relative Go–NoGo activity difference for a particular response, the more likely the corresponding column of thalamus will be disinhibited, and hence the corresponding action selected. Initially, network choices are determined by random synaptic weights favoring one or the other action and the additional intra-trial noise in unit activity favoring exploration of alternative actions, but choices stabilize via dopamine-dependent learning, as described next.

Dopamine modulates excitability in the Go and NoGo pathways and has opposing effects on the two populations. D1 receptor activity simulated in the Go units results in increased excitability while D2 receptor activity simulated in NoGo units has the opposite effect. Consequently, an increase in dopamine leads to increased Go activity relative to NoGo activity; this can influence both the choice process through acute changes in excitability and the learning process through consequent changes in synaptic efficacy during phasic changes in dopamine-related to positive and negative outcomes.

Several predictions of this model have been validated empirically across species, including effects of pharmacological manipulations, genetic variants, and optogenetic manipulations (Collins and Frank, 2014 for review and quantitative simulations) but as of yet, it has not explored the role of TANs. Furthermore, while it can account for general effects of dopaminergic manipulation on probabilistic reversal learning (Frank, 2005), it fails to appropriately modulate its learning as a function of environmental noise and internal action uncertainty. Finally, while BG RL models depend on DA-driven learning, they do not provide a mechanism that would allow striatal learning to occur selectively during reward prediction errors as opposed to various other factors that could also change DA levels (e.g., Howe et al., 2013). We investigated whether the TAN pause, by windowing dopaminergic RPEs, can provide such a permissive signal on learning.

The stereotypical TAN burst–pause response modulates excitability and presynaptic corticostriatal signaling in both MSN populations (Gerfen and Surmeier, 2011). In our model, a TAN pause was simulated at the onset of feedback to create a window around the dopaminergic reinforcement signal (Morris et al., 2004); to isolate the effects solely attributable to learning, TAN activity levels were constant during action selection itself (Figure 2, left; see Materials and methods). Cholinergic signaling within the striatum is extensive and complex, involving a diversity of cholinergic receptors. For tractability, we mainly focused our investigation on the effects of TAN signaling on MSN excitability, simulating the effects of M1- and M2-muscarinic acetylcholine receptors on MSNs and nicotinic receptors on GABAergic interneurons (we further explore the ACh effects on DA release in the final section). Changes in muscarinic activity modulate MSN excitability during TAN pauses, allowing the MSN population to be more or less excitable during reinforcement (see Materials and methods for details of biophysics and implementation in our model). Longer (or deeper) TAN pauses increase corticostriatal input and hence MSN activity in response to reinforcement, compared to shorter TAN pauses.

Figure 2

Download asset Open asset

We conducted a series of simulations in which this neural network was confronted with a two alternative, forced choice reversal learning task. On each trial, one of two input stimuli was presented and the network had to select one of two responses. For the initial simulations, one response was rewarded 80% of the trials and the other 20% of the trials. (Subsequent simulations described later varied the stochasticity of the environment.) The network was trained on 20 epochs of 20 trials for each stimulus, or 400 trials total. Performance was probed prior to the first training epoch as a baseline and following the completion of each training epoch. The first 10 training epochs comprised the 'acquisition' phase in which the reward schedule remained constant. Adaptive behavior in this context demands learning from unexpected outcomes during initial stages of the task but ignoring spurious outcomes (i.e. the 20% of negative outcomes) once the contingencies are well known. The reward contingencies were then reversed for the remaining 10 epochs ('reversal' phase) such that the response with the lower value during acquisition was now rewarded in 80% of trials. An adaptive learner would show renewed sensitivity to unexpected outcomes when the contingencies change, and then stabilize again asymptotically.

Because the TAN pause results in MSN disinhibition, it modulates their excitability during outcome presentation (when phasic dopaminergic signals are presented) via modulation of M1 and M2 receptors (see Materials and methods). TAN pauses elicit greater activation of the most excitable units, which then, due to activity-dependent plasticity, are most subject to dopaminergic learning. In turn, the resultant synaptic weight changes affect the subsequent distribution of active MSNs during action selection. This distribution can be quantified by the uncertainty in action selection across the MSN population, which we define here as the Shannon’s entropy based on the firing rates y of units associated with each action a across all time points t during action selection within a trial:

where p_a(t) reflects the population’s probability assigned to selecting action a based on normalized firings rates coding for that action (see Materials and methods). A population with low entropy has a few strongly active units representing primarily the dominant motor response, whereas a population with high entropy has many weakly active units representing multiple alternative responses. As the network learns, entropy across Go units declines over time and the network becomes more likely to strongly activate Go units associated with the correct response, and less likely to activate those for the opposing response (Figure 2, top right). Following the reversal, the entropy of the Go units increases as the network un-learns the previous correct response (making those units more weakly active) while beginning to activate those associated with the alternative response. While this pattern is evident in networks without TANs, it is more pronounced in TAN networks: the selective modulation of excitability during outcomes amplifies the effect of dopamine on Go activity for the dominant response, progressively differentiating from the less rewarded response and leading to low entropy Go representations (Figure 2, top right).

The decline in MSN entropy translates into reduced stochasticity in action selection, as the MSN population becomes dominated by a single strongly active population, providing stronger evidence for the frequently rewarded action. Consequently, TAN networks exhibit higher accuracy when a stochastic task is well learned than a network without TANs (Figure 3, left). However, this property alone reduces the flexibility of the network to respond to change points, because a single strongly active population is harder to unlearn, and the network will perseverate in selecting the previously learned response. This tradeoff will become evident below, where we show that TAN pause durations can be dynamically modulated to optimize the tradeoff between flexibility and stability.

Figure 3

Download asset Open asset

To directly study the impact of TANs on learning, we first parametrically varied the duration of the TAN pause exogenously (below we explore how this pause may be endogenously self-regulated by the network). Varying pause duration across networks reveals a behavioral trade-off between learning speed following probabilistic reversal and asymptotic accuracy (Figure 4, top row). During acquisition, learning speed is similar across a range of TAN pause durations, but networks with long pauses exhibit higher levels of asymptotic accuracy. Long pauses facilitate a disproportionate increase in synaptic efficacy for the most frequently rewarded response despite the occasional spurious outcome (Figure 5, bottom right), driving MSNs to focus their activity to a single dominant action (as quantified by MSN entropy; Figure 4, bottom). However, this same property makes such networks slow to reverse. Conversely, networks with short TAN pause durations learn more quickly following reversal but at a cost of asymptotic accuracy. Mechanistically, this occurs as short TAN pauses elicit weaker activation of spiny units, and hence elicit less activity-dependent plasticity, leading to a smaller learned divergence in synaptic weights between alternative responses (Figure 5, bottom right). This stunted divergence allows for continued co-activation of multiple response alternatives during subsequent action selection, and hence more stochastic choice, but this same property supports the network’s ability to quickly reinforce alternative actions following change points. Thus, although in these simulations the TAN pause modulates excitability only during window of the phasic dopaminergic reinforcement signal, it influences which MSNs learn and hence the population activity and action selection entropy thereof during subsequent choices.

Figure 4

Download asset Open asset

Figure 5

Download asset Open asset

The importance of this trade-off is most evident across different levels of predictable stochasticity, which was manipulated by varying the reward schedule. In a probabilistic reversal learning task in which the optimal action was rewarded with an 85% reward rate (suboptimal action rewarded 15%), networks with a short TAN pause outperformed networks with a longer TAN pause (Figure 6, left). Accuracy (percent choice of the optimal action) across all trials was 78% (SD=7%) for the shortest pause compared with 70% (SD=5%) for the longest TAN pause [t(148) = 7.3, p<3x10^–10, Cohen’s d = 1.2]. In a more stochastic task with only 40% reward rates for the optimal action (10% for the suboptimal action), the pattern is reversed: long pause networks obtained an overall accuracy level of 53% (SD = 10%) compared to 65% (SD = 10%) for long pause networks [t(148) = 8.1, p<3x10^–13, Cohen’s d = 1.3] (Figure 6, center). In both environments, overall accuracy varies parametrically with pause duration and degrades gracefully. These simulations show that, for each environment, the optimal pause duration is dependent on the cost of sensitivity to noise and the need for flexible behavior. The high MSN entropy generated by a short TAN pause allows for increased flexibility, which is advantageous in a highly deterministic environment with a single reversal, where spurious negative outcomes are less common, lowering the cost of sensitivity to noise. In contrast, the lower entropy MSN representation generated by longer TAN pauses (due to divergent weights) is advantageous when the task is highly stochastic. The ability to ignore spurious negative outcomes allows the network to maintain a stable estimate of an action value, which can be more important than behavioral flexibility. As a result, the performance of a network with fixed pause duration is dependent on the environmental statistics.

Figure 6

Download asset Open asset

Normative and algorithmic descriptions of TAN pause behavior

Before further analyzing biophysical mechanisms within the model—including a mechanism for computation of entropy, the effects of M1 receptor manipulations, and additional roles of the post–pause burst on DA release—we first develop higher level (computational and algorithmic) models that summarize the key trade-off identified above in functional terms (Marr and Poggio, 1976). While the neural network model makes empirical predictions at the biophysical level, the core computational and algorithmic problems solved by the network can be imbedded in simpler formulations. To describe the computational and algorithmic problems solved by the network, we compared the network’s behavior with two models: (1) an approximately Bayesian model that considers the higher-level computational problem of learning under uncertainty, and (2) an algorithmic model that more closely matches learning in the basal ganglia mechanistically, modified from Collins and Frank (2014) to include the role of TANs.

The core computational problem solved by the addition of TANs in the network addresses how to integrate noisy experiences in a changing environment. Behrens et al. (2007) noted that in a changing environment, a hierarchical representation of volatility can be used to adjust the learning rate in an optimal way. If rewards are distributed probabilistically with rate r and changes from trial to trial, an optimal agent can estimate the volatility of r as well as their distrust in the trial-to-trial volatility of the reward rate. It is unlikely that this hierarchical inference is implemented in the basal ganglia; thus, we consider here an approximation by which this computation need not be performed explicitly but where the striatum has access to its own uncertainty and adjust its learning rate accordingly.

In the tasks simulated, rewards were of equal value and delivered stochastically. In a stationary task with binomial outcomes, the posterior distribution of the reward rate for an optimal learner is a Beta distribution (Daw et al., 2005) parameterized by the counts of rewarded and non-rewarded trials. Because reversal tasks are, by definition, non-stationary, the Beta distribution does not represent the exact posterior distribution of expected values and is slow to adjust to a reversal: it becomes too certain about reward contingencies. A Bayesian treatment allowing for the possibility that the outcome statistics can change can be approximated using a mixture distribution combining the posterior of expected values with a uniform distribution (Nassar et al., 2010). In this approximation, the mixture component is the probability a change has occurred to some other unknown contingencies. Instead, we use a multiplicative parameter $\gamma$ to decay the counts of the Beta hyper-parameters towards the prior after each trial, an approximation that has been used previously to model rodent and human behavior (Daw et al., 2005; Doll et al., 2009). Decaying the counts multiplicatively maintains the mode expected value for each action but increases the variance (uncertainty) of the distribution. This effectively reduces the model’s confidence of the expected value without changing its best estimate. Hence, a faster decay rate allows for greater effective learning rate, analogous to the mechanism in the BG model, whereby MSN weights are prevented from overlearning.

We simulated the Bayesian model with both fixed values of $\gamma$ as well as a $\gamma$ that varied as a function of entropy in action selection. Models with fixed values of $\gamma$ are analogous to neural networks with a fixed duration TAN pause, both constituting static strategies to balance asymptotic accuracy and learning speed. Varying $\gamma$ as a function of action selection entropy is similar to the strategy employed by the neural network, as MSN entropy corresponds to the decision uncertainty of the network.

Over both manipulations, the Bayesian model shows the same qualitative pattern of behavior as the neural network (Figure 7, right). For fixed values of $\gamma ,$ we found the same tradeoff between asymptotic accuracy and learning speed following reversal. Likewise, varying the decay rate with uncertainty mitigated the effects of the trade-off, facilitating faster learning following reversal than possible with a slow, fixed decay rate without the cost of asymptotic accuracy associated with high, fixed decay (Figure 7, right, black line). Given that it is an idealized statistical model and not implementational, the Bayesian learner responds with an overall higher level of accuracy than the neural network. Unlike the Bayesian learner, the neural network does not learn an estimate of each action value directly but instead learns relative preferences for actions. Nevertheless, overall, the Bayesian learner provides a normative description of the neural network as the effect of manipulating the decay parameter $\gamma$ is qualitatively similar to what we see when manipulating TAN duration, suggesting that TAN duration affects the uncertainty of representations in MSNs.

Figure 7

Download asset Open asset

The OpAL model proposed by Collins and Frank (2014) provides a more algorithmic summary of the basal ganglia network, as an expansion of an actor–critic model to include opponent (D1/D2) actor values. The OpAL model provides a normative advantage over traditional RL while quantitatively capturing a variety of data across species implicating opponent processes in both learning and action selection, where dopamine manipulations affect the asymmetry with which humans and animals make decisions, in a model with few free parameters. In the OpAL model, reward prediction errors are computed within a critic that evaluates the expected value of each state, and are used to update these values and train the G and N actors, summarizing the population activity of Go and NoGo units with point values (see Materials and methods). To simulate the effect of TANs in OpAL, the G and N weights were decayed multiplicatively after each trial by either a constant rate (to simulate the effects of a fixed-duration TAN pause), or as a function of G and N entropy (to simulate the effects of the proposed feedback mechanism). A fast rate of decay (closer to zero) simulated a short TAN pause whereas a slow rate of decay (closer to one) simulated a long TAN pause.

Simulations over a range of fixed rates of decay shows a similar pattern as in the neural network and Bayesian learner, as slow rates of decay showed high asymptotic accuracy but with a decreased learning speed following reversal in a deterministic environment (Figure 8, left). Likewise, a faster rate of decay showed a similar pattern to networks with short TAN pause durations and showed increased flexibility after reversal at a cost of asymptotic accuracy. As in the neural network, the trade-off between asymptotic accuracy and speeded learning after reversal was marked by a divergence in the G weights (Figure 8, right). For models with a slow decay, the difference between G weights for the initially rewarded response and the initially sub-optimal response was much more pronounced than models with a faster decay. This divergence declined more rapidly in models with a slow decay, allowing these models to unlearn the previously rewarded response more quickly.

Figure 8

Download asset Open asset

Allowing the decay rate to vary as a function of uncertainty also showed the same pattern of behavior found in the neural network, mitigating the trade-off and resulting in both high asymptotic accuracy as well as speeded learning following reversal (Figure 8, left). Overall, varying decay with entropy resulted in higher reward rate compared with the next best performing model with fixed decay [t(198)=3.7, p<3x10^–4, Cohen’s d = 0.5]. Varying decay with entropy also optimized the divergence of G weights relative to fixed decay. The initial rate of divergence of the variable model is similar to that of a slow decay model, improving asymptotic accuracy. This rate of divergence declines as the model learns, preventing overlearning and facilitating faster learning following reversal. This is the same pattern observed in the divergence of Go weights in the neural model (Figure 5, bottom right), where varying pause duration with entropy optimized the divergence of weights relative to fixed duration pauses.

A local mechanism for entropy modulation of TAN duration

In the neural network simulations we proposed that TANs may have access to entropy over MSNs, given that they span large regions of the striatum and receive inputs from many MSNs and GABAergic interneurons. However, the biophysical details by which spiny neuron modulate TAN activity are not fully understood. Direct transmission of entropy may not be trivial as population entropy is a nonlinear function of the units’ activity. Here we consider a more explicit mechanism by which TANs directly approximate MSN entropy through synaptic integration. First, consider the minimalist case of just two MSNs coding for alternative actions. Here, the entropy is high when both are active or inactive, but low when either of the two units alone is active. In terms of a Boolean function, the entropy of the two MSNs is the logical opposite of an 'exclusive OR', a non-linear problem that typically requires interneurons to detect (Rumelhart and McClelland, 1986). While one potential mechanism for TANs to detect MSN entropy is by including interneurons, this non-linear detection is linearly solvable if the problem is split into two separate detections: the detection of when both neurons are active and the detection of when neither is active (Figure 8, top left). Consequently, a potential approximation for MSN entropy would be the detection of coincident activity (or inactivity) of pairs of MSNs. While an active readout of joint inactivity is implausible, this problem is solved by the opponent nature of MSN pathways in the BG (Frank, 2005; Kravitz et al., 2012), where the TANs could detect either coincident activity of two D1 MSNs signaling Go activity or two D2 MSNs signaling NoGo activity.

This scheme would require a synaptic organization such that two MSNs associated with different motor responses synapse close together on the dendrite of a TAN to facilitate coincidence detection (Figure 9, top right). Coincidence detection between distinct signals is thought to be an important mechanism both for cellular plasticity (Wang et al., 2000) as well as the coordination of sensory input (Kapfer et al., 2002). If synaptic signaling from both MSNs is needed to propagate an action potential to the cell body, then the pair of synapses can be thought of as a coincidence detector. In logical terms, the synapse pairs perform Boolean 'AND' detection (Figure 9, top left). Several pair of D1 synapses located on the dendrites of a single TAN could result in the summation of these signals in the cell body of the TAN, approximating half of the entropy function signaling that there is evidence for multiple motor responses. Similarly, multiple pairs of D2 synapses contributes to the other half of the entropy function, signaling that there is evidence against multiple responses. Thus, these two measures could effectively approximate decision uncertainty in the MSN population and are detectable by a single TAN via direct synapses from MSNs. In principle, other mechanisms within the striatum could transmit the population uncertainty (e.g., via further interactions between MSNs and fast-spiking interneurons). Our exercise below thus presents but a single plausible mechanism.

Figure 9

Download asset Open asset

We simulated the network varying the TAN pause with AND detection in the Go units (see Materials and methods). In a network with a fixed duration TAN pause (180 ms) and trained on an 80% reward schedule, AND detection in Go units followed a similar temporal pattern to Shannon’s entropy (Figure 9, center left). Both AND detection and entropy decline as the network learns the task and rise following the reversal of the reward contingencies. Not all summary statistics of MSN activity follow this pattern: for example, the simple summation of all activity in the Go population gradually declines over training and consequently, is not suitable for an approximation of entropy (Figure 9, center right). Notably, modulating TAN pause duration with the AND detection can also mitigate the behavioral trade-off between flexibility and stability, similar to the patterns observed using entropy (Figure 9, bottom right).

Simulated ablation of M1-receptors

In the neural network model, varying the duration of the TAN pause alters the degree to which MSNs are disinhibited during reward feedback. Mechanistically, this depends solely on M2 receptors, which are responsible for inhibition of the TANs in the model, and does not involve changes in the activity of M1 receptors (see Materials and methods). However, previous empirical studies have linked M1 receptors to learning after reversal. Both TAN ablations and muscarinic antagonists impair reversal learning but do not affect acquisition in deterministic tasks (Ragozzino et al., 2002; Witten et al., 2010), an effect that is specific to M1 receptors (McCool et al., 2008).

To investigate the effects of M1 receptors in the model, we simulated selective M1 receptor ablations (see Materials and methods). In our simulations, the simulated ablation of M1 receptors only modestly decreases performance during acquisition, resulting in similar asymptotic accuracy as a control model that does not contain TANs (Figure 3, left). Following reversal, the effects are much more pronounced as simulated M1 ablation results in severely degraded accuracy (Figure 3, right). These results are qualitatively consistent with empirical findings showing impairment in reversal learning following M1 antagonists. It is noteworthy that the impairment with simulated M1 ablations only occur as a consequence of the disinhibitory effects of the TAN pause that drive MSN activity to a lower entropic state: control networks simulated without TANs do not show the same degree of impairment (Figure 3, right). Without this decrease in entropy or the increase in NoGo excitability, the network is able to learn following the reversal. However, both effects combine to allow the network to adapt more flexibly, perseverating in the correct action when it is rewarded but more sensitive to negative feedback facilitating reorienting.

Figure 10

Download asset Open asset

To act purposefully, a learner needs to ignore predictable irrelevant feedback driven by environmental stochasticity, while also flexibly adapting to a changing environment. This problem is general to learning with several proposed solutions (Yu and Dayan, 2005; Behrens et al., 2007; Mathys et al., 2011). Here, we proposed how the striatum may incorporate uncertainty to an adaptive process in line with these general principals of learning with a model in which striatal cholinergic signaling promotes encoding uncertainty in the MSN population. Across all simulations, incorporating TANs increased asymptotic performance over the previously published BG model (Frank, 2005). Simulated lesions of M1-AChRs revealed a qualitatively similar pattern of behavior as reported in rodent ablation and optogenetic studies (McCool et al., 2008; Witten et al., 2010). Parametric manipulations of TAN pauses revealed a trade-off between asymptotic accuracy and flexible learning driven by changes in MSN entropy. As a result, a network with any fixed TAN behavior can perform well in one task but may perform poorly in others. MSN entropy decreased over time in networks with long, fixed duration TAN pauses, making them less sensitive to stochastic noise, and facilitating performance in highly stochastic, sparsely rewarded environments. In contrast, networks with short, fixed duration TAN pauses had relatively constant levels of MSN entropy, resulting in labile representations sensitive to noise. As these networks are sensitive to negative feedback, they were able to learn quickly following a reversal, particularly in more deterministic environments.

The behavioral trade-off seen in the network is a product of a fixed learning behavior in a stochastic and non-stationary environment. Optimal behavior in such environments requires balancing the need to create precise estimates of an action’s value with the flexibility to respond to sudden changes (Yu and Dayan, 2005; Behrens et al., 2007; Nassar et al., 2010). These goals are counter-posed, thus any fixed learning rule that treats each observation equally at the time of observation will not be able to achieve both. If a learner is sensitive to recent events, she will be quick to detect changes but sensitive to any noise in the observations. Conversely, if a learner considers a longer time window to determine the best course of action, she will be slower to detect a change but be able to ignore irrelevant outcomes. Computationally, this pattern can be described by decay rate in the approximately Bayesian model as a fast decay rate considers recent events more strongly than a slow decay rate, which considers a longer history of events.

These findings suggest that TANs may provide an effective way of annealing the learning rate at the population level by implicitly coding uncertainty in the spiny neuron population code and making learning in the basal ganglia more (though not perfectly) Bayesian. Short TAN pauses are computationally similar to a high learning rate (fast decay in the Bayesian model) and long TAN pauses are similar to a low learning rate (slow decay). These effects are driven by relative differences in disinhibition of MSNs during reinforcement, and are not dependent on temporal dynamics of the pause (the same tradeoff between flexible and stable learning was seen when the magnitude of the pause rather than duration was manipulated). During the TAN pause, the reduction in TAN activity lowers the level of MSN inhibition, increasing the differential of activity between response selection and feedback. The greater disinhibition associated with longer TAN pauses results in a higher differential of activity and stronger reinforcement, driving the MSN population towards a lower entropy representation. Conceptually, embedding uncertainty into the population code of spiny neurons is similar to proposals that probabilistic population codes are used to maintain distributions for Bayesian inference (Ma et al., 2006). In both cases, the entropy of the population code can represent uncertainty, but here we explore the implication of this notion for learning within the basal ganglia, rather than inference in sensory cortex.

The reduction in MSN entropy with learning results in a lower population-level learning rate. This happens because the reduction in entropy reflects a reduction in the number of units active during reinforcement, such that the highly active units representing the dominant (most rewarded) action will persist in the face of spurious outcomes. Highly active units require more reinforcement to unlearn the association while inactive units are not available for reinforcement due to activity-dependent plasticity. As a result, an MSN population with lower entropy learns more slowly. As previously pointed out with the basal ganglia acetylcholine-based entropy (BABE) model, a reduction in MSN entropy will also decrease stochastic exploration as a decrease in MSN entropy lowers noise in the evidence for action (Stocco, 2012). In this work, we provide a single mechanism that affects learning rate and exploration as a function of entropy, but we note that even the exploration effect is learned and is collinear with the change in learning rate: we manipulated TAN pauses only during reinforcement, the window in which pauses overlap with phasic dopamine responses (Morris et al., 2004). Both changes arise as changes in synaptic weights alter the entropic representations in future trials, allowing the network to anneal its learning rate and change its stochastic exploration adaptively with its own uncertainty.

Given the size and extensive connectivity of TANs within the striatum, it is plausible that TANs are responsive to MSN entropy or a correlate, such as the coincident detection of pairs of active Go or NoGo neurons associated with separate motor responses. TANs receive substantial GABAergic innervation from MSNs (Gonzales et al., 2013) and are regulated by the neuropeptides enkephalin and substance P from MSN axons (Gonzales and Smith, 2015). This connectivity appears to be involved in learning as Pavlovian learning has been associated with an increase of δ-opioid receptor expression in TANs in rodent accumbens (Bradfield et al., 2013; Laurent et al., 2014). We have proposed, given this connectivity and the potential for MSNs to represent action selection uncertainty within a population code, it is plausible that a measure of uncertainty could be approximated directly with signaling from MSNs to TANs.

Differences in TAN pause duration have been reported in several cases that lead us to the hypothesis that the TAN pause duration may be an important modulator of cellular activity. In a behavioral context, differences in TAN pause duration have been reported to the motivational valence of a stimuli (Ravel et al., 2003) and to the presence or omission of reward (Apicella et al., 2009). In vitro, application of dopamine can elongate the TAN pause response (Deng et al., 2007) and the duration of the pause has been linked to the magnitude of initial excitatory input (Doig et al., 2014). These results suggest the duration of the TAN pause may be an important regulator mechanism and we further hypothesize that the local collaterals from many spiny neurons across the striatum onto a single giant TAN would allow the TAN to reciprocally modulate MSN population activity.

However, an important caveat to this hypothesis is that while the MSN to TAN synaptic connections are established and influence inhibitory currents within TANs (Bolam et al., 1986; Chuhma et al., 2011), we are not aware of direct in vivo evidence that TAN pauses are modulated by MSN activity. This constitutes a core prediction of the model. We propose that coincident MSN activity is largely involved in regulating the TAN pause that occurs during novel and/or rewarding events (forming a window around phasic dopaminergic signaling) and may be less relevant at other periods of time. However, the exact influence of MSNs signaling on TAN activity and the mechanism for its transduction remain open to questions.

An alternate hypothesis is that pause duration is not modulated by MSN activity at all and reflects excitatory and dopamine input (Ding et al., 2010). In this case, we might still expect TAN activity to reflect uncertainty signals relevant to learning, relayed from other brain areas. The source of an uncertainty signal might originate through an explicit system that could signal a change-point in the reward rate (Behrens et al., 2007; Nassar et al., 2010; Wilson et al., 2010) or infer a change in a latent context or rule-structure (Gershman et al., 2010; Donoso et al., 2014). In principle, MSN entropy could also be influenced by hierarchical inputs from cortical areas that either further constrain action selection or increase noise, based on other factors such as perceived volatility. Human functional MRI (fMRI) studies have identified correlates of uncertainty processing in multiple areas, notably in the anterior cingulate cortex and striatum (Behrens et al., 2007; Bach et al., 2011), leaving open multiple possible mechanism to consider uncertainty at the whole brain level, including the top-down modulation of MSN entropy or TAN signals. Interestingly, a recent fMRI study have linked abstract prediction errors to the basal-forebrain, a separate source of cholinergic signaling (Iglesias et al., 2013). A final possibility is that uncertainty can be communicated to the TANs via more complex local striatal networks involving both MSNs and GABAergic interneurons, or an external source, such as a derivative of the dopamine learning signal. Nevertheless, while we have proposed that MSN synapses on TANs may directly relay an uncertainty cue, the core model prediction that TANs promote adaptive striatal learning via some representation of uncertainty is not itself sensitive to the source of the uncertainty cue.

Accordingly, the model makes several novel empirical predictions. A central prediction of the model is that MSN entropy should correlate with decision uncertainty during a task, which interacts with task stochasticity, and should covary with the degree of TAN pause inhibition. Similarly, we predict TAN pause duration (or equivalently, TAN firing rate) should correlate with decision uncertainty. Whereas previous studies have found a link between TAN activity and reversal learning (Ragozzino et al., 2002; Witten et al., 2010), our model makes the more specific prediction that M2 blockade would correspond with a decrease in overall accuracy during the performance of a probabilistic reversal learning task with multiple reward schedules. Notably, these manipulations were performed in the medial striatum, an area typically thought to be more involved in reversal than acquisition (Clarke et al., 2008). It is an open question whether TAN ablations would affect acquisition performance in other striatal areas, and in fact our model does predict changes in acquisition depending on the probabilities of reinforcement. We also expect M2 blockade would affect the tradeoff between exploration and exploitation by interfering with the striatum’s ability to anneal it’s own exploration policy with uncertainty. We would expect this effect to extend to overtraining; M2 blockade may lessen the effects of overtraining on stochastic tasks. We would expect optogentic silencing of TANs entirely to have similar effects.

While there are several previously proposed solutions to the general learning problem posed here (Yu and Dayan, 2005; Behrens et al., 2007; Mathys et al., 2011), the model we provide here is not meant as an alternative to these high level descriptions. Rather, we propose how a specific system, the striatum, can modulate its own behavior adaptively in line with Bayesian accounts and provide an interpretation of the role of TANs, tying together electrophysiological, pharmacological, lesion and behavioral data. This poses an additional theoretical question, as there is evidence to suggest that people learn in an adaptive way when faced with change points (Behrens et al., 2007; Nassar et al., 2010), and in different contexts in a process that likely involves the frontal cortex (Collins and Koechlin, 2012; Collins and Frank, 2013): why would the striatum consider uncertainty in a heuristic fashion if other systems are able to consider uncertainty explicitly? One benefit such a heuristic system that approximates optimal behavior could offer would be a lack of complexity. Optimal adaptation in a changing environment requires knowledge of the generative process underlying change, a problem that is complex and potentially ill-posed. Inference over the generative model may be too costly in highly complex, rapidly change or novel environments. In such an environment, striatal uncertainty could still be available and alter behavior in an adaptive way, encouraging exploration and weighing recent feedback strongly.

1. 1. Aisa B
   2. Mingus B
   3. O’Reilly R

   (2008) The emergent neural modeling system

   Neural Networks 21:1146–1152.

   https://doi.org/10.1016/j.neunet.2008.06.016

   • Google Scholar
2. 1. Aosaki T
   2. Graybiel A
   3. Kimura M

   (1994) Effect of the nigrostriatal dopamine system on acquired neural responses in the striatum of behaving monkeys

   Science 265:412–415.

   https://doi.org/10.1126/science.8023166

   • Google Scholar
3. 1. Aosaki T
   2. Kimura M
   3. Graybiel AM

   (1995)

   Temporal and spatial characteristics of tonically active neurons of the primate's striatum

   Journal of Neurophysiology 73:1234–1252.

   • Google Scholar
4. 1. Aosaki T
   2. Miura M
   3. Suzuki T
   4. Nishimura K
   5. Masuda M

   (2010) Acetylcholine-dopamine balance hypothesis in the striatum: an update

   Geriatrics & Gerontology International 10:S148–S157.

   https://doi.org/10.1111/j.1447-0594.2010.00588.x

   • Google Scholar
5. 1. Apicella P
   2. Deffains M
   3. Ravel S
   4. Legallet E

   (2009) Tonically active neurons in the striatum differentiate between delivery and omission of expected reward in a probabilistic task context

   European Journal of Neuroscience 30:515–526.

   https://doi.org/10.1111/j.1460-9568.2009.06872.x

   • Google Scholar
6. 1. Apicella P
   2. Ravel S
   3. Deffains M
   4. Legallet E

   (2011) The role of striatal tonically active neurons in reward prediction error signaling during instrumental task performance

   Journal of Neuroscience 31:1507–1515.

   https://doi.org/10.1523/JNEUROSCI.4880-10.2011

   • Google Scholar
7. 1. Ashby FG
   2. Crossley MJ

   (2011) A computational model of how cholinergic interneurons protect striatal-dependent learning

   Journal of Cognitive Neuroscience 23:1549–1566.

   https://doi.org/10.1162/jocn.2010.21523

   • Google Scholar
8. 1. Bach DR
   2. Hulme O
   3. Penny WD
   4. Dolan RJ

   (2011) The known unknowns: neural representation of second-order uncertainty, and ambiguity

   Journal of Neuroscience 31:4811–4820.

   https://doi.org/10.1523/JNEUROSCI.1452-10.2011

   • Google Scholar
9. 1. Barnes TD
   2. Kubota Y
   3. Hu D
   4. Jin DZ
   5. Graybiel AM

   (2005) Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories

   Nature 437:1158–1161.

   https://doi.org/10.1038/nature04053

   • Google Scholar
10. 1. Bayer HM
    2. Lau B
    3. Glimcher PW

    (2007) Statistics of midbrain dopamine neuron spike trains in the awake primate

    Journal of Neurophysiology 98:1428–1439.

    https://doi.org/10.1152/jn.01140.2006

    • Google Scholar
11. 1. Behrens TEJ
    2. Woolrich MW
    3. Walton ME
    4. Rushworth MFS

    (2007) Learning the value of information in an uncertain world

    Nature Neuroscience 10:1214–1221.

    https://doi.org/10.1038/nn1954

    • Google Scholar
12. 1. Bennett BD
    2. Wilson CJ

    (1999)

    Spontaneous activity of neostriatal cholinergic interneurons in vitro

    The Journal of Neuroscience 19:5586–5596.

    • Google Scholar
13. 1. Bolam JP
    2. Ingham CA
    3. Izzo PN
    4. Levey AI
    5. Rye DB
    6. Smith AD
    7. Wainer BH

    (1986) Substance p-containing terminals in synaptic contact with cholinergic neurons in the neostriatum and basal forebrain: a double immunocytochemical study in the rat

    Brain Research 397:279–289.

    https://doi.org/10.1016/0006-8993(86)90629-3

    • Google Scholar
14. 1. Bradfield LA
    2. Bertran-Gonzalez J
    3. Chieng B
    4. Balleine BW

    (2013) The thalamostriatal pathway and cholinergic control of goal-directed action: interlacing new with existing learning in the striatum

    Neuron 79:153–166.

    https://doi.org/10.1016/j.neuron.2013.04.039

    • Google Scholar
15. 1. Cachope R
    2. Mateo Y
    3. Mathur BN
    4. Irving J
    5. Wang H-L
    6. Morales M
    7. Lovinger DM
    8. Cheer JF

    (2012) Selective activation of cholinergic interneurons enhances accumbal phasic dopamine release: setting the tone for reward processing

    Cell Reports 2:33–41.

    https://doi.org/10.1016/j.celrep.2012.05.011

    • Google Scholar
16. 1. Calabresi P
    2. Centonze D
    3. Gubellini P
    4. Pisani A
    5. Bernardi G

    (1998) Blockade of M2-like muscarinic receptors enhances long-term potentiation at corticostriatal synapses

    European Journal of Neuroscience 10:3020–3023.

    https://doi.org/10.1111/j.1460-9568.1998.00348.x

    • Google Scholar
17. 1. Calabresi P
    2. Picconi B
    3. Tozzi A
    4. Di Filippo M

    (2007) Dopamine-mediated regulation of corticostriatal synaptic plasticity

    Trends in Neurosciences 30:211–219.

    https://doi.org/10.1016/j.tins.2007.03.001

    • Google Scholar
18. 1. Chuhma N
    2. Tanaka KF
    3. Hen R
    4. Rayport S

    (2011) Functional connectome of the striatal medium spiny neuron

    Journal of Neuroscience 31:1183–1192.

    https://doi.org/10.1523/JNEUROSCI.3833-10.2011

    • Google Scholar
19. 1. Clarke HF
    2. Robbins TW
    3. Roberts AC

    (2008) Lesions of the medial striatum in monkeys produce perseverative impairments during reversal learning similar to those produced by lesions of the orbitofrontal cortex

    Journal of Neuroscience 28:10972–10982.

    https://doi.org/10.1523/JNEUROSCI.1521-08.2008

    • Google Scholar
20. 1. Collins A
    2. Koechlin E

    (2012) Reasoning, learning, and creativity: frontal lobe function and human decision-making

    PLoS Biology 10:e1001293.

    https://doi.org/10.1371/journal.pbio.1001293

    • Google Scholar
21. 1. Collins AGE
    2. Frank MJ

    (2013) Cognitive control over learning: creating, clustering, and generalizing task-set structure

    Psychological Review 120:190–229.

    https://doi.org/10.1037/a0030852

    • Google Scholar
22. 1. Collins AGE
    2. Frank MJ

    (2014) Opponent actor learning (opAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive

    Psychological Review 121:337–366.

    https://doi.org/10.1037/a0037015

    • Google Scholar
23. 1. Cragg SJ

    (2006) Meaningful silences: how dopamine listens to the ACh pause

    Trends in Neurosciences 29:125–131.

    https://doi.org/10.1016/j.tins.2006.01.003

    • Google Scholar
24. 1. D'Angiulli A
    2. Runge M
    3. Faulkner A
    4. Zakizadeh J
    5. Chan A
    6. Morcos S

    (2013) Vividness of visual imagery and incidental recall of verbal cues, when phenomenological availability reflects long-term memory accessibility

    Frontiers in Psychology 4:1–25.

    https://doi.org/10.3389/fpsyg.2013.00001

    • Google Scholar
25. 1. Daw ND
    2. Niv Y
    3. Dayan P

    (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control

    Nature Neuroscience 8:1704–1711.

    https://doi.org/10.1038/nn1560

    • Google Scholar
26. 1. Deng P
    2. Zhang Y
    3. Xu ZC

    (2007) Involvement of ih in dopamine modulation of tonic firing in striatal cholinergic interneurons

    Journal of Neuroscience 27:3148–3156.

    https://doi.org/10.1523/JNEUROSCI.5535-06.2007

    • Google Scholar
27. 1. Ding JB
    2. Guzman JN
    3. Peterson JD
    4. Goldberg JA
    5. Surmeier DJ

    (2010) Thalamic gating of corticostriatal signaling by cholinergic interneurons

    Neuron 67:294–307.

    https://doi.org/10.1016/j.neuron.2010.06.017

    • Google Scholar
28. 1. Doig NM
    2. Magill PJ
    3. Apicella P
    4. Bolam JP
    5. Sharott A

    (2014) Cortical and thalamic excitation mediate the multiphasic responses of striatal cholinergic interneurons to motivationally salient stimuli

    Journal of Neuroscience 34:3101–3117.

    https://doi.org/10.1523/JNEUROSCI.4627-13.2014

    • Google Scholar
29. 1. Doll BB
    2. Hutchison KE
    3. Frank MJ

    (2011) Dopaminergic genes predict individual differences in susceptibility to confirmation bias

    Journal of Neuroscience 31:6188–6198.

    https://doi.org/10.1523/JNEUROSCI.6486-10.2011

    • Google Scholar
30. 1. Doll BB
    2. Jacobs WJ
    3. Sanfey AG
    4. Frank MJ

    (2009) Instructional control of reinforcement learning: a behavioral and neurocomputational investigation

    Brain Research 1299:74–94.

    https://doi.org/10.1016/j.brainres.2009.07.007

    • Google Scholar
31. 1. Donoso M
    2. Collins AGE
    3. Koechlin E

    (2014) Foundations of human reasoning in the prefrontal cortex

    Science 344:1481–1486.

    https://doi.org/10.1126/science.1252254

    • Google Scholar
32. 1. Frank MJ
    2. Doll BB
    3. Oas-Terpstra J
    4. Moreno F

    (2009) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation

    Nature Neuroscience 12:1062–1068.

    https://doi.org/10.1038/nn.2342

    • Google Scholar
33. 1. Frank MJ
    2. Seeberger LC
    3. O'Reilly RC

    (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism

    Science 306:1940–1943.

    https://doi.org/10.1126/science.1102941

    • Google Scholar
34. 1. Frank MJ

    (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated parkinsonism

    Journal of Cognitive Neuroscience 17:51–72.

    https://doi.org/10.1162/0898929052880093

    • Google Scholar
35. 1. Frank MJ

    (2006) Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making

    Neural Networks 19:1120–1136.

    https://doi.org/10.1016/j.neunet.2006.03.006

    • Google Scholar
36. 1. Gerfen CR
    2. Surmeier DJ

    (2011) Modulation of striatal projection systems by dopamine

    Annual Review of Neuroscience 34:441–466.

    https://doi.org/10.1146/annurev-neuro-061010-113641

    • Google Scholar
37. 1. Gershman SJ
    2. Blei DM
    3. Niv Y

    (2010) Context, learning, and extinction

    Psychological Review 117:197–209.

    https://doi.org/10.1037/a0017808

    • Google Scholar
38. 1. Goldberg JA
    2. Reynolds JNJ

    (2011) Spontaneous firing and evoked pauses in the tonically active cholinergic interneurons of the striatum

    Neuroscience 198:27–43.

    https://doi.org/10.1016/j.neuroscience.2011.08.067

    • Google Scholar
39. 1. Gonzales KK
    2. Pare J-F
    3. Wichmann T
    4. Smith Y

    (2013) GABAergic inputs from direct and indirect striatal projection neurons onto cholinergic interneurons in the primate putamen

    Journal of Comparative Neurology 521:2502–2522.

    https://doi.org/10.1002/cne.23295

    • Google Scholar
40. 1. Gonzales KK
    2. Smith Y
    3. Uhl GR

    (2015) Cholinergic interneurons in the dorsal and ventral striatum: anatomical and functional considerations in normal and diseased conditions

    Annals of the New York Academy of Sciences 1349:1–45.

    https://doi.org/10.1111/nyas.12762

    • Google Scholar
41. 1. Graybiel A
    2. Aosaki T
    3. Flaherty A
    4. Kimura M

    (1994) The basal ganglia and adaptive motor control

    Science 265:1826–1831.

    https://doi.org/10.1126/science.8091209

    • Google Scholar
42. 1. Hinton GE
    2. Sejnowski TJ

    (1983)

    Optimal perceptual inference

    Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition pp. 448–453.

    • Google Scholar
43. 1. Howe MW
    2. Tierney PL
    3. Sandberg SG
    4. Phillips PEM
    5. Graybiel AM

    (2013) Prolonged dopamine signalling in striatum signals proximity and value of distant rewards

    Nature 500:575–579.

    https://doi.org/10.1038/nature12475

    • Google Scholar
44. 1. Humphries MD
    2. Stewart RD
    3. Gurney KN

    (2006) A physiologically plausible model of action selection and oscillatory activity in the basal ganglia

    Journal of Neuroscience 26:12921–12942.

    https://doi.org/10.1523/JNEUROSCI.3486-06.2006

    • Google Scholar
45. 1. Iglesias S
    2. Mathys C
    3. Brodersen KH
    4. Kasper L
    5. Piccirelli M
    6. den Ouden HEM
    7. Stephan KE

    (2013) Hierarchical prediction errors in midbrain and basal forebrain during sensory learning

    Neuron 80:519–530.

    https://doi.org/10.1016/j.neuron.2013.09.009

    • Google Scholar
46. 1. Jocham G
    2. Klein TA
    3. Ullsperger M

    (2011) Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices

    Journal of Neuroscience 31:1606–1613.

    https://doi.org/10.1523/JNEUROSCI.3904-10.2011

    • Google Scholar
47. 1. Joshua M
    2. Adler A
    3. Mitelman R
    4. Vaadia E
    5. Bergman H

    (2008) Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials

    Journal of Neuroscience 28:11673–11684.

    https://doi.org/10.1523/JNEUROSCI.3839-08.2008

    • Google Scholar
48. 1. Kapfer C
    2. Seidl AH
    3. Schweizer H
    4. Grothe B

    (2002) Experience-dependent refinement of inhibitory inputs to auditory coincidence-detector neurons

    Nature Neuroscience 5:247–253.

    https://doi.org/10.1038/nn810

    • Google Scholar
49. 1. Kravitz AV
    2. Tye LD
    3. Kreitzer AC

    (2012) Distinct roles for direct and indirect pathway striatal neurons in reinforcement

    Nature Neuroscience 15:816–818.

    https://doi.org/10.1038/nn.3100

    • Google Scholar
50. 1. Lau B
    2. Glimcher PW

    (2008) Value representations in the primate striatum during matching behavior

    Neuron 58:451–463.

    https://doi.org/10.1016/j.neuron.2008.02.021

    • Google Scholar
51. 1. Laurent V
    2. Bertran-Gonzalez J
    3. Chieng BC
    4. Balleine BW

    (2014)  -opioid and dopaminergic processes in accumbens shell modulate the cholinergic control of predictive learning and choice

    Journal of Neuroscience 34:1358–1369.

    https://doi.org/10.1523/JNEUROSCI.4592-13.2014

    • Google Scholar
52. 1. Ma WJ
    2. Beck JM
    3. Latham PE
    4. Pouget A

    (2006) Bayesian inference with probabilistic population codes

    Nature Neuroscience 9:1432–1438.

    https://doi.org/10.1038/nn1790

    • Google Scholar
53. 1. Maia TV
    2. Frank MJ

    (2011) From reinforcement learning models to psychiatric and neurological disorders

    Nature Neuroscience 14:154–162.

    https://doi.org/10.1038/nn.2723

    • Google Scholar
54. Book

    1. Marr DC
    2. Poggio T

    (1976)

    From Understanding Computation to Understanding Neural Circuitry

    Massachusetts Institute of Technology Cambridge Artificial Intelligence Lab.

    • Google Scholar
55. 1. Mathys C
    2. Daunizeau J
    3. Friston KJ
    4. Stephan KE

    (2011) A bayesian foundation for individual learning under uncertainty

    Frontiers in Human Neuroscience 5:39.

    https://doi.org/10.3389/fnhum.2011.00039

    • Google Scholar
56. 1. Matsumoto N
    2. Minamimoto T
    3. Graybiel AM
    4. Kimura M

    (2001)

    Neurons in the thalamic CM-pf complex supply striatal neurons with information about behaviorally significant sensory events

    Journal of Neurophysiology 85:960–976.

    • Google Scholar
57. 1. McClure SM
    2. Berns GS
    3. Montague PR

    (2003) Temporal prediction errors in a passive learning task activate human striatum

    Neuron 38:339–346.

    https://doi.org/10.1016/S0896-6273(03)00154-5

    • Google Scholar
58. 1. McCool MF
    2. Patel S
    3. Talati R
    4. Ragozzino ME

    (2008) Differential involvement of M1-type and M4-type muscarinic cholinergic receptors in the dorsomedial striatum in task switching

    Neurobiology of Learning and Memory 89:114–124.

    https://doi.org/10.1016/j.nlm.2007.06.005

    • Google Scholar
59. 1. Montague PR
    2. Dayan P
    3. Sejnowski TJ

    (1996)

    A framework for mesencephalic dopamine systems based on predictive hebbian learning

    Journal of Neuroscience 16:1936–1947.

    • Google Scholar
60. 1. Morris G
    2. Arkadir D
    3. Nevet A
    4. Vaadia E
    5. Bergman H

    (2004) Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons

    Neuron 43:133–143.

    https://doi.org/10.1016/j.neuron.2004.06.012

    • Google Scholar
61. 1. Nassar MR
    2. Wilson RC
    3. Heasly B
    4. Gold JI

    (2010) An approximately bayesian delta-rule model explains the dynamics of belief updating in a changing environment

    Journal of Neuroscience 30:12366–12378.

    https://doi.org/10.1523/JNEUROSCI.0822-10.2010

    • Google Scholar
62. 1. O'Doherty JP
    2. Dayan P
    3. Friston K
    4. Critchley H
    5. Dolan RJ

    (2003) Temporal difference models and reward-related learning in the human brain

    Neuron 38:329–337.

    https://doi.org/10.1016/S0896-6273(03)00169-7

    • Google Scholar
63. 1. O'Reilly RC
    2. Hazy TE
    3. Herd SA

    (2015) The leabra cognitive architecture

    Oxford Handbooks Online.

    https://doi.org/10.1093/oxfordhb/9780199842193.013.8

    • Google Scholar
64. Book

    1. O’Reilly RC
    2. Munakata Y

    (2000)

    Computational Explorations in Cognitive Neuroscience

    Cambridge: MIT Press.

    • Google Scholar
65. 1. Payzan-LeNestour E
    2. Bossaerts P
    3. Behrens T

    (2011) Risk, unexpected uncertainty, and estimation uncertainty: bayesian learning in unstable settings

    PLoS Computational Biology 7:e1001048.

    https://doi.org/10.1371/journal.pcbi.1001048

    • Google Scholar
66. 1. Pessiglione M
    2. Seymour B
    3. Flandin G
    4. Dolan RJ
    5. Frith CD

    (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans

    Nature 442:1042–1045.

    https://doi.org/10.1038/nature05051

    • Google Scholar
67. 1. Ragozzino ME
    2. Jih J
    3. Tzavos A

    (2002) Involvement of the dorsomedial striatum in behavioral flexibility: role of muscarinic cholinergic receptors

    Brain Research 953:205–214.

    https://doi.org/10.1016/S0006-8993(02)03287-0

    • Google Scholar
68. 1. Ragozzino ME
    2. Mohler EG
    3. Prior M
    4. Palencia CA
    5. Rozman S

    (2009) Acetylcholine activity in selective striatal regions supports behavioral flexibility

    Neurobiology of Learning and Memory 91:13–22.

    https://doi.org/10.1016/j.nlm.2008.09.008

    • Google Scholar
69. 1. Rao RPN

    (2005)

    Hierarchical bayesian inference in networks of spiking neurons

    Advances in Neural Information Processing Systems 17:1113–1120.

    • Google Scholar
70. 1. Ratcliff R
    2. Frank MJ

    (2012) Reinforcement-based decision making in corticostriatal circuits: mutual constraints by neurocomputational and diffusion models

    Neural Computation 24:1186–1229.

    https://doi.org/10.1162/NECO_a_00270

    • Google Scholar
71. 1. Ravel S
    2. Legallet E
    3. Apicella P

    (2003)

    Responses of tonically active neurons in the monkey striatum discriminate between motivationally opposing stimuli

    The Journal of Neuroscience 23:8489–8497.

    • Google Scholar
72. 1. Reynolds JNJ
    2. Wickens JR

    (2002) Dopamine-dependent plasticity of corticostriatal synapses

    Neural Networks 15:507–521.

    https://doi.org/10.1016/S0893-6080(02)00045-X

    • Google Scholar
73. 1. Rumelhart DE
    2. McClelland JL
    3. the PDP Research Group

    (1986)

    Computational Models of Cognition and Perception

    Parallel distributed processing: Explorations in the microstructure of cognition, Computational Models of Cognition and Perception, (Vol. 1).

    • Google Scholar
74. 1. Samejima K
    2. Ueda Y
    3. Doya K
    4. Kimura M

    (2005) Representation of action-specific reward values in the striatum

    Science 310:1337–1340.

    https://doi.org/10.1126/science.1115270

    • Google Scholar
75. 1. Shen W
    2. Flajolet M
    3. Greengard P
    4. Surmeier DJ

    (2008) Dichotomous dopaminergic control of striatal synaptic plasticity

    Science 321:848–851.

    https://doi.org/10.1126/science.1160575

    • Google Scholar
76. 1. Shen W
    2. Hamilton SE
    3. Nathanson NM
    4. Surmeier DJ

    (2005) Cholinergic suppression of KCNQ channel currents enhances excitability of striatal medium spiny neurons

    Journal of Neuroscience 25:7449–7458.

    https://doi.org/10.1523/JNEUROSCI.1381-05.2005

    • Google Scholar
77. 1. Shen W
    2. Tian X
    3. Day M
    4. Ulrich S
    5. Tkatch T
    6. Nathanson NM
    7. Surmeier DJ

    (2007) Cholinergic modulation of Kir2 channels selectively elevates dendritic excitability in striatopallidal neurons

    Nature Neuroscience 10:1458–1466.

    https://doi.org/10.1038/nn1972

    • Google Scholar
78. 1. Stocco A

    (2012) Acetylcholine-based entropy in response selection: a model of how striatal interneurons modulate exploration, exploitation, and response variability in decision-making

    Frontiers in Neuroscience 6:18.

    https://doi.org/10.3389/fnins.2012.00018

    • Google Scholar
79. Book

    1. Sutton RS
    2. Barto AG

    (1998)

    Reinforcement Learning: An Introduction

    Cambridge: MIT Press.

    • Google Scholar
80. 1. Tan CO
    2. Bullock D

    (2008) A dopamine-acetylcholine cascade: simulating learned and lesion-induced behavior of striatal cholinergic interneurons

    Journal of Neurophysiology 100:2409–2421.

    https://doi.org/10.1152/jn.90486.2008

    • Google Scholar
81. 1. Threlfell S
    2. Lalic T
    3. Platt NJ
    4. Jennings KA
    5. Deisseroth K
    6. Cragg SJ

    (2012) Striatal dopamine release is triggered by synchronized activity in cholinergic interneurons

    Neuron 75:58–64.

    https://doi.org/10.1016/j.neuron.2012.04.038

    • Google Scholar
82. 1. Tsai H-C
    2. Zhang F
    3. Adamantidis A
    4. Stuber GD
    5. Bonci A
    6. de Lecea L
    7. Deisseroth K

    (2009) Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning

    Science 324:1080–1084.

    https://doi.org/10.1126/science.1168878

    • Google Scholar
83. 1. Wang SS
    2. Denk W
    3. Häusser M

    (2000) Coincidence detection in single dendritic spines mediated by calcium release

    Nature Neuroscience 3:1266–1273.

    https://doi.org/10.1038/81792

    • Google Scholar
84. 1. Wilson CJ
    2. Goldberg JA

    (2006) Origin of the slow afterhyperpolarization and slow rhythmic bursting in striatal cholinergic interneurons

    Journal of Neurophysiology 95:196–204.

    https://doi.org/10.1152/jn.00630.2005

    • Google Scholar
85. 1. Wilson RC
    2. Nassar MR
    3. Gold JI

    (2010) Bayesian online learning of the hazard rate in change-point problems

    Neural Computation 22:2452–2476.

    https://doi.org/10.1162/NECO_a_00007

    • Google Scholar
86. 1. Witten IB
    2. Lin S-C
    3. Brodsky M
    4. Prakash R
    5. Diester I
    6. Anikeeva P
    7. Gradinaru V
    8. Ramakrishnan C
    9. Deisseroth K

    (2010) Cholinergic interneurons control local circuit activity and cocaine conditioning

    Science 330:1677–1681.

    https://doi.org/10.1126/science.1193771

    • Google Scholar
87. 1. Woolf NJ
    2. Butcher LL

    (1981) Cholinergic neurons in the caudate-putamen complex proper are intrinsically organized: a combined evans blue and acetylcholinesterase analysis

    Brain Research Bulletin 7:487–507.

    https://doi.org/10.1016/0361-9230(81)90004-6

    • Google Scholar
88. 1. Yagishita S
    2. Hayashi-Takagi A
    3. Ellis-Davies GCR
    4. Urakubo H
    5. Ishii S
    6. Kasai H

    (2014) A critical time window for dopamine actions on the structural plasticity of dendritic spines

    Science 345:1616–1620.

    https://doi.org/10.1126/science.1255514

    • Google Scholar
89. 1. Yu AJ
    2. Dayan P

    (2005) Uncertainty, neuromodulation, and attention

    Neuron 46:681–692.

    https://doi.org/10.1016/j.neuron.2005.04.026

    • Google Scholar

Author details

1. Nicholas T Franklin

   Department of Cognitive, Linguistic and Psychological Sciences, Brown Institute for Brain Science, Brown University, Providence, United States

   Contribution

   NTF, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   No competing interests declared.

2. Michael J Frank

   Department of Cognitive, Linguistic and Psychological Sciences, Brown Institute for Brain Science, Brown University, Providence, United States

   Contribution

   MJF, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   For correspondence

   Michael_Frank@brown.edu

   Competing interests

   MJF: Reviewing editor, eLife.

   "This ORCID iD identifies the author of this article:" 0000-0001-8451-0523

• 2,916

  views

• 558

  downloads

• 77

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.