Humans and other animals have a remarkable ability to adapt their decision making to changes in their environment. An experiment called the “multi-armed bandit task” shows this process in action. The individual’s role in this task is to choose between multiple targets. One of these has a higher probability of reward than the other three, and individuals soon begin to favor this target over the others. If the identity of the most rewarded target changes, individuals adjust their responses accordingly. Crucially, however, individuals learn more quickly when the identity of the most rewarded target changes frequently. In other words, they learn faster in an uncertain world.

Changes in the strength of connections between neurons – called synapses – are thought to underlie such learning processes. Receiving a reward strengthens synapses in a process referred to as synaptic plasticity. However, the standard model of synaptic plasticity – in which synapses change from weak to strong or vice versa at a constant rate – struggles to explain why individuals learn more quickly under variable conditions.

An alternative model of learning is the cascade model, which incorporates ‘metaplasticity’. This assumes that the rateof synaptic plasticity can also vary; that is, synapses change their strength at different speeds. The cascade model is based on the observation that multiple biochemical signaling cascades contribute to synaptic plasticity, and some of these are faster than others. Kiyohito Iigaya therefore decided to test whether the cascade model could explain data from experiments such as the four-armed bandit task. While the cascade model was indeed more flexible than the standard model of synaptic plasticity, it still could not fully explain the observed results.

Iigaya solved the problem by introducing an external “surprise detection system” into the model. Doing so enabled the model to detect a sudden change in the environment and to rapidly increase the rate of learning, just as individuals do in real life. The surprise detection system allowed synapses to quickly forget what they had learned before, which in turn made it easier for them to engage in new learning. The next step is to identify the circuit behind the surprise detection system: this will require further theoretical and experimental studies.

From neurons to behavior, evidence shows that adaptation takes place over a wide range of timescales, with temporal dynamics often captured by power-law, or collection of multiple exponents, rather than a single exponent (Thorson and Biederman-Thorson, 1974; Ulanovsky et al., 2004; Corrado et al., 2005; Fusi et al., 2007; Kording et al., 2007; Wark et al., 2009; Lundstrom et al., 2010; Rauch et al., 2003; Pozzorini et al., 2013). On the other hand, single-exponent model analysis of behavioral data showed that the time constant of exponents (or learning rate) changed across trials (Behrens et al., 2007; Rushworth and Behrens, 2008; Soltani et al., 2006; Nassar et al., 2010; Nassar et al., 2012; Neiman and Loewenstein, 2013; McGuire et al., 2014). While theoretical and experimental studies strongly suggest that activity-dependent synaptic plasticity plays a crucial role in learning and adaptation in general (Martin et al., 2000; Kandel et al., 2000; Dayan and Abbott, 2001), the neural mechanisms behind flexible learning, especially in the case of decision making under uncertainty, has remained unclear. To address this issue, here we investigate the roles of synaptic plasticity within an established decision-making neural circuit model, and propose a model that can account for empirical data.

Standard learning models which use a single learning rate, $\alpha$, fail to capture multiple timescales of adaptation, including those described by a power-law, since these models can only store and update memory on a single timescale of $\tau \sim 1/\alpha$. This includes a well studied switch-like synaptic model of memory (Amit and Fusi, 1994; Fusi and Abbott, 2007) in which synapses make transitions between weak- and strong-efficacy states at a rate $\alpha$. It has been shown that its transition rate $\alpha$ effectively functions as the learning rate of systems with populations of such synapses in a decision making network (Soltani and Wang, 2006; Fusi et al., 2007; Iigaya and Fusi, 2013). Unlike the classical un-bounded synapses model, this switch-like model incorporates a biologically relevant assumption of bounded synaptic weights. However, by itself, the plausible assumption of bounded synapses fails to capture key phenomena of adaptive learning, including well-documented multiple timescales of adaptation (Thorson and Biederman-Thorson, 1974; Ulanovsky et al., 2004; Corrado et al., 2005; Fusi et al., 2007; Kording et al., 2007; Wark et al., 2009; Lundstrom et al., 2010; Rauch et al., 2003; Pozzorini et al., 2013).

It is however known that there are various chemical cascade processes taking place in synapses that affect synaptic plasticity (Citri and Malenka, 2008; Kotaleski and Blackwell, 2010). Those processes, in general, operate on a wide range of timescales (Zhang et al., 2012; Kramar et al., 2012). To capture this complex, multi-timescale synaptic plasticity in a minimum form, a complex – but still switch-like – synaptic model, the cascade model of synapses, has been proposed (Fusi et al., 2005). In the cascade model, synapses are still bounded in their strengths but assumed to be metaplastic, meaning that, in addition to the usual case of adaptable synaptic strengths, synapses are also permitted to change their rates of plasticity $\alpha$. The resulting model can efficiently capture the widely-observed power-law forgetting curve (Wixted and Ebbesen, 1991). However, application has been limited to studies of the general memory storage problem (Fusi et al., 2005; Savin et al., 2014), where synapses passively undergo transitions in response to uncorrelated learning events.

Indeed, recent experiments show that humans and other animals have a remarkable ability to actively adapt themselves to changing environments. For instance, animals can react rapidly to abrupt step-like changes in environments (Behrens et al., 2007; Rushworth and Behrens, 2008; Soltani et al., 2006; Nassar et al., 2010; Nassar et al., 2012; Neiman and Loewenstein, 2013; McGuire et al., 2014), or change their strategies dynamically (Summerfield et al., 2011; Donoso et al., 2014). While the original cascade model (Fusi et al., 2005) is likely to be able to naturally encode multiple timescales of reward information (Corrado et al., 2005; Fusi et al., 2007; Bernacchia et al., 2011; Iigaya et al., 2013; Iigaya, 2013), such active adaptation may also require external guidance, such as in the form of a surprise signal (Hayden et al., 2011; Garvert et al., 2015).

So far the computational studies of such changes in learning rates have largely been limited to optimal Bayesian inference models (e.g. Behrens et al., 2007). While those models can account for normative aspects of animal’s inference and learning, they provide limited insight into how probabilistic inference can be implemented in neural circuits.

To address these issues, in this paper we apply the cascade model of synapses to a well studied decision-making network. Our primary finding is that the cascade model of synapses can indeed capture the remarkable flexibility shown by animals in changing environments, but under the condition that synaptic plasticity is guided by a novel surprise detection system with simple, non-cascade type synapses. In particular, we show that while the cascade model of synapses is able to consolidate reward information in a stable environment, it is severely limited in its ability to adapt to a sudden change in the environment. The addition of a surprise detection system, which is able to detect such abrupt changes, facilitates adaptation by enhancing the synaptic plasticity of the decision-making network. We also shows that our model can capture other aspects of learning, such as spontaneous recovery of preference (Mazur, 1996; Gallistel et al., 2001).

Humans and other animals have a remarkable ability to adapt to a changing environment. The neural circuit mechanism underling such behavioral adaptation has remained, however, largely unknown. While one might imagine that the circuits underlying such remarkable flexibility must be very complex, the current work suggests that a relatively simple, well-studied decision-making network, when combined with a relatively simple model of synaptic plasticity guided by a surprise detection system, can capture a wide range of existing data.

We should stress that there have been extensive studies of modulation of learning in conditioning tasks in psychology, inspired by two very influential proposals. The first was by Mackintosh (Mackintosh, 1975), in which he proposed that learning should be enhanced if a stimulus predicts rewards. In other words, a reward-irrelevant stimulus should be ignored, while a reward-predictive stimulus should continue to be attended to. This can be interpreted in our model in terms of formations of stimulus-selective neural populations in the decision making circuit. In other words, such a process would be equated with a shaping of the network architecture itself. This modification is beyond the scope of the current work, and we leave it as future work. The other influential proposal was made by Pearce and Hall (Pearce and Hall, 1980). They proposed that learning rates should be increased when an outcome was unexpected. This indeed is at the heart of the model proposed here, where unexpected uncertainty enhanced synaptic plasticity and hence the learning rate. Since the Pearce-Hall model focused on the algorithmic level of computation while our work focusing more on neural implementation level of computation, our work complements the classical model of Pearce and Hall (Pearce and Hall, 1980). We should, however, stress again that how our surprise detection system can be implemented should still be determined in the future.

In relation to surprise, the problem of change-point detection has long been studied in relation to the modulation of learning rates in reinforcement learning theory and Bayesian optimal learning theory (Pearce and Hall, 1980; Adams and MacKay, 2007; Dayan et al., 2000; Gallistel et al., 2001; Courville et al., 2006; Yu and Dayan, 2005; Behrens et al., 2007; Summerfield et al., 2011; Pearson and Platt, 2013; Wilson et al., 2013). These models, however, provided limited insight into how the algorithms can be implemented in neural circuits. To fill this gap, we proposed a computation which is partially performed by bounded synapses, and we found that our model performs as well as a Bayesian learner model (Behrens et al., 2007). We should, however, note that we did not specify a network architecture for our surprise detection system. A detailed architecture for this, including connectivity between neuronal populations, requires more experimental evidence. For example, how the difference in reward rates (subtraction) were computed in the network needs to be further explored theoretically and experimentally. One possibility is a network that includes two neural populations (X and Y), each of whose activity is proportional to its synaptic weights. Then one way to perform subtraction between these populations would be to have a readout population that receives an inhibitory projection from one population (X) and an excitatory projection from the other population (Y). The activity of the readout neurons would then reflect the subtraction of signals that are proportional to synaptic weights (Y–X).

Nonetheless, the surprise detection algorithm that we propose was previously hinted by Aston and Cohen (Aston-Jones and Cohen, 2005), where they suggested that task-relevant values computed in the anterior cingulate cortex (ACC) and the orbitofrontal cortex (OFC) are somehow integrated on multiple timescales and combined at the locus coeruleus (LC), as they proposed that the phasic and tonic release of norepinephrine (NE) controls the exploitation-exploration tradeoff. Here we showed that this computation can be carried out mainly by synaptic plasticity. We also related our computation to the notions of unexpected and expected uncertainties, which have been suggested to be correlated with NE and Acetylcholine (Ach) release, respectively (Yu and Dayan, 2005). In fact, there is increasing evidence that the activity of ACC relates to the volatility of the environment (Behrens et al., 2007) or surprise signal (Hayden et al., 2011). Also, there is a large amount of experimental evidence that Ach can enhance synaptic plasticity (Gordon et al., 2005; Mitsushima et al., 2013). This could imply that our surprise signal could be expressed as the balance between Ach and NE. On the other hand, in relation to encoding reward history over multiple timescales, it is well known that the phasic activity of dopaminergic neurons reflects a reward prediction error (Schultz et al., 1997), while tonic dopamine levels may reflect reward rates (Niv et al., 2007); these signals could also play crucial roles in our multiple timescales of reward integration process. We also note that a similar algorithm for the surprise detection was recently suggested in a reduced Bayesian framework (Wilson et al., 2013).

In this paper, we assume that the surprise signals are sent when the incoming reward rate decreases unexpectedly, so that the cascade model synapses can increase the rate of plasticity and reset memory. However, there are other cases where surprise signals could be sent to modify the rates of plasticity. For example, when the incoming reward rate is dramatically increased, surprise signals could enhance the metaplastic transitions so that the memory of recent action values are rapidly consolidated. Also, in response to an unexpected punishment rather than reward, surprise signals could be sent to enhance the metaplastic transitions to achieve a one-shot memory (Schafe et al., 2001). Furthermore, the effect of the surprise signal may not be limited to reward-based learning. An unexpected recall of episodic memory could itself also trigger a surprise signal. This may explain some aspects of memory reconsolidation (Schafe et al., 2001).

Our model has some limitations. First, we mainly focused on a relatively simple decision making task, where one of the targets is more rewarding than the other and the reward rates for targets change at the same time. In reality, however, it is also possible that reward rates of different targets change independently. In this case it would be preferable to selectively change learning rates for different targets, which might be solved by incorporating an additional mechanism such as synaptic tagging (Clopath et al., 2008; Barrett et al., 2009). Second, although we assumed that the surprise signal would reset most of the accumulated evidence when reward-harvesting performance deteriorates, in many cases it would be better to keep accumulated evidence, such as to form distinct ’contexts’ (Gershman et al., 2010; Lloyd and Leslie, 2013). This would allow subjects to access it later. This type of operation may require further neural populations to be added to the decision making circuit that we studied. In fact, it has been shown that introducing neurons that are randomly connected to neurons in the decision making network can solve context dependent decision-making tasks (Rigotti et al., 2010; Barak et al., 2013). Those randomly connected neurons were reported in the prefrontal cortex (PFC) as ‘mixed-selective’ neurons (Rigotti et al., 2013). It would be interesting to introduce such neuronal populations to our model to study more complex tasks.

Also, distributing memory among different brain areas may also allow flexible access of memory on different timescales, or hierarchical structure of contexts, if the rates of synaptic plasticity are similarly distributed amongst different brain areas, with memory information being transferred from one area to another (Squire and Wixted, 2011). Indeed, it has recently been shown that such a partitioning could also be advantageous general memory performance (Roxin and Fusi, 2013), and this could be incorporated with relative ease into our model. One possibility is that the value signals computed by the cascade model synapses with a different range of timescales in distinct brain areas are combined to make decisions, so that the surprise signal is sent to the appropriate brain areas with the targeted rates of plasticity and contexts.

Some of the key features of our model remain to be tested as predictions. One is that the synapses encoding action values in the decision making network should change the level of plasticity itself. In other words, those synapses that reach the boundary of synaptic strength should become more resilient to change. For example, if rewards are given every trial from the same target, the synaptic strength targeting such target would reach the boundary, say after 100 trials. This means that the synaptic strength would remain the same, even after 1000 trials. However, the synapses after 1000 trials should be more resilient to change than synapse after 100 trials. Equally, the synapses that encode overall reward rates, or subject’s performance, in a surprise detection system should not make meta-plastic transitions. Thus, studying the nature of synaptic plasticity may allow us to dissociate the functions of circuits.

While we found that our model is robust to parameter changes, the effect of extreme parameter values may give insights into psychiatric and personality disorders. For example, if the threshold of the surprise signal, ${\displaystyle h}$, is extremely low, the model can become inflexible in the face of changes in the environment. On the other hand, if the threshold is extremely high, the model cannot consolidate the values of actions, leading to unstable behavior. As these sorts of maladaptive behaviors are common across different psychiatric and personality disorders, our model could potentially provide insights into the circuit level dynamics underlying aspects of these disorders (Deisseroth, 2014).

Our model consists of two systems: (1) the decision making network, which makes decisions according the actions values stored in plastic synapses (2) the surprise detection system, which computes expected uncertainties and unexpected uncertainties on multiple timescales to send a surprise signal to the decision making network, when the unexpected uncertainty exceeds the expected uncertainty.

The decision making network with cascade type synapses

Request a detailed protocol

The decision making network (Soltani and Wang, 2006; Fusi et al., 2007; Wang, 2008; Iigaya and Fusi, 2013) is illustrated in Figure 1A. In this neural circuit, two groups of excitatory neurons (decision populations), each of which is selective to an action of choosing each target stimuli (A or B), receive inputs from sensory neurons on each trial. Each of the excitatory populations are recurrently connected to sustain their activity during each trial. In addition, they inhibit with each other through a inhibitory neuronal population.

As a result of the inhibitory interaction, the firing rate of one population of excitatory neurons become much larger than the other population (winner take all process) (Wang, 2002). This is a stable state of this attractor network, and we assume that subject’s action is determined by the winning population (selecting A or B).

Soltani and Wang (Soltani and Wang, 2006) showed in simulations of a such network with spiking neurons that the decision of the attractor network is stochastic, but the probability of choosing a particular target can be well fitted by a sigmoid function of the difference between the synaptic input currents $I_A-I_B$ from the sensory neurons to the action selective populations A and B:

(2) ${\displaystyle {\displaystyle P_A=1-P_B=\frac{1}{e^{-\frac{I_A-I_B}{T}}+1},}}$

where $P_A$ is the probability of choosing target $A$ and the temperature $T$ is a free parameter determined by the amount of noise in the network.

The afferent currents $I_A$ and $I_B$ are proportional to the synaptic weights between the input population of neurons and the two decision populations of neurons. The current to a neuron that belongs to the decision of selecting target A can be expressed as:

(3) ${\displaystyle {\displaystyle I_A=\sum _{j=1}^N{w}_j^A{\nu }_j}}$

where the ${\displaystyle {\nu }_j}$’s are the firing rates of the $i$-th neuron (of the total of $N$ neurons) in the input population and ${\displaystyle w_j^A}$ is the synaptic weight to the population selective to A. An analogous expression holds for the $I_B$ and we assume that $N$ is the same for both populations. Assuming that the firing rates of input population is approximately to be uniform ${\nu }_j=\nu$, we can simplify the expression of the current:

(4) $I_A={\displaystyle \sum _{j=1}^N}{w}_j^A\nu =\nu N{⟨w⟩}_A$

where ${⟨w⟩}_A$ is the average synaptic weight to the population selective to A. Here we can assume $\nu N=1$ without any loss of generality, as we can rescale $T$ as $T/\nu N\to T$. Also any overlapping of selectivity or any other noise in those two decision making populations can be incorporated to the temperature parameter $T$ in our model.

Following (Fusi et al., 2005), the cascade model of synapses assumes that each synaptic strength is binary – either depressed or potentiated, with the value of 0 or 1, respectivey. This follows the important constrant of bounded synapses (Amit and Fusi, 1994; Fusi and Abbott, 2007), and it has been shown that having intermediate strength between 0 and 1 does not significantly improve model’s memory performance (Fusi and Abbott, 2007) or decision-making behavior (Iigaya and Fusi, 2013). In addition, the cascade model of synapses (Fusi et al., 2005; Soltani and Wang, 2006; Iigaya and Fusi, 2013) assumes synapses can take different levels of plasticity. Following (Iigaya and Fusi, 2013), we assume there are $m$ states in this dimension.

Instead of simulating the dynamics of all individual synapse, it is more convenient to keep track of the distribution of synapses over the synaptic state space:

(5) ${\displaystyle \sum _{i=1}^m{F}_i^{A-}+\sum _{i=1}^m{F}_i^{A+}=1,}$

where $F_i^{A-}$ ($F_i^{A+}$) is the fraction of synapses occupying the depressed (potentiated) state at the $i$’th level of the plasticity state in the population targeting the action of choosing $A$. The same can be written for the synapses targeting the neural population selective to target B. As we assume that the synaptic strength is 0 for the depressed states and 1 for the potentiated states, the total (normalized) synaptic strength can be expressed as

(6) ${\displaystyle ⟨w{⟩}_A=\sum _{i=1}^m{F}_i^{A+}.}$

Again, an analogous relation holds for the synaptic population between the input neurons and the neurons selective to choosing target B.

Hence the action of choosing A or B is determined by the decision making network as:

(7) ${\displaystyle {\displaystyle P_A=\frac{1}{e^{-\frac{\sum _{i=1}^m(F_i^{A+}-F_i^{B+})}{T}}+1}}}$

Thus the decision is biased by the synapses occupying the potentiated states, which reflects the memory of past rewards that is updated according to a learning rule. Here we apply the standard activity dependent reward-based learning rule (Fusi et al., 2007; Soltani and Wang, 2006; Soltani et al., 2006; Iigaya and Fusi, 2013) to the cascade model. This is schematically shown in Figure 6. When the network received a reward after choosing target A, the synapses between input population and the action selective population that is targeting the just rewarded action A (note that these neurons have a higher firing rates than the other population) make transitions as following.

${\displaystyle \begin{array}{}\text{(8)}& F_1^{A+}& \to F_1^{A+}+\sum _{i=1}^m{\alpha }_r^i{F}_i^{A-}-p_r^1{F}_1^{A+}& \text{(9)}& F_{1<i<m}^{A+}& \to F_{1<i<m}^{A+}+p_r^i{F}_{i-1}^{A+}-p_r^{i+1}{F}_i^{A+}& \text{(10)}& F_m^{A+}& \to F_m^{A+}-p_r^i{F}_{m-1}^{A+}& \text{(11)}& F_{1<i<m}^{A-}& \to F_{1<i<m}^{A-}-{\alpha }_r^i{F}_{1<i<m}^{A-}& \end{array}}$

where ${\alpha }_r^i$ is the transition probability to modify synaptic strength (between depressed 0 and 1) from the $i$’th level to the first level after rewards, and $p_r^i$ is the metaplastic transition probability from $i$’th (upper) level to $i+1$’th (lower) level after a reward. In words, the synapses at depressed states make stochastic transitions to the most plastic potentiated state, while the synapses that were already at potentiated states make stochastic transitions to deeper, or less plastic, states (see Figure 6).

Figure 6

Download asset Open asset

For the synapses tarting un-chosen population, we assume the opposite learning:

${\displaystyle \begin{array}{}\text{(12)}& F_1^{B-}& \to F_1^{B-}+\sum _{i=1}^m\gamma {\alpha }_r^i{F}_i^{B+}-\gamma p_r^1{F}_1^{B-}& \text{(13)}& F_{1<i<m}^{B-}& \to F_{1<i<m}^{B-}+\gamma p_r^i{F}_{i-1}^{B-}-\gamma p_r^{i+1}{F}_i^{B-}& \text{(14)}& F_m^{B-}& \to F_m^{B-}+\gamma p_r^i{F}_{m-1}^{B-}& \text{(15)}& F_{1<i<m}^{B+}& \to F_{1<i<m}^{B+}+\gamma {\alpha }_r^i{F}_i^{B+}& \end{array}}$

where $\gamma <1$ is the factor determining the probability of chaining states of synapses targeting an unchosen action at a given trial. In words, the synapses at potentiated states make stochastic transitions to the most plastic depressed state, while the synapses that were already at depressed states make stochastic transitions to deeper, or less plastic, states (see Figure 6).

Similarly, when the network received no reward after choosing target A, synapses change their states as:

${\displaystyle \begin{array}{}\text{(17)}& F_1^{A-}& \to F_1^{A-}+\sum _{i=1}^m{\alpha }_{nr}^i{F}_i^{A+}-p_{nr}^1{F}_1^{A-}& \text{(18)}& F_{1<i<m}^{A-}& \to F_{1<i<m}^{A-}+p_{nr}^i{F}_{i-1}^{A-}-p_{nr}^{i+1}{F}_i^{A-}& \text{(19)}& F_m^{A-}& \to F_m^{A-}+p_{nr}^i{F}_{m-1}^{A-}& \text{(20)}& {\displaystyle F_{1<i<m}^{A-}}& \to F_{1<i<m}^{A-}-{\alpha }_{nr}^i{F}_{1<i<m}^{A-}& \end{array}}$

and

${\displaystyle \begin{array}{}\text{(21)}& F_1^{B+}& \to F_1^{B+}+\sum _{i=1}^m\gamma {\alpha }_{nr}^i{F}_i^{B-}-p_{nr}^1{F}_1^{B+}& \text{(22)}& F_{1<i<m}^{B+}& \to F_{1<i<m}^{B+}+\gamma p_{nr}^i{F}_{i-1}^{B+}-\gamma p_{nr}^{i+1}{F}_i^{B+}& \text{(23)}& F_m^{B+}& \to F_m^{B+}+\gamma p_{nr}^i{F}_{m-1}^{B+}& \text{(24)}& F_{1<i<m}^{B-}& \to F_{1<i<m}^{B-}-\gamma {\alpha }_{nr}^i{F}_{1<i<m}^{B-}& \end{array}}$

where ${\alpha }_{nr}^i$ is the transition probability from the $i$’th state to the first state in case of no reward, and ${\displaystyle p_{nr}^i}$ is the metaplastic transition probability from $i$’th (upper) level to $i+1$’th (lower) level after no reward. Unless otherwise noted, in this paper we set ${\displaystyle {\alpha }_n^i={\alpha }_{nr}^i(={\alpha }_i)}$ and ${\displaystyle p_n^i=p_{nr}^i(=p_i)}$.

In Figure 5, we also simulating the effect of inter-session-interval (ISI). To do this, we simply assumed that random noisy events drive forgetting during the ISIs. This was simulated simply by letting synapses undergo what we define as forgetting transitions (Figure 7):

${\displaystyle \begin{array}{}\text{(25)}& F_1^{A+}& \to F_1^{A+}+\sum _{i=1}^m{\alpha }_i{F}_i^{A-}& \text{(26)}& F_{1<i<m}^{A-}& \to F_{1<i<m}^{A-}-{\alpha }_i{F}_{1<i<m}^{A-}& \end{array}}$

and

${\displaystyle \begin{array}{}\text{(27)}& F_1^{A-}& \to F_1^{A-}+\sum _{i=1}^m{\alpha }_i{F}_i^{A+}& \text{(28)}& F_{1<i<m}^{A+}& \to F_{1<i<m}^{A+}-{\alpha }_i{F}_{1<i<m}^{A+}.& \end{array}}$

In Figure 5, we assume the unit of ISI, ${\displaystyle T_{\text{ISI}}}$, is 100 repetition of these transitions. We found that our qualitative finding is robust against the setting of threshold value $h$. We did not allow metaplastic (downward) transitions during forgetting, since we focused on the forgetting aspect of ISI, which was sufficient to account for the data (Mazur, 1996).

Figure 7

Download asset Open asset

The surprise detection system

Request a detailed protocol

Here we describe our surprise detection system. We do not intend to specify detailed circuit architecture of the surprise detection system. Rather, we propose a simple computation algorithm that can be partially implementable by well-studied bounded synaptic plasticity. As detailed circuits of a surprise detection system have yet to be shown either theoretically or experimentally, we leave a problem of specifying the architecture of system to future studies.

In summary, this system (1) computes reward rates on different timescales (2) computes expected differences between the reward rates of different timescales (we call this as expected uncertainty) (3) compares the expected uncertainty with the current actual difference between reward rates (we call this unexpected uncertainty) (4) sends a surprise signal to the decision making network, if the unexpected uncertainty exceeds the expected uncertainty. As a result, the system receives an input of a reward or no-reward every trial, and sends an output of surprise or no-surprise to the decision making network.

It has been shown that a population of binary synapses can encode the rate of rewards on a timescale of $\tau =1/\alpha$, where $\alpha$ is the rate of synaptic plasticity (Rosenthal et al., 2001; Iigaya and Fusi, 2013). Here we use this property to monitor reward rates on multiple timescales, by introducing populations of synapses with different rates of plasticity. Since the goal of this system is to monitor incoming reward rates on which the cascade model synapses in the decision making network operates, we assume the total of $m$ populations of synapses, where $m$ is the same as the number of metaplastic states of the cascade model synapses. Accordingly, synapses in population $i$ have the plasticity rate of ${\alpha }_r^i$, which is the same rate as the cascade model’s transition rate at the $i$’th level. Crucially, we assume these synapses are not meta-plastic. They simply undergo reward-dependent stochastic learning; but importantly, this time they do so independent of a chosen action so that the system can keep track of overall performance.

It is again convenient to keep track of the distribution of synapses in the state space. We write the fraction of synapses at the depressed state is $G_i^{-}$, and the fraction of synapses at potentiated state is $G_i^{+}$:

(29) ${\displaystyle G_i^{-}+G_i^{+}=1,}$

Assuming that the synaptic strength is either 0 (depressed) or 1 (potentiated), the total synaptic strength $Z_i$ of population $i$ is simply

(30) $Z_i=G_i^{+}n,$

where $n$ is the total number of synapses. For simplicity, we assume each population has the same number of synapses. While $Z_i$ is the value that should be read out by a readout, without a loss of generality, we keep track of the normalized weight $R_i=Z_i/n=G_i^{+}$ as the synaptic strength.

The distribution changes according to a simple reward based plasticity rule (Iigaya and Fusi, 2013). When a network receives a reward,

${\displaystyle \begin{array}{}\text{(31)}& G_i^{+}& \to G_i^{+}+{\alpha }_r^i{G}_i^{-}& \text{(32)}& G_i^{-}& \to G_i^{-}-{\alpha }_r^i{G}_i^{-},& \end{array}}$

which means that the synapses at the depressed state make transitions to the potentiated state with a probability of ${\alpha }_r^i$. When the network received no reward, on the other hand,

${\displaystyle \begin{array}{}\text{(33)}& G_i^{-}& \to G_i^{-}+{\alpha }_{nr}^i{G}_i^{+}& \text{(34)}& G_i^{+}& \to G_i^{+}-{\alpha }_{nr}^i{G}_i^{+},& \end{array}}$

which means that the synapses at the potentiated state make transitions to the depressed state with a probability of ${\alpha }_{nr}^i$. The transition rate ${\alpha }_{nr}^i$ is designed to match the transition rate of the cascade model in case of no-reward. (In this paper we set ${\displaystyle {\alpha }_r^i={\alpha }_{nr}^i(={\alpha }^i)}$, as is also the case in the cascade model synapses in the decision making network.) These transitions take place independent of the taken action, and the synaptic strength $v_i=Z_i/n=G_i^{+}$ is a low-pass filtered (by bounded synapses) of reward rates on a timescale ${\tau }_i=1/{\alpha }^i$.

On each trial, the system also computes the expected uncertainty $u_{i,j}$ of reward rates between different timescales of synaptic populations. Note that for this we focus on the computational algorithm, and we do not specify the architecture of neural circuits responsible for this computation. As detailed circuits of a surprise detection system have yet to be shown either theoretically or experimentally, we leave a problem of specifying the architecture of system to future studies. The system learns the absolute value of the difference between the approximated reward rates $v_i$ and $v_j$ at a rate of $min({\alpha }^j,{\alpha }^i)$:

(35) ${\displaystyle u_{i,j}=u_{i,j}+min({\alpha }^i,{\alpha }^j)(|v_i-v_j|-u_{i,j}),}$

where we assume that the learning rate is a smaller rate of plasticity in the two populations. We call $u_{i,j}$ as the expected uncertainty between $i$ and $j$ (Yu and Dayan, 2005), representing the how different the reward rates of different timescales are expected to be. We also call the actual current difference $|v_i-v_j|$ as unexpected uncertainty between $i$ and $j$. Hence the expected uncertainty is the low-pass filtered unexpected uncertainty, both of which dynamically change over trials.

On each trial, the system also compares the expected uncertainty $u_{i,j}$ and unexpected uncertainty $|v_i-v_j|$ for each pair of $i$ and $j$. If the latter significantly exceeds the former, $|v_i-v_j|\gg u_{i,j}$, then the system sends an output of a surprise signal to the decision making network. For simplicity, we set the threshold $h$ as ${\displaystyle erf(\frac{v_i-v_j}{\sqrt{2}{u}_{i,j}})=h}$ when ${\displaystyle i>j}$, where ${\displaystyle erf(.)}$ is the error function. Note that the error function is sign sensitive. Thus when ${\displaystyle v_i>v_j}$, or when the reward rate is increasing locally in time, surprise signal is not sent if the threshold is set to be $h<0.5$. This threshold $h$ is a free parameter; but we confirmed that the system is robust over a wide range of $h$.

If a surprise signal is sent, because of the discrepancy between two timescales $i$ and $j$, $|v_i-v_j|\gg u_{i,j}$, the decision making network (cascade synapses) increase the rates of plasticity. Importantly this is done only for the levels of synapses that the surprise is detected (the lower levels do not change the rates of plasticity). This allows the decision-making network to keep information on different timescales as long as it is useful. For example, when a surprise was detected between $i$’th and $j$’th levels, we set the cascade model of transition rates

(36) ${\displaystyle {\displaystyle {\alpha }^k\to {\alpha }^1}}$

for $k\le j$ of the cascade model synapses. This allows the decision making network to reset the memory and adapt to a new environment. Note that this change of the rate of synopses is only for the cascade model synapses. The synapses in the surprise detection system do not change the rate of plasticity.

Figure 8 illustrates how the whole system of the decision making network and the surprise detection work together. We simulated our model in a two-choice VI schedule task with a total baiting probability of $0.4$. The reward contingency was reversed every 100 trials. The mean synaptic strength of each population $v_i$ is shown in Figure 8D, while each pair was compared separetly in Figure 8E–G. Surprises were detected mostly between $v_2$ and $v_3$, or between $v_1$ and $v_3$, (Figure 8I), but not between $v_1$ and $v_2$. This makes sense because the timescale of block change was 100 trial, which is similar to the timescale of $v_3$: ${\displaystyle 1/{\alpha }_3=25}$ trials. Thus the timescale of ${\displaystyle v_2}$ was too short to detect this change: ${\displaystyle 1/{\alpha }_3=25}$ trials. Thanks to the surprise signals, the cascade model of synapses were able to adapt to the sudden changes in contingency (Figure 8B,C). As a result, the choice probability also adapt to the environment (Figure 8A).

Figure 8

Download asset Open asset

1. 1. Adams RP
   2. MacKay DJ

   (2007)

   Bayesian online changepoint detection

   arXiv 0710.3742.

   • Google Scholar
2. 1. Amit DJ
   2. Fusi S

   (1994) Learning in neural networks with material synapses

   Neural Computation 6:957–982.

   https://doi.org/10.1162/neco.1994.6.5.957

   • Google Scholar
3. 1. Aston-Jones G
   2. Cohen JD

   (2005) An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance

   Annual Review of Neuroscience 28:403–450.

   https://doi.org/10.1146/annurev.neuro.28.061604.135709

   • Google Scholar
4. 1. Barak O
   2. Rigotti M
   3. Fusi S

   (2013) The sparseness of mixed selectivity neurons controls the generalization-discrimination trade-off

   Journal of Neuroscience 33:3844–3856.

   https://doi.org/10.1523/JNEUROSCI.2753-12.2013

   • Google Scholar
5. 1. Barrett AB
   2. Billings GO
   3. Morris RG
   4. van Rossum MC

   (2009) State based model of long-term potentiation and synaptic tagging and capture

   PLoS Computational Biology 5:e1000259.

   https://doi.org/10.1371/journal.pcbi.1000259

   • Google Scholar
6. 1. Behrens TE
   2. Woolrich MW
   3. Walton ME
   4. Rushworth MF

   (2007) Learning the value of information in an uncertain world

   Nature Neuroscience 10:1214–1221.

   https://doi.org/10.1038/nn1954

   • Google Scholar
7. 1. Bernacchia A
   2. Seo H
   3. Lee D
   4. Wang XJ

   (2011) A reservoir of time constants for memory traces in cortical neurons

   Nature Neuroscience 14:366–372.

   https://doi.org/10.1038/nn.2752

   • Google Scholar
8. 1. Citri A
   2. Malenka RC

   (2008) Synaptic plasticity: multiple forms, functions, and mechanisms

   Neuropsychopharmacology 33:18–41.

   https://doi.org/10.1038/sj.npp.1301559

   • Google Scholar
9. 1. Clopath C
   2. Ziegler L
   3. Vasilaki E
   4. Büsing L
   5. Gerstner W

   (2008) Tag-trigger-consolidation: a model of early and late long-term-potentiation and depression

   PLoS Computational Biology 4:e1000248.

   https://doi.org/10.1371/journal.pcbi.1000248

   • Google Scholar
10. 1. Corrado GS
    2. Sugrue LP
    3. Seung HS
    4. Newsome WT

    (2005) Linear-nonlinear-poisson models of primate choice dynamics

    Journal of the Experimental Analysis of Behavior 84:581–617.

    https://doi.org/10.1901/jeab.2005.23-05

    • Google Scholar
11. 1. Courville AC
    2. Daw ND
    3. Touretzky DS

    (2006) Bayesian theories of conditioning in a changing world

    Trends in Cognitive Sciences 10:294–300.

    https://doi.org/10.1016/j.tics.2006.05.004

    • Google Scholar
12. Book

    1. Dayan P
    2. Abbott LF

    (2001)

    Theoretical Neuroscience : Computational and Mathematical Modeling of Neural Systems

    Cambridge, Mass: Massachusetts Institute of Technology Press.

    • Google Scholar
13. 1. Dayan P
    2. Kakade S
    3. Montague PR

    (2000) Learning and selective attention

    Nature Neuroscience 3:1218–1223.

    https://doi.org/10.1038/81504

    • Google Scholar
14. 1. Deisseroth K

    (2014) Circuit dynamics of adaptive and maladaptive behaviour

    Nature 505:309–317.

    https://doi.org/10.1038/nature12982

    • Google Scholar
15. 1. Donoso M
    2. Collins AG
    3. Koechlin E

    (2014) Human cognition. Foundations of human reasoning in the prefrontal cortex

    Science 344:1481–1486.

    https://doi.org/10.1126/science.1252254

    • Google Scholar
16. 1. Fusi S
    2. Abbott LF

    (2007) Limits on the memory storage capacity of bounded synapses

    Nature Neuroscience 10:485–493.

    https://doi.org/10.1038/nn1859

    • Google Scholar
17. 1. Fusi S
    2. Asaad WF
    3. Miller EK
    4. Wang XJ

    (2007) A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales

    Neuron 54:319–333.

    https://doi.org/10.1016/j.neuron.2007.03.017

    • Google Scholar
18. 1. Fusi S
    2. Drew PJ
    3. Abbott LF

    (2005) Cascade models of synaptically stored memories

    Neuron 45:599–611.

    https://doi.org/10.1016/j.neuron.2005.02.001

    • Google Scholar
19. 1. Gallistel CR
    2. Mark TA
    3. King AP
    4. Latham PE

    (2001) The rat approximates an ideal detector of changes in rates of reward: Implications for the law of effect

    Journal of Experimental Psychology 27:354–372.

    https://doi.org/10.1037/0097-7403.27.4.354

    • Google Scholar
20. 1. Garvert MM
    2. Moutoussis M
    3. Kurth-Nelson Z
    4. Behrens TE
    5. Dolan RJ

    (2015) Learning-induced plasticity in medial prefrontal cortex predicts preference malleability

    Neuron 85:418–428.

    https://doi.org/10.1016/j.neuron.2014.12.033

    • Google Scholar
21. 1. Gershman SJ
    2. Blei DM
    3. Niv Y

    (2010) Context, learning, and extinction

    Psychological Review 117:197–209.

    https://doi.org/10.1037/a0017808

    • Google Scholar
22. 1. Gordon GR
    2. Baimoukhametova DV
    3. Hewitt SA
    4. Rajapaksha WR
    5. Fisher TE
    6. Bains JS

    (2005) Norepinephrine triggers release of glial ATP to increase postsynaptic efficacy

    Nature Neuroscience 8:1078–1086.

    https://doi.org/10.1038/nn1498

    • Google Scholar
23. 1. Hayden BY
    2. Heilbronner SR
    3. Pearson JM
    4. Platt ML

    (2011) Surprise signals in anterior cingulate cortex: neuronal encoding of unsigned reward prediction errors driving adjustment in behavior

    Journal of Neuroscience 31:4178–4187.

    https://doi.org/10.1523/JNEUROSCI.4652-10.2011

    • Google Scholar
24. 1. Herrnstein RJ

    (1961) Relative and absolute strength of response as a function of frequency of reinforcement

    Journal of the Experimental Analysis of Behavior 4:267–272.

    https://doi.org/10.1901/jeab.1961.4-267

    • Google Scholar
25. 1. Iigaya K

    (2013)

    Neural network models of decision making with learning on multiple timescales

    Ph.D. Thesis, Columbia University (New York, NY, USA).

    • Google Scholar
26. 1. Iigaya K
    2. Fusi S

    (2013) Dynamical regimes in neural network models of matching behavior

    Neural Computation 25:1–20.

    https://doi.org/10.1162/NECO_a_00522

    • Google Scholar
27. 1. Iigaya K
    2. Sugrue L
    3. Corrado GS
    4. Loewenstein Y
    5. Newsome W
    6. Fusi S

    (2013)

    Deviations from the matching law reflect reward integration over multiple timescales

    Cosyne Abstract.

    • Google Scholar
28. Book

    1. Kandel ER
    2. Schwartz JH
    3. Jessell TM

    (2000)

    Principles of Neural Science. Vol. 4

    New York: McGraw-Hill.

    • Google Scholar
29. 1. Kording KP
    2. Tenenbaum JB
    3. Shadmehr R

    (2007) The dynamics of memory as a consequence of optimal adaptation to a changing body

    Nature Neuroscience 10:779–786.

    https://doi.org/10.1038/nn1901

    • Google Scholar
30. 1. Kotaleski JH
    2. Blackwell KT

    (2010) Modelling the molecular mechanisms of synaptic plasticity using systems biology approaches

    Nature Reviews Neuroscience 11:239–251.

    https://doi.org/10.1038/nrn2807

    • Google Scholar
31. 1. Kramar EA
    2. Babayan AH
    3. Gavin CF
    4. Cox CD
    5. Jafari M
    6. Gall CM
    7. Rumbaugh G
    8. Lynch G

    (2012) Synaptic evidence for the efficacy of spaced learning

    PNAS 109:5121–5126.

    https://doi.org/10.1073/pnas.1120700109

    • Google Scholar
32. 1. Lau B
    2. Glimcher PW

    (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys

    Journal of the Experimental Analysis of Behavior 84:555–579.

    https://doi.org/10.1901/jeab.2005.110-04

    • Google Scholar
33. 1. Lloyd K
    2. Leslie DS

    (2013) Context-dependent decision-making: a simple Bayesian model

    Journal of the Royal Society Interface 10:20130069.

    https://doi.org/10.1098/rsif.2013.0069

    • Google Scholar
34. 1. Lundstrom BN
    2. Fairhall AL
    3. Maravall M

    (2010) Multiple timescale encoding of slowly varying whisker stimulus envelope in cortical and thalamic neurons in vivo

    Journal of Neuroscience 30:5071–5077.

    https://doi.org/10.1523/JNEUROSCI.2193-09.2010

    • Google Scholar
35. 1. Mackintosh NJ

    (1975) A theory of attention: Variations in the associability of stimuli with reinforcement

    Psychological Review 82:276–298.

    https://doi.org/10.1037/h0076778

    • Google Scholar
36. 1. Martin SJ
    2. Grimwood PD
    3. Morris RG

    (2000) Synaptic plasticity and memory: an evaluation of the hypothesis

    Annual Review of Neuroscience 23:649–711.

    https://doi.org/10.1146/annurev.neuro.23.1.649

    • Google Scholar
37. 1. Mazur JE

    (1996) Past experience, recency, and spontaneous recovery in choice behavior

    Animal Learning & Behavior 24:1–10.

    https://doi.org/10.3758/BF03198948

    • Google Scholar
38. 1. McGuire JT
    2. Nassar MR
    3. Gold JI
    4. Kable JW

    (2014) Functionally dissociable influences on learning rate in a dynamic environment

    Neuron 84:870–881.

    https://doi.org/10.1016/j.neuron.2014.10.013

    • Google Scholar
39. 1. Mitsushima D
    2. Sano A
    3. Takahashi T

    (2013) A cholinergic trigger drives learning-induced plasticity at hippocampal synapses

    Nature Communications 4:2760.

    https://doi.org/10.1038/ncomms3760

    • Google Scholar
40. 1. Nassar MR
    2. Rumsey KM
    3. Wilson RC
    4. Parikh K
    5. Heasly B
    6. Gold JI

    (2012) Rational regulation of learning dynamics by pupil-linked arousal systems

    Nature Neuroscience 15:1040–1046.

    https://doi.org/10.1038/nn.3130

    • Google Scholar
41. 1. Nassar MR
    2. Wilson RC
    3. Heasly B
    4. Gold JI

    (2010) An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment

    Journal of Neuroscience 30:12366–12378.

    https://doi.org/10.1523/JNEUROSCI.0822-10.2010

    • Google Scholar
42. 1. Neiman T
    2. Loewenstein Y

    (2013) Covariance-based synaptic plasticity in an attractor network model accounts for fast adaptation in free operant learning

    Journal of Neuroscience 33:1521–1534.

    https://doi.org/10.1523/JNEUROSCI.2068-12.2013

    • Google Scholar
43. 1. Niv Y
    2. Daw ND
    3. Joel D
    4. Dayan P

    (2007) Tonic dopamine: opportunity costs and the control of response vigor

    Psychopharmacology 191:507–520.

    https://doi.org/10.1007/s00213-006-0502-4

    • Google Scholar
44. 1. Pearce JM
    2. Hall G

    (1980) A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli

    Psychological Review 87:532–552.

    https://doi.org/10.1037/0033-295X.87.6.532

    • Google Scholar
45. 1. Pearson JM
    2. Platt ML

    (2013) Change detection, multiple controllers, and dynamic environments: insights from the brain

    Journal of the Experimental Analysis of Behavior 99:74–84.

    https://doi.org/10.1002/jeab.5

    • Google Scholar
46. 1. Pozzorini C
    2. Naud R
    3. Mensi S
    4. Gerstner W

    (2013) Temporal whitening by power-law adaptation in neocortical neurons

    Nature Neuroscience 16:942–948.

    https://doi.org/10.1038/nn.3431

    • Google Scholar
47. 1. Rauch A
    2. La Camera G
    3. Luscher HR
    4. Senn W
    5. Fusi S

    (2003) Neocortical pyramidal cells respond as integrate-and-fire neurons to in vivo-like input currents

    Journal of Neurophysiology 90:1598–1612.

    https://doi.org/10.1152/jn.00293.2003

    • Google Scholar
48. 1. Rescorla RA

    (2004) Spontaneous recovery

    Learning & Memory 11:501–509.

    https://doi.org/10.1101/lm.77504

    • Google Scholar
49. 1. Rigotti M
    2. Barak O
    3. Warden MR
    4. Wang XJ
    5. Daw ND
    6. Miller EK
    7. Fusi S

    (2013) The importance of mixed selectivity in complex cognitive tasks

    Nature 497:585–590.

    https://doi.org/10.1038/nature12160

    • Google Scholar
50. 1. Rigotti M
    2. Ben Dayan Rubin D
    3. Wang XJ
    4. Fusi S

    (2010) Internal representation of task rules by recurrent dynamics: the importance of the diversity of neural responses

    Frontiers in Computational Neuroscience 4:24.

    https://doi.org/10.3389/fncom.2010.00024

    • Google Scholar
51. 1. Rosenthal O
    2. Fusi S
    3. Hochstein S

    (2001) Forming classes by stimulus frequency: Behavior and theory

    PNAS 98:4265–4270.

    https://doi.org/10.1073/pnas.071525998

    • Google Scholar
52. 1. Roxin A
    2. Fusi S

    (2013) Efficient partitioning of memory systems and its importance for memory consolidation

    PLoS Computational Biology 9:e1003146.

    https://doi.org/10.1371/journal.pcbi.1003146

    • Google Scholar
53. 1. Rushworth MF
    2. Behrens TE

    (2008) Choice, uncertainty and value in prefrontal and cingulate cortex

    Nature Neuroscience 11:389–397.

    https://doi.org/10.1038/nn2066

    • Google Scholar
54. 1. Sakai Y
    2. Fukai T

    (2008) The actor-critic learning is behind the matching law: matching versus optimal behaviors

    Neural Computation 20:227–251.

    https://doi.org/10.1162/neco.2008.20.1.227

    • Google Scholar
55. 1. Savin C
    2. Dayan P
    3. Lengyel M

    (2014) Optimal recall from bounded metaplastic synapses: predicting functional adaptations in hippocampal area CA3

    PLoS Computational Biology 10:e1003489.

    https://doi.org/10.1371/journal.pcbi.1003489

    • Google Scholar
56. 1. Schafe GE
    2. Nader K
    3. Blair HT
    4. LeDoux JE

    (2001) Memory consolidation of Pavlovian fear conditioning: a cellular and molecular perspective

    Trends in Neurosciences 24:540–546.

    https://doi.org/10.1016/S0166-2236(00)01969-X

    • Google Scholar
57. 1. Schultz W
    2. Dayan P
    3. Montague PR

    (1997) A neural substrate of prediction and reward

    Science 275:1593–1599.

    https://doi.org/10.1126/science.275.5306.1593

    • Google Scholar
58. 1. Soltani A
    2. Lee D
    3. Wang XJ

    (2006) Neural mechanism for stochastic behaviour during a competitive game

    Neural Networks 19:1075–1090.

    https://doi.org/10.1016/j.neunet.2006.05.044

    • Google Scholar
59. 1. Soltani A
    2. Wang XJ

    (2006) A biophysically based neural model of matching law behavior: melioration by stochastic synapses

    Journal of Neuroscience 26:3731–3744.

    https://doi.org/10.1523/JNEUROSCI.5159-05.2006

    • Google Scholar
60. 1. Soltani A
    2. Wang XJ

    (2010) Synaptic computation underlying probabilistic inference

    Nature Neuroscience 13:112–119.

    https://doi.org/10.1038/nn.2450

    • Google Scholar
61. 1. Squire LR
    2. Wixted JT

    (2011) The cognitive neuroscience of human memory since H.M

    Annual Review of Neuroscience 34:259–288.

    https://doi.org/10.1146/annurev-neuro-061010-113720

    • Google Scholar
62. 1. Sugrue LP
    2. Corrado GS
    3. Newsome WT

    (2004) Matching behavior and the representation of value in the parietal cortex

    Science 304:1782–1787.

    https://doi.org/10.1126/science.1094765

    • Google Scholar
63. 1. Summerfield C
    2. Behrens TE
    3. Koechlin E

    (2011) Perceptual classification in a rapidly changing environment

    Neuron 71:725–736.

    https://doi.org/10.1016/j.neuron.2011.06.022

    • Google Scholar
64. 1. Thorson J
    2. Biederman-Thorson M

    (1974) Distributed relaxation processes in sensory adaptation

    Science 183:161–172.

    https://doi.org/10.1126/science.183.4121.161

    • Google Scholar
65. 1. Ulanovsky N
    2. Las L
    3. Farkas D
    4. Nelken I

    (2004) Multiple time scales of adaptation in auditory cortex neurons

    Journal of Neuroscience 24:10440–10453.

    https://doi.org/10.1523/JNEUROSCI.1905-04.2004

    • Google Scholar
66. 1. Wang XJ

    (2002) Probabilistic decision making by slow reverberation in cortical circuits

    Neuron 36:955–968.

    https://doi.org/10.1016/S0896-6273(02)01092-9

    • Google Scholar
67. 1. Wang XJ

    (2008) Decision making in recurrent neuronal circuits

    Neuron 60:215–234.

    https://doi.org/10.1016/j.neuron.2008.09.034

    • Google Scholar
68. 1. Wark B
    2. Fairhall A
    3. Rieke F

    (2009) Timescales of inference in visual adaptation

    Neuron 61:750–761.

    https://doi.org/10.1016/j.neuron.2009.01.019

    • Google Scholar
69. 1. Wilson RC
    2. Nassar MR
    3. Gold JI

    (2013) A mixture of delta-rules approximation to bayesian inference in change-point problems

    PLoS Computational Biology 9:e1003150.

    https://doi.org/10.1371/journal.pcbi.1003150

    • Google Scholar
70. 1. Wixted JT
    2. Ebbesen EB

    (1991) On the form of forgetting

    Psychological Science 2:409–415.

    https://doi.org/10.1111/j.1467-9280.1991.tb00175.x

    • Google Scholar
71. 1. Yu AJ
    2. Dayan P

    (2005) Uncertainty, neuromodulation, and attention

    Neuron 46:681–692.

    https://doi.org/10.1016/j.neuron.2005.04.026

    • Google Scholar
72. 1. Zhang Y
    2. Liu R-Y
    3. Heberton GA
    4. Smolen P
    5. Baxter DA
    6. Cleary LJ
    7. Byrne JH

    (2012) Computational design of enhanced learning protocols

    Nature Neuroscience 15:294–297.

    https://doi.org/10.1038/nn.2990

    • Google Scholar

Author details

1. Kiyohito Iigaya

   1. Gatsby Computational Neuroscience Unit, University College London, London, United Kingdom
   2. Center for Theoretical Neuroscience, College of Physicians and Surgeons, Columbia University, New York, United States
   3. Department of Physics, Columbia University, New York, United States

   Contribution

   KI, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   For correspondence

   kiigaya@gatsby.ucl.ac.uk

   Competing interests

   The author declares that no competing interests exist.

   "This ORCID iD identifies the author of this article:" 0000-0002-4748-8432

• 2,552

  views

• 513

  downloads

• 42

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.