Learning predictive cognitive maps with spiking neurons during behavior and replays
Abstract
The hippocampus has been proposed to encode environments using a representation that contains predictive information about likely future states, called the successor representation. However, it is not clear how such a representation could be learned in the hippocampal circuit. Here, we propose a plasticity rule that can learn this predictive map of the environment using a spiking neural network. We connect this biologically plausible plasticity rule to reinforcement learning, mathematically and numerically showing that it implements the TDlambda algorithm. By spanning these different levels, we show how our framework naturally encompasses behavioral activity and replays, smoothly moving from rate to temporal coding, and allows learning over behavioral timescales with a plasticity rule acting on a timescale of milliseconds. We discuss how biological parameters such as dwelling times at states, neuronal firing rates and neuromodulation relate to the delay discounting parameter of the TD algorithm, and how they influence the learned representation. We also find that, in agreement with psychological studies and contrary to reinforcement learning theory, the discount factor decreases hyperbolically with time. Finally, our framework suggests a role for replays, in both aiding learning in novel environments and finding shortcut trajectories that were not experienced during behavior, in agreement with experimental data.
Editor's evaluation
This is an important article that leverages a spiking network model of the hippocampal circuit to show how spiketimedependent plasticity can implement predictive reinforcement learning and form a predictive map of the environment. The authors provide a convincing and solid framework for understanding the prediction based learning rules that may be employed by the hippocampus to optimize an animal's behavior. This paper will be of interest to theoretical and experimental neuroscientists working on learning and memory as it provides new ways to connect computational models to experimental data that has yet to be fully explored from a reinforcement learning perspective.
https://doi.org/10.7554/eLife.80671.sa0Introduction
Mid twentieth century, Tolman proposed the concept of cognitive maps (Tolman, 1948). These maps are abstract mental models of an environment which are helpful when learning tasks and in decision making. Since the discovery of hippocampal place cells, cells that are activated only in specific locations of an environment, it is believed that the hippocampus can provide the substrate to encode such cognitive maps (O’Keefe and Dostrovsky, 1971; O’Keefe and Nadel, 1978). More evidence of the role of the hippocampus in behavior was found in numerous experimental studies, such as the seminal water maze experiments (Morris, 1981; Morris et al., 1982), radial arm maze experiments (Olton and Papas, 1979) as well as evidence of broader information processing beyond just cognitive maps (Wood et al., 1999; Eichenbaum et al., 1999; Aggleton and Brown, 1999; Wood et al., 2000).
While these place cells offer striking evidence in favour of cognitive maps, it is not clear what representation is actually learned by the hippocampus and how this information is exploited when solving and learning tasks. Recently, it was proposed that the hippocampus computes a cognitive map containing predictive information, called the successor representation (SR). Theoretically, this SR framework has some computational advantages, such as efficient learning, simple computation of the values of states, fast relearning when the rewards change and flexible decision making (Dayan, 1993; Stachenfeld et al., 2014; Stachenfeld et al., 2017; Russek et al., 2017; Momennejad et al., 2017). Furthermore, the SR is in agreement with experimental observations. Firstly, the firing fields of hippocampal place cells are affected by the strategy used by the animal to navigate the environment (known as the policy in machine learning), as well as by changes in the environment (Mehta et al., 2000; Stachenfeld et al., 2017). Secondly, reward revaluation — the ability to recompute the values of the states when rewards change — would be more effective than transition revaluation (Russek et al., 2017; Momennejad et al., 2017).
In this work, we study how this predictive representation can be learned in the hippocampus with spiketiming dependent synaptic plasticity (STDP). Using STDP at the mechanistic level, we show that the learning is equivalent to TD($\lambda $) on an algorithmic level. The latter is a wellstudied and powerful algorithm known from reinforcement learning (Sutton and Barto, 1998), which we will discuss in more detail below.
Our model can thus learn over a behavioral timescale while using STDP timescales in the millisecond range. We show mathematically that our proposed framework smoothly connects a temporally precise spiking code akin to replay activity with a rate based code akin to behavioral spiking. Subsequently, we show that the delaydiscounting parameter $\gamma $ allows us to consider time as a continuous variable, therefore we don’t need to discretize time as is usual in reinforcement learning (Doya, 1995; Doya, 2000). Moreover, the delaydiscounting in our model depends hyperbolically on time but exponentially on state transitions. We show how the $\gamma $ parameter can be modulated by neuronal firing rates and neuromodulation, allowing statedependent discounting and in turn enabling richer information in the SR, such as the encoding of salient states, landmarks, reward locations, etc. Finally, replays have long been speculated to be involved in learning models of the environment, supported by experiments (Johnson and Redish, 2007; Pfeiffer and Foster, 2013; Kay et al., 2020) and models (Hasselmo and Eichenbaum, 2005; Erdem and Hasselmo, 2012; Kubie and Fenton, 2012). Here, we investigate how replays could play an additional role in learning the SR cognitive map. Following properties of TD($\lambda $), we show how we can achieve both low bias and low variance by using replays, translating to both quicker initial learning and convergence to lower error. We show how we can use replays to learn offline. In this way, policies can be refined without the need for actual exploration.
Our framework allows us to make predictions about the roles of behavioral learning and replaylike activity and how they can be exploited in representation learning. Furthermore, we uncover a relation between STDP and a higher level learning algorithm. Our work therefore spans the three levels of analysis proposed by Marr, 2010. On the implementational level, our model consists of a feedforward network of excitatory neurons with biologically plausible spiketiming dependent plasticity. On the algorithmic level, we show that our model learns the successor representation using the TD($\lambda $) algorithm. On the computational theory level, our model tackles representation learning using cognitive maps.
Results
Cognitive maps are internal models of an environment which help animals to learn, plan and make decisions during task completion. The hippocampus has long been thought to provide the substrate for learning such cognitive maps (O’Keefe and Dostrovsky, 1971; O’Keefe and Nadel, 1978; Morris, 1981; Morris et al., 1982; Wood et al., 1999; Eichenbaum et al., 1999), and recent evidence points towards a specific type of representation learned by the hippocampus, the successor representation (SR) (Stachenfeld et al., 2017).
The successor representation
In this section, we will give an overview of the successor representation and its properties, especially geared toward neuroscientists. Readers already familiar with this representation may safely move to the next section.
To understand the concept of successor representation (SR), we can consider a spatial environment — such as a maze — while an animal explores this environment. In this setting, the SR can be understood as how likely it is for the animal to visit a future location starting from its current position. We further assume the maze to be formed out of a discrete number of states. Then, the SR can be more formally described by a matrix with dimension (${N}_{states}\times {N}_{states}$), where ${N}_{states}$ denotes the number of states in the environment and each entry ${R}_{ij}$ of this matrix describes the expected future occupancy of a state ${S}_{j}$ when the current state is ${S}_{i}$. In other words, starting from ${S}_{i}$, the more likely it is for the animal to reach the location associated with state ${S}_{j}$ and the nearer in the future, the higher the value of ${R}_{ij}$.
As a first example, we consider an animal running through a linear track. We assume the animal runs at a constant speed and always travels in the same direction — left to right (Figure 1a). We also split the track into four sections or states, ${S}_{1}$ to ${S}_{4}$, and the SR will be represented by a matrix with dimension ($4\times 4$). Since the animal always runs from left to right, there is zero probability of finding the animal at position $i$ if its current position is greater than $i$. Therefore, the lower triangle of the successor matrix is equal to zero (Figure 1b). Alternatively, if the animal is currently at position ${S}_{1}$, it will be subsequently found at positions ${S}_{2}$, ${S}_{3}$, and ${S}_{4}$ with probability 1. The further away from ${S}_{1}$, the longer it will take the animal to reach that other position. In terms of the successor matrix, we apply a discounting factor $\gamma $ ($\displaystyle 0<\gamma \le 1$) for each extra ‘step’ required by the animal to reach a respective location (Figure 1b).
Even though we introduced the linear track as an illustrative example, the SR can be learned in any environment (see Figure 1—figure supplement 1 for an example in an open field). Note that the representation learned by the SR is dependent not only on the structure of the environment, but also on the policy — or strategy — used by the animal to explore the environment. This is because the successor representation is not purely concerned with the physical distance between two areas in the environment, but rather it measures how long it usually takes to reach one place when starting from the other. In this first example, the animal applied a deterministic policy (always running from left to right), but the SR can also be learned for stochastic policies. Furthermore, the SR is a multistep representation, in the sense that it stores predictive information of multiple steps ahead.
Because of this predictive information, the SR allows sampleefficient relearning when the reward location is changed (Gershman, 2018). In reinforcement learning, we tend to distinguish between modelfree and modelbased algorithms. The SR is believed to sit inbetween these two modalities. In modelfree reinforcement learning, the aim is to directly learn the value of each state in the environment. Since there is no model of the environment at all, if the location of a reward is changed, the agent will have to first unlearn the previous reward location by visiting it enough times, and only then it will be able to relearn the new location. In modelbased reinforcement learning, a precise model of the environment is learned, specifically, singlestep transition probabilities between all states of the environment. Modelbased learning is computationally expensive, but allows a certain flexibility. If the reward changes location it is immediate to derive the updated values of the states. As we have seen, however, the SR can relearn a new reward location somewhat efficiently, although less so than modelbased learning. The SR can also be efficiently learned using modelfree methods and allows us to easily compute values for each state, which in turn can guide the policy (Dayan, 1993; Russek et al., 2017; Momennejad et al., 2017). This position between modelbased and modelfree methods makes the SR framework very powerful, and its similarities with hippocampal neuronal dynamics have led to increased attention from the neuroscience community. Finally, in our examples above we considered an environment made up of a discrete number of states. This framework can be generalised to a continuous environment represented by a discrete number of place cells.
Learning the successor representation in biologically plausible networks
We propose a model of the hippocampus that is able to learn the successor representation. We consider a feedforward network comprising of two layers. Similar to McNaughton and Morris, 1987; Hasselmo and Schnell, 1994; Mehta et al., 2000; Hasselmo et al., 2002, we assume that the presynaptic layer represents the hippocampal CA3 region and is alltoall connected to a postsynaptic layer  representing the CA1 network (Figure 1c). The synaptic connections from CA3 to CA1 are plastic such that the weight changes follow a spiketimingdependent plasticity (STDP) rule consisting of two terms: a weightdependent depression term for presynaptic spikes and a potentiation term for prepost spike pairs (Figure 1d).
For simplicity, we assume that the animal spends a fixed time $T$ in each state. During this time, a constant activation current is delivered to the CA3 neuron encoding the current location and, after a delay, to the corresponding CA1 place cell (see Materials and methods). On top of these fixed and locationdependent activations, the CA3 neurons can activate neurons in CA1 through the synaptic connections. In other words, the CA3 neurons are activated according to the current location of the animal, while the CA1 neurons have a similar locationdependent activity combined with activity caused by presynaptic neurons. The constant currents delivered directly to CA3 and CA1 neurons can be thought of as locationdependent currents from entorhinal cortex. These activations subsequently trigger plasticity at the synapses, and we can show analytically that, using the spiketiming dependent plasticity rule discussed above, the SR is learned in the synaptic weights (Figure 1e and f, and see Appendix).
Moreover, we find that, on an algorithmic level, our weight updates are equivalent to a learning algorithm known as TD($\lambda $), a powerful and wellknown algorithm in reinforcement learning that can be used to learn the successor representation. TD($\lambda $) is based on a mixed methodology, which is regulated by the parameter $\lambda $. At one extreme, when $\lambda =1$, the SR is estimated by taking the average of state occupancies over past trajectories. This type of algorithm is called TD(1) or Monte Carlo (MC). At the other extreme, when $\lambda =0$, the estimate of the SR is adjusted ‘online’, with every step of the trajectory, by comparing the observed position with its predicted value. This algorithm is equivalent to TD(0). For all values of $\lambda $ in between, the algorithm employs a mixture of both methodologies. The extreme cases of TD(1) and TD(0) have different strengths and weaknesses, as we will discuss in more detail in the next sections.
In practice, we prove analytically the mathematical equivalence of the dynamics of our spiking neural network, and the TD($\lambda $) algorithm (see Appendix). Our calculations essentially prove that, at each step, our neural network tracks the reinforcement learning algorithm, known to converge to the theoretical values of the SR. This equivalence guarantees that our neural network weights will eventually converge to the correct SR matrix. As a proof of principle, we show that it is possible to learn the SR for any initial weights (Figure 1—figure supplement 2), independently of any previous learning in the CA3 to CA1 connections.
Importantly, from our analytical derivations (see Appendix), we find that the $\lambda $ parameter depends on the behavioral parameter T (the time an animal spends in a state). We find that, the larger the time T, the smaller the value of $\lambda $ and viceversa. In other words, when the animal moves through the trajectory on behavioral timescales (large T compared to the synaptic plasticity timescales ${\tau}_{\text{LTP}}$), the network is learning the SR with TD($\lambda \sim 0$). For quick sequential activities (T → 0), akin to hippocampal replays, the network is learning the SR with TD($\lambda \sim 1$). As we will discuss below, this framework therefore combines learning based on rate coding as well as temporal coding. Furthermore, from our model follows the prediction that replays can also be used for learning purposes and that they are algorithmically equivalent to MC, whereas during behavior, the hippocampal learning algorithm is equivalent to TD($\lambda $). This strategy of using replays to learn is in line with recent experimental and theoretical observations (see Momennejad, 2020 for a review).
To validate our analytical results, we use again a linear track with a deterministic policy. Using our spiking model with either ratecode activity on behavioral timescales (Figure 2a top) or temporalcode activity similar to replays (Figure 2b top), we show that the synaptic weights across trials match the evolution of the TD($\lambda $) algorithm closely (Figure 2a and b middle). While convergence to the SR is guaranteed (Figure 2a and b bottom) due to the mathematical equivalence between our setup and TD($\lambda $) (Figure 2—figure supplement 1), the learning trajectory has more variance in the neural network case due to the noise introduced by the randomness of the spike times. This noise can be mitigated by averaging over a population of neurons. Moreover, due to the equivalence with TD($\lambda $), our setup is general for any type of task where discrete states are visited, in any dimension, and which may not need to be a navigation task (see e.g. Figure 1—figure supplement 1 for a 2D environment).
In summary, we showed how the network can learn the SR using a spiking neural model. We analytically showed how the learning algorithm is equivalent to TD($\lambda $), and confirmed this using numerical simulations. We derived a relationship between the abstract parameter $\lambda $ and the timescale T representing the animal’s behavior — and in turn the neuronal spiking — allowing us to unify rate and temporal coding within one framework. Furthermore, we predict a role for hippocampal replays in learning the SR using an algorithm equivalent to Monte Carlo.
Learning over behavioral timescales using STDP
An important observation in our framework is that the SR can be learned using the same underlying STDP rule over timescales ranging from replays up to behavior. One can now wonder how it is possible to learn relationships between events that are seconds apart during awake behavior, without any explicit error encoding signal typically used by the TD algorithm, and while the STDP rule is characterised by millisecond timescales (Figure 3a).
From a neuroscience perspective, this can be understood when considering the trajectory of the animal. Each time the animal moves from a position ${S}_{j1}$ to a position ${S}_{j}$, the CA3 cell encoding the location ${S}_{j1}$ stops firing and the CA3 cell encoding the location ${S}_{j}$ starts firing. Since in our example this transition is instantaneous, these cells are activating the same CA1 cells consecutively. Therefore, the change in the weight ${w}_{i,j1}$ depend on the synaptic weight of the subsequent state ${w}_{i,j}$ (Figure 3b, yellow depends on orange, orange depends on red, etc). Indeed, in our example of an animal in a linear track subdivided into four locations, the weights on the diagonal, such as ${w}_{4,4}$, are the first ones to be learned, since they are learned directly. The offdiagonal weights, such as ${w}_{3,4}$, ${w}_{2,4}$, and ${w}_{1,4}$, are learned consecutively more slowly as they are dependent on the subsequent synaptic weight. Eventually, weights between neurons encoding positions that are behaviorally far apart can be learnt using a learning rule on a synaptic timescale (Figure 3b).
From a reinforcement learning perspective, the TD(0) algorithm relies on a property called bootstrapping. This means that the successor representation is learned by first taking an initial estimate of the SR matrix (i.e. the previously learned weights), and then gradually adjusting this estimate (i.e. the synaptic weights) by comparing it to the states in the environment that the animal actually visits. This comparison is achieved by calculating a prediction error, similar to the widely studied one for dopamine neurons (Schultz et al., 1997). Since the synaptic connections carry information about the expected trajectories, in this case, the prediction error is computed between the predicted and observed trajectories (see Materials and methods).
The main point of bootstrapping, therefore, is that learning happens by adjusting our current predictions (e.g. synaptic weights) to match the observed current state. This information is available at each time step and thus allows learning over long timescales using synaptic plasticity alone. If the animal moves to a state in the environment that the current weights deem unlikely, potentiation will prevail and the weight from the previous to the current state will increase. Otherwise, the opposite will happen. It is important to notice that the prediction error in our model is not encoded by a separate mechanism in the way that dopamine is thought to do for reward prediction (Schultz et al., 1997). Instead, the prediction error is represented locally, at the level of the synapse, through the depression and potentiation terms of our STDP rule, and the current weight encodes the current estimate of the SR (see Materials and methods). Notably, the prediction error is equivalent to the TD($\lambda $) update. This mathematical equivalence ensures that the weights of our neural network track the TD($\lambda $) update at each state, and thus stability and convergence to the theoretical values of the SR. We therefore do not need an external vector to carry prediction error signals as proposed in Gardner et al., 2018; Gershman, 2018. In fact, the synaptic potentiation in our model updates a row of the SR, while the synaptic depression updates a column.
On the other extreme, for very fast timescales such as replays, TD(1) is equivalent to online Monte Carlo learning (MC), which does not bootstrap at all. Instead, MC samples the whole trajectory and then simply takes the average of the discounted state occupancies to update the SR (see Materials and methods). During replays, the whole trajectory falls under the plasticity window and the network can learn without bootstrapping. For all cases in between, the network partially relies on bootstrapping and we correspondingly find a $\lambda $ between 0 and 1.
In summary, in our framework, synaptic plasticity leads to the development of a successor representation in which synaptic weights can be directly linked to the successor matrix. In this framework, we can learn over behavioral timescales even though our plasticity rule acts on the scale of milliseconds, due to the bootstrapping property of TD algorithms.
Different discounting for space and time
In reinforcement learning, it is usual to have delaydiscounting: rewards that are further away in the future are discounted compared to rewards that are in the immediate future. Intuitively, it is indeed clear that a state leading to a quick reward can be regarded as more valuable compared to a state that only leads to an equal reward in the distant future. For tasks in a tabular setting, with a discrete state space and where actions are taken in discrete turns, such as for example chess or our simple linear track discussed in section ‘The Successor Representation’, one can simply use a multiplicative factor $\displaystyle 0<\gamma \le 1$ for each state transition. In this case the discount follows an exponential dependence, where rewards that are $n$ steps away are discounted by a factor of ${\gamma}^{n}$.
In order to still use the above exponential discount when time is continuous, the usual approach is to discretize time by choosing a unit of time. However, this would imply one can never remain in a state for a fraction of this unit of time, and it is not clear how this unit would be chosen. Our framework deals naturally with continuous time, through the monotonically decreasing dependence of the discount parameter $\gamma $ on the time an agent remains in a state, T. The dependence on T can be interpreted as an increased discounting the longer a state lasts.
In this way, instead of discounting by ${\gamma}^{n}$ when the agent stays $n$ units of time in a certain state, we would discount by $\gamma \left(n\cdot T\right)$. More generally, for any arbitrary time T, a discount corresponding to $\gamma \left(T\right)$ will be applied. This allows the agent to act in continuous time (Figure 3c and e). Interestingly, the dependence of $\gamma $ on T in our model is not exponential as in the tabular case. Instead, we have a hyperbolic dependence. This hyperbolic discount is well studied in psychology and neuroeconomics and appears to agree well with experimental results (Laibson, 1997; Ainslie, 2012).
The difference between a hyperbolic discount and an exponential discount lays in the fact that we will attribute a different value to the same temporal delay, depending on whether it happens sooner or later. A classic example is that, when given the choice, people tend to prefer 100 dollars today instead of 101 dollars tomorrow, while they tend to prefer 101 dollars in 31 days instead of 100 dollars in 30 days. They therefore judge the 1 day of delay differently when it happens later in time. Exponential discounting, on the other hand, always attributes the same value to the same delay no matter when it occurs.
Our model therefore combines two types of discounting: exponential when we move through space — when sequentially activating different place cells — and hyperbolic when we move through time — when we prolong the activity of the same place cell.
The discount factor $\gamma $ also depends on other parameters such as firing rate and STDP amplitudes (see Equation 22 in the Appendix). This gives our model the flexibility to encode statedependent discounting even when the trajectories and times spent in the states are the same. Such statedependent discounting can be useful to for example encode salient locations in the environment such as landmarks or reward locations (Figure 3c and f).
Biasvariance tradeoff
As discussed previously (section ‘Learning the successor representation in biologically plausible networks’), the TD($\lambda $) algorithm unifies the TD algorithm and the MC algorithm. In our framework, replaylike neuronal activations are equivalent to MC, while behaviorallike activity is equivalent to TD. In this section, we will discuss how the replays and behavior can work together when learning the cognitive map of an environment, leveraging the strengths of MC and TD.
The MC algorithm effectively works by averaging over the sampled trajectories. As such, the estimated SR matrix will be a close approximation of the theoretical value. The difference between the estimated and theoretical value is commonly referred to as bias. We can therefore say that the MC algorithm presents low bias. However, if the agent moves in the environment at random, the sampled trajectories will be quite different from each other. When taking the average, the estimated value will therefore fluctuate a lot. In this case, we say that the MC estimate has high variance as well (Figure 4A and B).
Unlike MC, the TD algorithm updates its estimate of the SR by comparing the current estimate of the SR with the actual state the agent transitioned to. Because of the dependence on the current estimate, this estimate will be incrementally refined with small updates. In this way, the SR estimate will not fluctuate much, and be lower in variance. However, by this dependence on the current estimate, we introduce a bias in the algorithm, which will be especially significant when our initial estimate of the SR is bad (Figure 4A and B). The TD algorithm therefore presents high bias and low variance.
We now apply these concepts to learning in a novel environment. Since the MC algorithm is unbiased by the initial estimate of the SR, replays should initially speed up learning in an unfamiliar environment. Later on, when the environment becomes familiar, the SR estimate is already closer to the exact value. At this point, we prefer to have low variance and thus the TD algorithm will be preferred. We confirm this logic using our spiking neural networks, and show how we can have both quick learning and low error at convergence if we proportionally have more replays at the first trials in a novel environment (Figure 4a–e). In contrast, when having an equal proportion of replays throughout the whole simulation, we do not yield as quick learning as MC and as low asymptotic error as TD (Figure 4—figure supplement 1). Interestingly, the pattern of proportionally more replays in novel environments versus familiar environments has also been experimentally observed (Cheng and Frank, 2008; Figure 4f). Please note that, while we implemented an exponentially decaying probability for replays after entering a novel environment, different schemes for replay activity could be investigated. Note also that other mechanisms besides the successor representation could account for these results, including modelbased reinforcement learning.
Leveraging replays to learn novel trajectories
In the previous section, the replays reactivated the same trajectories as seen during behavior. In this section, we extend this idea and show how in our model replays can be useful during learning even when the reactivated trajectories were not directly experienced during behavior.
For this purpose, we reproduce an placeavoidance experiment from Wu et al., 2017. In short, rats are allowed to freely explore a linear track on day 1. Half of the track is dark, while the other half is bright. On day 2, the animals did four trials separated by resting periods: in the first trial (pre), the animals were free to explore the track; in the second trial (shock), they started in the light zone and received two mild footshocks when entering in the shock zone; in the third and fourth trial (post and reexposure, respectively), they were allowed to freely explore the track again, but starting from the light zone or the shock zone respectively (Figure 5a). In the study, it was reported that during the post trial, animals tended to stay in the light zone and forward replays from the current position to the shock zone were observed when the animals reached the boundary between the light and the dark zone (Figure 5b and c).
We simulated a simplified version of this task. Our simulated agent moves through the linear track following a softmax policy, and all states have equal value during the first phase (pre) (Figure 5d, blue trajectories). Then, the agent is allowed to move through the linear track until it reaches the shock zone and experiences a negative reward. Finally, the third phase is similar as the first phase and the animal is free to explore the track. Two versions of this third phase were simulated. In one version, there are no replays (Figure 5d, orange trajectories in left panel), while in the second version a forward replay until the shock zone is simulated every time the agent enters the middle state (Figure 5d, orange trajectories in right panel, replays not shown). The replays affect the learning of the successor representation and the negative reward information is propagated towards the decision point in the middle of the track. The states in the dark zone therefore have lower value compared to the case without replays (Figure 5e). In turn, this different value affects the policy of the agent which now tends to avoid the dark zone all together, while the agent without replays still occupies many states of the dark zone as much as states in the light zone (Figure 5f). Moreover, even when doubling the amount of SR updates in the scenario without replays, the behavior of the agent remains unaltered (Figure 5—figure supplement 1). This shows that it is not the amount of updates, but the type of policy that is important when updating the SR, and how using a different policy in the replay activity can significantly alter behavior.
Our setup for this simulation is simplified, and does not aim to reproduce the complex decision making of the rats. Observe for example the peak of occupancy of the middle state by the animals (Figure 5c), which is not captured by our model because we assume the agent to spend the same amount of time in each state. Nonetheless, it is interesting to see how replaying trajectories that were not directly experienced before, in combination with a model allowing replays to affect the learning of a cognitive map, can substantially influence the final policy of an agent and the overall performance. This mental imagination of trajectories could be exploited to refine our cognitive maps, avoiding unfavourable locations or finding shortcuts to rewards. It is important to note here that, while we are suggesting a potential role for the SR in solving this task, the data itself would also be compatible with a modelbased strategy. In fact, experimental evidence suggests that humans may use a mixed strategy involving both modelbased reinforcement learning and the successor representation (Momennejad et al., 2017).
Discussion
In this article, we investigated how a spiking neural network model of the hippocampus can learn the successor representation. Interestingly, we show that the updates in synaptic weights resulting from our biologically plausible STDP rule are equivalent to TD($\lambda $) updates, a wellknown and powerful reinforcement learning algorithm.
Reinforcement learning
Our network learns the SR in the CA3CA1 weights. Since we have modeled neurons to integrate the synaptic EPSPs and generate spikes using an inhomogeneous Poisson process based on the depolarization, the firing rate is proportional to the total synaptic weights. Therefore, the successor representation can be read out simply by a downstream neuron. Moreover, since the value of a state is defined by the inner product between the successor matrix and the reward vector, it is sufficient for the synaptic weights to the downstream neuron to learn the reward vector, and the downstream neuron will then encode the state value in its firing rate (see Figure 5—figure supplement 2). While the neuron model used is simple, it will be interesting for future work to study analogous models with nonlinear neurons.
It is worth noting that, during learning, both presynaptic and postsynaptic layers receive external inputs representing the current state (Equation 10 and Equation 11 in Materials and methods). This may induce a distortion in the read out of the diagonal elements of the SR matrix (see Equations 13 and 15, and Figure 5—figure supplement 2). At a first glance, this may indicate that learning and reading out are antagonistic. However, there are multiple ways we could resolve this apparent conflict: (i) Since the external current in CA1 is present for only a fraction of the time T in each state, the readout might happen during the period of CA3 activation exclusively; (ii) The readout may be over the whole time T but becomes more noisy towards the end. Even in the case where the readout is noisy, the distortion would be limited to the diagonal elements of the matrix; (iii) Learning and readout may be separate mechanisms, where the CA3 driving current is present during readout only. This could be for instance signaled by neuromodulation (e.g. noradrenaline and acetylcholine are active during learning but not exploration Micheau and Marighetto, 2011; Hasselmo and Sarter, 2011; Robbins, 1997; TelesGrilo Ruivo and Mellor, 2013; PalaciosFilardo et al., 2021), or it could be that readout happens during replays; (iv) The weights to or activation functions of the readout neuron may learn to compensate for the distorted signal in CA1.
Furthermore, we can notice that the external inputs encoding the current state activate CA3 first, and CA1 later. The delay between these activations $\theta /T$ (Equation 10 and Equation 11 in Materials and methods) is an arbitrary parameter that can be adjusted. Varying this delay will change the reinforcement learning representation, especially parameters $\lambda $ and $\gamma $, but also the strength of the input current (see Figure 5—figure supplement 3). However, this will not impact the distortion of the diagonal elements of the SR matrix, which remains similar across various delay values $\theta /T$ (see Figure 5—figure supplement 4).
Biological plausibility
Uncovering a connection between STDP and TD($\lambda $) shows how, using minimal assumptions, a theoretically grounded learning algorithm can emerge from a biological implementation of plasticity. Similar learning rules have indeed been observed in the hippocampus (Shouval et al., 2002 and proposed on theoretical grounds Mehta et al., 2000; Waddington et al., 2012; van Rossum et al., 2012).
The TD algorithm is most commonly known in neuroscience for describing how reward prediction can be computed in the brain. More specifically, it is widely believed that dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) encode the prediction error between the observed and expected reward (Schultz et al., 1997), dopamine thus acts as a global signal that can be broadcasted to other areas of the brain like the striatum to compute the expected reward. In our model, the TD algorithm estimates the SR (i.e. expected future occupancy), rather than the value. However, since the prediction error for the SR is different for every synaptic connection (i.e. each pair of states), it is not clear how it could be carried by a global signal analogous to dopamine. The SR would need multiple signals, or a matrix transformation of the global signal. Furthermore, we would need to postulate that such error – or errors – are computed elsewhere in the brain. Instead, in our model, the prediction error simply emerges from the synaptic plasticity rule itself. Furthermore, thanks to the presynaptic depression, our STDP rule alone allows us to compute negative prediction errors, which still poses an open challenge for computation with dopamine because of the low baseline dopaminergic firing rate (Glimcher, 2011; Daw et al., 2002; Matsumoto and Hikosaka, 2007).
Our framework smoothly connects a temporally precise spiking code with a fully ratebased code, and anything in between. As we have proven mathematically, this translates in moving smoothly from Monte Carlo to Temporal Difference by means of TD($\lambda $). Fast spiking sequences (temporal code) can be used for consolidation of previous experiences using Monte Carlo learning, while the behavioral timescale activity (rate code) results in TD updates, allowing learning on the timescale of seconds even with plasticity timeconstants on the order of milliseconds. This type of Hebbian learning over behavioral timescale exploits the bootstrapping property of TD, and is different than the oneshot behavioral plasticity described in Bittner et al., 2017. However, these two mechanisms could be complementary, where the latter could play a more significant role in the formation of new place fields, while the former would be more relevant to shape the existing place fields to contain predictive information. Learning on behavioral timescales using STDP was also investigated in Drew and Abbott, 2006. The main difference between Drew and Abbott, 2006 and our work, is that the former relies on overlapping neural activity between the pre and postsynaptic neurons from the start, while in our case no such overlap is required. In other words, our setup allows us to learn connections between a presynaptic neuron and a postsynaptic neuron whose activities are separated by behavioral timescales initially. For this to be possible, there are two requirements: (1) the task needs to be repeated many times and (2) a chain of neurons are consecutively activated between the aforementioned presynaptic and postsynaptic neuron. Due to this chain of neurons, over time the activity of the postsynaptic neuron will start earlier, eventually overlapping with the presynaptic neuron.
In our work, we did not include theta modulation, but phase precession and theta sequences could be yet another type of activity within the TD lambda framework. A recent work (George et al., 2023) incorporated the theta sweeps into behavioral activity, showing it approximately learns the SR. Moreover, theta sequences allow for fast learning, playing a similar role as replays (or any other fast temporalcode sequences) in our work. By simulating the temporally compressed and precise theta sequences, their model also reconciles the learning over behavioral timescales with STDP. In contrast, our framework reconciles both timescales relying purely on ratecoding during behavior. Finally, their method allows to learn the SR within continuous space. It would be interesting to investigate whether these methods coexist in the hippocampus and other brain areas. Furthermore, (Fang et al., 2023) et al. recently showed how the SR can be learned using recurrent neural networks with biologically plausible plasticity.
There are three different neural activities in our proposed framework: the presynaptic layer (CA3), the postsynaptic layer (CA1), and the external inputs. These external inputs could for example be locationdependent currents from the entorhinal cortex, with timings guided by the theta oscillations. The dependence of CA1 place fields on CA3 and entorhinal input is in line with lesion studies (see e.g. Brun et al., 2008; Hales et al., 2014; O’Reilly et al., 2014). It would be interesting for future studies to further dissect the role various areas play in learning cognitive maps.
Notably, even though we have focused on the hippocampus in our work, the SR does not require predictive information to come from higherlevel feedback inputs. This framework could therefore be useful even in sensory areas: certain stimuli are usually followed by other stimuli, essentially creating a sequence of states whose temporal structure can be encoded in the network using our framework. Interestingly, replays have been observed in other brain areas besides the hippocampus (KurthNelson et al., 2016; Staresina et al., 2013). Furthermore, temporal difference learning in itself has been proposed in the past as a way to implement prospective coding (Brea et al., 2016).
Replays
We have also proposed a role for replays in learning the SR, in line with experimental findings and RL theories (Russek et al., 2017; Momennejad et al., 2017). In general, replays are thought to serve different functions, spanning from consolidation to planning (Roscow et al., 2021). Here, we have shown that when the replayed trajectories are similar to the ones observed during behavior, they play the role of speeding up and consolidating learning by regulating the biasvariance tradeoff, which is especially useful in novel environments. On the other hand, if the replayed trajectories differ from the ones experienced during wakefulness, replays can play a role in reshaping the representation of space, which would suggest their involvement in planning. Experimentally, it has been observed that replays often start and end from relevant locations in the environment, like reward sites, decision points, obstacles or the current position of the animal (Ólafsdóttir et al., 2015; Pfeiffer and Foster, 2013; Jackson et al., 2006; Mattar and Daw, 2017). Since these are salient locations, it is in line with our proposition that replays can be used to maintain a convenient representation of the environment. It is worth noticing that replays can serve a variety of functions, and our framework merely proposes additional beneficial properties without claiming to explain all observed replays. For example, in addition to forward replays, also reverse replays are ubiquitous in hippocampus (Pfeiffer, 2020). The reverse replays are not included in our framework, and it is not clear yet whether they play different roles, with some evidence suggesting that reverse replays are more closely tied to the reward encoding (Ambrose et al., 2016). Moreover, while indirect evidence supports the idea that replays can play a role during learning (Igata et al., 2021), it is not yet clear how synaptic plasticity is manifested during replays (Fuchsberger and Paulsen, 2022).
Learning flexibility
Multiple ideas from reinforcement learning, such as TD($\lambda $), statedependent discounting and the successor representation, emerge quite naturally from our simple biologically plausible setting. We propose in our work that time and space can be discounted differently. Moreover, the flexibility to change the discounting factor by modulating firing rates and plasticity parameters — which is ubiquitous in neural circuits — suggests that these mechanisms could be used to encode a variety of information in a cognitive map. Moreover, the specific dependence of the discount factor on the biological parameters leads to experimentally testable predictions. Indeed, our framework predicts welldefined changes in place fields after modulations of firing rates, speed of the agent or neuromodulation of the plasticity parameters (Figure 3). Importantly, the discount parameter also depends on the time spent in each state. This eliminates the need for time discretization, which does not reflect the continuous nature of the response of time cells (Kraus et al., 2013).
Limitations of the reinforcement learning framework
We have already outlined some of the benefits of using reinforcement learning for modeling behavior, including providing clear computational and algorithmic frameworks. However, there are several intrinsic limitations to this framework. For example, RL agents that only use spatial data do not provide complete descriptions of behavior, which likely arises from integrating information across multiple sensory inputs. Whereas an animal would be able to smell and see a reward from a certain distance, an agent exploring the environment would only be able to discover it when randomly visiting the exact reward location. Furthermore, the framework rests on fairly strict mathematical assumptions: typically the state space needs to be markovian, time and space need to be discretized (which we manage to evade in this particular framework) and the discounting needs to follow an exponential decay. These assumptions are simplistic and it is not clear how often they are actually met. Reinforcement Learning is also a sampleintensive technique, whereas we know that some animals, including humans, are capable of much faster or even oneshot learning.
Even though we have provided a neural implementation of the SR, and of the value function as its readout (see Figure 5—figure supplement 2), the whole action selection process is still computed only at the algorithmic level. It may be interesting to extend the neural implementation to the policy selection mechanism in the future.
Taken together, our work joins — in a single framework — a variety of concepts from the neuronal level over cognitive theories to reinforcement learning.
Materials and methods
The successor representation
Request a detailed protocolIn a tabular environment, we define the value of a state $s$ as being the expected cumulative reward that an agent will receive following a certain policy starting in $s$. The future rewards are multiplied by a factor $\displaystyle 0<{\gamma}^{n}\le 1$, where $n$ is the number of steps until reaching the reward location and $\displaystyle 0<\gamma \le 1$ is the delay discount factor. It is usual to use $\displaystyle 0<\gamma <1$, which ensures that earlier rewards are given more importance compared to later rewards. Formally, the value of a state $s$ under a certain policy $\pi $ is defined as
Here, $a$ denotes the action, $R(s,a)$ is the reward function and $P\left({s}^{\prime}s,a\right)$ is the transition function, i.e. the probability that taking an action $a$ in state $s$ will result in a transition to state ${s}^{\prime}$. Following (Dayan, 1993), we can decompose the value function into the inner product of reward function and successor matrix
with
This representation is known as the successor representation (SR), where each element ${M}_{ij}$ represents the expected future occupancy of state $j$ when in state $i$. By decomposing the value into the SR and the reward function (Equation 3), relearning the state values $V$ after changing the reward function is fast, similar to modelbased learning. At the same time, the SR can be learned in a modelfree manner, using for example temporal difference (TD) learning (Russek et al., 2017).
Derivation of the TD($\lambda $) update for the SR
Request a detailed protocolThe TD($\lambda $) update for the SR is then implemented according to (see e.g. Sutton and Barto, 1998)
Using ${\delta}_{i}^{TD}$ for the TD error at step $i$ and ${\delta}_{xy}$ for the Kronecker delta,
corresponds to the TD error for element $M(j+n,i+n)$ of the successor representation after the transition from state $j+n$ to state $j+n+1$. Combining Equations 5 and 6, we find
and
Neural network model
Plasticity rule
Request a detailed protocolThe synaptic plasticity rule (Figure 1d) consists of a weightdependent depression for presynaptic spikes and a spiketiming dependent potentiation, given by
Here, ${w}_{ij}$ represents the synaptic connection from presynaptic neuron $j$ to postsynaptic neuron $i$, $T{r}_{LTP}^{j}$ is the plasticity trace, a lowpass filter of the presynaptic spike train with time constant ${\tau}_{\text{LTP}}$, ${t}^{j}$ and ${t}^{i}$ are the spike times of the postsynaptic and presynaptic neuron respectively, ${A}_{\text{LTP}}$ and ${A}_{\text{LTD}}$ are the amplitudes of potentiation and depression respectively, ${\eta}_{\text{STDP}}$ is the learning rate for STDP and the $\delta (\cdot )$ denotes the Dirac delta function.
Place cell activation
Request a detailed protocolWe assume that each state in the environment is represented by a population of place cells in the network. In our model, this is achieved by delivering placetuned currents to the neurons. Whenever a state $S=j$ is entered, the presynaptic neurons encoding state $j$ start firing at a constant rate ${\rho}^{pre}$ for a time $\theta $, following a Poisson process with parameter ${\rho}_{h}^{pre}\left(t\right)$. The other presynaptic neurons are assumed to be silent:
where the Kronecker delta function is defined as ${\delta}_{hj}=1$ if $h=j$ and zero otherwise. Here we use the index $j$ to denote any neuron belonging to the population of neurons encoding state $j$. After a short delay, at time ${t}^{*}$, a similar current ${\rho}^{bias}$ is delivered to the postsynaptic neuron encoding state $j$, for a duration of time $\omega $.
Besides the placetuned input current, CA1 neurons receive inputs from the presynaptic layer (CA3). The postsynaptic potential ${\rho}_{i}^{post}$ when the agent is in state j is thus given by
with the first sum running over all ${N}_{pop}$ presynaptic neurons encoding state $j$, and the second sum over all presynaptic firing times ${t}_{k}^{f}$ of neuron $k$ happened before $t$. The excitatory postsynaptic current $\kappa $ is modeled as an exponential decay described as $\kappa \left(x\right)={\u03f5}_{0}{e}^{x/{\tau}_{m}}$ for $x\ge 0$ and zero otherwise. Each CA1 neuron $i$ fires following an inhomogeneous Poisson process with rate ${\rho}_{i}^{post}\left(t\right)$.
Note that, in most simulations we will use a single neuron in the population ${N}_{pop}=1$. In addition, we normally set ${t}^{*}=\theta $ and $\omega =T\theta $. However, we will keep these as explicit parameters for theoretical purposes.
Equivalence with TD($\lambda $)
Request a detailed protocolTotal plasticity update
Request a detailed protocolSince we have the mathematical equation for the plasticity rule, and CA3 and CA1 neurons follow an inhomogeneous Poisson process with timedependent firing rate, we can calculate analytically the average total weight change for the synapse ${w}_{ij}$, given a certain trajectory (details in the Appendix). Please notice that our calculation is based on Kempter et al., 1999, which takes into account the fact that our plasticity rule is sensitive to spike timing and involves a spikespike correlation term. We find that:
where $N$ is the number of states until the end of the trajectory and
Comparison with TD($\lambda $)
Request a detailed protocolComparing the total weight change due to STDP (Equation 13) to the TD($\lambda $) update (Equation 8), we can see that the two equations are very similar in form:
We impose ${w}_{ij}=M(j,i)$, and find:
where $A,B,{B}^{\prime}$ and $C$ are defined as in Equations 14, 15, and 16.
Hence, our plasticity rule is learning the Successor Representation through a TD($\lambda $) model with parameters:
To ensure the learning rate $\eta $ is positive, one condition resulting from Equation 21 is
Learning during normal behavior ($\displaystyle \theta >>{\tau}_{LTP}$)
Request a detailed protocolDuring normal behavior, we assume the placetuned currents are on larger timescales than the plasticity constants: $\displaystyle \theta ,\omega >>{\tau}_{LTP}$. We can see from Equations 14 and 16 that the factor $A$ grows linearly with $\theta $ while $C$ grows exponentially with $\theta $. From Equation 23, we then have
(See also Figure 2—figure supplement 1).
Learning during replays ($\displaystyle \theta <<{\tau}_{LTP}$)
Assumptions
Request a detailed protocolFor the replay model we assume the placetuned currents are impulses, which make the neurons emit exactly one spike at a given time. Specifically, we can make the duration of the placetuned currents go to 0,
while the intensity of the currents goes to infinity. For simplicity, we will take:
Furthermore, we assume that the contribution of the postsynaptic currents due to the single presynaptic spikes is negligible in terms of driving plasticity, allowing us to set
Calculations of TD parameters
Request a detailed protocolGiven the assumptions above, we can see from Equations 14 and 16 that:
For Equation 15, we can use the Taylor expansion for ${e}^{\frac{x}{\tau}}$ around $x=0$, such that: ${e}^{\frac{x}{\tau}}\approx 1+{\displaystyle \frac{x}{\tau}}$ :
Using Equations 21, 22, 23 and 18, we can calculate the parameters and constraints for the TD model:
As expected, the bootstrapping parameter $\lambda =1$ (see also Figure 2—figure supplement 1).
Alternative derivation of replay model
Place cell activation during replays
Request a detailed protocolWe model a replay event as a precise temporal sequence of spikes. Since every neuron represents a state in the environment, a replay sequence reproduces a trajectory of states. We assume that, when the agent is in state $S=j$, the neurons representing state $j$ fire ${n}_{pre}$ spikes at some point in the time interval $t\in [0,\sigma ]$, where the exact firing times are uniformly sampled. After a short delay, the CA1 neurons representing state $j$ fire ${n}_{post}$ spikes at a time uniformly sampled from the interval $[{t}^{*},{t}^{*}+\sigma ]$. The time between two consecutive state visits is $T$. The exact number of spikes in each replay event is random but small. Specifically, it is sampled from the set $\{0,1,2\}$ according to the probability vector
It is worth noting here that other implementations are possible but that we assume the average number of spikes in each state is 1, and that the average time between a presynaptic and a postsynaptic spike is ${t}^{*}$. The model could be further generalized for a higher number of average spikes per state.
Plasticity update
Request a detailed protocolWe can consider again our learning rule, composed of a positive prepost potentiation window and presynaptic weightdependent depression (Equation 9). Let’s consider the synapse ${w}_{ij}$, we can see that on average the total amount of depression will be determined by the number of times the state $j$ is visited in the trajectory replayed:
where ${N}_{j}$ is the number of times the state $j$ is visited. The amount of potentiation will be determined, instead, by the time difference between the postsynaptic and presynaptic firing times, which encode the distance between state $j$ and state $i$:
where ${n}_{k}^{ij}$ represents the number of times the agent visited state $i$ k steps after $j$. Combining the equations above we find that:
If we assume that the this value has converged to its stationary state, $\mathrm{\Delta}{w}_{ij}=0$;
Comparison with online Monte Carlo learning
Request a detailed protocolGiven the stable weight ${w}^{*}$ from Equation 30, we can impose that:
we find that the stable weight is:
which is the definition of the Successor Representation matrix (Equation 4). Indeed, ${w}_{ij}^{\star}$ is computing the sample mean of the discounted distance between states $i$ and $j$, which is equivalent to performing an everystate Monte Carlo or TD($\lambda $=1) update. Notably, from Equation 29, we have that the learning rate for the Monte Carlo update is given by:
Simulation details for Figure 2
Request a detailed protocolA linear track with four states is simulated. The policy of the agent in this simulation is to traverse the track from left to right, with one epoch consisting of starting in state 1 and ending in state 4. One simulation consists of 50 epochs, and we rerun the whole simulation ten times with different random seeds. Over these ten seeds, mean and standard deviation of the synaptic weights are recorded after every epoch.
Our neural network consists of two layers, each with a single neuron per state (as in Figure 1). Synaptic connections are made from each presynaptic neuron to all postsynaptic neurons, resulting in a 4by4 matrix which is initialized as the identity matrix. The plasticity rule and neuronal activations follow Equations 9–12.
The STDP parameters are listed in Table 1.
To obey Equation 24, we set ${A}_{\text{pre}}$ equal to the right hand side augmented with 5.
For the behavioral case, we choose T=100ms, $\theta $=80ms, $\omega =T\theta $, which correspond to TD($\lambda $) parameters $\lambda =0.21$, $\gamma =0.89$, $\eta =0.12$.
In the replay case, we have a sequence of single spike per neuron (see Figure 2b and section ‘Alternative derivation of replay model’). Following Equation 27, we choose $T=\mathrm{log}\left(\gamma \right){\tau}_{LTP}\approx $ 7ms, where $\gamma $ and ${\tau}_{LTP}$ are the same as in Table 1. We set $\theta =2$ ms and $\sigma =0.5$ ms. By setting the ${\eta}_{stdp}={\displaystyle \frac{\eta}{{A}_{LTP}\mathrm{exp}\left(\theta /{\tau}_{LTP}\right)}}$, the corresponding TD($\lambda $) parameters are $\lambda =1$, $\gamma =0.89$, $\eta =0.12$ just as in the behavioral case.
More details on the place cell activation during replays in our model can be found in section ‘Alternative derivation of replay model’. Using exactly one single spike per neuron with the above parameters would allow us to follow the TD(1) learning trajectories without any noise. For more biological realism, we choose ${p}_{1}=0.15$ in Equation 28, in order to achieve an equal amount of noise due to the random spiking as in the case of behavioral activity (see Figure 4—figure supplement 2).
Simulation details for Figure 3
Request a detailed protocolUsing the same neural network and plasticity parameters as the behavioral learning in Figure 2 (see previous section), we simulate the linear track in the following two situations:
The third state has T=200ms instead of 100ms. All other parameters remain the same as in Figure 2. Results plotted in Figure 3E.
The third state has ${\rho}_{\text{pre}}=0.2{\text{ms}}^{1}$ instead of $0.1{\text{ms}}^{1}$. All other parameters remain the same as in Figure 2. Results plotted in Figure 3F.
Simulation details for Figure 4
Request a detailed protocolA linear track with three states is simulated, and the agent has 50% probability to move left or right in each state (see Figure 4A). One epoch lasts until the agent reaches one of the STOP locations.
We then use the same neural network and plasticity parameters as used for Figure 2. We simulate three scenarios:
Only replaybased learning during all epochs (no behavioral learning). This scenario corresponds to MC STDP in Figure 4B and to Figure 4C.
Mixed learning using both behavior and replays. The probability for an epoch to be a replay is decaying over time following $\mathrm{exp}\left(i/6\right)$, with $i$ the epoch number. This scenario corresponds to Mix STDP in Figure 4B and to Figure 4E.
Only behavioral learning during all epochs (no replays). This scenario corresponds to TD STDP in Figure 4B and to Figure 4D.
Simulation details for Figure 5
Request a detailed protocolA linear track with 21 states is simulated. The SR is initialized as the identity matrix, and the reward vector (containing the reward at each state) is also initialized as the zero vector. We simulate the learning of the SR during behavior using the theoretical TD(0) updates and during replays using the theoretical TD(1) updates. The value of each state is then calculated as the matrixvector product between the SR and the reward vector, resulting in an initial value of zero for each state.
The policy of the agent is a softmax policy (i.e. the probability to move to neighboring states is equal to the softmax of the values of those neighboring states). The first time the agent reaches the leftmost state of the track (state 1), the negative reward of –2 is revealed, mimicking the shock in the actual experiments, and the reward vector is updated accordingly for this state.
We now simulate two scenarios: in the first scenario, the agent always follows the softmax policy and no replays are triggered (see Figure 5D, left panel). In the second scenario, every time the agent enters the dark zone from the light zone (i.e. transitions from state 12 to state 11 in our simulation), a replay is triggered from that state until the leftmost state (state 1) (see Figure 5D, right panel). Both scenarios are simulated for 2000 state transitions. We then run these two scenarios 100 times and calculate mean and standard deviation of state occupancies (Figure 5F).
Finally, since the second scenario has more SR updates than the first scenario, we also simulate the first scenario for 4000 state transitions (Figure 5—figure supplement 1) and show how the observed behavior of Figure 5 is unaffected by this.
Appendix 1
Analytical derivations for the total weight change in the behavioural model
Presynaptic rate during state $j$
Whenever a state $S=j$ is entered, the presynaptic neurons encoding state $j$ start firing at a constant rate ${\rho}^{pre}$ for a time $\theta $, following a Poisson process with parameter ${\rho}_{j}^{pre}\left(t\right)$:
The other presynaptic neurons are silent.
Postsynaptic rate during state $j$
The average postsynaptic rate can be calculated as follows. The probability of a presynaptic spike between $t$ and $t+dt$ is equal to ${\rho}_{j}\left(t\right)dt$. The size of the presynaptic population encoding state $j$ is equal to ${N}_{\text{pop}}$ and each excitatory postsynaptic potential (EPSP) is modeled by an immediate jump with amplitude ${\u03f5}_{0}{w}_{ij}$, followed by exponential decay of EPSP with time constant ${\tau}_{m}$.
Following Equation 12 in the main paper, reproduced below,
we find that the average postsynaptic potential at time $t$ is given by (assuming t=0 when entering the state $j$):
We assume that ${w}_{ij}\left(t\right)$ changes slowly compared to the timescale $\theta $ allowing us to consider the weight constant during that time. We can then approximate the average postsynaptic rate as:
If $\displaystyle {t}^{\star}<\theta$, both the first and the second term will contribute to the postsynaptic rate in the time between ${t}^{\star}$ and $\theta $.
LTP trace during state $j$
Given Equation 9 in the main paper, reproduced below,
and combined with Equation 35, we can calculate the evolution of the LTP trace for neuron $j$ during state $j$:
For $\displaystyle 0\le t<\theta$, the presynaptic neuron $j$ is active and therefore the trace builds up with the presynaptic spikes, for $t\ge \theta $, the trace decays exponentially with time constant ${\tau}_{LTP}$.
Total amount of LTP during state $j$
Following (Kempter et al., 1999), first we calculate the amount of LTP without taking into account spiketospike correlation:
The probability for a postsynaptic spike between $t$ and $t+dt$ is ${\overline{\rho}}_{i}^{post}\left(t\right)dt$. The amount of LTP due to a single spike at time $t$ is ${A}_{LTP}T{r}_{LTP}^{j}\left(t\right)$. Hence, combining Equations 37 and 38, the total amount of LTP during a state (i.e. between time 0 and $T$) becomes:
Following (Kempter et al., 1999), the amount of LTP due to the causal part (each presynaptic spike temporarily increase the probability of a postsynaptic spike) is given by:
Combining equations for the noncausal 40 and causal 41 parts, we get the total amount of LTP during a state (assuming $\displaystyle {\tau}_{m}<<{\tau}_{LTP})$:
Total amount of LTD during state $j$
There is a weightdependent depression for each presynaptic spike, hence the amount of LTD during a state is given by:
Total plasticity during state $j$
Combining Equations 42 and 43, we can calculate the total amount of plasticity during the time the agent spends in the current state $j$:
with
and
Plasticity due to states transitioning
Once the agent leaves state $j$, the decaying LTP trace can still cause potentiation due to the activity in the following states, $j+n$, with $n=1,2,\mathrm{\dots}$ . Given that the agent spends a time $T$ in each state, we find that the agent visits state $j+n$ during time $t\in [nT,nT+T)$. We will now calculate the contribution to plasticity due to these state transitions.
Postsynaptic rate during the new state $j+n$
During state $j+n$, the activity of the postsynaptic neurons is driven by the presynaptic neurons coding for $j+n$, and the bias current. We can thus generalize Equation 37 and find that the average postsynaptic rate ${\overline{\rho}}_{i}^{post}$ during state $j+n$ is:
LTP trace from state $j$, during the new state $j+n$
Following Equation 38, we find that the amplitude of the LTP trace from state $j$ during state $j+n$ is:
with $0<{t}^{\prime}<T$.
LTP due to state transitioning
We can then calculate the amount of LTP between the presynaptic neuron $j$ and the postsynaptic neuron $i$, when the agent is in state $j+n$. We refer to Equation 39 and find:
The amount of plasticity in state $j+n$ when starting from state $j$ is thus:
where
It is worth noting that the parameter B derived here is the same as Equation 46.
Summary: total STDP update
If we combine together Equations 44 and 49, we have that the total weight change for the synapse ${w}_{ij}$ is given by:
where $N$ is the number of states until the end of the trajectory and A, B, C are as defined in Equations 45, 46 and 50 respectively.
Analytical calculations for hyperbolic discounting
From Equation 22 in the main paper, we have that, in the behavioural model $\gamma =\left(1{\displaystyle \frac{C}{A}}\right){e}^{\frac{T}{{\tau}_{LTP}}}$. Here, we will derive an approximation to this value.
If we assume that $\displaystyle \theta >>{\tau}_{m},{\tau}_{LTP}$, we can approximate $A$ and $C$ as:
If we define $\psi $ such that $\theta +\psi =T$, we can rewrite and approximate the discount parameter as:
From Equation 56, we can see that the discount $\gamma $ follows a hyperbolic function if we increase the duration of the presynaptic current $\theta $. If, instead, we vary $\psi $, the discount becomes exponential (Figure 2—figure supplement 1a and b).
Notice that this analysis extends to the replay model. Following what was done after Equation 26, we can connect the behavioural model with the replay model by making $\theta ,{\u03f5}_{0}\to 0$, which implies $\psi \to T$. From Equation 56 we find that:
which is exactly the definition of $\gamma $ in the replay model (Equations 27 in Materials and methods). For replays, the discount is therefore strictly exponential.
Furthermore, using the same calculations and Equations 21 and 19 in the main paper, we can find approximated values for the other parameters too (Figure 2—figure supplement 1c and d).
Data availability
The current manuscript is a computational study, so no data have been generated for this manuscript. Modelling code is available on GitHub (https://github.com/jacopobono/learning_cognitive_maps_code, copy archived at swh:1:rev:d86b262545547353c7050bbc2d476c2f4a297989; Jacopo, 2023).
References

Episodic memory, amnesia, and the hippocampalanterior thalamic axisThe Behavioral and Brain Sciences 22:425–444.

Pure hyperbolic discount curves predict “ eyes open ” selfcontrolTheory and Decision 73:3–34.https://doi.org/10.1007/s1123801192725

Prospective coding by spiking neuronsPLOS Computational Biology 12:e1005003.https://doi.org/10.1371/journal.pcbi.1005003

Opponent interactions between serotonin and dopamineNeural Networks 15:603–616.https://doi.org/10.1016/s08936080(02)000527

BookTemporal difference learning in continuous time and spaceIn: Touretzky D, Mozer M, Hasselmo M, editors. Advances in Neural Information Processing Systems. Massachusetts, United States: MIT Press. pp. 1–10.

Reinforcement learning in continuous time and spaceNeural Computation 12:219–245.https://doi.org/10.1162/089976600300015961

A goaldirected spatial navigation model using forward trajectory planning based on grid cellsThe European Journal of Neuroscience 35:916–931.https://doi.org/10.1111/j.14609568.2012.08015.x

Modulation of hippocampal plasticity in learning and memoryCurrent Opinion in Neurobiology 75:102558.https://doi.org/10.1016/j.conb.2022.102558

Rethinking dopamine as generalized prediction errorProceedings of the Royal Society B 285:20181645.https://doi.org/10.1098/rspb.2018.1645

The successor representation: its computational logic and neural substratesThe Journal of Neuroscience 38:7193–7200.https://doi.org/10.1523/JNEUROSCI.015118.2018

Modes and models of forebrain cholinergic neuromodulation of cognitionNeuropsychopharmacology 36:52–73.https://doi.org/10.1038/npp.2010.104

Hippocampal sharp waves and reactivation during awake states depend on repeated sequential experienceThe Journal of Neuroscience 26:12415–12426.https://doi.org/10.1523/JNEUROSCI.411806.2006

SoftwareLearning_cognitive_maps_code, version swh:1:rev:d86b262545547353c7050bbc2d476c2f4a297989Software Heritage.

Neural ensembles in CA3 transiently encode paths forward of the animal at a decision pointThe Journal of Neuroscience 27:12176–12189.https://doi.org/10.1523/JNEUROSCI.376107.2007

Hebbian learning and spiking neuronsPhysical Review E 59:4498–4514.https://doi.org/10.1103/PhysRevE.59.4498

Linear lookahead in conjunctive cells: an entorhinal mechanism for vectorbased navigationFrontiers in Neural Circuits 6:20.https://doi.org/10.3389/fncir.2012.00020

Golden eggs and hyperbolic discountingThe Quarterly Journal of Economics 112:443–478.https://doi.org/10.1162/003355397555253

BookA Rational Model of Prioritized Experience ReplayAnn Arbor, USA: The University of Michigan.

Hippocampal synaptic enhancement and information storage within a distributed memory systemTrends in Neurosciences 10:408–415.https://doi.org/10.1016/01662236(87)900117

Acetylcholine and memory: a long, complex and chaotic but still living relationshipBehavioural Brain Research 221:424–429.https://doi.org/10.1016/j.bbr.2010.11.052

The successor representation in human reinforcement learningNature Human Behaviour 1:680–692.https://doi.org/10.1038/s4156201701808

Learning structures: predictive representations, replay, and generalizationCurrent Opinion in Behavioral Sciences 32:155–166.https://doi.org/10.1016/j.cobeha.2020.02.017

Spatial localization does not require the presence of local cuesLearning and Motivation 12:239–260.https://doi.org/10.1016/00239690(81)900205

BookThe Hippocampus as a Cognitive MapOxford, United Kingdom: Oxford University Press.https://doi.org/10.1016/j.neuron.2015.06.013

Spatial memory and hippocampal functionNeuropsychologia 17:669–682.https://doi.org/10.1016/00283932(79)900423

Relative contributions of CA3 and medial entorhinal cortex to memory in ratsFrontiers in Behavioral Neuroscience 8:292.https://doi.org/10.3389/fnbeh.2014.00292

Arousal systems and attentional processesBiological Psychology 45:57–71.https://doi.org/10.1016/s03010511(96)052222

Learning offline: memory replay in biological and artificial reinforcement learningTrends in Neurosciences 44:808–821.https://doi.org/10.1016/j.tins.2021.07.007

Predictive representations can link modelbased reinforcement learning to modelfree mechanismsPLOS Computational Biology 13:e1005768.https://doi.org/10.1371/journal.pcbi.1005768

The hippocampus as a predictive MAPNature Neuroscience 20:1643–1653.https://doi.org/10.1038/nn.4650

Reinforcement learning: an introductionIEEE Transactions on Neural Networks 9:1054–1056.https://doi.org/10.1109/TNN.1998.712192

Cholinergic modulation of hippocampal network functionFrontiers in Synaptic Neuroscience 5:2.https://doi.org/10.3389/fnsyn.2013.00002

Softbound synaptic plasticity increases storage capacityPLOS Computational Biology 8:e1002836.https://doi.org/10.1371/journal.pcbi.1002836

Triphasic spiketimingdependent plasticity organizes networks to produce robust sequences of neural activityFrontiers in Computational Neuroscience 6:88.https://doi.org/10.3389/fncom.2012.00088

Hippocampal awake replay in fear memory retrievalNature Neuroscience 20:571–580.https://doi.org/10.1038/nn.4507
Decision letter

Lisa M GiocomoReviewing Editor; Stanford School of Medicine, United States

Laura L ColginSenior Editor; University of Texas at Austin, United States

Michael E HasselmoReviewer; Boston University, United States
Our editorial process produces two outputs: (i) public reviews designed to be posted alongside the preprint for the benefit of readers; (ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.
Decision letter after peer review:
Thank you for submitting your article "Learning predictive cognitive maps with spiking neurons during behaviour and replays" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Laura Colgin as the Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Michael E. Hasselmo (Reviewer #2).
The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.
Essential revisions:
Based on the comments from all reviewers, I'd recommend focusing your revisions on the primary topic of improving the link between their model and experimental data. Specifically, this would include: Consideration (or modeling) of how the model would extend to 2D, discussion on how the neural activity would be used to perform computations, the limitations of RL as it relates to interpreting experimental data, and to more appropriately frame the work in the context of experimental studies (all reviewers had detailed suggestions for how to do this with text changes). I've provided highlights from the three reviewers below that apply to these concerns but note that reviewer 2 also provided a number of specific suggestions related to text/reference changes. All reviewer comments are also included at the bottom of this message.
Reviewer 1:
– The successor representation is learned at the level of synaptic weights between the two layers. It is not clear how it is read out into neural activity and exploited to perform actual computations, as both layers are assumed to be strongly driven by external inputs. This is a major limitation of this work.
– One of the results is that STDP at the timescale of milliseconds can lead to learning over behavioral timescales of seconds. This result seems related to Drew and Abbott PNAS 2006. In that work, the mapping between learning on micro and macro timescales in fact relied on precise tuning of plasticity parameters. It is not clear to which extent similar limitations apply here, and what is the precise relation with Drew and Abbott.
– Most of the results are presented at a formal, descriptive level relating plasticity to reinforcement learning algorithms. The provided examples are quite limited and focus on a simplified setting, a linear track. It would be important to see that the results extend to twodimensional environments, and to show how the successor representation is actually used (see first comment).
– The main text does not explain clearly how replays are implemented.
Reviewer 2:
I think the authors of this article need to be clear about the shortcomings of RL. They should devote some space in the discussion to noting neuroscience data that has not been addressed yet. They could note that most components of their RL framework are still implemented as algorithms rather than neural models. They could note that most RL models usually don't have neurons of any kind in them and that their own model only uses neurons to represent state and successor representations, without representing actions or action selection processes. They could note that the agents in most RL models commonly learn about barriers by needing to bang into the barrier in every location, rather than learning to look at it from a distance. The ultimate goal of research such as this should be to link cellular level neurophysiological data to experimental data on behavior. To the extent possible, they should focus on how they link neurophysiological data at the cellular level to spatial behavior and the unit responses of place cells in behaving animals, rather than basing the validity of their work on the assumption that the successor representation is correct.
Reviewer 3:
1. Could the authors elaborate more on the connection between the biological replays that are observed in a different context in the brain and the replays implemented in their model? Within the modeling context, when are replays induced upon learning in a novel environment, and what is the influence of replays when/if they are generated upon revisiting the previously seen/navigated environment?
2. The model is composed of CA1 and CA3, what are the roles of the other hippocampal subregions in learning predictive maps? From the reported results, it looks like it may be possible that predictionbased learning can be successfully achieved simply via the CA1CA3 circuit. Are there studies (e.g., lesioned) that show this causal relationship to behavior? Along this line, what are the potential limitations of the proposed framework in understanding the circuit computation adopted by the hippocampus?
3. Do the authors believe that the plasticity rules/computational principles observed within the 2layer model are specific to the CA1CA3 circuit? Can these rules be potentially employed elsewhere within the medial temporal lobe or sensory areas? What are the model parameters used that could suggest that the observed results are specific to hippocampusbased predictive learning?
4. The analytical illustration linking the proposed model with reinforcement learning is well executed. However, in practice, the actual implementation of reinforcement learning within the model is unclear. Given the sample task provided where animals are navigating a simple environment, how can one make use of valuebased learning to enhance behavior? Explicit discussion on the extent to which reinforcement learning is related to the actual computation potentially needed to navigate sensory environments (both learned and novel) would be really helpful in understanding the link between the model to reinforcement learning.
5. Subplots both within and across figures seem to be of very different text formatting and sizing (such as panel F in Figure 4 and Figure 5). Please reformat them accordingly.
Reviewer #2 (Recommendations for the authors):
Important: Note that the page numbers refer to the page in the PDF, which is their own page number1 (due to eLife adding a header page).
Page 3 – "smoothly… and anything in between" – this is overstated and should be removed.
Page 3 – "don't need to discretize time…". Here and elsewhere there should be citations to the work of Doya, NeurIPS 1995, Neural Comp 2000 on the modeling of continuous time in RL.
Page 3 – "using replays" – It is very narrowminded to assume that all replay do is set up successor representations. They could also be involved in modelbased planning of behavior as suggested in the work of Johnson and Redish, 2012; Pfeiffer and Foster, 2018; Kay et al. 2021 and modeled in Hasselmo and Eichenbaum, 2005; Erdem and Hasselmo, 2012 and Fenton and Kubie, 2012.
Page 3 – They assume that STDP can occur during replay, but evidence for STDP during replay is unclear. McNaughton's lab showed that LTP is less likely to be induced during the modulatory states during which sharpwave ripple replay events occur. They should look for citations that have actually shown LTP induction during the replay state.
Page 4 – Marr's three levels – They should remove this discussion about Marr's three levels as I think the implementation level is relatively sparse and the behavioral level is also relatively sparse.
Page 4 – "The hippocampus has long been thought" – It's astounding that the introduction only cites two experimental papers (O'Keefe and Dostrovsky, 1971; Mehta et al. 2000) and then the start of the Results section makes a statement like this and only cites Stachenfeld et al. 2017 as if it were an experimental paper. There are numerous original research papers that should be cited for the role of hippocampus in behavior. They should cite at least five or six so that the reader doesn't get the impression all o this work started with the holy paper by Stachenfeld et al. 2017. For example, they could again cite O'Keefe and Nadel, 1978 for the very comprehensive review of the literature up to that time, plus the seminal work of Morris et al. 1982 in Morris water maze and Olton, 1979 in 8arm radial maze and perhaps some of the work by Aggleton and by Eichenbaum on spatial alternation.
Page 5 – The description of successor representations is very one dimensional. They should mention how it can be expanded to two dimensions.
Page 5 – "Usually attributed to model based…". They cannot just talk about SR being model free. Since this section was supposed to be for neuroscientists, they need to clearly explain the distinction between model free and modelbased RL, and describe why successor representations are not just modelbased RL, but instead provide a lookup table of predictive state that does NOT involve modelbased planning of behavior. The blog of Vitay gives a much better overview that compares modelfree, modelbased and successor representations:
https://julienvitay.net/post/successor_representations/ – This needs more than just a citation – there should be a clear description of modelbased and modelfree RL in contrast to SR, and Vitay is an example of that.
Page 5 – Related to this issue – they need to repeatedly address the fact that Successor representations are just an hypothesis contrast with modelbased behavior, and repeatedly throughout the paper discuss that modelbased behavior could still be the correct accounting for all of the data that they address.
Page 5 – "similar to (Mehta et al. 2000)" – Learning in the CA3CA1 network has been modeled like this in many previous models that should be cited here including McNaughton and Morris, 1987; Hasselmo and Schnell, 1994; Treves and Rolls, 1994; Mehta et al. 2000; Hasselmo, Bodelon and Wyble, 2002.
Page 6 – Figure 1d looks like the net outcome of the learning rule in this example is longterm depression. Is that intended? Given the time interval between pre and post, it looks like it ought to be potentiation in the example.
Page 7 – They should address the problem of previously existing weights in the CA3 to CA1 connections. For example, what if there are preexisting weights that are strong enough to cause postsynaptic spiking in CA1 independent of any entorhinal input? How do they avoid the strengthening of such connections? (i.e. the problem of prior weights driving undesired learning on CA3CA1 synapses is addressed in Hasselmo and Schnell, 1994; Hasselmo, Bodelon and Wyble, 2002, which should be cited).
Page 7 – "Elegantly combines rate and temporal" – This is overstated. The possible temporal codes in the hippocampus include many possible representations beyond just one step prediction. They need to specify that this combines one type of possible temporal code. I also recommend removing the term "elegant" from the paper. Let someone else call your work elegant.
Page 7 – "replays for learning" – as noted above in experiments LTP has not been shown to be induced during the time periods of replay – sharpwave ripple replay events seem to be associated with lower cholinergic tone (Buzsaki et al. 1983; VandeCasteele et al. 2014) whereas LTP is stronger when Ach levels are higher (Patil et al. J. Neurophysiol. 1998). This is not an allornone difference, but it should be addressed.
Page 7 – "equivalent to TD…". Should this say "equivalent to TD(0)"?
Page 8 – "Bootstrapping means that a function is updated using current estimates of the same function…". This is a confusing and vague description of bootstrapping. They should try to give a clearer definition for neuroscientists (or reduce their reference to this).
Page 9 – Figure 2 – Do TD λ and TD zero really give equivalent weight outputs?
Page 8 – "that are behaviorally far apart" – I don't understand how this occurs.
page 10 – "dependency of synaptic weights on each other as discussed above." This was not made sufficiently clear either here or above.
Page 10 – "dependency of synaptic weights on each other" This also suggests a problem of stability if the weights can start to drive their own learning and cause instability – how is this prevented?
Page 10 – "average of the discounted state occupancies" – this would be uniform without discounting but what is the biological mechanism for the discounting that is used here?
Page 10 – "due to the bootstrapping" – again this is unclear – can be improved by giving a better definition of bootstrapping and possibly by referring to specific equation numbers.
Page 11 – "exponential dependence" what is the neural mechanism for this?
Page 11 – "Ainsley" is not a real citation in the bibliography. Should fix and also provide a clearer definition (or equation) for hyperbolic.
Page 11 – "elegantly combines two types of discounting" – how is useful? Also, let other people call your work elegant.
Page 11 – how does discounting depend on both firing rate and STDP  should provide some explanation or at least refer to where this is shown in the equations.
page 13 – "Cheng and Frank" – this is a good citation, but they could add more here on timing of replay events.
Page 15 – This whole section on the shock experiment starts with the assumption of a successor representation. As noted above, they need to explicitly discuss the important alternate hypothesis that the neural activity reflects modelbased planning that guides the behavior in the task (and could perhaps better account for the peak of occupancy at the border of light and dark).
Page 16 – "mental imagination" – rather than using it for modifying SR, why couldn't mental imagination just be used for modelbased behavior?
Page 17 – "spiking" – again, if they are going to refer to their model as a "spiking" model, they need to add some plots showing spiking activity.
Reviewer #3 (Recommendations for the authors):
I found the proposed modeling framework to be very exciting and of potential interest to not only computational neuroscientists but also to readers who are interested in neural mechanisms underlying learning in general. The manuscript is wellwritten and includes a detailed description and rationale of the model setups as well as the findings and their relevance to biological findings. That said, I have a few comments that I hope the authors could help address:
1. Could the authors elaborate more on the connection between the biological replays that are observed in a different context in the brain and the replays implemented in their model? Within the modeling context, when are replays induced upon learning in a novel environment, and what is the influence of replays when/if they are generated upon revisiting the previously seen/navigated environment?
2. The model is composed of CA1 and CA3, what are the roles of the other hippocampal subregions in learning predictive maps? From the reported results, it looks like it may be possible that predictionbased learning can be successfully achieved simply via the CA1CA3 circuit. Are there studies (e.g., lesioned) that show this causal relationship to behavior? Along this line, what are the potential limitations of the proposed framework in understanding the circuit computation adopted by the hippocampus?
3. Do the authors believe that the plasticity rules/computational principles observed within the 2layer model are specific to the CA1CA3 circuit? Can these rules be potentially employed elsewhere within the medial temporal lobe or sensory areas? What are the model parameters used that could suggest that the observed results are specific to hippocampusbased predictive learning?
4. The analytical illustration linking the proposed model with reinforcement learning is well executed. However, in practice, the actual implementation of reinforcement learning within the model is unclear. Given the sample task provided where animals are navigating a simple environment, how can one make use of valuebased learning to enhance behavior? Explicit discussion on the extent to which reinforcement learning is related to the actual computation potentially needed to navigate sensory environments (both learned and novel) would be really helpful in understanding the link between the model to reinforcement learning.
5. Subplots both within and across figures seem to be of very different text formatting and sizing (such as panel F in Figure 4 and Figure 5). Please reformat them accordingly.
[Editors’ note: further revisions were suggested prior to acceptance, as described below.]
Thank you for resubmitting your work entitled "Learning predictive cognitive maps with spiking neurons during behaviour and replays" for further consideration by eLife. Your revised article has been evaluated by Laura Colgin (Senior Editor) and a Reviewing Editor.
The manuscript has been improved but there are some remaining issues that need to be addressed, as outlined below:
Reviewer 1 makes two text suggestions that I believe would clarify the findings. There remains some lack of clarification around (1) how the second layer of the model mixes the successor representation with a representation of the current state itself and (2) justification for the difference in duration of the external inputs to the two layers. Reviewer 1 also suggests an additional figure, but I leave the decision to add (or not) this to the authors. Details regarding the requested clarifications are below.
Reviewer #1 (Recommendations for the authors):
The revised article has only partly resolved my confusion.
My main issue was the following: in the proposed feedforward model, the synaptic weights between the two layers learn the entries of the successor matrix. If external inputs were fed only to the first layer, the second layer would directly read out the successor representation (this is suggested in Figure 1 EF, but not explicitly mentioned in the text as far as I can tell). Instead, in the model, both layers are driven by external inputs representing the current state. This is crucial for learning, but it implies that the activity of the second layer mixes the successor representation with a representation of the current state itself. Learning and readout, therefore, seem antagonistic. It would be worth explaining this fact in the main text.
In their reply, the authors clarify that the external inputs drive the activity of the second layer only for a limited time (20%). As far as I can tell, in the text, this is mentioned explicitly only in the legend of Figure 5S2. That seems to imply that there is a large difference in the duration of the external inputs to the two layers. How can that be justified?
More importantly, it seems that varying the value of the delay should lead to a tradeoff between the accuracy of learning and the accuracy of the subsequent readout in the second layer. Is that the case? It would be useful to have a figure where the delay is varied.
https://doi.org/10.7554/eLife.80671.sa1Author response
Reviewer 1:
– The successor representation is learned at the level of synaptic weights between the two layers. It is not clear how it is read out into neural activity and exploited to perform actual computations, as both layers are assumed to be strongly driven by external inputs. This is a major limitation of this work.
We thank the reviewer for this important remark. Since we modelled our neurons to integrate the synaptic EPSPs and generated spikes using an inhomogeneous Poisson process based on the depolarization, the firing rate is proportional to the total synaptic weights. Therefore, the successor representation can be read out simply by a downstream neuron. Moreover, since the value of a state is defined by the inner product between the successor matrix and the reward vector, it is sufficient for the synaptic weights to the downstream neuron to learn the reward vector, and the downstream neuron will then encode the state value in its firing rate. We performed additional simulations and have added a supplementary figure (Figure 5—figure supplement S2) showing this setup.
We also want to note that the external inputs driving the activity affect the firing rate of the value neuron only during a limited amount of time (20% of the time in each state in our simulation of Figure 5—figure supplement S2). Moreover, this effect changes the estimate of the value quantitatively, but not qualitatively, i.e. the ranking of the states by value is not affected.
Practically, using the parameters in our simulation, one could either read out the correct estimate of the value during the first 80% of the time in a state, learn a correction to this perturbation in the weights to the value neuron, or simply use a policy that is based on the ranking of the values instead of the actual firing rate.
Besides the supplementary figure (Figure 5—figure supplement S2), we added the following paragraph in the Discussion section:
“Since we modelled our neurons to integrate the synaptic EPSPs and generate spikes using an inhomogeneous Poisson process based on the depolarization, the firing rate is proportional to the total synaptic weights. Therefore, the successor representation can be read out simply by a downstream neuron. Moreover, since the value of a state is defined by the inner product between the successor matrix and the reward vector, it is sufficient for the synaptic weights to the downstream neuron to learn the reward vector, and the downstream neuron will then encode the state value in its firing rate (see Figure 5—figure supplement S2). While the neuron model used is simple, it will be interesting future work to study analogous models with nonlinear neurons.”
– One of the results is that STDP at the timescale of milliseconds can lead to learning over behavioral timescales of seconds. This result seems related to Drew and Abbott PNAS 2006. In that work, the mapping between learning on micro and macro timescales in fact relied on precise tuning of plasticity parameters. It is not clear to which extent similar limitations apply here, and what is the precise relation with Drew and Abbott.
We thank the reviewer for pointing us to the interesting work by Drew and Abbott. We added the following paragraph in the Discussion section:
“Learning on behavioural timescales using STDP was also investigated in Drew 2006. The main difference between Drew 2006 and our work, is that the former relies on overlapping neural activity between the pre and postsynaptic neurons from the start, while in our case no such overlap is required. In other words, our setup allows us to learn connections between a presynaptic neuron and a postsynaptic neuron whose activities are separated by behavioural timescales initially. For this to be possible, there are two requirements: (1) the task needs to be repeated many times and (2) a chain of neurons are consecutively activated between the aforementioned presynaptic and postsynaptic neuron. Due to this chain of neurons, over time the activity of the postsynaptic neuron will start earlier, eventually overlapping with the presynaptic neuron.”
– Most of the results are presented at a formal, descriptive level relating plasticity to reinforcement learning algorithms. The provided examples are quite limited and focus on a simplified setting, a linear track. It would be important to see that the results extend to twodimensional environments, and to show how the successor representation is actually used (see first comment).
We thank the reviewer for the feedback, and have now included two new supplementary figures. Figure 5—figure supplement S2 shows the usefulness of our setup when learning the state values, as discussed above. Figure 1—figure supplement S1shows a simulation in a 2D environment. In fact, due to the exact link with TD(λ), our setup is general for any type of task, in any dimension, where states are visited and which may not need to be a navigation task.
We have added the following sentences in the second to last paragraph of section 2.2:
Moreover, due to the equivalence with TD(λ), our setup is general for any type of task where discrete states are visited, in any dimension, and which may not need to be a navigation task (see e.g. Figure 2d for a 2D environment).
– The main text does not explain clearly how replays are implemented.
We thank the reviewer for pointing out this issue. We have now updated the methods section 4.3.6 to be more clear about the implementation of the replays. We have also updated the description of replay generation in the methods of Figure 2 as follows:
“More details on the place cell activation during replays in our model can be found in section 4.3.6. Using exactly one single spike per neuron with the above parameters would allow us to follow the TD(1) learning trajectories without any noise. For more biological realism, we choose p1=0.15 in equation 28, in order to achieve an equal amount of noise due to the random spiking as in the case of behavioural activity (see Supplementary Figure S2).”
Reviewer 2:
I think the authors of this article need to be clear about the shortcomings of RL. They should devote some space in the discussion to noting neuroscience data that has not been addressed yet. They could note that most components of their RL framework are still implemented as algorithms rather than neural models. They could note that most RL models usually don't have neurons of any kind in them and that their own model only uses neurons to represent state and successor representations, without representing actions or action selection processes. They could note that the agents in most RL models commonly learn about barriers by needing to bang into the barrier in every location, rather than learning to look at it from a distance. The ultimate goal of research such as this should be to link cellular level neurophysiological data to experimental data on behavior. To the extent possible, they should focus on how they link neurophysiological data at the cellular level to spatial behavior and the unit responses of place cells in behaving animals, rather than basing the validity of their work on the assumption that the successor representation is correct.
We thank the reviewer for the important feedback. We have addressed the reviewer 2's concerns in the "Public Evaluation" section, which includes their detailed comments. In short, here, we made substantial changes to refer more thoroughly to the experimental literature, discuss the limitations of RL in general and the limitations of our proposed framework, discuss the link between neuronal data and behaviour, and finally, we made sure not to overstate the validity of the successor representation.
Reviewer 3:
1. Could the authors elaborate more on the connection between the biological replays that are observed in a different context in the brain and the replays implemented in their model? Within the modeling context, when are replays induced upon learning in a novel environment, and what is the influence of replays when/if they are generated upon revisiting the previously seen/navigated environment?
We thank the reviewer for the interesting questions. Within our modelling context, we speculate that replays may contribute to the learning of the SR. To understand why this may be beneficial, we show that in our framework, learning is faster in a novel environment when the proportion of replays is larger. Replays, because they rely on the MC algorithm, are great for learning quickly as they are fast to override obsolete weights. Specifically for Figure 4, we implemented a probability for replays that decays exponentially with the number of epochs in a novel environment, but other schemes that introduce more replays in novel environments than in familiar ones should lead to similar conclusions. We also show that, when replays are generated in familiar environments, they still contribute to learning the same SR but introduce more variance. We also argue that replays can be used to imagine novel trajectories (similar to ideas in modelbased planning) and thus update the SR without actually walking the trajectory (Figure 5). In summary, we believe that replays can functionally serve a variety of purposes, and our framework merely proposes additional beneficial properties without claiming to explain all observed replays. For example, our framework does not encompass reverse replays. We have updated the ‘Replays’ paragraph of the Discussion section to reflect this.
“We have also proposed a role for replays in learning the SR, in line with experimental findings and RL theories Russek 2017 Momennejad 2017. In general, replays are thought to serve different functions, spanning from consolidation to planning Roscow 2021. Here, we have shown that when the replayed trajectories are similar to the ones observed during behaviour, they play the role of speeding up and consolidating learning by regulating the biasvariance tradeoff, which is especially useful in novel environments. On the other hand, if the replayed trajectories differ from the ones experienced during wakefulness, replays can play a role in reshaping the representation of space, which would suggest their involvement in planning. Experimentally it has been observed that replays often start and end from relevant locations in the environment, like reward sites, decision points, obstacles or the current position of the animal FreyjaOlafsdottir 2015, Pfeiffer 2013, Jackson 2006,Mattar 2017. Since these are salient locations, it is in line with our proposition that replays can be used to maintain a convenient representation of the environment. It is worth noticing that replays can serve a variety of functions, and our framework merely proposes additional beneficial properties without claiming to explain all observed replays. For example, next to forward replays, also reverse replays are ubiquitous in hippocampus Pfeiffer 2020. The reverse replays are not
included in our framework, and it is not clear yet whether they play different roles, with some evidence suggesting that reverse replays are more closely tied to the reward encoding Ambrose 2016.”
2. The model is composed of CA1 and CA3, what are the roles of the other hippocampal subregions in learning predictive maps? From the reported results, it looks like it may be possible that predictionbased learning can be successfully achieved simply via the CA1CA3 circuit. Are there studies (e.g., lesioned) that show this causal relationship to behavior? Along this line, what are the potential limitations of the proposed framework in understanding the circuit computation adopted by the hippocampus?
We thank the reviewer for the feedback. We have added the following paragraph in the discussion to address this point:
“There are three different neural activities in our proposed framework: the presynaptic layer (CA3), the postsynaptic layer (CA1), and the external inputs. These external inputs could for example be locationdependent currents from the entorhinal cortex, with timings guided by the theta oscillations. The dependence of CA1 place fields on CA3 and entorhinal input is in line with lesion studies (see e.g. Brun2008, Hales2014, Oreilly2014). It would be interesting for future studies to further dissect the role various areas play in learning cognitive maps.”
3. Do the authors believe that the plasticity rules/computational principles observed within the 2layer model are specific to the CA1CA3 circuit? Can these rules be potentially employed elsewhere within the medial temporal lobe or sensory areas? What are the model parameters used that could suggest that the observed results are specific to hippocampusbased predictive learning?
We thank the reviewer for this important question. We refer to the following paragraph in the discussion regarding this topic:
“Notably, even though we have focused on the hippocampus in our work, the SR does not require predictive information to come from higherlevel feedback inputs. This framework could therefore be useful even in sensory areas: certain stimuli are usually followed by other stimuli, essentially creating a sequence of states whose temporal structure can be encoded in the network using our framework. Interestingly, replays have been observed in other brain areas besides the hippocampus KurthNelson2016, Staresina2013. Furthermore, temporal difference learning in itself has been proposed in the past as a way to implement prospective coding Brea2016”
4. The analytical illustration linking the proposed model with reinforcement learning is well executed. However, in practice, the actual implementation of reinforcement learning within the model is unclear. Given the sample task provided where animals are navigating a simple environment, how can one make use of valuebased learning to enhance behavior? Explicit discussion on the extent to which reinforcement learning is related to the actual computation potentially needed to navigate sensory environments (both learned and novel) would be really helpful in understanding the link between the model to reinforcement learning.
We thank the reviewer for this important remark. We have added Figure 5—figure supplement S2 and updated the discussion to address this point. Since we modelled our neurons to integrate the synaptic EPSPs and generated spikes using an inhomogeneous Poisson process based on the depolarization, the firing rate is proportional to the total synaptic weights. Therefore, the successor representation can be read out simply by a downstream neuron. Moreover, since the value of a state is defined by the inner product between the successor matrix and the reward vector, it is sufficient for the synaptic weights to the downstream neuron to learn the reward vector, and the downstream neuron will then encode the state value in its firing rate. We performed additional simulations and have added a supplementary figure (Figure 5—figure supplement S2) showing this setup.
Besides the supplementary figure (Figure 5—figure supplement S2), we added the following paragraph in the Discussion section:
“Since we modelled our neurons to integrate the synaptic EPSPs and generate spikes using an inhomogeneous Poisson process based on the depolarization, the firing rate is proportional to the total synaptic weights. Therefore, the successor representation can be read out simply by a downstream neuron. Moreover, since the value of a state is defined by the inner product between the successor matrix and the reward vector, it is sufficient for the synaptic weights to the downstream neuron to learn the reward vector, and the downstream neuron will then encode the state value in its firing rate (see Figure 5—figure supplement S2). While the neuron model used is simple, it will be interesting for future work to study analogous models with nonlinear neurons.”
5. Subplots both within and across figures seem to be of very different text formatting and sizing (such as panel F in Figure 4 and Figure 5). Please reformat them accordingly.
Thank you, we have reformatted the figures.
Reviewer #2 (Recommendations for the authors):
Important: Note that the page numbers refer to the page in the PDF, which is their own page number1 (due to eLife adding a header page).
Page 3 – "smoothly… and anything in between" – this is overstated and should be removed.
We thank the reviewer for the feedback. We have adapted the sentence as follows:
“We show mathematically that our proposed framework smoothly connects a temporally precise spiking code akin to replay activity with a rate based code akin to behavioural spiking.”
While we agree that our model implements only one type of temporal code, it is important to stress that there is a smooth transition between this temporal code and a pure rate encoding of the state. The larger the time T in each state, the less precise the spikes become organised in time. In all cases, the learning dynamics have the same fixed point, namely, they converge to the successor representation. We believe that this fact is nontrivial. Furthermore, we show how this smooth transition changes how the fixed point (SR) is reached. On one extreme (T=0), algorithmically it uses TD(1), on the other extreme (T=infinity) TD(0) and all intermediate cases implement a value of λ between 0 and 1.
Page 3 – "don't need to discretize time…". Here and elsewhere there should be citations to the work of Doya, NeurIPS 1995, Neural Comp 2000 on the modeling of continuous time in RL.
Thank you, we have added those citations
Page 3 – "using replays" – It is very narrowminded to assume that all replay do is set up successor representations. They could also be involved in modelbased planning of behavior as suggested in the work of Johnson and Redish, 2012; Pfeiffer and Foster, 2018; Kay et al. 2021 and modeled in Hasselmo and Eichenbaum, 2005; Erdem and Hasselmo, 2012 and Fenton and Kubie, 2012.
We thank the reviewer for the feedback, but assure that we did not mean to assume that is all replays do. In fact, we are well aware of other potential benefits of replays in memory consolidation, modelbased planning etc. We merely propose to add another potential benefit of replays to the existing hypotheses, namely that they can be used to learn and in doing so they can reduce bias and learn offline. We have adapted the ‘Replays’ paragraph in the discussion to reflect this and have also adapted the highlighted text to make this clearer:
“Finally, replays have long been speculated to be involved in learning models of the environment (Eichenbaum, 2005; Hasselmo and Erdem and Hasselmo, 2012 and Fenton and Kubie, 2012; Johnson and Redish, 2012; Pfeiffer and Foster, 2018; Kay et al. 2021;). Here, we investigate how replays could play an additional role in learning the SR cognitive map”
Page 3 – They assume that STDP can occur during replay, but evidence for STDP during replay is unclear. McNaughton's lab showed that LTP is less likely to be induced during the modulatory states during which sharpwave ripple replay events occur. They should look for citations that have actually shown LTP induction during the replay state.
Thank you for this comment. We have added the following sentence in the "Replay" paragraph of the discussion.
“Moreover, while indirect evidence supports the idea that replays can play a role during learning Igata2020, it is not yet clear how synaptic plasticity is manifested during replays Fuchsberger2022.”
Page 4 – Marr's three levels – They should remove this discussion about Marr's three levels as I think the implementation level is relatively sparse and the behavioral level is also relatively sparse.
We believe that this link between computation, algorithm and implementation is actually a strength of the proposed work. Typically, only one or at most two levels are discussed, without a more holistic view. We believe that our work (and e.g. George et al.) goes one step beyond by making the link between the implementation (spiking neurons with STDP), algorithm (TD learning) and computational theory (SR / predictive cognitive maps) explicit. While of course, these modelling studies are abstractions of reality, we believe this is not trivial and would like to maintain this paragraph to stimulate future research to bridge these levels as well.
Page 4 – "The hippocampus has long been thought" – It's astounding that the introduction only cites two experimental papers (O'Keefe and Dostrovsky, 1971; Mehta et al. 2000) and then the start of the Results section makes a statement like this and only cites Stachenfeld et al. 2017 as if it were an experimental paper. There are numerous original research papers that should be cited for the role of hippocampus in behavior. They should cite at least five or six so that the reader doesn't get the impression all o this work started with the holy paper by Stachenfeld et al. 2017. For example, they could again cite O'Keefe and Nadel, 1978 for the very comprehensive review of the literature up to that time, plus the seminal work of Morris et al. 1982 in Morris water maze and Olton, 1979 in 8arm radial maze and perhaps some of the work by Aggleton and by Eichenbaum on spatial alternation.
We agree and thank the reviewer for the suggestion. We have now expanded the citations of relevant experimental work.
Page 5 – The description of successor representations is very one dimensional. They should mention how it can be expanded to two dimensions.
To address this, we perform a new simulation in a 2D environment (Supplementary Figure 2) as well as a discussion on the generality of the approach to any dimension in section 2.1
“Even though we introduced the linear track as an illustrative example, the SR can be learned in any environment (see Figure 1—figure supplement S1 for an example in an open field)”.
Page 5 – "Usually attributed to model based…". They cannot just talk about SR being model free. Since this section was supposed to be for neuroscientists, they need to clearly explain the distinction between model free and modelbased RL, and describe why successor representations are not just modelbased RL, but instead provide a lookup table of predictive state that does NOT involve modelbased planning of behavior. The blog of Vitay gives a much better overview that compares modelfree, modelbased and successor representations:
https://julienvitay.net/post/successor_representations/ – This needs more than just a citation – there should be a clear description of modelbased and modelfree RL in contrast to SR, and Vitay is an example of that.
We have now extended this paragraph with a more thorough explanation of modelfree and modelbased RL, including an example of what happens when a reward location changes in all three cases.
Because of this predictive information, the SR allows sampleefficient relearning when the reward location is changed Gershman2018. In reinforcement learning, we tend to distinguish between modelfree and modelbased algorithms. The SR is believed to sit inbetween these two modalities. In modelfree reinforcement learning, the aim is to directly learn the value of each state in the environment. Since there is no model of the environment at all, if the location of a reward is changed, the agent will have to first unlearn the previous reward location by visiting it enough times, and then is able to relearn the new location. In modelbased reinforcement learning, a precise model of the environment is learned, specifically, singlestep transition probabilities between all states of the environment. Modelbased learning is computationally expensive, but allows a certain flexibility. If the reward changes location it is immediate to derive the updated values of the states. As we have seen, however, the SR can relearn a new reward location somewhat efficiently, although less so than modelbased learning. The SR can also be efficiently learned using modelfree methods and allows us to easily compute values for each state, which in turn can guide the policy Dayan1993, Russek2017, Momennejad2017. This position between modelbased and modelfree methods makes the SR framework very powerful, and its similarities with hippocampal neuronal dynamics have led to increased attention from the neuroscience community. Finally, in our examples above, we considered an environment made up of a discrete number of states. This framework can be generalised to a continuous environment represented by a discrete number of place cells.
Page 5 – Related to this issue – they need to repeatedly address the fact that Successor representations are just an hypothesis contrast with modelbased behavior, and repeatedly throughout the paper discuss that modelbased behavior could still be the correct accounting for all of the data that they address.
We thank the reviewer for pointing out that the successor representation is only one of the possible hypotheses.
We have proceeded to address this for the Frank and Cheng data (page 14, ‘Please note, however, that other mechanisms besides the Successor Representation could account for these results, including modelbased reinforcement learning.), and in the final section of the shock experiment (page 16, ‘It is important to note here that, while we are suggesting a potential role for the SR in solving this task, the data itself would also be compatible with a modelbased strategy. In fact, experimental evidence suggests that humans may use a mixed strategy involving both modelbased reinforcement learning and the successor representation [Momennejad et al., 2017].’).
Page 5 – "similar to (Mehta et al. 2000)" – Learning in the CA3CA1 network has been modeled like this in many previous models that should be cited here including McNaughton and Morris, 1987; Hasselmo and Schnell, 1994; Treves and Rolls, 1994; Mehta et al. 2000; Hasselmo, Bodelon and Wyble, 2002.
Thank you, we have added the citations
Page 6 – Figure 1d looks like the net outcome of the learning rule in this example is longterm depression. Is that intended? Given the time interval between pre and post, it looks like it ought to be potentiation in the example.
This depends on the prepost spike timing, and in this example the postsynaptic spike is too far from the presynaptic spike, leading to depression. When moving the postsynaptic spike closer to the presynaptic spike, potentiation would occur. The plasticity rule qualitatively results in three regions depending on the spike timing: depression (postpre), potentiation (prepost with small interval), and depression (prepost with large interval). Qualitatively it is in line with e.g. Shouval et al. 2002
Page 7 – They should address the problem of previously existing weights in the CA3 to CA1 connections. For example, what if there are preexisting weights that are strong enough to cause postsynaptic spiking in CA1 independent of any entorhinal input? How do they avoid the strengthening of such connections? (i.e. the problem of prior weights driving undesired learning on CA3CA1 synapses is addressed in Hasselmo and Schnell, 1994; Hasselmo, Bodelon and Wyble, 2002, which should be cited).
This is an important point, and since our model guarantees a stable fixed point of the learning dynamics, the initial conditions are not influencing the final convergence. To show this explicitly, we have performed a new simulation and added Figure 1—figure supplement S2 to illustrate this.
This is reflected in the manuscript:
“As a proof of principle, we show that it is possible to learn the SR for any initial weights (Figure 1—figure supplement S2), independently of any previous learning in the CA3 to CA1 connections.”
Page 7 – "Elegantly combines rate and temporal" – This is overstated. The possible temporal codes in the hippocampus include many possible representations beyond just one step prediction. They need to specify that this combines one type of possible temporal code. I also recommend removing the term "elegant" from the paper. Let someone else call your work elegant.
We thank the reviewer for the feedback. We have modified the manuscript to reflect the comments:
“As we will discuss below, this framework, therefore, combines learning based on rate coding as well as temporal coding.”
Page 7 – "replays for learning" – as noted above in experiments LTP has not been shown to be induced during the time periods of replay – sharpwave ripple replay events seem to be associated with lower cholinergic tone (Buzsaki et al. 1983; VandeCasteele et al. 2014) whereas LTP is stronger when Ach levels are higher (Patil et al. J. Neurophysiol. 1998). This is not an allornone difference, but it should be addressed.
Thank you for this comment. We have added the following sentence in the "Replay" paragraph of the discussion.
“Moreover, while indirect evidence supports the idea that replays can play a role during learning Igata2020, it is not yet clear how synaptic plasticity is manifested during replays Fuchsberger2022.”
Page 7 – "equivalent to TD…". Should this say "equivalent to TD(0)"?
We changed the phrasing in the manuscript to: “equivalent to TD(λ)”.
Page 8 – "Bootstrapping means that a function is updated using current estimates of the same function…". This is a confusing and vague description of bootstrapping. They should try to give a clearer definition for neuroscientists (or reduce their reference to this).
We have now changed the explanation in the manuscript in the corresponding section.
From a reinforcement learning perspective, the TD(0) algorithm relies on a property called bootstrapping. This means that the successor representation is learned by first taking an initial estimate of the SR matrix (i.e. the previously learned weights), and then gradually adjusting this estimate (i.e. the synaptic weights) by comparing it to the states in the environment the animal actually visits. This comparison is achieved by calculating a prediction error, similar to the widely studied one for dopamine neurons Schultz1997. Since the synaptic connections carry information about the expected trajectories, in this case, the prediction error is computed between the predicted and observed trajectories (see Methods).
The main point of bootstrapping, therefore, is that learning happens by adjusting our current predictions (e.g. synaptic weights) to match the observed current state. This information is available at each time step and thus it allows learning over long timescales using synaptic plasticity alone. If the animal moves to a state in the environment that the current weights deem unlikely, potentiation will prevail and the weight from the previous to the current state will increase. Otherwise, the opposite will happen. It is important to notice that the prediction error in our model is not encoded by a separate mechanism in the way that dopamine is thought to do for reward predictionSchultz1997. Instead, the prediction error is represented locally, at the level of the synapse, through the depression and potentiation terms of our STDP rule, and the current weight encodes the current estimate of the SR (see Methods). Notably, the prediction error updates result in a total update equivalent to the TD(\λ) update. This mathematical equivalence ensures that the weights of our neural network track the TD(\λ) update at each state, and thus stability and convergence to the theoretical values of the SR. We therefore do not need an external vector to carry prediction error signals as proposed in Gardner2018, Gershman2018. In fact, the synaptic potentiation in our model updates a row of the SR, while the synaptic depression updates a column.
Page 9 – Figure 2 – Do TD λ and TD zero really give equivalent weight outputs?
The theoretical value of the SR is the same, independently of the algorithm used to learn it. TD λ, for any value of λ, converges to this theoretical value. No tuning is needed for this, it is mathematically guaranteed that the algorithm converges. Thus, also for λ=0 the results are exact.
Page 8 – "that are behaviorally far apart" – I don't understand how this occurs.
See explanation above.
page 10 – "dependency of synaptic weights on each other as discussed above." This was not made sufficiently clear either here or above.
See explanation above.
Page 10 – "dependency of synaptic weights on each other" This also suggests a problem of stability if the weights can start to drive their own learning and cause instability – how is this prevented?
See explanation above. The dynamics have a fixed point of convergence (the SR). No matter what the initial weights are, it is guaranteed to converge to the SR. We illustrated that in the new Supplementary Figure 7.
Page 10 – "average of the discounted state occupancies" – this would be uniform without discounting but what is the biological mechanism for the discounting that is used here?
This is an important point. No extra mechanism is needed to induce the discount in our framework. Instead, the decaying LTP window is sufficient to drive the discounting of states further away.
Page 10 – "due to the bootstrapping" – again this is unclear – can be improved by giving a better definition of bootstrapping and possibly by referring to specific equation numbers.
We have now changed the explanation in the manuscript in the corresponding section, see comment above.
Page 11 – "exponential dependence" what is the neural mechanism for this?
In our case, it is the LTP window which exponentially decays.
Page 11 – "Ainsley" is not a real citation in the bibliography. Should fix and also provide a clearer definition (or equation) for hyperbolic.
Page 11 – "elegantly combines two types of discounting" – how is useful? Also, let other people call your work elegant.
Very true! We have now rephrased our statement. Hyperbolic discounting has been observed in neuroeconomics experiments, whereas reinforcement learning algorithms normally rely on exponential discounting for its favourable mathematical properties. This has been however considered confusing when applying reinforcement learning to behaviour. Our model uses exponential discounting at a neural level and in space, but can account for the findings with hyperbolic discounting.
Page 11 – how does discounting depend on both firing rate and STDP  should provide some explanation or at least refer to where this is shown in the equations.
We have now added a reference to the equation.
page 13 – "Cheng and Frank" – this is a good citation, but they could add more here on timing of replay events.
We added the following sentence regarding the replay timing:
“Please note that, while we implemented an exponentially decaying probability for replays after entering a novel environment, different schemes for replay activity could be investigated.”
Page 15 – This whole section on the shock experiment starts with the assumption of a successor representation. As noted above, they need to explicitly discuss the important alternate hypothesis that the neural activity reflects modelbased planning that guides the behavior in the task (and could perhaps better account for the peak of occupancy at the border of light and dark).
We have changed the text to reflect this suggestion.
Page 16 – "mental imagination" – rather than using it for modifying SR, why couldn't mental imagination just be used for modelbased behavior?
We believe it is indeed possible that mental imagination can be used for modelbased behaviour. However, it is possible that humans use a mixed strategy involving both modelbased learning and the successor representation, as suggested by the findings in Momennejad et al. 2017.
Page 17 – "spiking" – again, if they are going to refer to their model as a "spiking" model, they need to add some plots showing spiking activity.
One example of the spiking activity of our model can be found in Figure 2 panel A.
Reviewer #3 (Recommendations for the authors):
We have responded to the reviewer 3's concerns at the beginning of the rebuttal, in the "Essential Revision" section.
[Editors' note: further revisions were suggested prior to acceptance, as described below.]
Reviewer #1 (Recommendations for the authors):
The revised article has only partly resolved my confusion.
My main issue was the following: in the proposed feedforward model, the synaptic weights between the two layers learn the entries of the successor matrix. If external inputs were fed only to the first layer, the second layer would directly read out the successor representation (this is suggested in Figure 1 EF, but not explicitly mentioned in the text as far as I can tell). Instead, in the model, both layers are driven by external inputs representing the current state. This is crucial for learning, but it implies that the activity of the second layer mixes the successor representation with a representation of the current state itself. Learning and readout, therefore, seem antagonistic. It would be worth explaining this fact in the main text.
We thank the reviewer for the question, and we now added an extra explanation in the main text. In brief, the CA1 layer indeed mixes the SR with a representation of the current state. However, we argue that learning and readout are not necessarily antagonistic. We suggest here a few possibilities to resolve this conflict:
1. Since the external current in CA1 is present for only a fraction of the time T in each state, the readout might happen during the period of CA3 activation exclusively.
2. The readout may be over the whole time T but becomes noisier towards the end. It is worth noting that, even in the case where the readout is noisy, the distortion would be limited to the diagonal elements of the matrix (see equation 13 and 15 in the Methods section, and Figure 5 – supplement 2, panels B and C).
3. Learning and readout may be separate mechanisms, where during readout only the CA3 driving current is present. This could be for instance signaled by neuromodulation (e.g. acetylcholine has been associated to attention and arousal during spatial learning [1,2,3,4,5]), or it could be that readout happens during replays with a population of neurons.
4. The weights to or activation functions of the readout neuron may learn to compensate for the distorted signal in CA1.
We have now included this explanation in the main text in the Discussion, in the Reinforcement Learning subchapter.
In their reply, the authors clarify that the external inputs drive the activity of the second layer only for a limited time (20%). As far as I can tell, in the text, this is mentioned explicitly only in the legend of Figure 5S2. That seems to imply that there is a large difference in the duration of the external inputs to the two layers. How can that be justified?
More importantly, it seems that varying the value of the delay should lead to a tradeoff between the accuracy of learning and the accuracy of the subsequent readout in the second layer. Is that the case? It would be useful to have a figure where the delay is varied.
As suggested, we added two figures (Figure 5—figure supplement 3, Figure 5—figure supplement 4) where we varied this delay. In our model, the external inputs are first active in CA3 and afterwards in CA1 (equation 10 and 11). We model these inputs as step currents. We call the overall time in each state T, and say that the external current for CA3 has duration θ, while the external current for CA1 has duration ω omega. If we assume that there is no pause in between these two stimuli, we have ω = T – θ. We can therefore define the delay between the two stimuli as: θ/T.
As the reviewer points out, the delay value θ/T that we chose for Figure 5 – Supplement 2 was 0.8. However, this was an arbitrary choice and other values are possible. We explore here what happens when we vary the delay parameter θ/T:
– Impact on the reinforcement learning representation
The delay parameter θ/T influences the reinforcement learning representation, as it can be seen from equations 22 and 23. When the delay is longer, the value for γ increases and λ decreases (Figure 5 —figure supplement 3, panels bc). However, it is important to notice that γ and λ are determined by other biological values as well (see equations 22 and 23), such as the depression amplitude (Figure 5 —figure supplement 3, panels ef) or the decay of the potentiation learning window. The value we choose for the delay is therefore flexible: even if we want to learn the SR with certain fixed γ or λ, we can always vary other parameters in the spiking model.
– Impact on learning and readout
The strength of the placetuned input to CA1 also depends on the choice of the delay (see equation 18). Typically, the CA1 placetuned input strength increases with the delay, except for long values of T or low depression amplitudes, where the input remains fairly constant (Figure 5 —figure supplement 3, panels a and d respectively). Therefore, the longer the CA1 placetuned input lasts, the weaker it has to be to ensure the proper learning. Intuitively, this means that we can compensate for a shorter duration of the external current by increasing its strength. The distortion of the readout thus remains more or less constant independently of our choice of delay (Figure 5 —figure supplement 4). We therefore do not find a tradeoff between accuracy of learning and accuracy of readout, and the distortion in the value could be corrected for example by one of the mechanisms proposed above.
We addressed the above points in the Discussion, in the Reinforcement Learning subchapter.
References:
Micheau, J. and Marighetto, A. Acetylcholine and memory: a long, complex and chaotic but still living relationship. Behav. Brain Res. 221, 424–429 (2011)
Hasselmo, M. E. and Sarter, M. Modes and models of forebrain cholinergic neuromodulation of cognition. Neuropsychopharmacology 36, 52–73 (2011).
Robbins, T. W. Arousal systems and attentional processes. Biol. Psychol. 45, 57–71 (1997).
TelesGrilo Ruivo, L. M. and Mellor, J. R. Cholinergic modulation of hippocampal network function. Front. Synaptic Neurosci. 5, 2 (2013).
PalaciosFilardo, Jon, et al. "Acetylcholine prioritises direct synaptic inputs from entorhinal cortex to CA1 by differential modulation of feedforward inhibitory circuits." Nature communications 12.1 (2021): 116.
https://doi.org/10.7554/eLife.80671.sa2Article and author information
Author details
Funding
Wellcome Trust (200790/Z/16/Z)
 Claudia Clopath
Engineering and Physical Sciences Research Council (EP/R035806/1)
 Claudia Clopath
Simons Foundation (564408)
 Claudia Clopath
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication. For the purpose of Open Access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.
Senior Editor
 Laura L Colgin, University of Texas at Austin, United States
Reviewing Editor
 Lisa M Giocomo, Stanford School of Medicine, United States
Reviewer
 Michael E Hasselmo, Boston University, United States
Version history
 Preprint posted: August 17, 2021 (view preprint)
 Received: May 30, 2022
 Accepted: January 12, 2023
 Version of Record published: March 16, 2023 (version 1)
Copyright
© 2023, Bono, Zannone et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics

 945
 Page views

 119
 Downloads

 3
 Citations
Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading

 Neuroscience
The predictive nature of the hippocampus is thought to be useful for memoryguided cognitive behaviors. Inspired by the reinforcement learning literature, this notion has been formalized as a predictive map called the successor representation (SR). The SR captures a number of observations about hippocampal activity. However, the algorithm does not provide a neural mechanism for how such representations arise. Here, we show the dynamics of a recurrent neural network naturally calculate the SR when the synaptic weights match the transition probability matrix. Interestingly, the predictive horizon can be flexibly modulated simply by changing the network gain. We derive simple, biologically plausible learning rules to learn the SR in a recurrent network. We test our model with realistic inputs and match hippocampal data recorded during random foraging. Taken together, our results suggest that the SR is more accessible in neural circuits than previously thought and can support a broad range of cognitive functions.

 Computational and Systems Biology
 Ecology
High proportions of gut bacteria that produce their own food can be an indicator for poor gut health.