Neural learning rules for generating flexible predictions and computing the successor representation
Abstract
The predictive nature of the hippocampus is thought to be useful for memoryguided cognitive behaviors. Inspired by the reinforcement learning literature, this notion has been formalized as a predictive map called the successor representation (SR). The SR captures a number of observations about hippocampal activity. However, the algorithm does not provide a neural mechanism for how such representations arise. Here, we show the dynamics of a recurrent neural network naturally calculate the SR when the synaptic weights match the transition probability matrix. Interestingly, the predictive horizon can be flexibly modulated simply by changing the network gain. We derive simple, biologically plausible learning rules to learn the SR in a recurrent network. We test our model with realistic inputs and match hippocampal data recorded during random foraging. Taken together, our results suggest that the SR is more accessible in neural circuits than previously thought and can support a broad range of cognitive functions.
Editor's evaluation
This important work provides compelling evidence for the biological plausibility of the Successor Representation (SR) algorithm. The SR is a leading computational hypothesis to explore whether neural representations are consistent with the hypothesis that the neural networks in specific brain areas perform predictive computations. Establishing a biologically plausible learning rule for SR representations to form is of high significance in the field of neuroscience.
https://doi.org/10.7554/eLife.80680.sa0eLife digest
Memories are an important part of how we think, understand the world around us, and plan out future actions. In the brain, memories are thought to be stored in a region called the hippocampus. When memories are formed, neurons store events that occur around the same time together. This might explain why often, in the brains of animals, the activity associated with retrieving memories is not just a snapshot of what happened at a specific moment it can also include information about what the animal might experience next. This can have a clear utility if animals use memories to predict what they might experience next and plan out future actions.
Mathematically, this notion of predictiveness can be summarized by an algorithm known as the successor representation. This algorithm describes what the activity of neurons in the hippocampus looks like when retrieving memories and making predictions based on them. However, even though the successor representation can computationally reproduce the activity seen in the hippocampus when it is making predictions, it is unclear what biological mechanisms underpin this computation in the brain.
Fang et al. approached this problem by trying to build a model that could generate the same activity patterns computed by the successor representation using only biological mechanisms known to exist in the hippocampus. First, they used computational methods to design a network of neurons that had the biological properties of neural networks in the hippocampus. They then used the network to simulate neural activity. The results show that the activity of the network they designed was able to exactly match the successor representation. Additionally, the data resulting from the simulated activity in the network fitted experimental observations of hippocampal activity in Tufted Titmice.
One advantage of the network designed by Fang et al. is that it can generate predictions in flexible ways,. That is, it canmake both short and longterm predictions from what an individual is experiencing at the moment. This flexibility means that the network can be used to simulate how the hippocampus learns in a variety of cognitive tasks. Additionally, the network is robust to different conditions. Given that the brain has to be able to store memories in many different situations, this is a promising indication that this network may be a reasonable model of how the brain learns.
The results of Fang et al. lay the groundwork for connecting biological mechanisms in the hippocampus at the cellular level to cognitive effects, an essential step to understanding the hippocampus, as well as its role in health and disease. For instance, their network may provide a concrete approach to studying how disruptions to the ways neurons make and break connections can impair memory formation. More generally, better models of the biological mechanisms involved in making computations in the hippocampus can help scientists better understand and test out theories about how memories are formed and stored in the brain.
Introduction
To learn from the past, plan for the future, and form an understanding of our world, we require memories of personal experiences. These memories depend on the hippocampus for formation and recall (Scoville and Milner, 1957; Penfield and Milner, 1958; Corkin, 2002), but an algorithmic and mechanistic understanding of memory formation and retrieval in this region remains elusive. From a computational perspective, a key function of memory is to use past experiences to inform predictions of possible futures (Bubic et al., 2010; Wayne et al., 2018; Whittington et al., 2020; Momennejad, 2020). This suggests that hippocampal memory is stored in a way that is particularly suitable for forming predictions. Consistent with this hypothesis, experimental work has shown that, across species and tasks, hippocampal activity is predictive of the future experience of an animal (Skaggs and McNaughton, 1996; Lisman and Redish, 2009; Mehta et al., 1997; Payne et al., 2021; Muller and Kubie, 1989; Pfeiffer and Foster, 2013; Schapiro et al., 2016; Garvert et al., 2017). Furthermore, theoretical work has found that models endowed with predictive objectives tend to resemble hippocampal activity (Blum and Abbott, 1996; Mehta et al., 2000; Stachenfeld et al., 2017; Momennejad et al., 2017; Geerts et al., 2020; Recanatesi et al., 2021; Whittington et al., 2020; George et al., 2021). Thus, it is clear that predictive representations are an important aspect of hippocampal memory.
Inspired by work in the reinforcement learning (RL) field, these observations have been formalized by describing hippocampal activity as a predictive map under the successor representation (SR) algorithm (Dayan, 1993; Gershman et al., 2012; Stachenfeld et al., 2017). Under this framework, an animal’s experience in the world is represented as a trajectory through some defined state space, and hippocampal activity predicts the future experience of an animal by integrating over the likely states that an animal will visit given its current state. This algorithm further explains how, in addition to episodic memory, the hippocampus may support relational reasoning and decision making (Recanatesi et al., 2021; Mattar and Daw, 2018), consistent with differences in hippocampal representations in different tasks (Markus et al., 1995; Jeffery, 2021). The SR framework captures many experimental observations of neural activity, leading to a proposed computational function for the hippocampus (Stachenfeld et al., 2017).
While the SR algorithm convincingly argues for a computational function of the hippocampus, it is unclear what biological mechanisms might compute the SR in a neural circuit. Thus, several relevant questions remain that are difficult to probe with the current algorithm. What kind of neural architecture should one expect in a region that can support this computation? Are there distinct forms of plasticity and neuromodulation needed in this system? What is the structure of hippocampal inputs to be expected? A biologically plausible model can explore these questions and provide insight into both mechanism and function (Marr and Poggio, 1976; Frank, 2015; Love, 2021).
In other systems, it has been possible to derive biological mechanisms with the goal of achieving a particular network function or property (Zeldenrust et al., 2021; Karimi et al., 2022; Pehlevan et al., 2017; Olshausen and Field, 1996; Burbank, 2015; Aitchison et al., 2021; Földiák, 1990; Tyulmankov et al., 2022). Key to many of these models is the constraint that learning rules at any given neuron can only use information local to that neuron. A promising direction towards such a neural model of the SR is to use the dynamics of a recurrent neural network (RNN) to perform SR computations (Vértes and Sahani, 2019; Russek et al., 2017). An RNN model is particularly attractive as the hippocampus is highly recurrent, and its connectivity patterns are thought to support associative learning and recall (GardnerMedwin, 1976; McNaughton and Morris, 1987; Marr et al., 1991; Liu et al., 2012). However, an RNN model of the SR has not been tied to neural learning rules that support its operation and allow for testing of specific hypotheses.
Here, we show that an RNN with local learning rules and an adaptive learning rate exactly calculates the SR at steady state. We test our model with realistic inputs and make comparisons to neural data. In addition, we compare our results to the standard SR algorithm with respect to the speed of learning and the learned representations in cases where multiple solutions exist. Our work provides a mechanistic account for an algorithm that has been frequently connected to the hippocampus, but could only be interpreted at an algorithmic level. This networklevel perspective allows us to make specific predictions about hippocampal mechanisms and activity.
Results
The successor representation
The SR algorithm described in Stachenfeld et al., 2017 first discretizes the environment explored by an animal (whether a physical or abstract space) into a set of $n$ states that the animal transitions through over time (Figure 1A). The animal’s behavior can then be thought of as a Markov chain with a corresponding transition probability matrix $T}_{N\times N$ (Figure 1B). $T$ gives the probability that the animal transitions to a state ${s}^{\prime}$ from the state $s$ in one time step: ${T}_{ji}=P({s}^{\prime}=is=j)$. The SR matrix is defined as
Here, $\gamma \in (0,1)$ is a temporal discount factor. ${M}_{ji}$ can be seen as a measure of the occcupancy of state $i$ over time if the animal starts at state $j$, with $\gamma $ controlling how much to discount time steps in the future (Figure 1C). The SR of state $j$ is the $j$th row of $M$ and represents the states that an animal is likely to transition to from state $j$. Stachenfeld et al., 2017 demonstrate that, if one assumes each state drives a single neuron, the SR of $j$ resembles the population activity of hippocampal neurons when the animal is at state $j$ (Figure 1D). They also show that the $i$th column of $M$ resembles the place field (activity as a function of state) of a hippocampal neuron representing state $i$ (Figure 1E). In addition, the $i$th column of $M$ shows which states are likely to lead to state $i$.
Recurrent neural network computes SR at steady state
We begin by drawing connections between the SR algorithm (Stachenfeld et al., 2017) and an analogous neural network architecture. The input to the network encodes the current state of the animal and is represented by a layer of input neurons (Figure 1FG). These neurons feed into the rest of the network that computes the SR (Figure 1FG). The SR is then read out by a layer of output neurons so that downstream systems receive a prediction of the upcoming states (Figure 1FG). We will first model the inputs $\mathit{\varphi}$ as onehot encodings of the current state of the animal (Figure 1FG). That is, each input neuron represents a unique state, and input neurons are onetoone connected to the hidden neurons.
We first consider an architecture in which a recurrent neural network (RNN) is used to compute the SR (Figure 1F). Let us assume that the $T$ matrix is encoded in the synaptic weights of the RNN. In this case, the steady state activity of the network in response to input $\mathit{\varphi}$ retrieves a row of the SR matrix, $M}^{\u22ba}\mathit{\varphi$ (Figure 1F, subsection 4.14). Intuitively, this is because each recurrent iteration of the RNN progresses the prediction by one transition. In other words, the $t$th recurrent iteration raises $T$ to the $t$th power as in Equation 1. To formally derive this result, we first start by defining the dynamics of our RNN with classical rate network equations (Amarimber, 1972). At time $t$, the firing rate $\mathit{x}(t)$ of $N$ neurons given each neurons’ input $\mathit{\varphi}(t)$ follows the discretetime dynamics (assuming a step size $\mathrm{\Delta}t=1$)
Here, $\gamma $ scales the recurrent activity and is a constant factor for all neurons. The synaptic weight matrix $J\in {\mathcal{R}}_{N\times N}$ is defined such that ${J}_{ij}$ is the synaptic weight from neuron $j$ to neuron $i$. Notably, this notation is transposed from what is used in RL literature, where conventions have the first index as the starting state. Generally, $f$ is some nonlinear function in Equation 2. For now, we will consider $f$ to be the identity function, rendering this equation linear. Under this assumption, we can solve for the steady state activity $\mathit{x}}_{ss$ as
Equivalence between Equation 1 and Equation 3 is clearly reached when $J={T}^{\u22ba}$ (Russek et al., 2017; Vértes and Sahani, 2019). Thus, if the network can learn $T$ in its synaptic weight matrix, it will exactly compute the SR.
Here, the factor $\gamma $ represents the gain of the neurons in the network, which is factored out of the synaptic strengths characterized by $J$. Thus, $\gamma $ is an independently adjustable factor that can flexibly control the strength of the recurrent dynamics (see Sompolinsky et al., 1988). A benefit of this flexibility is that the system can retrieve successor representations of varying predictive strengths by modulating the gain factor $\gamma $. In this way, the predictive horizon can be dynamically controlled without any additional learning required. We will refer to the $\gamma $ used during learning of the SR as the baseline $\gamma $, or ${\gamma}_{B}$.
We next consider what is needed in a learning rule such that $J$ approximates ${T}^{\u22ba}$. In order to learn a transition probability matrix, a learning rule must associate states that occur sequentially and normalize the synaptic weights into a valid probability distribution. We derive a learning rule that addresses both requirements (Figure 1H, Appendix 2),
where $\eta $ is the learning rate. The first term in Equation 4 is a temporally asymmetric potentiation term that counts states that occur in sequence. This is similar to spiketiming dependent plasticity, or STDP (Bi and Poo, 1998; Skaggs and McNaughton, 1996; Abbott and Blum, 1996).
The second term in Equation 4 is a form of synaptic depotentiation. Depotentiation has been hypothesized to be broadly useful for stabilizing patterns and sequence learning (Földiák, 1990; Fiete et al., 2010), and similar inhibitory effects are known to be elements of hippocampal learning (Kullmann and Lamsa, 2007; Lamsa et al., 2007). In our model, the depotentiation term in Equation 4 imposes local antiHebbian learning at each neuron– that is, each column of $J$ is normalized independently. This normalizes the observed transitions from each state by the number of visits to that state, such that transition statistics are correctly captured. We note, however, that other ways of columnnormalizing the synaptic weight matrix may give similar representations (Appendix 7).
Crucially, the update rule (Equation 4) uses information local to each neuron (Figure 1H), an important aspect of biologically plausible learning rules. We show that, in the asymptotic limit, the update rule extracts information about the inputs $\mathit{\varphi}$ and learns $T$ exactly despite having access only to neural activity $\mathit{x}$ (Appendix 3). We will refer to an RNN using Equation 4 as the RNNSuccessor, or RNNS. Combined with recurrent dynamics (Equation 3), RNNS computes the SR exactly (Figure 1H).
As an alternative to the RNNS model, we consider the conditions necessary for a feedforward neural network to compute the SR. Under this architecture, the $M$ matrix must be encoded in the weights from the input neurons to the hidden layer neurons (Figure 1G). This can be achieved by updating the synaptic weights with a temporal difference (TD) learning rule, the standard update used to learn the SR in the usual algorithm. Although the TD update learns the SR, it requires information about multiple input layer neurons to make updates for the synapse from input neuron $j$ to output neuron $i$ (Figure 1I). Thus, it is useful to explore other possible mechanisms that are simpler to compute locally. We refer to the model described in Figure 1I as the feedforwardTD (FFTD) model. The FFTD model implements the canonical SR algorithm.
Evaluating SR learning by biologically plausible learning rules
To evaluate the effectiveness of the RNNS learning rule, we tested its accuracy in learning the SR matrix for random walks. Specifically, we simulated random walks with different transition biases in a 1D circular track environment (Figure 2A). The RNNS can learn the SR for these random walks (Figure 2B).
Because equivalence is only reached in the asymptotic limit of learning (i.e. $\mathrm{\Delta}J\to 0$), our RNNS model learns the SR slowly. In contrast, animals are thought to be able to learn the structure of an environment quickly (Zhang et al., 2021), and neural representations in an environment can also develop quickly (Monaco et al., 2014; Sheffield and Dombeck, 2015; Bittner et al., 2015). To remedy this, we introduce a dynamic learning rate that allows for faster normalization of the synaptic weight matrix, similar to the formula for calculating a moving average (Appendix 4). For each neuron, suppose that a trace $n$ of its recent activity is maintained with some time constant $\lambda \in (0,1)$,
If the learning rate of the outgoing synapses from each neuron $j$ is inversely proportional to ${n}_{j}\left(\eta =\frac{1}{{n}_{j}(t)}\right)$, the update equation quickly normalizes the synapses to maintain a valid transition probability matrix (Appendix 4). Modulating synaptic learning rates as a function of neural activity is consistent with experimental observations of metaplasticity (Abraham and Bear, 1996; Abraham, 2008; Hulme et al., 2014). We refer to this as an adaptive learning rate and contrast it with the previous static learning rate. We consider the setting where $\lambda =1$, so the learning rate monotonically decreases over time (Figure 2C). In general, however, the learning rate could increase or decrease over time if $\lambda <1$ (Figure 2C), and $n$ could be reset, allowing for rapid learning. Our learning rule with the adaptive learning rate is the same as in Equation 4, with the exception that $\eta =\text{min}(\frac{1}{{n}_{j}(t)},1.0)$ for synapses ${J}_{*j}$. This learning rule still relies only on information local to the neuron as in Figure 1H.
The RNNS with an adaptive learning rate normalizes the synapses more quickly than a network with a static learning rate (Figure 2D, Figure 2—figure supplement 1) and learns $T$ faster (Figure 2E, Figure 2—figure supplement 1). The RNNS with a static learning rate exhibits more of a tradeoff between normalizing synapses quickly (Figure 2D, Figure 2—figure supplement 1A) and learning $M$ accurately (Figure 2E, Figure 2—figure supplement 1). However, both versions of the RNNS estimate $M$ more quickly than the FFTD model (Figure 2F, Figure 2—figure supplement 1).
Place fields can form quickly, but over time the place fields may skew if transition statistics are consistently biased (Stachenfeld et al., 2017; Monaco et al., 2014; Sheffield and Dombeck, 2015; Bittner et al., 2015). The adaptive learning rate recapitulates both of these effects, which are thought to be caused by slow and fast learning processes, respectively. A low learning rate can capture the biasing of place fields, which develops over many repeated experiences. This is seen in the RNNS with a static learning rate (Figure 2G). However, a high learning rate is needed for hippocampal place cells to develop sizeable place fields in oneshot. Both these effects of slow and fast learning can be seen in the neural activity of an example RNNS neuron with an adaptive learning rate (Figure 2H). After the first lap, a sizeable field is induced in a oneshot manner, centered at the cell’s preferred location. In subsequent laps, the place field slowly distorts to reflect the bias of the transition statistics (Figure 2H). The model is able to capture these learning effects because the adaptive learning rate transitions between high and low learning rates, unlike the static version (Figure 2I).
Thus far, we have assumed that the RNNS learning rule uses pre→post activity over two neighboring time steps (Equation 4). A more realistic framing is that a convolution with a plasticity kernel determines the weight change at any synapse. We tested how this affects our model and what range of plasticity kernels best supports the estimation of the SR. We do this by replacing the pre→post potentiation term in Equation 4 with a convolution:
In the above equation, the full kernel $K$ is split into a pre→post kernel (${K}_{+}$) and a post→pre kernel (${K}_{}$). ${K}_{+}$ and ${K}_{}$ are parameterized as independent exponential functions, $A{e}^{t/\tau}$.
To systematically explore the space of plasticity kernels that can be used to learn the SR, we performed a grid search over the sign and the time constants of the pre→post and post→pre sides of the plasticity kernels. For each fixed sign and time constant, we used an evolutionary algorithm to learn the remaining parameters that determine the plasticity kernel. We find that plasticity kernels that are STDPlike are more effective than others, although plasticity kernels with slight post→pre potentiation work as well (Figure 2J). The network is sensitive to the time constant and tends to find solutions for time constants around a few hundred milliseconds (Figure 2JK). Our robustness analysis indicates the timescale of a plasticity rule in such a circuit may be longer than expected by standard STDP, but within the timescale of changes in behavioral states. We note that this also contrasts with behavioral timescale plasticity (Bittner et al., 2015), which integrates over a window that is several seconds long. Finally, we see that even plasticity kernels with slightly different time constants may give results with minimal error from the SR matrix, even if they do not estimate the SR exactly (Figure 2J). This suggests that, although other plasticity rules could be used to model longhorizon predictions, the SR is a reasonable –although not strictly unique– model to describe this class of predictive representations.
RNNS can compute the SR with arbitrary $\gamma}_{R$ under a stable regime of $\gamma}_{B$
We next investigate how robust the RNNS model is to the value of $\gamma $. Typically, for purposes of fitting neural data or for RL simulations, $\gamma $ will take on values as high as 0.9 (Stachenfeld et al., 2017; Barreto et al., 2017). However, previous work that used RNN models reported that recurrent dynamics become unstable if the gain $\gamma $ exceeds a critical value (Sompolinsky et al., 1988; Zhang et al., 2021). This could be problematic as we show analytically that the RNNS update rule is effective only when the network dynamics are stable and do not have nonnormal amplification (Appendix 2). If these conditions are not satisfied during learning, the update rule no longer optimizes for fitting the SR and the learned weight matrix will be incorrect.
We first test how the value of ${\gamma}_{B}$, the gain of the network during learning, affects the RNNS dynamics. The dynamics become unstable when ${\gamma}_{B}$ exceeds 0.6 (Figure 3—figure supplement 1). Specifically, the eigenvalues of the synaptic weight matrix exceed the critical threshold for learning when ${\gamma}_{B}>0.6$ (Figure 3A, ‘Linear’). As expected from our analytical results, the stability of the network is tied to the network’s ability to estimate $M$. RNNS cannot estimate $M$ well when ${\gamma}_{B}>0.6$ (Figure 3B, ‘Linear’). We explored two strategies to enable RNNS to learn at high $\gamma $.
One way to tame this instability is to add a saturating nonlinearity into the dynamics of the network. This is a feature of biological neurons that is often incorporated in models to prevent unbounded activity (Dayan and Abbott, 2001) Specifically, instead of assuming the network dynamics are linear ($f$ is the identity function in Equation 2), we add a hyperbolic tangent into the dynamics equation. This extends the stable regime of the network– the eigenvalues do not exceed the critical threshold until ${\gamma}_{B}>0.8$ (Figure 3A). Similar to the linear case, the network with nonlinear dynamics fits $M$ well until the critical threshold for stability (Figure 3b). These differences are clear visually as well. While the linear network does not estimate $M$ well for ${\gamma}_{B}=0.8$ (Figure 3B), the estimate of the nonlinear network (Figure 3c) is a closer match to the true $M$ (Figure 3D). However, there is a tradeoff between the stabilizing effect of the nonlinearity and the potential loss of accuracy in calculating $M$ with a nonlinearity (Figure 3—figure supplement 1).
We explore an alternative strategy for computing $M$ with arbitrarily high $\gamma $ in the range $0\le \gamma <1$. We have thus far pushed the limits of the model in learning the SR for different ${\gamma}_{B}$. However, an advantage of our recurrent architecture is that $\gamma $ is a global gain modulated independently of the synaptic weights. Thus, an alternative strategy for computing $M$ with high $\gamma $ is to consider two distinct modes that the network can operate under. First, there is a learning phase in which the plasticity mechanism actively learns the structure of the environment and the model is in a stable regime (i.e. ${\gamma}_{B}$ is small). Separately, there is a retrieval phase during which the gain ${\gamma}_{R}$ of the network can be flexibly modulated. By changing the gain, the network can compute the SR with arbitrary prediction horizons, without any changes to the synaptic weights. We show the effectiveness of separate network phases by simulating a 1D walk where the learning phase uses a small ${\gamma}_{B}$ (Figure 3E). Halfway through the walk, the animal enters a retrieval mode and accurately computes the SR with higher ${\gamma}_{R}$ (Figure 3E).
Under this scheme, the model can compute the SR for any $\gamma <1$ (Figure 3—figure supplement 1). The separation of learning and retrieval phases stabilizes neural dynamics and allows flexible tuning of predictive power depending on task context.
RNNS can be generalized to more complex inputs with successor features
We wondered how RNNS performs given more biologically realistic inputs. We have so far assumed that an external process has discretized the environment into uncorrelated states so that each possible state is represented by a unique input neuron. In other words, the inputs $\mathit{\varphi}$ are onehot vectors. However, inputs into the hippocampus are expected to be continuous and heterogeneous, with states encoded by overlapping sets of neurons (Hardcastle et al., 2017). When inputs are not onehot, there is not always a canonical groundtruth $T$ matrix to fit and the predictive representations are referred to as successor features (Barreto et al., 2017; Kulkarni et al., 2016). In this setting, the performance of a model estimating successor features is evaluated by the temporal difference (TD) loss function.
Using the RNNS model and update rule (Equation 4), we explore more realistic inputs $\mathit{\varphi}$ and refer to $\mathit{\varphi}$ as ‘input features’ for consistency with the successor feature literature. We vary the sparsity and spatial correlation of the input features (Figure 4A). As before (Figure 3H), the network will operate in separate learning and retrieval modes, where ${\gamma}_{B}$ is below the critical value for stability. Under these conditions, the update rule will learn
at steady state, where ${R}_{\mathit{\varphi}\mathit{\varphi}}(\tau )$ is the correlation matrix of $\mathit{\varphi}$ with time lag $\tau $ (Appendix 3). Thus, the RNNS update rule has the effect of normalizing the input feature via a decorrelative factor (${R}_{\mathit{\varphi}\mathit{\varphi}}{(0)}^{1}$) and mapping the normalized input to the feature expected at the next time step in a STDPlike manner (${R}_{\mathit{\varphi}\mathit{\varphi}}(1)$). This interpretation generalizes the result that $J={T}^{\u22ba}$ in the onehot encoding case (Appendix 3).
We wanted to further explore the function of the normalization term. In the onehot case, it operates over each synapse independently and makes a probability distribution. With more realistic inputs, it operates over a set of synapses and has a decorrelative effect. We first ask how the decorrelative term changes over learning of realistic inputs. We compare the mean value of the STDP term of the update (${x}_{i}(t){x}_{j}(t1)$) to the normalization term of the update (${x}_{j}(t1){\sum}_{k}{J}_{ik}{x}_{k}(t1)$) during a sample walk (Figure 4B). The RNNS learning rule has stronger potentiating effects in the beginning of the walk. As the model learns more of the environment and converges on the correct transition structure, the strength of the normalization term balances out the potentiation term. It may be that the normalization term is particularly important in maintaining this balance as inputs become more densely encoded. We test this hypothesis by using a normalization term that operates on each synapse independently (similar to Oja’s Rule, Oja, 1982, Appendix 5). We see that the equilibrium between potentiating and depressing effects is not achieved by this type of independent normalization (Figure 4B, Appendix 6).
We wondered whether the decorrelative normalization term is necessary for the RNNS to develop accurate representations. By replacing the decorrelative term with an independent normalization, features from nonadjacent states begin to be associated together and the model activity becomes spatially nonspecific over time (Figure 4C, top). In contrast, using the decorrelative term, the RNNS population activity is more localized (Figure 4C, bottom).
Interestingly, we noticed an additional feature of place maps as we transitioned from onehot feature encodings to more complex feature encodings. We compared the representations learned by the RNNS in a circular track walk with onehot features versus more densely encoded features. For both input distributions, the RNNS displayed the same skewing in place fields seen in Figure 2 (Figure 4—figure supplement 1). However, the place field peaks of the RNNS model additionally shifted backwards in space for the more complex feature encodings (Figure 4D). This was not seen for the onehot encodings (Figure 4D). The shifting in the RNNS model is consistent with the observations made in Mehta et al., 2000 and demonstrates the utility of considering more complex input conditions. A similar observation was made in Stachenfeld et al., 2017 with noisy state inputs. In both cases, field shifts could be caused by neurons receiving external inputs at more than one state, particularly at states leading up to its original field location.
RNNS estimates successor features even with naturalistic trajectories
We ask whether RNNS can accurately estimate successor features, particularly under conditions of natural behavior. Specifically, we used the dataset from Payne et al., 2021, gathered from foraging Tufted Titmice in a 2D arena (Figure 5A). We discretize the arena into a set of states and encode each state as in Section 2.5. Using positiontracking data from Payne et al., 2021, we simulate the behavioral trajectory of the animal as transitions through the discrete state space. The inputs into the successor feature model are the features associated with the states in the behavioral trajectory.
We first wanted to test whether the RNNS model was robust across a range of different types of input features. We calculate the TD loss of the model as a function of the spatial correlation across inputs $\mathit{\varphi}$ (Figure 5B). We find that the model performs well across a range of inputs but loss is higher when inputs are spatially uncorrelated. This is consistent with the observation that behavioral transitions are spatially local, such that correlations across spatially adjacent features aid in the predictive power of the model. We next examine the model performance as a function of the sparsity of inputs $\mathit{\varphi}$ (Figure 5C). We find the model also performs well across a range of feature sparsity, with lowest loss when features are sparse.
To understand the interacting effects of spatial correlation and feature sparsity in more detail, we performed a parameter sweep over both of these parameters (Figure 5D, Figure 5—figure supplement 1AF). We generated random patterns according to the desired sparsity and smoothness with a spatial filter to generate correlations. This means that the entire parameter space is not covered in our sweep (e.g. the topleft area with high correlation and high sparsity is not explored). Note that since we generate $\mathit{\varphi}$ by randomly drawing patterns, the special case of onehot encoding is also not included in the parameter sweep (onehot encoding is already explored in Figure 2). The RNNS seems to perform well across a wide range, with highest loss in regions of low spatial correlation and low sparsity.
We want to compare the TD loss of RNNS to that of a nonbiological model designed to minimize TD loss. We repeat the same parameter sweep over input features with the FFTD model (Figure 5E, Figure 5—figure supplement 1G). The FFTD model performs similarly to the RNNS model, with lower TD loss in regions with low sparsity or higher correlation. We also tested how the performance of both models is affected by the strength of ${\gamma}_{R}$ (Figure 5F). Both models show a similar increase in TD loss as ${\gamma}_{R}$ increases, although the RNNS has a slightly lower TD loss at high $\gamma $ than the FFTD model. Both models perform substantially better than a random network with weights of comparable magnitude (Figure 5—figure supplement 1D).
Unlike in the onehot case, there is no groundtruth $T$ matrix for nononehot inputs, so representations generated by RNNS and FFTD may look different, even at the same TD loss. Therefore, to compare the two models, it is important to compare representations to neural data.
RNNS fits neural data in a random foraging task
Finally, we tested whether the neural representations learned by the models with behavioral trajectories from Figure 5 match hippocampal firing patterns. We performed new analysis on neural data from Payne et al., 2021 to establish a dataset for comparison. The neural data from Payne et al., 2021 was collected from electrophysiological recordings in titmouse hippocampus during freely foraging behavior (Figure 6A). Payne et al., 2021 discovered the presence of place cells in this area. We analyzed statistics of place cells recorded in the anterior region of the hippocampus, where homology with rodent dorsal hippocampus is hypothesized (Tosches et al., 2018). We calculated the distribution of place field size measured relative to the arena size (Figure 6B), as well as the distribution of the number of place fields per place cell (Figure 6C). Interestingly, with similar analysis methods, Henriksen et al., 2010 see similar statistics in the proximal region of dorsal CA1 in rats, indicating that our analyses could be applicable across organisms.
In order to test how spatial representations in the RNNS are impacted by input features, we performed parameter sweeps over input statistics. As in Payne et al., 2021, we define place cells in the model as cells with at least one statistically significant place field under permutation tests. Under most of the parameter range, all RNNS neurons would be identified as a place cell (Figure 6D). However, under conditions of high spatial correlation and low sparsity, a portion of neurons (12%) do not have any fields in the environment. These cells are excluded from further analysis. We measured how the size of place fields varies across the parameter range (Figure 6E). The size of the fields increases as a function of the spatial correlation of the inputs, but is relatively insensitive to sparsity. This effect can be explained as the spatial correlation of the inputs introducing an additional spatial spread in the neural activity. Similarly, we measured how the number of place fields per cell varies across the parameter range (Figure 6F). The number of fields is maximal for conditions in which input features are densely encoded and spatial correlation is low. These are conditions in which each neuron receives inputs from multiple, spatially distant states.
Finally, we wanted to identify regions of parameter space that were similar to the data of Payne et al., 2021. We measured the KL divergence between our model’s place field statistics (Figure 6DE) and the statistics measured in Payne et al., 2021 (Figure 6BC). We combined the KL divergence of both these distributions to find the parameter range in which the RNNS best fits neural data (Figure 6G). This optimal parameter range occurs when inputs have a spatial correlation of $\sigma \approx 8.75$ cm and sparsity $\approx 0.15$. We note that the splithalf noise floor of the dataset of Payne et al., 2021 is a KL divergence of 0.12 bits (Figure 6—figure supplement 1E). We can visually confirm that the model fits the data well by plotting the place fields of RNNS neurons (Figure 6H).
We wondered whether the predictive gain (${\gamma}_{R}$) of the representations affects the ability of the RNNS to fit data. The KL divergence changes only slightly as a function of ${\gamma}_{R}$. Mainly, the KLdivergence of the place field size increases as ${\gamma}_{R}$ increases (Figure 6I), but little effect is seen in the distribution of the number of place fields per neuron (Figure 6J).
We next tested whether the neural data was better fit by representations generated by RNNS or the FFTD model. Across all parameters of the input features, despite having similar TD loss (Figure 5DE), the FFTD model has much higher divergence from neural data (Figure 6GI, Figure 6—figure supplement 1), similar to a random feedforward network (Figure 6—figure supplement 1E).
Overall, our RNNS model seems to strike a balance between performance in estimating successor features, similarity to data, and biological plausibility. Furthermore, our analyses provide a prediction of the input structure into the hippocampus that is otherwise not evident in an algorithmic description or in a model that only considers onehot feature encodings.
Discussion
Hippocampal memory is thought to support a wide range of cognitive processes, especially those that involve forming associations or making predictions. However, the neural mechanisms that underlie these computations in the hippocampus are not fully understood. A promising biological substrate is the recurrent architecture of the CA3 region of the hippocampus and the plasticity rules observed. Here, we showed how a recurrent network with local learning rules can implement the successor representation, a predictive algorithm that captures many observations of hippocampal activity. We used our neural circuit model to make specific predictions of biological processes in this region.
A key component of our plasticity rule is a decorrelative term that depresses synapses based on coincident activity. Such antiHebbian or inhibitory effects are hypothesized to be broadly useful for learning, especially in unsupervised learning with overlapping input features (LitwinKumar and Doiron, 2014; Sadeh and Clopath, 2021; Pehlevan et al., 2018; Payne et al., 2021). Consistent with this hypothesis, antiHebbian learning has been implicated in circuits that perform a wide range of computations, from distinguishing patterns (Földiák, 1990), to familiarity detection (Tyulmankov et al., 2022), to learning birdsong syllables (Mackevicius et al., 2020). This inhibitory learning may be useful because it decorrelates redundant information, allowing for greater specificity and capacity in a network (Sadeh and Clopath, 2021; Földiák, 1990). Our results provide further support of these hypotheses and predict that antiHebbian learning is fundamental to a predictive neural circuit.
We derive an adaptive learning rate that allows our model to quickly learn a probability distribution, and generally adds flexibility to the learning process. The adaptive learning rate changes such that neurons that are more recently active have a slower learning rate. This is consistent with experimental findings of metaplasticity at synapses (Abraham and Bear, 1996; Abraham, 2008; Hulme et al., 2014), and theoretical proposals that metaplasticity tracks the uncertainty of information (Aitchison et al., 2021). In RNNS, the adaptive learning rate improves the speed of learning and better recapitulates hippocampal data. Our adaptive learning rate also has interesting implications for flexible learning. Memory systems must be able to quickly learn new associations throughout their lifetime without catastrophe. Our learning rate is parameterized by a forgetting term $\lambda $ that controls the timescale in which environmental statistics are expected to be stationary. Although we fixed $\lambda =1$ in our simulations, there are computational benefits in considering cases where $\lambda <1$. This parameter provides a natural way for a memory system to forget gradually over time and prioritize recent experiences, in line with other theoretical studies that have also suggested that learning and forgetting on multiple timescales allow for more flexible behavior (Kaplanis et al., 2018; Fusi et al., 2007).
We tested the sensitivity of our network to various parameters and found a broad range of valid solutions. Prior work has sought to understand how an emergent property of a network could be generated by multiple unique solutions (Goldman et al., 2001; Prinz et al., 2004; Bittner et al., 2021; Hertäg and Clopath, 2022). It has been suggested that redundancy in solution space makes systems more robust, accounting for margins of error in the natural world (Marder and Goaillard, 2006; Marder and Taylor, 2011). In a similar vein, our parameter sweep over plasticity kernels revealed that a sizeable variety of kernels give solutions that resemble the SR. Although our model was initially sensitive to the value of $\gamma $, we found that adding biological components, such as nonlinear dynamics and separate network modes, broadened the solution space of the network.
Several useful features arise from the fact that RNNS learns the transition matrix $T$ directly, while separating out the prediction timescale, $\gamma $, as a global gain factor. It is important for animals to engage in different horizons of prediction depending on task or memory demands (Mattar and Lengyel, 2022; Bellmund et al., 2020). In RNNS, changing the prediction time horizon is as simple as increasing or decreasing the global gain of the network. Mechanistically, this could be accomplished by a neuromodulatory gain factor that boosts $\gamma $, perhaps by increasing the excitability of all neurons (Heckman et al., 2009; Nadim and Bucher, 2014). In RNNS, it was useful to have low network gain during learning (${\gamma}_{B}$), while allowing higher gain during retrieval to make longer timescale predictions (${\gamma}_{R}$). This could be accomplished by a neuromodulatory factor that switches the network into a learning regime (Pawlak et al., 2010; Brzosko et al., 2019), for example acetylcholine, which reduces the gain of recurrent connections and increases learning rates (Hasselmo, 1999; Hasselmo, 2006). The idea that the hippocampus might compute the SR with flexible $\gamma $ could help reconcile recent results that hippocampal activity does not always match high$\gamma $ SR (Widloski and Foster, 2022; Duvelle et al., 2021). Additionally, flexibility in predictive horizons could explain the different timescales of prediction observed across the anatomical axes of the hippocampus and entorhinal cortex (Jung et al., 1994; Dolorfo and Amaral, 1998; Brun et al., 2008; Kjelstrup et al., 2008; Strange et al., 2014; Poppenk et al., 2013; Brunec and Momennejad, 2022). Specifically, a series of successor networks with different values of $\gamma$ used in retrieval could establish a gradient of predictive timescales. Functionally, this may allow for learning hierarchies of state structure and could be useful for hierarchical planning (McKenzie et al., 2014; Momennejad and Howard, 2018; RibasFernandes et al., 2019).
Estimating $T$ directly provides RNNS with a means to sample likely future trajectories, or distributions of trajectories, which is computationally useful for many memoryguided cognitive tasks beyond reinforcement learning, including reasoning and inference (Ostojic and Fusi, 2013; Goodman et al., 2016). The representation afforded by $T$ may also be particularly accessible in neural circuits. Ostojic and Fusi, 2013 note that only few general assumptions are needed for synaptic plasticity rules to estimate transition statistics. Thus, it is reasonable to assume that some form of transition statistics are encoded broadly across the brain.
Interestingly, we also found that the recurrent network fit hippocampal data better than a feedforward network. An interesting direction for further work involves untangling which brain areas and cognitive functions can be explained by deep (feed forward) neural networks (Bonnen et al., 2021), and which rely on recurrent architectures, or even richer combinations of generative structures (Das et al., 2021). Recurrent networks, such as RNNS, support generative sequential sampling, reminiscent of hippocampal replay, which has been proposed as a substrate for planning, imagination, and structural inference (Foster and Wilson, 2006; Singer et al., 2013; Momennejad and Howard, 2018; Evans and Burgess, 2020; Kay et al., 2020).
There are inherent limitations to the approach of using a recurrent network to estimate the SR. For instance, network dynamics can be prone to issues of instability due to the recurrent buildup of activity. To prevent this instability, we introduce two different modes of operation, ‘learning’ and ‘retrieval’. An additional limitation is that errors in the estimated onestep transition can propagate over the course of the predictive rollout. This is especially problematic if features are more densely coded or more correlated, which makes onestep transition estimations more difficult. These insights into vulnerabilities of a recurrent network have interesting parallels in biology. Some hippocampal subfields are known to be highly recurrent (Schaffer, 1892; Ramón and Cajal, 1904; Miles and Wong, 1986; Le Duigou et al., 2014). This recurrency has been linked to the propensity of the hippocampus to enter unstable regimes, such as those that produce seizures (Sparks et al., 2020; Thom, 2014; Lothman et al., 1991; Knight et al., 2012). It remains an open question how a healthy hippocampus maintains stable activity, and to what extent the findings in models such as ours can suggest biological avenues to tame instability.
Other recent theoretical works have also sought to find biological mechanisms to learn successor representations, albeit with different approaches (Brea et al., 2016; de Cothi and Barry, 2020; Bono et al., 2023; Lee, 2022; George et al., 2023). For instance, the model from George et al., 2023 explores a feedforward network that takes advantage of theta phaseprecession to learn the SR. They analyze how place cells deform around boundaries and the function of the dorsalventral gradient in field size. The model introduced by Bono et al., 2023 uses a feedforward network with hippocampal replay. They explore how replay can modulate the biasvariance tradeoff of their SR estimate and apply their model to fearconditioning data. It is important to note that these mechanisms are not mutually exclusive with RNNS. Taken together with our work, these models suggest that there are multiple ways to learn the SR in a biological circuit and that these representations may be more accessible to neural circuits than previously thought.
Methods
Code availability
Code is posted on Github: https://github.com/chingf/srproject; Fang, 2022.
Random walk simulations
We simulated random walks in 1D (circular track) and 2D (square) arenas. In 1D simulations, we varied the probability of staying in the current state and transitioning forwards or backwards to test different types of biases on top of a purely random walk. In 2D simulations, the probabilities of each possible action were equal. In our simulations, one timestep corresponds to 1/3 second and spatial bins are assumed to be 5 cm apart. This speed of movement (15 cm/s) was chosen to be consistent with previous experiments. In theory, one can imagine different choices of timestep size to access different time horizons of prediction– that is, the choice of timestep interacts with the choice of $\gamma $ in determining the prediction horizon.
RNNS model
This section provides details and pseudocode of the RNNS simulation. Below are explanations of the most relevant variables:
$N$  Number of states, also equal to the number of neurons in the RNN 
$\mathit{x}$  Nlength vector of RNN neural activity 
$J$  $(N\times N)$ synaptic weight matrix 
$M$  $(N\times N)$ SR matrix 
$\mathit{\varphi}$  $N$length input vector into network 
$b$  binary variable indicating learning (0) or retrieval (1) mode 
${\gamma}_{B}$  Value of $\gamma$ the network uses to calculate $M$ in learning mode 
${\gamma}_{R}$  Value of $\gamma$ the network uses to calculate $M$ in retrieval mode 
$\mathit{n}$  Variable that tracks the activity of neurons integrated over time 
$\lambda $  Discount value the network uses to calculate $n$ 
$\mathit{\eta}$  Learning rate 
The RNNS algorithm is as follows:
Algorithm 1 RNNS. 
Inputs: $\mathit{\varphi}(t)$ for $t\in 1,\mathrm{\dots},T$ $b(t)$ for $t\in 1,\dots ,T$ Initialize: $J\leftarrow {\mathbf{0}}_{N\times N}$ $\mathit{n}\leftarrow {\mathbf{0}}_{N}$ $\mathit{x}(t)\leftarrow {\mathbf{0}}_{N}$ for $t\in 1,\mathrm{\dots},T$ for $t\mathrm{\in}\mathrm{1}\mathrm{,}\mathrm{\dots}\mathrm{,}T$ do if $b\mathrm{(}t\mathrm{)}\mathrm{=}\mathrm{=}\mathrm{1}$ then // Retrieval Mode $M}^{\u22ba}\leftarrow (1{\gamma}_{R}J{)}^{1$ $\mathit{x}(t)\leftarrow {M}^{\u22ba}\mathit{\varphi}(t)$ else // Learning Mode $M}^{\u22ba}\leftarrow (1{\gamma}_{B}J{)}^{1$ $\mathit{x}(t)\leftarrow {M}^{\u22ba}\mathit{\varphi}(t)$ $\mathit{n}\leftarrow \mathit{x}(t)+\lambda \mathit{n}$ // Learning rate update $\mathrm{\Delta}J\leftarrow \mathit{x}(t)\mathit{x}(t1{)}^{\u22ba}(J\mathit{x}(t1))\mathit{x}(t1{)}^{\u22ba}$ // Calculate weight update $\mathit{\eta}=\frac{1}{\mathit{n}}$ // Get learning rates (elementwise inversion) $\mathit{\eta}=\mathtt{\text{min}}(\mathit{\eta},1.0)$ // Learning rates can’t exceed 1.0 ${J}_{ij}\leftarrow {J}_{ij}+{\eta}_{j}\mathrm{\Delta}{J}_{ij}$ // Update synaptic weight matrix end if end for return $\mathit{x}$ 
RNNS with plasticity kernels
We introduce additional kernelrelated variables to the RNNS model above that are optimized by an evolutionary algorithm (see following methods subsection for more details):
$A}_{+},\phantom{\rule{thinmathspace}{0ex}}{\tau}_{+$  pre→ post side of the plasticity kernel ${K}_{+}(t)={A}_{+}{E}^{t/{\tau}_{+}}$ 
$A}_{},\phantom{\rule{thinmathspace}{0ex}}{\tau}_{$  As above, but for the post→ pre side 
${\alpha}_{d}$  Scaling term to allow for different selfsynapse updates 
${\alpha}_{n}$  Scaling term to allow for different learning rate updates 
We also define the variable ${t}_{k}=20$, which is the length of the temporal support for the plasticity kernel. The value of t_{k} was chosen such that ${e}^{{t}_{k}/\tau}$ was negligibly small for the range of $\tau $ we were interested in. The update algorithm is the same as in Algorithm 1, except lines 1516 are replaced with the following:
Algorithm 2 Plasticity kernel update 

$\mathit{n}\leftarrow {\alpha}_{n}\mathit{x}+\lambda \mathit{n}$ // Learning rate update $\mathit{k}}_{\mathbf{+}}\leftarrow {A}_{+}\sum _{{t}^{\mathrm{\prime}}=0}^{{t}_{k}}\mathit{x}(t{t}^{\mathrm{\prime}}){e}^{{t}^{\mathrm{\prime}}/{\tau}_{+}$ // Convolution with plasticity kernel $\mathit{k}}_{\mathbf{}}\leftarrow {A}_{}\sum _{{t}^{\mathrm{\prime}}=0}^{{t}_{k}}\mathit{x}(t{t}^{\mathrm{\prime}}){e}^{{t}^{\mathrm{\prime}}/{\tau}_{}$ $\mathrm{\Delta}{J}_{K}\leftarrow \mathit{x}(t){\mathit{k}}_{\mathbf{+}}^{\u22ba}+{\mathit{k}}_{\mathbf{}}\mathit{x}(t{)}^{\u22ba}$ // Calculate contribution to update from plasticity kernel $\mathrm{\Delta}{J}_{K}[ii]\leftarrow {\alpha}_{d}\mathit{x}(t){\mathit{k}}_{\mathbf{+}}^{\u22ba}$ // Updates to selfsynapses use separate scaling $\mathrm{\Delta}J\leftarrow \mathrm{\Delta}{J}_{K}(J\mathit{x}){\mathit{x}}^{\u22ba}$ // Calculate weight update 
Metalearning of RNN parameters
To learn parameters of the RNNS model, we use covariance matrix adaptation evolution strategy (CMAES) to learn the parameters of the plasticity rule. The training data provided are walks simulated from a random distribution of 1D walks. Walks varied in the number of states, the transition statistics, and the number of timesteps simulated. The loss function was the meansquared error (MSE) loss between the RNN $J$ matrix and the ideal estimated $T$ matrix at the end of the walk.
RNNS with truncated recurrent steps and nonlinearity
For the RNNS model with ${t}_{max}$ recurrent steps, lines 10 and 13 in Algorithm 1 is replaced with ${M}^{\u22ba}\leftarrow {\sum}_{t=0}^{{t}_{max}}{\gamma}^{t}{J}^{t}$.
For RNNS with nonlinear dynamics, there is no closed form solution. So, we select a value for ${t}_{max}$ and replace lines 10 and 13 in Algorithm 1 with an iterative update for ${t}_{max}$ steps: $\mathrm{\Delta}\mathit{x}=\mathit{x}+\gamma J\mathtt{\text{tanh}}({\mathit{x}}^{\mathrm{\prime}})+\mathit{\varphi}$. We choose ${t}_{max}$ such that $\gamma}_{max}^{t}<{10}^{4$.
RNNS with successor features
We use a tanh nonlinearity as described above. For simplicity, we set ${\gamma}_{B}=0$.
RNNS with independent normalization
As in Algorithm 1, but with the following in place of line 16
FFTD Model
In all simulations of the FFTD model, we use the temporal difference update. We perform a small grid search over the learning rate $\eta $ to minimize error (for SR, this is the MSE between the true $M$ and estimated $M$; for successor features, this is the temporal difference error). In the onehot SR case, the temporal difference update given an observed transition from state $s$ to state ${s}^{\prime}$ is:
for all synapses $j\to i$. Given arbitrarily structured inputs (as in the successor feature case), the temporal difference update is:
or, equivalently,
Generation of feature encodings for successor feature models
For a walk with $N$ states, we created $N$dimensional feature vectors for each state. We choose an initial sparsity probability $p$ and create feature vectors as random binary vectors with probability $p$ of being ‘on’. The feature vectors were then blurred by a 2D Gaussian filter with variance $\sigma $ with 1 standard deviation of support. The blurred features were then minsubtracted and maxnormalized. The sparsity of each feature vector was calculated as the L1 norm divided by $N$. The sparsity $s$ of the dataset then was the median of all the sparsity values computed from the feature vectors. To vary the spatial correlation of the dataset we need only vary $\sigma $. To vary the sparsity $s$ of the dataset we need to vary $p$, then measure the final $s$ after blurring with $\sigma $. Note that, at large $\sigma $, the lowest sparsity values in our parameter sweep were not possible to achieve.
Measuring TD loss for successor feature models
We use the standard TD loss function (Equation 18). To measure TD loss, at the end of the walk we take a random sample of observed transition pairs $(\mathit{\varphi},{\mathit{\varphi}}^{\mathrm{\prime}})$. We use these transitions as the dataset to evaluate the loss function.
Analysis of place field statistics
We use the open source dataset from Payne et al., 2021. We select for excitatory cells in the anterior tip of the hippocampus. We then select for place cells using standard measures (significantly placemodulated and stable over the course of the experiment).
We determined place field boundaries with a permutation test as in Payne et al., 2021. We then calculated the number of fields per neuron and the field size as in Henriksen et al., 2010. The same analyses were conducted for simulated neural data from the RNNS and FFTD models.
Behavioral simulation of Payne et al
We use behavioral tracking data from Payne et al., 2021. For each simulation, we randomly select an experiment and randomly sample a 28min window from that experiment. If the arena coverage is less than 85% during the window, we redo the sampling until the coverage requirement is satisfied. We then downsample the behavioral data so that the frame rate is the same as our simulation (3 FPS). Then, we divide the arena into a $14\times 14$ grid. We discretize the continuous X/Y location data into these states. This sequence of states makes up the behavioral transitions that the model simulates.
Place field plots
From the models, we get the activity of each model neuron over time. We make firing field plots with the same smoothing parameters as Payne et al., 2021.
Citation diversity statement
Systemic discriminatory practices have been identified in neuroscience citations, and a ‘citation diversity statement’ has been proposed as an intervention (Dworkin et al., 2020; Zurn et al., 2020). There is evidence that quantifying discriminatory practices can lead to systemic improvements in academic settings (Hopkins, 2002). Many forms of discrimination could lead to a paper being undercited, for example authors being less widely known or less respected due to discrimination related to gender, race, sexuality, disability status, or socioeconomic background. We manually estimated the number of male and female first and last authors that we cited, acknowledging that this quantification ignores many known forms of discrimination, and fails to account for nonbinary/intersex/trans folks. In our citations, firstlast author pairs were 62% malemale, 19% femalemale, 9% malefemale, and 10% femalefemale, somewhat similar to base rates in our field (biaswatchneuro.com). To familiarize ourselves with the literature, we used databases intended to counteract discrimination (blackinneuro.com, anneslist.net, connectedpapers.com). The process of making this statement improved our paper, and encouraged us to adopt less biased practices in selecting what papers to read and cite in the future. We were somewhat surprised and disappointed at how low the number of female authors were, despite being a femalefemale team ourselves. Citation practices alone are not enough to correct the power imbalances endemic in academic practice National Academies of Sciences, 2018 — this requires corrections to how concrete power and resources are distributed.
Appendix 1
Finding the conditions to retrieve from RNN steadystate activity
The successor representation is defined as
where $T$ is the transition probability matrix such that ${T}_{ji}=P({s}^{\prime}=is=j)$ for current state $s$ and future state ${s}^{\prime}$
For an RNN with connectivity $J$, activity $\mathit{x}$, input $\mathit{\varphi}$, and gain $\gamma \in [0,1]$, the (linear) discretetime dynamics equation is (Amarimber, 1972)
Furthermore, the steady state solution can be found by setting $\mathrm{\Delta}\mathit{x}=0$
Assume that $J={T}^{T}$ as a result of the network using some STDPlike learning rule where prepost connections are potentiated. The transposition is due to notational differences from the RL literature, where the $ij$ th index typically concerns the direction from state $i$ to state $j$. This is a result of differences in RL and RNN conventions in which inputs are leftmultiplied and rightmultiplied, respectively. Let $\gamma $ be a neuromodulatory factor that is applied over the whole network (and, thus, does not need to be encoded in the synaptic weights). Then, the equivalence to equation 12 becomes clear and our steady state solution can be written as:
This is consistent with the successor representation framework shown in Stachenfeld et al., 2017, where the columns of the $M$ matrix represent the firing fields of a neuron, and the rows of the $M$ matrix represent the network response to some input.
Appendix 2
Deriving the RNNS learning rule from TD Error and showing the learning rule is valid under a stability condition
Transitions between states $(s,{s}^{\prime})$ are observed as features $\varphi (s),\varphi ({s}^{\mathrm{\prime}})$ where $\mathit{\varphi}$ is some function. For notational simplicity, we will write these observed feature transitions as $(\mathit{\varphi},{\mathit{\varphi}}^{\mathrm{\prime}})$. A dataset $\mathcal{D}$ is comprised of these observed feature transitions over a behavioral trajectory. Successor features are typically learned by some function approximator $\psi (\mathit{\varphi};\theta )$ that is parameterized by $\theta $ and takes in the inputs $\mathit{\varphi}$. The SF approximator, $\psi $, is learned by minimizing the temporal difference (TD) loss function (Sutton and Barto, 2018):
for the current policy $\pi $. Here, the TD target is $\mathit{\varphi}+\gamma {\psi}^{\pi}({\mathit{\varphi}}^{\mathrm{\prime}};\theta )$. Analogous to the modelfree setting where the value function $V$ is being learned, $\mathit{\varphi}$ is in place of the reward $r$. Following these definitions, we can view the RNNS as the function approximator $\psi $:
For a single transition $(\mathit{\varphi},{\mathit{\varphi}}^{\mathrm{\prime}})$ we can write out the loss as follows:
For each observed transition, we would like to update $\psi $ such that the loss $L$ is minimized. Thus, we take the gradient of this temporal difference loss function with respect to our parameter $\theta =J$:
We can make the TD approximation (Sutton and Barto, 2018):
While ${\nabla}_{J}L(\theta )$ gives the direction of steepest descent in the loss, we will consider a linear transformation of the gradient that allows for a simpler update rule. This simpler update rule will be more amenable to a biologically plausible learning rule. We define this modified gradient as $D={\nabla}_{J}L(\theta )M$ where $M={(I\gamma J)}^{\u22ba}$. We must first understand the condition for $D$ to be in a direction of descent:
This expression is satisfied if $M+{M}^{\u22ba}$ is positive definite (its eigenvalues are positive). Thus, we find that our modified gradient points towards a descent direction if the eigenvalues of $M+{M}^{\u22ba}$ are positive. Interestingly, this condition is equivalent to stating that the recurrent network dynamics are stable and do not exhibit nonnormal amplification (Kumar et al., 2022; Murphy and Miller, 2009; Goldman, 2009). In other words, as long as the network dynamics are in a stable regime and do not have nonnormal amplification, our modified gradient reduces the temporal difference loss. Otherwise, the gradient will not point towards a descent direction.
We will use the modified gradient $D=({\mathit{x}}^{\mathrm{\prime}}J\mathit{x}){\mathit{x}}^{\u22ba}$ as our synaptic weight update rule. Our theoretical analysis explains much of the results seen in the main text. As the gain parameter ${\gamma}_{B}$ is increased, the network is closer to the edge of stability (the eigenvalues of $M$ are close to positive values, Figure 3A). Stability itself is not enough to guarantee that our update rule is valid. We need the additional constraint that nonnormal amplification should not be present (eigenvalues of $M+{M}^{\u22ba}$ are positive). In practice, however, this does not seem to be a mode that affects our network. That is, the ${\gamma}_{B}$ value for which the error in the network increases coincides with the ${\gamma}_{B}$ value for which the network is no longer stable (Figure 3B). Our theoretical analysis also shows that the gain ${\gamma}_{B}$ can always be decreased such that the eigenvalues of $M+{M}^{\u22ba}$ are positive and our update rule is valid (Figure 3E). At the most extreme, one can set ${\gamma}_{B}=0$ during learning to maintain stability (as we do in Figure 4 and onwards).
Appendix 3
Proving the RNNS update rule calculated on firing rates ($\mathit{x}$) depends only on feedforward inputs ($\mathit{\varphi}$) at steady state
We will show that our update rule, which uses $\mathit{x}$ (neural activity), converges on a solution that depends only on $\mathit{\varphi}$ (the feedforward inputs). We will also show that in the onehot case, we learn the SR exactly.
As a reminder, our learning rule for each $j\to i$ synapse is:
We can solve for the steady state solution of Equation 30 (set $\mathrm{\Delta}J=0$). Let $A={(1\gamma J)}^{1}$ for notational convenience, and recall that in steady state $\mathit{x}=A\mathit{\varphi}$. Let $\u27e8\mathit{x}\u27e9$ denote the average of $\mathit{x}$ over time.
Note that, since $A={(1\gamma J)}^{1}$, $J=\frac{1}{\gamma}(1{A}^{1})$.
Thus,
Therefore,
where ${R}_{\mathit{\varphi}\mathit{\varphi}}(\tau )$ is the autocorrelation matrix for some time lag $\tau $. Therefore, the RNNS weight matrix $J$ at steady state is only dependent on the inputs into the RNN over time.
In the case where $\mathit{\varphi}$ is onehot, we compute the SR exactly. This is because the steady state solution at each $j\to i$ synapse simplifies into the following expression:
This is the definition of the transition probability matrix and we see that $J={T}^{\u22ba}$. Note that the solution for ${J}_{ij}$ in Equation 41 is undefined if state $j$ is never visited. We assume each relevant state is visited at least once here.
Appendix 4
Deriving the adaptive learning rate update rule
This section explains how the adaptive learning rate is derived. The logic will be similar to calculating a weighted running average. Let ${d}_{ij}(t)$ be a binary function that is 1 if the transition from timestep $t1$ to timestep $t$ is state $j$ to state $i$. Otherwise, it is 0. Assume $\mathit{\varphi}$ is onehot encoded. Notice that in the onehot case, the RNNS update rule (Equation 4) simplifies to:
What $\eta $ should be used so $J$ approaches ${T}^{\u22ba}$ as quickly as possible? During learning, the empirical transition matrix, $T(t)$, changes at each timestep $t$, based on transitions the animal has experienced. Define the total number of times that state ${\varphi}_{j}$ happened prior to time $t$ as ${n}_{j}(t)={\sum}_{{t}^{\prime}=1}^{\mathrm{\infty}}{\varphi}_{j}(t{t}^{\prime})$, and define the running count of transitions from state j to state i as ${c}_{ij}(t)={\sum}_{{t}^{\prime}=1}^{\mathrm{\infty}}{d}_{ij}(t{t}^{\prime})$. We want $J(t)={T}^{\u22ba}(t)$, which necessitates
Note that ${n}_{j}(t)={n}_{j}(t1)+{\varphi}_{j}(t1)$, and ${c}_{ij}(t)={c}_{ij}(t1)+{d}_{ij}(t)$, which gives us
Therefore, comparing with Equation 42, we can see that a learning rate ${\eta}_{j}=\frac{1}{{n}_{j}(t)}$ will let $J={T}^{\u22ba}$ as quickly as possible. We have defined $n$ in terms of the inputs $\mathit{\varphi}$ for this derivation, but in practice the adaptive learning rate as a function of $\mathit{x}$ works well with the RNNS update rule (which is also a function of $\mathit{x}$). Thus, we use the adaptive learning rate defined over $\mathit{x}$ in our combined learning rule for increased biological plausibility.
In its current form, the update equation assumes transitions across all history of inputs are integrated. In reality, there is likely some kind of memory decay. This can be implemented with a decay term $\lambda \in (0,1)$:
$\lambda $ determines the recency bias over the observed transitions that make up the $T$ estimate. The addition of $\lambda $ has the added benefit that it naturally provides a mechanism for learning rates to modulate over time. If $\lambda =1$, the learning rate can only monotonically decrease. If $\lambda <1$, the learning rate can become strong again over time if a state has not been visited in a while. This provides a mechanism for fast learning of new associations, which is useful for a variety of effects, including remapping.
Appendix 5
Endotaxis model and the successor representation
The learning rule and architecture of our model is similar to a hypothesized “endotaxis” model (Zhang et al., 2021). In the endotaxis model, neurons fire most strongly near a reward, allowing the animal to navigate up a gradient of neural activity akin to navigating up an odor gradient. The endotaxis model discovers the structure of an environment and can solve many tasks such as spatial navigation and abstract puzzles. We were interested in similarities between RNNS and the learning rules for endotaxis, in support of the idea that SRlike representations may be used by the brain for a broad range of intelligent behaviors. Here, we outline similarities and differences between the two model architectures.
The endotaxis paper (Zhang et al., 2021) uses Oja’s rule in an RNN with placelike inputs. The SR can also be learned with an Ojalike learning rule. Oja’s rule is typically written as (Oja, 1982):
If we assume that there is a temporal asymmetry to the potentiation term (e.g., potentiation is more STDPlike than Hebbian), then we have
We then solve for the steady state solution of this equation, when $\mathrm{\Delta}{J}_{ij}=0$:
where $\u27e8\cdot \u27e9$ indicates the timeaverage of some term. Assume that the plasticity rule does not use $\mathit{x}$ exactly, but instead uses $\mathit{\varphi}$ directly. Given that inputs are onehot encodings of the animal’s state at some time $t$, the expression becomes
If we assume $T$ is symmetric, $J={T}^{\u22ba}$. Alternatively, if we use presynaptic normalization as opposed to the standard postsynaptic normalization of Oja’s rule (i.e., index $j$ instead of $i$ in the denominator), we also have $J={T}^{\u22ba}$. Thus, the steady state activity of a RNN with this learning rule retrieves the SR, as shown in subsection 4.14.
Appendix 6
Independent normalization and successor features
If we assume the same Ojalike rule as in Appendix 5, we can also arrive at a similar interpretation in the successor feature case as in Equation 7. By solving for the steady state solution without any assumptions about the inputs $\mathit{\varphi}$, we get the following equation:
where diag is a function that retains only the diagonal of the matrix. This expression provides a useful way to contrast the learning rule used in RNNS with an Ojalike alternative. While RNNS normalizes by the full autocorrelation matrix, an Ojalike rule only normalizes by the diagonal of the matrix. This is the basis of our independent normalization model in Figure 4BC.
Appendix 7
Comparing alternate forms of normalizing the synaptic weight matrix
The antiHebbian term of the RNNS learning rule normalizes the synaptic weight matrix into exactly a transition probability matrix. We wanted to test how important it was to use this exact normalization and whether other forms of the synaptic weight matrix could yield similar results. We simulated representations that would arise from different normalization procedures. For these tests, we simulate a random walk on a circular track, as in Figure 2, for 10 minutes of simulation time. A model where the synaptic weight matrix exactly estimates the transition probability matrix (as in Equation 4) will give the SR matrix (Appendix 7—figure 1A).
We test a model where the normalization term for the synaptic weight matrix is removed. Thus, $J$ will be equal to the count of observed transitions, i.e. ${J}_{ij}$ is equal to the number of experienced transitions from state $j$ to state $i$. We will refer to this as a count matrix. Without normalization, the values in the count matrix will increase steadily over the course of the simulation. This quickly results in unstable dynamics from the weights of the matrix being too large (Appendix 7—figure 1B). A simple way to prevent instability (specifically, ensure the maximum eigenvalue of the synaptic weight matrix is below 1) is to use an additional scaling factor $\alpha $ over the weights of the matrix, such that $J$ is multiplied by $\frac{1}{\alpha \text{max}(J)}$. A careful choice of a scaling value can ensure network activity remains stable within this walk, although this is not a generalizable solution as different scaling values may be needed for different random walks and tasks. However, even with this modification the representations above are not sufficiently predictive compared to the original SR (the offdiagonal elements of the SR are not captured well), and the activity strength seems to be unevenly distributed across the states (Appendix 7—figure 1CD). It is likely that depressing all synapses by the same factor (similar to Fiete et al., 2010) does not correct for differences in occupancies. In other words, states that happen to be visited more by chance are likely to dominate the dynamics, even if the transition statistics are identical across all states.
Final, as a further test of different ways of parameterizing the synaptic weight matrix, we examine the steady state neural activity when the count matrix is instead scaled in a rowbyrow fashion (Appendix 7—figure 1E). Specifically, we divide each row $i$ of the count matrix by the maximum of row $i$ (and some global scaling factor to ensure stability). Note that this is in contrast to $T$, where each row is divided by its sum. This is closer to the SR matrix expected if the synaptic weight matrix estimates $T$. We see there is a slight unevenness early on in learning in the diagonal of the matrix (Appendix 7—figure 1E). However, given enough observed transitions, the predictive representation looks reasonable and quite similar to the SR matrix.
Overall, we see that there are likely other formulations of the synaptic weight matrix that can give a representation similar to the SR. The important ingredient for this to happen appears to be some type of rowdependent normalization that is, neurons should have their synaptic weights normalized independently of each other. This ensures that occupancy is not conflated with transition statistics.
Data availability
The current manuscript is a computational study, so no data have been generated for this manuscript. Modelling code is publicly available on GitHub: https://github.com/chingf/srproject (copy archived at swh:1:rev:43320e9b8c15927c67849f768d2a9bf17f68a0ea).

Dryad Digital RepositoryNeural representations of space in the hippocampus of a foodcaching bird.https://doi.org/10.5061/dryad.pg4f4qrp7
References

Metaplasticity: the plasticity of synaptic plasticityTrends in Neurosciences 19:126–130.https://doi.org/10.1016/s01662236(96)80018x

Metaplasticity: tuning synapses and networks for plasticityNature Reviews. Neuroscience 9:387–390.https://doi.org/10.1038/nrn2356

Synaptic plasticity as Bayesian inferenceNature Neuroscience 24:565–571.https://doi.org/10.1038/s41593021008095

Characteristics of random nets of analog neuronlike elementsIEEE Transactions on Systems, Man, and Cybernetics SMC2:643–657.https://doi.org/10.1109/TSMC.1972.4309193

Sequence memory in the hippocampalentorhinal regionJournal of Cognitive Neuroscience 32:2056–2070.https://doi.org/10.1162/jocn_a_01592

Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell typeThe Journal of Neuroscience 18:10464–10472.

Conjunctive input processing drives feature selectivity in hippocampal CA1 neuronsNature Neuroscience 18:1133–1142.https://doi.org/10.1038/nn.4062

A model of spatial MAP formation in the hippocampus of the ratNeural Computation 8:85–93.https://doi.org/10.1162/neco.1996.8.1.85

Prospective coding by spiking neuronsPLOS Computational Biology 12:e1005003.https://doi.org/10.1371/journal.pcbi.1005003

Predictive representations in hippocampal and prefrontal hierarchiesThe Journal of Neuroscience 42:299–312.https://doi.org/10.1523/JNEUROSCI.132721.2021

Prediction, cognition and the brainFrontiers in Human Neuroscience 4:25.https://doi.org/10.3389/fnhum.2010.00025

Mirrored STDP implements autoencoder learning in a network of spiking neuronsPLOS Computational Biology 11:e1004566.https://doi.org/10.1371/journal.pcbi.1004566

What’s new with the amnesic patient h.m.?Nature Reviews. Neuroscience 3:153–160.https://doi.org/10.1038/nrn726

ConferenceAutumnsynth: synthesis of reactive programs with structured latent stateAdvances in Programming Languages and Neurosymbolic Systems Workshop; 2021.

BookTheoretical Neuroscience: Computational and Mathematical Modeling of Neural SystemsMassachusetts, United States: The MIT Press.

Neurobiological successor features for spatial navigationHippocampus 30:1347–1355.https://doi.org/10.1002/hipo.23246

Entorhinal cortex of the rat: topographic organization of the cells of origin of the perforant path projection to the dentate gyrusThe Journal of Comparative Neurology 398:25–48.

The extent and drivers of gender imbalance in neuroscience reference ListsNature Neuroscience 23:918–926.https://doi.org/10.1038/s415930200658y

The recall of events through the learning of associations between their partsProceedings of the Royal Society of London. Series B, Biological Sciences 194:375–402.https://doi.org/10.1098/rspb.1976.0084

The successor representation and temporal contextNeural Computation 24:1553–1568.https://doi.org/10.1162/NECO_a_00282

Global structure, robustness, and modulation of neuronal modelsThe Journal of Neuroscience 21:5229–5238.https://doi.org/10.1523/JNEUROSCI.211405229.2001

Neuromodulation: acetylcholine and memory consolidationTrends in Cognitive Sciences 3:351–359.https://doi.org/10.1016/S13646613(99)013650

The role of acetylcholine in learning and memoryCurrent Opinion in Neurobiology 16:710–715.https://doi.org/10.1016/j.conb.2006.09.002

Motoneuron excitability: the importance of neuromodulatory inputsClinical Neurophysiology 120:2040–2054.https://doi.org/10.1016/j.clinph.2009.08.009

A study on the status of women faculty in science at mit in AIP conference proceedingsAmerican Institute of Physics 628:103–106.https://doi.org/10.1063/1.1505288

Mechanisms of heterosynaptic metaplasticityPhilosophical Transactions of the Royal Society of London. Series B, Biological Sciences 369:1633.https://doi.org/10.1098/rstb.2013.0148

How environmental movement constraints shape the neural code for spaceCognitive Processing 22:97–104.https://doi.org/10.1007/s10339021010452

Comparison of spatial firing characteristics of units in dorsal and ventral hippocampus of the ratThe Journal of Neuroscience 14:7347–7356.

ConferenceContinual reinforcement learning with complex synapsesProceedings of the 35th International Conference on Machine Learning. pp. 2497–2506.

ConferenceLearning a biologically plausible linear controller for nonlinear systemsAPS March Meeting 2022.

LongTerm synaptic plasticity in hippocampal interneuronsNature Reviews. Neuroscience 8:687–699.https://doi.org/10.1038/nrn2207

BookNonnormality in neural networksIn: Jalali B, Kitayama K, editors. AI and Optical Data Sciences III. San Francisco, United States: SPIE. pp. 204–227.https://doi.org/10.1117/12.2613472

Recurrent synapses and circuits in the CA3 region of the hippocampus: an associative networkFrontiers in Cellular Neuroscience 7:262.https://doi.org/10.3389/fncel.2013.00262

Prediction, sequences and the hippocampusPhilosophical Transactions of the Royal Society of London. Series B, Biological Sciences 364:1193–1201.https://doi.org/10.1098/rstb.2008.0316

Formation and maintenance of neuronal assemblies through synaptic plasticityNature Communications 5:1–12.https://doi.org/10.1038/ncomms6319

Functional anatomy of hippocampal seizuresProgress in Neurobiology 37:1–82.https://doi.org/10.1016/03010082(91)90011o

Levels of biological plausibilityPhilosophical Transactions of the Royal Society of London. Series B, Biological Sciences 376:1815.https://doi.org/10.1098/rstb.2019.0632

Variability, compensation and homeostasis in neuron and network functionNature Reviews. Neuroscience 7:563–574.https://doi.org/10.1038/nrn1949

Multiple models to capture the variability in biological neurons and networksNature Neuroscience 14:133–138.https://doi.org/10.1038/nn.2735

Interactions between location and task affect the spatial and directional firing of hippocampal neuronsThe Journal of Neuroscience 15:7079–7094.

ReportFrom understanding computation to understanding neural circuitryMIT Artifical Intelligence Laboratory.

BookSimple memory: a theory for archicortexIn: Vaina L, editors. From the Retina to the Neocortex. Springer. pp. 59–128.https://doi.org/10.1007/9781468467758_5

Prioritized memory access explains planning and hippocampal replayNature Neuroscience 21:1609–1617.https://doi.org/10.1038/s415930180232z

Hippocampal synaptic enhancement and information storage within a distributed memory systemTrends in Neurosciences 10:408–415.https://doi.org/10.1016/01662236(87)900117

Excitatory synaptic interactions between CA3 neurones in the guineapig hippocampusThe Journal of Physiology 373:397–418.https://doi.org/10.1113/jphysiol.1986.sp016055

The successor representation in human reinforcement learningNature Human Behaviour 1:680–692.https://doi.org/10.1038/s4156201701808

Learning structures: predictive representations, replay, and generalizationCurrent Opinion in Behavioral Sciences 32:155–166.https://doi.org/10.1016/j.cobeha.2020.02.017

Attentive scanning behavior drives onetrial potentiation of hippocampal place fieldsNature Neuroscience 17:725–731.https://doi.org/10.1038/nn.3687

The firing of hippocampal place cells predicts the future position of freely moving ratsThe Journal of Neuroscience 9:4101–4110.

Neuromodulation of neurons and synapsesCurrent Opinion in Neurobiology 29:48–56.https://doi.org/10.1016/j.conb.2014.05.003

BookSexual Harassment of Women: Climate, Culture, and Consequences in Academic Sciences, Engineering, and MedicineWashington, United States: National Academies of Sciences.

A simplified neuron model as a principal component analyzerJournal of Mathematical Biology 15:267–273.https://doi.org/10.1007/BF00275687

Synaptic encoding of temporal ContiguityFrontiers in Computational Neuroscience 7:32.https://doi.org/10.3389/fncom.2013.00032

Timing is not everything: neuromodulation opens the STDP gateFrontiers in Synaptic Neuroscience 2:146.https://doi.org/10.3389/fnsyn.2010.00146

Blind nonnegative source separation using biological neural networksNeural Computation 29:2925–2954.https://doi.org/10.1162/neco_a_01007

Why do similarity matching objectives lead to hebbian/antihebbian networks?Neural Computation 30:84–124.https://doi.org/10.1162/neco_a_01018

Memory deficit produced by bilateral lesions in the hippocampal zoneA.M.A. Archives of Neurology and Psychiatry 79:475–497.https://doi.org/10.1001/archneurpsyc.1958.02340050003001

Longaxis specialization of the human hippocampusTrends in Cognitive Sciences 17:230–240.https://doi.org/10.1016/j.tics.2013.03.005

Similar network activity from disparate circuit parametersNature Neuroscience 7:1345–1352.https://doi.org/10.1038/nn1352

BookTextura Del Sistema Nervioso Del Hombre y de Los VertebradosImprenta y Librería de Nicolás Moya Madrid.

Subgoal and goalrelated reward prediction errors in medial prefrontal cortexJournal of Cognitive Neuroscience 31:8–23.https://doi.org/10.1162/jocn_a_01341

Beitrag Zur histologie Der ammonshornformationArchiv Für Mikroskopische Anatomie 39:611–632.https://doi.org/10.1007/BF02961541

Loss of recent memory after bilateral hippocampal lesionsJournal of Neurology, Neurosurgery, and Psychiatry 20:11–21.https://doi.org/10.1136/jnnp.20.1.11

Chaos in random neural networksPhysical Review Letters 61:259–262.https://doi.org/10.1103/PhysRevLett.61.259

The hippocampus as a predictive MAPNature Neuroscience 20:1643–1653.https://doi.org/10.1038/nn.4650

Functional organization of the hippocampal longitudinal axisNature Reviews. Neuroscience 15:655–669.https://doi.org/10.1038/nrn3785

Review: hippocampal sclerosis in epilepsy: a neuropathology reviewNeuropathology and Applied Neurobiology 40:520–543.https://doi.org/10.1111/nan.12150

Efficient and robust coding in heterogeneous recurrent networksPLOS Computational Biology 17:e1008673.https://doi.org/10.1371/journal.pcbi.1008673

The citation diversity statement: a practice of transparency, a way of lifeTrends in Cognitive Sciences 24:669–672.https://doi.org/10.1016/j.tics.2020.06.009
Decision letter

Srdjan OstojicReviewing Editor; École Normale Supérieure Paris, France

Timothy E BehrensSenior Editor; University of Oxford, United Kingdom

Stefano RecanatesiReviewer; University of Washington, United States

Arthur JulianiReviewer
Our editorial process produces two outputs: (i) public reviews designed to be posted alongside the preprint for the benefit of readers; (ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.
Decision letter after peer review:
Thank you for submitting your article "Neural learning rules for generating flexible predictions and computing the successor representation" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Timothy Behrens as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Stefano Recanatesi (Reviewer #1); Arthur Juliani (Reviewer #2); Srdjan Ostojic (Reviewer #3).
The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.
Main Comments:
1) The form of the plasticity rule in Equation 4 is motivated by the requirement that synaptic weights encode a properly normalised transition probability matrix (lines 9296). But why is the normalisation important? What would change if synaptic weights were simply monotonic functions of transition probabilities, without normalisation? Presumably that would allow for a broader range of plasticity rules.
2) As the results of the paper strongly rely on the normalizing term in Equation 4. One of the reviewers suggests potentially moving upfront part of the discussion of this term, and enlarging the paragraph that discusses the biological plausibility of this specific term. Clearly laying out, for the nonexpert reader, why it is biologically plausible compared to other learning rules. Also consider moving the required material to establish the novelty of such term: a targeted review of the relevant literature (current lines 358366 and 413433). This would allow the reader to understand immediately the significance and relative novelty of such term. For example, this reviewer personally wondered while reading the paper of how different was such term from the basic idea of Fiete et al., Neuron 2010 (DOI 10.1016/j.neuron.2010.02.003).
3) Related to the first point, the text insists on the fact that \γ is not encoded in the synaptic weights (eg line 89). Again, it is not entirely clear why this is important and justified, since γ is an adhoc factor in Equation 2. Presumably the proper normalisation of γ relies on the normalisation of J discussed above? It seems that this constraint could be relaxed.
4) As a consequence of the body of the text being devoted to the analysis of the design choices behind the proposed model, a relatively smaller portion of the work involves direct comparisons with neural data. In these comparisons, while it is apparent that there is a reasonable match between the proposed model and the empirical data, it is difficult to interpret these results. This is because it is unclear what should be expected of a good or bad model given the data being analyzed (TD error and KL divergence), and reasonable baselines to compare against are not presented outside of the traditional TD algorithm, which is shown to be comparable to the proposed RNN based method in a number of cases.
5) It would be useful to have a "limitations" paragraph in the discussion clearly outlining what this learning rule couldn't achieve. For example, Stachenfeld et al., Nat.Neuro. have many examples where the SR is deployed. Does the learning rule suggested by the authors would always work across the board, or are there limitations that could be highlighted where the framework suggested would not work well. No need to perform more experiments/simulations but simply to share insight regarding the results and the capability of the proposed learning rule.
Other comments/suggestions:
– Page 1: The introduction motivates this work with a discussion of hippocampal memory (storage and retrieval), but the work focuses on the SR which is inherently prospective. The first paragraph of the text could be revised to better make this connection beyond simply stating that the hippocampus is involved in both memory and future prediction.
– Page 2: The end of the introduction would be stronger if the motivation for an RNNs usage was tied to the literature on the known recurrent dynamics of the hippocampus. See for example: https://www.frontiersin.org/articles/10.3389/fncel.2013.00262/full
– Page 6: It is not clear the extent to which the FFTD model differs from a canonical tabular SR algorithm or linear SF algorithm. My understanding is that it is the same, but the presentation in Figure 1i for example makes this somewhat unclear.
– Pages 6 – 11: it may be of benefit to more strongly support the various modifications to the model with connections to known or hypothesized hippocampal neural dynamics.
– Page 14: It states that "We discretize the arena into a set of states and encode each state as a randomly drawn feature ϕ." If I understand correctly, these features are not completely random, and instead follow the distribution described in Section 2.5. As it currently reads, it seems that these features might be drawn from a uniform random distribution, which would be misleading.
– Page 14: In Section 2.6 there is an assumption that a certain level of TD error corresponds to good performance. It is not clear what should objectively be considered a reasonable TD error. This is especially difficult to interpret in the case where both the RNNS and FFTD models display comparable performance. Is there perhaps some other baseline you would expect to perform considerably worse?
– Page 17: In Figure 4 it is somewhat confusing that the KL divergence (subplots G and I) has reversed shading (dark for low values) compared to the other subplots. It would be easier to interpret these graphs if their color coding was more consistent.
– Page 18: Similar to the difficulty of interpreting the TD error results, it is not clear what a "good" or "bad" KL divergence from the neural data would be. Any hypotheses on how to ground the numbers provided here would help to improve the quality of the results.
– Page 20: It is mentioned that the predictive timescale may be a separate gain term which the hippocampus takes as input, but there is evidence that different regions of the hippocampus seem to operate on different timescales. See for example: https://www.jneurosci.org/content/42/2/299.abstract. Is there a way to reconcile these ideas?
– Page 23: Section 4.5 describes the procedure for learning the parameters of the weight update rule as CMAES. Mentioning the fact that an evolutionary algorithm is used for learning these weights would help to make Section 2.3 more clear.
– Figures 5DE and similar supplementary figures: if there is a parameter region that is unexplored then the color used for such region should be outside of the colormap. One of the reviewers suggests replacing white with gray for such region in these figures.
– Line 173: the text makes the distinction between an "SRlike" representation, and an "exact SR". What is the difference? Why is it important to have an exact of the SR in the neural activity, rather then eg a monotonic encoding of the SR?
– The RNN described in Equation 2 is not of the standard form (the nonlinearity is applied after the connectivity matrix, ie f(J x) instead of Jf(x)). Is this detail important? If not, why not use the more standard form to avoid confusion?
– A line of work in the Fusi lab has examined plasticity rules that lead to the encoding of transition probabilities (eg Fusi et al., Neuron 2007). In particular, a paper by the reviewing editor (Ostojic and Fusi Front Comp Neuro 2013) examined the encoding of transition probabilities using plasticity rules that look similar to this manuscript. This is mentioned just for information, the authors should decide if those papers are relevant.
– Figures 5DE and similar supplementary figures: if there is a parameter region that is unexplored then the color used for such region should be outside of the colormap. One of the reviewers suggests replacing white with gray for such region in these figures.
https://doi.org/10.7554/eLife.80680.sa1Author response
Main Comments:
1) The form of the plasticity rule in Equation 4 is motivated by the requirement that synaptic weights encode a properly normalised transition probability matrix (lines 9296). But why is the normalisation important? What would change if synaptic weights were simply monotonic functions of transition probabilities, without normalisation? Presumably that would allow for a broader range of plasticity rules.
The reviewer makes an interesting point that a range of possible rules may yield useful representations even if they do not learn the transition probability matrix exactly. We tested these ideas (see below) and found that normalization is generally important for maintaining stable dynamics in the recurrent network. Many forms of normalization can learn predictive representations similar to the SR, as long as the normalization is performed across rows of the weight matrix independently. We have added a few sentences of text (lines 107116) and a supplementary figure summarizing and showing these results. The details of our additional analyses and the text added to the manuscript are given below.
We constructed representations using different normalization procedures. For these tests, we simulated a random walk on a circular track, as in Appendix 7figure 1B, for 10 minutes of simulation time.
If the synaptic weight matrix estimates the normalized transition probability matrix (as in equation 4), the resulting SR matrix over the course of the walk is as Appendix 7figure 1A.
As an initial test of the role of normalization, we did the same simulation, removing normalization. Thus, $J$ will be equal to the count of observed transitions, i.e. $J_{ij}$ is equal to the number of experienced transitions from state $j$ to state $i$. We refer to this as a count matrix. Note that, without normalization, the values in the count matrix increase steadily over the course of the simulation. This quickly results in unstable dynamics due to the weights of the matrix being too large. We can quantify this instability by plotting the maximum eigenvalue of the weight matrix, where an eigenvalue >= 1 indicates instability (Sompolinsky et al., 1988), Appendix 7figure 1B.
A simple way to prevent instability is to scale the weights of the matrix by a constant factor such that the dynamics are in the stable regime (specifically, ensuring that the maximum eigenvalue of the synaptic weight matrix is below 1). A careful choice of a scaling value can ensure network activity remains stable within this walk, although this is not a generalizable solution as different scaling values will be needed for different random walks and tasks.
Informed by Appendix 7figure 1C, we chose a scaling factor of 1.75. and tested what neural representations look like throughout the walk.
Compared to the ground truth SR, the representations above are not very predictive (the offdiagonal elements of the SR are not captured well), and the activity strength seems to be unevenly distributed across states. For instance, there is more activity at state 8 than other states. It is likely that depressing all synapses by the same factor (similar in flavor to Fiete et al., 2010) does not correct for differences in occupancies. In other words, states that happen to be visited more by chance are likely to dominate the dynamics, even if the transition statistics are identical across all states (Appendix 7figure 1D).
As a further test of different ways of parameterizing the synaptic weight matrix, we can instead scale the count matrix in a rowbyrow fashion. Specifically, we divide each row $i$ of the count matrix by the maximum of row $i$ (and some global scaling factor to ensure stability). Note that this is in contrast to $T$, where each row is divided by its sum. The resulting steady state activity matrices (Appendix 7figure 1E).
This is closer to the SR matrix expected if the synaptic weight matrix estimates $T$. There is a slight unevenness early on in learning the diagonal of the matrix. However, given enough observed transitions, the predictive representation looks reasonable and quite similar to the SR matrix.
Overall, we see that the intuition of the reviewer is correct: there are likely other formulations of the synaptic weight matrix that can give a representation similar to the SR. The important ingredient for this to happen appears to be some form of rowdependent normalization—that is, neurons should have their synaptic weights normalized independently of each other. This ensures that occupancy is not conflated with transition statistics. We added the figures in appendix 7. We also added the following passage to the introduction of the plasticity rule in Results section 2.2 (and use the suggested reference to Fiete 2010):
“The second term in equation 4 is a form of synaptic depotentiation. Depotentiation has been hypothesized to be broadly useful for stabilizing patterns and sequence learning [37, 49], and similar inhibitory effects are known to be elements of hippocampal learning [50, 51]. In our model, the depotentiation term in equation 4 imposes local antiHebbian learning at each neuron—that is, each column of $J$ is normalized independently. This normalizes the observed transitions from each state by the number of visits to that state, such that transition statistics are correctly captured. We note, however, that other ways of columnnormalizing the synaptic weight matrix may give similar representations (Figure S1).”
2) As the results of the paper strongly rely on the normalizing term in Equation 4. one of the reviewers suggests potentially moving upfront part of the discussion of this term, and enlarging the paragraph that discusses the biological plausibility of this specific term. Clearly laying out, for the nonexpert reader, why it is biologically plausible compared to other learning rules. Also consider moving the required material to establish the novelty of such term: a targeted review of the relevant literature (current lines 358366 and 413433). This would allow the reader to understand immediately the significance and relative novelty of such term. For example, this reviewer personally wondered while reading the paper of how different was such term from the basic idea of Fiete et al., Neuron 2010 (DOI 10.1016/j.neuron.2010.02.003).
The reviewer points out that more context and clarity around the plasticity rule would be useful, particularly since an understanding of the plasticity rule is integral to the paper.
We would like to clarify that, although the RNNS is more biologically plausible than the FFTD learning rule, there is likely additional biological complexity/realism that can be added to the RNNS learning rule. We wanted to find the simplest rule that could capture the essence of the SR, which is why we focused on the particular form of the learning rule we used.
Aside from plausibility, a key aspect of the RNNS normalizing term (as discussed in Main Comment 1) is that it independently normalizes each column of the synaptic weight matrix. This is in contrast, say, to the Fiete et al., 2010 paper (which has a global depressive term) and other similar plasticity rules. We directly tested the effect of different normalizations on the RNN representations (see response to Main Comment 1), and find that columnspecific normalization is important for capturing transition statistics.
To make these subtle points more clear to the reader, we added additional sentences about biological realism and other forms of learning rules in the section where the learning rule is introduced:
“Crucially, the update rule (equation 4) uses information local to each neuron (Figure 1h), an important aspect of biologically plausible learning rules.
The second term in equation 4 is a form of synaptic depotentiation. Depotentiation has been hypothesized to be broadly useful for stabilizing patterns and sequence learning [37, 49], and similar inhibitory effects are known to be elements of hippocampal learning [50, 51]. In our model, the depotentiation term in equation 4 imposes local antiHebbian learning at each neuron that is, each column of $J$ is normalized independently. This normalizes the observed transitions from each state by the number of visits to that state, such that transition statistics are correctly captured. We note, however, that other ways of columnnormalizing the synaptic weight matrix may give similar representations (Figure S1).”
3) Related to the first point, the text insists on the fact that \γ is not encoded in the synaptic weights (eg line 89). Again, it is not entirely clear why this is important and justified, since γ is an adhoc factor in Equation 2. Presumably the proper normalisation of γ relies on the normalisation of J discussed above? It seems that this constraint could be relaxed.
The reviewer is correct that factorizing \γ as a separate factor from the synaptic weights (J) is a notational choice. We include the \γ as a factor distinct from the synaptic strengths for several reasons. The first is for consistency with previous literature. In the SR literature, \γ is factorized out of normalized transition matrices. Similarly, in RNN literature, it is typical to analyze a global gain factor (g), which determines the operating regime of the network (Sompolinsky et al., 1988). Our second is mechanistic. We interpret \γ as a measure of the gain of the network units, that is, as a physiological property of the neurons. The synaptic matrix J, on the other hand, measures the strengths of synapses. Keeping them separate allows for more flexibility, which leads to the third reason. Treating \γ as a separate factor allows the network to retrieve successor representations of different predictive strengths. Importantly, this dynamic predictive ability is achieved without changing any synaptic weights and without additional learning. In other words, a separate \γ allows us to decouple the learning and retrieval processes, providing more flexibility in using the SR.
To make this point more clearly in the text, we added the following lines to explain this rationale:
“Here, the factor $\γ$ represents the gain of the neurons in the network, which is factored out of the synaptic strengths characterized by J. Thus, $\γ$ is an independently adjustable factor that can flexibly control the strength of the recurrent dynamics (see [46]). A benefit of this flexibility is that the system can retrieve successor representations of varying predictive strengths by modulating the gain factor $\γ$. In this way, the predictive horizon can be dynamically controlled without any additional learning required.”
4) As a consequence of the body of the text being devoted to the analysis of the design choices behind the proposed model, a relatively smaller portion of the work involves direct comparisons with neural data. In these comparisons, while it is apparent that there is a reasonable match between the proposed model and the empirical data, it is difficult to interpret these results. This is because it is unclear what should be expected of a good or bad model given the data being analyzed (TD error and KL divergence), and reasonable baselines to compare against are not presented outside of the traditional TD algorithm, which is shown to be comparable to the proposed RNN based method in a number of cases.
The reviewer suggests conducting control analyses to help with interpretability of the TD error and KL divergence results. Specifically, they suggest comparing the performance of the RNN and FF network to good and bad models.
As an example of a “bad” model, we calculate the TD error and KL divergence of a feedforward network with weights randomly drawn from the distribution of weights of the FFTD model at the end of learning. We call this the Shuffle model, and it is representative of a model without learned structure but with a similar magnitude of weights as the FFTD model. Specifically, the shuffle model has much higher TD error than the RNN or FF network.
As an example of a “good” model, we calculate the KL divergence between randomly split halves of the dataset from Payne et al., (2021). Specifically, we compare the place field statistics of a random half of the neurons from Payne et al., (2021) with another random half. We calculate the KL divergence between the distributions calculated from each random half. This is repeated 500 times. We call this “Data” in the plot, and it is representative of a lower bound on KL divergence. Intuitively, it should not be possible to fit the data of Payne et al., as well as the dataset itself can. We compare this KL divergence to the KL divergence between each model and the neural data.
The KL divergence of the shuffle model is quite close to the FF network, suggesting that most of the “placelike” qualities of the FF network are more likely a reflection of the input features being “placelike” than the learned weights constructing these place fields. Place fields from the RNNS model are more similar to neural data than the FFTD or the Shuffle models. The analysis on split halves of the Payne et al., dataset shows a low KL divergence (specifically, around 0.12) that is much smaller than any of the models.
We add the two plots into the supplementary material (Figure S). Additionally, we report these results in the following sections of the main text:
In the section introducing TD error:
“We want to compare the TD loss of RNNS to that of a nonbiological model designed to minimize TD loss… Both models show a similar increase in TD loss as $\γ_R$ increases, although the RNNS has a slightly lower TD loss at high $\γ$ than the FFTD model. Both models perform substantially better than a random network with weights of comparable magnitude (Figure S5d).”
In the section introduced summed KL divergence:
“We combined the KL divergence of both these distributions to find the parameter range in which the RNNS best fits neural data (Figure 6g). This optimal parameter range occurs when inputs have a spatial correlation of $\σ \approx 8.75$ cm and sparsity $\approx 0.15$. We note that the splithalf noise floor of the dataset of Payne et al., is a KL divergence of $0.12$ bits (Figure S6E).
…
We next tested whether the neural data was better fit by representations generated by RNNS or the FFTD model. Across all parameters of the input features, despite having similar TD loss (Figure 5de), the FFTD model has much higher divergence from neural data (Figure 6gi, Figure S6), similar to a random feedforward network (Figure S6E).”
5) It would be useful to have a “limitations” paragraph in the discussion clearly outlining what this learning rule couldn’t achieve. For example, Stachenfeld et al., Nat.Neuro. have many examples where the SR is deployed. Does the learning rule suggested by the authors would always work across the board, or are there limitations that could be highlighted where the framework suggested would not work well. No need to perform more experiments/simulations but simply to share insight regarding the results and the capability of the proposed learning rule.
The reviewer suggests adding further conceptual clarity to the discussion by giving insight into the limitations of the model and its differences from the Stachenfeld et al., simulations. We anticipate that the RNNS can capture the results seen in Stachenfeld et al., since the SR learned by the algorithm in Stachenfeld et al., is identical to the representation learned by the RNNS in the onehot case.
However, using a recurrent architecture does impose limitations on how the learning rule is structured and how the network can be used. In particular, care must be given to avoid instability in the network due to buildup of recurrent activity. We proposed a ‘learning’ and ‘retrieval’ mode in the network precisely to control such instabilities. Furthermore, the recurrency of the network means that errors in transition structure can compound across a long horizon of prediction. This is especially problematic in the case of nononehot features, where greater errors in transition estimation are likely for more densely coded features.
We added a limitations paragraph in the discussion focusing on the limitations of using a recurrent network model (as opposed to a feedforward network). We also further tie these observations to biological evidence:
“There are inherent limitations to the approach of using a recurrent network to estimate the SR. For instance, network dynamics can be prone to issues of instability due to the recurrent buildup of activity. To prevent this instability, we introduce two different modes of operation, ``learning’’ and ``retrieval’’. An additional limitation is that errors in the estimated onestep transition can propagate over the course of the predictive rollout. This is especially problematic if features are more densely coded or more correlated, which makes onestep transition estimations more difficult. These insights into vulnerabilities of a recurrent network have interesting parallels in biology. Some hippocampal subfields are known to be highly recurrent [92, 93, 94, 95]. This recurrency has been linked to the propensity of the hippocampus to enter unstable regimes, such as those that produce seizures [96, 97, 98, 99]. It remains an open question how a healthy hippocampus maintains stable activity, and to what extent the findings in models such as ours can suggest biological avenues to tame instability.”
Other comments/suggestions:
– Page 1: The introduction motivates this work with a discussion of hippocampal memory (storage and retrieval), but the work focuses on the SR which is inherently prospective. The first paragraph of the text could be revised to better make this connection beyond simply stating that the hippocampus is involved in both memory and future prediction.
The reviewer makes an Important point, and in fact the connection between episodic memory and predictive maps in the hippocampus is an active area of research in the hippocampus field. We have edited the first paragraph of the introduction to better explain and flesh out the hypothesized connections between predictive coding in the hippocampus and its function in memory:
“To learn from the past, plan for the future, and form an understanding of our world, we require memories of personal experiences. These memories depend on the hippocampus for formation and recall [1, 2, 3], but an algorithmic and mechanistic understanding of memory formation and retrieval in this region remains elusive. From a computational perspective, a key function of memory is to use past experiences to inform predictions of possible futures [4, 5, 6, 7]. This suggests that hippocampal memory is stored in a way that is particularly suitable for forming predictions.”
– Page 2: The end of the introduction would be stronger if the motivation for an RNNs usage was tied to the literature on the known recurrent dynamics of the hippocampus. See for example: https://www.frontiersin.org/articles/10.3389/fncel.2013.00262/full
https://www.frontiersin.org/articles/10.3389/fncel.2013.00262/full
We add the following sentences in the introduction to further motivate the usage of RNNs with known hippocampal anatomy/dynamics, adding in the suggested reference:
“A promising direction towards such a neural model of the SR is to use the dynamics of a recurrent neural network (RNN) to perform SR computations [39, 40]. An RNN model is particularly attractive as the hippocampus is highly recurrent, and its connectivity patterns are thought to support associative learning and recall [41, 42, 43, 44]. However, an RNN model of the SR has not been tied to neural learning rules that support its operation and allow for testing of specific hypotheses.”
– Page 6: It is not clear the extent to which the FFTD model differs from a canonical tabular SR algorithm or linear SF algorithm. My understanding is that it is the same, but the presentation in Figure 1i for example makes this somewhat unclear.
Yes, you’re correct that the FFTD model is exactly the linear SR/SF algorithm. We’ve added clarifying sentences in the section introducing the FFTD model to emphasize this:
“As an alternative to the RNNS model, we consider the conditions necessary for a feedforward neural network to compute the SR. Under this architecture, the $M$ matrix must be encoded in the weights from the input neurons to the hidden layer neurons (Figure 1g). This can be achieved by updating the synaptic weights with a temporal difference (TD) learning rule, the standard update used to learn the SR in the usual algorithm… The FFTD implements the canonical SR algorithm.”
– Pages 6 – 11: it may be of benefit to more strongly support the various modifications to the model with connections to known or hypothesized hippocampal neural dynamics.
The reviewer suggests motivating the modifications to the model by connecting to known biological mechanisms. We currently make these connections in the Discussion section. To make these points earlier in the paper, we summarize some of these points made in the discussion and put them earlier in the paper. Specifically, we make the following additions:
In the section introducing the antiHebbian term:
“The second term in equation 4 is a form of synaptic depotentiation. Depotentiation has been hypothesized to be broadly useful for stabilizing patterns and sequence learning [37, 49], and similar inhibitory effects are known to be elements of hippocampal learning [50, 51].”
In the section introducing the adaptive learning rate:
“If the learning rate of the outgoing synapses from each neuron $j$ is inversely proportional to ${n}_{j}(\eta =\frac{1}{{n}_{j}(t)})$, the update equation quickly normalizes the synapses to maintain a valid transition probability matrix (Supplementary Notes 4). Modulating synaptic learning rates as a function of neural activity is consistent with experimental observations of metaplasticity [56, 57, 58]”
In the section adding a nonlinearity into network dynamics:
“One way to tame this instability is to add a saturating nonlinearity into the dynamics of the network. This is a feature of biological neurons that is often incorporated in models to prevent unbounded activity [60]. Specifically, instead of assuming the network dynamics are linear ($f$ is the identity function in equation 2), we add a hyperbolic tangent into the dynamics equation.”
– Page 14: It states that“"We discretize the arena into a set of states and encode each state as a randomly drawn feature ϕ”" If I understand correctly, these features are not completely random, and instead follow the distribution described in Section 2.5. As it currently reads, it seems that these features might be drawn from a uniform random distribution, which would be misleading.
Thank you for pointing this ou— it is true that the features in Section 2.6 are constructed the same as in Section 2.5. We’ve made this more explicitly clear in Section 2.6:
“We discretize the arena into a set of states and encode each state as in Section $2.5$.”
– Page 14: In Section 2.6 there is an assumption that a certain level of TD error corresponds to good performance. It is not clear what should objectively be considered a reasonable TD error. This is especially difficult to interpret in the case where both the RNNS and FFTD models display comparable performance. Is there perhaps some other baseline you would expect to perform considerably worse?
This comment is already raised and addressed in Main Comment 4.
– Page 17: In Figure 4 it is somewhat confusing that the KL divergence (subplots G and I) has reversed shading (dark for low values) compared to the other subplots. It would be easier to interpret these graphs if their color coding was more consistent.
Thanks for the clarifying suggestion. We reversed the color map for KL divergence.
– Page 18: Similar to the difficulty of interpreting the TD error results, it is not clear what a“"goo”" or“"ba”" KL divergence from the neural data would be. Any hypotheses on how to ground the numbers provided here would help to improve the quality of the results.
This comment is already raised and addressed in Main Comment 4.
– Page 20: It is mentioned that the predictive timescale may be a separate gain term which the hippocampus takes as input, but there is evidence that different regions of the hippocampus seem to operate on different timescales. See for example: https://www.jneurosci.org/content/42/2/299.abstract. Is there a way to reconcile these ideas?
The reviewer points out an interesting hippocampal finding that is not obviously explained in the RNNS model: some anatomical axes of the hippocampal formation appear to contain a continuum of predictive timescales in their neural activity (Dolorfo 1998, Brun 2008). This is a wellsupported finding with interesting functional implications, and thus is worth discussing and addressing in the paper.
One way to model an anatomical gradient of predictive timescales is to use a series of RNNS systems. Each of these systems would have a different $\γ$ values that is used during retrieval. Thus, despite these systems receiving the same feature inputs, each network can estimate the state of the animal across different timescales.
Alternatively, the gradient in timescales or granularity could exist on the input level. As in the first hypothesis, we can assume a series of RNNS systems, except all systems utilize the same $\γ$ value during retrieval. If each system receives inputs that encode different granularities of the animal’s state space (in a spatial example: perhaps one set of inputs uses a state space that divides the arena into quadrants, while another set of inputs uses a state space that divides the arena into a 10x10 grid), then each RNNS network will naturally develop representations across a continuum of scales.
Both these hypotheses can be functionally useful as a way to learn hierarchical structure and use that information for planning.
We choose to emphasize the first hypothesis (a gradient of $\γ$ values), and summarize this idea by add the following sentences into the discussion paragraph on flexible $\γ$:
“The idea that the hippocampus might compute the SR with flexible $\γ$ could help reconcile recent results that hippocampal activity does not always match high$\γ$ SR [79, 80]. Additionally, flexibility in predictive horizons could explain the different timescales of prediction observed across the anatomical axes of the hippocampus and entorhinal cortex [88, 89, 90, 91, 92]. Specifically, a series of successor networks with different values of γ used in retrieval could establish a gradient of predictive timescales. Functionally, this may allow for learning hierarchies of state structure and could be useful for hierarchical planning [93, 94, 95].”
– Page 23: Section 4.5 describes the procedure for learning the parameters of the weight update rule as CMAES. Mentioning the fact that an evolutionary algorithm is used for learning these weights would help to make Section 2.3 more clear.
We added additional sentences in Section 2.3 clarifying how parameters were learned:
“To systematically explore the space of plasticity kernels that can be used to learn the SR, we performed a grid search over the sign and the time constants of the pre > post and post > pre sides of the plasticity kernels. For each fixed sign and time constant, we used an evolutionary algorithm to learn the remaining parameters that determine the plasticity kernel.”
– figures 5DE and similar supplementary figures: if there is a parameter region that is unexplored then the color used for such region should be outside of the colormap. One of the reviewers suggests replacing white with gray for such region in these figures.
Thanks for the clarifying suggestion. We switched the color of the unexplored region from white to gray.
– Line 173: the text makes the distinction between an“"SRlik”" representation, and an“"exact S”". What is the difference? Why is it important to have an exact of the SR in the neural activity, rather then eg a monotonic encoding of the SR?
By “exact SR”, we mean the error between the steady state dynamics matrix and the SR matrix is precisely zero. “SRlike” was our loose way of referring to representation matrices with some amount of mean absolute error from the SR matrix that was still seemingly minimal but not zero.
The reviewer raises an important question about how crucial it is for the network to learn the SR exactly, versus other representations that may also capture longhorizon predictions.
This is similar in spirit to the question raised in Main Comment 1. We showed in an analysis for Main Comment 1 that it may not be necessary to learn the SR exactly, and that a range of possible rules may yield similar representations. The SR is convenient as a reasonable formalization of longhorizon predictions. The analysis referenced in line 173 (Figure 2J) also shows that plasticity kernels with varying time constants yield representations that are similar to the SR.
We clarify the statement previously in line 173 to remove the term “is SRlike” with “has minimal error from the SR matrix”. We also emphasize that the results of the analysis further supports that many predictive representations look similar to each other, and that an exactathemaatical equivalence to the SR is not the most important aspect of a predictive representation:
“Finally, we see that even plasticity kernels with slightly different time constants may give results with minimal error from the SR matrix, even if they do not estimate the SR exactly (Figure 2j). This suggests that, although other plasticity rules could be used to model longhorizon predictions, the SR is a reasonable—though not strictly uniqu— model to describe this class of predictive representations.”
– The RNN described in Equation 2 is not of the standard form (the nonlinearity is applied after the connectivity matrix, ie f(J x) instead of Jf(x)). Is this detail important? If not, why not use the more standard form to avoid confusion?
The reviewer points out that our equation uses $\mathrm{\Delta}x=x+f(Jx)+i$. It is standard to use either a firing rate representation ($\mathrm{\Delta}r=r+f(Jr+i)$) or a voltage representation ($\mathrm{\Delta}v=v+Jf(v)+i$). Choosing between the firing rate or voltage representation is not critical. Indeed, if input is not considered, these forms are equivalent up to a transformation (Miller and Fumarola 2012).
Nevertheless, the reviewer is correct that we used a nonconventional amalgamation of these two standard forms.
We reran analyses using Jf(x) instead of f(J x). We find that there is no obvious difference in the results generated. We updated the equation in the text to match the standard form. We then updated Figures 3, 5, 6, S3, S5, S6 to reflect this change in the model.
– A line of work in the Fusi lab has examined plasticity rules that lead to the encoding of transition probabilities (eg Fusi et al., Neuron 2007). In particular, a paper by the reviewing editor (Ostojic and Fusi Front Comp Neuro 2013) examined the encoding of transition probabilities using plasticity rules that look similar to this manuscript. This is mentioned just for information, the authors should decide if those papers are relevant.
Thanks for the recommendation— the Ostojic and Fusi paper is indeed quite relevant. It seems like equation 2 of the Ostojic and Fusi paper is the same as our plasticity rule under onehot encoding assumptions. In the case with more complex input features, the Ostojic and Fusi paper would be identical to the “Independent Normalization” model we present as comparison in Figure 4.
Overall, it is promising and exciting to find another study that arrives at similar conclusions: “Our study shows that synapses encode transition probabilities under general assumptions and this indicates that temporal contiguity is likely to be encoded and harnessed in almost every neural circuit in the brain.” (from the abstract of Ostojic and Fusi).
We have updated the discussion to include this reference:
“Estimating $T$ directly provides RNNS with a means to sample likely future trajectories, or distributions of trajectories, which is computationally useful for many memoryguided cognitive tasks beyond reinforcement learning, including reasoning and inference (Ostojic et al., 2013, Goodman et al., 2016). The representation afforded by $T$ may also be particularly accessible by neural circuits. Ostojic et al., (2013) note that only few general assumptions are needed for synaptic plasticity rules to estimate transition statistics. Thus, it is reasonable to assume that some form of transition statistics are encoded broadly across the brain.”
– Figures 5DE and similar supplementary figures: if there is a parameter region that is unexplored then the color used for such region should be outside of the colormap. One of the reviewers suggests replacing white with gray for such region in these figures.
This is a duplicate comment of another comment and has been fixed.
https://doi.org/10.7554/eLife.80680.sa2Article and author information
Author details
Funding
National Science Foundation (NeuroNex Award DBI1707398)
 LF Abbott
 Ching Fang
 Emily L Mackevicius
Gatsby Charitable Foundation
 LF Abbott
 Ching Fang
 Emily L Mackevicius
New York Stem Cell Foundation (Robertson Neuroscience Investigator Award)
 Ching Fang
 Dmitriy Aronov
 Emily L Mackevicius
National Institutes of Health (NIH Director's New Innovator Award (DP2AG071918))
 Ching Fang
 Dmitriy Aronov
 Emily L Mackevicius
Arnold and Mabel Beckman Foundation (Beckman Young Investigator Award)
 Ching Fang
 Dmitriy Aronov
 Emily L Mackevicius
National Science Foundation (Graduate Research Fellowship Program)
 Ching Fang
Simons Foundation (Society of Fellows)
 Emily L Mackevicius
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Acknowledgements
This work was supported through NSF NeuroNex Award DBI1707398, the Gatsby Charitable Foundation, the New York Stem Cell Foundation (Robertson Neuroscience Investigator Award), National Institutes of Health (NIH Director’s New Innovator Award (DP2AG071918)), and the Arnold and Mabel Beckman Foundation (Beckman Young Investigator Award). CF received support from the NSF Graduate Research Fellowship Program. ELM received support from the Simons Society of Fellows. We thank Jack Lindsey and Tom George for comments on the manuscript, as well as Stefano Fusi, William de Cothi, Kimberly Stachenfeld, and Caswell Barry for helpful discussions.
Senior Editor
 Timothy E Behrens, University of Oxford, United Kingdom
Reviewing Editor
 Srdjan Ostojic, École Normale Supérieure Paris, France
Reviewers
 Stefano Recanatesi, University of Washington, United States
 Arthur Juliani
Version history
 Preprint posted: May 19, 2022 (view preprint)
 Received: May 30, 2022
 Accepted: October 26, 2022
 Version of Record published: March 16, 2023 (version 1)
Copyright
© 2023, Fang et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics

 3,941
 Page views

 409
 Downloads

 8
 Citations
Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading

 Computational and Systems Biology
The hippocampus has been proposed to encode environments using a representation that contains predictive information about likely future states, called the successor representation. However, it is not clear how such a representation could be learned in the hippocampal circuit. Here, we propose a plasticity rule that can learn this predictive map of the environment using a spiking neural network. We connect this biologically plausible plasticity rule to reinforcement learning, mathematically and numerically showing that it implements the TDlambda algorithm. By spanning these different levels, we show how our framework naturally encompasses behavioral activity and replays, smoothly moving from rate to temporal coding, and allows learning over behavioral timescales with a plasticity rule acting on a timescale of milliseconds. We discuss how biological parameters such as dwelling times at states, neuronal firing rates and neuromodulation relate to the delay discounting parameter of the TD algorithm, and how they influence the learned representation. We also find that, in agreement with psychological studies and contrary to reinforcement learning theory, the discount factor decreases hyperbolically with time. Finally, our framework suggests a role for replays, in both aiding learning in novel environments and finding shortcut trajectories that were not experienced during behavior, in agreement with experimental data.

 Neuroscience
Investigating how, when, and what subjects learn during decisionmaking tasks requires tracking their choice strategies on a trialbytrial basis. Here we present a simple but effective probabilistic approach to tracking choice strategies at trial resolution using Bayesian evidence accumulation. We show this approach identifies both successful learning and the exploratory strategies used in decision tasks performed by humans, nonhuman primates, rats, and synthetic agents. Both when subjects learn and when rules change the exploratory strategies of winstay and loseshift, often considered complementary, are consistently used independently. Indeed, we find the use of loseshift is strong evidence that subjects have latently learnt the salient features of a new rewarded rule. Our approach can be extended to any discrete choice strategy, and its low computational cost is ideally suited for realtime analysis and closedloop control.