Neural dynamics of reversal learning in the prefrontal cortex and recurrent neural networks

We considered a reversal learning task that monkeys were trained to perform in a previous study (Bartolo and Averbeck, 2020). In our study, we trained the RNNs to perform a task similar to the task the monkeys performed. A reversal learning task was completed by executing a block of a fixed number of trials. In each trial, the subject (monkey or RNN) chose one of the two available options, and reward for the choice was delivered stochastically. At the beginning of a block, one option was designated as the high-value option, while the other option was designated as the low-value option. The high-value option was rewarded 70% of the time when chosen, and the low-value option was rewarded 30% of the time when chosen. The reward probability of two options was switched at a random trial near the mid-trial of a block, and the reversed reward schedule was maintained until the end of the block. The reversal of reward schedule occurred only once in our task, differing from other versions of reversal learning where reversal occurs multiple times across trials (Su and Cohen, 2022; Wang et al., 2018). The subject had to learn to switch its preferred choice to maximize reward. Because reward delivery was stochastic, the subject had to infer the reversal of reward schedule by accumulating evidence that a reversal had occurred.

The sequence of events occurring in a trial was structured similarly in the RNN and the monkey tasks. As described in Bartolo and Averbeck, 2020, in each trial, the monkeys were first shown a signal that required them to fixate for a variable time (400 - 800 ms). Then, two cues were presented to both sides of the fixation dot simultaneously. The monkeys made a saccade to an option to report their choice. After holding for 500 ms, the reward was delivered (see Methods Experiment setup for monkeys for details). In the RNNs, they were first stimulated by a cue signaling them to make a choice. After a short delay, the RNNs made a choice between two options. Then, the RNNs received feedback that signaled the choice it made and whether the outcome was rewarded (Figure 1A). This same trial structure was repeated across all trials in a block (see Methods Recurrent neural network for details).

Figure 1

Download asset Open asset

The RNN and the monkeys tasks had differences. For the RNNs, there were 36 trials in a block during testing (24 trials were used for training) and, for the monkeys, there were 80 trials in a block. The number of RNN trials was reduced to avoid GPU memory overflow issues when training with backpropagation-through-time. For the RNNs, the reversal of reward schedule occurred on a trial randomly chosen from the 10 middle trials of a block. For the monkeys, the reversal trial was chosen randomly from 20 middle trials of a block. The fixation period was not required in the RNNs since they performed the task without it; however, we examined the effects of having a fixation period on neural dynamics (see Figure 4D). In total, 40 successfully trained RNNs were analyzed in our study, where each RNN performed 20 blocks of reversal learning tasks. On the other hand, two monkeys performed in total of eight sessions of experiments, where each session consisted of 24 blocks of reversal learning.

To train RNNs, we set up a reward schedule where the high-value option at the start of a trial was randomly selected from two options. Since the initial high-value option was switched to the other option at a random trial (Figure 1, scheduled reversal), the RNNs had to learn to reverse their preferred choice to maximize reward. To learn to reverse, the RNNs were trained to mimic the outputs of a Bayesian inference model that was shown to capture the monkey’s reversal behavior in previous studies (Costa et al., 2015; Bartolo and Averbeck, 2020). We first let the RNNs perform the task by simulating the RNN dynamics starting at a random initial state and providing relevant stimuli, such as cue and feedback, at every trial. Once the RNNs completed a block of trials, the choice and reward outcomes of all trials in a block were fed into the Bayesian model to infer the trial at which reward schedule was reversed, referred to as the inferred scheduled reversal trial (Figure 1B, inferred reversal). Previous studies have shown that monkeys reverse their preferred choice (Figure 1B, behavioral reversal) a few trials after the scheduled reversal trial (Costa et al., 2015; Bartolo and Averbeck, 2020). Therefore, the target choice outputs (Figure 1B, target output), on which the RNNs were trained, were set to be the initial high-value option until a few trials after the inferred scheduled reversal trial, followed by an abrupt switch to the other option. The recurrent weights and the output weights of the RNNs were trained via supervised learning to minimize the cross-entropy between the RNN choice outputs and the target choice outputs (See Methods Training scheme for details on the RNN training).

After training, in a typical block, a trained RNN selected the initial high-value option, despite occasionally not receiving a reward, but abruptly switched its choice when consecutive no-reward trials persisted (Figure 1B, left). Such abrupt reversal behavior was expected as the RNNs were trained to mimic the target outputs of the Bayesian inference model (Figure 1B, right). The intrinsic time scale of the RNN (τ=20 ms in Equation 1 in Methods Recurrent neural network) was substantially faster than the duration of a trial (500 ms), thus the persistent behavior over multiple trials was a result of learning the task. Analyzing the choice outputs of all the trained RNNs showed that, as in the example discussed, they selected the high-value option with high probability before the behavioral reversal, at which time they abruptly switched their choice (Figure 1C, black). This abrupt reversal behavior was also found in the monkey’s behavior trained on the same task (Figure 1C, orange). The behavioral reversal was preceded by a gradually increasing number of no-reward trials in the RNNs and the monkeys (Figure 1D). The distribution of behavioral reversal trials (i.e. trial at which preferred choice was reversed) relative to the scheduled reversal trial (i.e. trial at which reward schedule was reversed) was in good agreement with the distribution of monkey’s reversal trials (Figure 1E).

Next, we analyzed the neural activity of neurons in the dorsolateral prefrontal cortex (PFC) of two monkeys and the activation of population of neurons in the RNNs. The spiking activity of PFC neurons was recorded while they performed the reversal learning task using eight microelectrode arrays. In each session, we recorded simultaneously from 573 to 1023 neurons (n=8 sessions, median population size 706). This neural data was collected and analyzed in a previous manuscript (Bartolo and Averbeck, 2020). For each PFC neuron, we counted the spikes it emitted in a 300 ms time window that moved in 20 ms increment to analyze its spiking activity over time. For the RNNs, 40 successfully trained RNNs, whose reward rates were close to 70%, were included in the analysis.

We examined the temporal dynamics of task-relevant neural activity, in particular activity encoding the choice and reversal probability. To capture task-relevant neural activity, we first identified population vectors that encoded the task variables using a method called targeted dimensionality reduction (Mante et al., 2013). It regresses the activity of each neuron at each time bin onto task variables across trials and finds the maximal population vector of each task variable. Then, neural activity representing the task variable is obtained by projecting the population activity onto the identified task vectors (see Methods Targeted dimensionality reduction for details). The spiking activity of PFC neurons was shown to encode the reversal of behavior in a previous study (Bartolo and Averbeck, 2020). Following this line of work, we focused on analyzing the trials around the behavioral reversal point in each block. We referenced the position of each trial to the behavioral reversal trial as a relative trial.

Within each trial, the block-averaged neural activity associated with choices and inferred reversal probability, denoted as $x_{choice}$ and $x_{rev}$, respectively, produced non-stationary dynamics (Figure 2A, left). Their activity level reached a maximum around the time of cue onset (white squares in Figure 2A, left), when the monkey and RNN were about to make a choice. Such rotational neural dynamics were found both in the PFC of monkeys and the RNNs.

Figure 2

Download asset Open asset

Across trials, the orientation of rotational trajectories shifted, indicating systematic changes in the choice and reversal probability activity. When the task-relevant activity at cue onset (or fixation) was analyzed, the points in the two-dimensional phase space ($x_{rev}$ and $x_{choice}$) shifted gradually across trials (Figure 2A, right). We found that reversal probability activity, $x_{rev}$, peaked at the reversal trial in the PFC and the RNN (Figure 2B). Choice activity, $x_{choice}$, on the other hand, decreased gradually over trials reflecting the changes in choice preference (Figure 2C). The inverted-V shape of $x_{rev}$ and the monotonic decrease of $x_{choice}$ over trials explained the counter-clockwise shift in the rotational trajectories observed in the two-dimensional phase space (Figure 2A).

A recent study found that, when a neural network was trained via reinforcement learning to perform a reversal learning task, the first two principal components of the network activity shifted gradually following a shape similar to $x_{rev}$ and $x_{choice}$ (see Figure 1 in Wang et al., 2018). These results suggest that the gradual shift in network states across trials (Figure 2B,C) could be a common feature that emerges in networks performing a reversal learning task, regardless of training methods. One main difference is that, by design, these neural networks lacked within-trial dynamics in contrast to our RNNs (Wang et al., 2018). In the following sections, we characterize the dynamics and properties of the non-stationary neural activity that our RNNs and the PFC of monkeys generated during the trials (Figure 2A).

We first asked if there were underlying dynamics that systematically changed the reversal probability activity, $x_{rev}$. Previous works have shown that accumulation of decision-related evidence can be represented as a line attractor in a stable subspace of network activity (Mante et al., 2013; Nair et al., 2023). We hypothesized that the gradual shift of $x_{rev}$ (Figure 2A, B) could be characterized similarly by a line attractor model, where $x_{rev}$ is explained by integrating reward outcomes across trials.

To test this idea, we set up a reward integration equation $x_{rev}^{k+1}(t)=x_{rev}^k(t)+R_{\pm }^k(t)$ that predicts the next trial’s reversal probability activity at time $t$ based on the current trial’s reversal probability activity at time $t$ and the reward outcome, therefore, predicting across-trial reversal probability by integrating reward outcomes. Here, $t$ is a time point within a trial (e.g. $t=t_{cue}$ is the time of cue onset), $x_{rev}^k(t)$ is the reversal probability activity at $t$ on trial $k$, and $R_{\pm }^k(t)$ is an estimate of the shift in reversal probability activity at $t$ driven by trial $k$’s reward outcome (+ if rewarded and - if not rewarded. See Methods Reward integration equation for details).

When the reward integration activity at cue onset ($t_{cue}$) was analyzed, the predicted reversal probability activity was in good agreement with the actual reversal probability activity of PFC and RNN (example blocks shown in Figure 3A,C; prediction accuracy of all blocks shown in Figure 3E). In addition, we found that $x_{rev}^k(t_{cue})$ responded to reward outcomes consistently with how reversal probability itself would be updated: no reward increased the reversal probability activity in the next trial (Figure 3B,D; no reward), while a reward decreased it (Figure 3B, D; reward). At the behavioral reversal trial ($k=0$), however, the reversal probability activity in the following trial ($k=1$) decreased regardless of the reward outcome at the reversal trial.

Figure 3

Download asset Open asset

When the reward integration equation was fitted to other time points in the trial (i.e. estimate $R_{\pm }^k(t)$ at other $t$ in the trial), the prediction accuracy remained stable over the trial duration (Figure 3F). This suggested that a line attractor model might be applicable throughout the trial. However, the reward integration equation is an algebraic relation and does not capture the dynamics of neural activity, such as the non-stationary activity during the trial (e.g. Figure 2). This observation led us to characterize the dynamics of reversal probability activity to assess whether it was compatible with the line attractor model.

We showed that the reversal probability activity encodes the accumulation of reward outcomes, which resembles a line attractor model (Figure 3). However, a direct application of the line attractor dynamics would imply that, when no decision-related evidence is presented within a trial, the reversal probability activity should remain stationary (Figure 4A, Stationary), which was incompatible with the non-stationary reversal probability activity observed during a trial (Figure 2A).

Figure 4 with 1 supplement see all

Download asset Open asset

To better characterize the dynamics of reversal probability activity, we analyzed how its derivative $d{x}_{rev}/dt$ changes throughout a trial. In the PFC of monkeys, we found that the reversal probability activity was stationary at the start of a trial: $d{x}_{rev}/dt$ was close to zero before and during the fixation (Figure 4B, Stationary). However, the non-stationary activity emerged at cue onset and was associated with task-related behavior, such as making a choice and receiving reward. Specifically, $d{x}_{rev}/dt$ increased rapidly at the cue onset, when a decision was made, followed by a sharp fluctuation and slow decay until the reward time (Figure 4B, Non-stationary).

We then analyzed the contraction factor of the reversal probability activity to assess whether the activity is contracting or expanding around its mean activity (Figure 4C; see Methods Contraction factor of reversal probability activity for details). We found that the contraction factor was less than 1 (i.e. contracting) before the fixation period (-2.5 to -1 s in Figure 4C, left; also see the right panel), became close to 1 (i.e. marginally stable) around fixation (-1 to 0 s), and exceeded 1 (i.e. expanding) at cue onset (0 s). This showed that the reversal probability activity is a point attractor (i.e. stationary and contracting) at the start of a trial but loses its stability as task-related behavior is executed.

In the RNNs, we found that the dynamics of reversal probability activity were similar to the PFC’s activity when an additional constraint was added to the RNN’s objective function to fix its choice output to zero before making a choice (i.e. fixation period was between fixation and cue-off lines in Figure 4D; indicated as Stationary period). With the fixation constraint, the reversal probability activity was stationary during the fixation period (i.e. $d{x}_{rev}/dt$ converged and stayed close to zero). This contrasted the RNNs trained without the fixation, which exhibited more dynamics before making a choice, suggesting the role of fixation in promoting stationary dynamics (Figure 4—figure supplement 1). The RNN’s non-stationary dynamics after cue-off was consistent with the PFC’s non-stationary activity regardless of the fixation constraint (Figure 4 and Figure 4—figure supplement 1, Non-stationary).

The contraction factor of reversal probability activity of the RNNs trained with the fixation constraint showed a trend that was similar to the PFC. It was less than 1 during fixation (fixation to cue-off in Figure 4E; also see the right panel) and became close to 1 immediately after cue-off when the decision was made (the point next to cue-off in Figure 4E). As in the PFC, the contraction factor shows that the RNN’s reversal probability activity is a point attractor at the start of a trial and becomes unstable as the RNN executes decision making.

Thus, analyzing $d{x}_{rev}/dt$ across a trial showed that the reversal probability activity consisted of two dynamic modes: a point attractor at the start of a trial, which was consistent with the line attractor model, followed by non-stationary dynamics during the trial, which deviated from the line attractor.

Next, we investigated whether the two activity modes are linked by common dynamics. In the RNNs, the cue applied at the end of the stationary period (Figure 4D, cue-on to cue-off) determined the initial state from which the non-stationary activity was generated by the recurrent dynamics. We wondered if the two activity modes of PFC also obeyed the same dynamic relationship, i.e., the non-stationary activity is generated by underlying dynamics with an initial condition given by the stationary state.

To test this hypothesis, we took the PFC activity at the start of fixation period (-0.5 s) as the initial state. Then, we trained a support vector regression (SVR) to infer the underlying dynamics, which used the PFC activity at fixation as the input and generated the remaining trajectory until the reward time as the output. The SVR model was trained on neural data from 20 trials around the behavioral reversal in 10 randomly selected blocks from a session and tested on the remaining blocks (approximately 10 blocks) from the same session, thus training separate SVR model for each session. This procedure was repeated 10 times to sample different sets of training blocks (see Methods Generating the PFC reversal probability trajectories from initial states for the details). The prediction error was quantified using the normalized mean-squared-error (MSE) between the SVR prediction and the actual reversal probability trajectory normalized by the amplitude of the trajectory.

Figure 4F (top) compares the SVR trajectory (red, prediction) generated from an initial state (green, initial state) and the actual PFC reversal probability trajectory (black, data) in trials around the behavioral reversal. We found that the SVR trajectories were able to capture the overall shape of data trajectories that had a bump shape and shifted up and down with the initial state. The normalized MSE of all the trials in the test dataset was 0.13, i.e., the mean error is 13% of the trajectory amplitude (Figure 4G).

To verify the role of initial state in generating trajectories, we trained a null SVR model, in which the initial states of trials in a block were randomly shuffled before training the model. In other words, a reversal probability trajectory was generated not from its own initial state, but from the initial state of a randomly chosen trial. The rest of the training procedure was identical as described above. We found that, although the null trajectory (Figure 4F, Null, prediction) resembled the overall shape of the data trajectory (Figure 4F, Null, data), i.e., increase towards cue onset and then decrease monotonically, it did not shift together with the initial states (Figure 4F, Null, initial state), as did the data trajectories. When the normalized MSE of the SVR and the null trajectories were compared, we found that in 86% of the test trials the SVR error was smaller than the null error (Figure 4H).

Together, our results show that the reversal probability activity consists of two activity modes linked by underlying dynamics. The stable stationary state (i.e. point attractor) at the start of a trial determines the initial condition from which the non-stationary dynamics, associated with task-related behavior, is generated. These findings suggest an extension of the standard line attractor model, in which points on the line attractor serve as initial states that launch non-stationary activity necessary to perform a task.

The previous section showed that the underlying neural dynamics, associated with task-related behavior, generates non-stationary activity during a trial. We next asked how distinct probabilistic values can be encoded in this non-stationary activity. In a stationary state, different positions encode different levels of decision-related evidence. When stationary points evolve through the non-stationary dynamics, as in our augmented model, they must remain separable in order to represent distinct values.

To address this question, we compared trajectories of adjacent trials to examine if the reward outcome drives the next trial’s trajectory away from the current trial’s trajectory, thus separating them. Analysis of PFC activity showed that not receiving a reward increased the next trial’s trajectory $x_{rev}^{k+1}(t)$, compared to the current trial’s trajectory $x_{rev}^k(t)$, over the entire trial duration until the next trial’s reward was revealed (Figure 5A, left; $R_{-}$). This was shown in the difference of adjacent trials’ trajectories being positive values, when not rewarded through most of the trial (Figure 5A, right; $R_{-}$). Moreover, across trials, the same trend was observed in all the trials except at the behavioral reversal trial, at which the reversal probability activity reached its maximum value and decreased in the following trial (Figure 5B, top; $R_{-}$). On the other hand, when a reward was received, the next trial’s trajectory was decreased, compared to the current trial’s trajectory over the entire trial duration until the next trial’s reward (Figure 5A, left; $R_{+}$). This was shown in the difference of adjacent trials’ trajectories being negative values, when rewarded through most of the trial (Figure 5A, right; $R_{+}$). Across trials, the same trend was observed in all the trials except at the trial preceding the behavioral reversal trial, at which the trajectory increased to the maximum value at the reversal trial (Figure 5, bottom; $R_{+}$). Additional analysis on $R_{-}$ and $R_{+}$ beyond the next trial’s reward time can be found in Figure 6—figure supplement 1.

Figure 5 with 1 supplement see all

Download asset Open asset

We then examined what type of activity the dynamic trajectories exhibited when separating away from the previous trial’s trajectory. Ramping activity is often observed in cortical neurons of animals engaged in decision-making (Li et al., 2015; Li et al., 2016; Finkelstein et al., 2021; Latimer et al., 2015; Zoltowski et al., 2019). We found that when no rewards were received, trajectories were separated from the previous trial’s trajectory by increasing their ramping rates towards the decision time ($d{R}_{-}/dt>0$ in Figure 5C, right). On the other hand, when rewards were received, trajectories were separated by decreasing their ramping rate ($d{R}_{+}/dt<0$ in Figure 5C, right). The increase (or decrease) in the ramping rates was observed in consecutive no reward (or reward) trials around the reversal trial (Figure 5C, left).

Consistent with the PFC activity, the trained RNN exhibited similar activity responses to reward outcomes: neural trajectories encoding reversal probability increased, when not rewarded, and decreased, when rewarded. The shift in trajectories persisted throughout the trial duration (Figure 5D) and ramping rates changed in agreement with the PFC findings (Figure 5E).

We examined the circuit dynamic motif of the trained RNN that separates neural trajectories. We projected the differential equation governing the network dynamics onto a one-dimensional subspace encoding reversal probability and analyzed the contribution of external and recurrent inputs to reversal probability dynamics: $\tau {\dot{x}}_{rev}=x_{rec}+x_{ext}\equiv x_{total}$ (see Methods Recurrent neural network for details). We found that the effect of the external input $x_{ext}$ was positive, reflecting that the external feedback input drives the reversal probability. On the other hand, the recurrent input $x_{rec}$ was negative, showing that it curtailed the external input (Figure 5F, external and recurrent). To analyze the effects of no-reward (or reward), we averaged the reversal probability activity over all trials (Fig.5F, reference) at which the subsequent trial was not (or was) rewarded. When no reward was received, $x_{ext}$ (Figure 5F, external, red) and $x_{rec}$ (Figure 5F, recurrent, magenta) were both amplified, compared to the reference, by approximately the same factor and resulted in increased total input: $x_{total}^{norew}={\gamma }^{norew}{x}_{total}$ with ${\gamma }^{norew}>1$ (Figure 5F, amplification, red and magenta). On the other hand, when reward was received, $x_{ext}$ (Figure 5F, external, blue) and $x_{rec}$ (Figure 5F, recurrent, cyan) were both suppressed, resulting in decreased total input: $x_{total}^{reward}={\gamma }^{reward}{x}_{total}$ with ${\gamma }^{reward}<1$ (Figure 5F, amplification, blue and cyan). This suggests a circuit dynamic motif, where positive external feedback balanced by recurrent inhibition drives the reversal probability dynamics. The total drive is amplified or suppressed, depending on reward outcomes, resulting in a trajectory that separates from the previous trial’s trajectory.

In sum, our findings show that dynamic neural trajectories encoding reversal probability are separated from the previous trial’s trajectory in response to reward outcomes, allowing them to represent distinct values of reversal probability during a trial.

We investigated if reversal probability trajectories across multiple trials maintained separability. To this end, we analyzed the mean behavior of trajectories in each trial (referred to as mean trajectory of a trial) and analyzed their separability across trials around the behavioral reversal. Since a mean trajectory of a trial was obtained by averaging over all reward outcomes of the previous trial, we compared how reward and no-reward contributed to modifying the next trial’s mean trajectory, which were quantified by $R_{+}^k$ and $R_{-}^k$ in Figure 5B, respectively.

We found that the effect of no-reward was larger than the effect of reward before the behavioral reversal. This is shown as the trace $R_{-}^k$ (no reward) lying above the trace $-R_{+}^k$ (reward) during a trial and across all pre-reversal trials (see pre-reversal trials -5 to -1 in Figure 6A; $R_{-}^k$ and $-R_{+}^k$ are positive traces since $R_{+}^k$ is mostly negative). The temporal averages of $R_{-}^k$ and $-R_{+}^k$ captured this systematic differences in the pre-reversal trials (Figure 6B, bottom), and the sum $R_{-}^k+R_{+}^k$ was positive during a trial in 80% of the pre-reversal trials (Figure 6C, top). The positivity of $R_{-}^k+R_{+}^k$ in time and across pre-reversal trials implied that the mean trajectories remained separable by increasing monotonically across the trials (Figure 6C, bottom). Consistent with this observation, the Spearman rank correlation of pre-reversal trajectories was stable in time (Figure 6E, pre).

Figure 6 with 1 supplement see all

Download asset Open asset

After the behavioral reversal, the effects of no-reward and reward were the opposite of pre-reversal trials. The trace $-R_{+}^k$ lied above the trace $R_{-}^k$ (see post-reversal trials 0 to 4 in Figure 6A), and the temporal average $⟨-R_{+}^k{⟩}_t$ was larger than $⟨R_{-}^k{⟩}_t$ (Figure 6B, bottom). This showed that $R_{-}^k+R_{+}^k$ was mostly negative during post-reversal trials. The fraction of trials, for which $R_{-}^k+R_{+}^k$ is negative, was close to 80% among the post-reversal trials (Figure 6D, top). The negativity of $R_{-}^k+R_{+}^k$ in time and across post-reversal trials implied that the post-reversal trajectories remained separable by decreasing monotonically across the trials (Figure 6D, bottom). The order of post-reversal trajectories was stable over the trial duration, similarly to the pre-reversal trials but in the reversed order (Figure 6E, post).

In the trained RNNs, the effects of reward outcomes on mean trajectories were consistent with the PFC findings: $R_{-}^k+R_{+}^k$ was positive and negative before and after the behavioral reversal, respectively (Figure 6F). Consequently, reversal probability trajectories of the RNNs maintained separability by shifting monotonically as in the PFC (Figure 6G,H). The order of trajectories was also stable over the trial duration (Figure 6I).

These findings show that the mean behavior of reversal probability trajectories is to shift monotonically across trials. It suggests that a family of separable neural trajectories around the reversal trial can represent varying estimates of reversal probability stably in time.

Here, we turned to the RNN to see if we could perturb activity within the reversal probability space and consequently perturb the network’s choice preference. We defined the reversal probability activity $x_{rev}$ as the activity in a neural subspace correlated to the behavioral reversal probability (Methods Targeted dimensionality reduction). However, it remains to be shown if the activity within this neural subspace can causally affect network’s behavioral outcomes.

Previous experimental works demonstrated that perturbing neural activity of medial frontal cortex (Murphy et al., 2022), specific cell types (Yi et al., 2024; Jeong et al., 2020), or neuromodulators (Su and Cohen, 2022; Hyun et al., 2023) affect the performance of reversal learning. In our perturbation experiments in the RNNs, the perturbation was tailored to be within the reversal probability space by applying an external stimulus aligned ($v_{+}$) or opposite ($v_{-}$) to the reversal probability vector. An external stimulus in a random direction was also applied as a control ($v_{rnd}$). All the stimuli were applied before the time of choice at the reversal trial or at preceding trials (Figure 7).

Figure 7

Download asset Open asset

We found that the deviation of perturbed reversal probability activity from the unperturbed activity peaked at the end of perturbation duration and decayed gradually (Figure 7B, red solid). The perturbed choice activity, however, deviated more slowly and peaked during the choice duration (Figure 7B, black solid). This showed that perturbation of the reversal probability activity had its maximal effect on the choice activity when the choice was made. The strong perturbative effects on the reversal probability and choice activity were not observed in the control (Figure 7B, dotted).

The perturbation in the aligned ($v_{+}$) and opposite ($v_{-}$) directions shifted the reversal probability activity along the same direction as the perturbation vector, as expected (Figure 7C, left). On the other hand, the choice activity was increased when the perturbation was in the opposite direction $v_{-}$. The perturbation in the aligned direction $v_{+}$ did not decreases the choice activity substantially, but its effect was significantly smaller than the increase seen in $v_{-}$ perturbation (Figure 7C, right; KS-test, p-value = 0.007).

We further analyzed if perturbing within the reversal probability space could affect the choice outcomes, specifically the behavioral reversal trial. We found that the reversal trial was delayed when $v_{-}$ stimulus was applied to reduce the reversal probability activity (Figure 7D, left). The effect of $v_{-}$ stimulus increased gradually with the stimulus strength and was significantly stronger than the $v_{+}$ or $v_{rnd}$ stimuli in delaying the reversal trial. Perturbation had the strongest effect when applied to the reversal trial, while perturbations on trials preceding the reversal showed appreciable but reduced effects (Figure 7D, right). When the $v_{+}$ stimulus was applied to trials preceding the reversal trial, the reversal was accelerated (Figure 7E, left). The effect of $v_{+}$ stimulus also increased with stimulus strength and was significantly stronger than the $v_{-}$ or $v_{rnd}$ stimuli in accelerating the reversal trial (Figure 7E, right).

We asked if perturbation of neural activity in PFC could exhibit similar responses. In other words, does increase (or decrease) in reversal probability activity lead to decrease (or increase) in choice activity in PFC? Although PFC activity was not perturbed by external inputs, we considered the residual activity of single trials, i.e. deviation of single trial neural activity around the trial-averaged activity, to be ‘natural’ perturbation responses. We fitted a linear model to the residual activity of reversal probability and choice and found that they were strongly negatively correlated (i.e. negative slope in Figure 7F) at the trial preceding the behavioral reversal trial. This analysis demonstrated the correlation between perturbation responses of reversal probability and choice activity. However, it remains to be investigated, through perturbation experiments, whether reversal probability activity is causally linked to choice activity in PFC and, moreover, to animal’s choice outcomes.

Reversal learning has been a behavioral framework for investigating how the brain supports flexible behavior (Butter, 1969; Costa et al., 2015; Groman et al., 2019; Bartolo and Averbeck, 2020; Su and Cohen, 2022; Hyun et al., 2023) and for elucidating neural mechanisms underlying mental health issues (Groman et al., 2022; Schoenbaum et al., 2006). It has been shown that multiple brain regions (cortical Groman et al., 2019; Bartolo and Averbeck, 2020; Wilson et al., 2010; Murphy et al., 2022; Yi et al., 2024; Jeong et al., 2020; Wilson et al., 2014 and subcortical Groman et al., 2019; Averbeck and O’Doherty, 2022), neuromodulators ; (Costa et al., 2015; Su and Cohen, 2022; Hyun et al., 2023), and different inhibitory neuron types (Yi et al., 2024; Jeong et al., 2020) are involved in reversal learning.

Despite these recent advances, the dynamics of neural activity in cortical areas during a reversal learning task have not been well characterized. In this study, we investigated how reversal probability is represented in cortical neurons by analyzing neural activity in the prefrontal cortex of monkeys and recurrent neural networks performing the reversal learning task. We found that the activity in a neural subspace encoding reversal probability represented integration of reward outcomes. This reversal probability activity had two activity modes: stable stationary activity at the start of trial, followed by non-stationary activity during the trial. There was underlying dynamics, associated with task-related behavior, that generated the non-stationary activity with an initial condition given by the stationary state. The existence of two activity modes suggested an extension of the standard line attractor model where non-stationary dynamics driven by task-related behavior link the attractor states. The non-stationary trajectories were separable, allowing distinct probabilistic values to be represented in dynamic trajectories. Perturbation experiments in the RNNs demonstrated a potential causal link between reversal the probability activity and choice outcomes.

RNNs with attractor dynamics have been investigated in various contexts as a neural implementation of normative models of decision-making and evidence integration (Ratcliff, 1978; Palmer et al., 2005; Shadlen and Newsome, 2001; Mazurek et al., 2003; Ratcliff et al., 2003). One perspective is to consider decision variables as discrete or continuous attractor states of an RNN. Then, the network activity converges to an attracting state as a decision is made. Biologically plausible network models (Wang, 2002; Seung, 1996) and neural recordings in cortical areas have been shown to exhibit discrete (Luo et al., 2023; Sutton, 1988; Genkin et al., 2023) and continuous (Wimmer et al., 2014) attractor dynamics. Another perspective, more closely related to our study, is to consider evidence integration as a movement of network state along a one-dimensional continuous attractor, as demonstrated in Inagaki et al., 2019; Nair et al., 2023; Sylwestrak et al., 2022 (see also continuous attractor dynamics in spatial mapping Bollimunta et al., 2012; Gardner et al., 2022; Sorscher et al., 2023; Hulse and Jayaraman, 2020).

In most of the studies, decision-related evidence was presented without significant interruption until the decision point (Luo et al., 2023; Sutton, 1988; Mante et al., 2013; Genkin et al., 2023). However, this was not the case in a reversal learning task with probabilistic rewards, as reward outcomes were revealed intermittently over multiple trials while task-related behavior must be performed within each trial. We showed that such multi-trial evidence integration promoted substantial non-stationary activity in the neural subspace encoding reversal probability. Therefore, the continuous attractor dynamics, in which the network state stays close to the attracting states, did not fully account for the observed neural dynamics. Instead, our findings suggest that separable dynamic trajectories in addition to attractor states could serve as a neural mechanism that represents accumulated evidence and accommodates non-stationary behaviors necessary to perform the task.

Our work demonstrated similarities in how reversal probability is represented in the PFC of monkeys and the RNNs. However, our approach, which compares the activity of RNNs trained on the task to the PFC activity, does not directly characterize the dynamic landscape of PFC activity. In particular, our analysis only shows that the PFC activity at the initial state of each trial can be characterized as a point attractor (Figure 4C, E). Although this result is compatible with a line attractor model, it does not demonstrate whether the initial states across trials collectively form a line attractor. In order to show the existence of a line attractor, it is necessary to identify the dynamics in the region of the phase space occupied by the set of point attractors.

Alternative approaches that infer the latent dynamics of spiking activity of a neural population (Kim et al., 2023; Kim et al., 2021; Chen et al., 2018; Sussillo et al., 2016) could allow us to address this question by deriving a low-dimensional system of differential equations that generates the PFC activity. If the inferred latent dynamics could generate the two activity modes observed in our data, it would allows us to characterize different aspects of the neural dynamics, such as attractor states, the role of external inputs, and the non-stationary dynamics, within a single dynamical system model.

Recent studies showed that intervening behaviors, such as introducing an intruder (Nair et al., 2023) or accumulating reward across trials (Sylwestrak et al., 2022), could produce neural trajectories that deviate from and retract to a line attractor. These studies are consistent with our finding in that their neural dynamics temporarily deviated from attractor states. However, in our study, we did not thoroughly investigate the neural activity from dynamical systems perspective. It remains as future work to characterize the dynamical landscape of how the separable dynamic trajectories observed in our study are augmented to the continuous attractor model and compare it to the previous works (Nair et al., 2023; Sylwestrak et al., 2022).

A number of relevant studies have trained RNNs to perform various decision-making tasks (Wang et al., 2018; Schaeffer et al., 2020; Song et al., 2017; Molano-Mazón et al., 2023). In a related work (Schaeffer et al., 2020), RNNs were trained to perform a change point detection task designed by the International Brain Laboratory (Findling, 2023). They showed that trained RNNs exhibited behavior outputs consistent with an ideal Bayesian observer without explicitly learning from the Bayesian observer. This finding shows that the behavioral strategies of monkeys could emerge by simply learning to do the task, instead of directly mimicking the outputs of Bayesian observer as done in our study.

The trained RNN in their work (Schaeffer et al., 2020) exhibited line attractor dynamics in contrast to our observation of stationary and non-stationary dynamics (Figure 4A). In another study, a line attractor-like dynamics, where the principal components of network activity moved gradually across trials, was observed in artificial agents trained to perform the reversal learning task via reinforcement learning (seeFigure 1 in Wang et al., 2018). One possible reason for the lack of non-stationary dynamics within a trial in these other studies is that a trial consisted of only one (Wang et al., 2018) or a few time points (Schaeffer et al., 2020), which limits the possible range of temporal dynamics RNNs can exhibit during a trial. This suggests that different setup of a task can lead to significantly different dynamics in the trained RNN. Moreover, it needs to be investigated whether such attractor dynamics are present in the neural recordings from mice performing the change point detection task.

The RNNs in our study were trained via supervised learning. However, in real life, animals most likely learn a reversal learning task via reinforcement learning (RL), i.e., learn the task from reward outcomes. Neuromodulators play a key role in mediating RL in the brain. In a recent study, dopamine-based RL was used to train artificial RNNs to conduct reversal learning tasks. It was shown that neural activity in RNNs and mice performing the same tasks were in good agreement (Wang et al., 2018). In addition, projections of serotonin from dorsal raphe nuclei (Hyun et al., 2023; Matias et al., 2017) and norepinephrine from the locus coeruleus (Su and Cohen, 2022) to the cortical areas were shown to be involved in reversal learning. Further studies with biologically plausible network models, including neuromodulatory effects (Harkin et al., 2023b; Wert-Carvajal et al., 2022) or formal RL theories incorporating neuromodulators (Harkin et al., 2023a) could provide further insights into the role of neuromodulators in reversal learning.

Our findings show that, when performing a reversal learning task, a cortical circuit represent reversal probability, not only in stable stationary states as in a line attractor model, but also in dynamic neural trajectories that can accommodate non-stationary task-related behaviors necessary for the task. Such neural mechanism demonstrates the temporal flexibility of cortical computation and opens the opportunity for extending existing neural model for evidence accumulation by augmenting temporal dynamics.

Here we formulate Bayesian models that infer the scheduled reversal trial or the behavior reversal trial.

Targeted dimensionality reduction (TDR) identifies population vectors that encode task variables explicitly or implicitly utilized in the experiment the subject or RNN performs (Mante et al., 2013). In this study, we were interested in identifying population vectors that encode choice preference and reversal probability. Once those task vectors were identified, we analyzed the neural activity projected to those vectors to investigate neural representation of task variables.

We describe how TDR was performed in our study (see Mante et al., 2013 for the original reference). First, we regressed the neural activity of each neuron at each time point onto task variables of interest. Then we used the matrix of regression coefficients (i.e. neuron by time) to identify the task vector. Let $y_{it}(k)$ be the spiking rate of neuron $i$ at time $t$ on trial $k$ where we have $N$ neurons and $M$ time points. We regressed the spiking activity on task variables of interest $z^v(k)$ where the task variables were $v\in$ {reversal probability, choice preference, direction, object, block type, reward outcome, trial number}. For each neuron-time pair, $(i,t)$, we performed linear regression over all trials $k\in [0,T]$ with a bias:

This regression analysis yielded an $N\times M$ coefficient matrix ${\beta }_{it}^v$ for each task variable, $v$. We considered this coefficient matrix as a population vector evolving in time: ${\mathit{\beta }}_t^v=({\beta }_{1t}^v,...,{\beta }_{Nt}^v)$. Then, a task vector was defined as the population vector ${\mathbf{w}}^v\in {\mathbb{R}}^N$ at which the $L_2$-norm $\Vert {\mathit{\beta }}_t^v{\Vert }_2$ achieved its maximum:

We performed QR-decomposition on the matrix of task vectors $W=[{\mathbf{w}}_{rev},{\mathbf{w}}_{choice},...]$ to orthogonalize the task vectors. Then, the population activity was projected onto each (orthogonalized) task vector to obtain the neural activity encoding each task variable:

where ${\mathbf{y}}_t(k)=(y_{1t}(k),...,y_{Nt}(k))$ is the population activity at time $t$ on trial $k$.

To derive the reward integration equation shown in Figure 3, we considered the neural activity in a subspace encoding the reversal probability:

We analyzed the neural activity at the time of cue onset $t=t_{on}$ and obtained a sequence of reversal probability activity across trials: $x_{rev}^0(t_{on}),\dots ,x_{rev}^K(t_{on})$. To set up the reward integration equation

we estimated the update $R_k^{\pm }(t_{on})$ driven by reward outcomes at each trial $k$. Specifically, the update term was defined as the block-average of the difference of reversal probability activity at adjacent trials:

Here, $b_k^{+}$ denotes all the blocks across sessions (or networks) in which reward was received at trial $k$. Similarly, $b_k^{-}$ denotes all the blocks in which reward was not received at trial $k$.

To predict $x_{rev}^k(t_{on})$, we set the initial value $x_{pred}^0(t_{on})=x_{rev}^0(t_{on})$ at trial 0 and sequentially predicted the following trials using Equation 29 with the update term from Equation 30. The same analysis was performed at different time points $t$. We derived integration equations for each time and assessed its prediction accuracy as shown in Figure 3F.

To evaluate the contribution of reward and no-reward outcomes on the average responses of $\mathrm{\Delta }{x}_{rev}^k(t)$ over blocks, we computed

where

with $b_k=b_k^{+}\cup b_k^{-}$ denote the fractions of reward and no-reward blocks at trial $k$. In Figure 5D and Figure 6A, the weighted responses, i.e., $f_k^{+}{R}_k^{+}(t)$ and $f_k^{-}{R}_k^{-}(t)$, were shown.

We defined a contraction factor to quantify whether the reversal probability activity $x_{rev}(t)$ contracts to or diverges from its mean activity on a short time interval. The contraction factor on the $n^{th}$ time interval $[nL,(n+1)L]$ of length $L$ was defined to be the coefficient of a one-dimensional autoregressive equation the mean-centered reversal probability activity $z_{rev}^{(n)}(t)$ satisfied.

where

The contraction factor $a$ of a time interval was estimated by performing a scalar linear regression without an intercept given the input data $(z_{rev}^{(n)}(0),\dots ,z_{rev}^{(n)}(L-1))$ and the output data $(z_{rev}^{(n)}(1),\dots ,z_{rev}^{(n)}(L))$.

We investigated if the PFC reversal probability trajectories $x_{rev}^k(t)$ of trial $k$ defined on the trial duration, $t\in [T_0,T_f]$, can be generated from their initial states (see Figure 4D). To test this idea, we trained support vector regression (SVR) model on the spiking activity of PFC neurons. The training data consisted of reversal probability trajectories $x_{rev}^k(t)$, $t\in [T_0,T_f]$ of 20 trials around the behavioral reversal trial in each block. About 50% of the blocks in an experiment session (specifically, 10 blocks) was randomly selected for training, and the remaining 50% of the blocks in the same session was used for testing. This procedure was repeated 10 times, training a different SVR model each time, to test models trained on different sets of blocks. For each experiment session, a different SVR model was trained.

To train an SVR model, reversal probability trajectory at each trial $k$ was divided into initial state and remaining trajectory:

where $T_0=-500\mathrm{m}\mathrm{s}$ (start of fixation) and $T_i=-300\mathrm{m}\mathrm{s}$ (end of initial state). Then, an SVR model $f$ was constructed that takes the initial state ${\mathbf{x}}_{init}^k$ and a time point $s$ as its inputs and produces an approximation of ${\mathbf{x}}_{traj}^k(s)$:

As described above, we combined 20 trials around the reversal trial ($k=-10,\dots ,10$) from 10 randomly selected blocks in a session (about 50% of blocks in a session) to train the SVR model. The radial basis function was used as a kernel, and the optimal hyperparameters of the SVR model were selected through cross-validation. We used the Python’s sklearn package to implement SVR.

The PFC activity $x_t^{rev}(k)$ encoding reversal probability was used to decode the behavioral reversal trial at which monkey reversed its preferred choice (see Figure 5—figure supplement 1). Our analysis is similar to the Linear Discriminant Analysis (LDA) performed in a previous study (Bartolo and Averbeck, 2020) at a fixed time point. Here, we applied LDA to time points across a trial.

For training, 90% of the blocks were randomly selected to train the decoder and remaining 10% of the blocks were used for testing. This was repeated 20 times. Input data to LDA was $x_t^{rev}(k)$ of $\mathrm{\Delta }k$ trials around the reverse trial, i.e., $k\in [k_{rev}-\mathrm{\Delta }k,\dots ,k_{rev}+\mathrm{\Delta }k]$ with $\mathrm{\Delta }k=10$. At each trial $k$, we took the activity vector ${\mathbf{x}}_{t_0}^{rev}(k)=(x_{t_0-\mathrm{\Delta }t}^{rev}(k),\dots ,x_{t_0+\mathrm{\Delta }t}^{rev}(k))$ around time $t_0$ with $\mathrm{\Delta }t=160\mathrm{m}\mathrm{s}$. The target output of LDA was a one-hot vector ${\mathbf{y}}^{target}$, whose element was 1 at the reversal trial $k_{rev}$ and 0 at other trials. The following input-output shows dataset of a block used for training:

Here, ${\mathbf{x}}_{t_0}^{rev}(k)\in {\mathbb{R}}^{T_{dec}}$ where $T_{dec}=2\mathrm{\Delta }t/\mathrm{\Delta }h+1$ denotes the number of time points around $t_0$ with time increment $\mathrm{\Delta }h=20\mathrm{m}\mathrm{s}$, and the one-hot vector ${\mathbf{y}}^{target}=[0,\dots ,1,\dots ,0]\in {\mathbb{R}}^{K_{dec}}$ where $K_{dec}=2\mathrm{\Delta }k+1$ denotes the number of trials around the reversal trial. As mentioned above, this analysis was repeated for time point $t_0$ across a trial.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

This research was supported by the Intramural Research Program of the National Institutes of Health: the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) and the National Institute of Mental Health (NIMH). This work utilized the computational resources of the NIH HPC Biowulf cluster (https://hpc.nih.gov).

1. Preprint posted: September 15, 2024
2. Sent for peer review: October 4, 2024
3. Reviewed Preprint version 1: December 6, 2024
4. Reviewed Preprint version 2: June 10, 2025
5. Version of Record published: September 23, 2025

You can cite all versions using the DOI https://doi.org/10.7554/eLife.103660. This DOI represents all versions, and will always resolve to the latest one.

This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.