Neural dynamics of reversal learning in the prefrontal cortex and recurrent neural networks

Reviewer #1 (Public review):

The authors aimed to investigate how the probability of a reversal in a decision-making task is represented in cortical neurons. They analyzed neural activity in the prefrontal cortex of monkeys and units in recurrent neural networks (RNNs) trained on a similar task. Their goal was to understand how the dynamical systems that implement computation perform a probabilistic reversal learning task in RNNs and nonhuman primates.

Major strengths and weaknesses:

Strengths:

(1) Integrative Approach: The study exemplifies a modern approach by combining empirical data from monkey experiments with computational modeling using RNNs. This integration allows for a more comprehensive understanding of the dynamical systems that implement computation in both biological and artificial neural networks.

(2) The focus on using perturbations to identify causal relationships in dynamical systems is a good goal. This approach aims to go beyond correlational observations.

Weaknesses:

(1) The description of the RNN training procedure and task structure lacks detail, making it difficult to fully evaluate the methodology.

(2) The conclusion that the representation is better described by separable dynamic trajectories rather than fixed points on a line attractor may be premature.

(3) The use of targeted dimensionality reduction (TDR) to identify the axis determining reversal probability may not necessarily isolate the dimension along which the RNN computes reversal probability.

Appraisal of aims and conclusions:

The authors claim that substantial dynamics associated with intervening behaviors provide evidence against a line attractor. The conclusion that this representation is better described by separable dynamic trajectories rather than fixed points on a line attractor may be premature. The authors found that the state was translated systematically in response to whether outcomes were rewarded, and this translation accumulated across trials. This is consistent with a line attractor, where reward input bumps the state along a line. The observed dynamics could still be consistent with a curved line attractor, with faster timescale dynamics superimposed on this structure.

Likely impact and utility:

This work contributes to our understanding of how probabilistic information is represented in neural circuits and how it influences decision-making. The methods used, particularly the combination of empirical data and RNN modeling, provide a valuable approach for investigating neural computations. However, the impact may be limited by some of the methodological concerns raised.

The data and methods could be useful to the community, especially if the authors provide more detailed descriptions of their RNN training procedures and task structure. However, reverse engineering of the network dynamics was minimal. Most analyses didn't take advantage of the full access to the RNN's update equations.

Reviewer #2 (Public review):

Summary:

In this work, the authors trained RNN to perform a reversal task also performed by animals while PFC activity is recorded. The authors devised a new method to train RNN on this type of reversal task, which in principle ensures that the behavior of the RNN matches the behavior of the animal. They then performed some analysis of neural activity, both RNN and PFC recording, focusing on the neural representation of the reversal probability and its evolution across trials. Given the analysis presented, it has been difficult for me to assess at which point RNN can reasonably be compared to PFC recordings.

Strengths:

Focusing on a reversal task, the authors address a challenge in RNN training, as they do not use a standard supervised learning procedure where the desired output is available for each trial. They propose a new way of doing that.

They attempt to confront RNN and neural recordings in behaving animals.

Weaknesses:

The design of the task for the RNN does not seem well suited to support the claim of the paper: no action is required to be performed by neurons in the RNN, instead, the choice of the animal is determined by applying a non-linearity to the RNN's readout (equation 7), no intervening behavior is thus performed by neurons on which the analysis is performed throughout the paper. Instead, it would have been nice to mimic more closely the task structure of the experiments on monkeys, with a fixation period where the read-out is asked to be at a zero value, and then asked to reach a target value (not just taking its sign), depending on the expected choice after a cue presentation period.

The comparison between RNN and neural data focuses on very specific features of neural activity. It would have been nice to see how individual units in the RNN behave over the course of the trial, do all units show oscillatory behavior like the readout shown in Figure 1B?

It would be nice to justify why it has been chosen to take a network of inhibitory neurons and to know whether the analysis can also be performed with excitatory neurons. All the analysis relies on the dimensionality reduction. It would have been nice to have some other analysis confirming the claim of the absence of a line attractor in the neural data. Or at least to better characterize this dimensionality reduction procedure, e.g. how much of the variance is explained by this analysis for instance?

It is thus difficult to grasp, besides the fact that reversal behavior is similar, to what extent the RNN is comparable to PFC functioning and to what extent we learn anything about the latter.

Other computational works (e.g. [1,2]) have developed procedures to train RNN on reversal-like tasks, it would have been nice to compare the procedure presented here with these other works.

(1) H Francis Song & Xiao-Jing Wang. Reward-based training of recurrent neural networks for cognitive and value-based tasks. eLife doi:10.7554/elife.21492.001.

(2) Molano-Mazón, M. et al. Recurrent networks endowed with structural priors explain suboptimal animal behavior. Current Biology 33, 622-638.e7 (2023).

Reviewer #3 (Public review):

Summary:

Kim et al. present a study of the neural dynamics underlying reversal learning in monkey PFC and neural networks. The concept of studying neural dynamics throughout the task (including intervening behaviour) is interesting, and the data provides some insights into the neural dynamics driving reversal learning. The modelling seems to support the analyses, but both the modelling and analyses also leave several open questions.

Strengths:

The paper addresses an interesting topic of the neural dynamics underlying reversal learning in PFC, using a combination of biological and simulated data. Reversal learning has been studied extensively in neuroscience, but this paper takes a step further by analysing neural dynamics throughout the trials instead of focusing on just the evidence integration epoch.

The authors show some close parallels between the experimental data and RNN simulations, both in terms of behaviour and neural dynamics. The analyses of how rewarded and unrewarded trials differentially affect dynamics throughout the trials in RNNs and PFC were particularly interesting. This work has the potential to provide new insights into the neural underpinnings of reversal learning.

Weaknesses:

Conceptual:

A substantial focus of the paper is on the within-trial dynamics associated with "intervening behaviour", but it is not clear whether that is well-modelled by the RNN. In particular, since there is little description of the experimental task, and the RNN does not have to do any explicit computation during the non-feedback parts of the trial, it is unclear whether the RNN 'non-feedback' dynamics can be expected to reasonably model the experimental data.

Data analyses:

While the basic analyses seem mostly sound, it seems like a potential confound that they are all aligned to the inferred reversal trial rather than the true experimental reversal trial. For example, the analyses showing that 'x_rev' decays strongly after the reversal trial, irrespective of the reward outcome, seem like they are true essentially by design. The choice to align to the inferred reversal trial also makes this trial seem 'special' (e.g. in Figure 2, Figure 5A), but it is unclear whether this is a real feature of the data or an artifact of effectively conditioning on a change in behaviour. It would be useful to investigate whether any of these analyses differ when aligned to the true reversal trial. It is also unsurprising that x_rev increases before the reversal and decreases after the reversal (it is hard to imagine a system where this is not the case), yet all of Figure 5 and much of Figure 4 is devoted to this point.

Most of the analyses focus on the dynamics specifically in the x_rev subspace, but a major point of the paper is to say that biological (and artificial) networks may also have to do other things at different times in the trial. If that is the case, it would be interesting to also ask what happens in other subspaces of neural activity, that are not specifically related to evidence integration or choice - are there other subspaces that explain substantial variance? Do they relate to any meaningful features of the experiment?

On a related note, descriptions of the task itself, the behaviour of the animal(s?), and the neural recordings are largely absent, making it difficult to know what we should expect from neural dynamics throughout a trial. In fact, we are not even told how many monkeys were used for the paper or which part of PFC the recordings are from.

Modelling:

There are a number of surprising and non-standard modelling choices made in this paper. For example, the choice to only use inhibitory neurons is non-conventional and not consistent with prior work. The authors cite van Vreeswijk & Sompolinsky's balanced network paper, but this and most other balanced networks use a combination of excitatory and inhibitory neurons.

It also seems like the inputs are provided without any learnable input weights (and the form of the inputs is not described in any detail). This makes it hard to interpret the input-driven dynamics during the different phases of a trial, despite these dynamics being a central topic of the paper.

It is surprising that the RNN is "trained to flip its preferred choice a few trials after the inferred scheduled reversal trial", with the reversal trial inferred by an ideal Bayesian observer. A more natural approach would be to directly train the RNN to solve the task (by predicting the optimal choice) and then investigate the emergent behaviour & dynamics. If the authors prefer their imitation learning approach (which should at least be motivated), it is also surprising that the network is trained to predict the reversal trial inferred using Bayesian smoothing instead of Bayesian filtering.

Author response:

We appreciate Reviewer 1’s observation that our findings (i.e., separable dynamic trajectories are systematically translated in response to whether outcomes are rewarded, and this translation is accumulated across trials) are consistent with a line attractor model. We agree with this assessment and, in the revised manuscript, will reframe our findings about the dynamic trajectories to address its consistency with a line attractor.

However, we would like to emphasize that a line attractor model does not account for the dynamic nature of reversal probability activity observed in the neural data. Line attractor, regardless of whether it is curved or straight, implies that the activity is fixed when no reward information is presented. The focus of our work is to highlight this dynamic nature of reversal probability activity and its incompatibility with the line attractor model.

This leads to the question of how we could reconcile the line attractor-like properties and the dynamic nature of reversal probability activity. In the revised manuscript, we will provide evidence for an augmented model that has an attractor state at the beginning of each trial, followed by dynamic activity during the trial. Such a model is an example of superposition of initial attractor states with fast within-trial dynamics, as pointed out by Reviewer 1.

We also thank Reviewer 2 and Reviewer 3 for their comments on how the manuscript could be improved. In the revised manuscript, we will provide detailed explanations to clarify the choice of network model, data analysis methods and experiment and model setups.

In addition, we would like to take this opportunity to point out potentially misleading statements in the reviews by Reviewer 2 and Reviewer 3. Reviewer 2 stated that “no action is required to be performed by neurons in the RNN, …, no intervening behavior is thus performed by neurons”. Reviewer 3 stated that “the RNN does not have to do any explicit computation during the non-feedback parts of the trial…”. These statements convey the message that the trained RNN does not perform any computation. In fact, the RNN is trained to make a choice during non-feedback period in response to feedback. This is the (and the only) computation RNN performs. “Intervening behavior” refers to the choice the RNN makes across trials until reversing its initially preferred choice. We think that this confusion might have happened because the meaning of the term “intervening behavior” was unclear. We will clarify this point in the revised manuscript.

Again, thank you for the insightful comments. We will provide a more detailed response to the reviews and revise the manuscript accordingly.