Revealing unexpected complex encoding but simple decoding mechanisms in motor cortex via separating behaviorally relevant neural signals

Reviewer #1 (Public Review):

This work seeks to understand how behaviour-related information is represented in the neural activity of the primate motor cortex. To this end, a statistical model of neural activity is presented that enables a non-linear separation of behaviour-related from unrelated activity. As a generative model, it enables the separate analysis of these two activity modes, here primarily done by assessing the decoding performance of hand movements the monkeys perform in the experiments. Several lines of analysis are presented to show that while the neurons with significant tuning to movements strongly contribute to the behaviourally-relevant activity subspace, less or un-tuned neurons also carry decodable information. It is further shown that the discovered subspaces enable linear decoding, leading the authors to conclude that motor cortex read-out can be linear.

Strengths:

In my opinion, using an expressive generative model to analyse neural state spaces is an interesting approach to understanding neural population coding. While potentially sacrificing interpretability, this approach allows capturing both redundancies and synergies in the code as done in this paper. The model presented here is a natural non-linear extension of a previous linear model (PSID) and

Weaknesses:

First, the model in the paper is almost identical to an existing VAE model (TNDM) that makes use of weak supervision with behaviour in the same way [1]. This paper should at least be referenced. If the authors wish they could compare their model to TNDM, which combines a state space model with smoothing similar to LFADS. Given that TNDM achieves very good behaviour reconstructions, it may be on par with this model without the need for a Kalman filter (and hence may achieve better separation of behaviour-related and unrelated dynamics).

Second, in my opinion, the claims regarding identifiability are overstated - this matters as the results depend on this to some extent. Recent work shows that VAEs generally suffer from identifiability problems due to the Gaussian latent space [2]. This paper also hints that weak supervision may help to resolve such issues, so this model as well as TNDM and CEBRA may indeed benefit from this. In addition however, it appears that the relative weight of the KL Divergence in the VAE objective is chosen very small compared to the likelihood (0.1%), so the influence of the prior is weak and the model may essentially learn the average neural trajectories while underestimating the noise in the latent variables. This, in turn, could mean that the model will not autoencode neural activity as well as it should, note that an average R2 in this case will still be high (I could not see how this is actually computed). At the same time, the behaviour R2 will be large simply because the different movement trajectories are very distinct. Since the paper makes claims about the roles of different neurons, it would be important to understand how well their single trial activities are reconstructed, which can perhaps best be investigated by comparing the Poisson likelihood (LFADS is a good baseline model). Taken together, while it certainly makes sense that well-tuned neurons contribute more to behaviour decoding, I worry that the very interesting claim that neurons with weak tuning contain behavioural signals is not well supported.

Third, and relating to this issue, I could not entirely follow the reasoning in the section arguing that behavioural information can be inferred from neurons with weak selectivity, but that it is not linearly decodable. It is right to test if weak supervision signals bleed into the irrelevant subspace, but I could not follow the explanations. Why, for instance, is the ANN decoder on raw data (I assume this is a decoder trained fully supervised) not equal in performance to the revenant distilled signals? Should a well-trained non-linear decoder not simply yield a performance ceiling? Next, if I understand correctly, distilled signals were obtained from the full model. How does a model perform trained only on the weakly tuned neurons? Is it possible that the subspaces obtained with the model are just not optimally aligned for decoding? This could be a result of limited identifiability or model specifics that bias reconstruction to averages (a well-known problem of VAEs). I, therefore, think this analysis should be complemented with tests that do not depend on the model.

Finally, a more technical issue to note is related to the choice to learn a non-parametric prior instead of using a conventional Gaussian prior. How is this implemented? Is just a single sample taken during a forward pass? I worry this may be insufficient as this would not sample the prior well, and some other strategy such as importance sampling may be required (unless the prior is not relevant as it weakly contributed to the ELBO, in which case this choice seems not very relevant). Generally, it would be useful to see visualisations of the latent variables to see how information about behaviour is represented by the model.

Summary:

This paper presents a very interesting analysis, but I have several concerns as to well the analysis supports the main conclusions. I think the work could benefit from an additional complementary analysis that seeks to confirm with another method if weakly tuned neurons indeed show an encoding that differs qualitatively from the strongly tuned ones.

[1] Hurwitz, Cole, et al. "Targeted neural dynamical modeling." Advances in Neural Information Processing Systems 34 (2021): 29379-29392.
[2] Hyvarinen, Aapo, Ilyes Khemakhem, and Hiroshi Morioka. "Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning." arXiv preprint arXiv:2303.16535 (2023).

Reviewer #2 (Public Review):

Li et al present a method to extract "behaviorally relevant" signals from neural activity. The method is meant to solve a problem which likely has high utility for neuroscience researchers. There are numerous existing methods to achieve this goal some of which the authors compare their method to, though there are notable omissions. However, I do believe that d-VAE is a promising approach that has its own advantages.

That being said, there are issues with the paper as-is. This could have been a straightforward "methods" paper describing their approach and validating it on different ground truth and experimental datasets. Instead, the authors focus on the neuroscientific results and their implications for brain mechanisms. Unfortunately, while the underlying method seems sound and performs well relative to the assessed competition, the scientific results and presentation they put forward were not sufficiently strong to support these claims, especially given the small amount of data (recordings of one monkey per task, with considerable variability between them).

Specific comments
- Is the apparently increased complexity of encoding vs decoding so unexpected given the entropy, sparseness, and high dimensionality of neural signals (the "encoding") compared to the smoothness and low dimensionality of typical behavioural signals (the "decoding") recorded in neuroscience experiments? This is the title of the paper so it seems to be the main result on which the authors expect readers to focus.

- I take issue with the premise that signals in the brain are "irrelevant" simply because they do not correlate with a fixed temporal lag with a particular behavioural feature hand-chosen by the experimenter. As an example, the presence of a reward signal in motor cortex [1] after the movement is likely to be of little use from the perspective of predicting kinematics from time-bin to time-bin using a fixed model across trials (the apparent definition of "relevant" for behaviour here), but an entire sub-field of neuroscience is dedicated to understanding the impact of these reward-related signals on future behaviour. Is there method sophisticated enough to see the behavioural "relevance" of this brief, transient, post-movement signal? This may just be an issue of semantics, and perhaps I read too much into the choice of words here. Perhaps the authors truly treat "irrelevant" and "without a fixed temporal correlation" as synonymous phrases and the issue is easily resolved with a clarifying parenthetical the first time the word "irrelevant" is used. But I remain troubled by some claims in the paper which lead me to believe that they read more deeply into the "irrelevancy" of these components.

- The authors claim the "irrelevant" responses underpin an unprecedented neuronal redundancy and reveal that movement behaviors are distributed in a higher-dimensional neural space than previously thought." Perhaps I just missed the logic, but I fail to see the evidence for this. The neural space is a fixed dimensionality based on the number of neurons. A more sparse and nonlinear distribution across this set of neurons may mean that linear methods such as PCA are not effective ways to approximate the dimensionality. But ultimately the behaviourally relevant signals seem quite low-dimensional in this paper even if they show some nonlinearity may help.

- Relatedly, I would like to note that the exercise of arbitrarily dividing a continuous distribution of a statistic (the "R2") based on an arbitrary threshold is a conceptually flawed exercise. The authors read too much into the fact that neurons which have a low R2 w.r.t. PDs have behavioural information w.r.t. other methods. To this reviewer, it speaks more about the irrelevance, so to speak, of the preferred direction metric than anything fundamental about the brain.

- there is an apparent logical fallacy that begins in the abstract and persists in the paper: "Surprisingly, when incorporating often-ignored neural dimensions, behavioral information can be decoded linearly as accurately as nonlinear decoding, suggesting linear readout is performed in motor cortex." Don't get me wrong: the equivalency of linear and nonlinear decoding approaches on this dataset is interesting, and useful for neuroscientists in a practical sense. However, the paper expends much effort trying to make fundamental scientific claims that do not feel very strongly supported. This reviewer fails to see what we can learn about a set of neurons in the brain which are presumed to "read out" from motor cortex. These neurons will not have access to the data analyzed here. That a linear model can be conceived by an experimenter does not imply that the brain must use a linear model. The claim may be true, and it may well be that a linear readout is implemented in the brain. Other work [2,3] has shown that linear readouts of nonlinear neural activity patterns can explain some behavioural features. The claim in this paper, however, is not given enough

- I am afraid I may be missing something, as I did not understand the fano factor analysis of Figure 3. In a sense the behaviourally relevant signals must have lower FF given they are in effect tied to the temporally smooth (and consistent on average across trials) behavioural covariates. The point of the original Churchland paper was to show that producing a behaviour squelches the variance; naturally these must appear in the behaviourally relevant components. A control distribution or reference of some type would possibly help here.

- The authors compare the method to LFADS. While this is a reasonable benchmark as a prominent method in the field, LFADS does not attempt to solve the same problem as d-VAE. A better and much more fair comparison would be TNDM [4], an extension of LFADS which is designed to identify behaviourally relevant dimensions.

[1] https://doi.org/10.1371/journal.pone.0160851
[2] https://doi.org/10.1101/2022.03.31.486635
[3] https://doi.org/10.1038/s41593-017-0028-6
[4] Hurwitz et al, Targeted Neural Dynamical Modeling, NeurIPS 2021.

Reviewer #3 (Public Review):

The authors develop a variational autoencoder (VAE), termed d-VAE (or distill VAE) that aims to tease apart the behaviorally relevant and irrelevant sections of each neuron's firing rate. The input to the VAE is the population activity for a given time step, and the output is the inferred behaviorally relevant section of the population activity at that time step. The residual is referred to as behaviorally irrelevant: total neural activity = behaviorally relevant + behaviorally irrelevant (x = x_r + x_i). The mapping from the raw neural signals (x) to the bottlenecked latent in the autoencoder (called z, z=f(x)) and back to the inferred behaviorally relevant single-neuron activities (x_r = g(z)) is applied per time step (does not incorporate any info from past/future time steps) and, critically, it is nonlinear (f and g are nonlinear feedforward neural networks). The key technical novelty that encourages x_r to encode behaviorally relevant information is a term added to the loss, which penalizes bad linear behavior decoding from the latent z. Otherwise the method is very similar to a prior method called pi-VAE, which should be explained more thoroughly in the manuscript to clearly highlight the technical novelty.

The authors apply their method to 3 non-human primate datasets to infer behaviorally relevant signals and contrast them with the raw neural signals and the residual behaviorally irrelevant signals. As a key performance metric, they compute the accuracy of decoding behavior from the inferred behaviorally relevant signals (x_r) using a linear Kalman filter (KF) or alternatively using a nonlinear feed forward neural network (ANN). They highlight 3 main conclusions in the abstract: first, that single neurons from which behavior is very poorly decodable do encode considerable behavior information in a nonlinear manner, which the ANN can decode. Second, they conclude from various analyses that behavior is occupying a higher dimensional neural space than previously thought. Third, they find that linear KF decoding and nonlinear ANN decoding perform similarly when provided with the inferred behaviorally relevant signals (x_r), from which they conclude that a linear readout must be performed in motor cortex.

The paper is well-written in many places and has high-quality graphics. The questions that it aims to address are also of considerable interest in neuroscience. However, unfortunately, several main conclusions, including but not limited to all 3 conclusions that are highlighted in the abstract, are not fully supported by the results due to confounds, some of which are fundamental to the method. Several statements in the text also seem inaccurate due to use of imprecise language. Moreover, the authors fail to compare with some more relevant existing methods that are specifically designed for extracting behaviorally relevant signals. In addition, for some of the methods they compare with, they do not use an appropriate setup for the benchmark methods, rendering the validation of the proposed method unconvincing. Finally, in many places imprecise language that is not accompanied with an operational definition (e.g., smaller R2 [of what], similar [per what metric]) makes results hard to follow, unless most of the text is read very carefully. Some key details of the methods are also not explained anywhere.