Split-trial analysis reveals the information capacity of neural population codes

Reviewer #1 (Public review):

The authors address a difficult and well-known problem in systems/computational neuroscience: how to estimate the magnitude of "information-limiting" noise. Existing approaches (direct Fisher-information estimation, decoding + Cramer-Rao, and large-N extrapolation) are data-hungry and unstable, which has left the field with conflicting empirical estimates across systems.

The central proposal - "split-trial analysis" - is simple and appealing. The recorded population is randomly partitioned into two non-overlapping halves; a decoder (continuous case) or classifier (binary case) is trained on each half using the same trials; and the covariance of the two halves' decoding errors is used to estimate the variance of the information-limiting noise. There is a clean mathematical derivation to support this conclusion (although there are a couple of mathematical errors in the methods section that should be fixed to avoid confusion on the part of the reader).

They benchmark the method in simulation against three prior methods (Moreno-Bote et al. 2014; Rumyantsev et al. 2020; Kafashan et al. 2021) and report substantially better sample efficiency, lower bias, and greater robustness. They then apply the method to three datasets: (1) mouse head-direction cells (Ajabi et al.), (2) mouse V1 (Stringer et al.), and (3) macaque PFC during a saccade task (Bartolo et al.).

This is a strong and timely contribution. The core idea is elegant, and the method appears to be more practical than existing alternatives in the finite-data regime that real experiments occupy. The three applications are well chosen, and each yields a non-trivial, biologically interpretable result. I am strongly supportive of the potential of this paper.

That said, the paper makes several strong empirical claims - most notably that prior V1 estimates were substantial overestimates, and that PFC information-limiting noise is temporally redundant - and the central estimator rests on an independence assumption whose finite-N validity is only partially characterized. Before these claims can be considered well supported, I would like the authors to address the following:

Major Points:

(1) The method relies on a key independence assumption that may not always be satisfied in the regime of finite neurons and trials. The author's main idea is to decompose the residuals of two decoders as follows:

X1 = delta + phi1
X2 = delta + phi2

The covariance is equal to the scale of information limiting noise, Var[delta], plus three terms:

Cov[X1, X2] = Var[delta] + Cov[delta, phi1] + Cov[delta, phi2] + Cov[phi1, phi2].

We can define phi1 as the part of X1 that is orthogonal to delta and likewise define phi2 as the part of X2 that is orthogonal to delta; thus, the cross terms evaluate to zero, and we are left with:

Cov[X1, X2] = Var[delta] + Cov[phi1, phi2]

Now the authors introduce an assumption that Cov[phi1, phi2] = 0. This leaves us with Cov[X1, X2] = Var[delta], but the question is: when is it justified to assume that Cov[phi1, phi2] = 0? For example, it is possible that

phi1 = c(N) * z + e1
phi2 = c(N) * z + e2

where z is another shared noise dimension that is not information limiting and e1 and e2 are truly independent. Here, c(N) is a constant that goes to zero as the number of neurons used to train the decoder, N, goes to infinity. Thus, in the limit of having very large neural populations at hand for the analysis, the author's assumption of Cov[phi1, phi2] = 0 can be justified. If the authors agree with this analysis, it would be nice to (a) flesh it out and include it in the methods / supplementary notes, and (b) to analyze in simulation how good this approximation is in finite N regimes. I suspect that the assumption works in finite N regimes if noise is low-dimensional, but that if there are many additional dimensions of correlation (i.e. many z's above), you will need a very large number of neurons before Cov[phi1, phi2] approaches zero.

Along these lines, another worthwhile analysis would be to report outcomes when the neural populations are sub-sampled further. Intuitively, it should fail once you subsample to only a handful of neurons, e.g. 3, but I'm curious where the breaking point is and whether the decline is graceful.

(2) In point 1, I raised the question of how the method behaves with a finite number of neurons. Another worry is that there is a finite number of trials. In particular, if you train two decoders on the same trials, I would worry that non-information-limiting fluctuations in those trials would induce correlations in the decoders that then would show up as correlations on the held-out test set. A more conservative approach would be to split trials into three disjoint subsets: a training set for decoder A, a training set for decoder B, and a common test set used to compute Cov[X1, X2].

As a concrete example, suppose that on the particular trials used for training, the animal happened to be more aroused when theta = 1 and less aroused when theta = 0, and that arousal added a fluctuation on top of the neural response. This arousal-related signal is not information-limiting - it would average away given enough trials - but because both decoders are fit to these same trials, each one adjusts its weights to partially discount the same spurious high-arousal/low-arousal trend. Their weights are now distorted in a correlated way, so when both are applied to the shared test set, their errors covary, and the method reads this shared-training artifact as information-limiting noise.

I think this dynamic should be acknowledged in the text and clarified in more detail. Ideally, simulations could be done to estimate how many trials are needed to average out this sort of confound, and similar to the suggestion in point 1 above, I would be interested in seeing what happens when the authors sub-sample trials before running their analysis. Together with point 1, the feedback is that I'd like to see more about "how many neurons and how many trials" are needed in order to trust your results. Similarly, are there diagnostics or resampling methods (e.g. bootstrapping) that could be helpful for a practitioner to know if they have enough neurons/trials?

(3) Unless I've fundamentally misunderstood something, there is an error on page 17 in the methods. There we find sigma2 = Var[delta] = ... = Cov[phi1, phi2], but I believe this is meant to be Cov[X1, X2]. Indeed, the method assumes that Cov[phi1, phi2] = 0, as discussed in point 1.

Additionally, on page 4, the authors introduce the main quantity as Cov[\hat{theta}_1, \hat{theta}_2] instead of Cov[X1, X2]. However, if theta is changing from trial to trial, then these two quantities are not technically equal to each other, so it would be more accurate to write down the conditioning on theta. That is, assuming conditionally unbiased decoders, Cov[X1, X2] = Cov[\hat{theta}_1, \hat{theta}_2 | theta] for a fixed theta.

More generally, I found it hard to wrap my head around the underlying math on my first read through the paper. The polarization identity, 1/4 * (Var(X1 + X2) - Var(X1 - X2)), seems like a very roundabout way to derive the method. This identity is very helpful for the deconvolution extension, but I would have thought that a simpler and more straightforward derivation would have just used the expansion, Cov[X1, X2] = Var[delta] + Cov[delta, phi1] + Cov[delta, phi2] + Cov[phi1, phi2], as I did in point 1. I suggest the authors revise the mathematical presentation for clarity.

Minor Points

(1) A very nice feature of the authors' method is that they make no parametric assumption on the distribution of noise. This is in contrast to Kanitscheider et al. [12]'s finite-sample bias correction using the inverse-Wishart distribution of $\hat\Sigma^{-1}$, which is derived under an assumption of multivariate Gaussianity. I think it is worth adding a sentence to highlight this feature of the model.

(2) Statistical inference claims (across sessions and population sizes) are supported by reported s.d.'s but no formal tests or confidence-interval-based comparisons. Given that several claims are comparative (split-trial < naive; V1 < prior reports; PFC stable over windows), please add appropriate uncertainty quantification (e.g., bootstrap CIs over sessions) and, where a difference is claimed, a test or effect size.

Reviewer #2 (Public review):

Le and Wei present a novel estimation method for information-limiting correlations. Information-limited correlations are shared noise fluctuations that affect neural encoding, but they can be hard to estimate (even to detect their presence) because they can be very small and buried under other common sources of variability that do not affect encoding. The newly proposed method bypasses two central limitations of previous approaches: extrapolation or assuming the noise structure to be Gaussian. The authors proposed a split-trial analysis where the population is split into two, and the correlations between the decoding errors arising from each population are computed. These correlations provide an unbiased measure of information-limiting correlations. The method is very simple and sound, and it is shown to deliver stable estimates with sensible magnitudes across several brain data sets. Further, even if the decoders are suboptimal, the method can detect the presence of information-limiting correlations, as only shared fluctuations of the two population decoders can possibly be observed if there are correlations that limit information.

Comments:

(1) The name "split-trial analysis" does not seem to reflect well the nature of the method introduced. I would propose something like "split-ensemble analysis" or "split-population decoding-correlation analysis".

(2) Previous work has proposed a related - but different - bootstrap method, which can be mentioned in the current paper (Nogueira et al, J of Neuroscience, 2020).

(3) The authors proposed a deconvolution method to study the shape of the distribution of information-limiting noise. An alternative would be to split neural populations into 3 or more subpopulations and compute 3rd- and 4th-order correlations between the decoding errors. This would lead to estimates of higher-order moments that can be compared to Gaussian ones and test for non-Gaussian distributions. Further, this N-split-ensemble method could be used to compare the deconvolution method results to test their consistency.

Reviewer #3 (Public review):

Summary:

In this manuscript, Le and Wei proposed a new method to identify differential correlations in real recordings (and simulations) that is based on splitting the simultaneously recorded population of neurons into two disjoint subpopulations. The method is based on evaluating the correlation between the decoded stimulus for each sub-population across trials. The authors validate their method on simulations and find the magnitude of differential correlations on three different publicly available datasets.

Strengths:

We think that this is a solid and relevant study for the computational neuroscience community, especially for the originality of the method and the fact that it seems to bypass the problem of very large populations to identify differential correlations. Overall, the results are novel and significant, and it addresses an important gap in the field. The main results are presented clearly and are easy to follow.

Weaknesses:

However, we believe that there are some additional analyses and clarifications that should be made to increase the clarity and impact of this study. In general, we believe that the authors should make a better effort to explain how their novel method depends on the number of trials and the number of neurons. More specifically:

Major

(1) The authors should show a realistic case for the covariance matrix in Figure 1. Currently, they are showing only Poisson noise (Figure 1c-e), only gain + Poisson (Figure 1f-h), and only differential correlations + Poisson (Figure 1i-k). They should show these same plots with a biologically realistic non-differential correlation structure (limited-range correlations, see Kanitscheider PNAS 2015). Perhaps even show the case for limited-range + gain + differential correlations. They should do the same for Figure 2.

(2) Throughout the manuscript, the role of population size (N) on the method is a bit confusing. Figures 1 and 2 give the impression that N is not particularly important, which is counterintuitive and surprising. We understand that that is one of the strengths of the split-trial method, but the authors should explain in much more detail in the results and methods the role of population size on their novel method. Why is large N crucial for the other methods, but not for them? There is a little bit of population-size dependency on Figures 3-5, especially on Figure 4g. The authors should explain in more detail those effects.

(3a) For dataset [27], the stimulus density was ~12 samples per deg for uniform sampling and ~1000 samples per deg for dense sampling. Figure S12 shows an overestimate of information-limiting noise when the number of trials used was significantly downsampled, which is, first of all, in disagreement with simulation results showing "when only a small number of trials are available to infer a large d-prime, split-trial analysis exhibits an under-estimation". It is true that we are not strictly in a binary classification task setting, but we are wondering if the authors have any justification for this result for [27].

(3b) Related to this point, the estimated info-limiting noise was 0.26 deg with all neurons and 0.6 deg with downsampling (we guess that is the first value of red lines in Figure S12). The only difference here, if we understand correctly, is the number of trials used. Otherwise, it's exactly the same neural responses used for estimation. So, a similar magnitude should be expected. If the latter is due to an insufficient number of trials used, would the same problem apply to the uniform sampling dataset? In other words, if there were more trials recorded with uniformly sampled stimuli, would the authors expect to see a further and significant decrease of sigma as well?