Split-trial analysis recovers information-limiting noise.

(a) Overview of split-trial analysis. The neural population is first partitioned into two halves, which are both used to train individual decoders on the same set of trials. The decoding errors from both decoders are then leveraged against each other to separate shared (information-limiting) noise from noise that is private to each split. (b) Simulation pipeline for evaluating the methods. (c,d,e) In this case, we construct a neural population code (Code 1) based on tuning curves and Poisson spiking noise. (c) shows the distribution of the noise correlations, which is centered around 0. (d) shows the scatter plot of the two decoders from the two splits. (e) The inferred variance of the information-limiting noise for split-trial analysis and a naive estimator based on direct decoding. The ground truth information-limiting noise is 0 in this case. (f,g,h) Similar convention as (c,d,e), but for a neural population code (Code 2) specified by tuning curves, Poisson spiking noise, and a population-wide gain fluctuation. No information-limiting noise is added. (i,j,k) Similar convention as (c,d,e), but for a neural population code (Code 3) specified by tuning curves, Poisson spiking noise, and non-zero information-limiting noise (σ = 5 deg) added through corrupting the stimulus.

Benchmarking the performance of split-trial analysis in recovering the ground truth d-prime in binary classification tasks.

(a) Scatter plot of the decision variable (DVs) for classifiers trained on two separate subpopulations of simulated neural datasets when d-prime =2. The distributions for each class are shown in gold and purple, and it can be observed that in higher noise (low d’) regimes, correlation between the two splits is more strongly observed. (b,c) Similar to (a), but for d-prime =6, 10, respectively. (d) We estimated d’ in simulated data using a variety of different approaches to estimate information limiting noise, using 500 samples/stimulus, when d-prime =2. (e,f) Similar to (d), but for d-prime = 6 and 10, respectively.

Results on head direction cells.

Data reanalyzed from [33].(a) According to the ring attractor model commonly used to model head-direction cells, noise in the internal compass of the animal should manifest in a way identical to information limiting noise. (b) Histogram displaying the estimated information-limiting noise σ over 42 head direction population recording sessions from [34]. Decoding was done using a ZIG decoder as in the original work, and we estimate a mean information-limiting σ of ≈ 8.1 deg. (c) Plotting the estimated information limiting σ on the x-axis, and the estimate from a naive decoder (such as a Bayesian one) on the y-axis, we observe that our technique not only reports a generally lower information limit, but also that the variance of our estimator is substantially lower for this system. (d) Scaling in estimates of information-limiting σ as cell population size varies in four example sessions for both the split-trial analysis and a naive estimator. (e) 2D histograms of the error density across two subpopulations for a single iteration used to generate the plots in (d), with Pearson’s correlation value noted.

Results on the orientation code in mice V1 from two different experimental paradigms.

Data reanalyzed from [27]. (a) Schematic of the uniform stimulus sampling method. (b) Example scatter plots of the subpopulation errors for sessions from the static orientation and dense discrimination datasets, respectively. The modest degree of correlation in both indicates the presence of information-limiting correlations in the considered datasets. Data from from [27]. (c) An example session’s estimated information-limiting σ plotted against population size for static orientation and dense discrimination, respectively. (d) The estimated information-limiting σ for the static orientation and dense discrimination datasets, taken at the full population sizes for each session. One session was not visualized due to its large error bar. See Fig. S10 for results for all 6 individual sessions. (e-g). Similar to (a-d), but for another dataset that densely sample the stimuli from a small range, i.e., [43,47] deg.

Results on the neural code in macaque PFC during a left/right saccade task.

Data reanalyzed from [25]. (a) Schematic of the structure of an individual trial. (b) Schematic of the analysis procedure. We gradually decrease the length of the time window used for the analysis. For each time window, we inferred the d-prime value based on split trial analysis and the standard classification analysis (naive approach). (c) Estimated d’ limit for PFC recording sessions from Bartolo et al., dataset consisted of two animals (labeled ‘V’ and ‘W’) across recording sessions for each animal. (d) Example DV scatter for a trial from session V1. (e) d-prime values in the limit of infinite number of neurons was estimated using both the split trial analysis (solid lines) and the naive approach (dashed lines) for increasingly large spike integration windows for the time period starting 300ms before reward delivery on successful trials. Each plot shows change in d’ limit as integration window is widened across different total population sizes (colors in legends), normalized by the mean value at the widest integration window. The curves in this panel indicate that, in this dataset, longer integration times do not substantively change the split trial method’s estimation of the information limit of the system in this task, as is the case for the naive approach. This indicates a strong and stable signal of information can be recovered in PFC with as little as 20ms spike data from the split analysis approach. (f-i) Individual plots for each session, in the same manner as panel (e).As was the case in panel (e), curves displayed estimated d’ using 25, 50, and all cells recorded in the session.

Histograms of the Pearson correlation values between the two splits for the three neural population codes studied in Fig 1.

For each code, we perform split-trial analysis by generating simulated response, decoding the two splits and calculating the correlation between the decoders from the two splits. The results show that when information-limiting noise is 0, correlation values are generally around 0. In the presence of information-limiting noise, correlation values are generally positive.

Split-trial analysis for another neural population code

We further constructed a fourth neural population code (Code 4) by incorporating both information-limiting noise (σ = 5 deg) and shared gain fluctuations (non-information limiting). We find that split-trial analysis can efficiently recover the information-limiting noise in the presence of both information-limiting and non-information limiting noise.

Validation of deconvolution based approach for estimating information-limiting noise.

Data is simulated in similar schemes as Fig. 3, but the information limiting σ is estimated using a deconvolution approach.Top row: data generated with a ground truth information-limiting σ = 3, bottom row: data generated with a ground truth information-limiting σ = 10. (a,d) The sum and difference of errors across both splits. (b,e) The estimated error distributions for the two datasets, with the estimated σ and MAE listed. (c,f) Using the deconvolving filter estimating the error distribution, we can reconstruct the original error sum distribution (solid blue) and compare it against the actual original distribution (dashed).

Benchmarking the performance of split-trial analysis and three alternative methods on inferred the ground truth d-prime (related to Fig. 2).

This figure shows the results for all sample/stim and cell population combinations used (related to Fig. 2). Note that the portions of the curves for which the method from [23] disappears due to a preponderance of unbounded information estimates.

Histograms of estimated d-prime limit for the method from [23] based on extrapolation.

The ground truth d-prime in this case is 2. For each panel, the median of the estimated d-prime values across 100 simulations was shown on the top. Note that, when there are few samples, the linear extrapolation method is highly unstable. Additionally, certain combinations of large cell count and large samples may be biased towards classifying a population as having unbounded information. This is likely due to the particular bias-corrected fisher information calculation developed used in the original report. When the estimated information is unbounded, such estimates were not shown in the plots (thus the sum of the counts in the histogram may be smaller than 100). In some case, none of the 100 simulations results in a finite estimate of the d-prime values, resulting in all NANs.

Histograms of estimated d-prime limit for the method from [23].

The ground truth d-prime in this case is 6. Same convention as Fig. S5.

Histograms of estimated d-prime limit for the method from [23].

The ground truth d-prime in this case is 10. Same convention as Fig. S5.

Individual session estimates for the information-limiting σ from [34] for both split and naive approaches

Plots were started from intermediate population sizes for visualization purposes.

Individual session estimates for the median absolute error (MAE) of information-limiting niose from [34].

Red: results based on split-trial analysis based on the deconvolution approach. Blue: results based on direct decoding using a ZIG Bayesian decoder[41]. Plots were started from intermediate population sizes for visualization purposes.

Results on the inferred standard deviation of information-limiting noise from the uniformly sampled orientation condition.

Results from 6 individual sessions were shown. Red: estimates from split-trial analysis. Blue: estimates from the direct (naive) decoding analysis. We observed that the naive estimate of σ for session 3 increases slightly after ∼ 10,000 cells. The odd level of variability in this session made display challenging so was excluded from the visualization of Fig. 4c. Error bars are standard deviation over all iterations.

Results on the inferred standard deviation of information-limiting noise from the densely sampled orientation condition.

Red: estimates from split-trial analysis. Blue: estimates from the direct (naive) decoding analysis.

Results on the inferred standard deviation of information-limiting noise from the densely sampled orientation condition when the dataset was downsampled.

Data from the dense discrimination sessions was downsampled to a range of different stimulus densities, up to 300, which is about the full dataset density. Notably, the split-trial estimates are stable and no longer appear to decrease after ∼150 samples/deg, suggesting that additional experimental trials would not affect the estimate.

Results on how d-prime values scale with the increasing integration windows.

Results from the split-trial analysis shown as solid line, showing relatively little change due to temporal integration. The four sessions have 770, 828, 509, 529 neurons, respectively.

Average normalized d-prime values across normalized sessions (n=4) when windows are 20ms wide and discrete steps are taken through the 300ms window.

D-prime values were inferred using split-trial analysis based on all neurons from individual sessions. The results show a trend that indicates information-limiting noise increases as reward delivery time approaches.

Non-normalized version of discrete windows individual sessions from the dataset from [25].

Despite early windows having higher values than later windows, they are typically lower than the (final) values recovered from the integrated calculations.