Brain activity measurements as sampling projections of the neural response patterns.

a. Neural activity response patterns can be viewed as points in a multidimensional metric space in which the dimensions correspond to the activity of single neurons. a. Example of a population of three neurons, each gray point represents the population response to one experimental condition. b. Projection of the neural response patterns onto the measured response space. Each point is recoded along a new set of axes given by the measurement channels. c. Measurement channels that reflect the activity of the neural population as weighted averages with only non-negative weights can be viewed as sampling axes that lie on the all-positive orthant. Panel c shows a set of response patterns with a spherical geometry (gray) and two example all-positive sampling axes (red). d. Recoding the points on the sphere along the sampling axes results in a geometry that is linearly stretched along the dimension of the neural population average. e. A set of non-negative sampling axes. Each red vector represents a measurement channel whose entries correspond to the neural sampling weights. These vectors can also be seen as points in the neural response space, represented by the red tips. The fact that all the weights are non-negative makes the sampling axes closely aligned with the all-1 vector.. f. Removing the mean activation from each measured pattern is equivalent to rigidly centering the tips of the original sampling weight vectors on the origin of the neural response space. This corrects the oversampling of the population-mean (all-1) axis, dispersing the effective axes along which the neural patterns are sampled, such that all directions are equally represented in expectation and the neural representational geometry is thus undistorted.

Representational dissimilarity matrices for an example randomly generated geometry and simulated measurements.

The left column shows squared Euclidean distances between the pairs of simulated neuronal patterns (top) and the absolute difference after averaging over all the neurons for each activity pattern (bottom). The middle column shows the distances between the patterns obtained after a random projection (top) and a simulation of voxel sampling (bottom). The right column shows the squared Euclidean distances between the pairs of activity patterns after subtracting the mean over the channels for each individual simulated experimental condition for random projection sampling (top) and simulated voxel sampling (bottom). For random projections (top), note that mean removal amounts to the removal of one of many dimensions of the representational space and does not appreciably alter the apparent representational geometry. For the voxel sampling model (bottom), mean removal corrects the overemphasis on the population-mean dimension (which results from averaging within voxels), revealing the otherwise hidden neural representational geometry.

Correlation between representational distances of ground-truth simulated activity patterns and simulated measured activity patterns.

Plots show the Pearson RDM correlation (vertical axes) between the simulated neural pattern RDM and the simulated measured pattern RDMs for five measurement models (lines) averaged across simulations. The shaded region around each individual line indicates the standard error of the mean across simulations. For each of a sample of randomly generated representational geometries, we simulate measuring the neuronal population varying the number of channels (horizontal axis). Panels show the RDM correlation between: a. the squared Euclidean distance RDM of the ground-truth simulated neuronal patterns and each of the squared Euclidean distance RDMs for simulated measured patterns. b. the squared Euclidean distance RDM of simulated neuronal patterns and the RDM of simulated measurement patterns after mean removal. c. the squared Euclidean distance RDM of the simulated neuronal patterns and the correlation distance RDM of the simulated measurement patterns. d. the RDMs obtained by taking the absolute differences of the average over all the simulated neurons/channels for each activity pattern. e. the RDM of simulated neuronal patterns after mean removal and the RDM of simulated measured patterns after mean removal. f. the correlation distance RDM of the simulated neuronal patterns and the correlation distance RDM of the simulated measurement patterns.

Pattern-mean removal alone is more effective than pattern-mean removal and pattern-variance normalization at correcting the population-mean overemphasis in the measured representational geometry.

We can compute the squared Euclidean distance on raw (red) or normalized measured patterns. Normalization can consist in subtracting the mean across measurement channels (e.g. voxels) from each pattern (beige) or in subtracting the mean and applying divisive normalization to scale the pattern variance to 1 (blue). When the squared Euclidean distance is computed after mean and variance normalization, it equals twice the Pearson correlation distance (derivation in Supplement Section 7.4) and thus yields the same RDM correlations (blue). Bars indicate the RDM correlation between the ground-truth neural RDM (squared Euclidean distance) and the RDMs containing the squared Euclidean distances among the measured patterns. Each subplot corresponds to a a different measurement model. If the measurements sample random projections (weights drawn i.i.d. from a 0-mean Gaussian, an unrealistic scenario shown in the left panel), the squared Euclidean distances of the measurement channels reflect the ground-truth neuronal representational geometry well. Mean removal hardly aflects the RDM correlation, but mean removal and variance normalization (implicit in the correlation distance) performs markedly worse. If the measurements sample non-negatively weighted averages (a scenario comparable to fMRI voxels, center panel), mean removal greatly improves the RDM correlation with the ground-truth neuronal RDM in the estimated RDM. Additional normalization of the pattern variance (or, equivalently, the use of correlation distance) markedly degrades the estimate of the neural representational geometry. If the measurements consist in a random subsample of the neurons (an idealization of the typically somewhat biased samples in neural recording experiments, right panel), the estimate of the representational geometry is highly accurate. Mean removal then slightly degrades the RDM correlation with the ground truth. Mean removal and variance normalization markedly degrade the RDM correlation with the ground truth. The error bars represent the standard error of the mean across simulations with different ground-truth representational geometries

Representational dissimilarity matrices for empirical data and simulated measurement patterns.

Panel a shows the RDM of the distances between the activity patterns generated from cell recordings of 674 neurons in monkey IT [23, 31], which we use as the ground-truth geometry for the simulated measurements. Panel b: Left column shows the RDM of squared Euclidean distances between simulated measurements taken with the subpopulation sampling model (top) and non-negative IWLCS model (bottom). Middle column shows the squared Euclidean distance RDMs taken from the simulated measurements after removing the mean from each measured pattern. The right column shows the RDMs obtained using the correlation distance on the simulated measurements, which implicitly removes the mean, and matches the analyses used in the empirical study.

Correlation of representational dissimilarities between monkey and human IT for empirical data and simulated measurements.

Panel a shows the results from the empirical data in the original study. Each dot marks the correlation between activity patterns for a pair of stimuli as measured in monkey IT (horizontal axis) and human IT (vertical axis). Simulation results are shown in panel b, where voxels were simulated as sampling a neural population with non-negatively weighted means. The simulated neural population encoded a representational geometry as observed in the monkey data, so the neural pattern dissimilarities (correlation distance: 1 — r, horizontal axes) match those of the actual neural recording data (left panel). These ground-truth dissimilarities are plotted against the dissimilarities among thesimulated voxel response patterns (correlation distance: 1 — r, vertical axis). Panel c shows results of a similar simulation, where the measurement model selects a random subset of neurons (subpopulation sampling). The concentration around the diagonal shows that the representational dissimilarities (1—r as before for both sampling) can be quite accurately estimated from a subpopulation of 200 neurons, if these neurons are sampled randomly.

When do measured representational distances reflect the neural representational geometry?

Three scenarios of brain-activity measurement conditions and their consequences for the resulting representational geometry. a. In the theoretical case of random projection sampling, the measured representational dissimilarities are an unbiased estimate of the neural pattern representational dissimilarities. b. For electrophysiological recordings, we consider two scenarios. In the first, the sampled cells are randomly selected from the underlying population, in which case the theoretical results and simulations in this study indicate that the apparent geometry remains undistorted. In the second scenario, where cells are spatially organized and measurements sample only a region of the population, the apparent representational geometry is expected to be distorted in complex ways, requiring separate modeling.. c. When sampling with fMRI voxels, if the underlying neural code is spatially structured at multiple scales, the geometry of the measured patterns will be distorted in complex ways that require modeling. In contrast, if the neural code is unstructured, two cases arise. First, if the mean activation across neurons is the same across all experimental conditions, the RDM derived from the measurements will provide an undistorted estimate of the neuronal RDM. Second, if the population mean varies across conditions, the apparent geometry will be linearly distorted along the population mean dimension. In this case, removing the mean activation from each measured pattern restores the underlying neural RDM in expectation.