Experimental neuroscience has recently made rapid progress with technologies for measuring neural population activity. Spatial and temporal resolution, as well as the coverage of measurements across the brains of animals and humans have all improved considerably (Parvizi and Kastner, 2018; Abbott et al., 2020; Wang and Xu, 2020; Allen et al., 2021; Guo et al., 2021; Uğurbil, 2021; Bandettini et al., 2021). Activity is measured using a wide range of techniques, including electrode recordings (Jun et al., 2017; Steinmetz et al., 2018; Parvizi and Kastner, 2018), calcium imaging (Wang and Xu, 2020), functional magnetic resonance imaging (fMRI; Allen et al., 2021; Uğurbil, 2021; Bandettini et al., 2021), and scalp electro- and magnetoencephalography (EEG and MEG; Baillet, 2017; Craik et al., 2019). In parallel to the advances in measuring brain activity, theoretical neuroscience has substantially scaled up brain-computational models that implement computational theories (e.g. Kriegeskorte, 2015; Kell et al., 2018; Kubilius et al., 2019; Zhuang et al., 2021). The engineering advances associated with deep learning (e.g. Paszke et al., 2019; Abadi et al., 2015) provide powerful tools for modeling brain information processing for complex, naturalistic tasks (LeCun et al., 2015). How to leverage the new big data to evaluate the new big models, however, is an open problem (Stevenson and Kording, 2011; Sejnowski et al., 2014; Smith and Nichols, 2018; Kriegeskorte and Douglas, 2018).

An important concept for understanding neural population codes is the concept of representational geometry (Shepard and Chipman, 1970; Edelman et al., 1998; Edelman, 1998; Norman et al., 2006; Diedrichsen and Kriegeskorte, 2017; Kriegeskorte et al., 2008a; Kriegeskorte et al., 2008b; Connolly et al., 2012; Xue et al., 2010; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al., 2014; Cichy et al., 2014; Haxby et al., 2014; Freeman et al., 2018; Kietzmann et al., 2019; Stringer et al., 2019; Chung et al., 2018; Chung and Abbott, 2021; Kriegeskorte and Wei, 2021). Neural activity patterns that represent particular pieces of mental content, such as the stimuli presented in a neurophysiological experiment, can be viewed as points in the multivariate neural population response space of a brain region. The representational geometry is the geometry of these points. The geometry is characterized by the matrix of distances among the points. This distance matrix abstracts from the roles of individual neurons and provides a summary characterization of the neural population code that can be directly compared among animals and between brain and model representations (e.g. a cortical area and a layer of a neural network model). The representational geometry provides a multivariate characterization of a neural population code that can be motivated as a generalization of linear decoding analyses. A linear decoder reveals a single projection of the geometry. The full distance matrix (when measured after a transform that renders the noise isotropic) captures what information is available in any linear projection (Kriegeskorte and Diedrichsen, 2019a).

A popular method for analyzing representational geometries (Kriegeskorte and Kievit, 2013) on which we build here is representational similarity analysis (RSA; Kriegeskorte et al., 2008a; Nili et al., 2014). RSA is a three-step process (Figure 1): In the first step, RSA characterizes the representational geometry of the brain region of interest (ROI) by estimating the representational distance for each pair of experimental conditions (e.g. different stimuli). The distance estimates are assembled in a representational dissimilarity matrix (RDM). We use the more general term ‘dissimilarity’ here to include dissimilarity measures that are not distances or metrics in the mathematical sense, such as crossvalidated distance estimators that can return negative values. This relaxation enables inclusion of measures that are not biased by the noise in the data (Kriegeskorte et al., 2007; Nili et al., 2014; Walther et al., 2016; Kriegeskorte and Diedrichsen, 2019a), returning values distributed symmetrically about 0, when the true distance is 0, but patterns are noisy estimates. In the second step, each model is evaluated by the accuracy of its prediction of the data RDM. To this end, an RDM is computed for each model representation. Each model’s prediction of the data RDM is evaluated using an RDM comparator, such as a correlation coefficient. In the third step, models are inferentially compared to each other in terms of their RDM prediction accuracy to guide computational theory.

Figure 1

Download asset Open asset

RSA is widely used (Kriegeskorte and Kievit, 2013; Haxby et al., 2014; Kriegeskorte and Diedrichsen, 2019a) and has gained additional popularity with the rise of image-computable representational models like deep neural networks (e.g. Krizhevsky et al., 2012; Yamins et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Mehrer et al., 2017; Kriegeskorte, 2015; Yamins and DiCarlo, 2016; Xu and Vaziri-Pashkam, 2021; Konkle and Alvarez, 2022; Cichy et al., 2016). There has been important recent progress with methods for estimating representational distances (step 1) as well as measures of RDM prediction accuracy (step 2). For RDM estimation, biased and unbiased distance estimators with improved reliability have been proposed (Nili et al., 2014; Cai et al., 2019; Walther et al., 2016). For quantification of the RDM prediction accuracy, the sampling distribution of distance estimators has been derived and measures of RDM prediction accuracy that take the dependencies between dissimilarity estimates into account have been proposed (Diedrichsen et al., 2020). However, existing statistical inference methods for RSA (step 3) have important limitations. Established RSA inference methods (Nili et al., 2014) provide a noise ceiling and enable comparisons of fixed models with generalization to new subjects and conditions. However, they cannot handle flexible models, can be severely suboptimal in terms of statistical power, and have not been thoroughly validated using simulated or real data where ground truth is known. Addressing these shortcomings poses three substantial challenges. (1) Model-comparative inference with generalization to new conditions is not trivial because new conditions extend an RDM and the evaluation depends on pairwise dissimilarities, thus violating independence assumptions. (2) Standard methods for statistical inference do not handle multiple random factors — subjects and conditions in RSA. (3) Flexible models, that is models that have parameters enabling them to predict different RDMs, are essential for RSA (Diedrichsen et al., 2018; Kriegeskorte and Diedrichsen, 2016). Evaluation of such models requires methods that are unaffected by overfitting to either subjects or conditions to avoid a bias in favor of more flexible models.

Here, we introduce a comprehensive methodology for statistical inference on models that predict representational geometries (Figure 1). We introduce novel bootstrapping methods that support generalization of model-comparative statistical inferences to new subjects, new conditions, or both simultaneously, as required to support the theoretical claims researchers wish to make. We also introduce a novel crossvalidation method for estimation of the RDM prediction accuracy of flexible models, that is models with parameters fitted to the data (Khaligh-Razavi and Kriegeskorte, 2014; Kriegeskorte and Diedrichsen, 2016). This is important, because theories do not always make a specific prediction for the representational geometry. There may be unknown parameters, such as the relative prevalences of different tuning functions (Khaligh-Razavi and Kriegeskorte, 2014; Jozwik et al., 2016) in the neural population or properties of the measurement process (Kriegeskorte and Diedrichsen, 2016). The combination of our 2-factor bootstrap and 2-factor crossvalidation methods enables statistical comparisons among fixed and flexible models that generalize across subjects and conditions.

We thoroughly validate the new inference methods using simulations and neural activity data. Extensive simulations based on deep neural network models and models of the measurement process enable us to test model-comparative inference in a setting where the ground-truth model (the one that actually generated the data) is known. These simulations confirm the validity of the inference procedures and their ability to generalize to the populations of subjects and/or conditions. We also validated the methods on real data from calcium imaging (mouse) and functional MRI (human). For both datasets, we confirm that conclusions generalize from an experimental dataset (a subset of the real data) to the entire dataset (which serves as a stand-in for the population). The statistical inference methodology described in this paper is available in a new open-source RSA toolbox written in Python (https://github.com/rsagroup/rsatoolbox, copy archived at Schütt, 2023).

We now introduce the 2-factor bootstrap procedure for model-comparative inference and the 2-factor crossvalidation procedure for unbiased evaluation of flexible models. This paper also introduces a new representational dissimilarity estimator for electrophysiological recordings of patterns of firing rates across a population of neurons, based on the KL-divergence between Poisson distributions (Appendix 2) and a faster alternative to the rank correlation ${\tau }_a$ as an RDM comparator (Nili et al., 2014), which we call ${\rho }_a$ (Appendix 3). The proposed inferential methods work for any representational dissimilarity measure and any RDM comparator. We evaluate alternative RDM comparators in terms of their power in Appendix 6. A complete description of all steps of the new methodology can be found in the Materials and methods (Full description of the RSA method).

Methods for inference on representational geometries

A simple approach to inferential comparison of two models is to compute the difference between the models’ performance estimates for each subject and use Student’s $t$-test (or a nonparametric alternative). However, inference then only takes the variability over subjects into account and thus does not justify generalization to different experimental conditions (e.g. different stimuli). Computational neuroscience usually pursues insights that generalize not only to a population of subjects but also to a population of conditions (Yarkoni, 2020). To support generalization to the population of conditions statistically, we require uncertainty estimates that treat the experimental conditions as a random sample from a population (Kriegeskorte et al., 2008a), whether or not the subjects are treated as a random sample.

For frequentist inference, the challenge is to estimate how variable the model-performance estimates would be if we repeated the experiment many times with new subjects and/or conditions. We would like to know (1) the variance of each model’s performance estimate and (2) the variance of the estimated performance difference for each pair of models. The variance of model-performance estimates enables us to statistically compare each model to a fixed value such as an RDM correlation of 0. The variance of our estimate of model-performance difference enables us to statistically compare two models to each other (see Frequentist tests for model evaluation and model comparison for details).

Estimating the variance of model-performance estimates for generalization to new subjects and conditions

To estimate the variance of model-performance estimates across repetitions of the experiment with new conditions, we use a bootstrap method. Bootstrap methods estimate the variance of experimental outcomes by sampling from the measured data with replacement, treating the measured data as an approximation to the population (Efron and Tibshirani, 1994). The population here is the set of experimental conditions of which the actual experimental conditions can be considered a random sample. Because the conditions do not have independent influences on the model evaluations, we cannot compute a sample variance across conditions as we can across subjects to replace the bootstrap.

When we bootstrap-resample conditions, we obtain RDMs of the same size as the original RDMs, but some of the conditions will be repeated. Here, we exclude the entries that correspond to the dissimilarity of any condition with itself from the comparisons between RDMs. Simulations confirm that this procedure yields a good estimate of how variable the results are when we sample new conditions with the same subjects (Figures 4a and 6g).

For simultaneous generalization to the populations of both conditions and subjects, we can employ a 2-factor bootstrap (Figure 1b) as introduced previously (Nili et al., 2014; Storrs et al., 2021). However, our simulations and theory here show that a naive 2-factor bootstrap approach triple-counts the variance contributed by the measurement noise (Methods, Estimating the uncertainty of our model-performance estimates, Figures 4c and 7c). This effect is not unique to RSA; a naive 2-factor bootstrap will triple-count variance related to the measurement noise for any type of experiment in which two factors (here subject and condition) jointly determine the experimental outcome. The true variance ${\sigma }_{both}^2$ of the experimental outcome when sampling both factors can be separated into a contribution from condition sampling (${\sigma }_{cond}^2$), a contribution from subject sampling (${\sigma }_{subj}^2$), and a contribution of the interaction of subjects and conditions or measurement noise (${\sigma }_{noise}^2$).

(1) ${\displaystyle \begin{array}{rl}{\sigma }_{both}^2& \approx {\sigma }_{subj}^2+{\sigma }_{cond}^2+{\sigma }_{noise}^2\end{array}}$

This decomposition is for the actual variance ${\sigma }_{both}^2$ across repeated experiments with new subjects and conditions. The variance ${\widehat{\sigma }}_{both}^2$ of the naive 2-factor bootstrap can likewise be decomposed into three additive terms (Online Methods, Estimating the uncertainty of our model-performance estimates), corresponding to subject sampling, condition sampling, and the interaction and/or noise. However, in the naive 2-factor bootstrap estimate ${\widehat{\sigma }}_{both}^2$, the independent noise contribution enters not only its own term, but also the two others. Thus, the original bootstrap estimate contains the noise variance component three times instead of once:

(2) ${\displaystyle \begin{array}{rl}{\hat{\sigma }}_{both}^2& \approx ({\sigma }_{subj}^2+{\sigma }_{noise}^2)+({\sigma }_{cond}^2+{\sigma }_{noise}^2)+{\sigma }_{noise}^2\\ & ={\sigma }_{subj}^2+{\sigma }_{cond}^2+3{\sigma }_{noise}^2\end{array}}$

This problem can be understood by considering the 1-factor bootstraps, which also contain the independent noise component although it has not been added explicitly:

(3) ${\displaystyle \begin{array}{rl}{\hat{\sigma }}_{subj}^2& \approx {\sigma }_{subj}^2+{\sigma }_{noise}^2\end{array}}$

(4) ${\displaystyle \begin{array}{rl}{\hat{\sigma }}_{cond}^2& \approx {\sigma }_{cond}^2+{\sigma }_{noise}^2\end{array}}$

When we bootstrap two factors, this automatic inclusion of the noise component happens three times. We confirmed this by both theory and simulation. The overestimate of the variance renders the naive 2-factor bootstrap conservative and not optimally powerful.

To correct the variance estimate, we introduce a novel corrected 2-factor bootstrap procedure to estimate the variance: We first compute the 1-factor bootstrap variance estimates ${\widehat{\sigma }}_{subj}^2$ and ${\widehat{\sigma }}_{cond}^2$. We also compute the naive 2-factor bootstrap estimate ${\widehat{\sigma }}_{both}^2$. We can then linearly combine the variances from these three bootstraps to cancel the surplus contribution from the measurement noise. This procedures yields a corrected 2-factor bootstrap estimate ${\widehat{\sigma }}_{c2f}^2$ that has approximately the right expected value:

(5) ${\displaystyle \begin{array}{rl}{\hat{\sigma }}_{c2f}^2& =2({\hat{\sigma }}_{subj}^2+{\hat{\sigma }}_{cond}^2)-{\hat{\sigma }}_{both}^2\\ & \approx {\sigma }_{subj}^2+{\sigma }_{cond}^2+{\sigma }_{noise}^2\end{array}}$

The approximations in these equations are due to $\frac{N-1}{N}$ factors that apply to the individual terms. We give the exact formulae including these factors in the methods section (Estimating the uncertainty of our model-performance estimates). We show in multiple simulations that this estimate approximates the correct variance better than the uncorrected 2-factor bootstrap (Figures 4c and 7c).

To stabilize the estimator and eliminate the possibility of a negative variance estimate, we bound the estimate from above and below. We use both ${\widehat{\sigma }}_{subj}^2$ and ${\widehat{\sigma }}_{stim}^2$ as lower bounds for the estimate as the variances they estimate are always smaller than the true variance. As an upper bound, we use ${\widehat{\sigma }}_{both}^2$, the naive, conservative estimate. Bounding slightly biases the variance estimate, but reduces its variability and ensures that it is strictly positive.

Evaluating the performance of flexible models

We often want to test flexible models, that is models that have parameters to be fitted to the brain-activity data. Two elements that often require fitting are weights for the model features and parameters of a measurement model. Feature weighting is required when a model is not meant to specify a priori how prevalent different tuning profiles are in the neural population or in the measured signals. For example, for deep neural network representations to match brain responses well, it is usually necessary to weight the features (e.g. Yamins et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Khaligh-Razavi et al., 2017; Storrs et al., 2021). A flexible measurement model may be necessary to account for the process of measurement, which may subsample, average, or distort neural responses. For example, fMRI voxels average the neural activity locally, which can be modeled with a parameter for the local averaging range, and electrophysiological recordings may preferentially sample certain classes of neurons (Kriegeskorte and Diedrichsen, 2016).

To avoid the bias in the model-performance estimates that can result from overfitting of flexible models, we use crossvalidation. Crossvalidation means that we partition the dataset into separate test sets. In each fold of crossvalidation, we then fit the models to all but one set and evaluate on the held-out set. Taking the average over the folds yields a single performance estimate. As for bootstrapping, crossvalidation is performed over both conditions and subjects so as to avoid overestimating the generalization performance of flexible models when tested on new subjects and new conditions drawn from the populations of subjects and conditions sampled in the actual experiment (Figure 1b).

Because the RDM for the test set must contain multiple values to allow a sensible comparison, the smallest possible number of conditions to perform crossvalidation is 6, which would yield three test conditions for twofold crossvalidation. For small numbers of conditions, we use twofolds. We use threefolds for ≥12 conditions, fourfolds for ≥24 conditions, and fivefolds for ≥40 conditions. These numbers seem to work reasonably well, but were chosen ad hoc.

To estimate our uncertainty about the crossvalidated model performances, we use the same bootstrap methods as for fixed models. To do so, we need to perform crossvalidation on each bootstrap sample. We call this procedure bootstrap-wrapped crossvalidation.

In any crossvalidation, different ways to partition the data into test sets lead to different overall evaluations of the models. When we partition the conditions set into disjoint test sets in RSA, this effect is particularly strong, because dissimilarities between conditions in separate test sets do not contribute to the evaluation in any fold. The variance in the evaluations created by this random assignment is generated by our analysis and would vanish if we performed repeated cycles of crossvalidation with all possible partitionings of the conditions set into test sets. Unfortunately, such exhaustive crossvalidation will usually be prohibitively expensive in terms of computation time, especially in bootstrap-wrapped crossvalidation.

We can estimate the variance without this surplus by sampling ${\displaystyle n_{cv}>1}$ different randomly chosen partitionings of the conditions set into crossvalidation test sets for each bootstrap sample. Each of the $n_{cv}$ partitionings into $k$ subsets defines a complete cycle of $k$-fold crossvalidation. The bootstrap-wrapped crossvalidation estimate of the variance of the model-performance estimates with $n_{cv}$ crossvalidation cycles will be larger than the variance ${\sigma }_{boot}^2$ of the exact mean performance over all possible partitionings of a dataset. When we assume that the variance ${\sigma }_{cv}^2$ of randomly chosen partitionings around the mean is equal for each bootstrap sample, the overall variance ${\sigma }_{bootcv,n_{cv}}^2$ is:

(6) ${\displaystyle {\sigma }_{bootcv,n_{cv}}^2={\sigma }_{boot}^2+\frac{{\sigma }_{cv}^2}{n_{cv}}}$

When we have more than one cycle of crossvalidation for each bootstrap sample, it is straightforward to compute an estimate for the variance we would have gotten if we had drawn only a single partitioning ${\sigma }_{bootcv,1}^2$. We can simply use only the $i$ th partitioning for each bootstrap to estimate the variance and average these estimates. Using these two variance estimates for 1 and $n_{c}v$ partitionings, we can simply solve for the variance contributions of the random partitioning and of the bootstrap:

(7) ${\displaystyle {\hat{\sigma }}_{cv}^2=\frac{n_{cv}}{n_{cv}-1}\left({\hat{\sigma }}_{bootcv,1}^2-{\hat{\sigma }}_{bootcv,n_{cv}}^2\right)}$

(8) ${\displaystyle \begin{array}{rl}{\hat{\sigma }}_{boot}^2& ={\hat{\sigma }}_{bootcv,n_{cv}}^2-\frac{{\hat{\sigma }}_{cv}^2}{n_{cv}}\\ & ={\hat{\sigma }}_{bootcv,n_{cv}}^2-\frac{{\hat{\sigma }}_{bootcv,1}^2-{\hat{\sigma }}_{bootcv,n_{cv}}^2}{n_{cv}-1}\end{array}}$

Thus, we can directly compute an estimate of the variance we expect for exhaustive crossvalidation from two or more crossvalidation cycles using random partitionings for each bootstrap sample. The repetition across bootstrap samples enables a stable estimate even for ${\displaystyle n_{cv}=2}$. The average estimate is independent of $n_{cv}$ (Figure 2a). We could invest computation in increasing either the number of bootstrap samples or the number of crossvalidation cycles per bootstrap sample. Our simulations show that the reliability of the bootstrap estimate of the variance of the model-performance estimate improves more when we increase the number of bootstrap samples than when we increase the number of crossvalidation cycles per bootstrap sample (Figure 2b). Thus, we recommend using only two crossvalidation cycles per bootstrap sample.

Figure 2

Download asset Open asset

This crossvalidation approach provides model-performance estimates that are not biased by overfitting of flexible models to either subjects or conditions. Fixed and flexible models with different numbers of parameters can be robustly compared with generalization over conditions and/or subjects. The method can handle any model that can be fitted efficiently enough (for the types of flexible models we actually implemented, see Methods, Flexible models).

Validation of the statistical inference methods

We validate the inference methods using simulations, functional MRI data, and neural data. First, we establish that the statistical tests for model comparison are valid, controlling the false-positive rate at the nominal level. This requires simulating data under the null hypothesis, where two models that predict distinct RDMs are exactly equal in their RDM prediction accuracy. We use a matrix-normal model to simulate this null scenario for model comparison. Second, we show that the estimates of our uncertainty about model performance correctly capture the true variability for different generalization schemes in more realistic simulated scenarios based on neural network models. In these simulations, we cannot simulate the null hypothesis of two models that predict the representational geometry equally accurately. We also use these more realistic simulations to evaluate the power afforded by different RDM comparators. Third, we validate the inference procedure for flexible models, confirming that our bootstrap-wrapped crossvalidation scheme correctly accounts for the overfitting of flexible models. Fourth, we validate the methods using real data, acquired with functional MRI in humans and calcium imaging in mice.

Validity of inferential model comparisons

A frequentist test is valid when the rate of false positives (i.e. the rate of positive results when the null hypothesis is true) does not exceed the specified error rate α (e.g. 5%). Here, we check the validity of model-comparative inference, where the null hypothesis is that the two models perform equally well at explaining the representational geometry. We simulate scenarios where two models predict distinct geometries, but perform equally well on average at predicting the true representational geometry.

To simulate situations where two different models perform equally well, we generated condition-response matrices (containing an activity level for each combination of condition and response channel) by sampling from matrix-normal density models. A matrix-normal distribution over matrices yields matrices with normally distributed cells whose covariance is separable into a covariance matrix across rows and one across columns. In our case, rows correspond to the experimental conditions (e.g. stimuli) and the columns correspond to measurement channels (e.g. neurons or voxels). For matrix-normal data, the covariance across conditions captures the similarity among condition-related response patterns and determines the expected squared Euclidean-distance RDM (Diedrichsen and Kriegeskorte, 2017). The covariance among channels only scales the covariance of the distance estimates. This relationship enables us to generate matrix-normal data for arbitrary choices of the expected squared Euclidean-distance RDM. To model the null hypothesis, we choose two models that predict distinct RDMs and generate data, such that the expected data RDM has equal Pearson correlation to both model RDMs (results in Appendix 1—figure 1; details in Appendix 1).

We first evaluated the bootstrap in the scenario, where the goal is to generalize across subjects only. All model-comparative subject-only bootstrap tests were found to be valid (Appendix 1—figure 1). Inflated false-positive rates were observed for subject-only bootstrap tests only when using a small sample of subjects (<20). For a small number of samples, bootstrapping is known to produce underestimates of the variance by a factor $\frac{n}{n-1}$ for $n$ samples (e.g. Efron and Tibshirani, 1994, chapter 5.3). In this scenario, we recommend using a $t$-test across subjects, which is more computationally efficient and more accurate than bootstrap methods for small numbers of subjects.

Next, we tested bootstrapping for generalization to new conditions. In this scenario, the bootstrap methods were all conservative, showing false-positive rates substantially below 5% (Appendix 1—figure 1). This is expected, because we did not include any random selection of conditions in our data simulation, but enforced the H₀ exactly for the measured conditions.

To assess how problematic it is to choose an inference method that ignores the variance due to condition sampling, we ran a simulation in which we sampled the conditions from a large pool. We generated two models that perform equally well on 1000 conditions using matrix-normal sampling and then sampled a smaller set of these conditions for the simulated experiment. In these simulations, all techniques that only take subjects into account as a random factor fail catastrophically (Appendix 1—figure 1), with false-positive rates growing with the number of simulated subjects and reaching 60% at 40 simulated subjects. In contrast, our bootstrap tests that include condition sampling all remain valid, including the uncorrected 2-factor bootstrap and our new corrected 2-factor bootstrap with false-positive rates below the nominal 5%. However, the uncorrected 2-factor bootstrap was extremely conservative.

We also validated the tests against chance performance, where a single model is tested and the null hypothesis is that its performance is at chance level. To do so, we performed similar matrix-normal data simulations, evaluating a model that predicts a specific randomly sampled RDM on matrix-normal data consistent with an independently sampled random expected data RDM. Results show that a $t$-test across subjects as well as the bootstrap $t$-test approaches provide valid inference (Appendix 1—figure 1, top row). The subject $t$-test and the corrected 2-factor bootstrap $t$-test avoid overly conservative false-positive rates.

We conclude that the tests are valid in these simple simulated H₀ scenarios, where we are able to estimate the false-positive rate. In more realistic simulations using neural network models and real data, we can no longer simulate distinct models that predict the data RDM equally well. We therefore restrict ourselves to evaluating our bootstrap estimate of the variance of model-performance estimates, assuming that the false-positive rates are adequately controlled when we use an accurate variance estimate.

Criteria for evaluation of inference procedures

To evaluate alternative inference procedures, we perform simulations that reveal (1) whether the estimates of the uncertainty of the model-performance estimates are accurate (ensuring the validity of the inferences), and (2) how sensitive different model comparison methods are to subtle differences between models (determining the power of the inferences). To measure whether our bootstrap methods correctly estimate the uncertainty of the model-performance estimates, we compute the relative uncertainty (RU). The RU is the standard deviation of the bootstrap distribution of model-performance estimates ${\sigma }_{boot}$ divided by the true standard deviation of model-performance estimates ${\sigma }_{true}$ as observed over repeated simulations:

(9) ${\displaystyle \mathrm{RU}=\frac{{\sigma }_{boot}}{{\sigma }_{true}}=\sqrt{\frac{\frac{1}{N}\sum _{i=1}^N{\sigma }_i^2}{{\sigma }_{true}^2}},}$

where ${\sigma }_i^2$ is the variance estimator of the bootstrap in simulated dataset $i$ of the $N$ simulations. Ideally, we would like the bootstrap-estimated variance to match the true variance such that the RU is 1.

To measure how sensitive our analysis is to differences in model performance (e.g. comparing layers of a deep neural network), we define the model discriminability as a signal-to-noise ratio (SNR). The signal is the magnitude of model-performance differences, which is measured as the variance across models of their average of performance estimates across simulations. The noise is the nuisance variation, which includes subject and condition sample variation along with measurement noise. The noise is measured as the average across models of the variance of performance estimates across simulations. This results in the following formula, in which ${\mathrm{Perf}}_{i,m}$ is the performance of model $m$ of $M$ in repetition $i$ of $N$ repetitions of the simulation:

(10) ${\displaystyle \mathrm{SNR}=\frac{{\mathrm{Var}}_m\left(\frac{1}{N}\sum _{i=1}^N{\mathrm{Perf}}_{i,m}\right)}{\frac{1}{M}\sum _{i=1}^M{\mathrm{Var}}_i({\mathrm{Perf}}_{i,m})}.}$

A higher SNR indicates greater sensitivity to differences in model performance: differences between models are larger relative to the variation of model-performance estimates over repeated simulations. Note that this measure does not depend on the accuracy of the bootstrap because the bootstrap estimates of the variances do not enter this statistic. The SNR exclusively measures how large differences between models are compared to the level of nuisance variation we simulate, which may include random sampling of conditions, subjects, or both (in addition to measurement noise).

Validity of generalization to new subjects and conditions

To test whether our inference methods correctly generalize to new subjects and conditions, we performed a simulation that includes random sampling of both subjects and conditions (Figure 3). We used the internal representations of the deep convolutional neural network model AlexNet (Krizhevsky et al., 2012) to generate fMRI-like simulated data. In each simulated scenario, one of the layers of AlexNet served as the true (data-generating) model, while all layers were considered as candidate models in the inferential model comparisons. We simulated true voxel responses as local averages of the activities of close-by units in the feature maps of layers of the model. The response of each simulated voxel was a local average of unit responses, weighted according to a 2D Gaussian kernel over the locations of the feature map multiplied by a vector of nonnegative random weights (drawn uniformly from the unit interval) across the features. We then simulated hemodynamic-response timecourses and added measurement noise. The covariance structure of the noise was determined by the overlap of the simulated voxels’ averaging regions over space and a first-order autoregressive model over time. The simulated data were subjected to a standard general linear model (GLM) analysis to estimate the condition-response matrix. Variation over conditions was generated by using randomly sampled natural images from ecoset (Mehrer et al., 2017) as input to the AlexNet model. Variation over subjects was generated by randomly choosing a new location and a new vector of feature weights for each voxel of a new simulated subject.

Figure 3

Download asset Open asset

We simulated N = 100 datasets for each parameter setting to estimate how variable the model-performance estimates truly are. In analysis, we must estimate our uncertainty about model performance from a single dataset. To estimate how accurate these estimates were, we compared the uncertainty estimates used by different inference procedures (including different bootstrap methods) to the true variability. This comparison is a enables us to validate our inference despite the fact that we cannot compute false-positive rates of the model comparison tests. Our neural-network-based simulations do not contain situations that correspond to the H₀ of two different models with equal performance, which would require that the data-generating neural network layer predicts an RDM equally similar to those predicted by two other model layers. As expected, the rate of erroneously finding an alternative model outperforming the true data-generating model was very low (not shown) whenever the type of bootstrap matches the simulated level of generalization because the true layer has a higher average performance than the other models. At the 5% uncorrected significance level, the proportion of cases where any other layer performed significantly better than the true (data-generating) layer was only 1.524%. This rate reflects the differences between the layers of AlexNet, the simulated variability due to subject, stimulus, and voxel sampling, the simulated noise level, and the number of layers. Tests against the best other layer (chosen based on all data) significantly favor this other layer in only 0.694% of cases. Multiple-comparison correction would reduce these model-selection error rates even further.

To test generalization to either new conditions or new subjects (but not both simultaneously), we kept the other dimension constant. When simulating condition sampling, the true variance across conditions is accurately estimated for 40 or more conditions (Figure 4a) and is overestimated by 1-factor bootstrap resampling of conditions (rendering the inference conservative) when we have less than about 40 conditions (Figure 4a). When simulating subject sampling, the true variance across subjects is accurately estimated for 20 or more subjects (Figure 4b) and is underestimated by 1-factor bootstrap resampling of subjects (invalidating the inference) when we have very few subjects (Figure 4b). This downward bias corresponds to the $\frac{n}{n-1}$ factor between the sample variance and the unbiased estimate for the population variance. Our implementation in the RSA toolbox uses this factor to correct the variance estimate.

Figure 4

Download asset Open asset

To test our corrected 2-factor bootstrap method’s ability to generalize to new subjects and new conditions simultaneously, we simulated sampling of both conditions (stimuli) and subjects in our simulations. The corrected 2-factor bootstrap estimates the overall variation caused by random sampling of subjects and conditions and by measurement noise much more accurately than the naive 2-factor bootstrap (Figure 4c). Cases where an incorrect model (not the data-generating model) significantly outperformed the true model occurred in only 0.3% of simulations with the corrected 2-factor bootstrap, even without any multiple-comparison correction. This proportion would be larger if the alternative models performed more similar to the true model than simulated here. The RSA toolbox adjusts for multiple comparisons, controlling either the familywise error rate or the false-discovery rate across all pairwise model comparisons.

Overall, we found that the new more powerful corrected 2-factor bootstrap method yields accurate estimates of the variance across the simulated populations of subjects and conditions when the dataset is large enough (≥20 subjects, ≥40 conditions) and the type of bootstrap matches the population sampling simulated (subject, condition, or both).

The model discriminability (SNR) increases monotonically with the number of measurements, affording greater power for model-comparative inference. Model discriminability increases with the amount of data according to a power law (straight line in log–log plot; Figure 4d–f). Such a relationship holds whether we increase the number of conditions, the number of subjects, or the number of repetitions per condition. This result is expected and validates the SNR as an indicator of model-comparison power. In general, increasing the number of measurements helps most for the factor that causes most variability of the performance estimates, rendering generalization harder. For example, in our deep-neural-network-based simulations, the variability over subjects is smaller than the variability across conditions (Figure 4g). In this simulation, it thus increases statistical power more to collect data for more conditions. When there is more variability across subjects, the opposite is expected to hold. An intermediate voxel size (Gaussian kernel width) yielded the highest model-performance discriminability as measured by the SNR (Figure 4h, see Appendix 5 for more discussion on this topic).

Validity of inference on flexible models

To validate inferential model comparisons involving flexible models, we made a variant of the deep neural network simulation in which we do not assume to know how voxels average local neural responses. As the simulated ground truth, we set the spatial weights for each voxel to a Gaussian with a standard deviation of 5% of the image size (full width at half maximum FWHM ≈ 11.77%) and randomly weighted the feature maps (with weights drawn independently for each voxel and feature map uniformly at random from the unit interval; details in Methods, Neural-network-based simulation).

We then used models that a researcher could generate without knowing the ground truth of how voxels average local features. As building blocks for the models, we computed RDMs for different voxel averaging pool sizes and for different methods to deal with averaging across feature maps. To capture voxel averaging across retinotopic locations, we smoothed the feature maps with Gaussians of different sizes. To capture voxel averaging across feature maps, we (1) generated RDMs computed after taking the average across feature maps at each location (avg), (2) computed the expected RDM for the weight sampling implemented in the simulation (weighted), or (3) computed RDMs without any feature-map averaging (full).

We combined these building blocks into two types of flexible model: selection models and nonnegative linear-combination models. In a selection model, fitting is implemented as selection of the best among a finite set of RDMs. Here we defined one selection model for each method of combining the feature maps. Each selection model contained RDMs computed for different sizes of the local averaging pool. In linear-combination models, fitting consists in finding nonnegative weights for a set of basis RDMs, so as to maximize RDM prediction accuracy. The RDMs contain estimates of the squared Mahalanobis distances, which sum across sets of tuned neurons that jointly form a population code. As component RDMs, we chose the four extreme cases of RDM generation: no pooling across space or averaging across the whole image, each paired with either ‘full’ or ‘avg’ treatment of the feature maps. The resulting four-RDM-component linear model approximates the effect of computing the RDM from voxels that reflect the average activity over retinotopic patches of different sizes (Kriegeskorte and Diedrichsen, 2016). For the averaging across feature maps, which uses random weights, there is a strong motivation for using a linear model: When the voxel activities are nonnegatively weighted averages of the underlying neurons with the weights drawn independently from the same distribution, the expected squared Euclidean RDM is exactly a linear combination of the RDM computed based on the univariate population-average responses and the RDM based on all neurons (Appendix 4; see also Carlin and Kriegeskorte, 2017). For comparison, we also included fixed RDM models, corresponding to component RDMs of the fitted models.

We found that our bootstrap-wrapped crossvalidation (corrected 2-factor bootstrap with adjustment for excess crossvaldation variance) yielded accurate estimates of the uncertainty. The relative uncertainties were close to 1 (Figure 5b). The model-performance discriminability (SNR) was primarily determined by how accurately the different models were able to recreate the true measurement model (Figure 5c). The highest SNRs were achieved when the assumed model matched exactly (weighted feature treatment and voxel size 0.05), but the model variants which allowed for some fitting still yield high SNRs. Analyses that take the averaging across space and features into account yielded the highest average model performance for the true model. In contrast, analyses that ignore averaging over space or features (the full feature set selection model and some of the fixed models) not only lead to lower SNRs (as seen in Figure 5c), but also systematically selected the wrong layer, because a higher average performance was achieved by a different layer than the one we used for generating the data (not shown).

Figure 5

Download asset Open asset

We conclude that when the true voxel sampling is unknown, flexible models are needed to account for voxel sampling, so as to enable us recover the underlying data-generating computational model with our model-comparative inference. Fixed models based on incorrect assumptions about the voxel sampling can lead to low model-performance discriminability (SNR) and even to incorrect inferences as to which model is the true model.

Validation with functional MRI data

The simulations presented so far validated all statistical inference procedures, but may not capture all aspects of the structure of real measurements of brain activity. To test our methods under realistic conditions, we used real human fMRI (this section) and mouse calcium-imaging data (next section). We resampled data from a large openly available fMRI experiment in which humans viewed pictures from ImageNet (Horikawa and Kamitani, 2017). These data contain various noise sources, individual differences, signal shapes, and distributions that are difficult to simulate accurately without using measured data. We therefore implemented a data-based simulation to create realistic synthetic data, whose ground-truth RDM we knew (Figure 6). By subsampling from this dataset, we generated smaller datasets to test inference with bootstrapping over conditions. We used the entire dataset as a stand-in for the population a researcher might wish to generalize to. For each cortical area, we computed the mean RDM using all data (all runs and subjects). Each area’s mean RDM served as a ground-truth RDM for datasets sampled from that area and as a model RDM for datasets sampled from all areas. The model comparison we attempted aims to recover which cortical area a dataset was subsampled from. The simulation enables us to check whether our uncertainty estimates are correct for model-performance estimates based on real data.

Figure 6

Download asset Open asset

We varied the strength of noise, the number of runs, and the number of conditions (i.e. viewed images). We did not vary the number of subjects because the original dataset contains only five subjects, which precludes informative resampling of subjects. To increase the variability of the resampled datasets beyond sampling from the 35 measurement runs and to vary the noise strength, we created new voxel timecourses for each sampled run while preserving the spatial structure and serial autocorrelation of the noise. To achieve this, we estimated a second-order autoregressive model ($AR(2)$) separately for each run’s GLM residuals, permuted the AR-model’s residuals and added the results to the GLM’s predicted timecourse (see Figure 6a–e and Methods, fMRI-data-based simulation for details). We repeated each simulated experiment 24 times and used the RU and the model-performance discriminability (SNR) as our evaluation criteria.

Results were largely similar to those of the neural-network-based simulations (Figure 6f, g). For the RU, which measures the accuracy of our bootstrap variance estimates, we see a convergence toward the expected ratio (dashed line at 1), validating the bootstrap procedure for real fMRI data. For the model-performance discriminability (SNR), we find the same power-law increase with the number of conditions and the number of runs used as data. These results suggest that the regions are discriminable on the basis of their RDMs estimated from fMRI given five subjects’ data when a sufficient number of stimuli (≥30) and runs (≥16) is used.

Validation with calcium-imaging data

We can also adjudicate among models of the representational geometry on the basis of direct neural measurements, such as electrophysiological recordings or calcium-imaging data. These measurement modalities have very different statistical properties than fMRI. To test our methods for this kind of data, we performed a resampling simulation based on a large calcium-imaging dataset of responses of mouse visual cortex to natural images (de Vries et al., 2020). This dataset contains recordings from six visual cortical areas: primary visual cortex (V1), laterointermediate (LM), posteromedial (PM), rostrolateral (RL), anteromedial (AM), and anterolateral (AL) visual area (Figure 7a).

Figure 7

Download asset Open asset

As in the previous section, we used the overall mean RDM for each area as a ground-truth model and subsampled the data to create simulated datasets for which we know the ground-truth RDM. We used different numbers of stimulus repetitions, neurons, mice, and stimuli to vary the amount of information afforded by each simulated dataset. We used the crossnobis estimator of representational dissimilarity for all analyses here. We repeated each simulated experiment 100 times and computed the RU to assess the correctness of our bootstrap uncertainty estimates and the model-discriminability SNR to determine which noise covariance estimators and RDM comparators afford most sensitivity to model differences.

We analyzed the overall discriminability of the brain areas (Figure 7b). Although cortical areas vary in the reliability of the estimated RDMs, they can be discriminated reliably when using all data. We used the RU to assess whether our bootstrap variance estimates are correct for these data (Figure 7c). We resampled all factors (subjects, stimuli, runs, and cells) to generate simulted datasets. Correspondingly, the analysis used bootstrapping over both subjects and stimuli. We observed correct variance estimates for the corrected 2-factor bootstrap. The uncorrected 2-factor bootstrap was conservative, substantially overestimating the true variance.

To understand how the model-comparative power depends on experimental parameters and analysis choices, we analyzed the model-discriminability SNR. We found that more subjects, more stimuli, more runs, and more cells all increased the SNR just as in our fMRI and neural-network-based simulations (Figure 7d). Furthermore, we find that taking the noise covariance into account for computing the crossnobis RDMs in the first-level analysis improves the SNR (Figure 7e). Univariate noise normalization (implemented by using a diagonal noise covariance matrix) is better than no noise normalization. Multivariate noise normalization is slightly better than univariate noise normalization (Walther et al., 2016). For multivariate noise normalization, we tested two different shrinkage estimators with different targets: a multiple of the identity and the diagonal matrix of variances. These two variants perform similarly. In addition, we find that different RDM comparators yield the best model discriminability for different cortical areas (Figure 7g). For some, cosine RDM similarity performs better, for others, Pearson RDM correlation performs better. The whitened RDM comparators are better on average, but there are cases where the unwhitened RDM comparators perform slightly better. Thus, it remains dependent on the concrete experiment (with a particular choice of conditions, tested models and underlying representational geometry), which RDM comparator affords the best power for model comparison (Diedrichsen et al., 2020).

We present new methods for inferential evaluation and comparison of models that predict brain representational geometries. The inference procedures enable generalization to new measurements, new subjects, and new conditions, treat flexible models correctly using crossvalidation, and work for any representational dissimilarity estimator and RDM comparator. For fixed as well as flexible models, our inference methods support all combinations of generalization: to new measurements using the same subjects and conditions, to new subjects, to new conditions, and to both new subjects and new conditions simultaneously. We validated the methods using simulated data as well as calcium-imaging and fMRI data, showing that the inferences are correct. The methods are available as part of an open-source Python toolbox (rsatoolbox.readthedocs.io).

The methods section for this paper is separated into two parts: First, we describe the RSA analysis pipeline we propose in full. In the second part, we describe the simulation methods we used to test our pipeline for this paper.

No new data were collected for this study. The code to run both the analysis we do and our simulations is available with our rsatoolbox (https://github.com/rsagroup/rsatoolbox, copy archived at Schütt, 2023). The data for the fMRI-based resampling analysis are available from Horikawa and Kamitani, 2017 and the data for the calcium imaging are available from the Allen institutes website.

1. 1. Horikawa T
   2. Kamitani Y

   (2017) Open Neuro

   ID ds001246. Generic decoding of seen and imagined objects using hierarchical visual features.

   https://openneuro.org/datasets/ds001246
2. 1. de Vries SEJ
   2. Lecoq JA
   3. Buice MA
   4. Groblewski PA
   5. Ocker GK
   6. Oliver M
   7. Feng D
   8. Cain N
   9. Ledochowitsch P
   10. Millman D
   11. Roll K
   12. Garrett M
   13. Keenan T
   14. Kuan L
   15. Mihalas S
   16. Olsen S
   17. Thompson C
   18. Wakeman W
   19. Waters J
   20. Williams D
   21. Barber C
   22. Berbesque N
   23. Blanchard B
   24. Bowles N
   25. Caldejon SD
   26. Casal L
   27. Cho A
   28. Cross S
   29. Dang C
   30. Dolbeare T
   31. Edwards M
   32. Galbraith J
   33. Gaudreault N
   34. Gilbert TL
   35. Griffin F
   36. Hargrave P
   37. Howard R
   38. Huang L
   39. Jewell S
   40. Keller N
   41. Knoblich U
   42. Larkin JD
   43. Larsen R
   44. Lau C
   45. Lee E
   46. Lee F
   47. Leon A
   48. Li L
   49. Long F
   50. Luviano J
   51. Mace K
   52. Nguyen T
   53. Perkins J
   54. Robertson M
   55. Seid S
   56. Shea-Brown E
   57. Shi J
   58. Sjoquist N
   59. Slaughterbeck C
   60. Sullivan D
   61. Valenza R
   62. White C
   63. Williford A
   64. Witten DM
   65. Zhuang J
   66. Zeng H
   67. Farrell C
   68. Bernard A
   69. Phillips JW
   70. Reid RC
   71. Koch C

   (2020) Allen Brain Atlas

   ID visualcoding. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex.

   https://observatory.brain-map.org/visualcoding/

1. Preprint

   1. Abadi M
   2. Agarwal A
   3. Barham P
   4. Brevdo E
   5. Chen Z
   6. Citro C
   7. Corrado GS
   8. Davis A
   9. Dean J
   10. Devin M
   11. Ghemawat S
   12. Goodfellow I
   13. Harp A
   14. Irving G
   15. Isard M
   16. Jia Y
   17. Jozefowicz R
   18. Kaiser L
   19. Kudlur M
   20. Levenberg J

   (2015) TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems

   arXiv.

   https://arxiv.org/abs/1603.04467

   • Google Scholar
2. 1. Abbott J
   2. Ye T
   3. Krenek K
   4. Gertner RS
   5. Ban S
   6. Kim Y
   7. Qin L
   8. Wu W
   9. Park H
   10. Ham D

   (2020) A nanoelectrode array for obtaining intracellular recordings from thousands of connected neurons

   Nature Biomedical Engineering 4:232–241.

   https://doi.org/10.1038/s41551-019-0455-7

   • PubMed
   • Google Scholar
3. 1. Abraham A
   2. Pedregosa F
   3. Eickenberg M
   4. Gervais P
   5. Mueller A
   6. Kossaifi J
   7. Gramfort A
   8. Thirion B
   9. Varoquaux G

   (2014) Machine learning for neuroimaging with scikit-learn

   Frontiers in Neuroinformatics 8:14.

   https://doi.org/10.3389/fninf.2014.00014

   • PubMed
   • Google Scholar
4. 1. Ali A
   2. Meilă M

   (2012) Experiments with Kemeny ranking: What works when?

   Mathematical Social Sciences 64:28–40.

   https://doi.org/10.1016/j.mathsocsci.2011.08.008

   • Google Scholar
5. 1. Allefeld C
   2. Görgen K
   3. Haynes JD

   (2016) Valid population inference for information-based imaging: From the second-level t-test to prevalence inference

   NeuroImage 141:378–392.

   https://doi.org/10.1016/j.neuroimage.2016.07.040

   • PubMed
   • Google Scholar
6. Preprint

   1. Allen EJ
   2. St-Yves G
   3. Wu Y
   4. Breedlove JL
   5. Dowdle LT
   6. Caron B
   7. Pestilli F
   8. Charest I
   9. Hutchinson JB
   10. Naselaris T
   11. Kay K

   (2021) A Massive 7T fMRI Dataset to Bridge Cognitive and Computational Neuroscience

   bioRxiv.

   https://doi.org/10.1101/2021.02.22.432340

   • Google Scholar
7. 1. Avants BB
   2. Epstein CL
   3. Grossman M
   4. Gee JC

   (2008) Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain

   Medical Image Analysis 12:26–41.

   https://doi.org/10.1016/j.media.2007.06.004

   • PubMed
   • Google Scholar
8. 1. Averbeck BB
   2. Latham PE
   3. Pouget A

   (2006) Neural correlations, population coding and computation

   Nature Reviews. Neuroscience 7:358–366.

   https://doi.org/10.1038/nrn1888

   • PubMed
   • Google Scholar
9. 1. Baillet S

   (2017) Magnetoencephalography for brain electrophysiology and imaging

   Nature Neuroscience 20:327–339.

   https://doi.org/10.1038/nn.4504

   • PubMed
   • Google Scholar
10. 1. Bandettini PA
    2. Huber L
    3. Finn ES

    (2021) Challenges and opportunities of mesoscopic brain mapping with fMRI

    Current Opinion in Behavioral Sciences 40:189–200.

    https://doi.org/10.1016/j.cobeha.2021.06.002

    • Google Scholar
11. 1. Behzadi Y
    2. Restom K
    3. Liau J
    4. Liu TT

    (2007) A component based noise correction method (CompCor) for BOLD and perfusion based fMRI

    NeuroImage 37:90–101.

    https://doi.org/10.1016/j.neuroimage.2007.04.042

    • PubMed
    • Google Scholar
12. 1. Bodurka J
    2. Ye F
    3. Petridou N
    4. Murphy K
    5. Bandettini PA

    (2007) Mapping the MRI voxel volume in which thermal noise matches physiological noise—Implications for fMRI

    NeuroImage 34:542–549.

    https://doi.org/10.1016/j.neuroimage.2006.09.039

    • Google Scholar
13. 1. Cadena SA
    2. Denfield GH
    3. Walker EY
    4. Gatys LA
    5. Tolias AS
    6. Bethge M
    7. Ecker AS

    (2019a) Deep convolutional models improve predictions of macaque V1 responses to natural images

    PLOS Computational Biology 15:e1006897.

    https://doi.org/10.1371/journal.pcbi.1006897

    • PubMed
    • Google Scholar
14. Conference

    1. Cadena SA
    2. Sinz FH
    3. Muhammad T
    4. Froudarakis E
    5. Cobos E
    6. Walker EY
    7. Reimer J
    8. Bethge M
    9. Tolias AS
    10. Ecker AS

    (2019b)

    How well do deep neural networks trained on object recognition characterize the Mouse visual system?

    Advances in Neural Information Processing Systems.

    • Google Scholar
15. 1. Cai MB
    2. Schuck NW
    3. Pillow JW
    4. Niv Y

    (2019) Representational structure or task structure? Bias in neural representational similarity analysis and a Bayesian method for reducing bias

    PLOS Computational Biology 15:e1006299.

    https://doi.org/10.1371/journal.pcbi.1006299

    • PubMed
    • Google Scholar
16. 1. Carlin JD
    2. Kriegeskorte N

    (2017) Adjudicating between face-coding models with individual-face fMRI responses

    PLOS Computational Biology 13:e1005604.

    https://doi.org/10.1371/journal.pcbi.1005604

    • PubMed
    • Google Scholar
17. 1. Chaimow D
    2. Yacoub E
    3. Uğurbil K
    4. Shmuel A

    (2018) Spatial specificity of the functional MRI blood oxygenation response relative to neuronal activity

    NeuroImage 164:32–47.

    https://doi.org/10.1016/j.neuroimage.2017.08.077

    • PubMed
    • Google Scholar
18. 1. Chung S
    2. Lee DD
    3. Sompolinsky H

    (2018) Classification and geometry of general perceptual manifolds

    Physical Review X 8:031003.

    https://doi.org/10.1103/PhysRevX.8.031003

    • Google Scholar
19. 1. Chung S
    2. Abbott LF

    (2021) Neural population geometry: An approach for understanding biological and artificial neural networks

    Current Opinion in Neurobiology 70:137–144.

    https://doi.org/10.1016/j.conb.2021.10.010

    • Google Scholar
20. 1. Cichy RM
    2. Pantazis D
    3. Oliva A

    (2014) Resolving human object recognition in space and time

    Nature Neuroscience 17:455–462.

    https://doi.org/10.1038/nn.3635

    • PubMed
    • Google Scholar
21. 1. Cichy RM
    2. Khosla A
    3. Pantazis D
    4. Torralba A
    5. Oliva A

    (2016) Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence

    Scientific Reports 6:27755.

    https://doi.org/10.1038/srep27755

    • PubMed
    • Google Scholar
22. Conference

    1. Cichy RM
    2. Roig G
    3. Andonian A
    4. Dwivedi K
    5. Lahner B
    6. Lascelles A
    7. Mohsenzadeh Y
    8. Ramakrishnan K
    9. Oliva A

    (2019) The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence

    2019 Conference on Cognitive Computational Neuroscience.

    https://doi.org/10.32470/CCN.2019.1018-0

    • Google Scholar
23. Preprint

    1. Cichy RM
    2. Dwivedi K
    3. Lahner B
    4. Lascelles A
    5. Iamshchinina P
    6. Graumann M
    7. Andonian A
    8. Murty NAR
    9. Kay K
    10. Roig G
    11. Oliva A

    (2021) The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion

    arXiv.

    https://doi.org/10.48550/ARXIV.2104.13714

    • Google Scholar
24. 1. Connolly AC
    2. Guntupalli JS
    3. Gors J
    4. Hanke M
    5. Halchenko YO
    6. Wu YC
    7. Abdi H
    8. Haxby JV

    (2012) The representation of biological classes in the human brain

    The Journal of Neuroscience 32:2608–2618.

    https://doi.org/10.1523/JNEUROSCI.5547-11.2012

    • PubMed
    • Google Scholar
25. 1. Cox RW
    2. Hyde JS

    (1997) Software tools for analysis and visualization of fMRI data

    NMR in Biomedicine 10:171–178.

    https://doi.org/10.1002/(sici)1099-1492(199706/08)10:4/5<171::aid-nbm453>3.0.co;2-l

    • PubMed
    • Google Scholar
26. 1. Craik A
    2. He Y
    3. Contreras-Vidal JL

    (2019) Deep learning for electroencephalogram (EEG) classification tasks: A review

    Journal of Neural Engineering 16:031001.

    https://doi.org/10.1088/1741-2552/ab0ab5

    • PubMed
    • Google Scholar
27. 1. Dale AM
    2. Fischl B
    3. Sereno MI

    (1999) Cortical Surface-Based Analysis

    NeuroImage 9:179–194.

    https://doi.org/10.1006/nimg.1998.0395

    • PubMed
    • Google Scholar
28. 1. de Vries SEJ
    2. Lecoq JA
    3. Buice MA
    4. Groblewski PA
    5. Ocker GK
    6. Oliver M
    7. Feng D
    8. Cain N
    9. Ledochowitsch P
    10. Millman D
    11. Roll K
    12. Garrett M
    13. Keenan T
    14. Kuan L
    15. Mihalas S
    16. Olsen S
    17. Thompson C
    18. Wakeman W
    19. Waters J
    20. Williams D
    21. Barber C
    22. Berbesque N
    23. Blanchard B
    24. Bowles N
    25. Caldejon SD
    26. Casal L
    27. Cho A
    28. Cross S
    29. Dang C
    30. Dolbeare T
    31. Edwards M
    32. Galbraith J
    33. Gaudreault N
    34. Gilbert TL
    35. Griffin F
    36. Hargrave P
    37. Howard R
    38. Huang L
    39. Jewell S
    40. Keller N
    41. Knoblich U
    42. Larkin JD
    43. Larsen R
    44. Lau C
    45. Lee E
    46. Lee F
    47. Leon A
    48. Li L
    49. Long F
    50. Luviano J
    51. Mace K
    52. Nguyen T
    53. Perkins J
    54. Robertson M
    55. Seid S
    56. Shea-Brown E
    57. Shi J
    58. Sjoquist N
    59. Slaughterbeck C
    60. Sullivan D
    61. Valenza R
    62. White C
    63. Williford A
    64. Witten DM
    65. Zhuang J
    66. Zeng H
    67. Farrell C
    68. Ng L
    69. Bernard A
    70. Phillips JW
    71. Reid RC
    72. Koch C

    (2020) A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex

    Nature Neuroscience 23:138–151.

    https://doi.org/10.1038/s41593-019-0550-9

    • PubMed
    • Google Scholar
29. 1. Diedrichsen J
    2. Kriegeskorte N

    (2017) Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis

    PLOS Computational Biology 13:e1005508.

    https://doi.org/10.1371/journal.pcbi.1005508

    • PubMed
    • Google Scholar
30. 1. Diedrichsen J
    2. Yokoi A
    3. Arbuckle SA

    (2018) Pattern component modeling: A flexible approach for understanding the representational structure of brain activity patterns

    NeuroImage 180:119–133.

    https://doi.org/10.1016/j.neuroimage.2017.08.051

    • PubMed
    • Google Scholar
31. 1. Diedrichsen J
    2. Berlot E
    3. Mur M
    4. Schütt HH
    5. Shahbazi M
    6. Kriegeskorte N

    (2020) Comparing representational geometries using whitened unbiased-distance-matrix similarity

    Neurons, Behavior, Data Analysis, and Theory 5:27664.

    https://doi.org/10.51628/001c.27664

    • Google Scholar
32. 1. Dumoulin SO
    2. Wandell BA

    (2008) Population receptive field estimates in human visual cortex

    NeuroImage 39:647–660.

    https://doi.org/10.1016/j.neuroimage.2007.09.034

    • PubMed
    • Google Scholar
33. 1. Edelman S

    (1998) Representation is representation of similarities

    The Behavioral and Brain Sciences 21:449–467.

    https://doi.org/10.1017/s0140525x98001253

    • PubMed
    • Google Scholar
34. 1. Edelman S
    2. Grill-Spector K
    3. Kushnir T
    4. Malach R

    (1998) Toward direct visualization of the internal shape representation space by fMRI

    Psychobiology 26:309–321.

    https://doi.org/10.3758/BF03330618

    • Google Scholar
35. Book

    1. Efron B
    2. Tibshirani RJ

    (1994) An Introduction to the Bootstrap

    CRC press.

    https://doi.org/10.1201/9780429246593

    • Google Scholar
36. 1. Ejaz N
    2. Hamada M
    3. Diedrichsen J

    (2015) Hand use predicts the structure of representations in sensorimotor cortex

    Nature Neuroscience 18:1034–1040.

    https://doi.org/10.1038/nn.4038

    • PubMed
    • Google Scholar
37. 1. Esteban O
    2. Markiewicz CJ
    3. Blair RW
    4. Moodie CA
    5. Isik AI
    6. Erramuzpe A
    7. Kent JD
    8. Goncalves M
    9. DuPre E
    10. Snyder M
    11. Oya H
    12. Ghosh SS
    13. Wright J
    14. Durnez J
    15. Poldrack RA
    16. Gorgolewski KJ

    (2019) fMRIPrep: a robust preprocessing pipeline for functional MRI

    Nature Methods 16:111–116.

    https://doi.org/10.1038/s41592-018-0235-4

    • PubMed
    • Google Scholar
38. 1. Fonov V
    2. Evans A
    3. McKinstry R
    4. Almli C
    5. Collins D

    (2009) Unbiased nonlinear average age-appropriate brain templates from birth to adulthood

    NeuroImage 47:S102.

    https://doi.org/10.1016/S1053-8119(09)70884-5

    • Google Scholar
39. 1. Freeman JB
    2. Stolier RM
    3. Brooks JA
    4. Stillerman BS

    (2018) The neural representational geometry of social perception

    Current Opinion in Psychology 24:83–91.

    https://doi.org/10.1016/j.copsyc.2018.10.003

    • PubMed
    • Google Scholar
40. 1. Friston KJ
    2. Diedrichsen J
    3. Holmes E
    4. Zeidman P

    (2019) Variational representational similarity analysis

    NeuroImage 201:115986.

    https://doi.org/10.1016/j.neuroimage.2019.06.064

    • PubMed
    • Google Scholar
41. 1. Glasser MF
    2. Coalson TS
    3. Robinson EC
    4. Hacker CD
    5. Harwell J
    6. Yacoub E
    7. Ugurbil K
    8. Andersson J
    9. Beckmann CF
    10. Jenkinson M
    11. Smith SM
    12. Van Essen DC

    (2016) A multi-modal parcellation of human cerebral cortex

    Nature 536:171–178.

    https://doi.org/10.1038/nature18933

    • PubMed
    • Google Scholar
42. 1. Gorgolewski K
    2. Burns CD
    3. Madison C
    4. Clark D
    5. Halchenko YO
    6. Waskom ML
    7. Ghosh SS

    (2011) Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python

    Frontiers in Neuroinformatics 5:13.

    https://doi.org/10.3389/fninf.2011.00013

    • PubMed
    • Google Scholar
43. Software

    1. Gorgolewski KJ
    2. Esteban O
    3. Markiewicz CJ
    4. Ziegler E
    5. Ellis DG
    6. Notter MP
    7. Jarecka D
    8. Johnson H
    9. Burns C
    10. Manhães-Savio A
    11. Hamalainen C
    12. Yvernault B
    13. Salo T
    14. Jordan K
    15. Goncalves M
    16. Waskom M
    17. Clark D
    18. Wong J
    19. Loney F
    20. Modat M

    (2018) Nipype

    Zenodo.

    https://doi.org/10.5281/zenodo.596855
44. 1. Greve DN
    2. Fischl B

    (2009) Accurate and robust brain image alignment using boundary-based registration

    NeuroImage 48:63–72.

    https://doi.org/10.1016/j.neuroimage.2009.06.060

    • Google Scholar
45. Conference

    1. Guo Z
    2. Wang L
    3. Ji B
    4. Xi Y
    5. Yang B
    6. Liu J

    (2021) Flexible, Multi-Shank Stacked Array for High-Density Omini-Directional Intracortical Recording

    2021 IEEE 34th International Conference on Micro Electro Mechanical Systems (MEMS).. pp. 540–543.

    https://doi.org/10.1109/MEMS51782.2021.9375160

    • Google Scholar
46. 1. Haxby JV
    2. Connolly AC
    3. Guntupalli JS

    (2014) Decoding neural representational spaces using multivariate pattern analysis

    Annual Review of Neuroscience 37:435–456.

    https://doi.org/10.1146/annurev-neuro-062012-170325

    • PubMed
    • Google Scholar
47. 1. Hebart MN
    2. Görgen K
    3. Haynes JD

    (2014) The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data

    Frontiers in Neuroinformatics 8:88.

    https://doi.org/10.3389/fninf.2014.00088

    • PubMed
    • Google Scholar
48. 1. Horikawa T
    2. Kamitani Y

    (2017) Generic decoding of seen and imagined objects using hierarchical visual features

    Nature Communications 8:15037.

    https://doi.org/10.1038/ncomms15037

    • PubMed
    • Google Scholar
49. 1. Jenkinson M
    2. Bannister P
    3. Brady M
    4. Smith S

    (2002) Improved optimization for the robust and accurate linear registration and motion correction of brain images

    NeuroImage 17:825–841.

    https://doi.org/10.1016/s1053-8119(02)91132-8

    • PubMed
    • Google Scholar
50. 1. Jozwik KM
    2. Kriegeskorte N
    3. Mur M

    (2016) Visual features as stepping stones toward semantics: Explaining object similarity in IT and perception with non-negative least squares

    Neuropsychologia 83:201–226.

    https://doi.org/10.1016/j.neuropsychologia.2015.10.023

    • PubMed
    • Google Scholar
51. 1. Jun JJ
    2. Steinmetz NA
    3. Siegle JH
    4. Denman DJ
    5. Bauza M
    6. Barbarits B
    7. Lee AK
    8. Anastassiou CA
    9. Andrei A
    10. Aydın Ç
    11. Barbic M
    12. Blanche TJ
    13. Bonin V
    14. Couto J
    15. Dutta B
    16. Gratiy SL
    17. Gutnisky DA
    18. Häusser M
    19. Karsh B
    20. Ledochowitsch P
    21. Lopez CM
    22. Mitelut C
    23. Musa S
    24. Okun M
    25. Pachitariu M
    26. Putzeys J
    27. Rich PD
    28. Rossant C
    29. Sun W-L
    30. Svoboda K
    31. Carandini M
    32. Harris KD
    33. Koch C
    34. O’Keefe J
    35. Harris TD

    (2017) Fully integrated silicon probes for high-density recording of neural activity

    Nature 551:232–236.

    https://doi.org/10.1038/nature24636

    • PubMed
    • Google Scholar
52. 1. Kay KN
    2. Naselaris T
    3. Prenger RJ
    4. Gallant JL

    (2008) Identifying natural images from human brain activity

    Nature 452:352–355.

    https://doi.org/10.1038/nature06713

    • PubMed
    • Google Scholar
53. 1. Kell AJE
    2. Yamins DLK
    3. Shook EN
    4. Norman-Haignere SV
    5. McDermott JH

    (2018) A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy

    Neuron 98:630–644.

    https://doi.org/10.1016/j.neuron.2018.03.044

    • PubMed
    • Google Scholar
54. 1. Kemeny JG

    (1959)

    Mathematics without numbers

    Daedalus 88:577–591.

    • Google Scholar
55. Book

    1. Kendall MG

    (1948)

    Rank Correlation Methods

    Griffin.

    • Google Scholar
56. 1. Khaligh-Razavi SM
    2. Kriegeskorte N

    (2014) Deep supervised, but not unsupervised, models may explain IT cortical representation

    PLOS Computational Biology 10:e1003915.

    https://doi.org/10.1371/journal.pcbi.1003915

    • PubMed
    • Google Scholar
57. 1. Khaligh-Razavi SM
    2. Henriksson L
    3. Kay K
    4. Kriegeskorte N

    (2017) Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models

    Journal of Mathematical Psychology 76:184–197.

    https://doi.org/10.1016/j.jmp.2016.10.007

    • PubMed
    • Google Scholar
58. 1. Kietzmann TC
    2. Spoerer CJ
    3. Sörensen LKA
    4. Cichy RM
    5. Hauk O
    6. Kriegeskorte N

    (2019) Recurrence is required to capture the representational dynamics of the human visual system

    PNAS 116:21854–21863.

    https://doi.org/10.1073/pnas.1905544116

    • Google Scholar
59. Software

    1. Kipnis A

    (2023) Fmri-simulations, version 2fd77d1

    Github.

    https://github.com/adkipnis/fmri-simulations
60. 1. Klein A
    2. Ghosh SS
    3. Bao FS
    4. Giard J
    5. Häme Y
    6. Stavsky E
    7. Lee N
    8. Rossa B
    9. Reuter M
    10. Chaibub Neto E
    11. Keshavan A

    (2017) Mindboggling morphometry of human brains

    PLOS Computational Biology 13:e1005350.

    https://doi.org/10.1371/journal.pcbi.1005350

    • PubMed
    • Google Scholar
61. 1. Konkle T
    2. Alvarez GA

    (2022) A self-supervised domain-general learning framework for human ventral stream representation

    Nature Communications 13:491.

    https://doi.org/10.1038/s41467-022-28091-4

    • PubMed
    • Google Scholar
62. Conference

    1. Kornblith S
    2. Norouzi M
    3. Lee H
    4. Hinton G

    (2019)

    Similarity of Neural Network Representations Revisited

    Proceedings of the 36th International Conference on Machine Learning.

    • Google Scholar
63. 1. Kriegeskorte N
    2. Goebel R
    3. Bandettini P

    (2006) Information-based functional brain mapping

    PNAS 103:3863–3868.

    https://doi.org/10.1073/pnas.0600244103

    • Google Scholar
64. 1. Kriegeskorte N
    2. Formisano E
    3. Sorger B
    4. Goebel R

    (2007) Individual faces elicit distinct response patterns in human anterior temporal cortex

    PNAS 104:20600–20605.

    https://doi.org/10.1073/pnas.0705654104

    • Google Scholar
65. 1. Kriegeskorte N
    2. Mur M
    3. Bandettini PA

    (2008a) Representational similarity analysis - connecting the branches of systems neuroscience

    Frontiers in Systems Neuroscience 2:4.

    https://doi.org/10.3389/neuro.06.004.2008

    • PubMed
    • Google Scholar
66. 1. Kriegeskorte N
    2. Mur M
    3. Ruff DA
    4. Kiani R
    5. Bodurka J
    6. Esteky H
    7. Tanaka K
    8. Bandettini PA

    (2008b) Matching categorical object representations in inferior temporal cortex of man and monkey

    Neuron 60:1126–1141.

    https://doi.org/10.1016/j.neuron.2008.10.043

    • PubMed
    • Google Scholar
67. 1. Kriegeskorte N
    2. Mur M

    (2012) Inverse MDS: Inferring dissimilarity structure from multiple item arrangements

    Frontiers in Psychology 3:245.

    https://doi.org/10.3389/fpsyg.2012.00245

    • PubMed
    • Google Scholar
68. 1. Kriegeskorte N
    2. Kievit RA

    (2013) Representational geometry: integrating cognition, computation, and the brain

    Trends in Cognitive Sciences 17:401–412.

    https://doi.org/10.1016/j.tics.2013.06.007

    • Google Scholar
69. 1. Kriegeskorte N

    (2015) Deep neural networks: A new framework for modeling biological vision and brain information processing

    Annual Review of Vision Science 1:417–446.

    https://doi.org/10.1146/annurev-vision-082114-035447

    • PubMed
    • Google Scholar
70. 1. Kriegeskorte N
    2. Diedrichsen J

    (2016) Inferring brain-computational mechanisms with models of activity measurements

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 371:20160278.

    https://doi.org/10.1098/rstb.2016.0278

    • PubMed
    • Google Scholar
71. 1. Kriegeskorte N
    2. Douglas PK

    (2018) Cognitive computational neuroscience

    Nature Neuroscience 21:1148–1160.

    https://doi.org/10.1038/s41593-018-0210-5

    • Google Scholar
72. 1. Kriegeskorte N
    2. Diedrichsen J

    (2019a) Peeling the onion of brain representations

    Annual Review of Neuroscience 42:407–432.

    https://doi.org/10.1146/annurev-neuro-080317-061906

    • PubMed
    • Google Scholar
73. 1. Kriegeskorte N
    2. Douglas PK

    (2019b) Interpreting encoding and decoding models

    Current Opinion in Neurobiology 55:167–179.

    https://doi.org/10.1016/j.conb.2019.04.002

    • PubMed
    • Google Scholar
74. 1. Kriegeskorte N
    2. Wei XX

    (2021) Neural tuning and representational geometry

    Nature Reviews. Neuroscience 22:703–718.

    https://doi.org/10.1038/s41583-021-00502-3

    • PubMed
    • Google Scholar
75. Conference

    1. Krizhevsky A
    2. Sutskever I
    3. Hinton GE

    (2012)

    Imagenet classification with deep convolutional neural networks

    Advances in neural information processing systems.

    • Google Scholar
76. Book

    1. Kubilius J
    2. Schrimpf M
    3. Kar K
    4. Rajalingham R
    5. Hong H
    6. Majaj N
    7. Issa E
    8. Bashivan P
    9. Prescott-Roy J
    10. Schmidt K
    11. Nayebi A
    12. Bear D
    13. Yamins DL
    14. DiCarlo JJ

    (2019)

    Brain-like object recognition with high-performing shallow recurrent Anns

    In: Wallach H, Larochelle H, Beygelzimer A, Alché-Buc F, Fox E, Garnett R, editors. Advances in Neural Information Processing Systems. Curran Associates, Inc. pp. 1–12.

    • Google Scholar
77. 1. Lanczos C

    (1964) Evaluation of Noisy Data

    Journal of the Society for Industrial and Applied Mathematics Series B Numerical Analysis 1:76–85.

    https://doi.org/10.1137/0701007

    • Google Scholar
78. 1. LeCun Y
    2. Bengio Y
    3. Hinton G

    (2015) Deep learning

    Nature 521:436–444.

    https://doi.org/10.1038/nature14539

    • PubMed
    • Google Scholar
79. 1. Ledoit O
    2. Wolf M

    (2004) Honey, i shrunk the sample covariance matrix

    The Journal of Portfolio Management 30:110–119.

    https://doi.org/10.3905/jpm.2004.110

    • Google Scholar
80. Conference

    1. Mehrer J
    2. Kietzmann TC
    3. Kriegeskorte N.

    (2017)

    Deep neural networks trained on ecologically relevant categories better explain human IT

    Conference on Cognitive Computational Neuroscience.

    • Google Scholar
81. 1. Mehrer J
    2. Spoerer CJ
    3. Jones EC
    4. Kriegeskorte N
    5. Kietzmann TC

    (2021) An ecologically motivated image dataset for deep learning yields better models of human vision

    PNAS 118:e2011417118.

    https://doi.org/10.1073/pnas.2011417118

    • PubMed
    • Google Scholar
82. 1. Naselaris T
    2. Kay KN
    3. Nishimoto S
    4. Gallant JL

    (2011) Encoding and decoding in fMRI

    NeuroImage 56:400–410.

    https://doi.org/10.1016/j.neuroimage.2010.07.073

    • PubMed
    • Google Scholar
83. 1. Nili H
    2. Wingfield C
    3. Walther A
    4. Su L
    5. Marslen-Wilson W
    6. Kriegeskorte N

    (2014) A toolbox for representational similarity analysis

    PLOS Computational Biology 10:e1003553.

    https://doi.org/10.1371/journal.pcbi.1003553

    • PubMed
    • Google Scholar
84. 1. Nili H
    2. Walther A
    3. Alink A
    4. Kriegeskorte N

    (2020) Inferring exemplar discriminability in brain representations

    PLOS ONE 15:e0232551.

    https://doi.org/10.1371/journal.pone.0232551

    • PubMed
    • Google Scholar
85. 1. Norman KA
    2. Polyn SM
    3. Detre GJ
    4. Haxby JV

    (2006) Beyond mind-reading: multi-voxel pattern analysis of fMRI data

    Trends in Cognitive Sciences 10:424–430.

    https://doi.org/10.1016/j.tics.2006.07.005

    • Google Scholar
86. 1. Parvizi J
    2. Kastner S

    (2018) Promises and limitations of human intracranial electroencephalography

    Nature Neuroscience 21:474–483.

    https://doi.org/10.1038/s41593-018-0108-2

    • PubMed
    • Google Scholar
87. Book

    1. Paszke A
    2. Gross S
    3. Massa F
    4. Lerer A
    5. Bradbury J
    6. Chanan G
    7. Killeen T
    8. Lin Z
    9. Gimelshein N
    10. Antiga L
    11. Desmaison A
    12. Kopf A
    13. Yang E
    14. DeVito Z
    15. Raison M
    16. Tejani A
    17. Chilamkurthy S
    18. Steiner B
    19. Fang L
    20. Bai J

    (2019)

    Pytorch: an imperative style, high-performance deep learning library

    In: Wallach H, Larochelle H, Beygelzimer A, d’ F, Fox E, Garnett R, editors. Advances in Neural Information Processing System. Curran Associates, Inc. pp. 8024–8035.

    • Google Scholar
88. 1. Pedregosa F
    2. Eickenberg M
    3. Ciuciu P
    4. Thirion B
    5. Gramfort A

    (2015) Data-driven HRF estimation for encoding and decoding models

    NeuroImage 104:209–220.

    https://doi.org/10.1016/j.neuroimage.2014.09.060

    • PubMed
    • Google Scholar
89. 1. Power JD
    2. Mitra A
    3. Laumann TO
    4. Snyder AZ
    5. Schlaggar BL
    6. Petersen SE

    (2014) Methods to detect, characterize, and remove motion artifact in resting state fMRI

    NeuroImage 84:320–341.

    https://doi.org/10.1016/j.neuroimage.2013.08.048

    • PubMed
    • Google Scholar
90. 1. Ramírez FM
    2. Cichy RM
    3. Allefeld C
    4. Haynes JD

    (2014) The neural code for face orientation in the human fusiform face area

    The Journal of Neuroscience 34:12155–12167.

    https://doi.org/10.1523/JNEUROSCI.3156-13.2014

    • PubMed
    • Google Scholar
91. 1. Ritchie JB
    2. Lee Masson H
    3. Bracci S
    4. Op de Beeck HP

    (2021) The unreliable influence of multivariate noise normalization on the reliability of neural dissimilarity

    NeuroImage 245:118686.

    https://doi.org/10.1016/j.neuroimage.2021.118686

    • PubMed
    • Google Scholar
92. 1. Satterthwaite TD
    2. Elliott MA
    3. Gerraty RT
    4. Ruparel K
    5. Loughead J
    6. Calkins ME
    7. Eickhoff SB
    8. Hakonarson H
    9. Gur RC
    10. Gur RE
    11. Wolf DH

    (2013) An improved framework for confound regression and filtering for control of motion artifact in the preprocessing of resting-state functional connectivity data

    NeuroImage 64:240–256.

    https://doi.org/10.1016/j.neuroimage.2012.08.052

    • PubMed
    • Google Scholar
93. 1. Schäfer J
    2. Strimmer K

    (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics

    Statistical Applications in Genetics and Molecular Biology 4:Article32.

    https://doi.org/10.2202/1544-6115.1175

    • PubMed
    • Google Scholar
94. Software

    1. Schütt H

    (2023) Representational Similarity Analysis 3.0, version swh:1:rev:01e767c432e77633fe31304201718afce6a6ff9c

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:60193eeb851ac071f455e3d5db14cd8baeae20fa;origin=https://github.com/rsagroup/rsatoolbox;visit=swh:1:snp:94465c330bf41b107120efa10a9829c4ba5a2b91;anchor=swh:1:rev:01e767c432e77633fe31304201718afce6a6ff9c
95. 1. Sejnowski TJ
    2. Churchland PS
    3. Movshon JA

    (2014) Putting big data to good use in neuroscience

    Nature Neuroscience 17:1440–1441.

    https://doi.org/10.1038/nn.3839

    • PubMed
    • Google Scholar
96. 1. Shepard RN
    2. Chipman S

    (1970) Second-order isomorphism of internal representations: Shapes of states

    Cognitive Psychology 1:1–17.

    https://doi.org/10.1016/0010-0285(70)90002-2

    • Google Scholar
97. 1. Smith SM
    2. Nichols TE

    (2018) Statistical challenges in “big data” human neuroimaging

    Neuron 97:263–268.

    https://doi.org/10.1016/j.neuron.2017.12.018

    • PubMed
    • Google Scholar
98. 1. Steinmetz NA
    2. Koch C
    3. Harris KD
    4. Carandini M

    (2018) Challenges and opportunities for large-scale electrophysiology with Neuropixels probes

    Current Opinion in Neurobiology 50:92–100.

    https://doi.org/10.1016/j.conb.2018.01.009

    • PubMed
    • Google Scholar
99. 1. Stevenson IH
    2. Kording KP

    (2011) How advances in neural recording affect data analysis

    Nature Neuroscience 14:139–142.

    https://doi.org/10.1038/nn.2731

    • PubMed
    • Google Scholar
100. Preprint

     1. Storrs KR
     2. Khaligh-Razavi SM
     3. Kriegeskorte N

     (2014) Noise Ceiling on the Crossvalidated Performance of Reweighted Models of Representational Dissimilarity: Addendum to Khaligh-Razavi & Kriegeskorte (2014)

     bioRxiv.

     https://doi.org/10.1101/2020.03.23.003046

     • Google Scholar
101. 1. Storrs KR
     2. Kietzmann TC
     3. Walther A
     4. Mehrer J
     5. Kriegeskorte N

     (2021) Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting

     Journal of Cognitive Neuroscience 33:2044–2064.

     https://doi.org/10.1162/jocn_a_01755

     • PubMed
     • Google Scholar
102. 1. Stringer C
     2. Pachitariu M
     3. Steinmetz N
     4. Carandini M
     5. Harris KD

     (2019) High-dimensional geometry of population responses in visual cortex

     Nature 571:361–365.

     https://doi.org/10.1038/s41586-019-1346-5

     • PubMed
     • Google Scholar
103. 1. Székely GJ
     2. Rizzo ML
     3. Bakirov NK

     (2007) Measuring and testing dependence by correlation of distances

     The Annals of Statistics 35:2769–2794.

     https://doi.org/10.1214/009053607000000505

     • Google Scholar
104. 1. Tong F
     2. Pratte MS

     (2012) Decoding patterns of human brain activity

     Annual Review of Psychology 63:483–509.

     https://doi.org/10.1146/annurev-psych-120710-100412

     • PubMed
     • Google Scholar
105. 1. Tustison NJ
     2. Avants BB
     3. Cook PA
     4. Zheng Y
     5. Egan A
     6. Yushkevich PA
     7. Gee JC

     (2010) N4ITK: improved N3 bias correction

     IEEE Transactions on Medical Imaging 29:1310–1320.

     https://doi.org/10.1109/TMI.2010.2046908

     • PubMed
     • Google Scholar
106. 1. Uğurbil K

     (2021) Ultrahigh field and ultrahigh resolution fMRI

     Current Opinion in Biomedical Engineering 18:100288.

     https://doi.org/10.1016/j.cobme.2021.100288

     • PubMed
     • Google Scholar
107. 1. Walther A
     2. Nili H
     3. Ejaz N
     4. Alink A
     5. Kriegeskorte N
     6. Diedrichsen J

     (2016) Reliability of dissimilarity measures for multi-voxel pattern analysis

     NeuroImage 137:188–200.

     https://doi.org/10.1016/j.neuroimage.2015.12.012

     • Google Scholar
108. 1. Wandell BA
     2. Winawer J

     (2015) Computational neuroimaging and population receptive fields

     Trends in Cognitive Sciences 19:349–357.

     https://doi.org/10.1016/j.tics.2015.03.009

     • Google Scholar
109. 1. Wang T
     2. Xu C

     (2020) Three-photon neuronal imaging in deep mouse brain

     Optica 7:947.

     https://doi.org/10.1364/OPTICA.395825

     • PubMed
     • Google Scholar
110. 1. Weldon KB
     2. Olman CA

     (2021) Forging a path to mesoscopic imaging success with ultra-high field functional magnetic resonance imaging

     Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 376:20200040.

     https://doi.org/10.1098/rstb.2020.0040

     • PubMed
     • Google Scholar
111. 1. Wu MCK
     2. David SV
     3. Gallant JL

     (2006) Complete functional characterization of sensory neurons by system identification

     Annual Review of Neuroscience 29:477–505.

     https://doi.org/10.1146/annurev.neuro.29.051605.113024

     • PubMed
     • Google Scholar
112. 1. Xu Y
     2. Vaziri-Pashkam M

     (2021) Limits to visual representational correspondence between convolutional neural networks and the human brain

     Nature Communications 12:2065.

     https://doi.org/10.1038/s41467-021-22244-7

     • PubMed
     • Google Scholar
113. 1. Xue G
     2. Dong Q
     3. Chen C
     4. Lu Z
     5. Mumford JA
     6. Poldrack RA

     (2010) Greater neural pattern similarity across repetitions is associated with better memory

     Science 330:97–101.

     https://doi.org/10.1126/science.1193125

     • PubMed
     • Google Scholar
114. 1. Yamins DLK
     2. Hong H
     3. Cadieu CF
     4. Solomon EA
     5. Seibert D
     6. DiCarlo JJ

     (2014) Performance-optimized hierarchical models predict neural responses in higher visual cortex

     PNAS 111:8619–8624.

     https://doi.org/10.1073/pnas.1403112111

     • PubMed
     • Google Scholar
115. 1. Yamins DLK
     2. DiCarlo JJ

     (2016) Using goal-driven deep learning models to understand sensory cortex

     Nature Neuroscience 19:356–365.

     https://doi.org/10.1038/nn.4244

     • PubMed
     • Google Scholar
116. 1. Yarkoni T

     (2020) The generalizability crisis

     The Behavioral and Brain Sciences 45:1–37.

     https://doi.org/10.1017/S0140525X20001685

     • PubMed
     • Google Scholar
117. 1. Young HP
     2. Levenglick A

     (1978) A consistent extension of condorcet’s election principle

     SIAM Journal on Applied Mathematics 35:285–300.

     https://doi.org/10.1137/0135023

     • Google Scholar
118. 1. Zhang Y
     2. Brady M
     3. Smith S

     (2001) Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm

     IEEE Transactions on Medical Imaging 20:45–57.

     https://doi.org/10.1109/42.906424

     • PubMed
     • Google Scholar
119. 1. Zhuang C
     2. Yan S
     3. Nayebi A
     4. Schrimpf M
     5. Frank MC
     6. DiCarlo JJ
     7. Yamins DLK

     (2021) Unsupervised neural network models of the ventral visual stream

     PNAS 118:e2014196118.

     https://doi.org/10.1073/pnas.2014196118

     • PubMed
     • Google Scholar

Author details

1. Heiko H Schütt

   Zuckerman Institute, Columbia University, New York, United States

   Present address

   Université du Luxembourg, Esch-Belval, Luxembourg

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing – original draft, Writing – review and editing

   For correspondence

   hs3110@columbia.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-2491-5710

2. Alexander D Kipnis

   Zuckerman Institute, Columbia University, New York, United States

   Present address

   Max Planck Institute for Biological Cybernetics, Tuebingen, Germany

   Contribution

   Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

3. Jörn Diedrichsen

   Western University, London, Canada

   Contribution

   Conceptualization, Software, Formal analysis, Supervision, Methodology, Writing – review and editing

   Competing interests

   Reviewing editor, eLife

   "This ORCID iD identifies the author of this article:" 0000-0003-0264-8532

4. Nikolaus Kriegeskorte

   Zuckerman Institute, Columbia University, New York, United States

   Contribution

   Conceptualization, Resources, Data curation, Software, Formal analysis, Supervision, Funding acquisition, Investigation, Visualization, Writing – original draft, Project administration, Writing – review and editing

   For correspondence

   N.Kriegeskorteg@columbia.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-7433-9005

• 2,134

  views

• 300

  downloads

• 5

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.