Figures and data
Representational Similarity Analysis (RSA).
Steps for classic RSA (cRSA) and trial-level RSA (tRSA). A) Neural Representational Similarity Matrix (RSMbrain) is generated by correlating multi-voxel activity patterns across all trials within a Region of Interest (ROI), reflecting similarity between neural responses. B) Model Representational Similarity Matrix (RSMmodel) is constructed by correlating of-interest stimulus properties across all trials. C) First-Level cRSA (top) and tRSA (bottom). For cRSA, the lower triangular parts (black outline) of RSMbrain and RSMmodel are compared, producing a single summary statistic (e.g., Spearman’s rho) across all trials. For tRSA, representational similarity values from RSMbrain and RSMmodel for the same trial (e.g., tennis ball; red dashed outline) are compared, producing a single representational strength estimate for that trial. D) Second-Level analysis for cRSA and tRSA. In cRSA, subject-level r values are submitted to a one-sample t-test to assess whether the values reliably exceed zero. In tRSA, a linear random effects model with random effects for subject and stimulus is fit, and hypothesis testing determines whether the estimated intercept is significantly greater than zero.
Trial count influences the reliability of correlations.
The spread of correlation estimates (y-axis) varies nonlinearly with the number of observations, or trial count (x-axis). Trial count ranges from 10 to 200 with increments of 1. For each trial count, observations were drawn from a bivariate normal distribution with the given ground truth Pearson correlation, and their empirical Pearson correlation coefficient was computed. Ten random samples were drawn for each trial count and ground truth combination. Both ground truth and estimated values in the scatter plots are Fisher-transformed (z).
Correspondence between classic and trial-level RSA.
A) Histograms of representational strength estimates obtained by averaging tRSA values. Each histogram depicted 10,000 iterations. Representational similarity matrices (RSMs) were simulated based on known ground truth representational strength values (cRSA). Solid vertical lines indicate the mean of the observations, and dashed vertical lines indicate the ground truth representational strength. B) Scatter plots of overall representational strength estimates, based on simulated activity patterns, produced by cRSA (x-axis) and tRSA (y-axis). Trial count (n) and measurement noise level (σ2) were both varied. 50 iterations were performed for each parameter combination. Solid slopes indicate where tRSA equaled cRSA (y = x). C) Histograms of overall representational strength estimates, based on simulated activity patterns for two separate conditions, produced by cRSA (gray) and across-condition tRSA (orange). Each histogram depicted 10,000 iterations. Trial counts were varied; for example, “20 : 80” means that Condition A contained 20 trials and Condition B contained 80 trials, which was an unbalanced scenario. Effect sizes were also varied, such that the ground truth representational strength would be equal between conditions (“A = B”) or stronger in Condition B (“A < B”). D) Histograms of condition differences in representational strength values in C.
Testing condition differences in representational strength with varying effect sizes.
The effect size of the condition difference in representation was manipulated by changing the noise level in each condition in small increments, with higher noise levels corresponding to lower ground truth representational strengths. When the noise level was the same for both conditions, Type I error rates (red) were computed as the proportion of significant contrasts across 10,000 iterations, regardless of sign of the estimate. Otherwise, proportions were computed separately for effects of the correct sign (+, or B>A; blue) and of the incorrect sign (-, or B<A). Asterisks indicate the significance level of the test of equal proportions between simulation results from cRSA (left) and tRSA (right). Significance annotation: *** p < 0.001, ** p < 0.01, * p < 0.05.
Testing condition differences in representational strength with varying designs.
The robustness of cRSA and tRSA were assessed in several variations of a base experimental design consisting of 40 subjects and 200 unique stimuli (randomly and evenly split into two sets). Simulations varied in A) number of subjects, B) trial count per subject, C) the ratio of trial counts between two conditions (with a fixed total), and D) the variance of trial count ratios across participants (even split overall). In each simulation, the statistical significance of the condition difference in representational strength was determined by a paired-sample t-test for cRSA and a linear mixed-effects model for tRSA, with α = 0.05. The proportion of significant contrasts was computed across 10,000 iterations. Type I error rates were computed regardless of the direction of the contrast. Type II error rates were computed only for the correct contrast (i.e., B > A). Error bars indicate standard errors. In the left column, dashed horizontal lines mark the nominal α level of 0.05, and dotted horizontal lines mark the critical values beyond which the estimated proportion would be significantly different from α. Asterisks indicate the significance of the deviation in error rates between cRSA and tRSA results, *** p < 0.001, ** p < 0.01, * p < 0.05.
Testing continuous modulations of representational strength with varying designs.
The robustness of cRSA and tRSA were assessed in several variations of a base experimental design consisting of 40 subjects and 200 unique stimuli presented once for each subject. Simulations varied in A) the true effect size of representational strength, B) number of subjects, C) trial count per subject, and D) the variance of trial count across subjects (assuming a fixed total). Three statistical tests were performed in each simulation: a paired- sample t-test on the difference between “high” and “low” conditions (median-split of rmeasured) for cRSA, a linear mixed-effects model on the same condition difference for tRSAdiscrete, and a linear mixed-effects model on the effect of continuous variable rmeasured for tRSAcontinuous. Statistical significance was determined with α = 0.05. The proportion of significant results was computed across 10,000 iterations. Type I error rates were computed regardless of the sign of the estimate. Type II error rates were computed only for the correct sign (+). Error bars indicate standard errors. In the left column for B through D, dashed horizontal lines mark the nominal α level of 0.05, and dotted horizontal lines mark the critical values beyond which the estimated proportion would be significantly different from α. Asterisks indicate the significance level of the test of equal proportions between each tRSA approach and cRSA, *** p < 0.001, ** p < 0.01, * p < 0.05.
tRSA outperforms cRSA in identifying representational regions during object perception.
A) Visualization of the Object Perception Task. The fMRI task required participants to view 114 labeled objects and rate how well the labels describe the objects on a scale of 1 to 4 (mean 3.59). B) Density plots (y-axis) of regional representational strengths (x-axis) for tRSA (model estimates, dark) and cRSA (mean correlation values, light). Regions with significant representational strength greater than zero (q < 0.05) are highlighted in red. C) Scatter plot showing the correlation between regional representational strengths from tRSA (model estimates, y-axis) and cRSA (mean correlation values, x-axis). Regions with significant tRSA representation are highlighted in red, and those significant in cRSA are outlined in red. The representational strength measures are highly correlated across regions (r = 0.98, p < 0.001). D) Bar graph of t-statistics for each of the 26 regions of interest. Representation t-statistics (y-axis) from cRSA (light) and tRSA (dark) across 26 regions of interest (x-axis). Regions significant after FDR correction (q < 0.05) are highlighted in red. E) Brain regions with significant representation. Regions in light red were identified by the tRSA and cRSA. Regions in dark red were identified by the tRSA method only.
Representation regions identified by cRSA and tRSA.
Unmodeled within- and between-participant variance in cRSA for mnemonic representations.
A) Top: Visualization of the main memory task. Participant recognition and judgement confidence was tested for 144 concepts (114 “old”, 30 “new”). Bottom: Stacked bar plots with the number of “Hit” (blue) and “Miss” (gray) trials (y- axes) for each participant (x-axis). Participants were sorted by their hit to miss ratio in descending order. B) Density plots of cRSA representational strength (y-axes) across the 26 ROIs (x-axes) for Hits (blue) and Misses (gray) for each participant, sorted by descending Hit-to-Miss count ratio. The Hit-to-Miss ratios were displayed in the top-left corner of each subplot, with the font color transitioning from blue (indicating more Hits) to gray (indicating more Misses). C) Scatter plot illustrating the influence of the Hit-to-Miss ratio (x-axis) on the participant CVFs (y-axis). The Spearman’s correlation was significant, with a coefficient of 0.45 and a p-value of 0.016. Two participants are marked X and were excluded from this correlation as their Hit-to Miss ratio is less than 1. D) Density plots (y-axis) of cRSA representational strength across participants for Hits (blue) and Misses (gray). E) Regions exhibiting significant cRSA mnemonic representation (Hits > Misses). Regions highlighted in blue were significant after FDR correction with q < 0.05. Regions in gray were significant at p < 0.05 but did not survive FDR correction.
Mnemonic representation regions identified by cRSA and tRSA.
tRSA outperforms cRSA in detecting mnemonic representations.
A) Bar graph of t Statistics for each of the 26 regions of interest. Mnemonic representation t-statistics (y-axis) from cRSA (light) and tRSA (dark) across 26 regions of interest (x-axis). Regions that are significant after FDR correction (q < 0.05) are highlighted in blue. B) Brain regions with significant mnemonic representation identified by tRSA are shown in blue. The region that was significant in cRSA but not in tRSA is depicted in light blue. C) Scatter plot of t-statistics calculated from tRSA (y-axis) and cRSA (x-axis), showing a significant Pearson correlation (r = 0.49, p = 0.010). Significant tRSA regions are highlighted in blue. Significant cRSA regions are outlined in light blue.
tRSA captures stimulus-level modulations on representation.
A) Distribution of Item Memorability. Density plot of memorability values demonstrates a wide distribution of memorability scores (i.e., the average confidence of remembering a particular item). This stimulus-level variance is an underexplored target of representational analyses. B) Cortical representational effects of Memorability. Relationship between item-level Memorability and representational strength, as assessed with tRSA, for each of the 26 regions of interest. Trial-level representational strength estimates from Object Perception were significantly modulated by a continuous, stimulus- level measure of Item Memorability in four regions (blue). The t-statistics for this modulation were plotted (y-axis) across regions of interest (x-axis).
Continuous modulation in tRSA.
Comparing across-condition and within-condition tRSA.
Each cell depicts descriptive statistics of different RSA methods from 10,000 simulations. Cells differ in raw trial counts (20, 80, or 320 in a given condition), balance of trial counts between conditions (balanced or unbalanced), and effect (A = B, A > B, or A < B). In each cell, the top scatter plots depict condition-level tRSA estimates (y-axis; blue, within-condition; orange, across-condition) against cRSA estimates (x-axis), whereas the bottom scatter plots depict the standard deviations of trial-level representational strength estimates from across-condition tRSA (y-axis) and within-condition tRSA (x-axis).
Behavioral results. Bold indicates participants excluded from Conceptual Retrieval Analyses.
Model selection using 3 criteria: Akaike information criterion (AIC), Bayesian information criterion (BIC), and log-liklihood ratio testing (LRT).
