Task and main analysis

(A) For the sensory task (left), participants viewed a randomly oriented grating for 9 seconds per trial (contrast phase-reversing at 5 Hz) and reported instances of contrast dimming. For the working memory task (right), participants remembered a briefly presented (500 ms) randomly orientated grating for 13 seconds, until a 3 second recall epoch (not depicted). (B) For each Region of Interest (ROI) we employed a split-half randomization procedure to create a Representational Similarity Matrix (RSM) for each participant. On each randomization fold, voxel patterns from all trials (300–340 for sensory, 324 for memory) were randomly split in half. For each half of trials, we averaged the voxel patterns for every degree in orientation space within a + 10° window. This resulted in 180 vectors with a length equal to the number of voxels for each split of the data. We then calculated the similarity between each vector (or degree) in one half of the data, to all vectors (or degrees) in the second half of the data, using a Spearman correlation coefficient. This resulted in a 180x180 similarity matrix on each fold. This randomization procedure was repeated 1000 times to generate the final RSM for each ROI and each participant. Across all folds, RSM’s are near-symmetrical around the diagonal, give-or-take some cross-validation noise. (C) Representational geometry of orientation during the sensory (top row) and working memory (bottom row) tasks, for retinotopically defined ROI’s (columns) across all participants. During the sensory task, the clear diagonal pattern in early visual areas V1–V3 indicates that orientations adjacent in orientation space are represented more similarly than orientations further away. During the memory task, similarity clusters strongly around oblique orientations (45° and 135°), contrasting starkly with the similarity patterns during perception. Note that the diagonal represents an inherent noise-ceiling, due to the cross-validation procedure used. This noise ceiling shows inhomogeneities across orientation space, demonstrating how certain orientations may be encoded with more noise than others. RSM’s are scaled to the range of correlations within each subplot to ease visual comparison of representational structure between sensory and memory tasks for all ROI’s (exact ranges are shown in Supplementary Figure 3). For early ROI’s (V1–V4), only visually responsive voxels are included in the analysis. Throughout, 0° (and 180°) denotes vertical, and 90° denotes horizontal.

Modeling the representational similarity of perceived and remembered orientations

(A) The distribution of visual orientation in the natural world is inhomogeneous, with higher prevalence of orientations closer to cardinal (90° & 180°) compared to oblique (45° & 135°). The function shown here approximates these input statistics, and is used to constrain both the veridical (in B) and categorical (in C) models. (B) The veridical model is based on the principle of efficient coding – the idea that neural resources are adapted to the statistics of the environment. We model this via 180 idealized orientation tuning functions with amplitudes scaled by the theoretical input statistics function (the top panel shows a subset of tuning functions for illustrational purposes). A vector of neural responses is simulated by computing the activity of all 180 orientation-tuned neurons to a given stimulus orientation. Representational similarity is calculated by correlating simulated neural responses to all possible orientations, resulting in the veridical model RSM (bottom panel). Note that while we chose to modulate tuning curve amplitude, there are multiple ways to warp the stimulus space (e.g., by applying non-uniform changes to gain, tuning width, tuning preference, etc.39,43). (C) In the categorical model, categorization is based on people’s subjective experience of relative similarity between orientations in different parts of orientation space: If orientations in part of the space appear quite similar, they are lumped together into the same category, while the most distinctive looking orientations serve as category boundaries. This is quantified via the “psychological distance” – the sum of derivatives along the input statistics function between any pair of orientations (see top panel). The insert shows an example of orientation-pairs near cardinal (in blue) and oblique (in red) that have the same physical distance, but different psychological distances. The psychological distance between each possible pair of orientations yields the categorical model’s RSM (bottom panel). (D) Fits of the veridical (grey) and categorical (teal) models for the sensory (top) and memory (bottom) tasks. During the sensory task, the veridical model better explains the data compared to the categorical model in almost all visual ROI’s (except IPS1–3), indicating a representational scheme that is largely in line with modeled early sensory responses. During the memory task, the categorical model gains increasingly more explanatory power over the veridical model along the visual hierarchy, and explains the data significantly better in V3, V3AB, V4, and IPS0. The Fisher transformed semi-partial correlations (on the y-axis) represent the unique contribution of each model after removing the variance explained by the other model via semi-partial correlations. Dots represent individual participants, and errorbars represent + 1 within-participant SEM. Asterisks indicate the significance level of post-hoc two-sided paired-sample t-tests (*p < 0.05; **p < 0.01; ***p < 0.001) comparing the two models in each ROI.

Generating and fitting the veridical and categorical models based on independent behavioral data

(A) During an independent psychophysical examination, a new set of participants (N=17) reported the orientation of briefly presented (200ms) and remembered (2s) single gratings by rotating a response dial with a computer mouse (i.e., via method-of-adjustment). For each possible stimulus orientation in the experiment (+1°), we calculated the mean absolute response error across all participants, and smooth the resulting function (Gaussian, over 10°). The absolute error−1 (y- axis) is plotted against the stimulus orientation shown to participants. From this psychophysical input function, the veridical and categorical models were generated as previously described (see Figure 2B & 2C). (B) Veridical and categorical models generated from the psychophysical input function (in A). (C) Fits of the veridical (in grey) and categorical (in teal) models based on the independent psychophysical data. During the sensory task (top), the veridical model better explains the data compared to the categorical model in all visual ROI’s except for IPS1–3. During the memory task (bottom), the categorical model better explains the data compared to the veridical model in V2, V2AB, and V4 (and marginally better in IPS0 with p = 0.053). The Fisher transformed semi-partial correlations (on the y-axis) represent the unique contribution of each model after removing the variance explained by the other model. Dots represent individual participants, and errorbars represent + 1 within-participant SEM. Asterisks indicate the significance level of post-hoc two-sided paired-sample t-tests (*p < 0.05; **p < 0.01; ***p < 0.001) comparing the two models in each ROI.

Ability to cross-decode using RSA

(A) Using across-task representational similarity analysis, we directly compare orientation response patterns recorded during the sensory task (y-axis), to those measured during the memory task (x-axis). Here we show V1 (left subplot) and IPS0 (right subplot) as example ROI’s. The across-task RSM in V1 shows a clear diagonal component, indicating similar response patterns for specific orientations in the sensory and memory tasks. In IPS0 such pattern similarity for matching orientations in the sensory and memory tasks is less evident. (B) We want to quantify the extent to which orientations held in working memory evoke response patterns that overlap with response patterns from those same orientations when viewed directly, and how this similarity drops at larger distances in orientation space. First, we center our across-task RSM’s on the remembered orientation (notice the x- axis), and then take the sum of correlations relative to the remembered orientation (plotted on top of the across-task RSM’s in grey). We call this the “correlation profile” of the remembered orientation. In V1 we see that correlations are highest between response patterns from matching perceived and remembered orientations (0° on the x-axis), explaining the ability to cross-decode between sensory and memory tasks as demonstrated in previous work (e.g.,1,31). By contrast, IPS0 shows a much flatter correlation profile. (C) Correlation profiles for all retinotopic ROI’s in our study, obtained by performing across-task RSA (left panel). Most ROI’s show a peaked correlation profile, indicative of shared pattern similarity between the same orientations when directly viewed and when remembered. The different offsets along the y-axis for different ROI’s reflect the overall differences in pattern similarity in different areas of the brain, with pattern similarity being highest in area V1. Shaded areas indicate + 1 SEM (D) To validate the ability to cross-decode using RSA, we directly compare this new approach (x-axis) to the multivariate analysis performed by Rademaker et al. in 2019 (y-axis). The latter used an inverted encoding model (IEM) that was trained on the sensory task, and tested on the delay period of the memory task. Both the correlation profiles from RSA, and the channel response functions from IEM yield more-or-less peaked functions over orientation space (relative to the remembered orientation) that can be quantified using the same fidelity metric (i.e., convolving with a cosine). Here, we show a high degree of consistency between the fidelity metrics derived with both approaches, and successful cross-generalization from the sensory to the memory task (as indexed by >0 fidelities) in many ROI’s. Each color represents a different ROI, and for each ROI we plot each of the six participants as an individual dot.

Second level RSA

(A) To compare how orientation is represented across different regions of visual cortex, RSM’s from fine-grained individual ROI’s (Supplementary Figure 1) were correlated in a 2nd level similarity analysis. For the sensory task (top panel), representational similarity is high among early visual areas; high among the various IPS regions; and high among LO regions. Similarity between these three clusters is relatively low. For the memory task (bottom panel) there is a slight shift in similarity compared to the sensory task, with V1 becoming less similar, and IPS0 becoming more similar, to areas V2–V4. Furthermore, the distinction between areas is generally less pronounced. (B) Representational similarity can also be used as an indicator of connectivity between ROI’s based on shared representational geometry: When the geometry is similar, the “connection” is stronger (indicated here by the width of the grey lines connecting different ROI’s). The sum of the strength of these connections in a given ROI (i.e., degree centrality) indicates to which extent a local representational geometry resembles that of other ROI’s. Degree centrality is highest in early visual cortex and lowest in IPS regions, indicating a higher conservation of geometry across early visual cortical regions.

Orientation representational geometry (as indexed by RSM’s) during sensory perception and working memory for all retinotopically defined ROI’s (across all participants) that were not already shown in Figure 1C. Here, ROI’s are organized by whether they are located in the dorsal or ventral stream (top and bottom two rows, respectively). Early visual areas V1–V3 were split by their dorsal and ventral portions – used as input to the second- level RSA analysis (Figure 5 of the main text). Areas IPS1–3 (in the dorsal stream) and LO (in the ventral stream) were split based on their respective sub-portions – and similarly used as input to the second-level RSA analyses. All RSM’s are scaled to the range of correlations within each subplot to ease visual comparison of representational structure.

With 180 possible target orientations, and a finite number of trials, some form of smoothing or binning is necessary for RSA to yield reliable correlations. In our main analysis we smooth over a window of +10°, which could in theory impact the geometry around categorical boundaries (90° and 180°). In particular, it could induce some smearing of the categorical pattern observed in the memory task. To ensure that this pattern does not critically depend on the way trials are combined, here we show the data for the memory task binned (instead of smoothed) into 12 bins of 15°. On top, we show RSM’s with bins centered on the 2 cardinals and the 2 obliques (see inset), meaning that the parts of orientation space highlighted in dark-red are bins that include a cardinal or an oblique orientation. On the bottom, we show the same analysis but with the bins shifted, such that they respect cardinal and oblique boundaries, and bins fall on either side. We observe that similarity in bins that include a cardinal (top row) is relatively low, presumably due to the relatively large psychological distance between orientations on different sides of a cardinal orientation, resulting in lower correlations. Nevertheless, there is relatively low similarity around cardinals also when we respect the categorical boundary (bottom row), implying these categorical effects are not impacted much by the specific binning approach. Overall, binning or smoothing do not drastically change the geometry (though of course, the resolution of the RSM is much lower with binning).

Exact ranges of correlations in the RSM’s from Figure 1C. To best show the representational structure for sensory and memory representations across ROI’s, and to ease comparison between them, the RSM’s in Figure 1C are scaled to the range (min-to-max) of correlations within each subplot. But the minimum and maximum correlations are not identical across subplots, therefore, correlation ranges across all participants (black rectangles) and individual participants (grey lines) are shown here for sensory (left) and memory (right) RSM’s.

Two alternative fitting approaches.

(A) Model weights when fitting the veridical and categorical models directly to the RSM’s (without first taking the residuals), and (B) model weights derived with a general linear regression (independent weights for each model). Irrespective of the fitting approach, the geometrical differences between our two tasks are captured by higher “veridical” weights in the sensory task, and more “categorical” weights in the memory task. For the “direct fitting” approach (in A) there is a significant 3-way interaction (model x ROI x task, F(7,35) = 2.413; p = 0.0398), which we followed up by post-hoc ANOVA’s within the sensory and memory task separately. There is a main effect of model in both the sensory (F(1,5) = 40.26; p = 0.001) and memory (F(1,5) = 12.47; p = 0.017) tasks that is not the same in all ROI’s (as indexed by model x ROI interactions for sensory F(7,35) = 3.262, p = 0.009 and memory F(7,35) = 2.791, p = 0.024 tasks). Similarly, for the general linear regression approach (in B) there is also a significant 3-way interaction (model x ROI x task, F(7,35) = 2.414; p = 0.0398), and main effects of model in both the sensory (F(1,5) = 40.28, p = 0.001) and memory (F(1,5) = 12.48, p = 0.017) tasks, and this difference between the models is not the same in all ROI’s (as indexed by model x ROI interactions for both sensory F(7,35) = 3.26, p = 0.009, and memory F(7,35) = 2.79, p = 0.024 tasks). Asterisks indicate the significance level of post-hoc two-sided paired- sample t-tests (*p < 0.05; **p < 0.01) comparing the two models in each ROI.

Model fits for the 3 different working memory distractor conditions. Overall, the results split by condition are qualitatively similar to the main results across all trails (Figure 2D, bottom panel). Two-way ANOVA’s comparing model and ROI showed that the categorical model did a better job at explaining the data in some of the ROI on trials without a distractor (model x ROI interaction: F(7,35) = 2.914; p = 0.016; main effect model: F(1,5) = 13.07; p = 0.015) and with a grating distractor (model x ROI interaction: F(7,35) = 4.3; p = 0.0016; main effect model: F(1,5) = 6.344; p = 0.053), indicating increasing differences between the veridical and categorical models along the visual hierarchy. While we see similar trends for the 108 trials with a noise distractor, these effects did not reach significance (model x ROI interaction: F(7,35) = 1.812; p = 0.116; main effect model: F(1,5) = 1.335; p = 0.3). Nevertheless, despite using only 1/3rd of the data in each of these sub-plots, the pattern in the data is highly consistent. Asterisks indicate the significance level of post-hoc two-sided paired-sample t-tests (*p < 0.05; **p < 0.01) comparing the two models in each ROI.

Orientation inhomogeneities of the representational geometry

(A) To examine the inhomogeneity or representational similarity throughout orientation space, we plot the diagonals of the RSM’s from in Figure 1C. During the sensory task, we see that similarity tends to be relatively high around vertical orientations (0°/180°) compared to horizontal orientations (90°). For both tasks, oblique orientations are represented relatively more similar, and cardinals less similar. This “oblique” like effect is much exacerbated in the memory task compared to the sensory task. (B) We use multidimensional scaling (MDS) to projects high dimensional response patterns into 2 dimensions, in order to better visualize of how orientation space is represented. During the sensory task there is an orderly geometrical progression of orientation space, with the highest similarity between adjacent orientations (and some clustering around cardinal orientations, especially 180°) in early visual areas V1–V3. There’s also a “pinching” of orientation space around the obliques (45° and 135° become very similar) in more anterior visual areas V3AB–IPS and LO. During the memory task, the orientation space geometry remains circular in all ROI’s, with notable clustering of similarity around the obliques.

Alternative model based on psychological distance.

(A) An alternative way to model the sensory and memory task RSM’s is to vary the degree of similarity that can be expected at cardinals or at obliques. By changing the exponent in the input statistics function f(x) = &|sin x| −1&exponent + 0.1 to a free parameter, and using the psychological distance between every pair of orientations (as in the categorical model), we can create a family of input statistics functions (left panel) that modulate the shape of the model RSM’s (right panels) such that we can span any orientation anisotropy ranging from highest similarity around cardinals to highest similarity around obliques (similar to the modeling approach in 57). Each of the input functions in the left panel matches an RSM in the right panels (with the input function overlaid in white). At an exponent of 0.55 we approximate a uniform diagonal RSM, or a “physical” model of orientation space. Note that by modulating the shape of the input function in this manner, we can retrieve models that look very similar to our veridical model (e.g., exponent = 0.4), and a model identical to our categorical model (exponent = 2) in this parametric RSM space. (B) We plot the best fitting exponent for the input function for all ROI’s (x-axis) and separately for the memory (dark blue) and sensory (light blue) tasks, and show that those differ significantly (ROI x task interaction: F(7,35) = 9.658; p < 0.001). For the sensory task the best fitting exponent stays close to 0.55 for all ROI’s, indicating that an RSM with a close-to uniform diagonal fits the data well. That said, the exponent does differ across ROI’s (main effect of roi, F(7,35) = 3.307; p = 0.008), showing that a model with slightly higher similarity around obliques (exponent > 0.55) does better in for example V1, while a model with slightly higher similarity at cardinals (exponent between 0 and 0.55) does better at for example V4. For the memory task we see a gradual increase in the exponent along the visual hierarchy (up to and including IPS0), indicating that a model with increasingly stronger similarity around oblique orientations (i.e., increasingly stronger categorization) is better at explaining the memory geometry for more anterior ROI’s (main effect of roi, F(7,35) = 9.213; p < 0.001). Best fitting exponents for individual subjects are shown as dots, and error bars indicate + 1 within-subject SEM.

Post-hoc statistics for two-sided paired t-tests from the theoretical input function based on the statistics in the natural world (in green) and from the psychophysical input function based on independent behavioral measurements (in blue). All significant cells are colored in a lighter shade for the purpose of quick visualization.