Experimental paradigm

(A) Participants were presented with a sequence of disks of different colors and at different locations. They were asked to memorize both the location and color of the sequence and later reproduce the full location and color sequences one after another by clicking the corresponding positions on the respective report rings. During the “recall location” phase, a grey ring appeared and participants prepared for subsequent location recall without motor movement, to ensure memory decoding without motor interventions. During the following “response period”, subjects serially selected memorized spatial locations on a “location ring”. Next, a color ring appeared (“recall color”) for subjects to be ready for subsequent color recall. They then clicked the remembered colors on a “color ring” (“response period”). (B) Aligned trajectory condition (AT) wherein the trajectory distances between consecutive items (i.e., 1st to 2nd, 2nd to 3rd) of location and color sequences are identical, although the two sequences occupy different locations within their respective rings. (C) Misaligned condition (MAT), wherein the trajectory distances between consecutive items are different for location and color sequences.

Behavioral performance

(A) Memory precision performance of location (black) and color (green) sequences for AT (dark color) and MAT (light color) conditions. Horizontal line in the boxplots denotes the median; box outlines denote the 25th and 75th percentiles; whiskers denote 1.5 × the interquartile range. Extreme values are denoted by crosses. (*: p < 0.05; **: p < 0.01; ***: p < 0.001). (B) Memory precision of 1st (purple), 2nd (turquoise) and 3rd (blue) items of location sequence, for AT (dark color) and MAT (light color) condtions. (C) Memory precision of 1st (purple), 2nd (turquoise) and 3rd (blue) items of color sequence, for AT (dark color) and MAT (light color) condtions. (D) Grand average (mean ± SEM) correlation coefficients of recalled trajectory error between location and color sequences, for 1st-to-2nd trajectory (brown), 2nd-to-3rd trajectory (brickred), and 1st-to-3rd trajectory (orange), under AT (dark color) and MAT (light color) conditions. Dots indicate individual participant. (E) Scatterplot of 1st-to-2nd trajectory memory error for location sequence (X-axis) and Color sequence (Y-axis) under AT condition. Note that the trajectory error of all trials within each subject was divided into 4 bins according to the location trajectory error, resulting in 33 (subject number)*4 (bins) dots in the plot. The brown line represents the best linear fit. (F) Same as E, but for 2nd-to-3rd trajectory. (G) Same as E, but for 1st-to-3rd trajectory.

Neural representation of memory contents during encoding period

(A) Grand average (mean ± SEM) neural decoding (slope of channel response) of location information for the 1st (purple), 2nd (turquoise) and 3rd (blue) disk as a function of time during the encoding period, for AT (left panel) and MAT conditions (right panel). Horizontal lines with corresponding colors denote significant time ranges (cluster-based permutation test, cluster-defining threshold p < 0.001, corrected significance level p < 0.001) (B) Same as A, but for color feature decoding.

Spontaneous color sequence replay during “recall location”

(A) Grand average (mean ± SEM) decoding performance for 1st (purple), 2nd (turquoise) and 3rd (blue) locations as a function of time during “recall locaiton” period, for AT (left panel) and MAT conditions (right panel). (B) Grand average (mean ± SEM) decoding performance for 1st (purple), 2nd (turquoise) and 3rd (blue) colors as a function of time during “recall locaiton” period, for AT (left panel) and MAT conditions (right panel). (Horizontal solid line: cluster-based permutation test, cluster-defining threshold p < 0.05, corrected significance level p < 0.05; Horizontal dashed line: marginal significance, cluster-defining threshold p < 0.1, 0.05 < cluster p < 0.1) (C) Grand average decoding performance within the respective significant time range, for 1st (purple), 2nd (turquoise) and 3rd (blue) colors, under AT (dark color) and MAT (lignt color) conditions. (D) Cross-correlation coefficient, calculated to quantify the extent of the neural representations of adjecent two items followed a forward (positive y) or backward (negative y) transition as a funciton of time lag, between 1st and 2nd colors (brown color) and between 2nd and 3rd (brickred color) colors, and their average (grey color). Dashed vertical line denotes the peak of the averaged cross-correlation time courses. Dashed horizontal lines denote the nonparametric statistical significance threshold (p < 0.05, permutation test). (E) Left panel: theoretical transition pattern for 3-item forward replay, i.e., 1st-2nd-3rd, characterized by cross-correlation at certain time lag. Right panel: empirical transitional pattern (actual cross-correlation matrix) at 130 ms time lag. A significant correlation was found between the two matrices (r = 0.690, p = 0.040), further confirming the forward replay of color sequence.

(A) Grand average (mean ± SEM) neural decoding of 1st-to-2nd as a function of time during the encoding period, for AT condition. (B) Same as A, but for 2nd-to-3rd trajectory. (C) Same as A, but for 1st-to-3rd trajectory. (D) Participants were divided into two groups (higher-correlation group and lower-correlation group), based on their 1st-to-2nd color-location trajectory memory behavioral correlation. Grand average (mean ± SEM) neural decoding of 1st-to-2nd trajectory as a function of time, for higher-correlation group (N =16; left panel) and lower-correlation group (N = 16; right panel). (Horizontal solid line: cluster-based permutation test, cluster-defining threshold p < 0.05, corrected significance level p < 0.05).

(A) Scatterplot of 1st-to-2nd trajectory memory error for location sequence (X-axis) and Color sequence (Y-axis) under MAT condition. Note that the trajectory error of all trials within each subject was divided into 4 bins according to the location trajectory error, resulting in 33 (subject number)*4 (bins) dots in the plot. The brown line represents the best linear fit. (B) Same as A, but for 2nd-to-3rd trajectory. (C) Same as A, but for 1st-to-3rd trajectory. (D) Grand average (mean ± SEM) neural decoding of 1st-to-2nd as a function of time during the encoding period, for MAT condition. (E) Same as A, but for 2nd-to-3rd trajectory. (F) Same as A, but for 1st-to-3rd trajectory.

(A) Left panel: grand average (mean ± SEM) time courses of the decoding performance for the 1st (purple), 2nd (turquoise) and 3rd (blue) WM colors during color recalling period. Right panel: grand average (mean ± SEM) time courses of the decoding performance in MAT condition. (B) Left panel: grand average (mean ± SEM) time courses of the decoding performance for the 1st, 2nd and 3rd WM locations during color recalling period. Right panel: grand average (mean ± SEM) time courses of the decoding performance in MAT condition. (Horizontal solid line: cluster-based permutation test, cluster-defining threshold p < 0.05, corrected significance level p < 0.05; Horizontal dashed line: marginal significance, cluster-defining threshold p < 0.1, 0.05 < cluster p < 0.1)

Participants were divided into two groups based on the average of trajectory decoding performance within the respective significant time range. (A) Grand average (mean ± SEM) time courses of the decoding performance for the 1st, 2nd and 3rd colors during location recalling period for high trajectory representation group. (B) Left panel: theoretical transition pattern for 3-item forward replay. Right panel: empirical transitional pattern (actual cross-correlation matrix) at 130 ms time lag definde in Figure 4D. A significant correlation was found between the two matrices (r = 0.715, p = 0.030), further confirming the forward replay of color sequence for high trajectory representation group. (C) Same as A, but for low trajectory representation group. (D) Same as B, but for low trajectory representation group, nonsignificant correlation between the two matrices was observed (r = 0.230, p = 0.553). (Horizontal solid line: time window with significnat activation (t-test, p < 0.05, without correction across time)