Non-feature-specific elevated responses and feature-specific backward replay in human brain induced by visual sequence exposure

Reviewer #1 (Public review):

Summary:

The study identifies two types of activation: one that is cue-triggered and non-specific to motion directions, and another that is specific to the exposed motion directions but occurs in a reversed manner. The finding that activity in the medial temporal lobe (MTL) preceded that in the visual cortex suggests that the visual cortex may serve as a platform for the manifestation of replay events, which potentially enhance visual sequence learning.

Evaluations:

Identifying the two types of activation after exposure to a sequence of motion directions is very interesting. The experimental design, procedures and analyses are solid. The findings are interesting and novel.

In the original submission, it was not immediately clear to me why the second type of activation was suggested to occur spontaneously. The procedural differences in the analyses that distinguished between the two types of activation need to be a little better clarified. However, this concern has been satisfactorily addressed in the revision.

Reviewer #2 (Public review):

This paper shows and analyzes an interesting phenomenon. It shows that when people are exposed to sequences of moving dots (That is moving dots in one direction, followed by another direction etc.), that showing either the starting movement direction, or ending movement direction causes a coarse-grained brain response that is similar to that elicited by the complete sequence of 4 directions. However, they show by decoding the sensor responses that this brain activity actually does not carry information about the actual sequence and the motion directions, at least not on the time scale of the initial sequence. They also show a reverse reply on a highly-compressed time scale, which is elicited during the period of elevated activity, and activated by the first and last elements of the sequence, but not others. Additionally, these replays seem to occur during periods of cortical ripples, similar to what is found in animal studies.

These results are intriguing. They are based on MEG recordings in humans, and finding such replays in humans is novel. Also, this is based on what seems to be sophisticated statistical analysis. The statistical methodology seems valid, but due to its complexity it is not easy to understand. The methods especially those described in figures 3 and 4 should be explained better.

Author response:

The following is the authors’ response to the original reviews.

Public Reviews:

Reviewer #1 (Public review):

Summary:

The study identifies two types of activation: one that is cue-triggered and nonspecific to motion directions, and another that is specific to the exposed motion directions but occurs in a reversed manner. The finding that activity in the medial temporal lobe (MTL) preceded that in the visual cortex suggests that the visual cortex may serve as a platform for the manifestation of replay events, which potentially enhance visual sequence learning.

Strengths:

Identifying the two types of activation after exposure to a sequence of motion directions is very interesting. The experimental design, procedures, and analyses are solid. The findings are interesting and novel.

Weaknesses:

It was not immediately clear to me why the second type of activation was suggested to occur spontaneously. The procedural differences in the analyses that distinguished between the two types of activation need to be a little better clarified.

We thank the reviewer for his/her summary and constructive feedback on our study. We appreciate the recognition of the strengths of our study.

The second type of activation, namely the replay of feature-specific reactivations, is considered spontaneous because it reflects internally driven neural processes rather than responses directly triggered by external stimuli. Unlike responses evoked by stimuli, spontaneous replay is not time-locked to stimulus onset. Instead, it arises from the brain's intrinsic activity, typically observed during offline periods (e.g., rest or blank period) when external stimuli are absent. This allows the neural system to reactivate and consolidate prior experiences without interference from ongoing external stimuli.

Replay is believed to be a key mechanism underlying various cognitive functions, such as memory consolidation (Gillespie et al., 2021; Gridchyn et al., 2020), learning (Igata et al., 2021), prediction and planning (Ólafsdóttir et al., 2018). Furthermore, the hippocampus and related cortical areas engage in replay to extract abstract relationships from sequential experiences, forming a "template" that can generalize across contexts (Liu et al., 2019). In our study, the feature-specific replay observed during blank periods likely reflects this process, supporting the integration of exposed motion direction sequences into cohesive memory representations and facilitating visual sequence learning.

We have extended the Discussion section to incorporate this explanation (Lines 440 - 447).

Regarding the second question, the procedural differences between the two types of activations lie in the classifiers used for the two analyses: a multiclass classifier for non-specific elevated responses and binary classifiers for feature-specific replay.

For the non-feature-specific elevated responses, we trained a five-class (with the labels of the four RDKs and the ITI (inter-stimulus interval)) classifier on the localizer data and tested on the blank period in the main phase. We attempted to decode motion direction information at each time point at the group level. However, the results revealed no feature-specific information at the group level during the blank period.

For the feature-specific replay, we employed the temporal delayed linear modeling (TDLM) to examine whether individual motion direction information was encoded in a sequential and spontaneous manner. Here, we first needed to train four binary classifiers, each was sensitive to only one motion direction (i.e., 0°, 90°, 180°, or 270°), as our aim was to quantify the evidence of feature-specific sequence in the subsequent analyses. For each classifier, positive instances were trials where the corresponding feature (e.g., 0°) was presented, while negative instances included trials with other features (e.g., 90°, 180°, and 270°) and an equivalent amount of null data from the ITI period (1–1.5 s).

We have clarified these methodological details in the Methods section (Pages 34 – 41).

Reviewer #2 (Public review):

This paper shows and analyzes an interesting phenomenon. It shows that when people are exposed to sequences of moving dots (that is moving dots in one direction, followed by another direction, etc.), showing either the starting movement direction or ending movement direction causes a coarse-grained brain response that is similar to that elicited by the complete sequence of 4 directions. However, they show by decoding the sensor responses that this brain activity actually does not carry information about the actual sequence and the motion directions, at least not on the time scale of the initial sequence. They also show a reverse reply on a highly compressed time scale, which is elicited during the period of elevated activity, and activated by the first and last elements of the sequence, but not others. Additionally, these replays seem to occur during periods of cortical ripples, similar to what is found in animal studies.

These results are intriguing. They are based on MEG recordings in humans, and finding such replays in humans is novel. Also, this is based on what seems to be sophisticated statistical analysis. However, this is the main problem with this paper. The statistical analysis is not explained well at all, and therefore its validity is hard to evaluate. I am not at all saying it is incorrect; what I am saying is that given how it is explained, it cannot be evaluated.

We thank the reviewer’s detailed evaluation as well as the acknowledgment of the novelty of our study.

To address the concern about the statistical analysis, in the revised manuscript, we have modified the Methods section to provide a more detailed explanation of the analytical pipeline, particularly for several important aspects such as decoding probability and TDLM. (Lines 646 – 657, Lines 682 – 734).

Below, we provide point-by-point responses to further elaborate on these revisions and address the reviewer’s comments.

Recommendations for the authors:

Reviewer #1 (Recommendations for the authors):

I have questions.

(1) Participants were exposed to a predefined sequence of motion directions either clockwise or counterclockwise. Is it possible that the observed replay is related to the activation of MST neurons? If a predetermined sequence is not in either clockwise or counterclockwise but is randomly determined like 0{degree sign}->180{degree sign}->270{degree sign}->90{degree sign}, would the same result be obtained?

We thank the reviewer for these thoughtful questions.

First, regarding the potential involvement of MST neurons, it is plausible that the observed replay might involve activity in motion-sensitive brain regions, including the medial superior temporal (MST) and even middle temporal (MT) areas. MST neurons, located in the extrastriate visual cortex, are highly direction-selective and are known for their sensitivity to complex motion patterns, such as rotations and expansions (Duffy & Wurtz, 1991; Saito et al., 1986). In our experiment, the use of RDKs with four distinct motion directions might elicit responses in MST neurons. However, due to the limited spatial resolution of MEG, we cannot provide direct evidence for this claim.

Second, regarding the impact of randomly ordered sequences, we believe that the replay patterns would still occur even if the sequences were randomly ordered (e.g., 0° → 180° → 270° → 90°). After a sequence is repeatedly exposed, the hippocampus has the capacity to encode abstract relationships in the sequence. Evidence supporting this view comes from previous studies. For example, Liu et al., (2019) showed that replay does not merely recapitulate visual experience but can also follow a sequence implied by learned abstract knowledge. In their study, participants were instructed that viewing pictures C→D, B→C, and A→B implies a true sequence of A→B→C→D. During subsequent testing, they observed replay events following this learned true sequence, even with novel visual stimuli, indicating that the brain maintains sequence knowledge independent of specific stimuli. Similarly, Ekman et al., (2023) showed that prediction-based neural responses could be observed when moving dots were presented in a random order rather than in a clockwise or counterclockwise order, which correspond to the four motion directions in our study.

Together, these studies suggest that replay mechanisms in the brain are flexible and can encode and reproduce abstract relationships between sequential stimuli, regardless of their specific spatial contents. Therefore, we believe that even if the sequence were randomly ordered, the same backward replay pattern would still be observed.

(2) Is it possible that the motion direction non-specific responses actually reflect the replay of another feature of the exposed sequence, namely, the temporally rhythmic presentations of the sequence, rather than suggested in the discussion?

We thank the reviewer for raising this insightful possibility.

There is substantial evidence that rhythmic stimulation can entrain neural oscillations, which in turn facilitates predictions about future inputs and enhances the brain's readiness for incoming stimuli (Barne et al., 2022; Herrmann et al., 2016; Lakatos et al., 2008, 2013). In our study, the temporally rhythmic presentation of the motion sequence may have entrained oscillatory activity in the brain, leading to periodic activation of sensory cortices. This rhythmic entrainment could account for the observed nonspecific responses by reflecting the brain's temporal predictions rather than specific feature replay.

It is important to note that, however, this interpretation is in line with our initial explanation that the non-feature-specific elevated responses likely reflect a general facilitation of neural processes for any upcoming stimuli, rather than being tied to specific stimuli. The rhythmic entrainment mechanism provides another way to understand how the temporal structure in the sequences might contribute to the non-feature-specific elevated responses.

We have revised the Discussion section to incorporate this interpretation, providing a more comprehensive account for the non-feature-specific elevated responses (Lines 428 – 439).

Reviewer #2 (Recommendations for the authors):

The main problem with the paper is that the sophisticated statistical methodology is not explained well and therefore its validity is hard to evaluate. I am not at all saying it is incorrect, what I am saying is that given how it is explained, it cannot be evaluated.

See below for detailed point-by-point responses.

The first part is clear. There are 4 directions of motion, and there can also be a blank screen. The random decoding accuracy would be 20%. The decoding methods from the sensors yielded a little above 50% accuracy. This is clearly about chance, but much less than one would get from electrode recording of motion-selective cells in the cortex. However, the concept and methods used here seem clear, in contrast to what comes next.

Indeed, in the first step, we aimed to validate the reliability of our decoding model by applying a leave-one-out cross validation scheme to the localizer data. Our results showed that the decoding accuracy exceeded 50%, demonstrating robust decoding performance. However, due to the noninvasive nature of MEG and its low spatial resolution, the recorded signals represent population-level activity that inherently includes more noise compared to electrode recordings of motion-selective neurons. Therefore, the decoding accuracy in our study is understandably lower than that obtained with electrode recordings.

Next, and most of the paper relies on this concept, they use the term decoding probability (Figure 2). What is the decoding probability measure (Turner 2023)? This is not explained in the methods section. I scanned the Turner et al 2023 paper referenced and could not find the term decoding probability there. In short, I have no idea what this means. What are these numbers between 0-0.3? How does this relate to accuracies above 50% reported? This is an important concept here, and it is used throughout the paper, so it makes it hard to evaluate the paper.

We apologize for the lack of clarity in our explanation of the term "decoding probability." Specifically, we used a one-versus-rest Lasso logistic regression model trained on the localizer data to decode the MEG signal patterns elicited by each motion direction during the main phase. The trained model could be used to predict a single label at each time point for each trial (e.g., labels 1 – 4 correspond to the four motion directions and label 5 corresponds to the ITI period). By comparing the predicted label with the true label across test trials, we could compute the time-resolved decoding accuracy as final reports.

Alternatively, rather than predicting a single label for each time point and each trial, the model can also output the probabilities associated with each label/class (e.g., we used the predict_proba function in scikit-learn). This results in a 5-column output, where each column represents the probability of the corresponding class, and the sum of the probabilities across the five columns equals 1. Finally, at each time point, averaging these probabilities across trials yields five values that indicate the likelihood of the predicted stimulus belonging to each class.

For example, Figure 2 in the manuscript depicts the decoding probabilities for the four RDKs (the probabilities for the ITI class are not shown in the figure). The number in a cell (between 0 and 0.3) indicates the probability of each class at a given time point (Figure 2A). The decoding probability does not have a direct relationship with the decoding accuracy. However, since there are five classes, the chance level of the decoding probability is 0.2. The highest probability among the five classes at a given time point determines the decoded label when computing the decoding accuracy.

For illustration, in the left panel of Figure 2B, at the onset of the first RDK (0 s), the mean decoding probabilities for the classes 0°, 90°, 180°, 270°, and the blank ITI are 5%, 4.1%, 4.0%, 4.5%, and 82.4%, respectively. Thus, the decoded label should be the blank ITI. In contrast, 0.4 s after the onset of the first RDK, the mean decoding probabilities for the five classes are 28.0%, 19.0%, 22.8%, 21.2%, and 9.0%, respectively. Therefore, the decoded label should be 0°.

We have revised the Methods section to explain this issue (Lines 646 – 657).

They did find compressed reversed reply events (Figures 3-4). This is again confusing for several reasons. First, because they use the same unexplained decoding probability measure. Second, the optimal time point defined above depends on the start time of a stimulus, but here the start time is random. Third, the TDLM algorithm is hard to understand. For example, what are the reactivation probabilities of Figure 3C? They do make an effort to explain this in the methods section (lines 652-697) but it's not clear enough from the outset. For example, what does the state X_j is this a vector of activity of sensors? Are these decoding probabilities of the different directions? What is it? Also, what is X_i vs X_i(\Delta t)? Frankly, despite their efforts, I am very confused. Additionally, the figures use the term reactivation probability, where is it defined? So again, the results seem interesting, but the methods are not explained well at all.

This paper must better explain the statistical methods so that they can be evaluated. This is not easy, these are relatively complex methods, but they must be explained much better so the validity of the paper can be examined.

Regarding the optimal time point, we defined it as the time point with the highest decoding accuracy, determined during the validation of the localizer data using a leave-one-out cross-validation scheme. This optimal time point was participant- and motion-direction-specific, as the latency to achieve the peak decoding accuracy varied across individuals and motion directions. For group-level visualization, we circularly shifted the data over time, aligning each optimal time point to a common reference point (arbitrarily set at 200 ms after stimulus onset). Importantly, however, these time points are unrelated to the data in the main phase, as the models were trained using the independent localizer data and then applied to each time point during the blank period in the main phase.

Regarding the TDLM algorithm, detailed descriptions of the algorithm have been provided in the revised Methods section (Line 683 – 735). Furthermore, we have included explanatory notes in the main text and figure legend to provide immediate context for terms such as "reactivation probability" (Lines 247 – 248, Lines 275 – 276).

This paper uses MEG in humans, a non-invasive technique. This allows for such results in humans. Indeed (if the methods are correct) these units can be decoded to provide statistically significant estimates of motion direction. Note, however, that the spatial resolution of MEG is limited. The decoding accuracies of above 50% are way above chance. Note however that if actual motion-sensitive neurons (e.g. area MT) were recorded, and even if the motion is far from 100% coherence, the decoding accuracy would approach 100%.

We agree with the reviewer that decoding accuracy would approach 100% if single-neuron data from motion-sensitive areas (e.g., area MT) were recorded, given the exceptionally high signal-to-noise ratio (SNR) of such data. However, two considerations inform the methodology of our study.

First, while single-neuron recordings provide invaluable insights, acquiring such data in humans is both ethically challenging and logistically impractical.

Non-invasive MEG, by contrast, offers a practical alternative that can achieve robust decoding of population-level activity with a reasonable SNR.

Second, the primary goal of our study was not merely to achieve high decoding accuracy but also to examine the replay of an exposed motion sequence in the human visual cortex. To achieve this, we first needed to train feature-specific models that can be used to decode the spontaneous reactivations of the four motion directions during the blank period. The ability to distinguish representations of the four motion directions was essential for calculating the “sequenceness” of the exposed motion sequence in the TDLM algorithm. While the absolute decoding accuracy of MEG data may not match that of single-neuron data, an important outcome was the successful construction of feature-specific models for the four motion directions (Figure 3B in the manuscript). These models provided a robust foundation for investigating sequential replay in the brain. These results also align with the broader goal of leveraging MEG data to study dynamic neural processes in humans, even in the face of its spatial resolution limitation.

Minor:

(1) Line 246 - there is no figure S2A, subplots are not labeled.

We have corrected this in the revised manuscript.

(2) Is Figure 3B referred to in the text? Same for 3C. This figure is there for explaining the statistical models used, but it is not well utilized.

We have modified the text to clarify this issue in the revised manuscript.

(3) English:

There are problems with the use of English in the paper, this should be corrected in the next version. A few examples are below.

Noises -> noise

- "along the motion path in visual cortex" What does this sentence mean? Is this referring to motion-sensitive areas in the brain? Please clarify.

There are many other examples. This is minor, but should be corrected.

We have corrected these errors in the revised manuscript.

References

Barne, L. C., Cravo, A. M., de Lange, F. P., & Spaak, E. (2022). Temporal prediction elicits rhythmic preactivation of relevant sensory cortices. European Journal of Neuroscience, 55(11–12), 3324–3339. https://doi.org/10.1111/ejn.15405

Ekman, M., Kusch, S., & de Lange, F. P. (2023). Successor-like representation guides the prediction of future events in human visual cortex and hippocampus. eLife, 12, e78904. https://doi.org/10.7554/eLife.78904

Gillespie, A. K., Maya, D. A. A., Denovellis, E. L., Liu, D. F., Kastner, D. B., Coulter, M. E., Roumis, D. K., Eden, U. T., & Frank, L. M. (2021). Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice. Neuron, 109(19), 3149-3163.e6. https://doi.org/10.1016/j.neuron.2021.07.029

Gridchyn, I., Schoenenberger, P., O’Neill, J., & Csicsvari, J. (2020). AssemblySpecific Disruption of Hippocampal Replay Leads to Selective Memory Deficit. Neuron, 106(2), 291-300.e6. https://doi.org/10.1016/j.neuron.2020.01.021

Herrmann, B., Henry, M. J., Haegens, S., & Obleser, J. (2016). Temporal expectations and neural amplitude fluctuations in auditory cortex interactively influence perception. NeuroImage, 124, 487–497. https://doi.org/10.1016/j.neuroimage.2015.09.019

Igata, H., Ikegaya, Y., & Sasaki, T. (2021). Prioritized experience replays on a hippocampal predictive map for learning. Proceedings of the National Academy of Sciences, 118(1), e2011266118. https://doi.org/10.1073/pnas.2011266118

Lakatos, P., Karmos, G., Mehta, A. D., Ulbert, I., & Schroeder, C. E. (2008). Entrainment of Neuronal Oscillations as a Mechanism of Attentional Selection. Science, 320(5872), 110–113. https://doi.org/10.1126/science.1154735

Lakatos, P., Musacchia, G., O’Connel, M. N., Falchier, A. Y., Javitt, D. C., & Schroeder, C. E. (2013). The Spectrotemporal Filter Mechanism of Auditory Selective Attention. Neuron, 77(4), 750–761. https://doi.org/10.1016/j.neuron.2012.11.034

Liu, Y., Dolan, R. J., Kurth-Nelson, Z., & Behrens, T. E. J. (2019). Human Replay Spontaneously Reorganizes Experience. Cell, 178(3), 640-652.e14. https://doi.org/10.1016/j.cell.2019.06.012

Ólafsdóttir, H. F., Bush, D., & Barry, C. (2018). The Role of Hippocampal Replay in Memory and Planning. Current Biology, 28(1), R37–R50. https://doi.org/10.1016/j.cub.2017.10.073