Neural representations of predicted events: Evidence from time-resolved EEG decoding

Reviewer #1 (Public Review):

Summary:

Li and colleagues describe an experiment whereby sequences of dots in different locations were presented to participants while electroencephalography (EEG) was recorded. By presenting fixed sequences of dots in different locations repeatedly to participants, the authors assumed that participants had learned the sequences during the experiment. The authors also trained classifiers using event-related potential (ERP) data recorded from separate experimental blocks of dots presented in a random (i.e., unpredictable) order. Using these trained classifiers, the authors then assessed whether patterns of brain activity could be detected that resembled the neural response to a dot location that was expected, but not presented. They did this by presenting an additional set of sequences whereby only one of the dots in the learned sequence appeared, but not the other dots. They report that, in these sequences with omitted stimuli, patterns of EEG data resembled the visual response evoked by a dot location for stimuli that could be expected, but were not presented. Importantly, this only occurred for an omitted dot stimulus that would be expected to appear immediately after the dot that was presented in these partial sequences.

This exciting finding complements previous demonstrations of the ability to decode expected (but not presented) stimuli in Blom et al. (2020) and Robinson et al. (2020) that are cited in this manuscript. It suggests that the visual system is able to generate patterns of activity that resemble expected sensory events, approximately at times at which an observer would expect them.

Strengths:

The experiment was carefully designed and care was taken to rule out some confounding factors. For example, gaze location was tracked over time, and deviations from fixation were marked, in order to minimise the contributions of saccades to above-chance decoding of dot position. The use of a separate block of dots (with unpredictable locations) to train the classifiers was also useful in isolating visual responses evoked by each dot location independently of any expectations that might be formed during the experiment. A large amount of data was also collected from each participant, which is important when using classifiers to decode stimulus features from EEG data. This careful approach is commendable and draws on best practices from existing work.

Weaknesses:

While there was clear evidence of careful experiment design, there are some aspects of the data analysis and results that significantly limit the inferences that can be drawn from the data. Both issues raised here relate to the use of pre-stimulus baselines and associated problems. As these issues are somewhat technical and may not be familiar to many readers, I will try to unpack each line of reasoning below. Here, it should be noted that these problems are complex, and similar issues often go undetected even by highly experienced EEG researchers.

Relevant to both issues, the authors derived segments of EEG data relative to the time at which each dot was presented in the sequences (or would have appeared when the stimuli were omitted in the partial sequences). Segments were derived that spanned -100ms to 300ms relative to the actual or expected onset of the dot stimulus. The 300ms post-stimulus time period corresponds to the duration of each dot in the sequence (100ms) plus the inter-stimulus interval (ISI) that was 200ms in duration before the next dot appeared (or would be expected to appear in the partial sequences). Importantly, a pre-stimulus baseline was applied to each of these segments of data, meaning that the average amplitude at each electrode between -100ms and 0ms relative to (actual or expected) stimulus onset was subtracted from each segment of data (i.e., each epoch in common EEG terminology). While this type of baseline subtraction procedure is commonplace in EEG studies, in this study design it is likely to cause problematic effects that could plausibly lead to the patterns of results reported in this manuscript.

First of all, the authors compare event-related potentials (ERPs) evoked by dots in the full as compared to partial sequences, to test a hypothesis relating to attentional tuning. They reported ERP amplitude differences across these conditions, for epochs corresponding to when a dot was presented to a participant (i.e., excluding epochs time-locked to omitted dots). However, these ERP comparisons are complicated by the fact that, in the full sequences, dot presentations are preceded by the presentation of other dots in the sequence. This means that ERPs evoked by the preceding dots in the full sequences will overlap in time with the ERPs corresponding to the dots presented at the zero point in the derived epochs. Importantly, this overlap would not occur in the partial sequence conditions, where only one dot was presented in the sequence. This essentially makes any ERP comparisons between full and partial sequences very difficult to interpret, because it is unclear if ERP differences are simply a product of overlapping ERPs from previously presented dots in the full sequence conditions. For example, there are statistically significant differences observed even in the pre-stimulus baseline period for this ERP analysis, which likely reflects the contributions ERPs evoked by the preceding dots in the full sequences, which are absent in the partial sequences.

The problems with interpreting this data are also compounded by the use of pre-stimulus baselines as described above. Importantly, the use of pre-stimulus baselines relies on the assumption that the ERPs in the baseline period (here, the pre-stimulus period) do not systematically differ across the conditions that are compared (here, the full vs. partial sequences). This assumption is violated due to the overlapping ERPs issue described just above. Accordingly, the use of the pre-stimulus baseline subtraction can produce spurious effects in the time period after stimulus onset (for examples see Feuerriegel & Bode, 2022, Neuroimage). This also makes it very difficult to meaningfully compare the ERPs following dot stimulus onset in these analyses.

The second issue relates to the use of pre-stimulus baselines and concerns the key finding reported in the paper: that EEG patterns corresponding to expected but omitted events can be decoded in the partial sequences. In the partial sequences, there are two critical epochs that were derived: One time-locked to the presentation of the dot, and another that was time-locked to 300ms after the dot was presented (i.e. when the next dot would be expected to appear). The latter epoch was used to test for representations of expected, but omitted, stimulus locations.

For the epochs in which the dots were presented, above-chance decoding can be observed spanning a training time range from around 100-300ms and a testing time range of a similar duration (see the plot in Figure 4b). This plot indicates that, during the time window of around 200-300ms following dot stimulus onset, the position of the dot can be decoded not only from trained classifiers using the same time windows spanning 200-300ms, but also using classifiers trained using earlier time windows of around 100-200ms.

This is important because the 200-300ms time period after dot onset in the partial sequences is the window used for pre-stimulus baseline subtraction when deriving epochs corresponding to the first successor representation (i.e., the first stimulus that might be expected to follow from the presented dot, but did not actually appear). In other words, the 200-300ms time window from dot onset corresponds to the -100 to 0 ms time window in the first successor epochs. Accordingly, the pattern that is indicative of the preceding, actually presented dot position would be subtracted from the EEG data used to test for the successor representation. Notably, the first successor condition would always be in another visual field quadrant (90-degree rotated or the opposite quadrant) as stated in the methods. In other words, the omitted stimulus would be expected to appear in the opposite vertical and/or horizontal visual hemifield as compared to the previously presented dot in these partial sequences.

This is relevant because ERPs tend to show reversed polarity across hemifields. For example, a stimulus presented in the right hemifield will have reversed polarity patterns at the same electrode as compared to an equivalent stimulus presented in the left hemifield (e.g., Supplementary Figure 3 in the comparable study of Blom et al., 2020). By subtracting the ERP patterns evoked by the presented dot in the partial sequences during the time period of 200-300ms (corresponding to the -100 to 0ms baseline window), this would be expected to bias patterns of EEG data in the first successor epochs to resemble stimulus positions in opposite hemifields. This could plausibly produce above-chance decoding accuracy in the time windows identified in Figure 5a, where the training time windows broadly correspond to the periods of above-chance decoding during 200-300ms from dot stimulus onset in Figure 4b.

In other words, the above-chance decoding of the first successor representation may plausibly be an artefact of the pre-stimulus baseline subtraction procedure used when deriving the epochs. This casts some doubt as to whether genuine successor representations were actually detected in the study. Additional tests for successor representations using ERP baselines prior to the presented dot in the partial sequences may be able to get around this, but such analyses were not presented, and the code and data were not accessible at the time of this review.

Although the study is designed well and a great amount of care was taken during the analysis stage, these issues with ERP overlap and baseline subtraction raise some doubts regarding the interpretability of the findings in relation to the analyses currently presented.

Reviewer #2 (Public Review):

Summary:

The authors investigated how predicted stimuli are represented in posterior regions of the brain by recording electroencephalography during a visual sequence learning task. After learning the spatial order in which four stimuli were presented within a fixed sequence, participants were shown partial sequences - i.e., sequences in which only one element of the sequence was presented. By examining the decoding accuracy of the omitted stimuli, the authors aimed to investigate whether anticipated stimuli are (pro)actively represented in the expected spatial location at the expected time.

Strengths and Weaknesses:

The study successfully replicated previous findings on omitted stimuli within a predicted sequence (Ekman et al., 2023), while providing novel information regarding the temporal dynamics of predictive representation. Nevertheless, this outcome is not entirely surprising, as similar temporal dynamics were observed in a previous study employing a different task (Kok et al., 2017). The high level of scientific rigor is evident, as demonstrated by the numerous control analyses. Additionally, the results are particularly intriguing in terms of discerning the nature of the prepared representation, spanning from early perceptual to late attentional representations. Unfortunately, this distinction is not investigated in detail, thus allowing for alternative interpretations of the results.

The connection between the findings and the literature on priority maps could benefit from further clarification. There is room for a clearer delineation of how much of the representation can be ascribed to a perceptual prediction mechanism versus an attentional (post-perceptual) spatial cueing process. Although the latter can be readily connected to the concept of a priority map guiding spatial attention, the relationship between the priority map and perceptual prediction remains somewhat ambiguous. Noteworthy, an explanation of the results in terms of spatial cueing does not necessarily require a perceptual predictive mechanism. The significant decoding of the location of the omitted stimulus might be attributed to the preceding stimulus orienting attention towards the following location. While this potential explanation was not explicitly addressed in the study, it presents an intriguing avenue for further investigation.

The study provides valuable insight into how omitted, yet predicted stimuli are represented in the brain and its dynamics. While the research is commendable, addressing the outlined limitations would enhance its impact in the field. Specifically, the spatial location decoding results do not disentangle between perceptual prediction (i.e., the features of the expected stimulus) and attentional processes (i.e., the cueing of the to-be-attended location),

Reviewer #3 (Public Review):

Summary:

The authors conducted a time-resolved EEG decoding study where they presented sequences of dot locations (4 locations onscreen) or single elements of those sequences, presented at the correct temporal epoch for if they had been presented in the full sequence. They were interested in examining whether presenting single items would activate representations of the anticipated following events that were never presented. Stimuli were presented for 100 ms and separated by 200 ms ISIs. They also had pattern estimation blocks with 600 ms ISIs. They found indeed, that anticipated events could be decoded at their correct moment in time, although future anticipated elements could not.

The decoding of presented dots was fairly confined to the diagonal of the decoding matrix (training time x testing time), suggesting little temporal generalisation. This was in contrast with successor representations which were temporally more diffuse. The subsequent successor could be decoded but not future successors.

Strengths:

I liked this paper. The design was simple and clean and the implications of the findings are clear. The authors achieved their aims with this design, with the results supporting the conclusions. The findings will be of interest to a range of researchers studying learning and perception mechanisms, as well as the more generic role of prediction in the brain.

Weaknesses:

The sample size is fairly low for an EEG study. The authors justify it according to a previous Hogendoorn study, but not according to effect sizes in that study and particular power values.

For understandable reasons, the long ISI blocks were presented before the main test blocks (I would have made the same decision) but there is the risk that participants then come to expect stimuli at larger temporal separations in the main blocks. I do wonder whether this is part of the reason for the greater temporal generalisation for anticipated event representations.

Additional context:

My memory of Ekman et al. 2017 is that single events (presented at position 1) elicited predictive activation of anticipated future events, but that there was a temporal compression. The present study appears to show no temporal compression but that the representations are activated at the correct moment in time. This seems like a potentially interesting difference and one with mechanistic implications for the field.