Anticipation of temporally structured events in the brain

To identify anticipatory signals in the brain, we examined TR-by-TR brain activity patterns during each of the six viewings of the movie clip. For each spherical searchlight within the brain volume, we fit an HMM jointly to all repetitions, to identify a sequence of event patterns common to all viewings and the timing of spatial pattern changes for each viewing. At each timepoint for each viewing, the HMM produced a probability distribution that describes the mixture of event patterns active at that timepoint. Computing the expected value of this distribution provides an index of how the brain transitions through event patterns on each viewing, allowing us to identify how this timing shifts within each region of the brain.

Our analysis revealed temporal shifts in event patterns in many brain regions, including lateral occipital cortex, angular and supramarginal gyri, lateral and anterior temporal lobe, lateral and medial prefrontal cortex (mPFC), and insular cortex (Figure 2). The magnitude of this shift varied along a posterior-to-anterior temporal hierarchy (Spearman’s rho = 0.58, p=0.0030), with the most anterior regions in the temporal pole and prefrontal cortex showing shifts of up to 15 s on subsequent viewings compared to the first viewing. This hierarchy persisted even when computed on the unthresholded anticipation map including voxels that did not meet the threshold for statistical significance (Spearman’s rho = 0.42, p=0.0028; see Figure 2—figure supplement 1). There were no significant correlations with the left-to-right axis (rho = 0.06, p=0.41 for thresholded map; rho = 0.12, p=0.29 for unthresholded map) or the inferior-to-superior axis (rho = 0.07, p=0.28 for thresholded map; rho = −0.11, p=0.73 for unthresholded map). We obtained a similar map when comparing the first viewing to just the sixth viewing alone (see Figure 2—figure supplement 2).

Figure 2 with 4 supplements see all

Download asset Open asset

We compared how this hierarchy of anticipation timescales related to the intrinsic processing timescales in each region during the initial viewing of the movie clip. Identifying the optimal number of HMM events for each searchlight, we observed a timescale hierarchy similar to that described in previous work, with faster timescales in sensory regions and slower timescales in more anterior regions (Figure 2—figure supplement 3a). Regions with longer intrinsic timescales also showed a greater degree of anticipation with repeated viewing (Figure 2—figure supplement 3b).

We also compared these results to those obtained by using a simple cross-correlation approach, testing for a fixed temporal offset between the responses to initial and repeated viewing. This approach did detect significant anticipation in some anterior regions, but was much less sensitive than the more flexible HMM fits, especially in posterior regions (Figure 2—figure supplement 4).

Our data-driven method for identifying event structure in fMRI data does not make use of information about the content of the stimulus, leaving open the question of how the HMM-identified event boundaries correspond to subjective event transitions in the movie. One possibility is that the brain’s event boundaries could start well-aligned with event boundaries in the movie and then shift earlier (indicating anticipation of upcoming stimulus content). Alternatively, they may initially lag behind stimulus boundaries (reflecting a delayed response time on initial viewing) and then shift to become better aligned with movie scene transitions on repeated viewings. Finally, both patterns may exist simultaneously in the brain, but in different brain regions.

We asked human raters to identify event transitions in the stimulus, labeling each ‘meaningful segment’ of activity (Figure 3). To generate a hypothesis about the strength and timing of event shifts in the fMRI data, we convolved the distribution of boundary annotations with a hemodynamic response function (HRF) as shown in Figure 4. We then explored alignment between these human-annotated event boundaries and the event boundaries extracted from the brain response to each viewing, as shown in Figure 1d. In each searchlight, we cross-correlated the brain-derived boundary timecourse with the event annotation timecourse to find the temporal offset that maximized this correlation.

Figure 3

Download asset Open asset

Figure 4

Download asset Open asset

We found three clusters in the middle temporal gyrus (MTG), fusiform gyrus (FG), and superior temporal sulcus (STS) in which the optimal lag for the repeated viewings was significantly earlier than for the initial viewing, indicating that the relationship between the brain-derived HMM event boundaries and the human-annotated boundaries was changing with repeated viewings (Figure 5). The HMM boundaries on the first viewing were significantly later than the annotated boundaries in FG and STS, while the optimal lag did not significantly differ from 0 in MTG (95% confidence intervals for the optimal lag, in seconds: MTG = [−0.27, 2.86]; FG = [0.14, 1.99]; STS = [1.48, 8.53]). The HMM boundaries on repeated viewings were significantly earlier than the annotated boundaries in all three regions (95% confidence intervals for the average optimal lag, in seconds: MTG = [−4.06, –1.83]; FG = [−1.56, –0.26]; STS = [−3.06, –1.69]).

Figure 5

Download asset Open asset

One region showing long-timescale anticipatory signals was the bilateral anterior insula. This region has been linked to anticipation of diverse categories of positive and negative outcomes (Liu et al., 2011), including outcomes that will be experienced by other people (Singer et al., 2009). The movie stimulus used in our experiment depicts an interview in which the protagonist is initially judged to have ‘zero’ experience but then ends up impressing the interviewer, allowing for anticipation of this unexpected social outcome only on repeat viewings. Other regions showing long timescales of anticipation include the medial prefrontal cortex (mPFC), which tracks high-level narrative schemas (Baldassano et al., 2018) and has been proposed to play a general role in event prediction (Alexander and Brown, 2014), and lateral prefrontal cortex, including the inferior frontal gyrus, which processes structured sequences across multiple domains (Uddén and Bahlmann, 2012).

We also observed shorter-timescale anticipation throughout lateral occipital and ventral temporal cortex, which, though primarily thought to process bottom-up visual information, also exhibits event-specific patterns during recall (Chen et al., 2017). A top-down memory-driven signal could be responsible for driving anticipatory activation in these regions during repeated movie viewing (Finnie et al., 2021). Future work incorporating eye-tracking measurements could determine whether anticipatory eye movements can account for the temporal shifts in these regions, or if this anticipation is separate from the representation of the current retinal input.

We did not observe widespread anticipatory signals in primary sensory areas, although some prior fMRI studies have been able to observe such signals in early regions such as V1 (Alink et al., 2010; Ekman et al., 2017; Hindy et al., 2016; Kok et al., 2012). One possibility is that the rich, ongoing sensory input dominated relatively small anticipatory signals in these regions. Paradigms involving periods without any sensory input (e.g., occasionally removing the audiovisual movie from the screen during repeated viewings) may be necessary to detect these subtle signals. Alternatively, ultra-fast fMRI sequences (Ekman et al., 2017) or alternative imaging modalities (discussed below) may be required to track anticipation at a subsecond scale.

Previous work has identified cumulatively longer timescales up the cortical hierarchy but has primarily focused on representations of the past. Lerner et al., 2011 demonstrated hierarchical cortical dynamics in participants who listened to variants of a 7 min narrative that was scrambled at different timescales (e.g., paragraphs, sentences, or words). Response reliability, measured as the correlation in BOLD activity timecourses across individuals, varied based on the timescale of scrambling, with higher-level brain regions responding consistently to only the more-intact narrative conditions. This led to the idea that higher-order brain regions contain larger ‘temporal receptive windows’ than lower-order areas, in that their activity at a given moment is influenced by relatively more of the past. Likewise, using intracranial EEG (iEEG), Honey et al., 2012 observed progressively longer temporal receptive windows in successive stages of the cortical hierarchy in participants who watched intact and scrambled versions of the movie Dog Day Afternoon. These findings can be described by the process memory framework (Hasson et al., 2015), where hierarchical memory timescales process, represent, and support longer and longer units of information. We found that this hierarchy also exists in the prospective direction, with the degree of anticipatory temporal shifts increasing from posterior-to-anterior regions of the brain. Furthermore, regions with longer intrinsic processing timescales showed further-reaching anticipation. These results extend the process memory framework, suggesting that the timescales in these regions are relevant not only for online processing and memory, but also for future anticipation or simulation.

Although prior work has uncovered anticipatory and predictive coding in the brain, most studies have examined fixed, shorter timescales of anticipation. Moreover, these shorter timescales have often been studied using simple, non-narrative stimuli such as objects moving across the screen, short visual sequences, and visual pattern completion tasks (Alink et al., 2010; Ekman et al., 2017; Gavornik and Bear, 2014; Hindy et al., 2016; Kok et al., 2012). Some studies have used dynamic movie stimuli, but anticipation was measured via correlations between initial and repeated viewing of a movie at a constant fixed lag of 2 s (Richardson and Saxe, 2020). Such an approach is not well suited to capturing dynamic levels of anticipation within and across brain regions.

Research investigating longer timescales of anticipation, such as learning future state representations in a maze task, examined single timescales up to 30 s ahead in OFC-VMPFC regions (Elliott Wimmer and Büchel, 2019). Some studies that use narrative stimuli have examined specially constructed texts in order to manipulate predictions about upcoming sentences; for example, work by Kandylaki et al., 2016 demonstrated that predictive processing of referents in narratives can be modulated by voice (passive vs. active) and causality (high vs. low). Our results show that in a naturalistic setting, in which structure exists at many timescales, anticipation at multiple levels can occur in parallel across different brain regions. We found anticipation up to approximately 15 s into the future with our 90 s stimulus, but future work with stimuli of longer duration could uncover even longer timescales of anticipation, on the scale of minutes. Simultaneously maintaining expectations at varying timescales could allow for flexible behaviors, because different timescales of anticipation may be helpful for a variety of tasks and actions. Taking action to avoid immediate harm or danger would require shorter timescales of prediction, whereas cultivating social relationships demands predictions on longer timescales.

These results are consistent with those of Baldassano et al., 2017, in which some participants listening to an audio narrative had advance knowledge of the high-level events of the story (because they had previously watched a movie version of the narrative). Using a similar HMM approach as in this paper, the authors observed shifts in event boundaries in higher-level regions including angular gyrus, posterior medial cortex, and mPFC. In the current study, however, participants were repeatedly exposed to an identical movie stimulus, allowing them to generate expectations at a broad range of timescales, including the timescales of fast-changing low-level visual features. This novel approach allowed us to observe for the first time that anticipation occurs in both low- and high-level regions, with shorter-timescale anticipation in visual occipital regions and the furthest-reaching anticipatory signals in prefrontal cortex.

Our model detects anticipation as temporal shifts in events, and though timepoints can reflect ‘mixed’ event assignments, it assumes that the underlying event patterns themselves (Figure 1b) are constant. This view of anticipation is complementary to other theories of predictive representations, in which event patterns themselves should change over time to incorporate future information. One example is the ‘successor representation’ model from the field of reinforcement learning, which describes a representation in which each state (here, event representation) comes to include features of future events, weighted by their likelihood of occurring and their distance into the future (Dayan, 1993). Successor representations can also be constructed at multiple scales (by changing the relative weighting of events near vs. far in the future). Such multi-scale representations are useful for goal-directed prediction that require multiple stages of planning (Momennejad and Howard, 2018; Brunec and Momennejad, 2019). Future work could explore how these two different theories could be integrated to model both mixing of event patterns and temporal shifts in the activation of these event patterns.

The current fMRI study is complementary to investigations of memory replay and anticipation that use MEG and iEEG. In an MEG study, Michelmann et al., 2019 found fast, compressed replay of encoded events during recall, with the speed of replay varying across the event. Furthermore, an iEEG investigation found anticipatory signals in auditory cortex when individuals listened to the same story twice (Michelmann et al., 2020). In another MEG study, Wimmer et al., 2020 found compressed replay of previously encoded information. Replay was forward when participants were remembering what came after an event, and backward when participants were remembering what came before an event. The forward replay observed in the Wimmer et al. study may be similar to the anticipatory signals observed in the current study, although there was no explicit demand on memory retrieval in our paradigm. Thus, one possibility is that the anticipatory signals observed in MEG or iEEG are the same as those we observe in fMRI, except that they are necessarily sluggish and smoothed in time when measured via a hemodynamic response. This possibility is supported by fMRI work showing evidence for compressed anticipatory signals, albeit at a slower timescale relative to MEG (Ekman et al., 2017).

An alternative possibility is that the anticipatory signals measured in our study are fundamentally different from those captured via MEG or iEEG. That could explain why we failed to find widespread anticipatory signals in primary visual or primary auditory cortex: the anticipatory signals in those regions might have been too fast to be captured with fMRI, particularly when competing with incoming, dynamic perceptual input. Future studies that obtain fMRI and MEG or iEEG in participants watching the same movie would be informative in that regard. It is possible that fMRI may be particularly well suited for capturing relatively slow anticipation of stable events, as opposed to faster anticipatory signals relating to fast sub-events. Nevertheless, advances in fMRI analyses may allow the detection of very fast replay or anticipation, closing the gap between these methods and allowing more direct comparisons (Wittkuhn and Schuck, 2021).

One limitation of the current work is the reliance on one movie clip. Movie clips of different durations might yield different results. For example, it is an open question whether the duration of anticipation scales with the length of the movie and playback speed or if the amount of anticipation is fixed (Lerner et al., 2014; Baumgarten et al., 2021). Furthermore, the content of the movie and how frequently event boundaries occur may change anticipation amounts. That said, anticipatory signals in naturalistic stimuli have been observed across multiple studies that use different movies and auditorily presented stories (e.g., Baldassano et al., 2017; Michelmann et al., 2020; also see Michelmann et al., 2019; Elliott Wimmer and Büchel, 2019; Wimmer et al., 2020). Thus, it is likely that anticipatory hierarchies will also replicate across different stimuli. There may nevertheless be important differences across stimuli. For example, the specific regions that are involved in anticipation may vary depending on what the most salient features of a movie or narrative are (e.g., particular emotional states, actions, conversations, or perceptual information).

The detection of varying timescales of anticipation in the brain can be applied to multiple domains and modalities of memory research. Future work could explore even shorter timescales using other neuroimaging modalities, or longer timescales using longer movies or narratives from TV series that span multiple episodes. Furthermore, the impact of top-down goals on the hierarchy of anticipation timescales could be explored by using different tasks that require different levels of anticipation, such as anticipating camera angle changes vs. location changes. Brain stimulation studies or studies of patients with brain lesions could also explore the extent to which anticipation in lower-level regions relies on feedback from higher-level regions (Auksztulewicz and Friston, 2016; Kiebel et al., 2008).

The increased use of naturalistic, dynamic stimuli in neuroscience, and the development of methods to analyze the resulting data, has opened many avenues for research exploring flexible, future-oriented behavior. Our results and analysis approach provide a new framework for studying how anticipatory signals are distributed throughout the cortex, modulated by prior memory, and adaptive for improving comprehension and behavior.

We used data collected by Aly et al., 2018. Thirty individuals (12 men, age: M = 23.0 years, SD = 4.2; education: M = 15.3 years, SD = 3.2; all right-handed) watched movie clips from The Grand Budapest Hotel while undergoing fMRI. None of the participants reported previously seeing this movie. We analyzed data from the Intact condition, during which participants watched a continuous 90 s clip from the movie in its original temporal order. This clip was watched six times, interspersed with other video clips that are not considered here. This Intact clip depicts an interview scene between the protagonist and his future employer inside of the Grand Budapest Hotel. Stimuli and data are available on OpenNeuro: https://openneuro.org/datasets/ds001545/versions/1.1.1.

Data were acquired on a 3T Siemens Prisma scanner with a 64-channel head/neck coil using a multiband echo planar imaging (EPI) sequence (repetition time = 1.5 s; echo time = 39 ms; flip angle = 50°; acceleration factor = 4; shift = 3; voxel size = 2.0 mm iso). T1-weighted structural images (whole-brain high-resolution; 1.0 mm iso) were acquired with an MPRAGE sequence. Field maps (40 oblique axial slices; 3 mm iso) were collected to aid registration. The fMRI scan took place over three experimental runs, each of which contained two presentations of the Intact movie clip (as well as other movie clips not considered here).

The first three EPI volumes of each run were discarded to allow for T1 equilibration. Data preprocessing was carried out in FSL, and included brain extraction, motion correction, high-pass filtering (max period = 140 s), spatial smoothing (3 mm FWHM Gaussian kernel), and registration to standard Montreal Neurological Institute (MNI) space. After preprocessing, the functional images for each run were divided into volumes that corresponded to each of the video clips presented within that run, and only the two Intact clips within each run are considered further. Finally, each voxel’s timecourse was z-scored to have zero mean and unit variance.

Fourteen individuals (nine men) were asked to mark event boundaries corresponding to the same 90 s Intact clip from The Grand Budapest Hotel as shown to the fMRI participants. Each participant was asked to pause the clip at the end of a meaningful segment and to record the time and a brief title corresponding to the segment (Figure 3). Specifically, they were given the following instructions: The movie clip can be divided into meaningful segments. Record the times denoting when you feel like a meaningful segment has ended. Pause the clip at the end of the segment, write down the time in the spreadsheet, and provide a short, descriptive title. Try to record segments with as few viewings of the movie clip as possible; afterward, record the number of times you viewed the clip. Although participants were allowed to watch the clip multiple times, they were instructed to minimize and report the number of viewings needed to complete the task. No participant reported watching the clip more than three times.

Group-averaged fMRI data were fit with the event segmentation model described by Baldassano et al., 2017. This HMM assumes that (1) events are a sequence of discrete states, (2) each event is represented in the brain by a unique spatial activity pattern, and (3) all viewings of the movie evoke the same sequence of activity patterns in the same order (though possibly with different timings). We fit the HMM jointly to all six viewings. This fitting procedure involved simultaneously estimating a sequence of event activity patterns that were shared across viewings, and estimating the probability of belonging to each of these events for every timepoint in all six datasets. The model was fit with seven events; this number was chosen to match the approximate timescale of the semantic events in the narrative, matching the mean number of events annotated by human observers (mean = 6.5).

After fitting the HMM, we obtain an event by timepoint matrix for each viewing, giving the probability that each timepoint belongs to each event. Note that because this assignment of timepoints to events is probabilistic, it is possible for the HMM to detect that the pattern of voxel activity at a timepoint reflects a mixture of multiple event patterns. This allows us to track subtle changes in the timecourse of how the brain is transitioning between events. We took the expectation over events at each timepoint, yielding curves showing the average event label at each timepoint for each viewing. To compute shifts in time between the first viewing and the average of repeated viewings, the area under the curve (AUC) was computed for each viewing. We then computed the amount of anticipation as the average AUC for repeated viewing (viewings 2–6) minus the AUC for the first viewing. In a supplementary analysis, we compared the first viewing to the last viewing only. To convert to seconds, we divide by the vertical extent of the graph (number of events minus 1) and multiplied by the repetition time (1.5 s). We then performed a one-tailed statistical test (described below) to determine whether this difference was significantly positive, indicating earlier event transitions with repeated viewing. Not only does this approach provide a way of quantifying anticipation, it gives us a trajectory of the most likely event at any given timepoint, as well as the onset and duration of each event.

We obtained whole-brain results using a searchlight analysis. We generated spherical searchlights spaced evenly throughout the MNI volume (radius = 5 voxels; stride = 5 voxels). We retained only the searchlights with at least 20 voxels which were inside a standard MNI brain mask and for which at least 15 participants had valid data for all viewings. We then used the SRM (Chen et al., 2015) to functionally hyperalign all participants into shared 10-dimensional space (jointly fitting the alignment across all six viewings) and averaged their responses together. This produced a 10 feature by 60 timepoint data matrix for each of the six viewings, which was input to the HMM analysis described above. After running the analysis in all searchlights, the anticipation in each voxel was computed as the average anticipation of all searchlights that included that voxel.

To assess statistical significance, we utilized a permutation-based null hypothesis testing approach. We constructed null datasets by randomly shuffling each participant’s six responses to the six presentations of the movie clip. The full analysis pipeline (including hyperalignment) was run 100 times, once on the real (unpermuted) dataset and 99 times on null (permuted) datasets, with each analysis producing a map of anticipation across all voxels. A one-tailed p-value was obtained in each voxel by fitting a normal distribution to the null anticipation values, and then finding the fraction of this distribution that exceeded the real result in this voxel (i.e., showed more anticipation than in our unpermuted dataset). Voxels were determined significant (q<0.05) after applying the Benjamini-Hochberg FDR correction, as implemented in AFNI (Cox, 1996).

To determine if anticipation systematically varied across the cortex in the hypothesized posterior-to-anterior direction, we calculated the Spearman’s correlation between the Y-coordinate of each significant (q<0.05) voxel (indexing the position of that voxel along the anterior/posterior axis) and the mean amount of anticipation in that voxel. To obtain a p-value, the observed correlation was compared to a null distribution in which the Spearman’s correlation was computed with the null anticipation values from the permutation analysis described above, in which the order of the viewings was randomly scrambled for each participant. For comparison, the correlation was also computed for the X (left-right) and Z (inferior-superior) axes. This analysis was repeated on unthresholded anticipation maps, to examine if this hierarchy remained even when including regions whose anticipation amounts did not reach statistical significance.

To relate the timescales of anticipation to the intrinsic timescales of brain regions during the first viewing, we fit the HMM on the first viewing alone, varying the number of events from 2 to 10. The HMM was trained on the average response from half of the participants (fitting the sequence of activity patterns for the events and the event variance) and the log-likelihood of the model was then measured on the average response in the other half of the participants. The training and testing sets were then swapped, and the log-likelihoods from both directions were averaged together. Hyperalignment was not used during this fitting process, to ensure that the training and testing sets remained independent. The number of events that yielded the largest log-likelihood was identified as the optimal number of events for that searchlight. The optimal number of events was then compared to the anticipation timescale in that region (from the main analysis), using Spearman’s correlation.

For comparison, we also ran a searchlight looking for anticipatory effects using a non-HMM cross-correlation approach. Within each searchlight, we obtained an average timecourse across all voxels and correlated the response to the first viewing with the average response to repeated viewings at differing lags. Using the same quadratic-fit approach for identifying the optimal lag described below, we tested whether the repeated-viewing timecourse was significantly ahead of the initial-viewing timecourse (relative to a null distribution in which the viewing order was shuffled within each subject). The p-values obtained were then corrected for FDR.

We compared the event boundaries identified by the HMM within each searchlight to the event boundaries annotated by human observers. To obtain an event boundary timecourse from the annotations, we convolved the number of annotations (across all raters) at each second with the HRF (Figure 4). Separately, we generated a continuous measure of HMM ‘boundary-ness’ at each timepoint by taking the derivative of the expected value of the event assignment for each timepoint, as illustrated in Figure 1d. Moments with high boundary strength indicate moments in which the brain pattern was rapidly switching between event patterns. We cross-correlated the HMM boundary strength timecourse for each viewing with the annotated event boundary timecourse, shifting the annotated timecourse forward and backward to determine the optimal temporal offset (with the highest correlation). We measured the timing of the peak correlation by identifying the local maximum in correlation closest to 0 lag, then fitting a quadratic function to the maximum correlation lag and its two neighboring lags and recording the location of the peak of this quadratic fit. This produced a continuous estimate of the optimal lag for each viewing. We measured the amount of shift between the optimal lag for the first viewing and the average of the optimal lags for repeated viewings, and obtained a p-value by comparing to the null distribution over maps with permuted viewing orders (as in the main analysis), then performed an FDR correction.

We identified three gray matter clusters significant at q<0.05. To statistically assess whether the optimal lags differed from 0 in the three searchlights maximally overlapping these three clusters, we repeated the cross-correlation analysis in 100 bootstrap samples, in which we resampled from the raters who generated the annotated event boundaries. We obtained 95% bootstrap confidence intervals for maximally correlated lag on the first viewing and for the average of the maximally correlated lags on repeated viewings.

Data preprocessing scripts and python code to reproduce all the results in this paper are available at https://github.com/dpmlab/Anticipation-of-temporally-structured-events (copy archived at swh:1:rev:8fbd488c04d47148f9a53048de5d05a90e1c1663). Results in MNI space can be viewed at https://identifiers.org/neurovault.collection:9584.

No external funding was received for this work.

We thank the Aly and Baldassano labs for their feedback and support during this project, Janice Chen for helpful conversations about prediction hierarchies, and our three reviewers for proposing many useful improvements to the analyses.

© 2021, Lee et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.