Anticipation of temporally structured events in the brain
- Columbia University, Department of Psychology, United States
- Dartmouth College, Department of Psychological and Brain Sciences, United States
Figures
Figure 1
Computing varying timescales of anticipatory signals by examining temporal shifts in events across multiple viewings of a movie.
(a) Given the voxel by time pattern of responses evoked by the movie clip on each viewing (darker colors indicate higher levels of activity), our goal is to model all viewings as a series of transitions through a shared sequence of event patterns. (b–c) By fitting a hidden Markov model (HMM) jointly to all viewings, we can identify this shared sequence of event patterns, as well as a probabilistic estimate of event transitions. Regions with anticipatory representations are those in which event transitions occur earlier in time for repeated viewings of a stimulus compared to the initial viewing, indicated by an upward shift on the plot of the expected value of the event at each timepoint. (d) Taking the temporal derivative of the event timecourse plot in (c) produces a measure of the strength of event shifts at each moment in time, allowing for comparison with event boundary annotations from human observers.
Figure 2 with 4 supplements
Timescales of anticipation vary across the cortical hierarchy.
(a) Multiple regions exhibited shifts in event timing between initial and repeated viewings, with event transitions shifting earlier in time with subsequent viewings. Across the brain, anticipation timescales varied from a few seconds to 15 s, with the longest timescale anticipatory signals in prefrontal cortex and the temporal pole. Anticipation followed a posterior-to-anterior hierarchy, with progressively anterior areas generating anticipatory signals that reach further into the future (Spearman’s rho = 0.58, p=0.0030). Statistical thresholding was conducted via a permutation test, with correction for false discovery rate (FDR), q<0.05. (b) Event by time plots for three sample regions from (a), selected post hoc for illustration. Because the HMM produces a probability distribution across states at each timepoint, which can reflect a combination of current and upcoming event representations, we plot the expected value of the event assignments at each timepoint. The upward shift from the first viewing to subsequent viewings indexes the amount of anticipation.
Figure 2—figure supplement 1
Unthresholded statistical map of anticipation timescales.
This figure shows the same analysis as in Figure 2a, but without statistical thresholding. The posterior-to-anterior hierarchy of anticipation persists in this unthresholded map (Spearman’s rho = 0.42, p=0.0028).
Figure 2—figure supplement 2
Timescales of anticipation when the first viewing is compared to the last viewing.
This figure shows the same analysis as that in Figure 2, except that the first viewing is compared to the last (6th) viewing only.
Figure 2—figure supplement 3
Timescales of anticipation as a function of the optimal number of events.
(a) This map depicts the optimal number of events in each brain region that shows statistically significant anticipation (i.e., masked by regions shown in Figure 2a). The optimal number of events was determined by maximizing the log-likelihood of the HMM on the first movie viewing. Purple indicates the fastest timescale (optimally fit by eight events, average event duration = 11 s) and yellow indicates the slowest timescale (optimally fit by two events, average event duration = 45 s). (b) Anticipation (Figure 2a) is shown binned by the optimal event timescale. Anticipation is further reaching in regions with longer timescales (Spearman’s rho = 0.319, p=0.00031).
Figure 2—figure supplement 4
Cross-correlation analysis of anticipation.
Cross-correlations were conducted between the activity timecourse of a given region for the first movie viewing and the average of the subsequent movie viewings. The regions depicted showed statistically significant shifts in activity, such that the peak correlation occurred when the subsequent viewing activity was shifted earlier in time. Statistical thresholding was conducted via a permutation test, with correction for false discovery rate (FDR), q<0.05. Colors depict the temporal shift of the peak in the cross-correlation.
Figure 3
An example of event annotations from The Grand Budapest Hotel.
Dotted lines demarcate events and phrases between the lines are brief titles given by one participant to describe each event. (Frames in this figure have been blurred to comply with copyright restrictions, but all participants were presented with the original unblurred version.)
Figure 4
Construction of behavioral boundary timecourse from human annotations.
The number of boundary annotations at each second of the movie clip (in gray) was convolved with a hemodynamic response function (HRF) to produce a continuous measure of boundary strength (black line).
Figure 5
Correlations between the brain’s event transitions and human-annotated event boundaries.
Cross-correlation plots show the correlation between the brain’s hidden Markov model (HMM) event boundaries and annotated event boundaries as the timecourses are shifted with respect to one another. The correlation at 0 lag indicates the similarity between the brain’s event boundaries and annotated event boundaries when the timecourses are aligned. Negative lags show the correlations when the human-annotated event timecourse is shifted earlier in time, and positive lags show the correlation when the human-annotated event timecourse is shifted later in time. Peaks in the cross-correlation plot indicate the lag that produced the highest correlation between the brain’s event boundaries and annotated event boundaries. On initial viewing, the HMM event boundaries for the fusiform gyrus and superior temporal sulcus lagged significantly behind the annotated event boundaries, while the timing of the peak correlation for the middle temporal gyrus did not significantly differ from 0 lag. On subsequent viewings, the HMM event boundaries in all three regions shifted to be significantly earlier than the initial viewing, with the timing of the peak correlation significantly preceding 0 lag.
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Anticipation of temporally structured events in the brain
eLife 10:e64972.
https://doi.org/10.7554/eLife.64972
