The influence of temporal context on vision over multiple time scales
- School of Psychology, The University of Sydney, Australia
- Queensland Brain Institute, The University of Queensland, Australia
Figures
Figure 1
Multiple scales of temporal context.
(a) Illustration of three scales of temporal context in a binary time series. At the micro scale, events can either stay the same over time (repeat; R) or change (alternate; A). At the meso scale, short sequences of events (e.g. 5) form patterns that vary in their regularity (e.g. four repeats of the same event seems more regular than a mixture of repeats and alternations between events). At the macro scale, general trends regarding the relative frequency of different events are formed over longer time periods. (b) To test the influence of temporal context on visual perception across different scales, participants were instructed to indicate the location of serially presented targets, which were randomly positioned on an imaginary circle centered on fixation. Participants performed a speeded binary judgement (e.g. left/right of fixation) on each trial and additionally reproduced the location of the target on 10% of trials. Trials were categorized as either (c, top) repeat (R) or (c, bottom) alternate (A) based on the location of the target relative to the previous target, according to three spatial reference planes: task-related (light cyan), task-unrelated (dark cyan), and stimulus-related (orange). (d) In Experiment 1, the location of targets was uniformly sampled such that repeat and alternate trials were equally likely. In Experiment 2, the probability of repeat and alternate trials was biased across the task-related and unrelated planes.
Figure 2 with 2 supplements
The influence of micro-scale temporal context on visual processing.
(a) Micro temporal context refers to the influence of the last event on the current event. We assessed its influence by comparing task performance for repeat and alternate presentations along task-related (light cyan), task-unrelated (dark cyan), and stimulus-related (orange) reference planes. (b–d) The difference (repeat – alternate) in (b) response time, (c) task accuracy, and (d) precision for the three reference planes. Asterisks indicate significant differences (*p<0.05, **p<0.01, ***p<0.001). (e) Classification accuracy of stimuli presented on different sides of task-related and unrelated planes, from re-analysis of previously published EEG data (Rideaux, 2024). (f) Same as (e), but split into repeat and alternate stimuli, along (left) task-related and (right) unrelated planes. Inset in (e) shows the EEG sensors included in the analysis (blue dots). Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ± SEM. Horizontal bars indicate cluster-corrected periods of significance (cyan and greyscale: above chance classification accuracy, pink: difference).
Figure 2—figure supplement 1
Decoding stimulus location and Δ location from EEG recordings.
(a) Topographic representation of task-related and unrelated stimulus location information produced using linear discriminant analysis to classify stimulus location, separately for each sensor, with time as the multivariate dimension. Blue dots indicate the sensors that were subsequently used in all other EEG analyses. (b) Channel responses for location and Δ location, produced by computing the dot product between the inverted model channel responses and the forward model, as a function of time. (c) The average channel response for location and Δ location between 0-500 ms following stimulus onset.
Figure 2—figure supplement 2
Removal of general micro temporal dependencies in EEG responses.
We found that there were differences in classification accuracy for repeat and alternate stimuli in the EEG data, even when stimulus labels were shuffled. This is likely due to temporal autocorrelation within the EEG data due to low frequency signal changes that are unrelated to the decoded stimulus dimension. This signal trains the decoder to classify temporally proximal stimuli as the same class, leading to a bias towards repeat classification. For example, in general, the EEG signal during trial one is likely to be more similar to that during trial two than during trial ten, because of low frequency trends in the recordings. If the decoder has been trained to classify the signal associated with trial one as a leftward stimulus, then it will be more likely to classify trial two as a leftward stimulus too. These autocorrelations are unrelated to stimulus features; thus, to isolate the influence of stimulus-specific temporal context, we subtracted the classification accuracy produced by shuffling the stimulus labels from the unshuffled accuracy (as presented in Figure 2e, f). We confirmed that using a stricter high-pass filter (0.7 Hz) removes this artifact, as indicated by the equal decoding accuracy between the two shuffled conditions. However, the stricter high-pass filter temporally smears the stimulus-related signal, which introduces other (stimulus-related) artifacts, e.g., above-chance decoding accuracy prior to stimulus presentation, that are larger and more complex, i.e., changing over time. Thus, we opted to use the original high pass filter (0.1 Hz) and apply a baseline correction. (a) The uncorrected classification accuracy along task related and unrelated planes. Note that these results are the same as the corrected version shown in Figure 2e, because the confound is only apparent when accuracy is grouped according to temporal context. (b) Same as (a), but split into repeat and alternate stimuli, along (left) task-related and (right) unrelated planes. Classification accuracy when labels are shuffled is also shown. Inset in (a) shows the EEG sensors included in the analysis (blue dots). (c, d) Same as (a, b), but on data filtered using a 0.7 Hz high-pass filter. Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ± SEM.
Figure 3 with 1 supplement
The influence of serial dependence on visual processing.
Serial dependence is associated with sensory stimuli being reported as more similar to previous stimuli and is typically assessed by measuring the perception of stimuli as a function of their distance to previous stimuli (Δ location). (a) Location reproduction bias, (b) binary task response time and (c) accuracy, and (d) reproduction precision as a function of distance from the previous stimulus. (e) Decoding accuracy for stimulus location, from re-analysis of previously published EEG data (Rideaux, 2024). The inset shows the EEG sensors included in the analysis (blue dots), and black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). (f) Decoding accuracy for location, as a function of time and Δ location. Bright colors indicate higher decoding accuracy; absolute accuracy values can be inferred from (e). (g–i) Average location decoding (g) accuracy, (h) precision, and (h) bias from 50 to 500ms following stimulus onset. Horizontal bar in (e) indicates cluster-corrected periods of significance; note, all time points were significantly above chance due to temporal smear introduced by high-pass filtering (see Figure 3—figure supplement 1 for full details). Note, the temporal abscissa is aligned across (e & f). Shaded regions indicate ± SEM.
Figure 3—figure supplement 1
Removal of general temporal dependencies in EEG responses for inverted encoding analyses.
As described in ‘Methods - Neural Decoding’, we used inverted encoding modelling of EEG recordings to estimate the decoding accuracy, precision, and bias of stimulus location. Just as in the linear discriminant classification analysis, we also found the influence of general temporal dependencies in the results produced by the inverted encoding analysis. In particular, there was increased decoding accuracy for targets with low Δ location. This was weakly evident in the period prior to stimulus presentation, but clearly visible when the labels were shuffled. These results are mirror those from the classification analysis, albeit in a more continuous space. However, whereas in the classification analysis it was straightforward to perform a baseline correction to remove the influence of general temporal dependency, the more complex nature of the accuracy, precision, and bias parameters over the range of time and Δ location makes this approach less appropriate. For example, the bias in the shuffled condition ranged from -180° to 180°, which when subtracted from the bias in the unshuffled condition would produce an equally spurious outcome, i.e., the equal opposite of this extreme bias. Instead for the inverted encoding analysis, we used the data high-pass filtered at 0.7 Hz. As with the classification analysis, this significantly reduced the influence of general temporal dependencies, as indicated by the results of the shuffled data analysis, but it also temporally smeared the stimulus-related signal, resulting in above chance decoding accuracy prior to stimulus onset. However, we were primarily interested in the pattern of accuracy, precision, and bias as a function of Δ location, and less concerned with the precise temporal dynamics of these changes. Thus, this was the more suitable approach to removing the general temporal dependencies in the inverted encoding analysis and the one that is presented in Figure 3. (a) Decoding accuracy as a function of time for the EEG data filtered using a 0.1 Hz high-pass filter. Inset shows the EEG sensors included in the analysis (blue dots), and black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). (b, c) The same as (a), but as a function of time and Δ location for (b) the original data and (c) data with shuffled labels. (d-f) Same as (a-c), but for data filtered using a 0.7 Hz high-pass filter. Shaded regions in (a, d) indicate ± SEM. Horizontal bars in (a, d) indicate cluster corrected periods of significance; note, all time points in (d) were significantly above chance. Note, the temporal abscissa is vertically aligned across plots (a-c and d-f).
Figure 4 with 2 supplements
The influence of meso-scale temporal context on visual processing.
(a) Illustration of all 16 possible sequences of repeat and alternate events for a series of five binary events. Sequences are arranged symmetrically such that those for which the final event is thought to be most expected are at the top and bottom and those for which the final event is least expected are in the middle. To control for differences between repeat and alternate events, we combined symmetric pairs, resulting in a total of eight sequences. (b) Response time, (c) task accuracy, and (d) precision as a function of sequence, across task-related and unrelated reference planes. Note, lower numbers on the abscissa are associated with sequences in which the final target stimulus is more expected. Asterisks indicate significant main effects of sequence (***p<0.001). Error bars indicate ± SEM; semi-transparent lines indicate linear fits to the data. (e) The correlation between pupil size and sequence as a function of time. Note, we did not analyze pupillometry data for micro temporal context due to the confounding effect of differential foveal luminance between repeat and alternate stimuli. (f) Classification accuracy of stimuli presented on different sides of task-related planes as a function of time, for each of the eight sequences, from re-analysis of previously published EEG data (Rideaux, 2024). (g) Same as (f), but for the task-unrelated plane. The inset in (f) shows the EEG sensors included in the analysis (blue dots). Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ± SEM. Horizontal bars indicate cluster-corrected periods of significant relationships between (e) pupil size or (f, g) classification accuracy and sequence order.
Figure 4—figure supplement 1
Meso-scale temporal context effects in Experiment 2.
(a) Response time, (b) task accuracy, and (c) precision as a function of sequence, across task-related and unrelated reference planes. Note, lower numbers on the abscissa are associated with more expected sequences. (d) Same as (c), but pooled across the four most expected (1-4) and unexpected sequences (1-8). Asterisks indicate significant correlations with sequence order (*p<.05, **p<.01, ***p<.001), hats indicate marginal significance (^p<.06). Error bars indicate ± SEM.
Figure 4—figure supplement 2
Removal of general meso temporal dependencies in EEG responses.
To remove any potential general sequential dependences from the meso-scale classification analysis of EEG recordings, we performed the same subtraction method as used in the micro-scale analysis. (a) Classification accuracy of stimuli presented on different sides of task-related planes as a function of time, for each of the eight sequences, from re-analysis of previously published EEG data (Rideaux, 2024). (b) Same as (a), but for the task-unrelated plane. (c) Same as (a), but with shuffled labels. Figure 4f, g shows (a, b) after removing shuffled accuracy (c). Inset in (a) shows the EEG sensors included in the analysis (blue dots). Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ± SEM.
Figure 5 with 1 supplement
The influence of macro temporal context on visual processing.
(a) Repeat (R) and alternate (A) stimuli were presented with unequal probabilities (counterbalanced across participants) along task-related and unrelated reference planes. The difference in performance between relatively frequent (expected; E) and infrequent (unexpected; U) stimuli was calculated to assess the influence of macro temporal context on visual processing. (b–d) The difference (expected – unexpected) in (b) response time, (c) task accuracy, and (d) reproduction precision between expected and unexpected stimuli, along task-related (TR) and unrelated (TU) planes. Note that we observed task accuracy differences associated with macro temporal context across the task-unrelated plane. In the micro and meso analyses, the expected outcome along the task-unrelated plane was never in conflict with that in the task-related plane. For example, if a stimulus is expected on the same (right) side of the display (repeat; task-related plane), then regardless of whether the stimulus appeared above or below fixation, there was a quadrant of the display that would satisfy both the task-related and unrelated expectations (bottom- or top-right). This is because the expectation was either for repeat stimuli (micro) or balanced between repeat and alternate stimuli (meso). By contrast, in the macro condition, where stimuli could be expected to alternate, this produced trials on which the task-related and unrelated expectations conflicted, such that there was no location that could satisfy both. Accuracy along the task-unrelated plane was reduced in these instances where there was conflict. Asterisks indicate significant differences (**p<0.01, ***p<0.001). (e) The difference (expected – unexpected) in pupil size as a function of time from stimulus presentation, along the task-related (TR) plane. Shaded region indicates ± SEM and the horizontal bar indicates a cluster-corrected period of significant difference.
Figure 5—figure supplement 1
Macro-scale temporal context effects in Experiment 3.
The difference (expected – unexpected) in (a) response time, (b) task accuracy, and (c) reproduction precision between expected and unexpected stimuli, along task-related (TR) and unrelated (TU) planes. Asterisks indicate significant differences (**p<.01, ***p<.001). (d) The difference (expected – unexpected) in pupil size as a function of time from stimulus presentation, along the task-related (TR) plane. Shaded region indicates ± SEM and the horizontal bar indicates a cluster corrected period of significant difference.
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
The influence of temporal context on vision over multiple time scales
eLife 14:RP106614.
https://doi.org/10.7554/eLife.106614.3
