Multiple scales of temporal context.

a) Illustration of three scales of temporal context in a binary time series. At the micro scale, events can either stay the same over time (repeat; R) or change (alternate; A). At the meso scale, short sequences of events (e.g., 5) form patterns that vary in their regularity (e.g., four repeats of the same event seems more regular than a mixture of repeats and alternations between events). At the macro scale, general trends regarding the relative frequency of different events are formed over longer time periods. b) To test the influence of temporal context on visual perception across different scales, participants were instructed to indicate the location of serially presented targets, which were randomly positioned on an imaginary circle centered on fixation. Participants performed a speeded binary judgement (e.g., left/right of fixation) on each trial and additionally reproduced the location of the target on 10% of trials. Trials were categorized as either (c, top) repeat (R) or (c, bottom) alternate (A) based on the location of the target relative to the previous target, according to three spatial reference planes: task-related (light cyan), task-unrelated (dark cyan), and stimulus-related (orange). d) In Experiment 1, the location of targets was uniformly sampled such that repeat and alternate trials were equally likely. In Experiment 2, the probability of repeat and alternate trials was biased across the task-related and unrelated planes.

The influence of micro-scale temporal context on visual processing.

a) Micro temporal context refers to the influence of the last event on the current event. We assessed its influence by comparing task performance for repeat and alternate presentations along task-related (light cyan), task-unrelated (dark cyan), and stimulus-related (orange) reference planes. b-d) The difference (repeat – alternate) in (b) response time, (c) accuracy, and (d) precision for the three references planes. Asterisks indicate significant differences (*p<.05, **p<.01, ***p<.001). e) Classification accuracy of stimuli presented on different sides of task-related and unrelated planes, from re-analysis of previously published EEG data17. f) Same as (e), but split into repeat and alternate stimuli, along (left) task-related and (right) unrelated planes. Inset in (e) shows the EEG sensors included in the analysis (blue dots). Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ±SEM. Horizontal bars indicate cluster corrected periods of significance (cyan and greyscale: above chance accuracy, pink: difference).

The influence of serial dependence on visual processing.

Serial dependence is associated with sensory stimuli being perceived as more similar to previous stimuli and is typically assessed by measuring the perception of stimuli as a function of their distance to previous stimuli (Δ location). a) Location reproduction bias, (b) binary task response time and (c) accuracy, and (d) reproduction precision as a function of distance from the previous stimulus. e) Decoding precision for stimulus location, from re-analysis of previously published EEG data 17. Inset shows the EEG sensors included in the analysis (blue dots), and black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). f) Decoding precision for location, as a function of time and Δ location. Bright colours indicate higher decoding precision; absolute precision values can be inferred from (e). Cluster corrected periods of decoding precision that is significantly above values from a shuffled comparison are bordered by semi-transparent white lines. Early (50-200 ms) and late (200-350 ms) periods are illustrated in green and pink, respectively. g-h) Average (g) location decoding precision and (h) bias during the early (green) and late (pink) period. i) Correlation between behavioural bias and location decoding bias as a function of time. Horizontal bars in (e & i) indicate cluster corrected periods of significance. Note, the temporal abscissa is aligned across (e, f, & i). Shaded regions indicate ±SEM.

The influence of meso-scale temporal context on visual processing.

a) Illustration of all 16 possible sequences of repeat and alternate events for a series of five binary events. Sequences are arranged symmetrically such that those for which the final event is thought to be most expected are at the top and bottom and those for which the final event is least expected are in the middle. To control for differences between repeat and alternate events, we combined symmetric pairs, resulting in a total of eight sequences. b) Response time, (c) accuracy, and (d) precision as a function of sequence, across task-related and unrelated reference planes. Note, lower numbers on the abscissa are associated with sequences in which the final target stimulus is more expected. Asterisks indicate significant main effects of sequence (***p<.001). Error bars indicate ±SEM; semi-transparent lines indicate linear fits to the data. e) The correlation between pupil size and sequence as a function of time. Note, we did not analyze pupillometry data for micro temporal context due the confounding effect of differential foveal luminance between repeat and alternate stimuli. f) Classification accuracy of stimuli presented on different sides of task-related planes as a function of time, for each of the eight sequences, from re-analysis of previously published EEG data 17. g) Same as (f), but for the task-unrelated plane. Inset in (f) shows the EEG sensors included in the analysis (blue dots). Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ±SEM. Horizontal bars indicate cluster corrected periods of significant relationships between (e) pupil size or (f, g) classification accuracy and sequence order.

The influence of macro temporal context on visual processing.

a) Repeat (R) and alternate (A) stimuli were presented with unequal probabilities (counterbalanced across participants) along task-related and unrelated reference planes. The difference in performance between relatively frequent (expected; E) and infrequent (unexpected; U) stimuli was calculated to assess the influence of macro temporal context on visual processing. b-d) The difference (expected – unexpected) in (b) response time, (c) accuracy, and (d) reproduction precision between expected and unexpected stimuli, along task-related (TR) and unrelated (TU) planes. Note that we observed accuracy differences associated with macro temporal context across the task-unrelated plane. In the micro and meso analyses, the expected outcome along the task-unrelated plane was never in conflict with that in the task-related plane. For example, if a stimulus is expected on the same (right) side of the display (repeat; task-related plane), then regardless of whether the stimulus appeared above or below fixation, there was a quadrant of the display that would satisfy both the task-related and unrelated expectations (bottom- or top-right). This is because the expectation was either for repeat stimuli (micro) or balanced between repeat and alternate stimuli (meso). By contrast, in the macro condition, where stimuli could be expected to alternate, this produced trials on which the task-related and unrelated expectations conflicted, such that there was no location that could satisfy both. Accuracy along the task-unrelated plane was reduced in these instances where there was conflict. Asterisks indicate significant differences (**p<.01, ***p<.001). e) The difference (expected – unexpected) in pupil size as a function of time from stimulus presentation, along the task-related (TR) plane. Shaded region indicates ±SEM and the horizontal bar indicates a cluster corrected period of significant difference.

Decoding stimulus location and Δ location from EEG recordings.

a) Topographic representation of task-related and unrelated stimulus location information produced using linear discriminant analysis to classify stimulus location, separately for each sensor, with time as the multivariate dimension. Blue dots indicate the sensors that were subsequently used in all other EEG analyses. b) Channel responses for location and Δ location, produced by computing the dot product between the inverted model channel responses and the forward model, as a function of time. c) The average channel response for location and Δ location between 0-500 ms following stimulus onset.

Removal of general micro temporal dependencies in EEG responses.

We found that there were differences in decoding accuracy for repeat and alternate stimuli in the EEG data, even when stimulus labels were shuffled. This is likely due to temporal autocorrelations within the EEG data that are unrelated to the decoded stimulus dimension. This signal causes the decoder to classify temporally proximal stimuli as the same class, leading to a bias towards repeat classification. For example, in general, the EEG signal during trial one will be more similar to that during trial two than during trial ten, because of low frequency trends in the recordings. If the decoder has been trained to classify the signal associated with trial one as a leftward stimulus, then it will be more likely to classify trial two as a leftward stimulus too. These autocorrelations are unrelated to stimulus features; thus, to isolate the influence of stimulus-specific temporal context, we subtracted the accuracy produced by shuffling the stimulus labels from the unshuffled accuracy (as presented in Figure 2e, f). a) Shows the uncorrected decoding accuracy along task related and unrelated planes. Note that these results are the same as the corrected version shown in Figure 2e, because the confound is only apparent when accuracy is grouped according to temporal context. b) Same as (a), but split into repeat and alternate stimuli, along (left) task-related and (right) unrelated planes. Decoding accuracy when labels are shuffled is also shown. Inset in (a) shows the EEG sensors included in the analysis (blue dots). Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ±SEM.

Decoding accuracy, but not precision, is confounded by general serial dependencies.

As described in Methods - Neural Decoding, we used inverted encoding modelling of EEG recordings to estimate the decoding accuracy and precision of stimulus location and Δ location. As shown in (a, b), there was a high correspondence between the relative magnitude of accuracy and precision, for both location and Δ location, over time. We also found that these parameters were very similar when further examined as a function of the distance of between current and previous stimulus location (c, d, g, h). To assess the influence of the general temporal dependencies found in the previous linear classification analysis between repeat and alternate stimuli, we calculated accuracy and precision on after shuffling the labels of the dataset. For decoding accuracy, we found a clear bias towards similar stimuli, i.e., those with small inter-trial offsets, for stimulus location (e), and less clear evidence for Δ location (i). Indeed, correlation analysis between the original and shuffled data decoding accuracy confirmed a significant positive relationship between the two for period before and after stimulus presentation (k). By contrast, for decoding precision, we found no evidence of these general temporal dependencies for either location or Δ location (f, j, l). Based on these findings, we assessed stimulus-specific serial dependence in the EEG recordings using decoding precision, which is uncontaminated by general serial dependence. However, this general dependence may explain the findings of a previous study. In particular, 19 found a repulsive pattern of biases associated with serial dependency in fMRI BOLD responses. The authors suggested that this may be due to adaptation, and provided a model in which this bias is reversed in higher cortical areas to produce the attractive bias associated with serial dependence. However, we find the same repulsive pattern of biases in the shuffled data (m, right), and in the pre-stimulus and late post-stimulus periods of the original data (m, top-left and bottom-left). By contrast, we found an attractive bias in the early post-stimulus period of the original data, consistent with serial dependence (m, middle-left). Thus, a more parsimonious explanation for the repulsive bias found by 19 is that they were observing the bias introduced by general temporal dependencies. Pre-stimulus (−120-0 ms), early (50-200 ms), and late (200-350 ms) periods are illustrated in orange, green, and pink, respectively. Note, the temporal abscissa is aligned across (a, c, e, g, I, k & b, d, f, h, j, l). Shaded regions indicate ±SEM. In our inverted encoding analysis, the precision of the early neural representation of stimulus location matched what we would expect from the previous binary analyses of micro temporal context, that is, highest precision when there was a large Δ location (i.e., angular distance between the target stimulus and the previous stimulus). However, behavioural precision showed an additional peak around small Δ location. This may be explained by the integration of location and Δ location representations, as Δ location, which was also peaked around small and large offsets, as shown in (g).

Meso-scale temporal context effects in Experiment 2.

a) Response time, (b) accuracy, and (c) precision as a function of sequence, across task-related and unrelated reference planes. Note, lower numbers on the abscissa are associated with more expected sequences. d) Same as (c), but pooled across the four most expected (1-4) and unexpected sequences (1-8). Asterisks indicate significant correlations with sequence order (*p<.05, **p<.01, ***p<.001), hats indicate marginal significance (^p<.06). Error bars indicate ±SEM.

Removal of general meso temporal dependencies in EEG responses.

To remove any potential general sequential dependences from the meso-scale classification analysis of EEG recordings, we performed the same subtraction method as used in the micro-scale analysis. (a) Classification accuracy of stimuli presented on different sides of task-related planes as a function of time, for each of the eight sequences, from re-analysis of previously published EEG data 17. b) Same as (a), but for the task-unrelated plane. (c) Same as (a), but with shuffled labels. Figure 4f, g shows (a, b) after removing shuffled accuracy (c). Inset in (a) shows the EEG sensors included in the analysis (blue dots). Black rectangles indicate the timing of stimulus presentations (solid: target stimulus, dashed: previous and subsequent stimuli). Shaded regions indicate ±SEM.

Macro-scale temporal context effects in Experiment 3.

a-c) The difference (expected – unexpected) in (a) response time, (b) accuracy, and (c) reproduction precision between expected and unexpected stimuli, along task-related (TR) and unrelated (TU) planes. Asterisks indicate significant differences (**p<.01, ***p<.001). d) The difference (expected – unexpected) in pupil size as a function of time from stimulus presentation, along the task-related (TR) plane. Shaded region indicates ±SEM and the horizontal bar indicates a cluster corrected period of significant difference.