The aim of the first experiment was to characterize the dependence of perceptual decisions on the short- and long-term stimulus history. To this end, twenty-three human observes completed a series of trials, in which they were presented with oriented Gabor stimuli, randomly sampled on each trial from the full range of possible orientations (0, 180°], and subsequently had to report the perceived orientation of each stimulus by adjusting a response bar (Figure 1). We then analyzed the dependence of the current adjustment response on each stimulus experienced in the 40 preceding trials, corresponding to a time window reaching ~3.5 min into the past. We found that adjustment responses were systematically attracted towards the stimulus presented on the previous trial (p<0.0001; Figure 2A). This attraction bias was maximal when orientations of successive stimuli differed by 18° (max. attraction = 1.66°), similar to previous reports of attractive serial dependence (Fischer and Whitney, 2014; Fritsche et al., 2017). However, the attractive influence of previous stimuli was short-lived and was only clearly observable between adjacent trials, corresponding to an inter-stimulus interval of about 5 s (Figure 2C). Conversely, stimuli encountered further back in the past exerted a repulsive bias on the current adjustment response. The repulsive bias was maximal for stimuli presented four trials in the past (Figure 2B, p<0.0001, max. repulsion = −1.07°) and subsequently decreased towards zero, albeit remaining negative up to 22 trials back, corresponding to a time window of about two minutes. Similar to the short-term attractive bias, the repulsive bias was tuned to the orientation difference between the current and past stimulus, such that current adjustment responses were maximally repelled for orientation differences of ~19° (average tuning width of 4- to 22-back trials). The results of Experiment 1 thus indicate that perceptual estimates are subject to both attractive and repulsive biases, operating over different timescales. While perceptual estimates are attracted towards immediately preceding stimuli, they are repelled from temporally distant stimuli. This repulsive bias potentially reaches back minutes into the past. The orientation tuning of the long-term repulsive bias is reminiscent of sensory adaptation (i.e. the negative tilt-aftereffect), which typically induces maximal repulsion for orientation differences of about 20° (Gibson and Radner, 1937; Jin et al., 2005). To shed more light on the nature of the long-term repulsive bias, we next investigated whether it is modulated by the spatial overlap between the current and temporally distant stimulus. Such a spatial specificity is a hallmark of classical sensory adaptation effects (Boi et al., 2011; Knapen et al., 2010; Mathôt and Theeuwes, 2013), and thus would point towards low-level sensory encoding as the origin of the repulsive bias.

Figure 1

Download asset Open asset

Figure 2

Download asset Open asset

Figure 2—source data 1

Results of Experiment 1: Estimation responses are attracted towards short-term, but repelled from long-term stimulus history.

The source data file contains a csv file with the 1- to 40-back DoG model parameter estimates and a csv file with p-values for testing the model’s amplitude parameters against zero. It also contains csv files with moving averages of response errors conditioned on the 1- and 4-back trial, respectively (participants x orientation). Furthermore, it contains a mat file with all 1- to 40-back moving averages of each participant.

https://cdn.elifesciences.org/articles/55389/elife-55389-fig2-data1-v2.zip

Download elife-55389-fig2-data1-v2.zip

Experiment 2 was similar to Experiment 1, with the exception that stimulus position could vary between two locations (10 visual degrees apart) on a trial-by-trial basis. Consequently, any past stimulus could be presented at the same or a different spatial location as the current stimulus. This dataset, comprising 24 human observers, has been previously described (Fritsche et al., 2017). We previously showed that adjustment responses were systematically biased towards the immediately preceding stimulus, with a similar magnitude when it was presented at the same or a different location (see Experiment 1 in Fritsche et al., 2017 and Figure 3A and B; same location: max. attraction = 1.15°, p=0.005; different location: max. attraction = 1.17°, p=0.0006; same vs. different: p=0.97). Here, we sought to investigate the spatial specificity of the long-term repulsive bias. To this end, we focused on the past 10 stimuli preceding the current stimulus, for which repulsive biases in Experiment 1 were strongest. In line with Experiment 1, we found that adjustment responses were significantly repelled away from stimuli seen 4 to 9 trials ago (Figure 3A). Moreover, stimuli presented within this time window, reaching from 26 to 60 s in the past, exerted a spatially specific bias on the current adjustment response. That is, the current adjustment response was more strongly repelled by a previous stimulus presented at the same compared to a different spatial location as the current stimulus (Figure 3C; same location: max. repulsion = −0.81°, p<0.0001; different location: max. repulsion = −0.39°, p=0.0006; same vs. different: p=0.006). This spatial specificity of the repulsive bias, exerted by stimuli seen tens of seconds in the past, points towards a sensory origin of the long-term repulsive bias, akin to the negative tilt-aftereffect. Moreover, the small remaining repulsion bias for stimuli presented at a different spatial location is consistent with previous findings, which suggested that weak tilt-aftereffects may occur at spatial locations further away from the adaptor stimulus (e.g. see Knapen et al., 2010; Mathôt and Theeuwes, 2013).

Figure 3

Download asset Open asset

Figure 3—source data 1

Results of Experiment 2: Long-term repulsive biases are spatially specific.

The source data file contains a csv file with the 1- to 10-back DoG model parameter estimates and a csv file with p-values for testing the model’s amplitude parameters against zero. It also contains two csv files with model parameters fit to 1-back or 4- to 9-back conditioned response errors, split by location change. Furthermore, it contains a mat file with all 1- to 10-back moving averages of each participant (same and different locations).

https://cdn.elifesciences.org/articles/55389/elife-55389-fig3-data1-v2.zip

Download elife-55389-fig3-data1-v2.zip

We have previously shown that the short-term positive bias exerted by the previous stimulus grows during the working memory retention period of the current trial, indicating a role of working memory for the positive bias (Experiment 4 in Fritsche et al., 2017 and Figure 4B; see also Bliss et al., 2017; Papadimitriou et al., 2015). One may surmise that if the long-term repulsive bias reflects a low-level encoding bias, affecting the earliest stages of visual processing, then in contrast to the short-term positive bias, long-term repulsion should not be modulated by the duration of the current working memory delay. To test this hypothesis, we analyzed an additional, previously published dataset involving perceptual estimation responses to oriented Gabor stimuli, in which we introduced a variable delay period of 50 or 3500 ms between stimulus offset and the estimation response (Fritsche et al., 2017). Similar to Experiment 1 and 2, we found that estimation responses were biased towards the stimulus orientation of the previous trial (Figure 4A, max. attraction = 1.25°, p<0.0001), but repelled from stimuli presented further in the past (max. repulsion of −0.81° for 3-back stimulus, p<0.0001). This further corroborates the existence of short-term attraction and long-term repulsion biases in perceptual estimates. Interestingly, in comparison to Experiment 1 and 2, it appeared that the transition from attractive to repulsive effects occurred earlier in time, such that repulsive effects were already visible for the 2-back trial (Figure 4A, max. repulsion = −0.46°, p=0.034). Importantly, trials in Experiment 3 were longer compared to the previous experiments, due to the working-memory delay period (8.25 s vs. 5.37 and 6.55 s trial lengths). Thus, one potential explanation for the earlier onset of repulsion biases could be that the decay of attractive and repulsive biases may depend on the elapsed time (i.e. seconds) between previous and current stimulus, rather than the number of intervening stimuli or decisions. We will return to a quantitative comparison of the temporal decay across the different experiments later.

Figure 4

Download asset Open asset

Figure 4—source data 1

Results of Experiment 3: Long-term repulsive biases are not strongly modulated by working memory delay.

The source data file contains a csv file with the 1- to 10-back DoG model parameter estimates and a csv file with p-values for testing the model’s amplitude parameters against zero. It also contains two csv files with model parameters fit to 1-back or 2- to 6-back conditioned response errors, split by memory delay duration. Furthermore, it contains a mat file with all 1- to 10-back moving averages of each participant (short and long memory delay).

https://cdn.elifesciences.org/articles/55389/elife-55389-fig4-data1-v2.zip

Download elife-55389-fig4-data1-v2.zip

The attraction bias towards the 1-back stimulus was stronger when the current memory delay was long compared to short (Figure 4B; short delay: max. attraction = 1.08°, p=0.001; long delay: max. attraction = 1.64°, p<0.0001; short vs. long: p=0.047; Fritsche et al., 2017). In contrast, for stimuli that exhibited overall repulsive biases (2- to six trials back, 17 to 50 s in the past), there was no significant difference in the magnitude of repulsion across memory delays (Figure 4C; short delay: max. repulsion = −0.47°, p=0.0002; long delay: max. repulsion = −0.75°, p<0.0001; short vs. long: p=0.055), although there was a trend towards stronger repulsion for current long delays. Indeed, there was a significant interaction between n-back trial (1-back vs. 2- to 6-back) and working memory delay (F(1,23) = 5.21, p=0.032, η² = 0.19), indicating that working memory delay has a stronger impact on short-term attraction than on long-term repulsion biases.

In order to quantify evidence for the null hypothesis of no modulation of the repulsive bias by working memory delay, we turned to a model-free analysis of averaging response errors for trials that were preceded by clockwise or counter-clockwise stimulus orientations, respectively (see Materials and methods). This allowed us to quantify serial dependence biases at the single participant level, which could subsequently be used for a Bayes Factor analysis. While this analysis provided moderate evidence for a modulation of the 1-back attraction bias by working memory delay (BF₁₀ = 7.25), which is in line with the model-based analysis, there was anecdotal (but no decisive) evidence for a lack of modulation for the long-term repulsive bias (BF₁₀ = 0.77).

To further test whether long-term repulsive biases arise at an early encoding stage, we turned to a different experimental paradigm, which sought to measure perception more directly compared to the stimulus reproduction technique employed in the previous experiments. Stimulus reproductions necessarily rely on memory of the perceived stimulus feature and thus may reflect a combination of early perceptual and late post-perceptual working memory and decision biases (Bliss et al., 2017; Fritsche et al., 2017; Papadimitriou et al., 2015). We have previously shown that when the perceived orientation of a stimulus is measured more directly, by an immediate comparison to a reference stimulus that is visible at the same time (Figure 5A), it is repelled from the previous stimulus in a spatially specific manner, likely reflecting sensory adaptation (see Fritsche et al., 2017 and Figure 5C ‘current inducer’). Here we asked, whether the perceived orientation of the current stimulus, as measured in direct comparison to a reference, would continue to be repelled from stimuli experienced further in the past. This analysis not only allowed us to investigate whether the long-term repulsive bias reflects a genuine low-level perceptual bias that directly alters the appearance of a stimulus, as would be expected of sensory adaptation, but also enabled us to assess the temporal decay of the repulsion effect while short-term attraction biases are absent or strongly reduced (Fritsche et al., 2017; but see Cicchini et al., 2017). In Experiment 4, the same 24 observers that participated in Experiment 2 performed two consecutive tasks on each trial. First, they were simultaneously presented with two Gabor stimuli and were cued to reproduce the orientation of one of the stimuli by adjusting a response bar. We termed the cued stimulus ‘inducer’, since it induced a perceptual decision at one location. In order to measure subsequent biases in perception, another two Gabor stimuli were presented (one at the previously cued location and one at a location in the opposite hemifield, 20 visual degrees apart) and observers judged which of the two stimuli was tilted more clockwise (two-alternative forced choice [2AFC], Figure 5A and B). To recapitulate, we previously found in the same dataset that in this direct perceptual comparison the perceived orientations of the 2AFC stimuli were repelled from the immediately preceding inducer in a spatially specific manner (see Fritsche et al., 2017 and Figure 5C ‘current inducer’; same location: repulsion −0.71°±0.10 SEM, t(23) = −6.91, p=5e-7, d = −1.41; different location: repulsion −0.17°±0.11 SEM, t(23) = −1.54, p=0.14, d = −0.31; same vs. different: t(23) = −3.85, p=0.0008, d = −0.79). Importantly, here we found that this repulsive bias was not limited to the inducer presented on the current trial (Figure 5C). On average, inducers presented on the 10 previous trials, corresponding to a time window of 10 to 100 s prior to the current trial, exerted a significantly repulsive bias on the perceived orientation of the current 2AFC stimuli, when presented at an overlapping spatial location (same location: repulsion −0.24°±0.05 SEM, t(23) = −5.24, p=3e-5, d = - 1.07), but not when presented at a different location 10 visual degrees away (different location: repulsion −0.09°±0.05 SEM, t(23) = −1.66, p=0.11, d = 0.34; same vs. different location: t(23) = −2.38, p=0.026, d = −0.49). The temporal decay of this spatially specific repulsion bias was captured by an exponential decay model, which revealed a half-life of about five trials, corresponding to 50 s. The presence of long-term repulsive biases in perceptual comparisons indicates that brief stimuli experienced tens of seconds ago can directly bias the perceived orientation of a current stimulus in a spatially specific manner. This further supports the view that the long-term repulsive biases occur early during perceptual processing during the encoding of sensory information. Moreover, in the absence of the attractive bias measured in perceptual estimates (Experiment 1–3), the repulsive adaptation effect is strongest for stimuli which are temporally closest to the current stimulus and decays with increasing time between previous and current stimulus, with a long half-life. The current finding suggests that attractive and repulsive biases may coexist in perceptual estimations (Experiment 1–3), but differ in their relative strength and rate of decay. While both attractive and repulsive biases are strongest for the most recent stimulus, an initially stronger attractive bias typically masks a weaker repulsive bias, leading to an overall attractive influence of the immediate past. However, since the attractive bias decays at a much faster rate compared to the repulsive bias, the net bias that a stimulus exerts on the current perceptual estimate switches from attractive to repulsive as the intervening time increases.

Figure 5

Download asset Open asset

Figure 5—source data 1

Task and results of Experiment 4: The long-term stimulus history directly biases the perceived orientation of current stimuli.

The source data file contains two csv files with estimated biases exerted by the previous 10 inducer stimuli, when the inducer was presented at the same or different spatial location, respectively (participants x n-back inducer).

https://cdn.elifesciences.org/articles/55389/elife-55389-fig5-data1-v2.zip

Download elife-55389-fig5-data1-v2.zip

What are the underlying computational principles that may lead to these concurrent attractive and repulsive biases in perceptual decisions? To shed more light on this question, we turned to computational modeling to explore whether the coexisting attractive and repulsive biases can be explained by an observer that exploits the temporal continuity of visual environment in order to optimize the efficient encoding and/or decoding of visual information.

Previous studies proposed that attractive biases in perceptual estimates arise from a probabilistically optimal strategy of decoding sensory information into a perceptual decision (Cicchini et al., 2018; van Bergen and Jehee, 2019). Attractive serial dependencies towards the previous stimulus orientation are well captured by such an observer model that estimates the current stimulus orientation by integrating a noisy sensory measurement of the current stimulus with a prior prediction about the upcoming stimulus orientation in a probabilistically optimal, that is Bayesian, manner (Figure 6A, orange box). If predictions are well matched to the temporal regularities in the environment, combining predictions based on previous stimuli with current sensory measurements generally produces more accurate stimulus estimates than those based on current sensory measurements alone. We implemented such an ideal observer model to test whether it could account for the short-term attractive and long-term repulsive biases in the empirical data. In this observer model, the prediction about the upcoming visual stimulus was formed by combining an internal model of naturally occurring orientation changes (Figure 6A, green box, transition distribution) with knowledge about previous stimulus orientations (i.e. previous posterior distributions). In contrast to the ideal observer model by van Bergen and Jehee, 2019, we here assumed that the observer does not only rely on a prediction formed on the most recent trial, but utilizes a weighted mixture of previous predictions, where the relative contribution of previous predictions is determined by an exponential integration time constant (Figure 6A, blue box, temporal integration weights). Such a mixture prior accounts for the fact that statistical regularities in natural environments do not only exist between immediately consecutive orientation samples, but over extended timescales, with the most recent stimuli also being the most predictive of the current stimulus. In contrast to an observer that only relies on the prediction based on the previous trial, an observer utilizing such a mixture prior has been shown to adequately capture positive serial dependencies beyond the most recent trial (Kalm and Norris, 2018). We found that this observer model indeed captured attractive serial dependencies. However, it was not able to produce the long-term repulsive biases found in Experiments 1–3 (Figure 7 and Figure 7—figure supplement 2). Thus, an observer that exploits the temporal stability of the environment via Bayes-optimal decoding alone cannot explain the pattern of attractive and repulsive biases. Furthermore, the inability of the Bayesian ideal observer to capture both attraction and repulsion biases provides evidence that the long-term repulsion biases are not an artifact resulting from short-term attraction biases and random fluctuations in the input sequence (Maus et al., 2013).

Figure 6

Download asset Open asset

Figure 7 with 5 supplements see all

Download asset Open asset

Figure 7—source data 1

Empirical biases and ideal observer predictions.

The source data file contains csv files with the empirical bias amplitudes of 1- to 20-back stimuli, as well as the predicted bias amplitudes of the ideal observer with efficient encoding and Bayesian decoding and the observer with Bayesian decoding alone. Three separate files are provided for Experiment 1–3, respectively. Furthermore, for each experiment we provide two mat files with the model fitting results of the two ideal observer models.

https://cdn.elifesciences.org/articles/55389/elife-55389-fig7-data1-v2.zip

Download elife-55389-fig7-data1-v2.zip

As described in the Introduction, repulsive adaptation biases have been proposed to arise from efficient encoding of visual information (Barlow and Földiák, 1989; Clifford et al., 2000; Müller et al., 1999; Stocker and Simoncelli, 2006). For instance, the sensory system may allocate sensory coding resources such that it is optimally adapted to represent the current stimulus (Wei and Stocker, 2017; Wei and Stocker, 2015). For temporally stable visual input, in which current stimulus features are similar to previous ones, an optimally efficient allocation of encoding resources entails that more resources should be dedicated to stimulus features encountered in the recent history. As a consequence, the most likely stimulus, as predicted from the sensory history, is represented with highest fidelity. Here, we implemented an observer model that predicted the orientation of an upcoming stimulus based on previous sensory measurements and subsequently allocated sensory encoding resources such that the mutual information between the predicted stimulus and its sensory representation was maximized (Figure 6B; Wei and Stocker, 2015 and ‘Efficient encoding model with history-independent decoding’ in Materials and methods). For this observer, repulsive adaptation biases then result not from changes to the prior, but from changes in the sensory representation of the current stimulus; in other words in the likelihood function. Due to biased allocation of encoding resources, the likelihood function of a current stimulus often becomes asymmetric, with a long tail away from the most likely predicted stimulus orientation (Wei and Stocker, 2015; Figure 6B). Any subsequent ‘readouts’ of this likelihood function by downstream decoders, which take the full likelihood function into account, will consequently be biased away from the predicted stimulus.

We found that such an observer with optimally efficient encoding at the likelihood stage, but no influence of the sensory history on the decoding stage (i.e. a uniform, history-independent decoding prior), is able to produce long-term repulsive biases. However, this observer additionally results in repulsive and not attractive biases in response to the most recently experienced stimuli (Figure 7—figure supplement 2). Therefore, an observer with efficient coding alone is not able to capture the transition from short-term attractive to long-term repulsive biases observed in the empirical data.

Neither an observer in which sensory history influenced only Bayesian decoding nor only efficient encoding was able to reproduce the pattern of attractive and repulsive biases that we observed in perceptual decisions. An intriguing possibility is that the coexisting attractive and repulsive biases might arise from a concurrent optimization of both encoding and decoding according to the sensory history. We tested this idea by developing an ideal observer model with both efficient encoding and history-dependent Bayes optimal decoding of sensory information (see ‘Full efficient-encoding-Bayesian-decoding model’ in Materials and methods). In this observer, efficient coding determined the shape of the likelihood function during the encoding of the current stimulus orientation, whereas Bayesian decoding prescribed how this likelihood function was subsequently integrated with a prior prediction about the current stimulus orientation. Importantly, we allowed predictions used for optimizing encoding and decoding to be based on potentially distinct transition distributions and integration time constants. This was motivated by the possibility that encoding transition models in low-level sensory circuits could be relatively inflexible and learned over the organism’s lifetime, whereas higher-level transition models used for decoding are likely context- and task-dependent (Fischer et al., 2020) and learned quickly, within minutes to hours (Braun et al., 2018).

We found that this ideal observer model with efficient encoding and history-dependent Bayesian decoding was able to faithfully reproduce short-term attraction and long-term repulsion biases across all three experiments (Figure 7 and Figure 7—figre supplement 1). It predicted estimation biases better than observers, in which sensory history only influenced Bayesian decoding or only efficient encoding (Figure 7—figre supplement 3; cross-validated prediction accuracy, Exp. 1: r = 0.58 vs. 0.29 and 0.27; Exp. 2: r = 0.53 vs. 0.38 and 0.20; Exp. 3: r = 0.66 vs. 0.32 and 0.42). Furthermore, the observer captured estimation biases far better than an observer in which efficient encoding and Bayesian decoding were based on the same predictions, using the identical transition distribution and integration time constant (see “Efficient-encoding-Bayesian-decoding model (single prior)” in Materials and methods; cross-validated prediction accuracy, Exp. 1: r = 0.58 vs. 0.29; Exp. 2: r = 0.53 vs. 0.20; Exp. 3: r = 0.66 vs. 0.42). Crucially, we found that predictions for efficient encoding and Bayesian decoding were integrated over different timescales. Bayesian decoding was based on predictions with a short half-life, thus only integrating information of the most recent trials (decoding t_1/2: Exp. 1: 4.59 s; Exp. 2: 4.47 s; Exp. 3: 3.94 s). Conversely, predictions used for efficient encoding were integrated over a longer timescale, as indicated by substantially longer half-lives (encoding t_1/2: Exp. 1: 40.10 s; Exp. 2: 18.94 s; Exp. 3: 30.99 s). Moreover, we observed that transition distributions, reflecting internal models of orientation changes occurring in the environment, were broader at the encoding compared to decoding stage across all three experiments (Exp. 1: ${\sigma }_{tr}$= 11.28° vs 8.35°; Exp. 2: ${\sigma }_{tr}$= 13.02° vs 9.14°; Exp. 3: ${\sigma }_{tr}$= 21.77° vs 9.75°). This suggests that the observer expects orientation information to be more variable at encoding compared to decoding. One speculative explanation is that the encoding transition model captures more variable orientation changes in spatially localized, retinotopic regions of the visual field, whereas the decoding transition model captures more stable orientation transitions between successively attended features across a broad region of visual space.

Finally, we sought to corroborate evidence for the involvement of efficient encoding principles in the present experiments. The current efficient coding scheme prescribes that more encoding resources should be dedicated to stimulus features encountered in the recent history. Next to inducing repulsive biases, such a reallocation of encoding resources would increase the fidelity of stimulus representations that are similar to recently encountered orientations, by narrowing the width of the likelihood function. From this follows the prediction that estimation responses should be less variable when current and previous stimulus orientation are similar. We indeed observed a marked reduction in response variability when the current and 1-back stimulus were of similar orientation (SD reduction – Exp. 1: −0.99°±0.18; Exp. 2: −0.59°±0.14; Exp. 3: −0.62°±0.12; Figure 7—figure supplement 4; for details see Materials and methods). However, variability estimates for temporally more distant stimuli were noisy and did not exhibit a consistent reduction for similarly oriented n-back stimulus orientations (Figure 7—figure supplement 4), so we did not find evidence for a slow decay in variability, as we reported for the repulsive bias.

Together, the current findings indicate that the coexisting attractive and repulsive biases in perceptual decisions may originate from the concurrent optimization of efficient encoding and Bayes-optimal decoding of visual information. Furthermore, our model suggests that human observers use distinct predictions, accumulated over different timescales of the perceptual history, to optimize encoding and decoding of sensory information in a stable environment.

Across three perceptual estimation experiments, we found that while the current estimation response is attracted towards the most recently perceived stimuli, it is simultaneously repelled from stimuli seen further in the past. Based on our ideal observer model, we estimate that the attractive influence of a previous stimulus decreases exponentially and is reduced to half of its initial magnitude after ~4.5 s. This exponential half-life is in line with previous studies, which found attractive serial dependencies to decay over a relatively short time period of about 10–15 s (Fischer and Whitney, 2014; Suárez-Pinilla et al., 2018; Van der Burg et al., 2019). Conversely, timescale estimates of repulsive adaptation effects in response to brief stimuli have been more variable. Several studies have reported short-lived adaptation, persisting no more than a few seconds (Kanai and Verstraten, 2005; Pastukhov and Braun, 2013; Pastukhov and Braun, 2013; Pavan et al., 2012). However, recent studies found that initial attraction biases are replaced by long-term repulsive biases that extend many trials into the past (Gekas et al., 2019; Gordon et al., 2019; Suárez-Pinilla et al., 2018), potentially lasting for minutes (Chopin and Mamassian, 2012; but see Maus et al., 2013). Our current findings are in line with these latter reports of long-term repulsive adaptation. We estimate the repulsive bias reaches half of its initial magnitude after 20 to 40 s and this is thus marked by a distinctly slower decay compared to the short-term attraction bias. Importantly, several of the previous studies reporting long-term repulsive biases measured perceptual decisions in two-alternative forced choice paradigms (Gordon et al., 2019), while employing repeated presentations of a limited set of stimuli (Chopin and Mamassian, 2012; Gekas et al., 2019). In the current study we demonstrate that long-term repulsive biases are similarly present in two-alternative forced choice (Experiment 4) and continuous report paradigms (Experiment 1–3), even when presenting continuously varying stimuli from the full range of possible orientations. Thus, repulsive biases are unlikely to arise due to artifacts of the measurement method or necessitate the repeated presentation of the same stimuli, but present a genuine perceptual bias arising from the brief exposure to individual stimuli experienced seconds to minutes in the past.

Notably, opposite effects to ours have also been reported in the literature (Dekel and Sagi, 2015). In particular, Dekel and Sagi observed short-term repulsive biases followed long-term attraction, when observers adapted to natural image stimuli. However, note that the long-term attraction biases were likely explained by response correlations, which are absent in the present experiments (see Dekel and Sagi, 2015, sections 3.4 and 4.4.).

One outstanding question is whether short-term attractive and long-term repulsive biases decay as a function of physical time, or as a function of intervening visual events. We estimated relatively similar half-lives of the temporal decay of both attractive and repulsive biases across estimation experiments, even though experiments differed in their average trial durations (5.37, 6.55 and 8.25 s). This may suggest the decay is mainly determined by intervening time and not the frequency of stimulus presentations. However, estimation experiments also differed in other aspects of experimental design, complicating a direct comparison between experiments. Furthermore, the only source of temporal variability within the current experiments was introduced via the self-paced estimation responses, leading to only minor temporal jitter across trials (in other words, intervening time and intervening trials were very highly correlated). This renders the current experiments unsuitable for distinguishing the role of elapsed time and intervening stimulus presentations for the history decay. Future studies may investigate this question more specifically, by experimentally dissociating time and the frequency of stimulus presentations.

Previous studies proposed that attractive biases in perceptual estimates towards the previous stimulus arise from a probabilistically optimal strategy of decoding sensory information into a perceptual decision (Cicchini et al., 2018; Kalm and Norris, 2018; van Bergen and Jehee, 2019). In a stable environment, integrating current noisy sensory measurements with information about previous stimuli generally results in more accurate perceptual estimates, compared to relying on noisy sensory measurements alone. Effectively, such a Bayesian integration of previous and current visual information would smooth perceptual representations over time, thereby promoting visual stability in the face of disruptive factors, such as eye blinks and external and internal noise. However, in the current study, we show that the temporal pattern of history biases is more complex, consisting of a superposition of concurrent attractive and repulsive biases, which cannot be explained by an observer model based on Bayesian decoding of visual information alone. Importantly, this not only shows that a Bayesian observer model is inadequate to explain the full pattern of history dependencies in perceptual estimates, but the inability of the Bayesian observer to capture both short-term attraction and long-term repulsion biases also indicates that the long-term repulsion biases are not an artifact resulting from short-term attraction biases in combination with random input fluctuations (Maus et al., 2013). Rather, the spatial specificity (Experiment 2) and the presence in immediate perceptual comparisons to a reference stimulus (Experiment 4) suggest a low-level sensory origin of the long-term repulsive bias, akin to sensory adaptation (Gibson and Radner, 1937).

Prominent theories of sensory adaptation posit that repulsive biases may arise as a consequence of optimally efficient encoding of visual information in a stable environment. In particular, repulsive biases may arise due to self-calibration and decorrelation (Barlow and Földiák, 1989; Clifford et al., 2000; Müller et al., 1999), or an increase in the signal-to-noise ratio for measurements similar to recently perceived stimuli (Stocker and Simoncelli, 2006). In line with the latter proposal, we here developed a model in which repulsive adaptation biases arise as a consequence of efficient allocation of sensory coding resources (Wei and Stocker, 2017; Wei and Stocker, 2015). Since in a temporally stable visual environment new visual input is likely to be similar to recent input, the visual system can exploit the sensory history to strategically allocate sensory coding resources such that the mutual information between the likely physical stimulus and its sensory representation is maximized. Such an efficient allocation of sensory resources increases sensitivity to small changes in the input, in line with previous findings of improved discriminability following adaptation (Clifford et al., 2001; Mattar et al., 2018; Regan and Beverley, 1985). At the same time, it leads to a repulsive bias of perceptual estimates, away from previous stimulus orientations, thus producing classical sensory adaptation effects.

Across three experiments involving perceptual estimation tasks, we show that an ideal observer with efficient encoding together with Bayesian decoding can adequately capture the pattern of short-term attraction and long-term repulsion biases in perceptual estimates. This observer model provides a framework for explaining how the visual system exploits temporal regularities in the input to achieve sensitivity to changes in the environment, while at the same time maintaining stable visual processing of visual information. Importantly, however, our model indicates that efficient encoding and Bayesian decoding are based on distinct priors, accumulating previous stimulus information over different timescales and employing distinct internal models of naturally occurring input changes. In particular, at the encoding stage, the prediction about the upcoming stimulus is based on many previous stimulus encodings, reaching far into the past. Conversely, the prediction used for Bayesian decoding is solely based on previous stimulus information encountered in the most recent past (previous 2–3 trials). Moreover, the transition distribution, reflecting an internal model of naturally occurring orientation changes, is broader at the encoding compared to the decoding stage, suggesting that the observer expects successive orientation samples to be less stable at encoding compared to decoding. As a consequence, the efficient allocation of encoding resources is less biased to any particular previous stimulus, but more broadly influenced by the long-term history of visual stimulation. In contrast, at decoding, the relatively stronger contribution of the short-term history to the Bayesian decoding prior leads to a dominating attractive influence of the most recent stimuli, but no influence of long-term stimulus history.

We may speculate on the reasons for the distinct timescales and transition models for efficient encoding and Bayesian decoding. At encoding, the long integration timescale and broad transition model, which effectively diffuse and smooth prior information across time, may reflect the need to adapt to long-term perturbations in visual processing or slowly changing input statistics without over-adjusting to the rapidly changing low-level input at spatially localized, retinotopic regions of the visual field. In contrast, the more narrow transition model and short integration timescale at decoding may be tuned to more stable orientation transitions between briefly attended features across a broad region of visual space. However, such explanations remain speculative at this moment. More extensive characterizations of the spatiotemporal statistics of natural visual input and visual information processing will be needed to advance our understanding of the principles underlying the distinct encoding and decoding priors. In this regard, it will also be interesting to study whether observers can flexibly modify encoding and/or decoding transition models in response to experimentally induced temporal regularities in visual input. There is evidence that human observers can indeed adapt to simple temporal regularities after relatively brief exposure (Braun et al., 2018), and our ideal observer model provides a framework to understand changes in internal transition models across multiple stages of the perceptual decision-making process.

The current model in which attraction biases are due to Bayesian decoding of sensory information provides a parsimonious explanation for the increase of attractive serial dependence biases with working memory delay (Experiment 3; Bliss et al., 2017; Papadimitriou et al., 2015), and the absence of short-term attraction biases in the perceptual comparison task of Experiment 4. In particular, Bayesian decoding posits that the attractive serial dependence bias is dependent on the relative widths of sensory likelihood and prior. When sensory likelihoods are relatively narrow, briefly after the initial encoding of the sensory stimulus, the prior has little influence on the resulting posterior, leading only to small attraction biases. As the sensory representation degrades during working memory retention, and the likelihood becomes increasingly broader, the influence of the prior increases. As a result, the posterior will be increasingly pulled towards the prior, leading to an increase in attraction during working memory retention. Importantly, in Experiment 4, observers could make their perceptual decision about the perceptual comparison stimuli while the stimuli were simultaneously presented on the screen, minimizing the influence of working memory noise and presumably leading to relatively narrow sensory likelihoods at the time of the perceptual decision. As outlined above, this would lead to a strong reduction or even absence of attraction biases in this scenario, consistent with the absence of attraction biases in Experiment 4. Furthermore, in agreement with this proposal, it has recently been found that attractive biases can arise early after stimulus offset when visual stimulation is very weak (Manassi et al., 2018), leading to a broad likelihood function, and are stronger for more uncertain stimulus representations (Cicchini et al., 2018; van Bergen and Jehee, 2019). Whether such a Bayesian decoding computation operates continuously during working memory retention, causing systematic drifts in working memory representations over time (Panichello et al., 2019; Wolff et al., 2020), or occurs only once during the final read-out of the working memory representation is an exciting question for future research.

The current model bears some similarity to a recently proposed two-process model of serial dependence in which repulsive biases are mediated by sensory adaptation, in the form of gain changes in low-level perceptual neural populations, whereas attractive biases arise due to persistent read-out biases at a higher-level decisional units (Pascucci et al., 2019). A key distinction to the model by Pascucci et al. is that the current model of repulsive adaptation and attractive serial dependence is based on a normative framework of efficient encoding and Bayesian decoding of visual information. At the same time, the current model is agnostic to the neural implementation underlying efficient encoding and Bayesian decoding. In fact, at the level of neural population tuning, efficient encoding could be realized by multiple alternative mechanisms, involving changes in the location, width and/or gain of neural tuning curves of sensory neurons (Wei and Stocker, 2015). Similarly, Bayesian decoding could be achieved by combining predictions with sensory information in low-level sensory circuits via feedback mechanisms or later integration in mid- and high-level visual areas. The model by Pascucci et al., which models a potential neural implementation of history biases, and the current model are therefore not mutually exclusive, but rather reside at two different levels of explanation. Notably, however, if human visual processing indeed follows the computational goals of efficient encoding and Bayesian decoding of visual information, this will put important constraints on the neural mechanisms that need to fulfill these goals.

The general premise that repulsive biases arise at a low-level encoding stage, whereas attractive biases arise at a higher-level decoding stage is well supported by recent empirical findings. For instance, it has been shown that in contrast to repulsive adaptation, attractive serial dependence biases are strongly dependent on attention (Fischer and Whitney, 2014; Fritsche and de Lange, 2019; Suárez-Pinilla et al., 2018), and are modulated by subjective confidence (Samaha et al., 2019; Suárez-Pinilla et al., 2018). Furthermore, in contrast to repulsive adaptation biases, which exhibit a clear spatial specificity (Figures 3 and 5; Knapen et al., 2010), attractive serial dependencies appear to have a very broad spatial tuning (Collins, 2019; Fischer and Whitney, 2014; Fritsche et al., 2017) and have been shown do increase during the post-perceptual working-memory delay period (Bliss et al., 2017; Fritsche et al., 2017; Papadimitriou et al., 2015). A recent study showed that attractive serial dependencies are abolished when backward-masking the previous inducer stimulus, suggesting a role of high-level feedback processing in attractive serial-dependence (Fornaciai and Park, 2019). Conversely, repulsive adaptation was not affected by masking, consistent with feedforward changes in local, low-level neural circuits underlying adaptation. Moreover, a recent study found that attractive serial dependences can be modulated by contextual features, such as stimulus color or serial position in a trial, suggesting a more selective integration of prior information at the higher-level decoding stage (Fischer et al., 2020). Finally, there is evidence that repulsive adaptation and attractive serial dependence map onto distinct cortical networks (Schwiedrzik et al., 2014). While adaptation was confined to posterior regions, comprising early visual cortex, a more widespread network of higher-level visual and fronto-parietal areas was involved in attractive serial dependence.

Despite the above empirical evidence that points towards low- and high-level visual processing as the source of repulsive and attractive history biases, one may wonder to which extent the current biases could be explained by temporal dependencies in low-level motor processing instead. There are several arguments against a systematic influence of motor biases in the present experiments. First, in each trial of the perceptual estimation tasks (Experiments 1–3), the initial orientation of the response bar was randomly selected from the range of all possible orientations (0180], and thus the motor actions required for adjusting the response bar to a particular orientation were independent across trials. Second, several previous studies have shown that an explicit reproduction response is not necessary to elicit an attraction bias on the next trial (Czoschke et al., 2019; Fischer and Whitney, 2014; Suárez-Pinilla et al., 2018), that perceptual decisions are attracted towards the presented stimulus orientation rather than a mirrored response orientation (Cicchini et al., 2017), and that attraction biases occur even when performing a different task (and response) on the previous stimulus (Fritsche and de Lange, 2019). Together, these studies strongly suggest that short-term attractive serial dependencies are not due to low-level motor biases. Furthermore, it seems unlikely that low-level motor biases would lead to larger attraction biases with increasing working memory delay (Experiment 3) and that they would induce spatially specific long-term repulsion biases (Experiments 2 and 4). Additional evidence against an involvement of motor biases in the long-term repulsion effect is provided by Experiment 4. In this experiment, participants alternate between a reproduction task and 2-AFC task, which require very different motor responses. Furthermore, stimuli in Experiment 4 were counterbalanced such that simple motor biases in the 2-AFC task (e.g. a previous response to a leftward tilted stimulus in the reproduction task induces a bias to press the left response button in the 2-AFC task) would not induce systematic biases in the analysis of the perceptual comparison data. Taken together, while there is evidence for temporal dependencies in low-level motor responses (Pape and Siegel, 2016), the above points provide strong arguments against a systematic influence of such dependencies on the current estimate of short-term attraction and long-term repulsion biases in perceptual decisions.

Finally, next to short-term attraction between successive stimuli with similar orientations, in Experiment 2 and 3 we observed systematic short-term repulsion when the orientation difference between current and previous stimuli was large (>60°; see Figures 3B and 4B), in line with previous findings (Fritsche and de Lange, 2019; van Bergen and Jehee, 2019). In the current study, we specifically focused on short-term attraction biases between similar stimuli and the current ideal observer model does not capture the repulsion biases between highly dissimilar stimuli. We previously found that while the attraction bias between stimuli with similar orientations were strongly dependent on attention to previous stimulus features, the repulsion bias for dissimilar stimuli was not modulated by attention (Fritsche and de Lange, 2019). This dissociation speaks against a model in which attraction of similar and repulsion of dissimilar stimuli are mediated by the same Bayesian decoding process. Rather, the short-term repulsion effect could reflect an encoding bias, that is sensory adaptation. While the repulsion bias of the previous stimulus is indeed partly captured by the efficient encoding component in Experiment 3 (Figure 7, lower row), it is not accounted for in Experiment 2 (Figure 7, middle row). Furthermore, in Experiment 1, repulsion by previous dissimilar stimuli is absent, even though long-term repulsion effects are clearly present (Figure 7, upper row). This suggests that long-term and short-term repulsion biases are (at least partially) independent. Taken together, more research is necessary to elucidate the nature of the repulsion biases for large orientation differences and their potential relationship to efficient encoding and Bayesian decoding of sensory information.

It should be noted that current ideal observer model with efficient encoding and Bayesian decoding is based on several assumptions. For instance, we assumed a particular efficient coding scheme in which the observer maximizes the mutual information between physical stimuli and sensory representations (Wei and Stocker, 2015). Due to the efficient reallocation of encoding resources to stimulus features encountered in the recent history, this efficient encoding scheme makes the prediction that estimation responses should be less variable when current and previous stimulus orientation are similar. While we indeed observed a short-term reduction of response variability when current and previous stimuli were similar, variability estimates for temporally more distant stimuli were noisy and did not exhibit a consistent reduction for similarly oriented n-back stimulus orientations, so we did not find evidence for a slow decay in variability, as we reported for the repulsive bias. Thus, we were not able to fully confirm the predicted changes in encoding fidelity for temporally more distant trials. Future studies may specifically test whether long-term repulsive biases are accompanied by reductions in discrimination thresholds, as predicted from the efficient coding scheme based on the maximization of mutual information. There exist alternative formulations of efficient coding, such as minimizing redundancy (Barlow, 1961) or reconstruction error (Wang et al., 2012). Encoding efficiency may also depend on the specific task context for which the encoded information is used. Since low-level orientation information is likely used for many different tasks, the current efficient coding scheme in which a more generic information criterion is optimized may represent a good general strategy for the visual system (Wei and Stocker, 2015). A further assumption is that the observer reads out the mean of the posterior distribution at the Bayesian decoding stage. This presents the optimal strategy for minimizing the expected squared error (Ma, 2019), but it is unclear whether the visual system employs this strategy. It is important to note that different encoding schemes and read-out rules may or may not lead to similar predictions. More work is needed to explore the validity of these assumptions and the consequences for modeling attractive and repulsive history dependencies.

Notably, the effects of adaptation on neural tuning are diverse, and may depend on many stimulus parameters such as stimulus size and duration in non-trivial ways (Patterson et al., 2014; Patterson et al., 2013). Several neural mechanisms, such as stimulus-specific fatigue and suppressive normalization, have been proposed to underly these neural tuning changes (Solomon and Kohn, 2014) and have been related to functional explanations of adaptation, such as efficient coding (Schwartz et al., 2007), modulations of stimulus salience (McDermott et al., 2010; Wissig et al., 2013), and prediction (Chopin and Mamassian, 2012). Importantly, the current study cannot explicitly rule out such alternative explanations of repulsive adaptation biases. Rather, here we seek to illustrate a principled way in which the visual system can exploit the temporal stability in the environment across multiple processing stages. Future studies will need to investigate how population-level changes in neural responses may support these different functional explanations of adaptation, respectively.

Our results demonstrate that visual processing is concurrently influenced by repulsive encoding and attractive decoding biases, operating over distinct timescales. While attractive decoding biases are confined to the immediately preceding stimuli, repulsive encoding biases decay at a distinctly slower rate. The temporal pattern of short-term attraction and long-term repulsion can be well captured by an ideal observer model with efficient encoding and Bayesian decoding of visual information in a stable environment. This model illustrates a principled way in which observers can exploit the temporal stability in the environment across multiple processing stages to concurrently achieve efficient and stable visual processing.

Participants were recruited from the institute’s subject pool and received either monetary compensation or study credits. All participants provided written informed consent prior to the start of the experiments. All experiments were approved by the Radboud University Institutional Review Board (CMO region Arnhem-Nijmegen, The Netherlands; Protocol CMO2014/288). Participants reported normal or corrected-to-normal visual acuity. Stimuli were generated with the Psychophysics Toolbox for MATLAB and were displayed on a 24’' flat panel display (Benq XL2420T, resolution 1920 × 1080, refresh rate: 60 Hz). Participants viewed the stimuli from a distance of 53 cm in a dimly lit room, resting their head on a table-mounted chinrest. All data analyses were performed with MATLAB.

The data of Experiment 2 were collected as part of a previously published study, where we reported on the bias exerted by the previous stimulus (Experiment 1 in Fritsche et al., 2017). Here, we reanalyzed this dataset to investigate biases exerted by stimuli encountered further in the past, when presented at the same or different spatial location as the current stimulus.

The data of Experiment 3 were collected as part of a previously published study, where we reported on the bias exerted by the previous stimulus (Experiment 4 in Fritsche et al., 2017). Here, we reanalyzed this dataset to investigate biases exerted by stimuli encountered further in the past, when the working memory delay on the current trial was either short or long. Experiment 4 was conducted in two separate sessions. Experimental sessions were conducted on different days, but scheduled no longer than three days apart. The experimental procedures were similar across the two sessions.

The data of Experiment 4 were collected as part of a previous study, where we reported on the bias exerted by the history-inducing stimulus on the current (Experiment 2 in Fritsche et al., 2017). Here, we reanalyzed this dataset to investigate biases exerted by inducer stimuli encountered further in the past.

After obtaining a noisy sensory measurement of the current stimulus, $p\left(m^{\left(t\right)}\right|{\theta }^{\left(t\right)})$, the observer seeks to infer the stimulus orientation, which most likely caused the measurement. Importantly, when doing so, the observer does not only rely on the current sensory measurement, but makes use of predictions based on knowledge about previous stimulus orientations (van Bergen and Jehee, 2019). While, in principle, encoding and decoding stages could utilize the same predictions about the upcoming stimulus (see ‘Encoding-decoding model (single prior)’ below), here we allowed encoding and decoding to be based on separate predictions, involving separate transition models and integration time constants. For instance, transition models in low-level sensory circuits, encoding visual information, could be acquired over the organism’s lifetime, whereas higher-level transition models used for decoding could be context dependent (Fischer et al., 2020) and learned quickly, within minutes to hours (Braun et al., 2018).

Following van Bergen and Jehee, 2019, predictions at the decoding stage were computed by convolving the distribution of knowledge about the previous stimulus ${\displaystyle p\left({\theta }^{\left(t\right)}|m^{\left(t_0:t-1\right)}\right)}$ with the distribution of changes that can occur between two consecutive measurements, as given by the transition model ${\displaystyle p_{decoding}\left({\theta }^{\left(t\right)}|{\theta }^{\left(t-1\right)}\right)}$:

Importantly, even though the Bayesian inference process is recursive and consequently the prediction ${\displaystyle p\left({\theta }^{\left(t\right)}|m^{\left(t_0:t-1\right)}\right)}$ theoretically contains information about all sensory measurements since the starting point of the inference process, it has been previously found that such a model does not adequately capture positive serial dependencies beyond the most recent trial (Kalm and Norris, 2018). Therefore, when predicting the upcoming stimulus orientation, the current observer does not only rely on the most recent prediction, but utilizes an exponentially weighted mixture of previous predictions (Kalm and Norris, 2018):

with weights

Similar to encoding, the exponential decay function is defined over elapsed trials, where t denotes the index of the current trial and i denotes the index of a previous trial. Subsequently, the ideal observer combines this prediction with the current sensory measurement according to Bayes, resulting in a posterior distribution:

We assume that the observer reports the mean of this posterior distribution as their best estimate of the viewed stimulus orientation. Reporting the mean presents the optimal strategy for minimizing the expected squared error (Ma, 2019).

It should be noted that in this formulation of the ideal observer, the prior of the current trial consists of a mixture of previous trials’ posteriors, from which previous perceptual decisions were estimated, convolved with a transition distribution. Therefore, the likelihood of the current trial is pulled towards previous posteriors, more closely reflecting previous decisions, rather than previous likelihoods, reflecting previous stimuli.

We compared four different models in their ability to capture the temporal pattern of short-term attraction and long-term repulsion biases in our perceptual estimation experiments. While all of these models consisted of an encoding and decoding stage, they differed in how the sensory history influenced encoding and decoding, respectively:

We fitted all four models to the empirical serial dependence biases of Experiment 1 to 3, respectively. In particular, models were fit to the 1- to 20-back group-average serial dependence curves of each experiment. In a first step, we performed a grid search over a wide range of parameter combinations and evaluated the sum of squared residuals (SSR) between predicted and observed serial dependence curves. Subsequently, we used the parameter combination with the lowest SSR as a starting point for further numerical minimization of SSRs using MATLAB’s lsqcurvefit algorithm. To ensure plausible parameter estimates we constrained parameters to ${\displaystyle 0<{\sigma }_s<60}$,${\displaystyle {\sigma }_{td}>0}$, $0\le p_{same}\le 1$, ${\displaystyle {\tau }_{encoding}>0}$, ${\displaystyle {\tau }_{decoding}>0}$. Parameters of the encoding and decoding transition models of the full encoding-decoding model were constrained to the same ranges.

In order to evaluate the models’ ability to capture the empirically observed serial dependence biases, we employed a cross-validation procedure. Separately for each experiment, we split the data of each participant into two halves (odd- versus even-numbered blocks) and computed the group-average 1- to 20-back serial dependence curves for each half. We then conducted the model fitting on the first half and testing on the second half. The models’ performance was evaluated based on the Pearson correlation between the predicted and observed 1- to 20-back serial dependence curves of the testing dataset. This cross-validation procedure accounts for the increased flexibility of the full encoding-decoding model, due to its greater number of free parameters.

In order to establish a baseline level of chance prediction accuracy of each model, we employed a permutation procedure. Specifically, we randomly permuted the models’ predicted responses to the stimuli in the testing dataset, thereby abolishing the trial correspondence between model and participant errors, and computed the correlation between the resulting permuted model predictions and observed serial dependence curves. This procedure was repeated 1000 times and p-values were taken as the proportion of permutation correlations, which exceeded the true correlation between predicted and observed serial dependence curves.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Human subjects: The study followed institutional guidelines of the local ethics committee (CMO region Arnhem-Nijmegen, The Netherlands; Protocol CMO2014/288), including informed consent of all participants.

© 2020, Fritsche et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.