Introduction

Predictive processing is a framework for understanding brain function. It proposes that the brain constructs internal models of the world based on regularities in past sensory input and sensorimotor loops to predict upcoming sensory input. The brain does this by minimizing the mismatches between predicted and actual sensory input. These mismatches, known as prediction errors, play a dual role in both updating internal representations and internal models (Friston, 2005; Keller and Sterzer, 2024; Rao and Ballard, 1999).

Predictions about sensory input can be made based on a variety of sources of other information the brain has available. Certain stimuli can be predicted from the preceding sensory input – experimentally this can be used, e.g., in oddball paradigms to trigger stimulus history based prediction errors (Garrido et al., 2009). Predictions can be crossmodal. Certain sounds are associated with specific types of visual input. Experimentally visual stimuli can be coupled to sounds to then trigger audiovisual prediction errors in mouse visual cortex (Garner and Keller, 2022). Predictions can also be based on memory. Not seeing a stimulus in a specific spatial location can result in visuospatial prediction errors in mouse cortex (Fiser et al., 2016). However, the strongest predictor - by consistency of coupling - is self-generated movement. Every movement is directly coupled to sensory feedback throughout life.

An essential ingredient to the computation of visuomotor prediction error responses are motor-related predictions of visual feedback (Keller and Mrsic-Flogel, 2018). In the mouse, evidence for such predictions have come from the discovery of a strong motor-related modulation of activity in visual cortex (Keller et al., 2012; Niell and Stryker, 2010; Saleem et al., 2013). These motor-related signals are likely in part driven by motor-related predictions arriving from motor areas of cortex (Leinweber et al., 2017). However, the overall level of this motor-related activity is much higher than one would expect simply from predictions of visual feedback that are compared against visual input.

The more precise the prediction and comparison, the less motor-related activity should be detectable in visual cortex. This is evident in the auditory system of a songbird, which relies on very precise sensorimotor error detection for vocal learning, where there is much less motor modulation of auditory responses (Keller and Hahnloser, 2009). Similarly, there is very little movement related modulation of activity in visual cortex of non-human primates (Liska et al., 2024; Talluri et al., 2023). In humans, movement-related modulation of EEG activity in visual cortex has been recently described (Cheng and Nordin, 2025), but it is still unclear if visual responses are modulated by these signals. If indeed the precision of visuomotor coupling determines the amount of motor modulation of visual responses, we would expect to find less of it in humans than in the mouse.

Imprecise comparisons of motor related predictions and visual input should manifest as increases in motor-related activity and smaller prediction error response. Based on this, we would expect to find strong visuomotor mismatch responses in humans, similar to – or stronger than - those observed in mice (Keller et al., 2012). Based on this and prior work demonstrating that there are selective responses in human visual cortex to a mismatch between actual hand position and that of a virtual hand (Stanley and Miall, 2007), we expected to find strong EEG responses to a break in coupling between locomotion and visual feedback.

Results

Visual responses

To quantify visual responses in freely moving human participants we combined a virtual reality headset with an 8-electrode wireless EEG recording system (Methods; Figure 1A). Participants were shown a virtual environment that consisted of a large empty floor space with a square checkerboard pattern fixed to their viewing angle directly in front of them (Figure 1B). The checkerboard covered a visual angle of 53° and contained a red fixation dot in the center. Participants were asked to fixate on the red dot throughout all measurements. The checkerboard reversed white-black at intervals sampled from a uniform distribution between 2 and 4 s (Figure 1C). Recordings were split into a passive session during which participants were seated on a chair, and an active session during which they were instructed to walk around a rectangular floor area of 5 by 7 m. During the entire experiment we tracked the location of participants (Figure 1D).

Figure 1.

Walking triggered movement related artifacts in the EEG signals (Figure S1). The strength of these movement artifacts varied as a function of hairstyle and gait of the participants. All participants were encouraged to go ‘gentle’ to reduce this problem. We excluded all data with high movement related artifacts from further analysis (Methods; visual paradigm: 19 of 48 recordings, visuomotor mismatch paradigm: 15 of 48 recordings).

We could detect multiphasic responses to the checkerboard inversion in occipital EEG electrodes (Figures 1E and S2). These responses were comparable to previous reports (Drislane, 2007) and exhibited a positive peak at around 88 ms (P1). Interestingly, when participants were walking, the same responses showed an early negative deflection with a peak latency of 48 ms that preceded P1 (Figure 1F).

While participants were instructed to fixate on the red dot, it is possible that there are systematic differences in eye movements between walking and sitting conditions. Our setup did not allow for concurrent eye tracking; thus we cannot exclude the possibility that differences in eye movements contribute to the differences in EEG responses we observe. However, given the relatively rapid onset of response differences (40 ms), we suspect that stimulus triggered eye movement cannot account for these early differences.

Visuomotor mismatch responses

To measure visuomotor mismatch responses, participants were instructed to walk around in a virtual corridor (Figure 2A). The corridor was 1 m wide, 2.4 m high and of an oval shape covering approximately 7 by 5 m (Figure 2B). Virtual movement in the corridor was coupled to the movement of participants. We refer to this as closed loop. To trigger visuomotor mismatches, we briefly (0.5 s) halted the coupling between movement of the participants and visual feedback in the virtual corridor at random times (every 10 to 15 s; Methods; Figure 2C). Throughout all experiments, we tracked the virtual position of participants as they were walking around the corridor (Figure 2D).

Figure 2.

Anecdotally, the perception some participants reported experiencing during these visuomotor mismatches was a very salient sudden movement of the entire visual world. In the words of one participant: “The world suddenly flew forward! – Are you printing this? – Hi Mom!”. This was reminiscent of a case report of a patient with a lesion to the lateral rectus muscle described by von Helmholtz (von Helmholtz, 1867). The lateral rectus muscle moves the eye laterally. The patient, upon attempting to move the affected eye laterally, reported seeing the world rapidly moving in the direction of the intended eye movement. One interpretation of this is that the combination of movement and absence of resulting visual flow change results in the perception that the world must be moving.

In the EEG recordings of occipital electrodes, we found strong responses triggered by these visuomotor mismatches (Figure 2E). Responses were dominated by a positive component peaking at 180 ms. Following the 0.5 s halting of the visual stimulus, the virtual location was updated to match the actual location of participants, and the visual flow coupling was resumed. This resulted in a sudden change of visual stimulus combined with a visual flow onset and drove what is likely a visual response in the EEG signal (Figure 2E).

To quantify how much of the visuomotor mismatch response can be explained by visual input alone, we exposed participants to a replay of the visual flow that was self-generated in the preceding closed loop session, including the visual flow halts that constitute visuomotor mismatch in the closed loop condition. We refer to this as open loop. For these experiments, participants were seated on a chair. To reduce the likelihood of triggering nausea in participants, roll and pitch movements of the head were removed from this replay. We found measurable responses to visual flow playback halts (Figure 2F), but these were much smaller than visuomotor mismatch responses (Figure 2G). In the few participants who volunteered to experience playback with pitch and roll movements, playback halt responses were not different from those in the reduced playback condition (Figure S3). The difference in response amplitude between mismatch and playback response could not be explained by a simple movement related gain of visual responses, as the response to the re-onset of visual flow was similar in both closed and open loop conditions (Figure 2H). Note that the mismatch responses and playback halt responses shown here do not come from fully overlapping pools of participants (Table S1). A subset of sessions had to be excluded because of recording noise (visuomotor mismatch: 15 of 48, playback: 4 of 48), while some participants aborted the playback recording due to developing nausea (11 of 48; Methods). A participant-wise comparison of mismatch and playback halt responses is shown in Figure S4.

Although participants reduced their walking speed in response to the visuomotor mismatch, this reduction was merely a trend and did not reach statistical significance (Figure 2I). However, given the comparably slow time course of reduction in walking speed, these behavioral changes cannot explain visuomotor mismatch responses.

Distribution of visuomotor mismatch response strength

In mice, visuomotor mismatch responses originate in primary visual cortex and propagate to a network of areas across dorsal cortex (Heindorf and Keller, 2023; Takeuchi et al., 2024). Thus, we expected to find larger and faster responses in the two occipital electrodes. To test this, we compared the visuomotor mismatch responses across the eight recorded locations in both response strength and timing (Figures 3A and 3B). We indeed found that the strongest responses were observed at occipital electrodes O1 and O2, though significant responses were also present at frontal (Fp1) and central (C3) sites (Figure 3C). However, while there was a trend toward faster responses in occipital cortex, we found no evidence of differences in response latency across the different electrodes (Figures 3D and 3E).

Figure 3.

Comparison of visual responses and visuomotor mismatch responses

The time course of visual responses (Figure 1E) and visuomotor mismatch responses (Figure 2E) differed in that they appeared to exhibit reversed polarity (Figure 4A). Despite this difference in polarity, the initial peaks occurred at similar latencies. The total power of the visuomotor response was higher than that of the visual response (Figure 4B) but given that we did not optimize either the visuomotor mismatch stimulus nor the visual stimulus for maximum response, this comparison is more qualitative than quantitative in nature. We include it here primarily to illustrate how the visuomotor mismatch response compares to a more conventional visual response.

Figure 4.

Comparison of visuomotor mismatch responses and auditory oddball mismatch responses

To further contrast the strength of visuomotor mismatch to other frequently used prediction error signals, we also recorded EEG responses in an auditory oddball paradigm. Mismatch responses recorded with this paradigm can be thought of as a stimulus history prediction error (Garrido et al., 2009; Näätänen et al., 2007). We implemented an oddball paradigm composed of a sequence of identical tones that was occasionally disrupted by a tone of a different frequency (Figure 5A). We used pure tones of 1 kHz and 1.2 kHz frequency, and alternated in blocks which tone was standard and which was the deviant (Methods). To enable direct comparison with visuomotor mismatch responses, recordings were made from occipital electrodes O1 and O2. Participants watched short silent movies in the VR headset while listening to the tone sequences delivered in headphones. To isolate the oddball driven mismatch response, we subtracted the response to the tone when presented as a standard from the response to the same tone when presented as an deviant. The resulting difference waveform closely resembled those reported in previous studies (Figures 5B and 5C) (Näätänen et al., 2007). Comparing these oddball mismatch responses to the visuomotor mismatch response, we found that both had similar dynamics with a dominant positive peak at around 180 ms (Figure 5D). Comparing the total power of the two responses, we found that visuomotor mismatch responses were significantly larger than oddball mismatch responses (Figure 5E). Also here the comparison of total power is more qualitative than quantitative as neither stimulus was optimized for maximum EEG response, and we include it to illustrate how typical variants of these responses compare.

Figure 5.

Discussion

The utility of paradigms for measuring prediction errors in humans primarily comes from their relevance to psychiatric disorders, where disruptions in prediction error processing are widely implicated (Adams et al., 2013; Fletcher and Frith, 2009; Kirihara et al., 2020; Qela et al., 2025; Sterzer et al., 2018). Various types of prediction error responses are being explored as potential biomarkers in clinical research, serving to monitor disease progression, evaluate symptoms, and provide insight into the underlying neural mechanisms of these conditions.

The most prominent of these prediction errors is mismatch negativity (MMN). MMN is the first component of the oddball mismatch response (Näätänen et al., 1978). MMN has been shown to be reduced or abnormal in patients with schizophrenia spectrum disorders (Todd et al., 2012; Umbricht and Krljes, 2005), autism spectrum disorders (Chen et al., 2020; Dunn et al., 2008; Schwartz et al., 2018), speech disorders (El Hatal de Souza et al., 2020), depression (Tseng et al., 2021), and bipolar disorders (Chitty et al., 2013). More generally, it is thought that many of these disorders can be related to changes in predictive processing. In the case of schizophrenia, for example, the symptoms are thought to arise from an imbalance in the strength of high and low level predictions (priors) (Schmack et al., 2017, 2015a, 2015b, 2013). Further testing these ideas will require experiments that trigger identified functional responses in humans, for which we have a circuit level understanding based on cell type specific recordings.

We were able to measure robust visuomotor mismatch responses in freely moving humans. These signals cannot be attributed to changes in participants’ behavior or to simple visual offset responses. Notably, the visuomotor mismatch responses exhibited a markedly different temporal profile compared to purely visual responses. What might account for this difference? One possibility is that visuomotor mismatch signals may rely more strongly on input from other cortical areas than visual responses and are thus delayed relative to visual responses. A more interesting interpretation is that visual responses and visuomotor mismatch responses are both prediction errors of different types. A central tenet of the cortical circuit for predictive processing is the split into separate populations of neurons that compute positive and negative prediction errors (Keller and Mrsic-Flogel, 2018; Rao and Ballard, 1999). In this interpretation, a visuomotor mismatch response is a negative prediction error, while the response to a visual stimulus is a positive prediction error.

Visuomotor prediction errors are likely computed in layer 2/3 of mouse visual cortex (Jordan and Keller, 2020), and are also preferentially detectable in superficial layers of human visual cortex (Thomas et al., 2024). In the mouse, positive and negative prediction errors are likely computed in separate populations of neurons in visual cortex (Keller and Mrsic-Flogel, 2018; O’Toole et al., 2023). These two cell types have a different depth distribution, with negative prediction error neurons more superficial and positive prediction error neurons located deeper in cortex (O’Toole et al., 2023). Given that EEG signals are thought to exhibit positive or negative deflections as a function of depth of the source (Cohen, 2017; Kirschstein and Köhling, 2009), the polarity reversal of visual and visuomotor mismatch responses (Figure 4) may be the consequence of different populations of cells being activated and inhibited. Visuomotor mismatch should activate negative prediction error neurons and inhibit positive prediction error neurons, while the reverse is the case for a visual stimulus.

The fact that there is also a strong visual response to the visual flow re-onset following visuomotor mismatch means that our visuomotor mismatch paradigm might allow us to measure both negative and positive prediction error responses. More intriguingly, there is a differential modulation by walking of the two responses (Figures 2G and 2H). In theory, visuomotor mismatch drives a combination of two negative prediction errors. One is based on a movement related prediction. The other on a stimulus-history related prediction. Both the self-motion as well as the ongoing visual flow would predict a continuation of visual flow. A visuomotor mismatch violates both predictions. During passive observation there is no movement-related prediction, but the stimulus history-based prediction is still violated.

Intriguing is the similarity in timing and polarity of the playback halt and the difference between oddball and standard response (Figure 5D). An oddball response is always a combination of both a positive and a negative prediction error. There is a negative prediction error for the absence of the expected standard tone as well as a positive prediction error for the presence of the unexpected oddball tone. Thus, it is conceivable that the subtraction of standard response from the oddball response, isolates the component of the oddball response that corresponds to a negative prediction error.

One of the key challenges in systems neuroscience is translating findings from animal models to humans. Although recent animal studies have provided detailed insights into the circuit level implementation of predictive processing in the cortex (Keller and Mrsic-Flogel, 2018), and psychological and psychiatric conditions have long been described within this framework (Keller and Sterzer, 2024; Sterzer et al., 2018), translation to human research has been limited by a lack of methods to record cell type specific signals in human experiments. With an understanding from animal models of how positive and negative prediction errors are computed, and which cortical layers are involved, an approach based on functionally identified responses might be most promising. One step in this direction is to use experimental paradigms that can cleanly separate e.g. positive and negative prediction error responses. As argued above, we speculate that the visuomotor mismatch paradigm as we have used it here, is one possible way to achieve this.

Methods

Participants

The study was approved by Ethikkommission Nordwest- und Zentralschweiz (Project-ID: 2024-02458). Participants signed a written informed consent before participation and were not financially compensated for their participation. 46 healthy adults (19 males, 27 females) participated in the study, ranging in age from 18 to 65 years. The age distribution was as follows: 19 participants aged 18-30, 18 aged 31-40, 5 aged 41-50, and 4 aged 51-65. Two participants went through the recordings twice, resulting in 48 recorded sessions in total. None of the participants reported a prior diagnosis of movement disorders, vestibular dysfunction, or epilepsy. Experience with virtual reality (VR) technology ranged from beginner to advanced, with most participants reporting minimal prior experience.

EEG recordings in humans

We integrated a wireless EEG recording system with a VR headset. The EEG recording system was composed of a wet electrode cap and a Cyton biosensing board from OpenBCI. This allowed us to record 8 EEG channels (45 sessions: FP1, FP2, C3, C4, P3, P4, O1 and O2; 3 sessions: FP1, FP2, T3, T4, T5, T6, O1 and O2) at a 250 Hz sampling rate. Electrode labels follow the international 10-20 system: FP = frontopolar, C = central, P = parietal, O = occipital, and T = temporal; odd numbers indicate the left hemisphere, and even numbers the right (Figure 3A). All EEG data were wirelessly (via Bluetooth) transmitted to a nearby computer. The VR headset was a Meta Quest 3, with a virtual environment developed in the Unity engine (Unity Technologies). Virtual 3D objects were designed in Fusion 360 (Autodesk). Synchronization between EEG recording and VR headset was performed by connecting an auditory output of the VR headset to the Cyton board to exchange synchronization triggers. This allowed us to synchronize EEG data with VR events offline. Auxiliary signals, including the participant’s position, stimulus trigger timing, and type, were recorded directly on the headset at a sampling rate of approximately 100 Hz.

Visual responses

Visual stimuli were presented using the VR headset. We used a reversing square checkerboard stimulus to drive visually evoked potentials. The checkerboard reversed colors at random intervals (between 2 and 4 s). In virtual space, the checkerboard measured 0.5 by 0.5 m and was positioned 0.5 m in front of participants’ eyes. This resulted in a horizontal and vertical coverage by the checkerboard of approximately 53° of visual angle. Each visual stimulation session lasted 4 min, during which the checkerboard reversed colors between 78 and 82 times. For half of the session, participants viewed the stimulus while seated; for the other half, they were instructed to walk freely within a 7 by 5 m empty floor space. The order of sitting and walking conditions was randomized across participants.

Visuomotor mismatch responses

For the experiments measuring visuomotor mismatch responses, we used a 3D virtual corridor with vertical gratings on the walls. The corridor measured 1 m in width, 2.4 m in height, and had an oval shape of 7 by 5 m. Visuomotor mismatches were introduced at random intervals every 10 to 15 s as participants walked through the corridor. The session lasted 5 min and included between 22 and 26 visuomotor mismatch events. During these events, the coupling between the participant’s movement and the visual feedback in the VR headset was briefly interrupted, i.e. the visual scene was frozen for 0.5 s. Because participants continued to move during this time, they were teleported to their current position in the virtual space following visuomotor mismatch event.

Playback halt responses

To quantify how much of the visuomotor mismatch response could be explained by visual input alone, participants were asked to passively observe a replay of the visual flow they had self-generated during the preceding closed loop session. We quickly learned, however, that watching 5 minutes of playback in the VR headset triggered nausea in most participants. Thus, we started experimenting with changes to the playback to minimize the risk of triggering nausea. One such modification we settled on to use for experiments was a playback version that omitted pitch and roll movements of the head. Full playback involved six degrees of freedom (6DOF): 3D position in space, plus pitch, yaw, and roll angles of the head. The constrained playback included only four degrees of freedom (4DOF): the 3D position in space, but only yaw movements of the head. A subset of participants volunteered to view the full playback (Figure S3). Playback sessions were conducted while participants were seated and lasted 5 min, matching the duration of the closed loop session. Approximately 23% of participants reported strong nausea during the beginning of the 4DOF playback session, at which point the recordings were terminated.

Mismatch negativity

To measure mismatch negativity we used an auditory oddball paradigm. Auditory stimuli consisted of two pure tones: 1 kHz and 1.2 kHz. They were presented at 60 to 70 dB sound pressure level in a counterbalanced design, in which each tone served as the standard in one condition and as the deviant in the other. Tones were 50 ms in duration, including 5 ms linear rise and fall ramps. Stimuli were delivered in a pseudorandom order, with deviant tones comprising 15% of all trials and preceded by at least five standard tones. The inter-trial interval was selected from a uniform distribution of between 500 and 600 ms. Participants were seated and listened to the tones while watching a silent movie that was not related to the tone sequences.

EEG Signal Analysis

Data analysis was done using custom-written MATLAB scripts. EEG signals were band-pass filtered between 0.2 and 100 Hz. To remove power line noise, a band-stop filter was applied between 49 and 51 Hz. Movement of the participants triggered all varieties of movement related artifacts in the EEG recordings. The strength of these artifacts depended on a variety of factors: impedance of the electrodes, hair style of participants, gait pattern, and likely others. To reduce data contaminated by excessive movement artifacts, trials with a maximum absolute response amplitude exceeding 100 μV were discarded from further analysis (Figure S5). Data from each electrode were included in the final analysis only if at least 15 triggers remained after exclusion of triggers with excessive movement artifacts (Table S1). To compare average response strength across conditions, a 100 ms analysis window was used, centered on the peak of the respective responses: the visuomotor mismatch event recorded at occipital electrodes O1 and O2 (Figures 2E, 2F, and S4), the mean visuomotor mismatch response across all electrodes (Figure 3C), the mean visual response across all electrodes (Figure S2C) or playback halt response in the 6DOF condition (Figure S3B). Signal power was compared by calculating the mean squared amplitude within a 0 - 0.5 s analysis window following stimulus onset (Figures 4B and 5E).

Statistical tests

All statistical analyses were conducted using hierarchical bootstrap (Saravanan et al., 2020). Bootstrap resampling enables statistical comparisons across conditions without assuming a specific distribution of the EEG data. For analysis in Figures 1, 2, 4, 5, S3, and S4, we averaged signals from electrodes O1 and O2, and treated the result as a single data point per participant. For the analysis shown in Figures 3C, 3E, and S2C we included signals from all electrodes and used a nested bootstrap to account for multiple data points originating from the same participant. We first resampled the data with replacement at the level of participants, followed by resampling at the level of electrodes. For each resampled population, we computed the mean response and repeated this procedure 10000 times. The p value was estimated as the fraction of bootstrap samples in which the sample mean violated the tested hypothesis. See Table S1 for all information on number of participants or electrodes used for all analyses shown.

Acknowledgements

We thank all participants for taking part in the study. We thank all members of the Keller lab for discussion and support. We thank Philipp Sterzer for feedback on the manuscript. This project has received funding from the Swiss National Science Foundation (GBK), the Novartis Research Foundation (GBK), and the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 865617) (GBK).

Additional information

Funding

Swiss National Science Foundation

Novartis Foundation

European Research Council

https://doi.org/10.3030/865617