EEG responses to visual stimulation are modulated by walking.

(A) Participant wearing the setup for wireless EEG recordings and a virtual reality headset. An Open BCI EEG electrode cap is combined with META QUEST 3. Inset: First person view of the virtual environment. Red fixation dot was enlarged in figure to make it visible. (B) A third person view of the virtual scene used to study visual responses to a reversing checkerboard pattern. The black line on the ground indicates the safety boundary within which the participant was instructed to walk around during the movement block of the paradigm. The grid indicates 1 by 1 m squares. (C) Participants viewed a reversing checkerboard in two sessions: During half of the trials, they remained seated, and during the other half they walked within a defined safety boundary. Checkerboard reversed colors at random intervals every 2 to 4 s. (D) 3D trajectory of an example participant during the visual experiment. Movement was recorded using accelerometers integrated into the VR headset. (E) Visually evoked potentials measured in the sitting session on the occipital electrodes O1 and O2. Responses recorded on other electrodes are presented in Figure S3. Solid black line represents the mean, and shading indicates the SEM across participants. Dashed vertical red line indicates the time of the checkerboard reversal. P1 - positive peak at around 88 ms. (F) As in E, but while participants were walking. Arrow indicates an early negative deflection preceding the P1 component. Inset: Comparison of visual evoked potentials measured in the sitting E and walking F sessions. The horizontal bar above the plot marks time bins in which responses differ significantly (black: p < 0.05) or do not differ (gray: p > 0.05).

Visuomotor mismatch elicits strong EEG responses.

(A) Participant navigating a virtual tunnel. Inset: First person view of the tunnel. (B) Third person view of the virtual environment used to study visuomotor mismatch responses. The grid indicates 1 by 1 m squares. (C) In closed loop sessions, the walking speed of participants was coupled to movement in the virtual corridor. Visuomotor mismatches were introduced by briefly halting the visual flow for 0.5 s. Following each visuomotor mismatch event, the view was updated to the participants’ current position (brief peaks in visual flow speed after the mismatch event) and visuomotor coupling was resumed. (D) Trajectory of an example participant during the visuomotor mismatch paradigm. (E) Responses to visuomotor mismatches recorded from occipital electrodes (O1 and O2). Solid black line represents the mean, and shading indicates the SEM across participants. Dashed vertical red lines are onset and offset of the visuomotor mismatch. The gray shaded areas mark the analysis windows used to quantify response strength in G and H. Analysis windows of 100 ms were centered on the peak of the visuomotor mismatch response and the teleport response. (F) As in E, but for visual flow playback halt responses recorded from occipital electrodes. (G) Comparison of the response strength to visuomotor mismatches and playback halts. Here and elsewhere: Boxes mark median, quartiles, and range of data not considered outliers. Each circle represents data from one participant. ***: p<0.001. See Table S1 for all statistical information. (H) Comparison of the response strength to teleport events following visuomotor mismatches in the closed loop session and playback halts in the open loop session. n.s.: not significant. See Table S1 for all statistical information. (H) Average walking speed of participants during visuomotor mismatches. Inset: Temporally expanded view. The horizontal bar above the plot marks time bins where walking speed differs significantly (black: p < 0.05) or does not differ (gray: p > 0.05) from baseline.

Visuomotor mismatch responses are most prominent over occipital cortex.

(A) Top down view of EEG electrode locations on the head. (B) Visuomotor mismatch responses measured on electrodes shown in A. Solid black lines represent the mean, and shading indicates the SEM across participants. Dashed vertical red lines are onset and offset of the visuomotor mismatch. (C) Comparison of the response strength to visuomotor mismatches measured on electrodes shown in A. Average response strength was calculated within a 100 ms window centered on the peak of the average visuomotor mismatch response across all electrodes. Boxes mark median, quartiles, and range of data not considered outliers. Each circle represents data from an individual participant. ***: p<0.001, **: p<0.01, *: p<0.05, n.s.: not significant. See Table S1 for all statistical information. (D) The responses shown in B, averaged over pairs of electrodes and overlaid. Solid black lines represent mean and shading SEM across electrodes. Dashed vertical red lines are onset and offset of the visuomotor mismatch. (E) Comparison of the latency to half maximum response for the 4 electrode pairs. **: p<0.01, n.s.: not significant. See Table S1 for all statistical information.

Visuomotor mismatch responses show experience-related changes predominantly over frontal electrodes.

(A) Average walking speed during visuomotor mismatch events in early versus late trials. The solid black line shows the mean and the shaded area the SEM across participants. Vertical red dashed lines indicate mismatch onset and offset. The horizontal bar above the plot marks time bins in which the walking speed in early trials differs significantly (black, p < 0.05) or not (gray, p > 0.05) from that in late trials. (B) Visuomotor mismatch responses in early versus late trials over frontal electrodes. (C) As in B, but for occipital electrodes. (D) Comparison of the difference between visuomotor mismatch response in early and late trials for frontal and occipital electrodes. Average response strength was calculated within a 100 ms window centered on the peak of the average visuomotor mismatch response in early trials above frontal or occipital electrodes. Boxes mark median, quartiles, and range of data not considered outliers. *: p<0.05. See Table S1 for all statistical information.

Visuomotor mismatch responses have reversed polarity and more power compared to visual responses.

(A) Comparison of visual (Figure 1E) and visuomotor mismatch (Figure 2E) responses recorded from all electrodes. Solid black line represents the mean, and shading indicates the SEM across participants. Dashed vertical red lines are onset (visual and mismatch) and offset (mismatch) of the stimuli. (B) Comparison of the power of visual and visuomotor mismatch responses, calculated within a 0 - 0.5 s time window following stimulus onset. Boxes mark median, quartiles, and range of data not considered outliers. Each circle represents data from an individual participant. ***: p<0.001, *: p<0.05. See Table S1 for all statistical information.

Visuomotor mismatch responses are larger than auditory oddball mismatch responses but have similar temporal dynamics

(A) Design of the auditory oddball paradigm with examples of the silent films participants were exposed to. (B) Top: Auditory responses to the 1 kHz tone presented as a standard versus as a deviant. Bottom: Oddball mismatch response calculated by subtracting the average response over trials in which the tone was presented as a standard from those when the tone was presented as a deviant. Solid black lines represent the mean, and shading indicates the SEM across participants. Dashed vertical red line is the onset of the auditory stimulus. (C) As in B, but for the 1.2 kHz tone. (D) Comparison of visuomotor mismatch, oddball mismatch (average over data shown in panels B and C) and playback halt responses recorded from occipital electrodes. Solid black lines represent the mean response, with shading indicating SEM across electrodes. Dashed vertical red lines are onset (visual, mismatch) and offset (mismatch) of the stimuli. (E) Comparison of the power of visuomotor mismatch, oddball mismatch response (average over data shown in panels B and C) and playback halt responses, calculated within a 0 s - 0.5 s time window following stimulus onset. Boxes mark median, quartiles, and range of data not considered outliers. Each circle represents data from an individual participant. ***: p<0.001; *p<0.05. See Table S1 for all statistical information.

Movement onsets result in increases in variance in EEG activity.

(A) We included only data in which the EEG signals remained below an exclusion threshold of 100 µV. Most of the movement related variance in the EEG activity is likely a movement artifact. Example of an EEG signal (black line) at movement onset that reached exclusion threshold (100 µV). Overlaid is the walking speed of the participant (green line). (B) As in A, but for an example of an EEG signal at movement onset that did not reach exclusion threshold (100 µV). (C, D) As in A, but for examples of EEG signals at movement onset with minimal movement contamination.

Examples of rejected and valid trials based on maximum signal amplitude in the visuomotor mismatch paradigm.

(A) Example of an EEG response contaminated by an eye blink artifact (arrow). Dashed vertical red lines are onset and offset of the visuomotor mismatch. This trial was removed. (B, C) As in A, but for examples of EEG responses contaminated by walking artifacts. These trials were removed. (D, E) As in A, but for an example of an EEG response that met the inclusion criteria (amplitude < 100 µV). (F) Histogram of maximum trial amplitudes. The red dashed line marks the threshold for inclusion. 53 trials with amplitudes exceeding 1500 µV are not shown on the histogram.

Visual responses are strongest over occipital cortex.

(A) Top down view of EEG electrode locations on the head. (B) Visual evoked responses measured on electrodes shown in A. Solid black lines represent the mean, and shading indicates the SEM across participants. Dashed vertical red line is the onset of the checkerboard inversion. (C) Comparison of the response strength of visual evoked potentials measured on electrodes shown in A. Average response strength was calculated within a 100 ms window centered on the peak of the average visual response across all electrodes. Boxes mark median, quartiles, and range of data not considered outliers. Each circle represents data from an individual participant. ***: p<0.001, **: p<0.01, n.s.: not significant. See Table S1 for all statistical information.

Visuomotor mismatch responses are bigger than playback halt responses even when the coupling is full.

(A) Responses to visuomotor mismatches recorded from occipital electrodes. Solid black line represents the mean, and shading indicates the SEM across participants. Dashed vertical red lines are onset and offset of the visuomotor mismatch. (B) As in A, but showing playback halt responses to full six degrees of freedom (6DOF) visual flow playback, recorded over occipital electrodes (C) Comparison of the response strength to visuomotor mismatches and 6DOF playback halts. Average response strength was calculated within a 100 ms analysis window centered on the peak of the visuomotor mismatch response. Boxes mark median, quartiles, and range of data not considered outliers. Each data point corresponds to one participant and lines connect mismatch and playback halt responses from the same participant. **: p<0.01. See Table S1 for all statistical information. (D) Comparison of 6DOF and 4DOF playback halt responses recorded from occipital electrodes. Solid black lines represent the mean, and shading indicates the SEM across participants. Dashed vertical red lines are onset and offset of the visuomotor mismatch. (E) Comparison of the response strength to 6DOF and 4DOF playback halts. Average response strength was calculated within a 100 ms analysis window centered on the peak of the playback halt response in the 6DOF condition. Each circle represents data from an individual recording session. Boxes mark median, quartiles, and range of data not considered outliers. n.s.: not significant.

Mismatch and playback halt responses obtained from the same participants.

(A) Responses to visuomotor mismatches recorded from occipital electrodes (O1 and O2). Solid black line represents the mean, and shading indicates the SEM across participants. The gray shaded areas mark the analysis windows used to quantify response strength in C. Dashed vertical red lines are onset and offset of the visuomotor mismatch. As in Figure 2E, F, but only including data from participants for which we have both closed and open loop data. (B) As in A, but for visual flow playback halt responses recorded from occipital electrodes. (C) Comparison of the response strength to visuomotor mismatch and playback halts (32 participants 4DOF and 7 participants 6DOF). Average response strength was calculated within a 100 ms analysis window centered on the peak of the visuomotor mismatch response. Boxes mark median, quartiles, and range of data not considered outliers. Each data point corresponds to one participant and lines connect mismatch and playback halt responses from the same participant. ***: p<0.001. See Table S1 for all statistical information.

Adaptation over sessions does not explain the difference between visuomotor mismatch and playback halt responses.

(A) Playback halt responses recorded from occipital electrodes in the reversed sequence experiment where the open loop session was presented first, followed by the closed loop session. The visual flow played back was the self-generated visual flow of the previous participants’ closed loop session. Note, this implies that visual flow statistics are no longer matched for closed and open loop for the same participant. Solid black line represents the mean, and shading indicates the SEM across participants. Dashed vertical red lines are onset and offset of the visual playback halt. (B) As in A, but for visuomotor mismatch responses. (C) Comparison of the response strength to visuomotor mismatches and playback halts in the reverse sequence session. Average response strength was calculated within a 100 ms analysis window centered on the peak of the visuomotor mismatch response. Boxes mark median, quartiles, and range of data not considered outliers. ***: p<0.001. See Table S1 for all statistical information. (D) Playback halt and visuomotor mismatch responses as a function of progress through the session, when open loop session preceded closed loop session. Progress in the paradigm is measured as the percentage of total visuomotor mismatch or playback halt events. (E) As in D, but when closed loop was preceding the open loop session.

Visuomotor mismatch response cannot be explained by eye blinks or changes in eye movement speed.

(A) Average horizontal eye position during visuomotor mismatch. Solid black line represents the mean, and shading indicates the SEM across participants. Vertical red dashed lines indicate mismatch onset and offset. The horizontal bar above the plot marks time bins in which the response differs significantly (black, p < 0.05) or not (gray, p > 0.05) from the baseline. (B) Average vertical eye position during visuomotor mismatch. (C) Average eye-movement speed during visuomotor mismatch. (D) Average probability of eye blinks during visuomotor mismatch. 0 – eyes open, 1 – eyes fully closed. (E) Responses to visuomotor mismatches recorded from occipital electrodes (O1 and O2).

Visuomotor mismatch responses exceed auditory oddball mismatch responses at electrodes T3–T4

(A) Top-down view of EEG electrode positions on the scalp, including the temporal electrodes T3, T4, T5, T6. (B) Auditory responses to the 1 kHz tone presented as a standard versus as a deviant. Bottom: Oddball mismatch response calculated by subtracting the average response over trials in which the tone was presented as a standard from those when the tone was presented as a deviant. Solid black lines represent the mean, and shading indicates the SEM across participants. Dashed vertical red line is the onset of the auditory stimulus. (C) As in B, but for the 1.2 kHz tone. (D) Comparison of visuomotor mismatch and oddball mismatch responses (average over data shown in panels A and B) recorded from temporal electrodes. Solid black lines represent the mean response, with shading indicating SEM across electrodes. Dashed vertical red lines are onset (visual, mismatch) and offset (mismatch) of the stimuli. (E) Comparison of the power of visuomotor mismatch and oddball mismatch response (average over data shown in panels A and B), calculated within a 0 s - 0.5 s time window following stimulus onset. Boxes mark median, quartiles, and range of data not considered outliers. Each circle represents data from an individual participant. ***: p<0.001, *: p<0.05. See Table S1 for all statistical information.

Visuomotor mismatch responses exceed auditory oddball mismatch responses at electrodes T5–T6.

(A) Auditory responses to the 1 kHz tone presented as a standard versus as a deviant. Bottom: Oddball mismatch response calculated by subtracting the average response over trials in which the tone was presented as a standard from those when the tone was presented as a deviant. Solid black lines represent the mean, and shading indicates the SEM across participants. Dashed vertical red line is the onset of the auditory stimulus. (B) As in A, but for the 1.2 kHz tone. (C) Comparison of visuomotor mismatch and oddball mismatch responses (average over data shown in panels A and B) recorded from posterior temporal electrodes. Solid black lines represent the mean response, with shading indicating SEM across electrodes. Dashed vertical red lines are onset (visual, mismatch) and offset (mismatch) of the stimuli. (D) Comparison of the power of visuomotor mismatch and oddball mismatch response (average over data shown in panels A and B), calculated within a 0 s - 0.5 s time window following stimulus onset. Boxes mark median, quartiles, and range of data not considered outliers. Each circle represents data from an individual participant. ***: p<0.001, n.s.: not significant.

Statistics.

We used hierarchical bootstrap (Saravanan et al., 2020) for all comparisons. We recorded a total of 91 sessions. The numbers in the table indicate the subset of these we could include for each analysis. Note, this differs for electrode location and condition. Exclusion reasons were a) recording is too noisy, or b) participant aborted the recording (in the case of open loop session).

Excluded trials by paradigm and session.

Indicated are the number and percentage of excluded trials as: excluded trials/all trials, % of excluded trials, for frontal (FP1-2), central (C3-4), parietal (P3-4), and occipital (O1-2) electrodes. See Figure 3A for electrode positions