Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement
- School of Psychology, University of Sydney, Australia
- Queensland Brain Institute, The University of Queensland, Australia
- School of Psychology, The University of Queensland, Australia
Figures
Figure 1
Behavioural performance is improved for audiovisual stimuli.
(A) Average accuracy of responses across participants in the behavioural session at each stimulus location for each stimulus condition, fitted to a psychometric curve. Steeper curves indicate greater sensitivity in identifying stimulus location. (B) Average sensitivity across participants in the behavioural task, estimated from psychometric curves, for each stimulus condition. The red cross indicates estimated performance assuming optimal (maximum likelihood estimation [MLE]) integration of unisensory cues. (C) Average behavioural sensitivity across participants in the EEG session for each stimulus condition. Error bars indicate ±1 SEM, n = 41.
Figure 2 with 1 supplement
Audiovisual event-related potentials (ERPs) follow an additive principle.
Average ERP amplitude for each modality condition. Five plots represent the different stimulus locations, as indicated by the grey inset, and the final plot (bottom-right) shows the difference between the summed auditory and visual responses and the audiovisual response. Shaded error bars indicate ±1 SEM, n = 41. Orange horizontal bars indicate cluster corrected periods of significant difference between visual and audiovisual ERP amplitudes.
Figure 2—figure supplement 1
Correlations reveal a positive relationship between eye position and stimulus position in all conditions.
Correlations (Spearman’s rho) between average eye position and stimulus position. Shaded error bars indicate ±1 SEM, n = 41. Coloured horizontal bars indicate cluster corrected periods that are significantly different from chance (0). Cluster corrected analysis revealed no significant differences between the conditions. If consistent saccades to audiovisual stimuli were responsible for the non-linear decoding shown in Figure 5, we would expect a greater correlation between horizontal eye position and stimulus location for the audiovisual condition compared with the unisensory conditions.
Figure 3
Spatiotemporal representation of audiovisual location.
(A) Accuracy of locations decoded from neural responses for each stimulus condition. Shaded error bars indicate ±1 SEM, n = 41. Coloured horizontal bars indicate cluster corrected periods that showed a significant difference from chance (0). (B) Topographic decoding performance in each condition during critical period (grey inset in (A)).
Figure 4 with 2 supplements
Super-additive multisensory interaction in multivariate patterns of electroencephalography (EEG) activity.
(A) Decoding sensitivity in each stimulus condition across the epoch. Overall trends closely matched decoding accuracy. (B) Predicted (optimal sensitivity through maximum likelihood estimation [MLE] and aggregate A+V) and actual audiovisual sensitivity across the epoch. Coloured horizontal bars indicate cluster corrected periods where actual sensitivity significantly exceeded that which was predicted. Shaded error bars indicate ±1 SEM, n = 41.
Figure 4—figure supplement 1
Decoding sensitivity from frontal electrodes.
Predicted (optimal sensitivity through maximum likelihood estimation [MLE] and aggregate A+V) and actual audiovisual decoding sensitivity for decoding stimulus locations from frontal neural activity. Shaded error bars indicated ±1 SEM, n = 41. Sensitivity is markedly lower than that from posterior neural activity (Figure 5), suggesting eye movements are not a primary contributor to non-linear multisensory enhancements.
Figure 4—figure supplement 2
Channel activity.
To represent the channel activity of the forward model, we reconstructed the tuning curves from each of the five channel responses for each location (±[0, 7.5, 15]°). They were computed by taking the dot product of the channel responses and the forward model. Vertical dashed lines indicate the stimulus positions. Channel responses are colour-coded, and shaded regions indicate ±1 SEM.
Figure 5
Audiovisual decoding sensitivity is significantly positively correlated with behavioural sensitivity.
Correlations (Spearman’s rho) are shown between decoding and behavioural sensitivity from the electroencephalography (EEG) session (150–250 ms post-stimulus onset) for each stimulus condition, with a line of best fit.
Figure 6
Experimental design of behavioural and electroencephalography (EEG) sessions.
(A) An example trial for the audiovisual condition in the behavioural session. Each trial consisted of a (centred) reference stimulus and a target stimulus presented at one of eight locations along the horizontal meridian of the display. (B) An example trial for the audiovisual condition in the EEG session. The top row displays the possible locations of stimuli. In each trial, participants were presented with 20 stimuli that were each spatially localised to one of five possible locations along the horizontal meridian. The task was to determine if there were more stimuli presented to the left or right of fixation.
Author response image 1
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement
eLife 13:RP97230.
https://doi.org/10.7554/eLife.97230.3
