1. Neuroscience
Download icon

Frontal cortex selects representations of the talker’s mouth to aid in speech perception

  1. Muge Ozker  Is a corresponding author
  2. Daniel Yoshor
  3. Michael S Beauchamp  Is a corresponding author
  1. Baylor College of Medicine, United States
  2. Michael E. DeBakey Veterans Affairs Medical Center, United States
Short Report
Cite this article as: eLife 2018;7:e30387 doi: 10.7554/eLife.30387
5 figures, 6 tables and 1 additional file

Figures

Visual cortex responses to speech.

(A) The speech stimuli consisted of audiovisual recordings of a female talker speaking words (AV) edited so that only the visual portion of the recording was presented (Vis) or only the auditory portion of the recording was presented (Aud). Subjects were instructed to fixate the talker’s mouth (AV and Vis conditions) or a fixation point presented at the same screen location as the talker’s mouth (Aud condition). (B) In eight subjects, a total of 73 responsive occipital electrodes were identified. Electrode locations (black circles) are shown on a posterior view (top) and a medial view (bottom) of the left hemisphere of a template brain. (C) Broadband responses (70–150 Hz) to AV (solid line), Vis (dashed line) and Aud (dotted line) speech averaged across electrodes. The red arrow indicates the first time point (400 ms) at which the response to Vis speech was significantly greater than the response to AV speech. On the x-axis, time zero corresponds to the onset of the video (V) followed by the onset of the talker’s mouth movements (M) at 200 ms and onset of auditory speech (A) at 283 ms (orange arrows). (D) The broadband response enhancement (Vis – AV) measured from 200 to 1500 ms. One symbol per electrode (symbols jittered along x-axis for improved visibility).

https://doi.org/10.7554/eLife.30387.002
Retinotopic organization of speech responses in the visual cortex.

(A) Medial view of a cortical surface model of the left hemisphere brain of a single subject (anonymized subject ID YAI). Posterior electrode e81 (red circle) was located superior to the calcarine sulcus on the occipital pole (red circle) while anterior electrode e65 (blue circle) was located inferior to the calcarine on the medial wall of the hemisphere. The receptive field mapping stimulus consisted of a small checkerboard presented at random screen locations while subjects performed a letter detection task at fixation (not shown). (B) The responses evoked by the mapping stimulus in electrodes e81 (left panel) and e65 (right panel). Color scales corresponds to the amplitude of the visual evoked response at each location in the visual field, with the crosshairs showing the center of the visual field and the red and blue circles showing the center of a two-dimensional Gaussian fitted to the response. Electrode e81 had a central receptive field (eccentricity at RF center of 2.5°) while electrode e65 had a peripheral receptive field (eccentricity 10.9°). (C) The receptive field location for the two sample electrodes, e81 (red circle) and e65 (blue circle) are shown on the speech stimulus. Subjects were instructed to fixate the talker’s mouth and electrodes were classified as representing the mouth region of the talker’s face (red circles; less than 5° from the center of the mouth, white dashed line) or as non-mouth (blue circles;>5°). Electrode locations of mouth (red) and non-mouth (blue) electrodes shown on posterior and medial brain views. Response enhancement (Vis-AV) for each individual electrode; inset bar graph shows mean values (±standard error). (D) In a control speech experiment, a white fixation crosshair was presented on the talker’s shoulder, moving the mouth of the talker’s face to the visual periphery. Electrodes were classified as representing the mouth of the talker (red circles; less than 5° from the the center of the mouth, white dashed line) or non-mouth (blue circles;>5°). Mouth electrodes were in the periphery of the visual field (mean eccentricity of 9.5°). Response enhancement (Vis-AV) for each individual electrode; inset bar graph shows mean values (±standard error).

https://doi.org/10.7554/eLife.30387.003
Functional connectivity with the frontal cortex.

(A) Average broadband responses for each Vis speech trial (200–1500 ms, 70–150 Hz) measured simultaneously in a single frontal (green) and a single visual cortex (red) electrode. Trials were ranked based on their response amplitudes (y axis) and shown with respect to their presentation orders (x axis). The Spearman rank correlation between the amplitude time series of the two electrodes was high (ρ = 0.57). (B) Left hemisphere of a template brain showing all frontal (green circles), visual cortex mouth (red) and non-mouth (blue) electrodes. Frontal electrodes were located near the precentral sulcus (dashed line). Plot shows Spearman rank correlation between each frontal-visual electrode pair; inset bar graph shows mean values (±standard error).

https://doi.org/10.7554/eLife.30387.006
Frontal cortex responses.

(A) Broadband responses to AV (solid line), Vis (dashed line) and Aud (dotted line) speech averaged across frontal electrodes (locations shown on inset brain). (B) Scatter plot showing average responses (from 200 to 1500 ms) to Vis and AV speech for all frontal electrodes (green circles). Black diagonal line indicates the line of equality. (C) Broadband responses to AV (solid line) and Vis (dashed line) speech in a single frontal (green trace) and visual cortex (red trace) electrode. Dashed black lines depicts the onset of mouth movements and the onset of auditory speech (note expanded scale on x-axis compared with A). Vertical green line indicates the first time point at which the frontal response to Vis speech was enhanced relative to AV speech. Red line indicates the same for visual cortex. Black arrow highlights latency difference. (D) Enhancement latency for all frontal (green circles) and visual (red circles) electrodes. Letter codes along y-axis show different subjects. Vertical green and red lines indicate the average enhancement latency across all frontal and visual electrodes respectively.

https://doi.org/10.7554/eLife.30387.008
Auditory cortex responses and connectivity.

(A) Broadband responses to AV (solid line), Vis (dashed line) and Aud (dotted line) speech across auditory electrodes located on the STG (purple circles on inset brain). (B) The broadband response enhancement (Aud – AV) with one symbol per electrode (symbols jittered along x-axis for improved visibility). Electrodes were divided into two groups, those showing a large (>=10%) enhancement (orange circles) and those showing little or no enhancement (yellow circles). (C) Anatomical distribution of STG electrodes by Aud – AV amplitude. (D) Frontal connectivity of STG electrodes by Aud – AV amplitude.

https://doi.org/10.7554/eLife.30387.009

Tables

Table 1
LME for Amplitude in Visual Cortex
https://doi.org/10.7554/eLife.30387.004
Fixed effects:EstimateStd. errorDFt-valuep-value
Baseline988.790.511.210−16
A Speech−835.9134.5−14.210−16
A Speech x Peripheral RF479.7133.54.910−6
V Speech265.4132.94.810−6
Peripheral RF−6515.290.5−4.310−5
V Speech x Peripheral RF−249.4132.9−2.60.01
Table 2
LME for Amplitude in Visual Cortex (Control Experiment)
https://doi.org/10.7554/eLife.30387.005
Fixed effects:EstimateStd. errorDFt-valuep-value
Baseline3573532.710.110−11
V Speech507326.710−7
Non-mouth RF−2875532.7-510−5
V Speech x Non-mouth RF−411232−3.510−3
Table 3
LME for Frontal-Visual Cortex Connectivity
https://doi.org/10.7554/eLife.30387.007
Fixed effects:EstimateStd. errorDFt-valuep-value
Baseline0.230.03179.17.410−12
Peripheral RF−0.240.05179.1−4.510−5
A Speech−0.10.04141.1−2.50.01
A Speech x Peripheral RF0.160.07136.52.40.02
V Speech x Peripheral RF−0.10.06134−1.60.1
V Speech0.00030.04134−0.0071
Table 4
LME for Amplitude in Auditory Cortex
https://doi.org/10.7554/eLife.30387.010
Fixed effects:EstimateStd. errorDFt-valuep-value
Baseline414.988.38.410−12
V Speech−32583.2−6.410−8
A Speech15.184.10.20.8
Table 5
LME for Frontal-Auditory Cortex Connectivity
https://doi.org/10.7554/eLife.30387.011
Fixed effects:EstimateStd. errorDFt-valuep-value
Baseline0.040.0483.61.10.3
V Speech−0.10.0483.6−2.30.02
A Speech0.020.0484.40.40.7
Table 6
LME for Amplitude in Frontal Cortex
https://doi.org/10.7554/eLife.30387.012
Fixed effects:EstimateStd. errorDFt-valuep-value
Baseline336.712.1510−4
V Speech205.214.13.910−3
A Speech-75.814.4−1.10.3

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)