1. Neuroscience

A tradeoff between acoustic and linguistic feature encoding in spoken language comprehension

  1. Filiz Tezcan  Is a corresponding author
  2. Hugo Weissbart
  3. Andrea E Martin
  1. Language and Computation in Neural Systems Group, Max Planck Institute for Psycholinguistics, Netherlands
  2. Donders Centre for Cognitive Neuroimaging, Radboud University, Netherlands
https://doi.org/10.7554/eLife.82386
Share this article
Cite this article
  1. Filiz Tezcan
  2. Hugo Weissbart
  3. Andrea E Martin
(2023)
A tradeoff between acoustic and linguistic feature encoding in spoken language comprehension
eLife 12:e82386.
https://doi.org/10.7554/eLife.82386
  2. Download BibTeX
  3. Download .RIS
7 figures, 19 tables and 1 additional file

Figures

Figure 1
Download asset Open asset
Schematic of TRF models and features used in models.

(A) All speech features are divided into high and low word entropy conditions (B) TRFs for each brain source are generated by the linear regression model that estimates the source localized MEG signal from speech features, then they are averaged over sources.

Figure 2
Download asset Open asset
Model accuracy comparison between Dutch and French stories (n=24).

(A) Accuracy improvement (averaged over the sources in whole brain) by each feature for Dutch Stories (B) Accuracy improvement (averaged over the sources in whole brain) by each feature for French Stories. Braces in Figure A and B shows the significance values of the contrasts (difference between consecutive models, ****<0.0001, ***<0.001, **<0.01, *<0.05) in linear mixed effect models (Tables 2 and 3). Error bars show within subject standard errors. (C) Source points where accuracies of base acoustic model were significantly different than 0 (D) Source points where reconstruction accuracies of the model were significantly different than previous model. Accuracy values shows how much each linguistic feature increased the reconstruction accuracy compared to the previous model.

Figure 3 with 1 supplement
Download asset Open asset
First 4 Dutch Story Parts (n=24).

Light orange and light green represent Low Word Entropy condition, dark orange and dark green represent High Word Entropy condition for Dutch and French stories, respectively. (A) Reconstruction accuracy interaction between word entropy and language for acoustic features (B) Reconstruction accuracy interaction between word entropy and language for phoneme features (Braces in Figure A and B indicate the significant different between high and low entropy word conditions, ****<0.0001, ***<0.001, **<0.01, *<0.05. Error bars shows within subject standard errors.) (C-D) Acoustic Edge TRFs on left hemisphere (LH) and right hemisphere (RH) (E-F) Phoneme Features TRFs on LH and RH. Lines on the graphs in Figure C-F show the mean and shaded areas show the standard error of the mean. (G) Sources where the main effect of Language and Word Entropy, and interaction are found.

Figure 3—figure supplement 1
Download asset Open asset
First 4 Dutch Stories (n=24).

(A) Phoneme Onset TRF (B) Phoneme Surprisal TRF (C) Phoneme Entropy TRF Lines on the graphs in Figure A-C show the mean and shaded areas show the standard error of the mean. (D) Sources where the main effect of Language and Word Entropy, and interaction are found.

Figure 4 with 1 supplement
Download asset Open asset
Second 4 Dutch Story Parts (n=24).

Light orange and light green represent Low Word Entropy condition, dark orange and dark green represent High Word Entropy condition for Dutch and French stories, respectively. (A) Reconstruction accuracy interaction between word entropy and language for acoustic features. (B) Reconstruction accuracy interaction between word entropy and language for phoneme features. (Braces in Figure A and B indicate the significant different between high and low entropy word conditions, ****<0.0001, ***<0.001, **<0.01, *<0.05. Error bars shows within subject standard errors.) (C-D) Acoustic Edge TRFs on left hemisphere (LH) and right hemisphere (RH). (E-F) Phoneme Features TRFs on LH and RH. Lines on the graphs in Figure C-F show the mean and shaded areas show the standard error of the mean. (G) Sources where the main effect of Language and Word Entropy, and interaction are found.

Figure 4—figure supplement 1
Download asset Open asset
Next 4 Dutch Stories (n=24).

(A) Phoneme Onset TRF (B) Phoneme Surprisal TRF (C) Phoneme Entropy TRF Lines on the graphs in Figure A-C show the mean and shaded areas show the standard error of the mean.(D) Sources where the main effect of Language and Word Entropy, and interaction are found.

Figure 5
Download asset Open asset
Accuracy improvement by each linguistic feature calculated by subtracting the model accuracy of previous model from the model which also has the feature of interest for each time window in High and Low Word Entropy Conditions (n=24).

(A) Phoneme Onset (Left Hemisphere – LH on the left, Right Hemisphere RH on the right). (B) Phoneme Surprisal. (C) Phoneme Entropy. Lines on the graphs in Figure A-C show the mean and shaded areas show the standard error of the mean.

Figure 6
Download asset Open asset
Accuracy improvement by each linguistic feature calculated by subtracting the model accuracy of previous model from the model which also has the feature of interest for each time window (n=24).

High (Left) and Low (Right) Word Entropy Conditions for French (Below) and Dutch (Top) Stories.

Author response image 1
Download asset Open asset
Full Model.

Light orange and light green represent Low Word Entropy condition, dark orange and dark green represent High Word Entropy condition for Dutch and French stories, respectively. (Braces indicate the significant different between high and low entropy word conditions, **** <0.0001, *** <0.001, ** <0.01, * < 0.05 ) (A) Reconstruction accuracy interaction between word entropy and language for acoustic features (B) Reconstruction accuracy interaction between word entropy and language for phoneme features (C-D) Acoustic Edge TRFs on LH and RH (E-F) Phoneme Features TRFs on LH and RH (G) Sources where the main effect of Language and Word Entropy, and interaction are found.

Tables

Table 1
LMM results of reconstruction accuracies for Dutch and French stories.
EstimateStd. Errort valuePr(>|t|)
(Intercept)2.86E-032.38E-0412.002.16E-11***
Language (French - Dutch)–7.70E-041.57E-04–4.915.64E-05***
Phon. Onset – Acoustic5.50E-051.14E-054.823.04E-06***
Phon. Surprisal – Phon. Onset7.90E-051.14E-056.917.51E-11***
Phon. Entropy – Phon. Surprisal1.33E-041.14E-0511.632.00E-16***
Word Frequency – Phon. Entropy1.48E-041.14E-0512.952.00E-16***
Language: Phon. Onset – Acoustic–3.64E-051.62E-05–2.252.55E-02*
Language: Phon. Surprisal – Phon. Onset–5.97E-051.62E-05–3.702.88E-04***
Language: Phon. Entropy – Phon. Surprisal–1.04E-041.62E-05–6.459.83E-10***
Language:Word Frequency – Phon. Entropy–1.30E-041.62E-05–8.079.12E-14***

  1. ****<0.0001, ***<0.001, **<0.01, *<0.05.

Table 2
LMM results of reconstruction accuracies for Dutch stories.
EstimateStd. ErrorDft valuePr(>|t|)
(Intercept)2.86E-032.38E-0423.111.992.14E-11***
Phoneme Onset – Acoustic5.50E-051.49E-0592.03.703.63E-04***
Phoneme Surprisal – Phoneme Onset7.90E-051.49E-0592.05.327.41E-07***
Phoneme Entropy – Phoneme Surprisal1.33E-041.49E-0592.08.943.82E-14***
Word Frequency - Phoneme Entropy1.48E-041.49E-0592.09.962.84E-16***
Table 3
LMM results of reconstruction accuracies for French stories.
EstimateStd. ErrorDft valuePr(>|t|)
(Intercept)2.09E-032.09E-0423.010.007.54E-10***
Phoneme Onset – Acoustic1.86E-056.34E-0692.02.944.15E-03**
Phoneme Surprisal – Phoneme Onset1.93E-056.34E-0692.03.043.09E-03**
Phoneme Entropy – Phoneme Surprisal2.88E-056.34E-0692.04.541.70E-05***
Word Frequency - Phoneme Entropy1.76E-056.34E-0692.02.786.54E-03**

  1. ****<0.0001, ***<0.001, **<0.01, *<0.05.

Table 4
LMM results of accuracy improvement by linguistic features for Dutch stories.
EstimateStd. Errordft valuePr(>|t|)
(Intercept)4.22E-056.83E-0631.586.186.83E-07***
hemisphere_right–1.06E-055.26E-06167.00–2.024.55E-02*

  1. ****<0.0001, ***<0.001, **<0.01, *<0.05.

Table 5
LMM results of accuracy improvement by linguistic features for French stories.
EstimateStd. Errordft valuePr(>|t|)
(Intercept)5.16E-063.05E-0641.241.699.80E-02.
hemisphere_right–1.52E-063.10E-06167.00–0.496.26E-01

  1. ****<0.0001, ***<0.001, **<0.01, *<0.05.

Table 6
LMM results of reconstruction accuracies for Phoneme features.
EstimateStd. Errordft valuePr(>|t|)
(Intercept)8.68E-048.17E-0526.410.625.11E-11***
French–8.49E-046.75E-0535.1–12.581.46E-14***
Low Word Entropy–2.69E-044.21E-05118.0–6.383.65E-09***
Right Hemisphere1.22E-049.96E-0523.01.232.33E-01
French: Low Word Entropy3.74E-045.96E-05118.06.295.69E-09***
Table 7
LMM results of reconstruction accuracies for Acoustic Edges.
EstimateStd. Errordft valuePr(>|t|)
(Intercept)6.83E-051.66E-0532.84.112.45E-04***
French8.85E-051.10E-05141.08.033.48E-13***
Low Word Entropy–6.08E-051.10E-05141.0–5.521.61E-07***
Right Hemisphere4.11E-051.94E-0523.02.124.49E-02*
French: Low Word Entropy–9.54E-051.56E-05141.0–6.128.76E-09***

  1. ****<0.0001, ***<0.001, **<0.01, *<0.05.

Table 8
LMM results of reconstruction accuracies for Phoneme Features (Next 4 Dutch Story Parts).
EstimateStd. ErrorDft valuePr(>|t|)
(Intercept)8.14E-057.03E-0639.111.573.36E-14***
French–8.00E-055.62E-06141.0–14.232.00E-16***
Low Word Entropy–5.00E-055.62E-06141.0–8.892.61E-15***
Right Hemisphere–5.32E-067.61E-0623.0–0.704.91E-01
French: Low Word Entropy3.67E-057.95E-06141.04.618.87E-06***
Table 9
LMM results of reconstruction accuracies for Acoustic Edges (Next 4 Dutch Story Parts).
EstimateStd. Errordft valuePr(>|t|)
(Intercept)8.38E-051.70E-0532.04.932.44E-05***
French7.70E-051.09E-05141.07.067.15E-11***
Low Word Entropy–8.18E-051.09E-05141.0–7.496.76E-12***
Right Hemisphere3.30E-051.92E-0523.01.729.89E-02
French: Low Word Entropy–7.44E-051.54E-05141.0–4.823.66E-06***

  1. ****<0.0001, ***<0.001, **<0.01, *<0.05.

Table 10
Stimuli.
Story PartLanguageDurationSpeakerParts used in analysis
Anderson_S01_P01NL4 min 58 sWoman 1Dutch Part 1
Anderson_S01_P02NL5 min 17 sWoman 1Dutch Part 1
Anderson_S01_P03NL4 min 49 sWoman 1Dutch Part 1
Anderson_S01_P04NL5 min 50 sWoman 1Dutch Part 1
Grimm_23_1NL5 min 3 sWoman 2Dutch Part 2
Grimm_23_2NL5 min 32 sWoman 2Dutch Part 2
Grimm_23_3NL5 min 2 sWoman 2Dutch Part 2
Grimm_20_1NL6 min 6 sWoman 2Dutch Part 2
ANGE_part1FR4 min 34 sWoman 3French Part 1
BALL_part1FR4 min 58 sWoman 3French Part 1
EAUV_part1FR5 min 43 sMan 1French Part 1
EAUV_part2FR6 min 1 sMan 1French Part 1
Table 11
Model names and speech features in models.
SpectrogramAcoustic EdgePhoneme OnsetPhoneme SurprisalPhoneme EntropyWord Frequency
Acoustic
Phoneme Onset
Phoneme Surprisal
Phoneme Entropy
Word Frequency
Author response table 1
LMM results of reconstruction accuracies for Phoneme Features.
EstimateStd. Errordft valuePr(>|t|)
(Intercept)3.29E-055.45E-0643.076.033.34E-07****
French-2.48E-054.06E-06164.00-6.126.75E-09****
Low Word Entropy-2.00E-054.06E-06164.00-4.922.10E-06****
Right Hemisphere-3.73E-062.87E-06164.00-1.301.95E-01
French: Low Word Entropy1.09E-055.74E-06164.001.905.96E-02.
Author response table 2
LMM results of reconstruction accuracies for Acoustic Edges.
EstimateStd. Errordft valuePr(>|t|)
(Intercept)4.01E-043.82E-0532.4110.505.85E-12****
French1.48E-052.16E-05140.000.694.93E-01
Low Word Entropy-5.78E-062.64E-05140.00-0.228.27E-01
Right Hemisphere9.76E-054.96E-0528.021.975.91E-02.
French: Low Word Entropy-5.88E-053.05E-05140.00-1.935.60E-02.
Author response table 3
LMM results of reconstruction accuracies for Dutch (first 2 parts) and French (first 2 parts) stories.
EstimateStd. Errort valuePr(>|t|)
(Intercept)2.92E-032.72E-0410.721.99E-10***
Language (French – Dutch)-6.44E-041.52E-04-4.233.13E-04***
Phon. Onset – Acoustic5.57E-051.23E-054.521.13E-05***
Phon. Surprisal – Phon. Onset8.10E-051.23E-056.575.04E-10***
Phon. Entropy – Phon. Surprisal1.25E-041.23E-0510.102.00E-16***
Word Frequency – Phon. Entropy1.22E-041.23E-059.882.00E-16***
Language: Phon. Onset – Acoustic-4.17E-051.74E-05-2.391.80E-02*
Language: Phon. Surprisal – Phon. Onset-6.73E-051.74E-05-3.861.58E-04***
Language: Phon. Entropy – Phon. Surprisal-1.06E-041.74E-05-6.096.30E-09***
Language:Word Frequency – Phon. Entropy-1.27E-041.74E-05-7.298.80E-12***
Author response table 4
LMM results of reconstruction accuracies for Dutch (first 2 parts) and French (second 2 parts) stories.
EstimateStd. Errort valuePr(>|t|)
(Intercept)2.92E-032.72E-0410.722.00E-10***
Language (French- Dutch)-1.09E-031.75E-04-6.202.45E-06***
Phon. Onset – Acoustic5.57E-051.18E-054.714.79E-06***
Phon. Surprisal – Phon. Onset8.10E-051.18E-056.861.03E-10***
Phon. Entropy – Phon. Surprisal1.25E-041.18E-0510.552.00E-16***
Word Frequency – Phon. Entropy1.22E-041.18E-0510.322.00E-16***
Language: Phon. Onset – Acoustic-4.38E-051.67E-05-2.629.49E-03**
Language: Phon. Surprisal – Phon. Onset-7.34E-051.67E-05-4.401.87E-05***
Language: Phon. Entropy – Phon. Surprisal-1.17E-041.67E-05-7.024.25E-11***
Language:Word Frequency – Phon. Entropy-1.39E-041.67E-05-8.321.91E-14***
Author response table 5
LMM results of reconstruction accuracies for Phoneme Features (First 2 parts of Dutch stories and first 2 parts of French stories).
EstimateStd. Errordft valuePr(>|t|)
(Intercept)4.92E-057.45E-0656.116.601.56E-08***
French-5.92E-056.46E-06164.00-9.172.00E-16***
Low Word Entropy-4.89E-056.46E-06164.00-7.572.62E-12***
Right Hemisphere-3.68E-064.57E-06164.00-0.814.21E-01
French: Low Word Entropy4.13E-059.13E-06164.004.521.19E-05***
Author response table 6
LMM results of reconstruction accuracies for Acoustic Edges (First 2 parts of Dutch stories and first 2 parts of French stories).
EstimateStd. Errordft valuePr(>|t|)
(Intercept)1.45E-043.40E-0545.164.271.00E-04***
French-2.16E-062.61E-05164.00-0.089.34E-01
Low Word Entropy3.58E-052.61E-05164.001.371.72E-01
Right Hemisphere6.05E-051.85E-05164.003.281.27E-03**
French: Low Word Entropy-1.20E-043.69E-05164.00-3.241.43E-03**
Author response table 7
LMM results of reconstruction accuracies for Phoneme Features (First 2 parts of Dutch stories and second 2 parts of French stories).
EstimateStd. Errordft valuePr(>|t|)
(Intercept)4.97E-056.73E-0669.147.392.55E-10***
French-6.57E-056.37E-06164.00-10.312.00E-16***
Low Word Entropy-4.89E-056.37E-06164.00-7.671.45E-12***
Right Hemisphere-4.77E-064.51E-06164.00-1.062.91E-01
French: Low Word Entropy3.78E-059.01E-06164.004.204.38E-05***
Author response table 8
LMM results of reconstruction accuracies for Acoustic Edges (First 2 parts of Dutch stories and second 2 parts of French stories).
EstimateStd. Errordft valuePr(>|t|)
(Intercept)1.41E-043.06E-0547.694.603.11E-05***
French-4.81E-062.43E-05164.00-0.208.44E-01
Low Word Entropy3.58E-052.43E-05164.001.471.43E-01
Right Hemisphere6.87E-051.72E-05164.004.009.69E-05***
French: Low Word Entropy-1.33E-043.44E-05164.00-3.871.55E-04***

Additional files

MDAR checklist
https://cdn.elifesciences.org/articles/82386/elife-82386-mdarchecklist1-v1.docx

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Filiz Tezcan
  2. Hugo Weissbart
  3. Andrea E Martin
(2023)
A tradeoff between acoustic and linguistic feature encoding in spoken language comprehension
eLife 12:e82386.
https://doi.org/10.7554/eLife.82386