When we understand spoken language, we transform a continuous physical signal into structured meaning. In order to achieve this, we likely capitalize on stored, previously-learned linguistic representations and other forms of knowledge. During this process, systematic changes shown by the acoustic signal are categorized by the brain as phonemes, while the similar ordering of consecutive phonemes creates words and, a certain ordering of words creates phrases and sentences. As these transformation occur, the phase of the neural signal is thought to align with temporal landmarks in both acoustic and linguistic dimensions; this phenomenon is referred to as neural tracking (Daube et al., 2019; Luo and Poeppel, 2007; Di Liberto et al., 2015; Brodbeck et al., 2018; Broderick et al., 2018; Daube et al., 2019; Keitel et al., 2018; Donhauser and Baillet, 2020; Gillis et al., 2021; Kaufeld et al., 2020a; Weissbart et al., 2020; Brodbeck et al., 2022; Heilbron et al., 2020; Coopmans et al., 2022; Slaats et al., 2023; Ten Oever et al., 2022a; Zioga et al., 2023). Neural tracking has been proposed as a mechanism to segment the acoustic input into linguistic units such as phoneme, syllables, morphemes (Giraud and Poeppel, 2012; Ghitza, 2013; Ding et al., 2016) by aligning the neural excitability cycles to the incoming input. One way to quantify the degree of neural tracking of speech features is using a linear encoding modelling also known as Temporal Response Functions (TRF). Adding sub-lexical, lexical, and phrase- and sentence-level predictors, either quantified with information theoretic metrics (e.g. surprisal and entropy), or syntactic annotations, to the encoding model improves the reconstruction accuracy of neural responses on distinct sources in the brain (see Brodbeck et al., 2022), and surprisal and entropy predictors have been shown to modulate the neural response at different time intervals (see Donhauser and Baillet, 2020). However, whether acoustic features and higher level linguistic units are encoded with similar veracity during comprehension, and how they impact each other’s encoding is still poorly understood. In this study, we investigated the effect of language comprehension in a first language (Dutch) on the tracking of acoustic and phonemic features, contrasted with a statistically familiar but uncomprehended language (French), during naturalistic story listening. Contrasting a comprehended language with a language that is familiar, but not understood, allows us to separate neural signals related to speech processing vs. those related to speech processing in the service of language comprehension. We asked three questions: (1) whether phoneme-level features contribute to neural encoding even when acoustic contributions are carefully controlled, as a function of language comprehension, (2) whether sentence- and discourse-level constraints on lexical information operationalized as word entropy impacted the encoding of acoustic and phoneme-level features, and (3) whether tracking of acoustic landmarks (viz., acoustic edges) was enhanced or suppressed as a function of comprehension. We found that acoustic features are enhanced when the spoken language stimulus is not understood, but that phonemic features are more strongly encoded when language is understood, consistent with an account where gain modulation, in the form of enhancement or suppression, expresses the behavioral goal of the brain (Martin, 2016; Martin, 2020). We found that both phonemic and acoustic features are more strongly encoded when the context is less constraining, as exemplified by word entropy. Finally, and in contrast with extant arguments in the literature about the pre-activation or enhancement of low-level representations with the contextual information (DeLong et al., 2005; McClelland and Rumelhart, 1981 Rumelhart and McClelland, 1982, Nieuwland et al., 2018; Nieuwland, 2019 ), we found that acoustic-edge processing appears to be suppressed when language comprehension occurs, compared to when it does not, and both acoustic and phonemic features are suppressed when entropy is low within a comprehended language (viz., the context is more constraining and a word is more expected). This pattern of results is consistent with acoustic edges being important for auditory and speech processing, but speaks against their functioning as computational landmarks during language comprehension, the behavioral and neural goal of speech processing.

During comprehension, contextual information from higher level linguistic units such as words, phrases, and sentences likely affects the neural representation and processing of lower level units like phonemes and acoustic features. A wealth of behavioral and neural data from both spoken and visual language comprehension contexts has been leveraged to illustrate the incremental processing of incoming perceptual input within existing sentence and discourse context. During spoken language comprehension, acoustic, phonemic, phonological, and prosodic information is dynamically integrated with morphemic, lexical, semantic, and syntactic information (e.g., Bai et al., 2022; Kaufeld et al., 2020b; Kaufeld et al., 2020c; Marslen-Wilson and Welsh, 1978; Friederici, 2002; Hagoort, 2013; Martin et al., 2017; Oganian et al., 2023; Martin, 2016; Martin, 2020). Models with top-down and bottom-up information flow have been invoked to account for how sensory information could be integrated with internally-generated knowledge or information, expressed in many forms in different models, as statistical priors, distributional knowledge, or as abstract structure (McClelland and Rumelhart, 1981; Rumelhart and McClelland, 1982; Rao and Ballard, 1999; Lee and Mumford, 2003; Friston, 2005; Martin, 2016; Martin and Doumas, 2019; Martin, 2020; Ten Oever and Martin, 2021; Ten Oever et al., 2022a; Ten Oever et al., 2022b). In word recognition, models such as the Interactive Activation Model (McClelland and Rumelhart, 1981) and TRACE (Rumelhart and McClelland, 1982), feedback connections from word-level to phoneme-level enable a faster activation of phonemes in words than phonemes in nonwords by either enhancing the representation of expected phonemes or suppressing the competing phonemes. Even though these models can account for phenomena such as the word-superiority effect, faster detection of a letter on a masked visual image when presented in a word than in an unpronounceable nonword or in isolation (Reicher, 1969; Wheeler, 1970; Mewhort, 1967), they do not incorporate effects of sentence- and discourse- level constraints. It has also been demonstrated that mispronounced or ambiguous phonemes are more likely to be missed by participants during a detection task when the context is more constraining, indicating that top-down processing constraints interact with bottom-up sensory information to reduce the number of possible word candidates as the acoustic input unfolds (Marslen-Wilson and Welsh, 1978; Martin et al., 2017; Martin and Doumas, 2017).

Even though the effects of sentence-level and word-level context on sub-lexical representations were widely studied with behavioral experiments, studies that investigated the neural readout of the perceptual modulation by the context are still very limited for non-degraded or naturally produced spoken stimuli. Martin, 2016; Martin, 2020 suggested that cues from each hierarchical level are weighted according to their reliabilities and integrated simultaneously by gain modulation in the form of selective amplification and inhibition. When the acoustic edges of the speech signal reach a threshold for the activation of a particular phoneme representation, the sensory representations of the acoustic edges are inhibited. On a slower timescale, the resulting representations of phonemes activate items in the lexicon which then, in turn, suppress the phonemic representations once lexical access is achieved. Intelligibility of the speech signal determines the reliability of the acoustic signal, and sentence- and discourse-level constraint determines the reliability of lexical access and structure formation. According to this theoretical model, when context is less constraining, sub-lexical representations are less inhibited compared to in a highly constraining context.

Linear encoding models have made it possible to investigate how certain features of phonemes and words affect the neural response during naturalistic language comprehension. However, one of the important challenges in this modeling method is the difficulty of understanding whether the features tracked by the brain are only acoustic changes or changes in linguistic units as well (Daube et al., 2019). For example, the beginning of a word may coincide with the beginning of a phoneme, that is also a sudden change in the envelope of the acoustic signal. It has also been argued that neural tracking reflects the convolution of evoked responses to the change in acoustic envelope referred to as acoustic edges (Oganian et al., 2023). To overcome this, features that can cause changes in brain signal should be modeled together as multivariate and to isolate the effect of each feature, features should be added to the models in an incremental way (Brodbeck et al., 2021). Most of the studies using the linear encoding models showed the tracking of either the envelope of the acoustic signal or linguistic units at both word and phoneme levels however only a few of them controlled the acoustic features. In this study, we added each phoneme feature on top of the acoustic base model incrementally and compared the reconstruction accuracies only explained by that feature.

Another limitation of linear encoding method is that only the linear relationship between the changes in the features and the amplitude of the brain signal can be observed. While modeling the linear relationship of the conditional probability of phonemes in words with the brain signal, the effect of phoneme features on the brain signal is assumed to be temporally and topologically constant. However, previous studies have shown that the intensity of the sound stimulus, and the constraints introduced on the words by the context of the sentence has an effect on the latency of the brain signal. Although it is not possible to continuously model this dynamic relationship with linear models, stimulus features can be separated according to certain properties and modeled separately (Drennan and Lalor, 2019). Thus, it is possible to model how sentence and discourse context affects the tracking over lower-level features by separating the predictors into high and low constraining context conditions, denoted by differences in word entropy. Figure 1 shows predictors are separated into low and high word entropy conditions.

Figure 1

Download asset Open asset

To this end, we made use of an MEG dataset of native Dutch speakers with no or minimal comprehension of French (i.e. self-report of no comprehension,<1 year education in French, chance performance on comprehension questions) that was recorded while they listened to audiobooks in Dutch and French. We then used the linear encoding modeling method to predict the MEG signal from the acoustic and phoneme-level speech features including the spectrogram, acoustic edges, phoneme onset, phoneme surprisal and phoneme entropy, separately, for words with high and low word entropy. Word entropy was used to quantify the sentence- and discourse-level constraint for each word given the preceding 30 words. Phoneme surprisal quantifies the conditional probability of each phoneme in a word calculated according to the probability distribution over the lexicon weighted by the occurrence frequency of each word. Phoneme entropy reflects how constraining each phoneme is by quantifying the uncertainty about the next phoneme with each unfolding phoneme. If sentence comprehension modulates phoneme-level processing then the reconstruction accuracies should differ in two important contrasts. First, there should be an effect of comprehended language on whether words with high and low entropy affect phoneme processing, and within the comprehended language, high and low entropy words should differently affect phoneme features. For example, when participants are listening to an audiobook in a language they can understand, they can encounter a word like ‘interval’ or ‘internet’ in a high or low constraining sentence. The occurrence frequency of the word ‘internet’ is higher than ‘interval’ and the conditional probability of hearing the phoneme ‘n’ after ‘ɪnˈtɜː(r)’ is also higher than phoneme ‘v’. That means the surprisal of phoneme ‘n’ which is calculated by the negative log probability, is lower. Representations for phonemes might be different when the reliabilities of contextual cues are different. In a sentence such as ‘You can find anything on the internet’ phoneme features might be more important and more informative compared to a sentence like ‘On campus, you can connect to wireless internet’ because in the first sentence it is more difficult to predict which word is coming given the previous words before the word ‘internet’ in the sentence, so participants may rely more on the phoneme-level features.

Our aim in this current study was to investigate the following questions: (1) Can we measure neural tracking of phoneme- and word-level features when acoustic features are properly controlled? (2) Can we demonstrate that products of language comprehension (viz., words in a sentence and story context) modulate the encoding of lower-level linguistic cues (acoustic and phoneme level features) under normal listening conditions? And (3) does the encoding of acoustic information change as a function of comprehension, reflecting perceptual modulation of neural encoding in the service of behavioral goals? In order to address these questions, we chose to compare the neural response between a comprehended first language (Dutch) and an uncomprehended but acoustically familiar one (French). Having a high-familiarity, statistically known control language allows us to model the neural response to acoustic and phoneme-level features and contrast it to a case where there is comprehension versus where there is not. This is stronger control than examining the neural response to an unknown, statistically unfamiliar language, whose modulation of the neural response may mask the degree to which acoustic and phoneme-level processing dominate the neural signal during language comprehension.

In this study, we investigated whether phoneme-level features are tracked over and above acoustic features and acoustic edges, and whether word entropy affects the tracking of acoustic- and phoneme-level features as a function of language comprehension. We used linear encoding models to quantify the degree of neural tracking while native Dutch speakers listened to audiobooks in Dutch and French. Participants were familiar with the prosodic and phonotactic statistical regularities of the French language, but did not comprehend the stories, allowing for the contrast of speech processing during and in the absence of comprehension in a statistically familiar auditory and perceptual context. First, we examined whether phoneme features are tracked by the brain signal even when acoustic features are controlled. Results showed that all phoneme features (i.e. phoneme onset, phoneme surprisal, phoneme entropy) and word frequency increased the averaged reconstruction accuracy for both Dutch and French stories when acoustic features were controlled, extending the findings of previous studies (Brodbeck et al., 2018; Gillis et al., 2021; Brodbeck et al., 2022). An interaction between language and encoding model showed that reconstruction accuracy improvement from the addition of phoneme-level features and word frequency was greater in Dutch stories, suggesting that when language is comprehended, internally-generated linguistic units, e.g., phonemes, are tracked over and above acoustic edges (Meyer et al., 2020). Our findings contradict the results of Daube et al., 2019, who used articulatory features and phoneme onsets as phonetic features and showed that phoneme-locked responses can be explained by the encoding of acoustic features. It is possible that information theoretic metrics, such as surprisal and entropy, are better suited to capture the transformation of acoustic features into abstract linguistic units.

Comparison of the reconstruction accuracies on the source level showed that, for Dutch stories, accuracy improvement by the linguistic features were left lateralized. However, for uncomprehended French stories, we couldn’t find a significant lateralization effect. The Dutch-speaking participants we selected were familiar with the acoustics and phonotactics of French, and thus with the statistical distribution of those features, even though they cannot comprehend the language (as evidenced by their chance-level behavioral performance, educational background, and self-report). This comparison of neural signals in acoustically-familiar contexts set the bar higher for detecting neural sensitivity to phoneme-level features in a comprehended language, over and above any general statistical sensitivity the brain might show during perception and perceptual adaptation. The slight increase in reconstruction accuracy on the phoneme-level in French is likely due to the participants’ general familiarity with the phonotactic properties of the language, though, given the duration of the experiment, participants also could have learned the phonotactic distribution of the stories, as well as become sensitive to high-frequency words that they may recognize, but which do not lead to sentence and discourse comprehension (e.g. the most frequent words in the stories were le, la, un, une, les, des – all forms of the definite and indefinite determiner the/a); statistical regularities can be acquired very quickly even by naive listeners (Saffran et al., 1996).

We performed a word-level analysis to examine the second question: do higher level linguistic constraints, as formed during comprehension, interact with the neural encoding lower-level information (acoustic and phonemic features) under normal listening conditions? We operationalized constraint as word entropy, or the uncertainty around word predictability given the previous 30 words. When the word entropy is high, the next word is less reliably predictable given the previous words, so the context is less constraining. However, word frequency and predictability are known to be highly correlated (Cohen Priva and Jaeger, 2018; in our study, the correlation between entropy and word frequency was French stories: R=0.336, p<0.00001; Dutch stories: R=0.343, p<0.00001). To control for the responses correlated with word frequency, we also added word frequencies in our full model and compared the reconstruction accuracy improvement explained by acoustic and phoneme features. When we compared the reconstruction accuracies for low and high entropy words, we found that tracking performance of phoneme and acoustic features in high entropy words was higher in Dutch stories. This suggests that sentence and discourse constraint, as expressed in word entropy, modulates the encoding of sub-lexical representations, consistent with top-down models of language comprehension. Results showed that when context was more reliable (i.e. on low entropy words) the neural contribution of acoustic and phoneme-level features is downregulated, inhibited, or not enhanced. When sentence- and discourse-level context constrains lexical representations, sub-lexical representations are inhibited by gain modulation as they are no longer as important for thresholding (Martin, 2016; Martin, 2020).

We also found an interaction between language comprehension and word entropy in the opposite direction for acoustic edges and phoneme features: the effect of sentence and discourse-level context was larger on phoneme features for a comprehended language, and it was larger on acoustic features for the uncomprehended language. In the uncomprehended language, low entropy words were also the most frequent words presented, such as le, la, un, une. Modulation of acoustic edges by context in this situation could be related to statistical 'chunking' of the acoustic signal for frequent words, essentially reflecting recognition of those single function words in the absence of language comprehension. In contrast, when a language is understood, contextual information strongly modulates the tracking of phoneme features. Yet, when the next word is predicted from the context, phoneme features are not as informative as when the context is less containing, and they are downregulated or suppressed such that their encoding does not contribute as much to the composition of the neural signal.

To investigate how language comprehension and word entropy modulated acoustic and phoneme-level linguistic features as they unfolded in time, we analyzed the weights of the TRFs. The amplitude of the power of weights were greater for phoneme-level features in Dutch stories than in French stories, however it was lower for acoustic edges. This suggests that when the language is not comprehended, acoustic features may be more dominant in the neural response, as linguistic features simply are not available. This pattern of results suggest that language comprehension might suppress neural tracking of acoustic edges, or it suppresses the neural representation of edges. If this is so, then language comprehension can be seen as a neural filter over sensory and perceptual input in service of the transformation of that input into linguistic structures. A similar effect was also shown for increasing speech rate; tracking performance of linguistic features decreased and tracking performance of acoustic features increased with decreasing intelligibility (Verschueren et al., 2022).

We also found that phoneme features generated a peak at around 80ms and between 200 and 600ms in the comprehended language, whereas in an uncomprehended language, they only peaked at around 80ms. This dual-peak pattern could be attributable to the internal generation of morphemic units or to lexical access in the comprehended language. Similar to the results we found in reconstruction accuracies, comparison of low and high entropy word TRFs showed that when the word entropy was higher (viz., in a low-constraining sentence and discourse context), the weights of acoustic edges and phoneme-level features were higher. This result suggests that when the higher level cues are not available or reliable, lower-level cues at the phoneme- and acoustic-level may be upregulated or enhanced, or allowed to propagate in order to comprise more of the neural bandwidth. While our results are consistent with Molinaro et al., 2021. – we provide support for a cost minimization perspective rather than the perception facilitation perspective discussed in Molinaro et al. – it is important to note that Molinaro et al. only examined the tracking of acoustic features, specifically the speech envelope, using the Phase Locking Value, and did not examine the contribution of lower-level linguistic features. Secondly, Molinaro et al. use a condition-based experimental design in contrast to our naturalistic stimulus approach. In our study, our aim was to investigate the dynamics of encoding both acoustic and linguistic features, and we utilized a multivariate linear regression method on low and high constraining words which ‘naturally’ occurred in our audiobook stimulus across languages. Our results revealed a trade-off between the encoding of acoustic and linguistic features that was dependent on the level of comprehension. Specifically, in the comprehended language, the predictability of the following word had a greater influence on the tracking of phoneme features as opposed to acoustic features, while in the uncomprehended language, this trend was reversed. Similarly, Donhauser and Baillet, 2020 showed that when sentence context is not reliable, phonemic features (i.e. phoneme surprisal and entropy) are enhanced. Here we examined the effects word entropy on acoustic and phonemic representations in comprehended and uncomprehended languages; we found that comprehension modulates acoustic edges for a shorter period of time than phoneme features when language is comprehended. The interaction between language comprehension and word entropy for acoustic edge encoding was located around auditory cortex in both hemispheres, whereas for phoneme features, it was located in left frontal cortex and right temporal cortex. The presence of a later interaction for phoneme onsets compared to acoustic edges could be due to the sustained representations of phonemes until lexical ambiguity is resolved (Gwilliams et al., 2020) – when the sentence context is not very informative, phonemic representations transformed from acoustic features may persist or play a larger role in the neural response until the word is recognized or until a structure is built.

In this study, to disassociate the effects of word entropy and word frequency, we modeled word frequency with an assumption that entropy and frequency have linearly additive effects; however, previous studies have shown that they interact during late stages of word processing (Fruchter and Marantz, 2015; Huizeling et al., 2022). In a naturalistic listening paradigm, even though it is an instrumental way to study language comprehension, it is not fully possible to control for word frequency or to completely dissociate it from any sentence context effects.

Another limitation of our study is that linear regression modeling does not allow the dissociation of the weights of highly correlated features. When we compared the TRF weights of phoneme onset, surprisal and entropy separately between the set of the first four and the second four Dutch story parts, we saw that TRF weights of these features showed differences (Figure 3—figure supplement 1, Figure 4—figure supplement 1). However, acoustic-edge features and averaged phoneme-level features were highly consistent between different story parts, as acoustic and averaged phoneme features are not as correlated as phoneme onset, phoneme surprisal and phoneme entropy (Figure 3 and Figure 4). Thus, it would be problematic to interpret how language comprehension and sentence and discourse level constraint affects these features separately. As alternative solution to this problem, we also fitted different models with varying time lags separated by 50ms with a 100ms sliding window between –100 and 800ms and also compared the reconstruction accuracies of these models instead of TRF weights. Results showed that word entropy modulated the encoding of phoneme onset and phoneme surprisal for a longer time, starting earlier compared to phoneme entropy.

The raw data used in this study are available from the Donders Institute Data Repository (https://doi.org/10.34973/a65x-p009) By the time of the submission of the paper, the following subjects had both MRI and MEG data: sub-001, sub-003, sub-004, sub-008, sub-009, sub-010, sub-011, sub-013, sub-014, sub-015, sub-017, sub-018, sub-019, sub-020, sub-021, sub-023, sub-025, sub-026, sub-027, sub-028, sub-029, sub-030, sub-032, and sub-033. They were included in the analysis. Analysis code is shared on the following link. https://github.com/tezcanf/Scripts_for_publication.git (copy archived at swh:1:rev:23b3c7ad9dcabae84dca2a017ee8260081a9d18c).

1. 1. Martin AE

   (2023) Donders Institute Data Repository

   Constructing sentence-level meaning: an MEG study of naturalistic language comprehension.

   https://doi.org/10.34973/a65x-p009

1. 1. Bai F
   2. Meyer AS
   3. Martin AE

   (2022) Neural dynamics differentially encode phrases and sentences during spoken language comprehension

   PLOS Biology 20:e3001713.

   https://doi.org/10.1371/journal.pbio.3001713

   • PubMed
   • Google Scholar
2. 1. Brainard DH

   (1997) The Psychophysics toolbox

   Spatial Vision 10:433–436.

   https://doi.org/10.1163/156856897X00357

   • PubMed
   • Google Scholar
3. 1. Brodbeck C
   2. Hong LE
   3. Simon JZ

   (2018) Rapid transformation from auditory to linguistic representations of continuous speech

   Current Biology 28:3976–3983.

   https://doi.org/10.1016/j.cub.2018.10.042

   • PubMed
   • Google Scholar
4. Preprint

   1. Brodbeck C
   2. Das P
   3. Gillis M
   4. Kulasingham JP
   5. Bhattasali S
   6. Gaston P
   7. Resnik P
   8. Simon JZ

   (2021) Eelbrain: A Python toolkit for time-continuous analysis with temporal response functions

   bioRxiv.

   https://doi.org/10.1101/2021.08.01.454687

   • Google Scholar
5. 1. Brodbeck C
   2. Bhattasali S
   3. Cruz Heredia AAL
   4. Resnik P
   5. Simon JZ
   6. Lau E

   (2022) Parallel processing in speech perception with local and global representations of linguistic context

   eLife 11:e72056.

   https://doi.org/10.7554/eLife.72056

   • PubMed
   • Google Scholar
6. 1. Broderick MP
   2. Anderson AJ
   3. Di Liberto GM
   4. Crosse MJ
   5. Lalor EC

   (2018) Electrophysiological correlates of semantic dissimilarity reflect the comprehension of natural, narrative speech

   Current Biology 28:803–809.

   https://doi.org/10.1016/j.cub.2018.01.080

   • PubMed
   • Google Scholar
7. 1. Cohen Priva U
   2. Jaeger TF

   (2018) The interdependence of frequency, predictability, and Informativity in the segmental domain

   Linguistics Vanguard 4:s2.

   https://doi.org/10.1515/lingvan-2017-0028

   • Google Scholar
8. 1. Coopmans CW
   2. de Hoop H
   3. Hagoort P
   4. Martin AE

   (2022) Effects of structure and meaning on cortical tracking of linguistic units in naturalistic speech

   Neurobiology of Language 3:386–412.

   https://doi.org/10.1162/nol_a_00070

   • PubMed
   • Google Scholar
9. 1. Daube C
   2. Ince RAA
   3. Gross J

   (2019) Simple acoustic features can explain phoneme-based predictions of cortical responses to speech

   Current Biology 29:1924–1937.

   https://doi.org/10.1016/j.cub.2019.04.067

   • PubMed
   • Google Scholar
10. 1. DeLong KA
    2. Urbach TP
    3. Kutas M

    (2005) Probabilistic word pre-activation during language comprehension inferred from electrical brain activity

    Nature Neuroscience 8:1117–1121.

    https://doi.org/10.1038/nn1504

    • PubMed
    • Google Scholar
11. Preprint

    1. de Vries W
    2. Nissim M

    (2020) As good as new. How to successfully recycle English GPT-2 to make models for other languages

    arXiv.

    https://arxiv.org/abs/2012.05628

    • Google Scholar
12. 1. Di Liberto GM
    2. O’Sullivan JA
    3. Lalor EC

    (2015) Low-frequency cortical entrainment to speech reflects phoneme-level processing

    Current Biology 25:2457–2465.

    https://doi.org/10.1016/j.cub.2015.08.030

    • PubMed
    • Google Scholar
13. 1. Ding N
    2. Melloni L
    3. Zhang H
    4. Tian X
    5. Poeppel D

    (2016) Cortical tracking of hierarchical linguistic structures in connected speech

    Nature Neuroscience 19:158–164.

    https://doi.org/10.1038/nn.4186

    • PubMed
    • Google Scholar
14. 1. Donhauser PW
    2. Baillet S

    (2020) Two distinct neural timescales for predictive speech processing

    Neuron 105:385–393.

    https://doi.org/10.1016/j.neuron.2019.10.019

    • PubMed
    • Google Scholar
15. 1. Drennan DP
    2. Lalor EC

    (2019) Cortical tracking of complex sound envelopes: modeling the changes in response with intensity

    ENeuro 6:ENEURO.0082-19.2019.

    https://doi.org/10.1523/ENEURO.0082-19.2019

    • PubMed
    • Google Scholar
16. 1. Friederici AD

    (2002) Towards a neural basis of auditory sentence processing

    Trends in Cognitive Sciences 6:78–84.

    https://doi.org/10.1016/s1364-6613(00)01839-8

    • PubMed
    • Google Scholar
17. 1. Friston K

    (2005) A theory of cortical responses

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 360:815–836.

    https://doi.org/10.1098/rstb.2005.1622

    • PubMed
    • Google Scholar
18. 1. Fruchter J
    2. Marantz A

    (2015) Decomposition, lookup, and recombination: MEG evidence for the full decomposition model of complex visual word recognition

    Brain and Language 143:81–96.

    https://doi.org/10.1016/j.bandl.2015.03.001

    • PubMed
    • Google Scholar
19. 1. Ghitza O

    (2013) The Theta-syllable: a unit of speech information defined by cortical function

    Frontiers in Psychology 4:138.

    https://doi.org/10.3389/fpsyg.2013.00138

    • PubMed
    • Google Scholar
20. 1. Gillis M
    2. Vanthornhout J
    3. Simon JZ
    4. Francart T
    5. Brodbeck C

    (2021) Neural markers of speech comprehension: measuring EEG tracking of linguistic speech representations, controlling the speech acoustics

    The Journal of Neuroscience 41:10316–10329.

    https://doi.org/10.1523/JNEUROSCI.0812-21.2021

    • PubMed
    • Google Scholar
21. 1. Giraud AL
    2. Poeppel D

    (2012) Cortical oscillations and speech processing: emerging computational principles and operations

    Nature Neuroscience 15:511–517.

    https://doi.org/10.1038/nn.3063

    • PubMed
    • Google Scholar
22. Preprint

    1. Gwilliams L
    2. King JR
    3. Marantz A
    4. Poeppel D

    (2020) Neural dynamics of phoneme sequences: position-invariant code for content and order

    bioRxiv.

    https://doi.org/10.1101/2020.04.04.025684

    • Google Scholar
23. 1. Hagoort P

    (2013) MUC (memory, unification, control) and beyond

    Frontiers in Psychology 4:416.

    https://doi.org/10.3389/fpsyg.2013.00416

    • PubMed
    • Google Scholar
24. Website

    1. Hart MS

    (1971) Project Gutenberg. Project Gutenberg

    Accessed July 21, 2021.

    https://www.gutenberg.org
25. Preprint

    1. Heilbron M
    2. Armeni K
    3. Schoffelen JM
    4. Hagoort P
    5. de Lange FP

    (2020) A hierarchy of linguistic predictions during natural language comprehension

    bioRxiv.

    https://doi.org/10.1101/2020.12.03.410399

    • Google Scholar
26. 1. Huizeling E
    2. Arana S
    3. Hagoort P
    4. Schoffelen JM

    (2022) Lexical frequency and sentence context influence the brain’s response to single words

    Neurobiology of Language 3:149–179.

    https://doi.org/10.1162/nol_a_00054

    • PubMed
    • Google Scholar
27. 1. Kaufeld G
    2. Bosker HR
    3. Ten Oever S
    4. Alday PM
    5. Meyer AS
    6. Martin AE

    (2020a) Linguistic structure and meaning organize neural oscillations into a content-specific hierarchy

    The Journal of Neuroscience 40:9467–9475.

    https://doi.org/10.1523/JNEUROSCI.0302-20.2020

    • PubMed
    • Google Scholar
28. 1. Kaufeld G
    2. Naumann W
    3. Meyer AS
    4. Bosker HR
    5. Martin AE

    (2020b) Contextual speech rate influences morphosyntactic prediction and integration

    Language, Cognition and Neuroscience 35:933–948.

    https://doi.org/10.1080/23273798.2019.1701691

    • Google Scholar
29. 1. Kaufeld G
    2. Ravenschlag A
    3. Meyer AS
    4. Martin AE
    5. Bosker HR

    (2020c) Knowledge-based and signal-based cues are weighted flexibly during spoken language comprehension

    Journal of Experimental Psychology. Learning, Memory, and Cognition 46:549–562.

    https://doi.org/10.1037/xlm0000744

    • PubMed
    • Google Scholar
30. Website

    1. Kearns J

    (2015) Librivox free public domain audiobooks

    Accessed July 21, 2021.

    https://librivox.org/
31. 1. Keitel A
    2. Gross J
    3. Kayser C

    (2018) Perceptually relevant speech tracking in auditory and motor cortex reflects distinct linguistic features

    PLOS Biology 16:e2004473.

    https://doi.org/10.1371/journal.pbio.2004473

    • PubMed
    • Google Scholar
32. 1. Keuleers E
    2. Brysbaert M
    3. New B

    (2010) SUBTLEX-NL: A new measure for Dutch word frequency based on film subtitles

    Behavior Research Methods 42:643–650.

    https://doi.org/10.3758/BRM.42.3.643

    • PubMed
    • Google Scholar
33. 1. Kleiner M
    2. Brainard D
    3. Pelli D

    (2007)

    What’s new in Psychtoolbox-3

    Psychology 36:1–16.

    • Google Scholar
34. 1. Lee TS
    2. Mumford D

    (2003) Hierarchical Bayesian inference in the visual cortex

    Journal of the Optical Society of America. A, Optics, Image Science, and Vision 20:1434–1448.

    https://doi.org/10.1364/josaa.20.001434

    • PubMed
    • Google Scholar
35. Software

    1. Louis A

    (2020) BelGPT-2: a GPT-2 model pre-trained on French corpora

    GitHub.

    https://github.com/antoiloui/belgpt2
36. 1. Luo H
    2. Poeppel D

    (2007) Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex

    Neuron 54:1001–1010.

    https://doi.org/10.1016/j.neuron.2007.06.004

    • PubMed
    • Google Scholar
37. 1. Marslen-Wilson WD
    2. Welsh A

    (1978) Processing interactions and lexical access during word recognition in continuous speech

    Cognitive Psychology 10:29–63.

    https://doi.org/10.1016/0010-0285(78)90018-X

    • Google Scholar
38. 1. Martin AE

    (2016) Language processing as cue integration: grounding the psychology of language in perception and neurophysiology

    Frontiers in Psychology 7:120.

    https://doi.org/10.3389/fpsyg.2016.00120

    • PubMed
    • Google Scholar
39. 1. Martin AE
    2. Doumas LAA

    (2017) A mechanism for the cortical computation of hierarchical linguistic structure

    PLOS Biology 15:e2000663.

    https://doi.org/10.1371/journal.pbio.2000663

    • PubMed
    • Google Scholar
40. 1. Martin AE
    2. Monahan PJ
    3. Samuel AG

    (2017) Prediction of agreement and phonetic overlap shape sublexical identification

    Language and Speech 60:356–376.

    https://doi.org/10.1177/0023830916650714

    • PubMed
    • Google Scholar
41. 1. Martin AE
    2. Doumas LA

    (2019) Predicate learning in neural systems: using oscillations to discover latent structure

    Current Opinion in Behavioral Sciences 29:77–83.

    https://doi.org/10.1016/j.cobeha.2019.04.008

    • Google Scholar
42. 1. Martin AE

    (2020) A compositional neural architecture for language

    Journal of Cognitive Neuroscience 32:1407–1427.

    https://doi.org/10.1162/jocn_a_01552

    • PubMed
    • Google Scholar
43. 1. McClelland JL
    2. Rumelhart DE

    (1981) An interactive activation model of context effects in letter perception: I. An account of basic findings

    Psychological Review 88:375–407.

    https://doi.org/10.1037/0033-295X.88.5.375

    • Google Scholar
44. 1. Mewhort DJK

    (1967) Familiarity of letter sequences, response uncertainty, and the tachistoscopic recognition experiment

    Canadian Journal of Psychology / Revue Canadienne de Psychologie 21:309–321.

    https://doi.org/10.1037/h0082983

    • Google Scholar
45. 1. Meyer L
    2. Sun Y
    3. Martin AE

    (2020) Synchronous, but not Entrained: exogenous and endogenous cortical rhythms of speech and language processing

    Language, Cognition and Neuroscience 35:1089–1099.

    https://doi.org/10.1080/23273798.2019.1693050

    • Google Scholar
46. 1. Molinaro N
    2. Lizarazu M
    3. Baldin V
    4. Pérez-Navarro J
    5. Lallier M
    6. Ríos-López P

    (2021) Speech-brain phase coupling is enhanced in low contextual semantic predictability conditions

    Neuropsychologia 156:107830.

    https://doi.org/10.1016/j.neuropsychologia.2021.107830

    • PubMed
    • Google Scholar
47. 1. New B
    2. Pallier C
    3. Ferrand L
    4. Matos R

    (2001) Une base de Données Lexicales Du Français Contemporain sur Internet: LEXIQUE

    L’Année Psychologique 101:447–462.

    https://doi.org/10.3406/psy.2001.1341

    • Google Scholar
48. 1. Nieuwland MS
    2. Politzer-Ahles S
    3. Heyselaar E
    4. Segaert K
    5. Darley E
    6. Kazanina N
    7. Von Grebmer Zu Wolfsthurn S
    8. Bartolozzi F
    9. Kogan V
    10. Ito A
    11. Mézière D
    12. Barr DJ
    13. Rousselet GA
    14. Ferguson HJ
    15. Busch-Moreno S
    16. Fu X
    17. Tuomainen J
    18. Kulakova E
    19. Husband EM
    20. Donaldson DI
    21. Kohút Z
    22. Rueschemeyer S-A
    23. Huettig F

    (2018) Large-scale replication study reveals a limit on probabilistic prediction in language comprehension

    eLife 7:e33468.

    https://doi.org/10.7554/eLife.33468

    • PubMed
    • Google Scholar
49. 1. Nieuwland MS

    (2019) Do ‘Early’Brain responses reveal word form prediction during language comprehension? A critical review

    Neuroscience & Biobehavioral Reviews 96:367–400.

    https://doi.org/10.1016/j.neubiorev.2018.11.019

    • Google Scholar
50. 1. Oganian Y
    2. Kojima K
    3. Breska A
    4. Cai C
    5. Findlay A
    6. Chang E
    7. Nagarajan SS

    (2023)

    Phase alignment of low-frequency neural activity to the amplitude envelope of speech reflects evoked responses to acoustic edges, not oscillatory Entrainment

    Journal of Neuroscience 43:3909–3921.

    • Google Scholar
51. 1. Oostenveld R
    2. Fries P
    3. Maris E
    4. Schoffelen JM

    (2011) Fieldtrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data

    Computational Intelligence and Neuroscience 2011:156869.

    https://doi.org/10.1155/2011/156869

    • PubMed
    • Google Scholar
52. 1. Pelli DG

    (1997)

    The VideoToolbox software for visual psychophysics: transforming numbers into movies

    Spatial Vision 10:437–442.

    • PubMed
    • Google Scholar
53. 1. Rao RPN
    2. Ballard DH

    (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects

    Nature Neuroscience 2:79–87.

    https://doi.org/10.1038/4580

    • PubMed
    • Google Scholar
54. 1. Reicher GM

    (1969) Perceptual recognition as a function of meaningfulness of stimulus material

    Journal of Experimental Psychology 81:275–280.

    https://doi.org/10.1037/h0027768

    • PubMed
    • Google Scholar
55. 1. Rumelhart DE
    2. McClelland JL

    (1982)

    An interactive activation model of context effects in letter perception: part 2. The contextual enhancement effect and some tests and extensions of the model

    Psychological Review 89:60–94.

    • PubMed
    • Google Scholar
56. 1. Saffran JR
    2. Aslin RN
    3. Newport EL

    (1996) Statistical learning by 8-month-old infants

    Science 274:1926–1928.

    https://doi.org/10.1126/science.274.5294.1926

    • PubMed
    • Google Scholar
57. Conference

    1. Schiel F

    (1999)

    Automatic phonetic transcription of non-prompted speech

    International Conference on Statistical Language and Speech Processing.

    • Google Scholar
58. Conference

    1. Schiel F

    (2015)

    A Statistical Model for Predicting Pronunciation

    International Congress of Phonetic Sciences.

    • Google Scholar
59. 1. Slaats S
    2. Weissbart H
    3. Schoffelen JM
    4. Meyer AS
    5. Martin AE

    (2023) Delta-band neural responses to individual words are modulated by sentence processing

    The Journal of Neuroscience 1:JN-RM-0964-22.

    https://doi.org/10.1523/JNEUROSCI.0964-22.2023

    • PubMed
    • Google Scholar
60. 1. Smith SM
    2. Nichols TE

    (2009) Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and localisation in cluster inference

    NeuroImage 44:83–98.

    https://doi.org/10.1016/j.neuroimage.2008.03.061

    • PubMed
    • Google Scholar
61. Conference

    1. Strunk J
    2. Schiel F
    3. Seifart F

    (2014)

    Untrained forced alignment of transcriptions and audio for language documentation corpora using WebMAUS

    International Conference on Language Resources and Evaluation.

    • Google Scholar
62. 1. Ten Oever S
    2. Martin AE

    (2021) An oscillating computational model can track pseudo-rhythmic speech by using linguistic predictions

    eLife 10:e68066.

    https://doi.org/10.7554/eLife.68066

    • PubMed
    • Google Scholar
63. 1. Ten Oever S
    2. Carta S
    3. Kaufeld G
    4. Martin AE

    (2022a) Neural tracking of phrases in spoken language comprehension is automatic and task-dependent

    eLife 11:e77468.

    https://doi.org/10.7554/eLife.77468

    • PubMed
    • Google Scholar
64. 1. Ten Oever S
    2. Kaushik K
    3. Martin AE

    (2022b) Inferring the nature of linguistic computations in the brain

    PLOS Computational Biology 18:e1010269.

    https://doi.org/10.1371/journal.pcbi.1010269

    • PubMed
    • Google Scholar
65. 1. Verschueren E
    2. Gillis M
    3. Decruy L
    4. Vanthornhout J
    5. Francart T

    (2022) Speech understanding oppositely affects acoustic and linguistic neural tracking in a speech rate manipulation paradigm

    The Journal of Neuroscience 42:7442–7453.

    https://doi.org/10.1523/JNEUROSCI.0259-22.2022

    • PubMed
    • Google Scholar
66. 1. Weissbart H
    2. Kandylaki KD
    3. Reichenbach T

    (2020) Cortical tracking of surprisal during continuous speech comprehension

    Journal of Cognitive Neuroscience 32:155–166.

    https://doi.org/10.1162/jocn_a_01467

    • PubMed
    • Google Scholar
67. 1. Wheeler DD

    (1970) Processes in word recognition

    Cognitive Psychology 1:59–85.

    https://doi.org/10.1016/0010-0285(70)90005-8

    • Google Scholar
68. 1. Zioga I
    2. Weissbart H
    3. Lewis AG
    4. Haegens S
    5. Martin AE

    (2023) Naturalistic spoken language comprehension is supported by alpha and beta oscillations

    The Journal of Neuroscience 43:3718–3732.

    https://doi.org/10.1523/JNEUROSCI.1500-22.2023

    • PubMed
    • Google Scholar

Author details

1. Filiz Tezcan

   Language and Computation in Neural Systems Group, Max Planck Institute for Psycholinguistics, Nijmegen, Netherlands

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   Filiz.TezcanSemerci@mpi.nl

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3327-0181

2. Hugo Weissbart

   Donders Centre for Cognitive Neuroimaging, Radboud University, Nijmegen, Netherlands

   Contribution

   Conceptualization, Data curation, Supervision, Validation, Visualization, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2820-3865

3. Andrea E Martin

   1. Language and Computation in Neural Systems Group, Max Planck Institute for Psycholinguistics, Nijmegen, Netherlands
   2. Donders Centre for Cognitive Neuroimaging, Radboud University, Nijmegen, Netherlands

   Contribution

   Conceptualization, Resources, Formal analysis, Supervision, Funding acquisition, Validation, Methodology, Project administration, Writing - review and editing

   Competing interests

   Reviewing editor, eLife

   "This ORCID iD identifies the author of this article:" 0000-0002-3395-7234

• 810

  views

• 144

  downloads

• 9

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.