Human speech comprehension involves a complex set of processes that transform an auditory input into the speaker’s intended meaning, wherein each word is sequentially recognized and integrated with the preceding words to obtain a coherent interpretation (Marslen-Wilson and Tyler, 1980; Choi et al., 2021; Lyu et al., 2019). Crucially, rather than simple linear concatenation, individual words are combined according to the non-linear and often discontinuous structure embedded in an utterance as it is delivered over time (Everaert et al., 2015). For example, in the sentence “The boy who chased the cat was …”, it is the structurally close word ‘boy’, rather than the linearly close word ‘cat’, that is combined with ‘was’. However, the neural dynamics underpinning the incremental construction of a structured interpretation from a spoken sentence is still unclear.

Previous neuroimaging studies on the structure of language primarily focused on syntax (Matchin and Hickok, 2020), contrasting grammatical sentences against word lists or sentences with syntactic violations (Law and Pylkkänen, 2021; Nelson et al., 2017), manipulating the syntactic complexity in sentences (Pallier et al., 2011), or studying artificial grammatical rules elicited by structured, unintelligible strings (Friederici et al., 2006). Nevertheless, finding the structure in an unfolding sentence also depends on the constraints jointly placed by other linguistic properties and non-linguistic information such as the broad world knowledge (Bever, 1970; Tyler and Marslen-Wilson, 1977).

Unlike the two-stage model (Frazier, 1987; Frazier and Rayner, 1982) which posits an initial parsing stage relying solely on syntax, the constraint-based approach to sentence processing (MacDonald et al., 1994; Trueswell and Tanenhaus, 1994) proposes that speech comprehension is concurrently governed by multiple types of probabilistic constraints (e.g. syntax, semantics, world knowledge), generated by individual words as they are sequentially heard. There is no delay in the utilization of these multifaceted constraints once they become available, neither is a fixed priority assigned to one type of constraint over another; rather, it is the interpretative coherence of all available constraints that forms the basis for successful language comprehension (Altmann, 1998). Although lexical constraints of individual words can be estimated from large corpora data, it has been challenging to model the dynamic interplay between various types of linguistic and non-linguistic constraints in a specific context, especially at the sentence level and beyond.

Contemporary deep language models (DLMs) have made great strides in a wide array of natural language processing tasks, including text generation, parsing, and translation (Vaswani, 2017; Devlin et al., 2019; Brown, 2020; Ouyang et al., 2022). While current DLMs are still imperfect in terms of human-level language understanding related to reasoning and complex physical or social situations (Bisk et al., 2020), they are arguably valuable models of general linguistic capacities due to their ability to identify and leverage relevant statistical regularities of linguistic and non-linguistic world knowledge present in massive training data (Linzen and Baroni, 2021; Pavlick, 2022). Human language comprehension requires a contextualized integration of multifaceted constraints (Tyler and Marslen-Wilson, 1977; Marslen-Wilson, 1975; Kuperberg, 2007). In this regard, DLMs excel in flexible combination of different types of features (e.g. syntactic structure and semantic meaning) embedded in their rich internal representations (Linzen and Baroni, 2021; Pavlick, 2022; Bengio et al., 2021; Manning et al., 2020). Their deep contextualized representations capture the distributed regularities that jointly determine the coherent interpretation of a given sentence, providing context-dependent composition and quantitative measures of the underlying sentential structure. These properties relate back to Elman’s recurrent neural network (Elman, 1990; Elman, 1993) which automatically picks up and encodes lexical syntactic/semantic information in the hidden states.

Recent studies have revealed an overall congruence between language representations in DLMs and those observed in the human brain while processing the same spoken or written input (Schrimpf et al., 2021; Goldstein et al., 2022; Heilbron et al., 2022; Toneva et al., 2022; Caucheteux et al., 2022; Caucheteux and King, 2022; Caucheteux et al., 2023), suggesting the potential value of DLMs as a computational tool to investigate the neural basis of language comprehension. To move beyond comparing the similarities between entire model hidden states and brain activity, probing techniques that can extract specific contents from DLMs (Hewitt and Liang, 2019; Tenney, 2019) make it possible to study the neural dynamics relevant to processing such specific information. The important advance here is that we can leverage the deep learning strengths of DLMs to create rigorously quantified models of the broader and multifaceted constraint environment in which a structured interpretation is constructed. Such models can be compared, dynamically, with more restricted and interpretable factors that capture the specific linguistic combinatorial constraints necessary for successful language comprehension.

Here, we take this approach further by designing sentences with contrasting linguistic structures and using a structural probe technique (Hewitt and Manning, 2019) to extract word-by-word contextualized representations of sentential structures from a widely used DLM, namely, BERT (Devlin et al., 2019). This provides the neurocomputational specificity required to elucidate the neural dynamics underlying the incremental construction of a structured interpretation from an unfolding spoken sentence. After a detailed evaluation of BERT structural measures according to the hypothesized constraint-based approach and human behavioural results, we used spatiotemporal searchlight representational similarity analysis (ssRSA) (Kriegeskorte et al., 2008) to test these quantitative structural measures and relevant lexical properties against source-localized EMEG data recorded while participants were listening to the same sentences. Our findings reveal how the structured interpretation of a spoken sentence is incrementally built under multifaceted probabilistic constraints in the brain.

We constructed 60 sets of sentences with varying sentential structures (see ‘Materials and methods’) and presented them to human listeners. We also input them word-by-word to BERT to extract incremental structural representations. These natural spoken sentences were constructed to balance off specifically linguistic constraints on interpretation against varying non-linguistic constraints as the sentence is incrementally interpreted, providing a realistic simulation of real-life language use. In each stimulus set, there are two target sentences differing only in the transitivity of the first verb (Verb1) encountered, that is, how likely it is that Verb1 takes a direct object see [a] and [b] below and Figure 1:

1. The dog found in the park was covered in mud.

2. The dog walked in the park was covered in mud.

Figure 1

Download asset Open asset

In the first sentence, Verb1 (i.e. ‘found’) has high transitivity (HiTrans) and strongly prefers a direct object (e.g. ball), while in the second sentence, Verb1 (i.e. ‘walked’) has relatively low transitivity (LoTrans) and is often used without a following direct object. Critically, the structural interpretation of these sentences is ambiguous at the point Verb1 is encountered, and the preferred human resolution of this ambiguity depends on the real-time integration of linguistic and non-linguistic constraints as more of the sentence is heard. In the example above, the sequence “The dog found …” could initially have either an active interpretation – where the dog has found something – or a passive interpretation – where the dog is found by someone (Figure 1). Because ‘find’ is primarily a transitive verb, the human listener is likely to be biased towards an initial active interpretation. Similarly, the sequence “The dog walked …”, where walk is primarily used as an intransitive verb (without a direct object), could also bias the listener to an active interpretation, where the dog is doing the walking, rather than the less frequent passive interpretation where someone is taking the dog for a walk (i.e. walking the dog).

This initial structural interpretation up to Verb1 does not, however, just depend on linguistic knowledge such as Verb1 transitivity. It also depends on non-linguistic information, that is, how likely the subject is (or is not) to adopt the active (agent) role to perform the specified action (Dowty, 1991; Marslenwilson et al., 1993), that is, ‘thematic role’ properties of the subject noun. Although it could be reflected by statistical regularities in language, thematic role preference hinges more on world knowledge, plausibility, or real-world statistics. So, regardless of Verb1 transitivity, the active interpretation should be more strongly favoured in “The king found/walked …” given the higher agenthood of the ‘king’ and thus the greater implausibility of a passive interpretation involving a ‘king’ relative to a ‘dog’. Hence, the word-by-word interpretation of the sentential structure – and of the real-world event structure evoked by this interpretation – is determined by the constraints jointly placed by the subject noun and Verb1, which is manifested by the interpretative coherence between non-linguistic world knowledge (i.e. thematic role preference) and linguistic knowledge (i.e. verb transitivity).

As the sentence evolves, and the prepositional phrase ‘in the park’ that follows Verb1 is incrementally processed, there is further modulation of the preferred interpretation, again reflecting both Verb1 transitivity and the plausibility of the event being constructed. Specifically, the passive interpretation will become more preferred in a HiTrans sentence, given the absence of an expected direct object for the highly transitive Verb1, so Verb1 tends to be interpreted as a passive verb [i.e. the head of a reduced relative clause in “The dog (that was) found in the park …”]. Conversely, in a LoTrans sentence, the active interpretation of Verb1 is strengthened by the incoming prepositional phrase, which is in accord with the verb’s intransitive use and the event conjured up by the sequence of words heard so far (e.g. “The dog walked in the park …”). Hence, these two sentence types are likely to differ in the structural interpretation preferred by the end of the prepositional phrase. However, with the appearance of the actual main verb (e.g. ‘was covered’ in the example sentences), the active interpretation of Verb1 as the main verb will be completely rejected, which resolves the potential ambiguity and confirms the passive interpretation in both HiTrans and LoTrans sentences.

In brief, understanding these complex sentences require listeners to integrate discontinuous words to solve a long-distance dependency between the subject noun and the actual main verb separated by an intervening clause. This engages the neurobiological processes of integration across different lexical constraints and multiple levels of the sentence processing system. For example, the incremental building, maintenance, and update of sentential structure over time might primarily involve activity in the fronto-temporal regions (Friederici, 2012), while estimating the plausibility of the event interpreted from the sentence with prior knowledge of the world may elicit neural responses in the default mode network (DMN) (Yeshurun et al., 2021).

Human incremental structural interpretations

As the first step, and to quantify how the stimulus sentences exemplified a constraint-based account of incremental structural interpretation, we conducted two pre-tests where participants listened to sentence fragments, starting from sentence onset and continuing either until the end of Verb1 or to the end of the prepositional phrase (Figure 2A), and then produced a continuation to complete the sentence (see ‘Materials and methods’). Based on the continuations provided by the listeners at these two gating points, we can infer their online structural interpretations.

Figure 2

Download asset Open asset

In the continuations after Verb1, a direct object was more likely to be found in HiTrans sentences, indicating a transitive use of Verb1, while an opposite pattern was found for a PP continuation, indicating an intransitive use of Verb1 (Figure 2B). As expected, the probability of a main verb (MV) in the continuations after the prepositional phrase was lower in LoTrans sentences (Figure 2C), suggesting that listeners preferred the active interpretation and tended to interpret Verb1 as the main verb by the end of the prepositional phrase in LoTrans sentences, and vice versa in HiTrans sentences. Crucially, neither of the two pre-tests resulted in a complete separation between HiTrans and LoTrans sentences; instead, they were characterized by two different but overlapping probabilistic distributions. This finding suggests that passive and active interpretations varied in plausibility in each sentence type before the actual main verb was presented, reflecting the probabilistic constraints jointly placed by the combination of the specific subject noun, Verb1, and the prepositional phrase in each sentence.

To relate these human interpretative preferences to the broader landscape of distributional language data, we developed corpus-based measures of the thematic role preference of the subject noun (i.e. how likely it is interpreted as an agent that conducts an action) and the transitivity of Verb1 in each sentence, from which we derived a passive index and an active index (see ‘Materials and methods’). These indices separately capture the interpretative coherence between these two lexical properties towards passive and active interpretations. Both high subject noun agenthood and low Verb1 transitivity coherently preferred an active interpretation as the prepositional phrase was heard (i.e. a high active index), and vice versa for the passive interpretation (i.e. a high passive index). In accord with the constraint-based hypothesis, we found that human interpretative preference for the two types of sentences was significantly correlated with the lexical constraints placed by the subject noun and Verb1 (Figure 2D).

Incremental structural representations extracted from BERT

Next, we extracted structural representations at various positions in the same sentences from BERT and evaluated them according to the constraint-based hypothesis and human behavioural results. This evaluation is needed to motivate the use of BERT structural measures to reveal how the structured interpretation of a spoken sentence is incrementally built in the brain.

Typically, the structure of a sentence can be represented by a dependency parse tree (MacCartney and Manning, 2006), where words are situated at different depths given their structural dependency (Figure 3A). Each edge links two structurally proximate words as being the head and the dependent separately (e.g. a verb and its direct object). However, such a parse tree is context-free, that is, it only captures the syntactic relation between each pair of words and abstracts away from the specific lexical (and higher order) contents of the sentence that constrain its structural interpretation. This context-free parse depth is always the same for words at the same position in sentences with the same structured interpretation (e.g. ‘found’ and ‘walked’ in either of the two parse trees in Figure 3A).

Figure 3 with 3 supplements see all

Download asset Open asset

To obtain structural measures that also encode the specific lexical contents in a sentence, we adopted a structural probing technique (Hewitt and Manning, 2019) to reconstruct a sentence’s structure by estimating each word’s parse depth based on their contextualized representations generated by BERT (see ‘Materials and methods’). Note that BERT is a multi-layer DLM (24 layers in the version used in this study) which may distribute different aspects of its computational solutions over multiple layers. Accordingly, we trained a structural probing model for each layer and selected the one with the most accurate structural representations while also including its neighbouring layers to cover relevant upstream and downstream information. Following this strategy, we used the BERT structural measures obtained from layers 12–16 with the best performance achieved in layer 14 (see Figure 3—figure supplement 1 and ‘Materials and methods’).

We input each sentence word-by-word to the trained BERT structural probing models, focusing on the incremental structural representation being built as it progressed from Verb1 to the main verb (see examples in Figure 3B and C). Note that we defined the first word after the prepositional phrase as the main verb since its appearance is sufficient to resolve the intended structure where Verb1 is a passive verb. We found that, for each type of sentences, the BERT parse depth of words at the same position formed a distribution ranging around the corresponding context-free parse depths in either the passive or the active interpretation (see Figure 3—figure supplement 2), suggesting a word-specific rather than position-specific structural representation. In addition, we quantified the contributions of words at different positions to the variances encoded in BERT parse depth vectors. Our analysis revealed that content words contributed significantly more than function words (i.e. the determiners) (see Figure 3—figure supplement 3).

Then we visualized BERT’s word-by-word structural measures, focusing on the dependency between the subject noun and Verb1 that is core to the current interpretation of the sentence – whether the subject noun is the agent or the patient of Verb1. To this end, we built a three-dimensional (3D) vector including BERT parse depths of the first three words up to Verb1 for each sentence (e.g. “The dog found …”). This 3D vector was updated incrementally with each additional word in the input, thereby capturing the dynamic interpretation of the structural dependency between the subject noun and Verb1, influenced by the context provided by subsequent words in a specific sentence. Like the probabilistic interpretation found within each type of sentences in human listeners, trajectories of individual HiTrans and LoTrans sentences are considerably distributed and intertwined (see the upper panel of Figure 3D), implying that BERT structural interpretations are sensitive to the idiosyncratic contents in each sentence.

To make sense of these trajectories, we also vectorized the context-free parse depth of the first three words indicating passive and active interpretations (Figure 3A) separately and located them in the 3D vector space as landmarks (hollow triangle and circle in Figure 3D), so that the plausibility of either interpretation can be estimated by a sentence’s distance from its landmark. As shown by the trajectories of the median BERT parse depth of the two sentence types (see the lower panel of Figure 3D), in general, HiTrans sentences continuously moved towards the passive interpretation landmark after Verb1, with a significant change of distances detected at the main verb (see the orange bars in Figure 3E). LoTrans sentences started by approaching the active interpretation landmark but were reorientated to the passive counterpart with the appearance of the actual main verb, with significant changes of distances detected at both Verb1 and main verb (see the purple bars in Figure 3E). These results resemble the pattern of human interpretative preference observed in the continuation pre-tests (Figure 2C), where the passive and active interpretations were separately preferred in HiTrans and LoTrans sentences by the end of the prepositional phrase in a probabilistic manner, before the passive interpretation was established with the appearance of the actual main verb.

BERT structural measures are correlated with constraints driving human interpretation

To further assess whether BERT’s preferences for structural interpretation align with the constraints considered by human listeners during speech comprehension, we correlated BERT structural measures with relevant corpus-based measures and human behavioural data (see ‘Materials and methods).

We first focused on BERT’s interpretative mismatch quantified as the distance between an unfolding sentence and each of the two landmarks in the model space, which was dynamically updated as the sentence unfolded (Figure 3D). Consistently, from the incoming prepositional phrase to the main verb, sentences that are closer to the passive landmark in the model space have higher Verb1 transitivity, a higher passive index but a lower active index, while sentences closer to the active interpretation landmark exhibited higher active index and lower passive index (Figure 4A and B). Moreover, at the beginning of the prepositional phrase, the change of distance towards either interpretation landmark between two consecutive words is also correlated with these constraints (Figure 4C and D), suggesting an immediate update in the structural interpretation in combination with the accumulated constraints from the preceding subject noun and Verb1.

Figure 4 with 3 supplements see all

Download asset Open asset

Moreover, we found that both the incremental BERT parse depth vectors as a whole (which are captured by their principal components) and the BERT parse depth of Verb1 (which is the most indicative marker of the interpretation preferred) are correlated with the constraints placed by the subject noun and Verb1 (Figure 4E–H). Moreover, the significant effects consistently found as the sentence unfolds suggest that properties of preceding words are used to constrain the interpretation of the upcoming input, which is key to resolving discontinuous structural dependencies. In addition, we found that BERT structural interpretations were also correlated with the main verb probability in the continuation pre-test which directly reflects human interpretation preference (black bars in Figure 4).

Overall, these results illustrated, at which position in a sentence, relevant lexical constraints started being encoded by BERT, which also validated the contextualized BERT structural measures in terms of the constraint-based hypothesis and human behavioural results, and motivated the use of them to probe the neural processes involved during the incremental structural interpretation of spoken sentences.

Neural dynamics of incremental structural interpretation

To study how the structured interpretation of a spoken sentence is built word-by-word in the brain, we used ssRSA to test the incremental BERT structural measures in source-localized EMEG collected when the same sentences were delivered to human listeners (see Figure 5 for the pipeline of ssRSA). This combination of methods gains improved neurocomputational specificity by probing the spatiotemporally resolved neural activity with detailed structural representations rather than the entire hidden states of BERT. We compared the representational geometry of BERT structural measures with that of neural responses inside a spatiotemporal searchlight moving across the cortical surface, significant model fits showed when and where the incremental structural interpretations or relevant lexical constraints emerge and update in the brain. Given the probabilistic interpretations in BERT and human listeners reported above, we combined HiTrans and LoTrans sentences as one group to increase the range of pair-wise dissimilarity to be modelled in ssRSA.

Figure 5

Download asset Open asset

We began with the BERT parse depth vector containing the parse depth of each word in an incremental input, providing a dynamic structural representation updated as the sentence unfolded. Then, we tested the interpretative mismatch between the incremental BERT parse depth vector and the corresponding context-free parse depth vector for the passive or the active interpretation. The degree of this mismatch is proportional to the evidence for or against the two interpretations, that is, the smaller the distance, the more positively loaded this interpretation. Besides these two measures based on the entire incremental input, we also focused on Verb1 since the potential structural ambiguity lies in whether Verb1 is interpreted as a passive verb or the main verb. Given the context-free parse depth of Verb1 that is 2 in the passive interpretation and 0 in the active interpretation (Figure 3A), with each incoming later word, an increased BERT Verb1 parse depth towards 2 or a decreased value towards 0 reflects separately the preference biased to a passive or an active interpretation (Figure 7—figure supplement 1). All quantitative measures tested in ssRSA are summarized in Supplementary file 1.

For the listener’s neural activity, we focused on three critical epochs in each sentence: (a) Verb1 – when its structural dependency with the preceding subject noun was initially established despite potential ambiguity; (b) the preposition – when the initial structural interpretation started being updated, to be either strengthened or weakened by the incoming preposition phrase; and (c) main verb – when the intended passive interpretation was finally confirmed. We aligned the continuous EMEG data to the onset of Verb1, the preposition, and the main verb respectively and obtained three 600 ms epochs.

We found that the incremental BERT parse depth vectors exhibited significant fits to brain activity consistently in all three epochs as the corresponding word was being heard at that time (Figure 6A–C). In Verb1 epoch, effects in bilateral frontal and anterior-to-middle temporal regions started immediately from Verb1 onset and continued until the uniqueness point – the point at which the word has been uniquely identified. These early effects could be due to the different subject nouns included in the BERT parse depth vectors. While the BERT parse depth of Verb1 per se showed similar effects but with greater duration which peaked exactly at Verb1 uniqueness point (Figure 6—figure supplement 3). As the sentence unfolded, effects of BERT parse depth vectors were found in the left fronto-temporal regions in the two later epochs, starting after the recognition of the preposition or the main verb separately.

Figure 6 with 6 supplements see all

Download asset Open asset

Turning to the interpretative mismatch for the two possible interpretations, we only observed significant effects of the mismatch for active interpretation in Verb1 epoch (Figure 6D). However, it was the mismatch for passive interpretation that fitted brain activity in the preposition and main verb epochs (Figure 6E and F, marginal significance in main verb epoch with cluster-wise p=0.06). These results suggest that listeners, in general, tended to have an initial preference for an active interpretation (even before the recognition of Verb1) but might start favouring a passive interpretation when the prepositional phrase began to be heard. This finding is consistent with the tendency to process the first noun encountered at the beginning of a sentence as the agent (Bever, 1970; Jackendoff, 2002; Mahowald et al., 2023). Note that our approach does not constitute a direct test for the hypothesis of parallel parsing, as we did not uncover evidence supporting the maintenance of parallel representations of different syntactic structures in the brain; rather, we only found one preferred structure in each epoch.

Effects of the BERT parse depth vectors and those of the interpretative mismatch for the preferred structural interpretation have substantial overlaps in terms of their spatiotemporal patterns in the brain, characterized primarily by a transition from bilateral to left-lateralized fronto-temporal regions as the sentence unfolds. Across the three epochs, the most sustained effects were observed in the left inferior frontal gyrus (IFG) and the anterior temporal lobe (ATL). Notably, with the identification of the actual main verb, effects of the eventually resolved structure also involved regions in the left prefrontal and inferior parietal regions (Figure 6C) which belong to the multiple-demand network (Duncan, 2010). The involvement of the prefrontal regions could be indicative of the varying working memory demands (e.g. the different number of open nodes in the sentence structures for active and passive interpretations before the actual main verb is recognized) for building the structure of the unfolding sentence (Nelson et al., 2017; Pallier et al., 2011).

Structural ambiguity resolution probed using BERT Verb1 parse depth

As mentioned above, the potential ambiguity between a passive and an active interpretation centres around whether Verb1 is considered as a passive verb or the main verb, which is resolved upon the appearance of the actual main verb. We probed how this is implemented in the brain using the dynamic BERT parse depth of Verb1. Specifically, the cognitive demands required by this resolution process can be characterized by the change between the updated BERT parse depth of Verb1 when the actual main verb is presented and its initial value when Verb1 is first encountered (see Figure 7—figure supplement 1 for the dynamic change of BERT Verb1 parse depth).

We first tested the change of Verb1 parse depth in the main verb epoch. Significant fits to brain activity emerged in the left posterior temporal and inferior parietal regions upon the main verb uniqueness point, and then extended to more anterior temporal regions (Figure 7A). After the main verb offset, the declining effects of the Verb1 parse depth change in the left anterior temporal region seamlessly overlapped with the arising effects of the updated Verb1 parse depth (Figure 7B and C). These results indicate that the recognition of the actual main verb immediately triggered an update of the previous interpretation of Verb1, with the resolved interpretation emerged in the left temporal lobe and was later delivered to the right posterior temporal and parietal areas. It is also worth noting that the left hippocampus was activated for both measures of Verb1 parse depth after the actual main verb is recognized, suggesting that the episodic memory of experienced events might contribute to the updating of structural interpretations (Bicknell et al., 2010; Altmann and Ekves, 2019; Metusalem et al., 2012). These results address the dynamic update of structured interpretation by focusing on the BERT parse depth of Verb1, which complements those of the interpretative mismatch based on the incremental BERT parse depth vector incorporating constraints of all the words heard so far (Figure 6F).

Figure 7 with 2 supplements see all

Download asset Open asset

Emergent structural interpretations driven by multifaceted constraints in the brain

Next, we further asked how the multifaceted constraints, considered by human listeners and encoded in BERT parse depths, drive the structured interpretation in the brain? When and where in the brain do these constraints emerge? How are their neural effects related to those of the final resolved sentential structure? To address these questions, we first tested the subject noun thematic role properties obtained from corpus data. Significant effects of agenthood and patienthood were found in the preposition epoch (Figure 8A) and in the main verb epoch (Figure 8B) separately. Notably, effects of the subject noun itself preceded those of incremental BERT parse depth vectors modelling the sentence fragments in the same epoch (compare Figure 8A with Figure 6B and Figure 8B with Figure 6C). These findings indicate that subject noun thematic role might be evaluated before building the overall structural interpretation of the utterance delivered so far. Specifically, the initial preference for an active interpretation during Verb1, while present as the preposition started (i.e. subject noun agenthood in Figure 8A), was superseded by the preference for a passive interpretation as the rest of the prepositional phrase (Figure 6A) and the main verb (Figure 6B) were heard.

Figure 8 with 3 supplements see all

Download asset Open asset

Despite being jointly constrained by subject noun thematic role preference and Verb1 transitivity in a probabilistic manner, the structural interpretation temporarily held just before the recognition of the actual main verb could differ across sentences (e.g. passive interpretation in “The dog found in the park …” and active interpretation in “The dog walked in the park …”). Therefore, in contrast to the passive or active index specialized for one particular structural interpretation, we constructed a non-directional index that merely quantifies the degree of interpretative coherence for one interpretation, whether passive or active (see ‘Materials and methods’ for details). Thus, a higher value only indicates greater interpretative coherence between the subject noun and Verb1 regardless of which interpretation is preferred.

Effects of this non-directional measure of interpretative coherence appeared very soon after the main verb onset in both hemispheres and lasted till its offset (Figure 8C), suggesting an immediate evaluation of the previously integrated constraints from the subject noun and Verb1 when a listener realized that the sentence had not finished yet. Moreover, these effects roughly co-occurred with the effects of subject noun patienthood (compare Figure 8B and C), indicating that a patient role for the subject noun was considered as the main verb was being recognized. Intriguingly, the most sustained regions associated with this non-directional index, including the left ATL, angular gyrus (AG), and precuneus, are also the classical areas of the DMN. This finding is consistent with recent claims that the DMN integrates external input with internal prior knowledge to make sense of an input stimulus such as speech (Yeshurun et al., 2021). In particular, precuneus and AG have been found to be involved in building thematic relationships and event structures (Baldassano et al., 2017; Humphreys et al., 2021).

Following the declining effects of the non-directional index upon the recognition of the main verb, we found significant effects of the passive index in right anterior fronto-temporal regions (Figure 8D), suggesting that the intended passive interpretation was eventually established in all sentences. Previous studies have revealed that the relatively narrow sentence-specific information and the broad world knowledge are processed in the left and right hemispheres separately (Jung-Beeman, 2005; Metusalem et al., 2016; Troyer et al., 2022). Relevant to this, in the main verb epoch, we found effects of the BERT parse depth vector and those of the passive index in the left and right hemispheres, respectively, arising almost at the same time as the main verb was recognized (compare Figure 6C and Figure 8D). Therefore, a critical question is whether and how the online structural interpretation of a specific sentence is facilitated by the interpretative coherence conjured up from lexical constraints that also depend on broad world knowledge (e.g. thematic role).

To address this question, we adopted non-negative matrix factorization to decompose the whole-brain ssRSA fits of the passive index and the BERT parse depth vector found in the main verb epoch into two sets of components given their temporal synchronizations (see ‘Materials and methods’). We then conducted multivariate Granger causality analyses (GCA) to infer directed connections among them. We found only GC connections from the components of passive index to those of BERT parse depth vector (Figure 8E). Specifically, we identified information flows from the right hemisphere components of the passive index to the left hemisphere components of BERT parse depth vector, suggesting that a sentence’s structure represented in the left hemisphere might be influenced by the coarse estimate of the event plausibility concurrently determined by broad world knowledge in the right hemisphere (Jung-Beeman, 2005; see Figure 8—figure supplement 2 for more details).

In this study, we investigated the neural dynamics involved in constructing structured interpretations from speech. We combined spatiotemporally resolved brain activity of human listeners, quantitative structural representations derived from a DLM (i.e. BERT), and corpus-based and behavioural measures. Our study revealed the emergence and update of a structured interpretation, jointly constrained by different lexical properties related to both linguistic and non-linguistic world knowledge, in an extensive set of brain regions beyond the core fronto-temporal language network. Specifically, our results show (1) a shift from bi-hemispheric lateral frontal-temporal regions to left-lateralized regions in representing the current structured interpretation as a sentence unfolds, (2) a pattern of sequential activations in the left lateral temporal regions, updating the structured interpretation as syntactic ambiguity is resolved, and (3) the influence of lexical interpretative coherence activated in the right hemisphere over the resolved sentence structure represented in the left hemisphere. These findings provide empirical evidence for the constraint-based approach to sentence processing and deepen the understanding of specific spatiotemporal patterning and neuro-computational properties underpinning incremental speech comprehension.

Using artificial neural networks (ANNs) to study the neural substrates of human cognition complements the long-time pursuit of generative rules and interpretable models (Kriegeskorte and Douglas, 2018). ANNs have informed our understanding of various cognitive processes in the brain by providing quantifiable predictions that aim to connect behaviours and relevant neural activity (Yamins and DiCarlo, 2016; Rabovsky et al., 2018; Donhauser and Baillet, 2020; Yang et al., 2019; Kietzmann et al., 2019; Bao et al., 2020; Sheahan et al., 2021; Giordano et al., 2023; Doerig et al., 2023). This is crucial for quantifying the outcome of complex, interrelated constraints that arise in specific contexts, such as spoken sentences, and constructing the representational geometry to be probed in the brain. Where DLMs are concerned, recent studies have systematically compared the internal representations of DLMs to those observed in the human brain during language processing, which highlights the importance of predictive coding and contextual information (Schrimpf et al., 2021; Goldstein et al., 2022; Heilbron et al., 2022; Toneva et al., 2022; Caucheteux et al., 2022; Caucheteux and King, 2022; Caucheteux et al., 2023). Furthermore, these studies have motivated the use of DLMs as a computational tool, or hypothesis, to study the neural substrates of language.

Here we asked a more specific question, that is, how a sequence of spoken words is incrementally structured and coherently interpreted in the brain? Our goal was to use quantitative measures of sentence structure that capture the interplay between different types of constraints that simultaneously influence this process. As a potential solution, we extracted detailed structural measures specific to the contents in each sentence from the hidden states of BERT, which was trained on massive corpora from real-life language use. Although DLMs such as BERT are not specifically designed to parse sentences, they can learn from training corpora the multi-dimensional properties related to sentence structure and dependency (Manning et al., 2020). In line with this, our analyses confirmed that BERT structural measures incorporate relevant lexical constraints and that they exhibit both behavioural and neural alignments with human listeners.

Taking advantage of the contextualized BERT structural measures, our ssRSA results provide neural evidence for the construction of a coherent interpretation driven by the interaction between linguistic and non-linguistic knowledge evoked by individual words as they are heard sequentially in a spoken sentence. Specifically, neural representations of an unfolding sentence’s structure initially emerged in bilateral fronto-temporal regions and became left-lateralized when more complex syntactic properties, rather than canonical linear adjacency, were considered to build a structured interpretation (e.g. beyond Verb1 in our stimulus sentences). Meanwhile, we found right-hemisphere activations associated with broad world knowledge, which is essential for understanding the intended meaning conveyed by the speaker (Bicknell et al., 2010). In addition to the core fronto-temporal language network, we found that the multiple-demand network and the DMN were also involved during online construction of structured interpretations, which may reflect additional cognitive demands for resolving potential structural ambiguity and evaluating the plausibility of underlying events (Smallwood et al., 2021).

Moreover, our results show that, compared to corpus-based and behavioural measures, BERT structural measures are more effective in fitting listeners’ neural activity, possibly due to its advanced ability in modelling specific contexts within each sentence. Nevertheless, it is important to recognize the important role of corpus-based and behavioural measures as explanatory variables. They are crucial not only in interpreting BERT measures but also in understanding their alignment with listeners’ neural activity. This includes, for instance, the temporal sequence of activations of key lexical constraints and the emerging structure of a sentence (e.g. effects of subject noun patienthood leading those of BERT parse depth vector in the MV epoch, as seen in Figures 8B and 6C) and the spatial distribution of their model fits in the brain (e.g. contrasting model fits of passive index and BERT parse depth vector in the MV epoch across different hemispheres, as shown in Figures 8D and 6C). Such an integrative approach allows for a more comprehensive understanding of the complex mental processes underpinning speech comprehension, which takes advantages of the interpretability of traditional measures and the deep contextualized representations of DLMs.

There are two points to note about the use of BERT. Firstly, unlike autoregressive DLMs trained using left-to-right attention and the next-word prediction task, BERT is trained to predict masked words in a sentence with a bidirectional attention mechanism. The additional right-to-left attention provides updated representations of preceding words every time an incoming word is added to the input (e.g. representation of ‘dog’ in “The dog …” is different from that in “The dog found …”). This feature of BERT is useful for tracking the dynamic change of the representation of a specific word as its context evolves (e.g. Verb1 in this study), particularly in sentences with structural ambiguity. Although autoregressive DLMs also update hidden states as the input unfolds and could be used to study complex sentential structures (Jurayj et al., 2022), the updated contextual effects are reflected in the hidden states of the right-most incoming word, while those of the preceding words on the left remain unchanged (i.e. the representation of ‘dog’ is the same in “The dog …” and “The dog found …”). This is different from BERT, where the updated contextual effects are reflected in the hidden states of all preceding words.

Secondly, although we input each sentence word-by-word to BERT, however, unlike human listeners or recurrent neural networks, BERT process two consecutive inputs (e.g. “The dog …” and “The dog found …”) independently, and there is no direct relationship between the hidden states of these two inputs. In fact, human listeners would not start over from the beginning of a sentence as it unfolds word-by-word, but update it continually as each word is heard and use whatever information currently available to build a coherent interpretation (McRae and Matsuki, 2013). This difference, however, does not impede our objective of extracting contextualized structural representations at critical points within a sentence. In the case of BERT, the representation of each word is continuously updated in a bidirectional manner as a new word is added. This process accounts for the constraints imposed by all the words of the input and their interactions, forming a coherent interpretation. Nonetheless, for other hypotheses in speech comprehension, such as parallel parsing and how various grammatically correct sentence structures are maintained and compete in the brain, DLMs with recurrent memory might be more suitable. Such models can better stimulate the continuous, dynamic updating of interpretations that characterizes human sentence processing.

In summary, recent developments in DLMs have shown great potential in capturing the dynamic interplay between syntax, semantics, and broader world knowledge that is essential for successful language comprehension. The empirical evidence from this study supports the notion that DLMs, when utilized as potential models of brain computation and integrated with advanced neuroimaging techniques within a well-defined framework, can offer significant insights into human cognition (Kriegeskorte and Douglas, 2018; Doerig et al., 2023). Future DLMs, especially those with more human-like model architecture (McClelland et al., 2020) and subjected to rigorous evaluation (Binz and Schulz, 2023), hold the potential to shed light on the neural implementation of various incremental processes that support the rapid transition from sound to meaning in the brain.

Preprocessed E/MEG data, scripts and relevant materials are available in the following repository https://osf.io/7u8jp/.

1. 1. Lyu B
   2. Marslen-Wilson WD
   3. Fang Y
   4. Tyler LK

   (2024) Open Science Framework

   Finding structure during incremental speech comprehension.

   https://doi.org/10.17605/OSF.IO/7U8JP

1. 1. Altmann GT

   (1998) Ambiguity in sentence processing

   Trends in Cognitive Sciences 2:146–152.

   https://doi.org/10.1016/S1364-6613(98)01153-X

   • Google Scholar
2. 1. Altmann GTM
   2. Ekves Z

   (2019) Events as intersecting object histories: A new theory of event representation

   Psychological Review 126:817–840.

   https://doi.org/10.1037/rev0000154

   • PubMed
   • Google Scholar
3. Book

   1. Baayen RH

   (1993)

   The CELEX lexical data base on CD-ROM

   Academia Press.

   • Google Scholar
4. 1. Baldassano C
   2. Chen J
   3. Zadbood A
   4. Pillow JW
   5. Hasson U
   6. Norman KA

   (2017) Discovering event structure in continuous narrative perception and memory

   Neuron 95:709–721.

   https://doi.org/10.1016/j.neuron.2017.06.041

   • PubMed
   • Google Scholar
5. 1. Bao P
   2. She L
   3. McGill M
   4. Tsao DY

   (2020) A map of object space in primate inferotemporal cortex

   Nature 583:103–108.

   https://doi.org/10.1038/s41586-020-2350-5

   • PubMed
   • Google Scholar
6. 1. Barnett L
   2. Seth AK

   (2014) The MVGC multivariate Granger causality toolbox: a new approach to Granger-causal inference

   Journal of Neuroscience Methods 223:50–68.

   https://doi.org/10.1016/j.jneumeth.2013.10.018

   • PubMed
   • Google Scholar
7. 1. Baroni M
   2. Bernardini S
   3. Ferraresi A
   4. Zanchetta E

   (2009) The WaCky wide web: a collection of very large linguistically processed web-crawled corpora

   Language Resources and Evaluation 43:209–226.

   https://doi.org/10.1007/s10579-009-9081-4

   • Google Scholar
8. 1. Bengio Y
   2. Lecun Y
   3. Hinton G

   (2021) Deep learning for AI

   Communications of the ACM 64:58–65.

   https://doi.org/10.1145/3448250

   • Google Scholar
9. Book

   1. Bever TG

   (1970)

   The cognitive basis for linguistic structures

   In: Hayes JR, editors. Cognition and the Development of Language. John Wiley. pp. 55–518.

   • Google Scholar
10. 1. Bicknell K
    2. Elman JL
    3. Hare M
    4. McRae K
    5. Kutas M

    (2010) Effects of event knowledge in processing verbal arguments

    Journal of Memory and Language 63:489–505.

    https://doi.org/10.1016/j.jml.2010.08.004

    • PubMed
    • Google Scholar
11. 1. Binz M
    2. Schulz E

    (2023) Using cognitive psychology to understand GPT-3

    PNAS 120:e2218523120.

    https://doi.org/10.1073/pnas.2218523120

    • PubMed
    • Google Scholar
12. Conference

    1. Bisk Y
    2. Holtzman A
    3. Thomason J
    4. Andreas J
    5. Bengio Y
    6. Chai J
    7. Lapata M
    8. Lazaridou A
    9. May J
    10. Nisnevich A
    11. Pinto N
    12. Turian J

    (2020) Experience Grounds Language

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 8718–8735.

    https://doi.org/10.18653/v1/2020.emnlp-main.703

    • Google Scholar
13. Conference

    1. Brown T

    (2020) Language models are few-shot learners

    Advances in Neural Information Processing Systems. pp. 1877–1901.

    https://doi.org/10.48550/arXiv.2005.14165

    • Google Scholar
14. 1. Caucheteux C
    2. Gramfort A
    3. King JR

    (2022) Deep language algorithms predict semantic comprehension from brain activity

    Scientific Reports 12:16327.

    https://doi.org/10.1038/s41598-022-20460-9

    • PubMed
    • Google Scholar
15. 1. Caucheteux C
    2. King JR

    (2022) Brains and algorithms partially converge in natural language processing

    Communications Biology 5:134.

    https://doi.org/10.1038/s42003-022-03036-1

    • PubMed
    • Google Scholar
16. 1. Caucheteux C
    2. Gramfort A
    3. King JR

    (2023) Evidence of a predictive coding hierarchy in the human brain listening to speech

    Nature Human Behaviour 7:430–441.

    https://doi.org/10.1038/s41562-022-01516-2

    • PubMed
    • Google Scholar
17. 1. Choi HS
    2. Marslen-Wilson WD
    3. Lyu B
    4. Randall B
    5. Tyler LK

    (2021) Decoding the real-time neurobiological properties of incremental semantic interpretation

    Cerebral Cortex 31:233–247.

    https://doi.org/10.1093/cercor/bhaa222

    • PubMed
    • Google Scholar
18. Conference

    1. Devlin J
    2. Chang MW
    3. Lee K
    4. Toutanova K

    (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4171–4186.

    https://doi.org/10.18653/v1/N19-1423

    • Google Scholar
19. 1. Doerig A
    2. Sommers RP
    3. Seeliger K
    4. Richards B
    5. Ismael J
    6. Lindsay GW
    7. Kording KP
    8. Konkle T
    9. van Gerven MAJ
    10. Kriegeskorte N
    11. Kietzmann TC

    (2023) The neuroconnectionist research programme

    Nature Reviews. Neuroscience 24:431–450.

    https://doi.org/10.1038/s41583-023-00705-w

    • PubMed
    • Google Scholar
20. 1. Donhauser PW
    2. Baillet S

    (2020) Two distinct neural timescales for predictive speech processing

    Neuron 105:385–393.

    https://doi.org/10.1016/j.neuron.2019.10.019

    • PubMed
    • Google Scholar
21. 1. Dowty D

    (1991) Thematic proto-roles and argument selection

    Language 67:547–619.

    https://doi.org/10.1353/lan.1991.0021

    • Google Scholar
22. 1. Duncan J

    (2010) The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour

    Trends in Cognitive Sciences 14:172–179.

    https://doi.org/10.1016/j.tics.2010.01.004

    • PubMed
    • Google Scholar
23. 1. Elman JL

    (1990) Finding structure in time

    Cognitive Science 14:179–211.

    https://doi.org/10.1016/0364-0213(90)90002-E

    • Google Scholar
24. 1. Elman JL

    (1993) Learning and development in neural networks: the importance of starting small

    Cognition 48:71–99.

    https://doi.org/10.1016/0010-0277(93)90058-4

    • PubMed
    • Google Scholar
25. 1. Everaert MBH
    2. Huybregts MAC
    3. Chomsky N
    4. Berwick RC
    5. Bolhuis JJ

    (2015) Structures, not srings: linguistics as part of the cognitive science

    Trends in Cognitive Sciences 19:729–743.

    https://doi.org/10.1016/j.tics.2015.09.008

    • PubMed
    • Google Scholar
26. 1. Frazier L
    2. Rayner K

    (1982) Making and correcting errors during sentence comprehension: eye movements in the analysis of structurally ambiguous sentences

    Cognitive Psychology 14:178–210.

    https://doi.org/10.1016/0010-0285(82)90008-1

    • Google Scholar
27. 1. Frazier L

    (1987) Syntactic processing: evidence from dutch

    Natural Language and Linguistic Theory 5:519–559.

    https://doi.org/10.1007/BF00138988

    • Google Scholar
28. 1. Friederici AD
    2. Bahlmann J
    3. Heim S
    4. Schubotz RI
    5. Anwander A

    (2006) The brain differentiates human and non-human grammars: functional localization and structural connectivity

    PNAS 103:2458–2463.

    https://doi.org/10.1073/pnas.0509389103

    • PubMed
    • Google Scholar
29. 1. Friederici AD

    (2012) The cortical language circuit: from auditory perception to sentence comprehension

    Trends in Cognitive Sciences 16:262–268.

    https://doi.org/10.1016/j.tics.2012.04.001

    • PubMed
    • Google Scholar
30. 1. Giordano BL
    2. Esposito M
    3. Valente G
    4. Formisano E

    (2023) Intermediate acoustic-to-semantic representations link behavioral and neural responses to natural sounds

    Nature Neuroscience 26:664–672.

    https://doi.org/10.1038/s41593-023-01285-9

    • PubMed
    • Google Scholar
31. 1. Goldstein A
    2. Zada Z
    3. Buchnik E
    4. Schain M
    5. Price A
    6. Aubrey B
    7. Nastase SA
    8. Feder A
    9. Emanuel D
    10. Cohen A
    11. Jansen A
    12. Gazula H
    13. Choe G
    14. Rao A
    15. Kim C
    16. Casto C
    17. Fanda L
    18. Doyle W
    19. Friedman D
    20. Dugan P
    21. Melloni L
    22. Reichart R
    23. Devore S
    24. Flinker A
    25. Hasenfratz L
    26. Levy O
    27. Hassidim A
    28. Brenner M
    29. Matias Y
    30. Norman KA
    31. Devinsky O
    32. Hasson U

    (2022) Shared computational principles for language processing in humans and deep language models

    Nature Neuroscience 25:369–380.

    https://doi.org/10.1038/s41593-022-01026-4

    • Google Scholar
32. 1. Guggenmos M
    2. Sterzer P
    3. Cichy RM

    (2018) Multivariate pattern analysis for MEG: a comparison of dissimilarity measures

    NeuroImage 173:434–447.

    https://doi.org/10.1016/j.neuroimage.2018.02.044

    • PubMed
    • Google Scholar
33. 1. Hämäläinen MS
    2. Ilmoniemi RJ

    (1994) Interpreting magnetic fields of the brain: minimum norm estimates

    Medical & Biological Engineering & Computing 32:35–42.

    https://doi.org/10.1007/BF02512476

    • PubMed
    • Google Scholar
34. 1. Heilbron M
    2. Armeni K
    3. Schoffelen JM
    4. Hagoort P
    5. de Lange FP

    (2022) A hierarchy of linguistic predictions during natural language comprehension

    PNAS 119:e2201968119.

    https://doi.org/10.1073/pnas.2201968119

    • PubMed
    • Google Scholar
35. 1. Henson RN
    2. Mouchlianitis E
    3. Friston KJ

    (2009) MEG and EEG data fusion: simultaneous localisation of face-evoked responses

    NeuroImage 47:581–589.

    https://doi.org/10.1016/j.neuroimage.2009.04.063

    • PubMed
    • Google Scholar
36. Conference

    1. Hewitt J
    2. Liang P

    (2019) Designing and Interpreting Probes with Control Tasks

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 2733–2743.

    https://doi.org/10.18653/v1/D19-1275

    • Google Scholar
37. Conference

    1. Hewitt J
    2. Manning CD

    (2019) A structural probe for finding syntax in word representations

    Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4129–4138.

    https://doi.org/10.18653/v1/N19-1419

    • Google Scholar
38. 1. Humphreys GF
    2. Lambon Ralph MA
    3. Simons JS

    (2021) A unifying account of angular gyrus contributions to episodic and semantic cognition

    Trends in Neurosciences 44:452–463.

    https://doi.org/10.1016/j.tins.2021.01.006

    • PubMed
    • Google Scholar
39. Book

    1. Jackendoff RS

    (2002) Foundations of Language: Brain, Meaning, Grammar, Evolution

    Academic Press.

    https://doi.org/10.1093/acprof:oso/9780198270126.001.0001

    • Google Scholar
40. 1. Jung-Beeman M

    (2005) Bilateral brain processes for comprehending natural language

    Trends in Cognitive Sciences 9:512–518.

    https://doi.org/10.1016/j.tics.2005.09.009

    • PubMed
    • Google Scholar
41. Conference

    1. Jurayj W
    2. Rudman W
    3. Eickhoff C

    (2022) Garden Path Traversal in GPT-2

    Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. pp. 305–313.

    https://doi.org/10.18653/v1/2022.blackboxnlp-1.25

    • Google Scholar
42. 1. Kietzmann TC
    2. Spoerer CJ
    3. Sörensen LKA
    4. Cichy RM
    5. Hauk O
    6. Kriegeskorte N

    (2019) Recurrence is required to capture the representational dynamics of the human visual system

    PNAS 116:21854–21863.

    https://doi.org/10.1073/pnas.1905544116

    • PubMed
    • Google Scholar
43. 1. Klimovich-Gray A
    2. Tyler LK
    3. Randall B
    4. Kocagoncu E
    5. Devereux B
    6. Marslen-Wilson WD

    (2019) Balancing prediction and sensory input in speech comprehension: The spatiotemporal dynamics of word recognition in context

    The Journal of Neuroscience 39:519–527.

    https://doi.org/10.1523/JNEUROSCI.3573-17.2018

    • PubMed
    • Google Scholar
44. 1. Kocagoncu E
    2. Clarke A
    3. Devereux BJ
    4. Tyler LK

    (2017) Decoding the cortical dynamics of sound-meaning mapping

    The Journal of Neuroscience 37:1312–1319.

    https://doi.org/10.1523/JNEUROSCI.2858-16.2016

    • PubMed
    • Google Scholar
45. Conference

    1. Korhonen A
    2. Krymolowski Y
    3. Briscoe T

    (2006)

    A large subcategorization lexicon for natural language processing applications

    Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). pp. 1015–1020.

    • Google Scholar
46. 1. Kriegeskorte N
    2. Mur M
    3. Bandettini P

    (2008) Representational similarity analysis - connecting the branches of systems neuroscience

    Frontiers in Systems Neuroscience 2:4.

    https://doi.org/10.3389/neuro.06.004.2008

    • PubMed
    • Google Scholar
47. 1. Kriegeskorte N
    2. Douglas PK

    (2018) Cognitive computational neuroscience

    Nature Neuroscience 21:1148–1160.

    https://doi.org/10.1038/s41593-018-0210-5

    • PubMed
    • Google Scholar
48. 1. Kuperberg GR

    (2007) Neural mechanisms of language comprehension: challenges to syntax

    Brain Research 1146:23–49.

    https://doi.org/10.1016/j.brainres.2006.12.063

    • PubMed
    • Google Scholar
49. 1. Law R
    2. Pylkkänen L

    (2021) Lists with and without syntax: a new approach to measuring the neural processing of syntax

    The Journal of Neuroscience 41:2186–2196.

    https://doi.org/10.1523/JNEUROSCI.1179-20.2021

    • PubMed
    • Google Scholar
50. Conference

    1. Li T
    2. Jawale PA
    3. Palmer M
    4. Srikumar V

    (2020) Structured Tuning for Semantic Role Labeling

    Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

    https://doi.org/10.18653/v1/2020.acl-main.744

    • Google Scholar
51. 1. Linzen T
    2. Baroni M

    (2021) Syntactic structure from deep learning

    Annual Review of Linguistics 7:195–212.

    https://doi.org/10.1146/annurev-linguistics-032020-051035

    • Google Scholar
52. 1. Lyu B
    2. Choi HS
    3. Marslen-Wilson WD
    4. Clarke A
    5. Randall B
    6. Tyler LK

    (2019) Neural dynamics of semantic composition

    PNAS 116:21318–21327.

    https://doi.org/10.1073/pnas.1903402116

    • PubMed
    • Google Scholar
53. Conference

    1. MacCartney B
    2. Manning CD

    (2006)

    Generating typed dependency parses from phrase structure parses

    Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). pp. 449–454.

    • Google Scholar
54. 1. MacDonald MC
    2. Pearlmutter NJ
    3. Seidenberg MS

    (1994) The lexical nature of syntactic ambiguity resolution [corrected]

    Psychological Review 101:676–703.

    https://doi.org/10.1037/0033-295x.101.4.676

    • PubMed
    • Google Scholar
55. 1. Mahowald K
    2. Diachek E
    3. Gibson E
    4. Fedorenko E
    5. Futrell R

    (2023) Grammatical cues to subjecthood are redundant in a majority of simple clauses across languages

    Cognition 241:105543.

    https://doi.org/10.1016/j.cognition.2023.105543

    • PubMed
    • Google Scholar
56. 1. Manning CD
    2. Clark K
    3. Hewitt J
    4. Khandelwal U
    5. Levy O

    (2020) Emergent linguistic structure in artificial neural networks trained by self-supervision

    PNAS 117:30046–30054.

    https://doi.org/10.1073/pnas.1907367117

    • PubMed
    • Google Scholar
57. 1. Marcus MP
    2. Santorini B
    3. Marcinkiewicz MA

    (1993)

    Building a large annotated corpus of english: the penn treebank

    Computational Linguistics 19:313–330.

    • Google Scholar
58. 1. Marslen-Wilson WD

    (1975) Sentence perception as an interactive parallel process

    Science 189:226–228.

    https://doi.org/10.1126/science.189.4198.226

    • PubMed
    • Google Scholar
59. 1. Marslen-Wilson W
    2. Tyler LK

    (1980) The temporal structure of spoken language understanding

    Cognition 8:1–71.

    https://doi.org/10.1016/0010-0277(80)90015-3

    • PubMed
    • Google Scholar
60. 1. Marslenwilson WD
    2. Tyler LK
    3. Koster C

    (1993) Integrative processes in utterance resolution

    Journal of Memory and Language 32:647–666.

    https://doi.org/10.1006/jmla.1993.1033

    • Google Scholar
61. 1. Matchin W
    2. Hickok G

    (2020) The cortical organization of syntax

    Cerebral Corte 30:1481–1498.

    https://doi.org/10.1093/cercor/bhz180

    • PubMed
    • Google Scholar
62. 1. McClelland JL
    2. Hill F
    3. Rudolph M
    4. Baldridge J
    5. Schütze H

    (2020) Placing language in an integrated understanding system: next steps toward human-level performance in neural language models

    PNAS 117:25966–25974.

    https://doi.org/10.1073/pnas.1910416117

    • PubMed
    • Google Scholar
63. Book

    1. McRae K
    2. Matsuki K

    (2013)

    Constraint-based models of sentence processing

    In: van Gompel RPG, editors. Current Issues in the Psychology of Language. Sentence Processing. Psychology Press. pp. 51–77.

    • Google Scholar
64. 1. Metusalem R
    2. Kutas M
    3. Urbach TP
    4. Hare M
    5. McRae K
    6. Elman JL

    (2012) Generalized event knowledge activation during online sentence comprehension

    Journal of Memory and Language 66:545–567.

    https://doi.org/10.1016/j.jml.2012.01.001

    • PubMed
    • Google Scholar
65. 1. Metusalem R
    2. Kutas M
    3. Urbach TP
    4. Elman JL

    (2016) Hemispheric asymmetry in event knowledge activation during incremental language comprehension: a visual half-field ERP study

    Neuropsychologia 84:252–271.

    https://doi.org/10.1016/j.neuropsychologia.2016.02.004

    • PubMed
    • Google Scholar
66. 1. Mosher JC
    2. Leahy RM
    3. Lewis PS

    (1999) EEG and MEG: forward solutions for inverse methods

    IEEE Transactions on Bio-Medical Engineering 46:245–259.

    https://doi.org/10.1109/10.748978

    • PubMed
    • Google Scholar
67. Conference

    1. Mrini K
    2. Dernoncourt F
    3. Tran QH
    4. Bui T
    5. Chang W
    6. Nakashole N

    (2020) Rethinking Self-Attention: Towards Interpretability in Neural Parsing

    Findings of the Association for Computational Linguistics.

    https://doi.org/10.18653/v1/2020.findings-emnlp.65

    • Google Scholar
68. 1. Nelson MJ
    2. El Karoui I
    3. Giber K
    4. Yang X
    5. Cohen L
    6. Koopman H
    7. Cash SS
    8. Naccache L
    9. Hale JT
    10. Pallier C
    11. Dehaene S

    (2017) Neurophysiological dynamics of phrase-structure building during sentence processing

    PNAS 114:E3669–E3678.

    https://doi.org/10.1073/pnas.1701590114

    • PubMed
    • Google Scholar
69. Conference

    1. Ouyang L
    2. Wu J
    3. Jiang X
    4. Almeida D
    5. Wainwright C
    6. Mishkin P
    7. Zhang C
    8. Agarwal S
    9. Slama K
    10. Ray A
    11. Schulman J
    12. Hilton J
    13. Kelton F
    14. Miller L
    15. Simens M
    16. Askell A
    17. Welinder P
    18. Christiano PF
    19. Leike J
    20. Lowe L

    (2022)

    Training language models to follow instructions with human feedback

    36th Conference on Neural Information Processing Systems (NeurIPS 2022).

    • Google Scholar
70. 1. Pallier C
    2. Devauchelle AD
    3. Dehaene S

    (2011) Cortical representation of the constituent structure of sentences

    PNAS 108:2522–2527.

    https://doi.org/10.1073/pnas.1018711108

    • PubMed
    • Google Scholar
71. 1. Pavlick E

    (2022) Semantic structure in deep learning

    Annual Review of Linguistics 8:447–471.

    https://doi.org/10.1146/annurev-linguistics-031120-122924

    • Google Scholar
72. 1. Rabovsky M
    2. Hansen SS
    3. McClelland JL

    (2018) Modelling the N400 brain potential as change in a probabilistic representation of meaning

    Nature Human Behaviour 2:693–705.

    https://doi.org/10.1038/s41562-018-0406-4

    • PubMed
    • Google Scholar
73. 1. Sarvas J

    (1987) Basic mathematical and electromagnetic concepts of the biomagnetic inverse problem

    Physics in Medicine and Biology 32:11–22.

    https://doi.org/10.1088/0031-9155/32/1/004

    • PubMed
    • Google Scholar
74. 1. Schrimpf M
    2. Blank IA
    3. Tuckute G
    4. Kauf C
    5. Hosseini EA
    6. Kanwisher N
    7. Tenenbaum JB
    8. Fedorenko E

    (2021) The neural architecture of language: integrative modeling converges on predictive processing

    PNAS 118:e2105646118.

    https://doi.org/10.1073/pnas.2105646118

    • PubMed
    • Google Scholar
75. 1. Sheahan H
    2. Luyckx F
    3. Nelli S
    4. Teupe C
    5. Summerfield C

    (2021) Neural state space alignment for magnitude generalization in humans and recurrent networks

    Neuron 109:1214–1226.

    https://doi.org/10.1016/j.neuron.2021.02.004

    • PubMed
    • Google Scholar
76. 1. Smallwood J
    2. Bernhardt BC
    3. Leech R
    4. Bzdok D
    5. Jefferies E
    6. Margulies DS

    (2021) The default mode network in cognition: a topographical perspective

    Nature Reviews. Neuroscience 22:503–513.

    https://doi.org/10.1038/s41583-021-00474-4

    • PubMed
    • Google Scholar
77. 1. Taulu S
    2. Simola J

    (2006) Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements

    Physics in Medicine and Biology 51:1759–1768.

    https://doi.org/10.1088/0031-9155/51/7/008

    • Google Scholar
78. Conference

    1. Tenney I

    (2019)

    What do you learn from context? Probing for sentence structure in contextualized word representations

    7th International Conference on Learning Representations.

    • Google Scholar
79. 1. Toneva M
    2. Mitchell TM
    3. Wehbe L

    (2022) Combining computational controls with natural text reveals aspects of meaning composition

    Nature Computational Science 2:745–757.

    https://doi.org/10.1038/s43588-022-00354-6

    • PubMed
    • Google Scholar
80. 1. Troyer M
    2. McRae K
    3. Kutas M

    (2022) Wrong or right? Brain potentials reveal hemispheric asymmetries to semantic relations during word-by-word sentence reading as a function of (fictional) knowledge

    Neuropsychologia 170:108215.

    https://doi.org/10.1016/j.neuropsychologia.2022.108215

    • PubMed
    • Google Scholar
81. Book

    1. Trueswell JC
    2. Tanenhaus MK

    (1994)

    Toward a Lexicalist framework of constraint-based syntactic ambiguity resolution

    In: Trueswell JC, editors. Perspectives on Sentence Processing. Routlege. pp. 1–25.

    • Google Scholar
82. 1. Tyler LK
    2. Marslen-Wilson WD

    (1977) The on-line effects of semantic context on syntactic processing

    Journal of Verbal Learning and Verbal Behavior 16:683–692.

    https://doi.org/10.1016/S0022-5371(77)80027-3

    • Google Scholar
83. Preprint

    1. Vaswani A

    (2017) Attention is all you need

    arXiv.

    https://doi.org/10.48550/arXiv.1706.03762

    • Google Scholar
84. 1. Winkler AM
    2. Ridgway GR
    3. Douaud G
    4. Nichols TE
    5. Smith SM

    (2016) Faster permutation inference in brain imaging

    NeuroImage 141:502–516.

    https://doi.org/10.1016/j.neuroimage.2016.05.068

    • PubMed
    • Google Scholar
85. Preprint

    1. Wolf T

    (2019) Huggingface’s transformers: state-of-the-art natural language processing

    arXiv.

    https://doi.org/10.48550/arXiv.1910.03771

    • Google Scholar
86. 1. Yamins DLK
    2. DiCarlo JJ

    (2016) Using goal-driven deep learning models to understand sensory cortex

    Nature Neuroscience 19:356–365.

    https://doi.org/10.1038/nn.4244

    • PubMed
    • Google Scholar
87. 1. Yang GR
    2. Joglekar MR
    3. Song HF
    4. Newsome WT
    5. Wang XJ

    (2019) Task representations in neural networks trained to perform many cognitive tasks

    Nature Neuroscience 22:297–306.

    https://doi.org/10.1038/s41593-018-0310-2

    • PubMed
    • Google Scholar
88. 1. Yeshurun Y
    2. Nguyen M
    3. Hasson U

    (2021) The default mode network: where the idiosyncratic self meets the shared social world

    Nature Reviews. Neuroscience 22:181–192.

    https://doi.org/10.1038/s41583-020-00420-w

    • PubMed
    • Google Scholar

Author details

1. Bingjiang Lyu

   Changping Laboratory, Beijing, China

   Contribution

   Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Visualization, Writing – original draft, Writing – review and editing

   For correspondence

   bingjiang.lyu@gmail.com

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8554-5138

2. William D Marslen-Wilson

   Centre for Speech, Language and the Brain, Department of Psychology, University of Cambridge, Cambridge, United Kingdom

   Contribution

   Conceptualization, Writing – original draft, Methodology

   Competing interests

   No competing interests declared

3. Yuxing Fang

   Centre for Speech, Language and the Brain, Department of Psychology, University of Cambridge, Cambridge, United Kingdom

   Contribution

   Supervision, Writing – review and editing

   Competing interests

   No competing interests declared

4. Lorraine K Tyler

   Centre for Speech, Language and the Brain, Department of Psychology, University of Cambridge, Cambridge, United Kingdom

   Contribution

   Conceptualization, Writing – original draft, Data curation, Project administration, Methodology

   For correspondence

   lkt10@cam.ac.uk

   Competing interests

   No competing interests declared

• 807

  views

• 99

  downloads

• 3

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.