Large language models poorly capture how word order shapes representations of sentence meaning in the human brain
- The Centre for Brain, Mind and Markets, Faculty of Business and Economics, The University of Melbourne, Australia
- Faculty of Social Data Science, Hitotsubashi University, Japan
Figures
Figure 1
Summary of study methods for constructing stimuli, computing model representations, and collecting fMRI and behavioural data.
(a) We construct 108 handcrafted sentences, designed to enable systematic variation in sentence meaning while controlling for lexical similarity. Here, we show the corresponding 108×108 design matrix colour-coded with the type of each sentence pair. Sentence pairs in the six blocks along the diagonal are the primary pairs of interest in this study. (b) All sentences were encoded using each of the four computational models of sentence meaning which we examine in this study. (c) We then computed representational similarity matrices of the 108 stimuli for each of the four models. More similar sentence pairs are shown in blue, and less similar in red. (d) Study pipeline for the fMRI experiment, in which participants were presented one sentence at a time for 2–7 s depending on sentence length. Multiple choice comprehension questions were interspersed randomly to assess attention. After scanning, data was processed and brain activity patterns were used to compute a neural representational similarity matrix for each participant. Correlations were then computed between the model and brain representational similarity analysis (RSA) matrices. (e) Study pipeline for behavioural experiment, in which online participants were each shown 112 sentence pairs and asked to rate their semantic similarity. Ratings were averaged over participants to compute a similarity matrix. The correlation was then computed between the model and behavioural RSA matrices.
Figure 2
Model correlations with brain activity for all sentence pairs and the block diagonal subset of sentence pairs.
Partial correlations between representational similarity analysis (RSA) matrices of five computational models (Random, Mean, Transformer, VerbNet-CN, and AMR-Smatch) and the brain RSA matrix, controlling for differences in sentence length. ‘Human’ refers to behavioural ratings. Blue bars show correlations computed over all stable voxels (excluding visual regions V1–V4), while orange bars show correlations for stable voxels in the language network. Notation for statistical significance: * for p<0.05, ** for p<0.01, and *** for p<0.001, with Bonferroni correction for three independent comparisons. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all four computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown for comparison, regression lines (red) are computed over the block diagonal pairs only.
Figure 3
Comparison of sentence representations and model correlations across brain regions.
(a) Representational similarity analysis (RSA) matrices for various cortical regions, computed controlling for differences in sentence length. (b) RSA matrices for various cortical regions, computed controlling for differences in sentence length and minimum sentence length. (c) Searchlight RSA for the VerbNet-CN model using 8 mm radius showing cortical regions of interest, with those forming part of the language network underlined. RSA correlations are thresholded at z=2. (d) Partial correlations controlling for differences in sentence length by cortical region, with each individual participant shown as blue dots, and the simple average over individual correlations shown as a bar. (e) Partial correlations controlling for differences in sentence length computed using the group-averaged RSA matrix, shown by cortical region. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants.
Figure 4
Correlations between individual fMRI data and Llama 3 layers.
Average correlations between representational similarity analysis (RSA) matrices of each layer of Llama 3 and brain RSA matrix of each participant. Mean-CN (CN) and VerbNet-CN hybrid (VN) models are also shown for comparison. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) RSA matrices for the Mean-CN model along with selected layers of the Llama 3 model, computed controlling for differences in sentence length.
Figure 5
Comparison of model correlations across subregions of the language network.
(a) Regions within the language network. (b) Partial correlations controlling for differences in sentence length by brain region, with each individual participant shown as blue dots, and the simple average over individual correlations shown as a bar. (c) Partial correlations controlling for differences in sentence length computed using the group-averaged representational similarity analysis (RSA) matrix, shown by language network region. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants.
Figure 6
Behavioural ratings of sentence similarity show similar results to fMRI results, but with higher absolute correlations.
(a) (Left) Average correlations between representational similarity analysis (RSA) matrices of four computational models and human-rated similarities using all sentence pairs. (Right) Average correlations between RSA matrices of four computational models and human-rated similarities using only diagonal sentence pairs. Error bars show standard error of the mean computed over 100 bootstraps. (b) Scatterplots showing the relationship between model similarities (horizontal axis) and human rated similarities (vertical axis) for all four computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. The 45 degree line (black) shows a hypothetical line of perfect fit between model and human similarities.
Appendix 1—figure 1
Representational similarity analysis (RSA) matrices for all computational models.
Representational similarity matrices for all 21 models, as well as human and GPT-4 ratings, and three control variables related to sentence length and visual similarity. For vector-based models, two similarity metrics were used: cosine similarity (‘cs’) and dimension insensitive Euclidean metric (DIEM) similarity (‘dm’). Block diagonal patterns are visible for all models aside from random embeddings, indicating all models are sensitive to lexical similarity. Most transformers show only limited similarity to sentence structure, as evident from the lack of distinction between off-diagonal ‘swapped’ sentence pairs and on-diagonal ‘modified’ and ‘substituted’ sentence pairs. By contrast, the graph-based and the VerbNet-CN hybrid models generally show greater sensitivity to such structural differences.
Appendix 1—figure 2
Correlations between all computational models and brain activity.
Partial correlations between representational similarity analysis (RSA) matrices of all computational models and the brain RSA matrix, controlling for differences in sentence length. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 3
Correlations between behavioural ratings of sentence similarity and model similarities for all computational models.
(a) Correlations between representational similarity analysis (RSA) matrices of all computational models and human-rated similarities computed over all sentence pairs (top) and block diagonal sentence pairs (bottom). Error bars show 95% confidence intervals calculated by bootstrap resampling over raters. (b) Scatterplots showing the relationship between model similarities (horizontal axis) and human rated similarities (vertical axis) for all computational models. Each dot corresponds to a single pairwise similarity. The 45-degree line (black) shows a hypothetical line of perfect fit between model and human similarities.
Appendix 1—figure 4
Model correlations with brain activity for all sentence pairs and the block diagonal subset of sentence pairs, with an additional control for minimum sentence length.
Partial correlations between representational similarity analysis (RSA) matrices of five computational models and the brain RSA matrix, controlling for differences in sentence length and minimum sentence length. ‘Human’ refers to behavioural ratings. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. Notation for statistical significance: * for p<0.05, ** for p<0.01, and *** for p<0.001, with Bonferroni correction for three independent comparisons. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all four computational models. Each dot corresponds to a single pairwise similarity after regressing out the effects of sentence length similarity and minimum sentence length. Note that the displayed correlation for the AMR-Smatch is inaccurate owing to the discrete nature of SMATCH similarities, which leads to inconsistencies when using regression residuals to estimate brain similarities after controlling for length differences and the minimum sentence length. Similarities have been rescaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 5
Comparison of sentence representations and model correlations across brain regions, with an additional control for minimum sentence length.
(a) Representational similarity matrices for various cortical regions, computed controlling for differences in sentence length and minimum sentence length. (b) Representational similarity matrices for various cortical regions, computed controlling for difference in sentence length and minimum sentence length. Block diagonal structure is visible for all regions outside the visual cortex. (c) Searchlight representational similarity analysis (RSA) for hybrid model using 8 mm radius showing cortical regions of interest, with those part of the language network underlined. RSA correlations are thresholded at z=2. (d) Partial correlations controlling for differences in sentence length and minimum sentence length by cortical region, with the simple average over individual correlations shown as a bar. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (e) Partial correlations controlling for differences in sentence length and minimum sentence length computed using the group-averaged RSA matrix, shown by cortical region. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants.
Appendix 1—figure 6
Correlations between individual fMRI data and Llama 3 layers, with an additional control for minimum sentence length.
Average correlations between representational similarity analysis (RSA) matrices of each layer of Llama 3 and brain RSA matrix of each participant, controlling for the RSA similarity of visual cortex activity averaged over participants. Mean-CN (CN) and VerbNet-CN hybrid (VN) models are also shown for comparison. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) RSA matrices for the Mean-CN and VerbNet-CN models, along with selected layers of Llama 3, computed with an additional control for minimum sentence length.
Appendix 1—figure 7
Comparison of model correlations across subregions of the language network, with an additional control for minimum sentence length.
(a) Regions within the language network. (b) Partial correlations controlling for differences in sentence length and minimum sentence length shown by language network subregion, with each individual participant shown as blue dots, and the simple average over individual correlations shown as a bar. (c) Partial correlations controlling for differences in sentence length and minimum sentence length computed using the group-averaged representational similarity analysis (RSA) matrix, shown by language network subregion. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants.
Appendix 1—figure 8
Correlations between all computational models and brain activity, with an additional control for minimum sentence length.
Partial correlations between representational similarity analysis (RSA) matrices of all computational models and the brain RSA matrix, controlling for differences in sentence length and minimum sentence length. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 9
Model correlations with brain activity for all sentence pairs and the block diagonal subset of sentence pairs, with visual similarity control.
Partial correlations between representational similarity analysis (RSA) matrices of five computational models and the brain RSA matrix, controlling for the RSA similarity of visual cortex activity averaged over participants. ‘Human’ refers to behavioural ratings. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. Notation for statistical significance: * for p<0.05, ** for p<0.01, and *** for p<0.001, with Bonferroni correction for three independent comparisons. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all four computational models. Each dot corresponds to a single pairwise similarity after regressing out the subject-averaged visual cortex similarities. Note that the displayed correlation for the AMR-Smatch is inaccurate owing to the discrete nature of SMATCH similarities, which leads to inconsistencies when using regression residuals to estimate brain similarities. Similarities have been rescaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 10
Correlations between all computational models and brain activity, with visual similarity control.
Partial correlations between representational similarity analysis (RSA) matrices of all computational models and the brain RSA matrix, controlling for the RSA similarity of visual cortex activity averaged over participants. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 11
Comparison of sentence representations and model correlations across brain regions, with visual similarity control.
(a) Partial correlations controlling for differences in sentence length and minimum sentence length by cortical region, with the simple average over individual correlations shown as a bar. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (b) Partial correlations controlling for differences in sentence length and minimum sentence length computed using the group-averaged representational similarity analysis (RSA) matrix, shown by cortical region. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants.
Appendix 1—figure 12
Correlations between individual fMRI data and Llama 3 layers, with visual similarity control.
Average correlations between representational similarity analysis (RSA) matrices of each layer of Llama 3 and brain RSA matrix of each participant, controlling for the RSA similarity of visual cortex activity averaged over participants. Mean-CN (CN) and VerbNet-CN hybrid (VN) models are also shown for comparison. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) RSA matrices for the Mean-CN and VerbNet-CN models, along with selected layers of Llama 3, computed controlling for visual similarity.
Appendix 1—figure 13
Model correlations with brain activity for all sentence pairs and the block diagonal subset of sentence pairs using the dimension insensitive Euclidean metric (DIEM) similarity metric.
Partial correlations between representational similarity analysis (RSA) matrices of five computational models and the brain RSA matrix, controlling for differences in sentence length. ‘Human’ refers to behavioural ratings. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. Notation for statistical significance: * for p<0.05, ** for p<0.01, and *** for p<0.001, with Bonferroni correction for three independent comparisons. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all four computational models. Each dot corresponds to a single pairwise similarity after regressing out the effects of sentence length similarity and minimum sentence length. Note that the displayed correlation for the AMR-Smatch is inaccurate owing to the discrete nature of SMATCH similarities, which leads to inconsistencies when using regression residuals to estimate brain similarities after controlling for length differences and the minimum sentence length. Similarities have been rescaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 14
Correlations between all computational models and brain activity using dimension insensitive Euclidean metric (DIEM) similarity metric.
Partial correlations between representational similarity analysis (RSA) matrices of all computational models and the brain RSA matrix, controlling for differences in sentence length. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 15
Correlations between all computational models and brain activity using dimension insensitive Euclidean metric (DIEM) similarity metric, with an additional control for minimum sentence length.
Partial correlations between representational similarity analysis (RSA) matrices of all computational models and the brain RSA matrix, controlling for differences in sentence length and minimum sentence length. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 16
Correlations between all computational models and brain activity using dimension insensitive Euclidean metric (DIEM) similarity metric, with visual similarity control.
Partial correlations between representational similarity analysis (RSA) matrices of all computational models and the brain RSA matrix, controlling for the RSA similarity of visual cortex activity averaged over participants. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Appendix 1—figure 17
Average correlations between representational similarity analysis (RSA) matrices of each layer of AMRBart and the brain RSA matrix of each participant.
Mean-CN (Mn) and VerbNet-CN hybrid (VN) models are also shown for comparison. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) RSA matrices for the Mean-CN and VerbNet-CN models, along with selected layers of the AMRBart.
Appendix 1—figure 18
Average correlations between representational similarity analysis (RSA) matrices of each layer of ERNIE 2.0 and brain RSA matrix of each participant.
Mean-CN (Mn) and VerbNet-CN hybrid (VN) models are also shown for comparison. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) RSA matrices for the Mean-CN and VerbNet-CN models, along with selected layers of the ERNIE 2.0.
Appendix 1—figure 19
Comparison of model correlations with brain activity when varying the stability score threshold.
Partial correlations between representational similarity analysis (RSA) matrices of five computational models and the brain RSA matrix, controlling for differences in sentence length. Bars are colour-coded based on the threshold used for voxel inclusion. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Correlations between RSA matrix of each individual participant and the group-averaged RSA matrix, organised by the voxel selection method used for computing the RSA matrix. Each dot corresponds to the correlation for a single participant.
Appendix 1—figure 20
Comparison of model correlations with brain activity when varying the stability score threshold.
Partial correlations between representational similarity analysis (RSA) matrices of five computational models and the brain RSA matrix, controlling for differences in sentence length and minimum sentence length. Bars are colour-coded based on the threshold used for voxel inclusion. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Correlations between RSA matrix of each individual participant and the group-averaged RSA matrix, organised by the voxel selection method used for computing the RSA matrix. Each dot corresponds to the correlation for a single participant.
Appendix 1—figure 21
Correlation between participant accuracy in attention check questions and model similarity to brain representations.
Scatterplot showing the accuracy score for each participant on attention check questions presented during the scanning run (horizontal axis) against the model correlation with all sentence pairs computed over all stable cortical voxels excluding V1–V4 (vertical axis). Each dot shows the results for a single participant, with all 38 participants included who achieved a score on the questions above chance level (25%).
Appendix 1—figure 22
Model correlations with brain activity for all sentence pairs and the block diagonal subset of sentence pairs, using the final 3 s of each stimulus.
Partial correlations between representational similarity analysis (RSA) matrices of five computational models (Random, Mean, Transformers, Hybrid, and Graph) and the brain RSA matrix, controlling for differences in sentence length and using the final 3 s for each stimulus. Blue bars show correlations computed over all stable (excluding visual regions V1–V4), while green bars show correlations for stable voxels in the language network. Notation for statistical significance: * for p<0.05, ** for p<0.01, and *** for p<0.001, with Bonferroni correction for three independent comparisons. (a) Partial correlations for each individual participant shown as blue dots, with the simple average over individual correlations shown as a bar. (b) Partial correlations computed using the group-averaged RSA matrix. Error bars show 95% confidence intervals calculated by bootstrap resampling over participants. (c) Scatterplots showing the relationship between model similarities (horizontal axis) and group-average neural similarities (vertical axis) for all four computational models. Each dot corresponds to a single pairwise similarity, scaled to between 0 and 1 for visualisation. While all sentence pairs are shown, regression lines (red) are computed over the block diagonal pairs only.
Tables
Table 1
Explanation of the process of constructing sentences used in the study.
Added or altered elements in the second sentence in each pair are italicised. The final two columns represent approximate relative similarities intended for each sentence pair type, though there will be variation due to the precise details of each sentence.
| Type | Example sentence pair | Lexical similarity | Overall semantic similarity |
|---|---|---|---|
| Same | The cameraman brought the equipment to the director. The cameraman brought the new equipment to the director. | Highest | Highest |
| Modified | The cameraman brought the equipment to the director. The cameraman begrudgingly brought the awkward equipment up from the basement to the director. | Moderate | High |
| Substituted | The cameraman brought the equipment to the director. The cameraman sold the equipment to the director. | Moderate | Moderate |
| Modified and substituted | The cameraman sold the equipment to the director. The cameraman begrudgingly brought the awkward equipment up from the basement to the director. | Low | Moderate |
| Swapped (simple case) | The cameraman brought the equipment to the director. The director brought the cameraman to the equipment. | High | Moderate |
| Swapped (with substitution) | The cameraman brought the equipment to the director. The director sold the cameraman’s equipment. | Moderate | Low |
| Swapped (with modifiers) | The cameraman brought the equipment to the director. The renowned director brought the impatient cameraman to the equipment from his last project. | Low | Low |
| Different | The cameraman brought the equipment to the director. The psychologist warned the secretary about the patient. | Lowest | Lowest |
Table 2
Summary of models of sentence meaning analysed in this study.
| Model name | Explanation | Citation |
|---|---|---|
| InferSent | A bi-directional LSTM trained on a variety of natural language inference tasks. | Conneau et al., 2017 |
| USE | Universal Sentence Encodings, based on a transformer architecture trained on a range of language tasks. Sentence embedding constructed by averaging all token embeddings from the final layer. | Cer et al., 2018 |
| ERNIE 2.0 | A transformer based on the BERT architecture trained using multi-task learning. Sentence embedding constructed by averaging all token embeddings from the final layer. | Sun et al., 2020 |
| SentBERT | A sentence transformer model with additional training to predict paired sentences from a large dataset. We used the mpnet embeddings. | Reimers and Gurevych, 2019 |
| DefSent-cls | A RoBERTa-large transformer adapted to produce sentence embeddings with fine-tuning on about 100,000 words paired with their dictionary definitions. | Tsukagoshi et al., 2021 |
| AMRBart | A sentence transformer architecture trained to encode AMR graphs. | Bai et al., 2022 |
| S3BERT | A sentence transformer based on SentBERT with extra training to use AMR graph-based metrics to construct an overall similarity score. | Opitz and Frank, 2022b |
| OpenAI | Embeddings provided from the OpenAI API model text-embedding-ada-002, based on a large transformer with additional fine-tuning from human feedback. | Neelakantan et al., 2022 |
| Llama 3 | A decoder-only transformer architecture trained using multi-task learning with additional fine-tuning from human feedback. Embeddings extracted from the 8B model. Sentence embedding constructed by averaging all token embeddings from the final layer. | Grattafiori et al., 2024 |
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Large language models poorly capture how word order shapes representations of sentence meaning in the human brain
eLife 14:RP108442.
https://doi.org/10.7554/eLife.108442.4
