Abstract
Language is acquired and processed in complex and dynamic naturalistic contexts, involving simultaneous processing of connected speech, faces, bodies, objects, etc.. How words and their associated concepts are encoded in the brain during real-world processing is still unknown. Here, the representational structure of concrete and abstract concepts was investigated during movie watching to address the extent to which brain responses dynamically change depending on visual context. First, across contexts, concrete and abstract concepts are shown to encode different experience-based information in separable sets of brain regions. However, these differences are reduced when multimodal context is considered. Specifically, the response profile of abstract words becomes more concrete-like when these are processed in visual scenes highly related to their meaning. Conversely, when the visual context is unrelated to a given concrete word, the activation pattern resembles more that of abstract conceptual processing. These results suggest that while concepts generally encode habitual experiences, the underlying neurobiological organisation is not fixed but depends dynamically on available contextual information.
eLife assessment
Kewenig et al. present a timely and valuable study that extends prior research investigating the neural basis of abstract and concrete concepts by examining how these concepts are processed in a naturalistic stimulus: during movie watching. The authors provide convincing evidence that the varying strength of the relationship between a word and a particular visual scene is associated with a change in the similarity between the brain regions active for concrete and abstract words. This work makes a contribution that will be of general interest within any field that faces the inherent challenge of quantifying context in a multimodal stimulus.
Significance of findings
valuable: Findings that have theoretical or practical implications for a subfield
- landmark
- fundamental
- important
- valuable
- useful
Strength of evidence
convincing: Appropriate and validated methodology in line with current state-of-the-art
- exceptional
- compelling
- convincing
- solid
- incomplete
- inadequate
During the peer-review process the editor and reviewers write an eLife assessment that summarises the significance of the findings reported in the article (on a scale ranging from landmark to useful) and the strength of the evidence (on a scale ranging from exceptional to inadequate). Learn more about eLife assessments
Humans acquire, and process language in situated multimodal contexts, through dynamic interactions with their environment. For example, children may learn what the word ‘tiger’ means primarily via sensory-motor experience: they see one on TV, or they are told that a tiger looks like a big cat. Conversely, the experience required for understanding the more abstract concept ‘good’ will likely include an evaluation of the rational and emotional motives underscoring intentional actions. Consequently, while more concrete concepts have external physical references (they refer to objects or actions that are easily perceived in the world) (Wiemer-Hastings, 2010), more abstract concepts do not necessarily have such reference (they generally refer more to cultural and societal constructs or peoples’ inner states of mind) (Villani, 2019). Is this difference reflected in concrete and abstract representations in the brain during naturalistic processing? And are these static or can they change as a function of the multimodal contexts in which processing occurs?
Most studies of concrete and abstract processing are not naturalistic in that they present words or sentences isolated from the rich contexts in which we usually process them. Collectively, these studies suggest that concrete and abstract concepts engage separate brain regions involved in processing different types of information. (Bedny & Thompson-Schill, 2006; Binder et al., 2009; Sabsevitz et al., 2005; Wang et al., 2010; Vigliocco et al, 2014). Concrete words engage regions involved in experiential processing (Barsalou et al., 2003; Pulvermuller & Fadiga, 2010). For example, motor related cortices activate during the processing of action verbs like ‘throw’ (Hauk & Pulvermüller, 2004), or action related nouns like ‘hammer’ (Vigliocco et al, 2006; Kiefer & Pulvermüller, 2012), auditory cortices for sound-related words like ‘telephone’ (Goldberg et al., 2006; Kiefer et al., 2008), and visual cortices for colour-related words like ‘yellow’ (Hsu et al., 2011; Simmons et al., 2007). These results are consistent with the view that we learn and neurobiologically encode concrete concepts in terms of the sensory and motor experiences associated with their referents.
In contrast, some studies of abstract concepts have found greater activation in brain regions associated with general linguistic processing (J. R. Binder, 2005; Mellet, 1998; Noppeney et al., 2004; Sabsevitz et al., 2005). These findings suggest that abstract concepts are learnt by understanding their role in linguistic context, including semantic relationships with other words (e.g. ‘democracy’ is understood through its relationships to words like ‘people’, ‘parliament’, ‘politics’, etc., e.g. Recchia & Jones, 2012). However, neurobiological data also support the view that subcategories of abstract concepts retain sensorimotor information (Harpaintner et al, 2022, 2020 & 2018; Fernandino et al., 2022) as well as social information (Villani et al., 2019; Conca et al, 2021) and internal/interoceptive/affective experiences (Oosterwijk et al., 2015; Vigliocco et al., 2014), which are also important for learning abstract concepts (Ponari et al., 2018). Thus, abstract concepts constitute a more heterogeneous category (Roversi et al., 2013; Zdrazilova et al., 2018; Villani et al., 2019; Muraki et al., 2020; Muraki et al., 2020; Kiefer et al., 2022)
A limitation of these studies is that they have only investigated processing of decontextualized concepts (see e.g., Table 1 in a recent review by Del Maschio et al., 2022). That is, they (often implicitly) assume that conceptual representations in the brain are the product of a stable set of regions processing different types of information depending on whether a concept is concrete or abstract. However, this dichotomy may not account for the way in which we typically process concepts (Lebois et al., 2015), given that the information encoded during conceptual processing depends on the contextual information available. For example, the situated (i.e., contextualised in the discourse but also in the physical setting in which processing occurs) processing of concrete concepts like ‘chair’ could be linked to many abstract internal elements like goals (’I want to rest’), motivations (’I have been standing for two hours’), emotions (’I would like to feel comfortable’), and theory of mind (’is that older person more in the need of this chair than me?’). Conversely, an abstract concept like ‘truth’ is no longer particularly abstract when used in reference to a perceived physical situation (such as ‘snowing’) that matches the utterance’s meaning (’it is true that it is snowing’). Here, ‘truth’ refers to a concrete state of the world (Barsalou et al., 2018).
Indeed, previous work has postulated flexible conceptual processing in experiential brain circuits (Binder & Desai, 2011; Pulvermüller, 2018). Behavioural data support the view that contextual information can affect conceptual processing (e.g., Chambers et al., 2004; Cooper, 1974; Tanenhaus et al., 1995). For example, when an object is depicted in a visual context consistent with its use, the action associated with using the object is more readily available than when the context is more consistent with picking the object up (Kalenine et al., 2014). There is also neurobiological evidence that objects visually present in a situation can influence conceptual processing (Hoffman et al., 2013; Yee & Thompson-Schill, 2016). For example, task-related colour-congruency of objects correlates with less activation of brain regions involved in colour perception during processing – likely because less retrieval of detailed colour knowledge was necessary (Hsu et al., 2011). Dynamic, context-dependent recruitment of visual and motor-related areas during semantic processing has also been established (Hoenig et al, 2008; van Dam et al, 2012; Popp et al, 2019). An understanding of conceptual knowledge as static and context-independent is insufficient to account for these dynamics (Pulvermüller, 2018).
However, no previous study has addressed whether the brain areas associated with concrete and abstract concepts are fixed or recruited in a more dynamic way during semantic processing. The present study aims to fill this gap and test the following two predictions. First, we submit that results from previous investigations of conceptual processing, which generally depict a stable dichotomy between concrete and abstract words, reflect the average experiential information of the type of situational context in which concepts are habitually experienced. Therefore, we predict that the neurobiological representation of concrete concepts, will be related to associated brain regions, because they retain experiences related to their physical referents that are predominantly characterised by sensory and motor information, (Pulvermuller, 2018; Willems et al., 2010). In contrast, because their representations mostly reflect information related to internal/interoceptive/affective experience as well as linguistic information, we expect abstract concepts to activate brain regions associated with emotional, interoceptive, and general linguistic processing (Reinboth & Farkaš, 2022).
Second, the reviewed work also suggests that these habitual representations are not necessarily stable and might change during naturalistic processing depending on the specific contextual information available. We specify two context conditions: a concept is displaced if its context offers little or no visual information related to the concept’s external sensory-motor features. In contrast, a concept is situated, if its visual context contains objects related to its meaning. We predict that when a concrete concept is processed in displaced situations (e.g., ‘cat’ processed when discussing the general character traits of cats vs dogs), response profiles will shift towards more internalised processing shared with abstract concepts. In contrast, when an abstract concept is processed in a situated context (for example the word ‘love’ processed in a scene with people kissing), its representation will more heavily draw on regions involved in processing external, visual information that otherwise characterise more concrete concepts.
We propose that this erosion of the concrete/abstract dichotomy for contextualised processing shows that both concrete and abstract concepts draw on information related to experience (external and internal) as well as linguistic association. Which associated neurobiological structures are engaged during processing depends dynamically on the contextual information available. This way of thinking about the representational nature of conceptual knowledge may help reconcile contradictory evidence concerning the involvement of different brain areas during conceptual processing (for example as discussed in Patterson et al., 2007; Hofmann et al., 2018).
Materials and Methods
The present study analysed the ‘Naturalistic Neuroimaging Database’ (NNDb) (Aliko et al., 2020). All code is made available on a designated repository under https://github.com/ViktorKewenig/Naturalistic_Encoding_Concepts
Participants and Task
The Naturalistic Neuroimaging Database (Aliko et al., 2020, https://openneuro.org/datasets/ds002837/versions/2.0.0) includes 86 right-handed participants (42 females, range of age 18–58 years, M = 26.81, SD = 10.09 years) undergoing fMRI while watching one of 10 full-length movies selected across a range of genres. All had unimpaired hearing and (corrected) vision. None had any contraindication for magnetic resonance imaging (MRI), history of psychiatric or neurological disorder, or language-related learning disabilities. All participants gave informed consent, and the study was approved by the University College London Ethics Committee.
Data Acquisition and Preprocessing
Functional and anatomical images were obtained using a 1.5T Siemens MAGNETOM Avanto, equipped with a 32-channel head coil. Whole-brain images were captured, each consisting of 40 slices per volume at an isotropic resolution of 3.2mm. These were obtained using a multiband echo-planar imaging (EPI) sequence with no in-plane acceleration, a multiband factor of 4x, a repetition time of 1 second, an echo time of 54.8 milliseconds, and a flip angle of 75 degrees. Each study participant yielded a number of brain volumes equivalent to movie runtime in seconds. Due to software constraints limiting the EPI sequence to 1 hour of continuous scanning, there were mandatory breaks during the movie for all participants.
The data were preprocessed with AFNI (Cox 1996) and included despiking, slice-time correction, coregistration, blurring and nonlinear alignment to the MNI152 template brain. The time series underwent smoothing using an isotropic full-width half-maximum of 6mm, with detrending accomplished through regressors for motion, white matter, cerebrospinal fluid, and run length. Adjustments were made to account for breaks in movie viewing, and artefacts identified by spatial independent component analysis were regressed out. Detailed information on data acquisition and preprocessing is available in Aliko et al. 2020 and on openneuro.org.
Materials
All words in the movies were annotated using automated approaches with a machine learning based speech-to-text transcription tool from Amazon Web Services (AWS; https://aws.amazon.com/transcribe/). The resulting transcripts contained on and offset timings for individual words. However, as not all words were transcribed or accurately transcribed, timings were subsequently corrected manually.
Concrete and abstract words were selected for the present study from existing Brysbaert et al (2013) norms. In this database, 37,058 words were rated for concreteness on a scale from 0 (not experience based) to 5 (experience based) by over 4,000 participants. We median split only content words on this scale to yield our set of concrete and abstract words. These were matched for word frequency within 1 SD of mean log frequency (3.61), using the SUTBLEX (US) corpus (Brybaert & New, 2009), which contains frequency counts for 72,286 words. Concrete and abstract words were also matched to be within 1SD from mean length measured as number of letters (4.81).
After this matching process, we were left with more concrete words than abstract words in all movies (783 on average for concrete words; 440 for abstract words). To maintain equal numbers in the subsequent analysis, we randomly selected a subset of 440 concrete words in each movie to match the amount of abstract words, leaving us with 880 words (half concrete, half abstract) per movie on average. Mean concreteness rating for the resulting set of concrete words was 3.22, compared to 1.83 for abstract words. The final mean log frequency and mean length for the final set of concrete words was 3.69 and 4.91 compared to 3.46 and 5.27 for the final set of abstract words and were not significantly different as determined by t-tests (all ps > .45). Surprisal given preceding linguistic context (within the same time window) was extracted with a customised script making use of the predictive processing nature of GPT-2 (Radford et al., 2019), which has been shown to be a good model of processing difficulty (Wilcox et al., 2020; Hao et al., 2020; Merkx and Frank, 2021; Schrimpf et al., 2021). Mean surprisal of concrete words was 22.19 bits, mean surprisal of abstract words was 21.86 bits. A t-test revealed that this difference was not significant (t=1.23, p=0.218). Mean semantic diversity of concrete words was 1.92 and 1.96 of abstract words. This difference was also not significant (t=-1.20, p=0.230).
We used luminance and loudness to control for visual and acoustic properties of the movies that might vary more or less for concrete or abstract words. These ‘low-level’ features might be correlated with other potentially confounding auditory and visual variables. For example, luminance correlates significantly with stimulus intensity and contrast (Johnson & Casson, 1995) and loudness correlates with pitch (Wengenroth et al, 2014), prosody (E Couper-Kuhlen et al, 2004), and speaking rate (Kuhlmann et al, 2021). Thus, luminance and loudness for each frame in the movie was measured using the ‘Librosa’ package for music and audio analysis in Python (McFee et al, 2015). We then averaged these measures across the full duration of each word. Mean luminance for concrete words was 0.72, compared to 0.65 for abstract words. These were significantly different (t(4798) = 9.13 p < 0.001). The mean loudness for concrete words was 0.69, compared to 0.77 for abstract words. These were also significantly different (t(4798) = 9.86, p < 0.001).
For the analysis looking at conceptual processing within context, we similarly wanted to check for collinear variables in the 2 second context window preceding each word, which could have confounding effects. In particular, we looked at surprisal given linguistic context, as well as the visual variables motion (optical flow), colour saturation, and spatial frequency. We extracted the visual features for each frame in the 2s context window preceding each label using the scikit-image package (Walt et al., 2014).
Conceptual Processing Across Contexts
In this analysis we tested the prediction that when contextual information is averaged away, the neurobiological organisation of conceptual processing will reflect brain systems involved in experiential and linguistic information processing, broadly in line with previous studies. Specifically, sensory and motor system engagement for concrete concepts and internal/interoceptive/affective and more general linguistic processing system engagement for abstract concepts. All statistical analyses on the preprocessed NiFTI files were carried out in AFNI (Cox, 1995, 1997). Individual AFNI programs used are indicated parenthetically or in italics in subsequent descriptions.
Deconvolution Analysis
We used an amplitude (also known as parametric) modulated deconvolution regression to estimate activity associated with concrete and abstract words from the preprocessed fMRI data. Specifically, we estimated four sets of amplitude modulated impulse response functions (IRF) for 1) abstract words; 2) concrete words; 3) remaining words; and 4) other time points. Both concrete and abstract words included word onset and five modulators, two of interest and five nuisance modulators. These were the independent ratings of concreteness and abstractness and luminance, loudness, duration, word frequency, and speaking rate (calculated as the number of phonemes divided by duration) for each word. We also estimated the IRFs in the same manner and with the same amplitude modulators for all the remaining words in the movie that were not of interest to our hypothesis. Finally, we generated IRFs (without amplitude modulators) for all time points which did not include any speech. The deconvolution model also included general linear tests for 1) abstract words under the curve; 2) concrete words under the curve; 3) contrasts between concrete and abstract words at each timepoint (for a comparison of significant timing differences see figure S9).
In contrast to a standard convolution based regression analysis, deconvolution does not assume a canonical hemodynamic response function. Instead, an IRF is estimated over a 20s time-window from stimulus onset at one second steps using multiple basis functions. This produces a better understanding of shape differences between individual hemodynamic response functions and achieves higher statistical power at both individual and group level (Chen et al, 2015). Furthermore, there might be differences in timing for the processing of concrete and abstract words (Kroll & Nerves, 1986). In particular, the ‘concreteness effect’ indicates that concrete words are processed faster and more accurately than abstract words (Paivio et al., 1994; Jessen et al., 2000; Fliessbach et al., 2006). These timing differences can be captured by our approach. We chose a 20s time window because this should be sufficient to capture the hemodynamic response function for each word. We selected ‘Csplin’ over the Tent function to deconvolve the BOLD signal because this function offers more interpolation between time points, which might result in a more precise estimate of the individual response function (but is computationally more costly).
Furthermore, traditional ‘main effect’ analysis confounds various non-specific processes, such as acoustic processing, which co-vary with each presented word (especially during dynamic, naturalistic stimuli). In contrast, amplitude modulation allows us to isolate regions that exhibit activational fluctuations specifically in relation to the concreteness modulator beyond the ‘main effect’ and fluctuations in the amplitude response caused by other modulators included in our model. Including nuisance modulators can help serve as controls, mitigating potentially confounding effects - in our case the significant differences between luminance and loudness. By adjusting for these sensory attributes, we ensure that the final betas from this analysis represent the estimated BOLD response specifically associated with concreteness.
Group Level Analysis
We then used the 20 amplitude modulated beta-coefficients from the concrete-abstract contrasts in a linear mixed effects model for group level analysis using ‘3dLME’ (Chen et al., 2013). The model included the factors ‘contrast’ with levels ‘abstract’ and ‘concrete’ and ‘time’ with 20 levels. The model also included a centred covariate for age of participant, and covariates for gender (2 levels) and movie ID (10 levels). Besides a random intercept for participant we included a control implemented in ‘3dLME’ for the potentially auto-correlative structure of residuals to make sure that we model the true effect estimates of the multiple basis function (Hefley et al., 2017). The final model formula was: contrast + age + gender + movie. We included 20 general linear tests, one for each contrast between concrete and abstract activation at each of the 20 timepoints, because we wanted to see how the amplitude of the activation associated with concreteness and abstractness changes over time. We thought that the timing and/or amplitude of the response for concrete and abstract words might vary and that this might be particularly true of the subsequent context analysis. We provide information on timing differences in the supplementary material (Figure S9).
Correction for Multiple Comparisons
To correct for multiple comparisons in the LME, we used a multi-threshold approach rather than choosing an arbitrary P value at the individual voxel level threshold. In particular, we used a cluster simulation method to estimate the probability of noise-only clusters using the spatial autocorrelation function from the LME residuals (’3dFWHMx’ and ‘3dClustSim’). This resulted in the cluster sizes to achieve a corrected alpha value of 0.01 at 9 different P values (i.e., 0.05, 0.02, 0.01, 0.005, 0.002, 0.001, 0.0005, 0.0002, and 0.0001). We thresholded each map at the corresponding z-value for each of these nine p-values and associated cluster sizes. We then combined the resulting maps, leaving each voxel with its original z-value. For additional protection and presentation purposes, we use a minimum cluster size 20 voxels for all results (using ‘3dMerge’). For tables, we determined the centre of mass for each of these clusters using ‘3dCM’. See Cox, 2017 for a validation of a related method and Skipper et al., 2022 for an earlier application.
Analyses of experiential features
In order to more closely characterise the functional specificity of the spatial activation maps from the preceding LME analysis, we carried out the following two additional analyses. In both, the goal is to determine whether brain activity associated with concrete and abstract word modulation relates to separable experiential domains that roughly map onto the aforementioned sensory-motor vs internal/interoceptive/affective/linguistic distinction, respectively.
1. Meta-Analytic Descriptions
The resulting coordinates of the centre of mass of each cluster associated with modulation of concreteness and abstractness were inputed into Neurosynth (https://neurosynth.org/, Yarkoni et al., 2011), an online tool that includes activation maps of 14,371 neuroscientific studies (accessed April, 2023). Neurosynth automatically mines all words in titles and abstracts of these articles and performs a two-way ANOVA, testing for the presence of a non-zero association between terms reporting activation that overlaps with the input location. We scraped all terms with z scores above 3.12 (p < 0.001) (excluding those related to specific brain regions and nondescript terms related to methods, tasks or results) and repeated this procedure for each concrete and abstract cluster to determine functionally associated terms. We then tested whether any of these terms were more important for concrete or abstract words across clusters using a Kruskal Wallis test. We did not correct for multiple comparisons, as this analysis was exploratory in nature and we did not have a prediction about how many terms we would end up with.
2. Peak and Valley Analysis
The meta-analytic approach can only provide relatively general functional descriptions of concrete and abstract words as it is based only on high frequency terms in published titles and abstracts. To provide more precise functional specificity, we used a variant of the ‘reverse correlation’ method (Hasson et al., 2004), called the ‘Peaks and Valleys Analysis’ (Hasson et al., 2008; Skipper et al., 2009). For each participant, this analysis averaged the timeseries of voxels within clusters of modulated activity associated with concreteness and abstractness and relates this directly to features of the perceived stimulus. The approach assumes that, if a brain region encodes certain features, e.g. sensorimotor features, valence, or arousal, then activity will rise (creating peaks) in that region when the feature is present in the stimulus and fall (resulting in valleys) when it is absent.
We first extracted the averaged timeseries for each activation cluster for the concrete and abstract modulations across voxels using 3dMerge. Next, we determined peaks and valleys by calculating the discrete difference ‘Δ’ along the timeseries ‘x’ for each value ‘i’ using the ‘Numpy’ Python package (Harris et al, 2020) (Figure 1, (1)), where Δx[i] = x[i + 1] - x[i]. Given that the canonical model of the hemodynamic response function is said to peak at around 6s after stimulus onset for stimuli of our length, we extracted the words that were mentioned at each peak and valley in a given cluster’s timeseries with a 5- and 6-seconds lag (Figure 1, (2)). We then used the Lancaster sensorimotor norms (Lynott et al, 2019) and norms for valence and arousal (Warriner, 2013) to determine a 13-dimensional experience-based representation for each word (Figure 1, (3)), which included the dimensions: ‘Auditory’, ‘Gustatory’, ‘Haptic’, ‘Interoception’, ‘Visual’, ‘Hand_Arm’, ‘Foot_Leg’, ‘Torso’, ‘Mouth’, ‘Head’, ‘Olfactory’, ‘Valence’, and ‘Arousal’.
Specifically, for concrete clusters, we expected significantly more sensory-motor features (i.e., ‘Foot_Leg’, ‘Hand_Arm’, ‘Haptic’, ‘Visual’ and ‘Torso’ in the corpora) to be associated with peaks rather than valleys in the timeseries compared to abstract clusters. Conversely, we expected significantly more experiential features related to internal/interoceptive/affective processing (i.e., ‘Interoception’, ‘Valence’, and ‘Arousal’ in the corpora) to be associated with peaks compared to valleys for abstract relative to concrete clusters. It was not clear to us whether the dimensions (’Auditory’, ‘Head’, and ‘Gustatory’) were more related to internal/interoceptive/affective or sensory-motor processing. Therefore, we made no predictions for those.
For each of these dimensions, we created two categorical arrays, one for peaks and one for valleys, noting down 0 if the word mentioned at a peak or valley was not highly rated on the dimension and 1 if it was rated highly. This was defined as a deviation of at least one standard deviation from the mean. Given the distributional nature of this data, we then conducted a Kruskal Wallis test between these arrays to determine whether a given experiential dimension occurred significantly more with peaks than valleys in the averaged timeseries of a cluster (Figure 1, (4)). We repeated this procedure for a 4s, 5s and a 6s timeseries lag and conducted a cosine-similarity test between each result (using the ‘Sklearn’ package in Python (Pedregosa et al, 2011) in order to determine if they were significantly different. The results for the 5s and 6s lag converge but not for 4s. This was expected, because the delay of the HRF is somewhere between 5 and 6s. In the main results, we randomly decided between presenting the 5s and 6s lag. The 5s lag is now displayed in figure S2, the 4s lag in figure S1.
Conceptual Processing in Context
The previous analyses tested whether, when activation is considered across contexts, concrete and abstract words are processed in separable brain regions encoding different types of experiential information. Here, we test if these response profiles dynamically change depending on the objects present in the viewing context. We predict that when abstract concepts are situated in highly related contexts (e.g., the word “love” is processed while watching two people kissing), they engage neurobiological regions that are usually involved in processing concrete concepts and are related to processing of external visual information. Conversely, when concrete concepts are displaced from the visual context (e.g., processing the word “apple” while watching the interior of a house), we predict them to engage more abstract-like regions in the brain that are related to processing of internal/interoceptive/affective information. Note that we chose to do the analysis in two stages (first across context, then within context) because only a subset of the 440 words used in our analysis across context for concrete and abstract words are related to the objects present in the scene in a way that situates them in visual context (see below). We wanted to have as much power as possible for the first deconvolution/LME to look beyond the effects of context.
Estimating Contextual Situatedness
To test our predictions, we estimated a measure of contextual situatedness for each concrete and abstract word included in the first analysis. To that end, we utilised two pre-trained visual recognition models, Faster R-CNN (Ren et al., 2015) and OmniSource (Duan et al., 2020), to extract object features using computer vision toolboxes (Chen et al., 2019; MMCV Contributors, 2018; MMAction2 Contributors, 2020), respectively. For each prediction frame (about every four frames, i.e., 4*0.04 = 0.16 s), the object recognition model generated a list of detected object labels and kept those that had a prediction confidence of 90% or greater (Figure 2, (1)). Then, we excluded all objects that were recognized at least 3 standard deviations more often by the model compared to the mean recognition rate of objects (which was 682 appearances), because they would bias our measure of situatedness. These labels were ‘person’ (17,856 appearances per movie on average), ‘chair’ (9,718 appearances per movie on average) and ‘tie’ (8,123 appearances per movie on average). After exclusion of these labels, the final object features were represented as the average of the vectorized object labels using GloVe (Pennington et al, 2014 (Figure 2, (2)), which represents the meaning of each label via global co-occurrence statistics.
We then estimated a representation of a 2s (or 60 frames) context window, which should capture the immediate visual context leading up to each concrete and abstract word. We extracted all the labels of objects visually present in each frame within that window. Finally, we calculated the cosine similarity between the vector representation of each word and its context average, using the ‘Sklearn’ package in Python (Pedregosa et al, 2011), to estimate contextual situatedness for each concrete and abstract word (a value c between 0 and 1), Figure 2, (3)). After separating into situated (c > 0.6) and displaced words (c < 0.4), we were left with (on average for each movie) 164 abstract situated words, 201 abstract displaced words, 215 concrete situated words and 172 concrete displaced words. Given that, as concerns observations, the abstract situated condition was the limiting condition, we randomly selected words from the abstract displaced, concrete situated and concrete displaced conditions to have an equal number of 164 words in each condition (on average per movie).
Though we use visual nuisance regressors, we note that there may be additional confounding visual information when estimating contextual situatedness: high situatedness may correlate positively with the number of objects present and therefore ‘naturally’ engage visual processing more for abstract situated concepts. To alleviate this concern, we counted the number of objects in the abstract situated (8315 objects across movies) and abstract displaced (7443 across movies) conditions. The difference between the two (872) is not statistically significant (H(2)=4.1, p < .09)
Deconvolution Analysis
The deconvolution analysis was as described previously except that the sets of concrete and abstract words were broken into four equal subsets of regressors, i.e., situated concrete, displaced concrete, situated abstract, and displaced abstract words and modulators with 164 words per each set. The four contrasts included were: (1) abstract situated vs abstract displaced, (2) concrete situated vs concrete displaced, (3) abstract situated vs concrete situated and (4) abstract displaced vs concrete displaced. Mean concreteness rating for the resulting sets of concrete words were 3.39 for displaced words and 3.35 for situated words, compared to 1.84 for abstract situated words and 1.71 for abstract displaced words. The mean log frequency and mean length for concrete words was 4.91 and 4.88 for situated words and 5.33 and 4.91 for displaced words, compared to 5.11 and 5.08 for abstract situated words and 5.20 and 5.27 for abstract displaced words. Mean surprisal ratings for concrete situated words were 21.98 bits, 22.02 bits for the displaced concrete words, 22.10 for the situated abstract words and 22.25 for the abstract displaced words. Mean semantic diversity ratings were 1.88 for the concrete situated words, 2.19 for the concrete displaced words, 2.03 for the abstract situated words, and 1.95 for the abstract displaced words. As concerns visual variables, mean optical flow was 0.85, mean colour saturation was 0.33 and mean spatial frequency was 0.02 for the 2s context windows before abstract situated words. Mean optical flow was 0.81, mean color saturation was 0.35 and mean spatial frequency was 0.04 for the 2s context windows before abstract displaced words. Mean optical flow was 0.76, mean color saturation was 0.41 and mean spatial frequency was 0.10 for the 2s context windows before concrete situated words. Mean optical flow was 0.86, mean colour saturation was 0.33 and mean spatial frequency was 0.05 for the 2s context windows before concrete displaced words. Pairwise T-tests between all of the mentioned measures for all groups revealed no significant differences.
Group Level Analysis
A linear mixed effects model for group level analysis was conducted on the 20 amplitude modulated betas from each condition (concrete situated, abstract situated, concrete displaced, situated displaced) using ‘3dLME’ (Chen et al., 2013). The model included factors ‘word_type’ (concrete and abstract), ‘context’ (displaced and situated), and ‘time’ (20 levels) and all possible interactions between these factors. We again included covariates for age, gender, and movie, a random intercept for participant, and a control of the auto-correlative structure of residuals. The final model formula was: word_type * context * time + age + gender + movie. We had no prediction on whether the timing and/or amplitude of the response for concrete and abstract words would vary in the present analysis. Therefore we included 4 general linear tests, 1 for each contrast across time: 1 for the abstract situated vs. abstract displaced contrast, 1 for the concrete situated vs. concrete displaced contrast, 1 for the abstract situated vs abstract displaced contrast, and 1 for the concrete situated vs concrete displaced contrast.
Results
Conceptual Processing Across Contexts
Consistent with previous studies, we predicted that across naturalistic contexts, concrete and abstract concepts are processed in a separable set of brain regions. To test this, we contrasted concrete and abstract modulators at each time point of the IRF (Figure 3). This showed that concrete produced more modulation than abstract processing in parts of the frontal lobes, including the right posterior inferior frontal gyrus (IFG) and the precentral sulcus (Figure 3, red). Known for its role in language processing and semantic retrieval, the IFG has been hypothesised to be involved in the processing of action-related words and sentences, supporting both semantic decision tasks and the retrieval of lexical semantic information (Bookheimer, 2002; Hagoort, 2005). The precentral sulcus is similarly linked to the processing of action verbs and motor-related words (Pulvermüller, 2005). In the temporal lobes, greater modulation occurred in the bilateral transverse temporal gyrus and sulcus, planum polare and temporale. These areas, including primary and secondary auditory cortices, are crucial for phonological and auditory processing, with implications for the processing of sound-related words and environmental sounds (Binder et al., 2000). The superior temporal gyrus (STG) and sulcus (STS) also showed greater modulation for concrete words and these are said to be central to auditory processing and the integration of phonological, syntactic, and semantic information, with a particular role in processing meaningful speech and narratives (Hickok & Poeppel, 2007). In the parietal and occipital lobes, more concrete modulated activity was found bilaterally in the precuneus, which has been associated with visuospatial imagery, episodic memory retrieval, and self-processing operations and has been said to contribute to the visualisation aspects of concrete concepts (Cavanna & Trimble, 2006). More activation was also found in large swaths of the occipital cortices (running into the inferior temporal lobe), and the ventral visual stream. These regions are integral to visual processing, with the ventral stream (including areas like the fusiform gyrus) particularly involved in object recognition and categorization, linking directly to the visual representation of concrete concepts (Martin, 2007). Finally, subcortically, the dorsal and posterior medial cerebellum were more active bilaterally for concrete modulation. Traditionally associated with motor function, some studies also implicate the cerebellum in cognitive and linguistic processing, including the modulation of language and semantic processing through its connections with cerebral cortical areas (Stoodley & Schmahmann, 2009).
Conversely, activation for abstract was greater than concrete words in the following regions (Figure 3, blue): In the frontal lobes, this included right anterior cingulate gyrus, lateral and medial aspects of the superior frontal gyrus. Being involved in cognitive control, decision-making, and emotional processing, these areas may contribute to abstract conceptualization by integrating affective and cognitive components (Shenhav et al., 2013). More left frontal activity was found in both lateral and medial prefrontal cortices, and in the orbital gyrus, regions which are key to social cognition, valuation, and decision-making, all domains rich in abstract concepts (Amodio & Frith, 2006). In the parietal lobes, bilateral activity was greater in the angular gyri (AG) and inferior parietal lobules, including the postcentral gyrus. Central to the default mode network, these regions are implicated in a wide range of complex cognitive functions, including semantic processing, abstract thinking, and integrating sensory information with autobiographical memory (Seghier, 2013). In the temporal lobes, activity was restricted to the STS bilaterally, which plays a critical role in the perception of intentionality and social interactions, essential for understanding abstract social concepts (Frith & Frith, 2003). Subcortically, activity was greater, bilaterally, in the anterior thalamus, nucleus accumbens, and left amygdala for abstract modulation. These areas are involved in motivation, reward processing, and the integration of emotional information with memory, relevant for abstract concepts related to emotions and social relations (Haber & Knutson, 2010, Phelps & LeDoux, 2005).
Finally, there was overlap in activity between modulation of both concreteness and abstractness (Figure 3, yellow). The overlap activity is due to the fact that we performed general linear tests for the abstract/concrete contrast at each of the 20 timepoints in our group analysis. Consequently, overlap means that activation in these regions is modulated by both concrete and abstract word processing but at different time-scales. In particular, we find that activity modulation associated with abstractness is generally processed over a longer time-frame (for a comparison of significant timing differences see figure S9). In the frontal, parietal, and temporal lobes, this was primarily in the left IFG, AG, and STG, respectively. Left IFG is prominently involved in semantic processing, particularly in tasks requiring semantic selection and retrieval and has been shown to play a critical role in accessing semantic memory and resolving semantic ambiguities, processes that are inherently time-consuming and reflective of the extended processing time for abstract concepts (Thompson-Schill et al., 1997; Wagner et al., 2001; Hofman et al., 2015). The STG, particularly its posterior portion, is critical for the comprehension of complex linguistic structures, including narrative and discourse processing. The processing of abstract concepts often necessitates the integration of contextual cues and inferential processing, tasks that engage the STG and may extend the temporal dynamics of semantic processing (Ferstl et al., 2008; Vandenberghe et al., 2002). In the occipital lobe, processing overlapped bilaterally around the calcarine sulcus, which is associated with primary visual processing (Kanwisher et al., 1997; Kosslyn et al., 2001).
Meta-Analytic Results
Overall, these results suggest that concrete modulation engages sensory and motor regions more, whereas abstract words engage regions more associated with semantic as well as internal/interoceptive/affective processing. Both categories overlap (though necessarily at different time points) in regions typically associated with word processing. However, these interpretations are based on informal reverse inference. To more formally and quantitatively evaluate this distinction between concrete and abstract words, we employed meta-analytic description and reverse correlation analyses. Both test whether brain regions involved in concrete and abstract conceptual processing reflect different types of habitual experience (i.e., sensory-motor vs internal/interoceptive/affective).
Term-based labelling demonstrates that significantly more concrete clusters are related to the term ‘Movement’ compared to abstract clusters (H(2) = 12.4, p < 0.001; Figure 4, red). In contrast, abstract clusters are more related to terms that are arguably associated with internal/interoceptive/affective processing compared to concrete activation clusters, i.e., ‘Autobiographical Memory’, ‘Nausea’, ‘Pain’, ‘Reward/Motivation’, and ‘Valence’ (all ps < 0.05; Figure 4). Finally, ‘Language’ was the only term more associated with overlapping clusters than either concrete (H(2) = 7, p < 0.001) or abstract clusters (H(2) = 4, p = 0.045; Figure 4). For meta-analytic associations of each individual cluster, see (Table S1).
Peaks and Valleys Results
Comparing dimensions for abstract vs concrete modulated clusters, we found significantly more concrete compared to abstract clusters associated with the dimension ‘Torso’ (H(2)=7, p<0.001). Three concrete clusters were associated with ‘Haptic’ and ‘Mouth’, which was also significantly more than for abstract clusters (all tests H(2)=5.2, all ps=0.02). Two concrete clusters with ‘Foot_Leg’ compared to 0 abstract clusters was not significant, but the mean was in the expected direction (H(2)=3.4, p=0.06) All concrete clusters are displayed in Figure 5 (red). Conversely, eight abstract clusters were significantly more associated with the dimension ‘Valence’ than for concrete clusters (H(2)=8.3, p < .001). Three abstract clusters were associated with the dimension ‘Auditory’, which was not significantly different from concrete clusters (H(2)=1.9, p=0.17). All abstract clusters are displayed in Figure 5 (blue). Finally, five clusters in which modulation through concreteness and abstractness overlapped (though at different time points) were significantly more associated with the dimension ‘Mouth’ (compared to 2 concrete clusters (H(2)= 5.1, p=0.03) and 0 abstract clusters (H(2)=7.8, p < .001). All overlap clusters are displayed in Figure 5 (yellow). For all results of the peak and valley tests for each individual cluster, see Table S2.
Conceptual Processing in Context
Activation associated with the main effect of word_type overlapped with processing of concrete and abstract words across context in superior temporal sulcus, superior temporal gyrus and middle temporal gyrus (bilateral), in angular gyrus (bilateral), in the central sulcus and precentral and postcentral gyrus (right hemisphere), in lateral and medial frontal cortices as well as in the occipital lobe (see figure 6, A). Activation for the main effect of context was found bilaterally in posterior temporal lobe at the intersection with occipital lobe, as well as in nodes of the default mode network (DMN), including precuneus, medial prefrontal regions and angular gyrus (figure 6, B). The interaction between word_type and context modulated activity in the main nodes of the DMN (amongst other regions), including precuneus, medial prefrontal regions, and angular gyrus (all bilaterally, see figure 6, C). Indeed, the thresholded interaction map with 1501 voxels was ‘decoded’ using the Neurosynth package, where the Pearson correlation is computed between the vectorized map and all the maps in the Neurosynth database. The top four associated terms (excluding brain regions or methodological terms) were the ‘DMN’ (r(1500) = 0.194, p < 0.001), ‘Default Mode’ (r(1500) = 0.219, p < 0.001), and ‘Default’ (r(1500) = 0.226, p < 0.001) as well as ‘Semantic Control’ (r(1500) = 0.206, p < 0.001).
To better understand the nature of this interaction and how it relates to response profiles associated with concreteness and abstractness across contexts, we contrasted concrete vs abstract modulation in situated and displaced conditions (collapsing across timepoints, as we had no prediction about timing differences). This comparison is displayed in figure 7. We then extracted the resulting activation maps as masks and spatially correlated them with the brain mask obtained from the activation map contrasting concreteness and abstractness across contexts. This comparison is displayed in Figure 8.
To quantify this comparison, we used cosine similarity to calculate spatial correlation measures between the unthresholded results from contrasting concrete and abstract modulations in displaced and situated context with the unthresholded contrasts between concrete and abstract modulations across contexts (Figure 3, red is the thresholded version of this map). This shows that in situated context, the contrasted modulation by abstractness overlaps more with concreteness across context (r(72964) = 0.64, p < 0.001) compared to displaced concreteness (r(72964) = 0165, p = 0.476). Concreteness across contexts overlaps with situated abstractness (Figure 7, A, red) bilaterally in the fusiform, occipital lobe, inferior and superior parietal lobules and with displaced concreteness bilaterally in the occipital lobe, as well as large swaths of the superior temporal gyrus (Figure 8, B).
Conversely, in displaced context, the contrasted modulation by concreteness (Figure 7, B, blue) overlaps more with the pattern of activity modulated by abstractness across context (r(72,964) = 0.49, p < 0.001) compared to situated abstractness (r(72,964) = 0.21, p < 0.001). Abstractness across contexts overlaps with displaced concreteness bilaterally in large portions of the inferior parietal lobule, including supramarginal gyrus and post superior temporal sulcus, up to the intersection of the occipital and temporal lobes, lingual gyrus in particular (Figure 8, A). Overlap activation could also be found bilaterally in anterior thalamus and medial prefrontal regions.
Discussion
Conceptual processing is typically investigated in experiments where words are stripped away from their naturally occurring context: most studies use isolated words, and sometimes sentences (see Table 1 in Maschio et al, 2022). However, conceptual processing in its ecology occurs in rich multimodal contexts. Our study investigated naturalistic conceptual processing during movie-watching to begin to understand the effect of multimodal context on the neurobiological organisation of real-world conceptual representation.
Conceptual Processing Across Contexts
First, we asked where in the brain concrete and abstract concepts are processed across different contexts as well as the type of information they encode. Given the hypothesis that conceptual representations reflect contextual information, we expected a set of regions that correspond to the most typical set of experiences (e.g., as encountered during word learning in development) to activate across different contexts. Specifically, we expected concrete conceptual encoding to activate regions more involved in sensory and motor processing and abstract conceptual encoding to activate regions associated with more internal/interoceptive/affective as well as general linguistic processing (Fernandino et al., 2019; Meteyard et al., 2012; J. R. Binder, 2005).
Indeed, we found a general tendency for concrete and abstract words to activate regions associated with different experiences (Figure 3). Consistent with prior work, concrete words were associated with multiple regions involved in sensory and motor processing (Mkrtychian et al., 2019), including most of the visual system (Gao et al., 2019) and the right frontal motor system (Pulvermuller, 2005). In contrast, abstract words engaged regions typically associated with internal/interoceptive/affective processing (anterior thalamus, somatosensory cortex)(Harpaintner et al., 2018; Villani et al., 2021), autobiographical memory (anterior medial prefrontal regions) (Conca, Borsa, et al., 2021), and emotional processing and regulation (anterior medial prefrontal regions, orbital prefrontal cortex, dorsolateral prefrontal cortex, nucleus accumbens and amygdala) (Vigliocco, 2014; Conca, Catricala, et al., 2021).
Consistent with this, both meta-analytic and peak and valley analyses showed that concrete regions were more associated with sensory-motor properties (e.g., ‘Movement’ and ‘Hand_Arm’) whereas abstract regions were more associated with internal/interoceptive/affective properties (e.g., ‘Valence’; Figures 2-4). Together, these results provide evidence from naturalistic processing that concrete and abstract concepts encode different types of experiences (Vigliocco et al., 2009; Wiemer-Hastings & Xu, 2005; Kiehl et al., 1999).
At the level of brain regions, our study aligns with previous literature identifying distinct brain regions engaged in processing abstract versus concrete words. Specifically, our results show greater activation for concrete words in temporo-parieto-occipital regions. These areas include the bilateral middle temporal gyrus, the left fusiform gyrus, and the bilateral angular gyrus, among others. Conversely, our study found that abstract word processing preferentially engages a network of regions within the left hemisphere, including the inferior frontal gyrus (IFG), superior and middle temporal gyri, and the inferior parietal lobule.
However, the regions involved in processing concrete and abstract concepts across contexts did not imply a fully dichotomous encoding of experiences. First, we found that regions involved in sensory (mostly in visual cortices) and motor processing are involved in processing both types of words (Figure 3). Moreover, we found overlap activation in regions associated with language processing in general (Tang et al, 2021, Figure 3). Such results are in line with proposals in which both concrete and abstract representations rely on experiential information as well as their linguistic relationships with other words (e.g., Andrews, Vigliocco & Vinson, 2009; Vigliocco et al., 2009, Piantadosi & Hill, 2022). This latter hypothesis is also supported by our Peaks and Valleys analysis, more specifically that information related to ‘Mouth’ (i.e. the language organ) drives activation in overlap clusters. This is furthermore evidence against hypotheses in which the mouth is specifically associated with abstract concepts (Borghi, 2016).
Conceptual Processing in Context
Though results across contexts presumably represent a form of experiential central tendency, the behavioural, neuroimaging, and electrophysiological literature suggests that conceptual representations might not be stable and may vary as a function of context (Elman, 1995; Spivey & Dale, 2006 in Cai & Vigliocco, 2018; Kutas & Federmeier, 2011; Yee & Thompson-Schill, 2016, Deniz et al., 2023). For this reason, we conducted a second set of analyses with the goal of understanding the extent to which representations associated with concrete or abstract conceptual processing in the brain change as a function of context (Barsalou et al., 2018).
We find that brain activation underlying concrete and abstract conceptual processing fundamentally changes as a function of visual context. We compared the activation profiles of concrete and abstract concepts in displaced and situated contexts with the activations obtained when collapsing across contexts. Our results show that concrete concepts become more abstract-like in displaced contexts with less relevant visual information (Figure 6, B). Overlap between activation for concrete concepts in displaced conditions and abstract concepts across context (Figure 7, A) can be found in ACC, thalamus, and large swaths of the anterior, middle and posterior temporal lobe. We propose that this is because, when a concrete concept is processed in displaced context, its representation will relate more to internal/interoceptive variables and linguistic associations, which are usually encoded by abstract concepts. Conversely, abstract concepts become more concrete-like when they are highly situated (Figure 6, A). Overlap between activation for abstract concepts in situated conditions and concrete concepts across context (Figure 7, B) can be found in fusiform and the occipital lobe (bilateral). We propose that this is because an abstract concept processed in a situated context relates more to external visual information, which is usually encoded by concrete concepts. A consequence of this finding is that the concrete/abstract distinction is neurobiologically less stable than might be assumed. Brain regions ‘switch alliance’ during concrete or abstract word processing depending on context.
What is the neurobiological mechanism behind contextual modulation of conceptual encoding in the brain? Our results indicate that variance in visual context interacted with word-type (both concrete and abstract) in regions commonly defined as the DMN, as well as a set of prefrontal regions associated with semantic control (Ralph et al, 2017; Hoffman et al, 2018) (Figure S3, S5). Recent literature on the role of the DMN suggests that these regions reactivate memories (Crittenden et al., 2015, Konishi et al., 2015, Murphy et al., 2019, Murphy et al., 2018, Sormaz et al., 2018, Spreng et al., 2014, Vatansever et al., 2017) and contexts-specific information (Hahamy et al., 2023), possibly to form contextually relevant situation models (Chen et al., 2017, Keidel et al., 2018, Ranganath and Ritchey, 2012, Smith et al., 2021, Smith et al., 2018), in order to guide semantic cognition (Binder et al, 1999; Binder & Fernandino, 2015; Yeshurun et al., 2021; Tong et al, 2022).
Breaking up the interaction between word_type and context, we find that the DMN is especially involved in displaced conditions for both concrete and abstract conceptual processing (see figure 6, A and B (blue)). These results fit well with evidence suggesting that the DMN supports conceptual processing especially when displaced from sensorimotor input (Murphy et al., 2018; Lanzoni et al., 2020; Wang et al., 2020). Accordingly, the DMN is most strongly activated in the displaced conditions involving abstract concepts. Given their inherent lack of sensorimotor information, abstract concepts offer a greater degree of displacement than their concrete counterparts, thereby demanding a higher engagement of the DMN in these conditions.
In considering the impact of visual context on the neural encoding of concepts generally, it is furthermore essential to recognize that the mechanisms observed may extend beyond visual processing to encompass more general sensory processing mechanisms. The human brain is adept at integrating information across sensory modalities to form coherent conceptual representations, a process that is critical for navigating the multimodal nature of real-world experiences (Barsalou, 2008; Smith & Kosslyn, 2007). While our findings highlight the role of visual context in modulating the neural representation of abstract and concrete words, similar effects may be observed in contexts that engage other sensory modalities. For instance, auditory contexts that provide relevant sound cues for certain concepts could potentially influence their neural representation in a manner akin to the visual contexts examined in this study. Future research could explore how different sensory contexts, individually or in combination, contribute to the dynamic neural encoding of concepts, further elucidating the multimodal foundation of semantic processing.
Conceptual Processing and Language
The exact relationship between concepts and language remains an open question, but it is undisputed that, as determinants of meaning, concepts are necessary for language (Jackendoff, 2002; Bloom, 2002; Fauconnier & Mark, 2002). The present study examined language-driven conceptual processing, as we looked at brain activation during word processing. Our results imply that the underlying neurobiological processes are dynamically distributed and contextually determined. This view fits well with models of ‘natural’ organisation of language in the brain where it is argued that language processing more generally is a whole brain process whose patterns of activation are determined by available context (Skipper, 2014; 2015). These more distributed regions may be averaged away when indiscriminately analysed together and following thresholding because (i) they are more variable given they are associated with different experiences (as we have seen here), linguistic categories (e.g., ‘formulaic speech’; see Skipper et al., 2022), and processes (e.g., different types of syntax) (ii) there are individual differences in all of these (e.g., Wang et al, 2022; Skipper et al, 2022; Skipper, 2015). These suppositions are supported by the fact that concrete and abstract modulation overlaps in typical perisylvian ‘language regions’ (Figure S7).
Conclusions
Our work emphasises the merits of investigating conceptual processing in naturalistic multimodal contexts. This paves the way for future analyses systematically quantifying different types of contexts (e.g. in terms of related objects, actions, emotions, or social interactions) and examining how these can affect conceptual processing in the brain. Such work might further our understanding of the neurobiology of conceptual processing in naturalistic settings by clarifying what type of contexts affect processing and how. This may inform the recent development of multimodal large language models, where processing depends on context beyond purely text based information (Driess, 2023) - especially in naturalistic settings (Kewenig et al., 2023). Apart from commercial applications, gaining a better understanding of the mechanisms underlying naturalistic conceptual processing in the brain might bear important implications for clinical domains, e.g., by informing progress towards helping patients who lost the ability to speak by real-time semantic reconstruction of non-invasive brain recordings with the help of large language models (Tang et al, 2023).
Funding
This work was supported in part by the European Research Council Advanced Grant (ECOLANG, 743035); Royal Society Wolfson Research Merit Award (WRM\R3\170016) to GV; and Leverhulme award DS-2017-026 to VK and GV.
Acknowledgements
VK would like to thank Bangjie Wang for help in using image recognition software, and Dr Sarah Aliko for help with neuroimaging analysis.
Supplementary Material
Complete Results for Meta-Analysis & Peak and Valley Analysis
Peak and Valley Analysis at different lags
Peak and Valley Analysis at the Individual Participant Level (5s lag)
Test of Nonlinearity for Peak and Valley Analysis
Comparison between Overlap and Language Regions
Time Course of Activation
Code
We implemented our data analysis in Bash, Python and R. Our code will be provided as online supplemental material upon publication and hosted openly on a dedicated Github repository under: https://github.com/ViktorKewenig/Naturalistic_Encoding_Concepts
References
- A naturalistic neuroimaging database for understanding the brain using ecological stimuliSci Data 7https://doi.org/10.1038/s41597-020-00680-2
- Comprehension and semantic flexibilityJournal of Verbal Learning and Verbal Behaviour 13:471–481https://doi.org/10.1016/S0022-5371(74)80024-1
- Context-independent and context-dependent information in conceptsMemory & Cognition 10:82–93https://doi.org/10.3758/BF03197629
- The instability of graded structure: Implications for the nature of conceptsConcepts and conceptual development: Ecological and intellectual factors in categorization :101–140
- Moving beyond the distinction between concrete and abstract conceptsPhilos Trans R Soc Lond B Biol Sci 373https://doi.org/10.1098/rstb.2017.0144
- Grounding Conceptual Knowledge in Modality Specific SystemsTrends in Cognitive Sciences 7
- Situating Abstract ConceptsGrounding Cognition :129–163https://doi.org/10.1017/cbo9780511499968.007
- Neuroanatomically separable effects of imageability and grammatical class during single-word comprehensionBrain Lang 98:127–139https://doi.org/10.1016/j.bandl.2006.04.008
- Distinct Brain Systems for Processing Concrete and Abstract ConceptsJournal of Cognitive Neuroscience 17
- Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studiesCereb Cortex 19:2767–2796https://doi.org/10.1093/cercor/bhp055
- How children learn the meanings of words
- Functional MRI of language: new approaches to understanding the cortical organization of semantic processingAnnu Rev Neurosci 25:151–88https://doi.org/10.1146/annurev.neuro.25.112701.142946
- Grounding abstractness: Abstract concepts and the activation of the mouthFrontiers in Psychology 7
- Concreteness ratings for 40 thousand generally known English word lemmasBehav Res Methods 46:904–911https://doi.org/10.3758/s13428-013-0403-5
- Word ProcessingLanguage and Thought 3
- Brains and algorithms partially converge in natural language processingCommun Biol 5https://doi.org/10.1038/s42003-022-03036-1
- Actions and affordances in syntactic ambiguity resolutionJ Exp Psychol Learn Mem Cogn 30:687–696https://doi.org/10.1037/0278-7393.30.3.687
- Linear mixed-effects modeling approach to fmri group analysisNeuroimage 73:176–90https://doi.org/10.1016/j.neuroimage.2013.01.047
- Detecting the subtle shape differences in hemodynamic responses at the group levelFront Neurosci 9https://doi.org/10.3389/fnins.2015.00375
- The multidimensionality of abstract concepts: A systematic reviewNeurosci Biobehav Rev 127:474–491https://doi.org/10.1016/j.neubiorev.2021.05.004
- In search of different categories of abstract concepts: a fMRI adaptation studySci Rep 11https://doi.org/10.1038/s41598-021-02013-8
- The Control of Eye Fixation by the Meaning of Spoken LanguageCognitive Psychology 6
- AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance NeuroimagesComputers and Biomedical Research 29
- Software tools for analysis and visualization of fMRI dataNmr in Biomedicine 10
- Semantic Representations During Language Comprehension are Affected by ContextJournal of Neuroscience 43:3144–3158https://doi.org/10.1523/JNEUROSCI.2459-21.2023
- Evidence for the Concreteness of Abstract Language: A Meta-Analysis of Neuroimaging StudiesBrain Sciences 12https://doi.org/10.3390/brainsci12010032
- BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1https://doi.org/10.18653/v1/N19-1423
- PaLM-E: An Embodied Multimodal Language ModelarXiv https://doi.org/10.48550/arXiv.2303.03378
- Omni-source Webly-supervised Learning for Video RecognitionarXiv https://doi.org/10.48550/arXiv.2003.13042
- Eye Movements as a Window into Real-Time Spoken Language Comprehension in Natural Contexts J Psycholinguist Res 24:409–436https://doi.org/10.1007/BF02143160
- The way we think: Conceptual blending and the mind’s hidden complexities
- Functional specificity for high-level linguistic processing in the human brainProc Natl Acad Sci U S A 108:16428–16433https://doi.org/10.1073/pnas.1112937108
- Distinguishing abstract from concrete concepts in supramodal brain regionsNeuropsychologia :102–110https://doi.org/10.1016/j.neuropsychologia.2019.05.032
- The Organization of Language in the BrainScience 170
- Perceptual knowledge retrieval activates sensory brain regionsJ Neurosci 26:4917–4921https://doi.org/10.1523/JNEUROSCI.5389-05.2006
- Shared computational principles for language processing in humans and deep language modelsNat Neurosci 25:369–380https://doi.org/10.1038/s41593-022-01026-4
- On Broca, brain, and binding: a new frameworkTrends Cogn Sci 9:416–23https://doi.org/10.1016/j.tics.2005.07.004
- The Semantic Content of Abstract Concepts: A Property Listing Study of 296 Abstract WordsFront Psychol 9https://doi.org/10.3389/fpsyg.2018.01748
- The grounding of abstract concepts in the motor and visual system: An fMRI studyCortex 124:1–22https://doi.org/10.1016/j.cortex.2019.10.014
- Time course of brain activity during the processing of motor- and vision-related abstract concepts: flexibility and task dependencyPsychological Research 86:2560–2582https://doi.org/10.1007/s00426-020-01374-5
- Intersubject Synchronization of Cortical Activity During Natural VisionScience 303
- Improving the analysis, storage and sharing of neuroimaging data using relational databases and distributed computingNeuroImage 39:693–706
- Somatotopic Representation of Action Words in Human Motor and Premotor CortexNeuron 41:301–307https://doi.org/10.1016/s0896-6273(03)00838-9
- Deep Residual Learning for Image RecognitionarXiv https://doi.org/10.48550/arXiv.1512.03385
- The dual stream model of speech and language processingHandb Clin Neurol 185:57–69https://doi.org/10.1016/B978-0-12-823384-9.00003-7
- Semantic diversity: a measure of semantic ambiguity based on variability in the contextual usage of wordsBehav Res Methods 45:718–730https://doi.org/10.3758/s13428-012-0278-x
- Concepts, control, and context: A connectionist account of normal and disordered semantic cognitionPsychological Review 125:293–328https://doi.org/10.1037/rev0000094
- Differing Contributions of inferior prefrontal and anterior temporal cortex to concrete and abstract conceptual knowledgeCortex 63:250–266https://doi.org/10.1016/j.cortex.2014.09.00
- Conceptual flexibility in the human brain: Dynamic recruitment of semantic maps from visual, motion and motor-related areasJournal of Cognitive Neuroscience 20
- Color, Context, and Cognitive Style: Variations in Color Knowledge Retrieval as a Function of Task and Subject VariablesJournal of Cognitive Neuroscience 23:2544–2457https://doi.org/10.1162/jocn.2011.21619
- Foundations of Language: Brain, Meaning, Grammar, Evolutionhttps://doi.org/10.1093/acprof:oso/9780198270126.001.0001
- The neural correlates of semantic control revisitedNeuroimage 224
- The role of semantic diversity in lexical organizationCan J Exp Psychol 66:115–24https://doi.org/10.1037/a0026727
- Visual context modulates potentiation of grasp types during semantic object categorizationPsychon Bull Rev 21:645–651https://doi.org/10.3758/s13423-013-0536-7
- Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational ModelsArxiv Preprint
- Conceptual representations in mind and brain: theoretical developments, current evidence and future directionsCortex 48:805–825https://doi.org/10.1016/j.cortex.2011.04.006
- The sound of concepts: four markers for a link between auditory and conceptual brain systemsJ Neurosci 28:12224–12230https://doi.org/10.1523/JNEUROSCI.3579-08.2008
- Differential temporo-spatial pattern of electrical brain activity during the processing of abstract concepts related to mental states and verbal associationsNeuroImage 252https://doi.org/10.1016/j.neuroimage.2022.119036
- Neural pathways involved in the processing of concrete and abstract wordsHum. Brain Mapp 7:225–233https://doi.org/10.1002/(SICI)1097-0193(1999)7:4<225::AID-HBM1>3.0.CO;2-P
- Lexical access for concrete and abstract wordsJournal of Experimental Psychology: Learning, Memory, and Cognition 12:92–107https://doi.org/10.1037/0278-7393.12.1.92
- Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP)Annu Rev Psychol 62:621–47https://doi.org/10.1146/annurev.psych.093008.131123
- Are Automatic Conceptual Cores the Gold Standard of Semantic Processing? The Context-Dependence of Spatial Meaning in Grounded Congruency EffectsCogn Sci 39:1764–1801https://doi.org/10.1111/cogs.12174
- Multimodal foundation models are better simulators of the human brainarXivLabs (preprint) https://doi.org/10.48550/arXiv.2208.08263
- The Lancaster Sensorimotor Norms: multidimensional measures of perceptual and action strength for 40,000 English wordsBehav Res Methods 52:1271–1291https://doi.org/10.3758/s13428-019-01316-z
- Cortical anatomy of mental imagery of concrete nouns based on their dictionary definitionNeuroReport 9
- Coming of age: A review of embodiment and the neuroscience of semanticsCortex 48:788–804
- Concrete vs. Abstract Semantics: From Mental Representations to Functional Brain MappingFront Hum Neurosci 13https://doi.org/10.3389/fnhum.2019.00267
- Heterogeneity in abstract verbs: An ERP studyBrain and Language 211https://doi.org/10.1016/j.bandl.2020.104863
- Heterogenous abstract concepts: Is ‘ponder’ different from ‘dissolve’?Psychological Research 86:2478–2494https://doi.org/10.1007/s00426-020-01398-x
- The neural areas that control the retrieval and selection of semanticsNeuropsychologia 42:1269–1280https://doi.org/10.1016/j.neuropsychologia.2003.12.014
- Concepts in context: Processing mental state concepts with internal or external focus involves different neural systemsSoc Neurosci 10:294–307https://doi.org/10.1080/17470919.2014.998840
- Where do you know what you know? The representation of semantic knowledge in the human brainNat Rev Neurosci 8:976–87https://doi.org/10.1038/nrn2277
- GloVe: Global Vectors for Word RepresentationProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing 11https://doi.org/10.3115/v1/D14-1162
- The Calgary semantic decision project: concrete/abstract decision data for 10,000 English wordsBehav Res Methods 49:407–417https://doi.org/10.3758/s13428-016-0720-6
- Meaning Without Reference in Large Language ModelsArchive (preprint) https://doi.org/10.48550/arXiv.2208.02957
- Processing of action and sound verbs in context: An fMRI studyTranslational Neuroscience 10:200–222https://doi.org/10.1515/tnsci-2019-0035
- Brain mechanisms linking language and actionNature Reviews Neuroscience 6
- Neurobiological mechanisms for semantic feature extraction and conceptual flexibilityTopics in Cognitive Science 10:590–620https://doi.org/10.1111/tops.12367
- Neural reuse of action perception circuits for language, concepts and communicationProg Neurobiol 160:1–44https://doi.org/10.1016/j.pneurobio.2017.07.001
- Active perception: sensorimotor circuits as a cortical basis for languageNat Rev Neurosci 11:351–360https://doi.org/10.1038/nrn2811
- Ultimate Grounding of Abstract Concepts: A Graded AccountJournal of Cognition 5https://doi.org/10.5334/joc.214
- A marriage is an artefact and not a walk that we take together: An experimental study on the categorization of artefactsReview of Philosophy and Psychology 4:527–542https://doi.org/10.1007/s13164-013-0150-7
- Modulation of the semantic system by word imageabilityNeuroimage 27:188–200https://doi.org/10.1016/j.neuroimage.2005.04.012
- A common neural substrate for perceiving and knowing about colorNeuropsychologia 45:2802–2810https://doi.org/10.1016/j.neuropsychologia.2007.05.002
- Reorganization of the Neurobiology of Language After Sentence OverlearningCereb Cortex 32:2447–2468https://doi.org/10.1093/cercor/bhab354
- Gestures orchestrate brain networks for language understandingCurr Biol 19:661–667https://doi.org/10.1016/j.cub.2009.02.051
- The NOLB model: a model of the natural organization of language and the brainCognitive Neuroscience of Natural Language Use :101–134https://doi.org/10.1017/cbo9781107323667.006
- Integration of visual and linguistic information in spoken language comprehensionScience 268
- Semantic reconstruction 1103 of continuous language from non-1 invasive brain recordingsBioarchive
- Effects of repetition and competition on activity in left prefrontal cortex during word generationNeuron 23:513–522
- Flexibility in embodied lexical-semantic representationsHuman Brain Mapping 33:2322–2333https://doi.org/10.1002/hbm.21365
- Attention is All You NeedarXivLabs (preprint) https://doi.org/10.48550/arXiv.1706.03762
- The role of semantics and grammatical class in the neural representation of wordsCereb Cortex 16:1790–1796https://doi.org/10.1093/cercor/bhj115
- Toward a theory of semantic representationLanguage and Cognition 1:219–247https://doi.org/10.1515/LANGCOG.2009.011
- The neural representation of abstract words: the role of emotionCereb Cortex 24:1767–1777https://doi.org/10.1093/cercor/bht025
- Varieties of abstract concepts and their multiple dimensionsLanguage and Cognition 11:403–430https://doi.org/10.1017/langcog.2019.23
- Sensorimotor and interoceptive dimensions in concrete and abstract conceptsJ Mem Lang 116https://doi.org/10.1016/j.jml.2020.104173
- Recovering meaning: Left prefrontal cortex guides controlled semantic retrievalNeuron 31:329–338
- Neural representation of abstract and concrete concepts: a meta-analysis of neuroimaging studiesHum Brain Mapp 31:1459–1468https://doi.org/10.1002/hbm.20950
- Norms of valence, arousal, and dominance for 13,915 English lemmasBehav Res Methods 45:1191–1207https://doi.org/10.3758/s13428-012-0314-x
- The neural organization of semantic control: Tms evidence for a distributed network in left inferior frontal and posterior middle temporal gyrusCereb Cortex 21:1066–75
- Content Differences for Abstract and Concrete ConceptsCognitive Science 29:719–736https://doi.org/10.1207/s15516709cog0000_33
- Body-specific representations of action verbs: neural evidence from right- and left-handersPsychol Sci 21:67–74https://doi.org/10.1177/0956797609354072
- Putting concepts into contextPsychon Bull Rev 23:1015–1027https://doi.org/10.3758/s13423-015-0948-7
- The default mode network: where the idiosyncratic self meets the shared social worldNat Rev Neurosci 22:181–192https://doi.org/10.1038/s41583-020-00420-w
- Communicating abstract meaning: Concepts revealed in words and gesturesPhilosophical transactions of the Royal Society of London. Series B, Biological sciences 373https://doi.org/10.1098/rstb.2017.0138
Article and author information
Version history
- Preprint posted:
- Sent for peer review:
- Reviewed Preprint version 1:
- Reviewed Preprint version 2:
Copyright
© 2024, Kewenig et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics
- views
- 717
- downloads
- 9
- citations
- 0
Views, downloads and citations are aggregated across all versions of this paper published by eLife.