1 Significance Statement

Understanding how the brain processes social actions is critical to understanding the social mind. This study leverages a novel EEG-fMRI fusion method to reveal, for the first time, the spatiotemporal hierarchy of social action representations in the lateral visual pathway from early visual cortex to the superior temporal sulcus. Notably, we can decode communicative interactions in the anterior superior temporal sulcus (aSTS) within 200 ms shortly following decoding of visuospatial features about the social scene. This finding underscores the brain’s remarkable efficiency in extracting social meaning. Together, these results advance our understanding of the neural computations underlying social perception and highlight the lateral visual stream as a specialized pathway for dynamic social perception.

2 Introduction

Perceiving the actions of others is essential in daily life. Among the most common actions we witness are social actions that involve two or more people, like talking, dancing, or gesturing (1; 2; 3). The brain primarily represents actions of others in lateral regions including the lateral occipital temporal cortex (LOTC) and the superior temporal sulcus (STS) (2; 3). In these regions, one of the key organizing features of actions is their sociality, or the extent to which an action is directed at another person (2; 3; 4). Other related work has identified regions in the STS that respond selectively to social interactions (5; 6; 7; 8). These findings, combined with research on dynamic face processing, have led to new ideas about a third visual pathway projecting laterally from primary visual cortex, through LOTC, to the STS (9). The lateral visual pathway is hypothesized to be separate from the classic ventral and dorsal visual pathways (10; 11) and to be specialized for understanding dynamic social content.

Features of social interactions are hypothesized to be hierarchically processed along the lateral visual pathway (12). This is supported by studies with simple, controlled stimuli showing that there are increasingly abstract social action representations along the lateral surface. For instance, mid-level visual cues indicative of social actions, or “social primitives”, such as whether two bodies are facing (13; 14) or moving toward one another (15) are represented in the body-selective extrastriate body area (EBA), a mid-level region in the lateral pathway. More anterior regions along the STS are selective for more abstract, high-level social action information, including the presence and valence of social interactions (5; 6; 7; 8). Further, a recent comprehensive fMRI study investigated the organization of low-to-high level social features. This work found a posterior-to-anterior gradient of increasingly abstract social feature representations in the lateral visual pathway (6). Related work has found a temporal hierarchy of action representations using electroencephalography (EEG) (1) and magnetoencephalography with source-reconstruction (16).

While the prior research provides evidence for hierarchical representations of social actions, none of these studies have been able to address how information flows through these lateral visual regions due to the use of methods that either have high temporal or high spatial resolution but not both. To investigate the question of how representations in these regions evolve over time, in the current study, we use EEG-fMRI fusion to combine the high-temporal resolution of EEG with the high-spatial resolution of fMRI. EEG-fMRI fusion is a previously developed method (17) that has been used to investigate how object (18) and social representations (19) evolve in space and time in the human brain. Here, we make several significant methodological advances to the original EEG- fMRI fusion approach. We adopt an encoding model approach that (1) links EEG and fMRI in a cross-validated manner allowing us to determine the generalizability of representations in held-out stimuli, and (2) defines a hypothesis space of the tuning profile of different regions of interest (ROIs) to stimulus features and to EEG at each time point (20; 21).

We combine our previously published fMRI data (6) with new EEG recordings in separate subjects while they watched the same video clips as the fMRI participants (Figure 1A). Using EEG decoding and EEG-fMRI fusion, our results support a hierarchy of features in computing social interactions but also suggest that the lateral stream may not be a strict feedforward hierarchy.

Figure 1:

3 Results

3.1 Social Action Videos Evoke Reliable EEG Responses

A group of participants (n = 20) participated in an EEG experiment in which their neural activity was recorded with a 64-channel EEG, while they viewed short videos that were used in our prior study of social action perception (6). This stimulus set comprises 250 three-second videos depicting social actions, which are split into 200 videos for training and 50 videos for evaluation. Here we maintained the defined train-test split for the stimuli but reduced the videos to the central 500 ms to facilitate temporal time-locking in EEG. In the EEG experiment, videos in the training set were repeated four times, and videos in the test set were repeated sixteen times to ensure a high signal-tonoise ratio of the target data.

After minimal preprocessing, temporal resampling of the EEG data to 400 Hz, and temporal smoothing (see 5.4.1), we estimated the signal quality of EEG data by calculating the split-half reliability in the test set. Overall, the reliability is significant in the 72–438, 458–622, and 658–800 ms time ranges, revealing that there is detectable signal in EEG (Figure S1) and that the data are of high enough quality for the encoding/decoding framework adopted below.

3.2 Early-Late Temporal Dissociation in the Lateral Pathway

To understand the time course of neural processing across the lateral stream, we used an encoding approach to fuse the EEG data with our previously collected fMRI data (6). At each time point for each EEG participant, we predicted the average ROI response in fMRI data from each subject. Our fMRI dataset (6) includes voxel-wise fMRI neural responses (β values) to each video from four participants. The fMRI data includes both anatomically defined regions of interest (ROIs), such as the early visual cortex (EVC) and the motion-selective middle temporal area (MT), as well as functionally defined ROIs in the lateral stream. The lateral ROIs consist of the extrastriate body area (EBA), which processes bodies and their spatial relationships (22; 13), the lateral occipital cortex (LOC), which is selective for objects (23) and is involved in processing object-directed actions (3), and posterior and anterior social-interaction selective regions in the superior temporal sulcus (pSTS and aSTS) (5; 8; 7; 6).

Following the encoding, we averaged the prediction across EEG participants and fMRI participants. Our results show a dissociation between early activation in the most posterior region (EVC: t_onset = 60 ms) and late activation in all other lateral regions of the brain (MT: t_onset = 310 ms, LOC: t_onset = 130 ms, EBA: t_onset = 112 ms, pSTS-SI: t_onset = 125 ms, and aSTS-SI: t_onset = 115 ms, Figure 2). The onset in MT is surprisingly late, and MT has low overall prediction, pointing to low correspondence between EEG responses and fMRI responses in MT.

Figure 2:

From these results, we do not find the classical latency profile of feedforward hierarchical computations (24; 9) as mid-level regions (EBA and LOC) are significantly predicted at the same time as anterior regions in the STS, suggesting information in mid- and high-level regions comes online at the same time.

Latency estimations are known to be influenced by the overall decodability of information (e.g., the higher decoding in EVC compared to other reginos). We performed an additional analysis to increase our statistical power and minimize bias due to overall decodability. To do this, we binned the decoding performance, variance distribution, and permutation distribution within-participant in 50 ms bins from 0 to 300 ms. We then averaged each distribution across participants. These results mirror the time-resolved analysis, with EVC first decodable in the 50-100 ms interval, and all other ROIs decodable by the 100–150 ms interval, confirming this early-late distinction (Figure S2).

We see similar trends in whole-brain voxelwise encoding analyses, with high EVC prediction by 100 ms, and onset of prediction in all other regions by 150 ms (Figure 3 and Figure S3). Together, these results suggest that lateral visual regions may not be organized in a strict feedforward hierarchy as has previously been suggested (6; 9). This is in contrast to the ventral stream, where, by 100 ms, there is also prediction in posterior ventral temporal regions, suggesting that the ventral stream may be organized in a more strict feedforward hierarchy than the lateral stream (Figure 3 and Figure S3).

Figure 3:

3.2.1 A Spatiotemporal Hierarchy of Social Action Features Across the Lateral Stream

Our stimulus set includes human behavioral ratings that capture various visual and social scene features. These rated dimensions include mid-level features about the spatial layout of the scene and the configuration between people, including descriptions of the scene’s size (spatial expanse), the proximity of individuals in the video (agent distance), the degree to which the individuals are facing each other (facingness), and the extent to which an action involves an object (object directedness). As well as high-level features about the social content, including the extent to which people were engaged in a joint or physical interaction (joint action), the degree of communication between people (communication), and affective characteristics (valence and arousal). The ratings were obtained using a Likert scale from at least ten subjects, and our prediction target is the average of the participants’ ratings. In addition to human annotations, we also used AlexNet-conv2 and motion energy as models of low-level visual processing (25) and motion activations in MT (26), respectively.

We then asked how these social action features arise in the brain over space and time using a novel two-step regression method (Figure 1B). Briefly, the two-step procedure involves predicting the fMRI responses from the 64-channel EEG activity at each time point (as in Section 3.2). Then we used the stimulus features to predict the EEG-predicted fMRI activity (). This follows the same logic as the procedure for removing nuisance regressors, but takes an opposite approach to “regress in” the EEG-relevant fMRI signals. We scored the regression by predicting the EEG-predicted fMRI activity () to the held-out test videos and then correlating the actual fMRI response (fMRI_test) in the test set to this prediction at each time point (). A more detailed description of the procedure is available in Section 5.4.5.

Using this method, we find that EVC represents low-level visual and motion features with a short latency (AlexNet-conv2: t_onset = 68 ms, motion energy: t_onset = 60 ms, Figure 4A and Figure S4). However, feature representations in MT are relatively late and have poor overall prediction (motion energy: t_onset = 140 ms and arousal: t_onset = 305 ms, Figure S5), which suggests low correspondence between EEG and fMRI signals in MT.

Figure 4:

Mid-level lateral regions represent social primitive features later than low-level features in EVC. In particular, LOC activity was predictable by motion energy (t_onset = 268 ms), spatial expanse (t_onset = 160 ms), and agent distance (t_onset = 160 ms, Figure 4B and Figure S6), and EBA by motion energy (t_onset = 302 ms, Figure S7).

In contrast, high-level lateral regions in the STS represent both social primitive and social interaction features. pSTS-SI was predicted by spatial expanse (t_onset = 148 ms) and agent distance (t_onset = 148 ms, Figure S8) while aSTS-SI was predicted by spatial expanse (t_onset = 215 ms), agent distance (t_onset = 150 ms), and communication (t_onset = 195 ms, Figure 4C and Figure S9).

Social primitive features are represented in both LOTC (EBA and LOC) and STS (pSTS-SI and aSTS-SI) at a similar latency, mirroring the similar EEG-fMRI time course in (Section 3.2. Importantly, however, communication representations are present only in aSTS-SI, not lower-level regions, and have a later onset and peak time than social primitive features in any ROI. Together, these support the idea that social information is extracted hierarchically along the lateral stream (Figure S10).

4 Discussion

Building on existing theories (3; 9), we hypothesized that the perception of social interactions relies on hierarchical computations in the lateral visual pathway (12). Here, we examined the spatiotemporal flow of information through lateral visual regions as a unique window into the underlying neural computations. Using novel methods in EEG-fMRI fusion, we found a temporal hierarchy in feature decoding—more abstract features were decodable at longer latencies in fMRI fused responses in the lateral stream. These results provide novel support for a spatiotemporal hierarchy of social action features along lateral visual regions.

4.1 Early Representations of Communicative Actions in the STS

We found that communicative interactions are decodable from EEG signals and are represented in anterior STS, supporting our hypothesis that representing communication among others is a computational goal of the lateral visual stream (6). Relatedly, we have hypothesized that representing the social interactions between others is a visual process (12). Here we show for the first time that communicative interactions can be decoded from the human STS within 200 ms of video onset. This fast representation is consistent with estimates of feedforward visual processing (24; 27), providing novel empirical support for the theory that social interactions are extracted by the human visual system. While prior work failed to find such rapid decoding in natural stimuli (28; 1), our ability to detect early social interaction signals seems to be due to the power of our novel methods in EEG-fMRI fusion and its advantages over whole brain M/EEG decoding (see below).

4.2 Broad Representations of Social Primitive Features Across the Lateral Stream

Decoding of communication follows shortly after decoding mid-level social features, such as the distance between agents, which have been referred to as “social primitives” or precursors of social interaction in prior work (12; 6). This lends further support to the theory that social interactions are extracted via these mid-level primitives.

Our results, however, find that these representations of social primitives are spatially distributed across both mid- and high-level lateral stream regions. In a recent transcranial magnetic stimulation (TMS) study, only EBA was causally implicated in the perception of social primitives, in direct contrast to nearby regions in LOTC (29). One key difference between Gandolfo et al. (29) and the current study is the use of dynamic versus static stimuli. Itis possible that social primitives of people in motion are represented broadly, while social primitives in static images are only represented in the EBA.

Another possible reason that we find broad representation of social primitives in the current study is that social primitives are correlated with geometric scene structure in the stimulus set (6). As a result, it may be that scene content, rather than social primitives, is broadly represented throughout the visual cortex. Previous work did find representations of scene structure and social primitives in STS regions before controlling for shared variance among features (6). Future analyses controlling for shared variance among features may elucidate whether the lateral visual stream broadly represents social primitive features.

4.3 Two-Step Regression to Characterize Feature Representations in Space and Time

In the current study, we proposed a novel method for investigating the spatiotemporal dynamics of representations in the human brain non-invasively. We do this by performing a two-step regression in which we first predict fMRI responses from whole-brain EEG activity and then use stimulus features to predict the EEG-predicted fMRI representations. Using this method, we found that only low-level visual features (features from the second convolutional layer of AlexNet or a motion energy model) are predictive of EVC responses through time (Figure S4). This validates the specificity and utility of the method for investigating spatiotemporal representations in the human brain.

Somewhat counterintuitively, we find that some information is decoding better with this method than direct decoding from the EEG activity. For example, MT was poorly predicted by EEG directly and the onset latency was much later than would be expected based on electrophysiology (Figures ??) (24), but following the two-step regression, we find early motion energy decoding in MT consistent with feedforward processing. These results suggest that the two-step regression procedure—in addition to linking neural signals in space and time—may also effectively denoise the EEG signal by considering only the information relevant for a given region of interest.

This two-step regression procedure can be applied to any other dataset or question if there are fMRI and EEG responses for the same stimuli. This neural data can be combined with stimulus features as is done here, but it could also be extended to other applications such as investigating the time course of any behavior in a region of interest or investigating how neural networks align with brain regions through time. Further, unlike invasive electrophysiology, this method enables the characterization of the temporal dynamics with dense, whole-brain spatial resolution.

While this is a powerful method to characterize the spatiotemporal dynamics of representations in the human brain non-invasively, the method is still limited by the noise of the recording methodologies, particularly EEG. We partially mitigate this limitation by having multiple stimulus repetitions, but it may be less suited to research questions in which averaging across trials is not possible. The method is also limited by the poor coverage of EEG signals in deeper cortical regions and sub-cortical structures. Future research comparing the spatiotemporal profile found via EEG-fMRI fusion with invasive recordings can shed light on the specificity of this method under different conditions.

4.4 Beyond Feedforward Hierarchies

Our findings suggest that the processing of social action features from posterior to anterior brain regions does not adhere to a strict feedforward hierarchy. Information in mid- and high-level regions comes online at similar time points and both regions have representations of mid-level, social primitive features. The broad representation of mid-level features could be explained by skip connections from EVC to the most anterior regions directly. Alternatively, information in separate but nearby cortical regions may be blurred due to the limitations of EEG-fMRI fusion described above.

Broadly, our results are in line with prior research indicating that even the ventral visual stream may not be accurately described by a simple feedforward hierarchy. The hypothesized hierarchy in the ventral stream in macaques (containing V1, V2, V4, and posterior inferotemporal cortex (IT), central IT, and anterior IT) (24) is largely based on observations of longer latencies of responses in more anterior regions and on retinotopic re-representation of the visual field with increasingly large receptive field sizes (24). However, a large body of work has highlighted that regions along the ventral visual stream contain many feedback connections and interconnections with dorsal and lateral regions (30; 31). In light of these complexities, more research is needed to characterize the anatomical and functional profile of lateral visual regions.

4.5 Conclusion

Here, we investigated the spatiotemporal organization of features in regions of the lateral visual stream. Replicating prior work, we found that communicative actions are represented in the STS (6), and we shed novel light on the computations underlying social perception. We also find, for the first time, that communicative interactions are rapidly extracted by the human brain in natural scenes. Finally, we find strong evidence that social interactions are computed via hierarchical computation of low- and mid-level social visual features in the lateral visual pathway. Future research should aim to better characterize the anatomical and functional profile of lateral regions to further refine our models of the neural basis of human social perception.

5 Materials & Methods

5.1 Reproducibility

For reproducibility, all code is available publicly on GitHub: github.com/Isik-lab/SIEEG_analysis.git.

5.2 Stimuli and fMRI

5.2.1 Stimulus Set and Feature Annotations

Videos, feature annotations, and fMRI are from our prior dataset (6). Here we shortened the videos from 3 s to the central 500 ms to facilitate time locking of the EEG signal as in other EEG-fMRI fusion work (19). The stimulus set includes behavioral annotations of features of the visual and social scene (see Section 3.2.1 for a full description of the stimulus features).

Similar to our previous study, in addition to the behavioral ratings, we also extracted the activations from the second convolutional layer of a pytorch implementation (32) of an ImageNet (33)-trained AlexNet (34) as a model of early visual processing (25) (referred to throughout as AlexNet-conv2). The activations were extracted for every frame and then averaged across frames.

Motion energy was estimated with an Adelson and Bergen model (26) implemented in pymoten (35) using the default pyramid with a temporal window of 10 frames. The motion energy was then averaged across spatiotemporal windows.

Both AlexNet-conv2 and motion energy were estimated on the full 3 s video because they were only used to predict the EEG-predicted fMRI activity and participants in the fMRI saw the 3 s videos. We reduced the dimensionality of AlexNet-conv2 and the motion energy using principal components (PC) analysis implemented in scikit-learn (36) to the number of samples of the training set. We learned the PCs in the training set and applied the learned components to the test set.

5.2.2 fMRI

In our prior experiment (6), participants (n = 4) viewed the 3 s videos in the fMRI scanner. The videos were divided into 200 videos in a training set that were repeated 9-10 times and a 50 video test set that was repeated 18-20 times. The reliability of the fMRI data was estimated as the splithalf correlation across repetitions for every voxel in the brain. In our prior study as well as in the current investigation, analyses are always limited to voxels that were determined to have a significant correlation (p < 0.05, uncorrected).

In addition to the main videos, the participants also completed a battery of functional localizer tasks to enable functional localization of ROIs in the ventral and lateral stream (see Section 3.2 for a description of the ROIs investigated in the current study).

5.3 EEG Experiment

5.3.1 Participants

Participants (n = 21, 5 Males, M = 21.4 years, SD = 2.9 years) gave informed consent prior to participation in accordance with the Johns Hopkins University Institutional Review Board and were either given course credit or monetary compensation for their time. One participant was excluded from subsequent analyses due to excessive movement (final sample, n = 20).

5.3.2 Experimental Procedure

Based on previous studies (19; 1), continuous EEG recordings with a sampling rate of 1000 Hz were made with a 64-channel Brain Products ActiCHamp system using actiCAP slim electrode caps in a Faraday chamber. Electrode impedances were kept below 25 kΩ when possible, and the Cz electrode was used as an online reference.

Participants were seated upright while viewing the videos on a back-projector screen situated approximately 60 cm away.

During the experiment, participants viewed 250 500 ms clips of social actions. The training-test split (200 and 50 videos in train and test sets, respectively) from McMahon et al. (6) was maintained. Videos in the training set were repeated four times in total, and videos in the test set were repeated sixteen times.

The experiment consisted of four sections. For a given section, the 200 training videos were randomly divided into four blocks, and the test videos were shuffled and presented in one block. One section consisted of the four training blocks and the test block repeated four times.

During each block, five “catch” videos were randomly sampled from 50 videos that depicted crowds of people and were randomly interspersed among the other videos. The participants’ task was to hit a button on the crowd trials. These trials were not analyzed in any subsequent analyses.

Between blocks, participants were told that they could take a short break to rest their eyes, but that they should remain still and hit a button when they were ready to continue. Between each of the four sections, participants were given longer breaks. They were told to rest as long as needed and announce when they were ready to continue.

Stimuli were presented with an Epson Home Cinema 3800 projector with a 60 Hz refresh rate. Videos were shown on a black background and subtended approximately 16 × 16 degrees of visual angle. Between trials, a white fixation cross was displayed that disappeared when the videos began. Participants were instructed to fixate between trials but were told that they could move their eyes during the videos. A photodiode was used to accurately track on-screen stimulus presentation times and account for projector lag. The paradigm was implemented in MATLAB R2021b using the Psychophysics Toolbox (37).

5.4 Analysis

5.4.1 EEG preprocessing

Minimal EEG data preprocessing was performed using MATLAB R2023b and FieldTrip (38). The EEG data were aligned to stimulus onset and cut to 1.2 s (0.2 s pre-stimulus to 1 s post-stimulus onset), baseline-corrected using the 0.2 s prior to stimulus onset, high-pass filtered at 0.1 Hz, and low-pass filtered to 60 Hz. Data were resampled to 400 Hz and temporally smoothed over five consecutive samples (12.5 ms windows). Finally, catch trials and false alarm trials were removed.

5.4.2 EEG reliability

We estimated the signal quality of EEG data by calculating the split-half reliability in the test set. For each EEG participant, channel, and time sample, even and odd presentations of videos were split in half and then averaged. The correlation was then computed between the two halves of the data across videos. The variance of the reliability was estimated using bootstrapped resampling of the videos 5000 times and recomputing the even-odd correlation. Reliability and variance distributions were averaged across channels and participants.

5.4.3 ROI response decoding

Within each EEG participant at each time point, we used the 64-channel EEG activity of each participant at each time point to predict the average response in each ROI for each of the fMRI participants using ridge regression. Given the high correlation between the signal recorded by separate EEG electrodes, we first used principal components analysis (PCA) to rotate and orthogonalize the EEGchannel features without reducing the dimensionality. The ridge penalty (varied between 10^-5 and 10³⁰) was fit using optimized leave-one-out prediction implemented in the DeepJuice python package (39) in the training set, and evaluation was performed in the test set. Performance was evaluated as the correlation between the predicted and actual ratings for each feature.

When plotting the time course of decoding, we further smoothed the data using a 25 ms sliding average window. This was done for visualization purposes alone and is not reflected in any of the statistical results. This smoothing procedure was done for all other time course plotting.

5.4.4 Voxelwise fMRI encoding with EEG

To perform whole-brain analyses, we predicted the voxelwise fMRI response following smoothing to a 12 mm full-width-half-max Gaussian kernel to increase signal-to-noise ratio. We performed the same regression procedure as in (Section 5.4.3) to predict the voxelwise activity for each fMRI participant. Visualizations are done by averaging across EEG participants for individual fMRI participants.

5.4.5 Two-Step EEG-Feature fMRI Encoding

Inspired by the regression procedures used by others (40; 41), we devised a novel regression framework to map between EEG, fMRI, and feature annotations (Figure 1B). The procedure involved two regression steps. In step 1, we performed five-fold cross-validation from EEG to fMRI in the training set. We predicted the response for the held-out stimuli and did this across all folds. This results in a predicted fMRI training set based on the EEG data. In other words, we generated a matrix of the fMRI signal that was predictable by EEG. In step 2, we fit a regression with a train/test split with the video annotations (either human ratings or image-computed low-level visual features) as the predictor and the EEG-predicted fMRI response from step 1 as the target. In this way, the method maps features to the EEG-predicted portion of the fMRI signal. We scored the regression by predicting the response in the fMRI test set using the held-out test ratings or visual feature representations and correlating the predicted with the actual response. This procedure allows us to predict the fMRI response in a shared EEG-rating space.

5.4.6 Statistical Analysis

To estimate the distribution of variance of our model performance, at every time point, we performed bootstrap resampling over the model predictions and the true response over 5000 iterations. Similarly, to estimate a statistical null distribution, we performed permutation shuffling between the predicted and true response over 5000 iterations at each time point. We averaged both the bootstrap and permutation distributions first over EEG participants and then over fMRI participants for group-level analyses. Group-level variance was estimated as the 95% confidence interval of the bootstrapped distributions, and group-level significance was the one-tailed test of true relative to the average null distribution. Significance was cluster-corrected for multiple comparisons using the maximum cluster sum across time windows and regressions performed, α = 0.05, cluster-setting α = 0.05.

Supplemental Figures

Figure S1:

Figure S2:

Figure S3:

Figure S4:

Figure S5:

Figure S6:

Figure S7:

Figure S8:

Figure S9:

Figure S10:

Data availability

All EEG, fMRI, and behavior data have been made publicly available and are available here: https://osf.io/2vnw4/overview. The code for the analyses presented in this paper is openly accessible at: https://github.com/Isik-lab/SIEEG_analysis/.

Acknowledgements

This work was funded in part by NSF GRFP DGE-1746891 awarded to E.M. and NIMH R01MH132826 awarded to L.I.

Additional information

Funding

HHS | NIH | National Institute of Mental Health (NIMH) (R01MH132826)

• Leyla Isik

National Science Foundation (NSF) (RFP DGE-1746891)

• Emalie McMahon