A Spatiotemporal Hierarchy for Social Interaction Perception in the Lateral Visual Stream

Reviewer #1 (Public review):

Summary:

The manuscript presents a study combining human functional MRI (fMRI), electroencephalography (EEG), and human judgement data within a two-step regression framework to determine when several regions of interest (ROIs) along the so-called third visual pathway for social perception are activated, and which aspects of visual representations of social interactions they encode. The goal is to determine whether social interactions are encoded through a hierarchy of representation along the third pathway, and whether this hierarchy is implemented in a feedforward manner.

The first analysis linking EEG to fMRI data identifies early activation (within 60 ms) in early visual cortex (EVC) and later (onset between 112 and 310 ms) activations in selected, more anterior ROIs along the pathway, i.e., motion-selective MT, extrastriate body area (EBA), object-selective lateral occipital cortex (LOC), and posterior and anterior social-interaction superior temporal sulcus regions (pSTS and aSTS). The second analysis linking EEG-fMRI data to stimulus features derived either computationally from the images or from human judgments shows that EVC activity arises early (within 68 ms) and is predicted by low-level visual features and motion energy. Activity in all other ROIs emerges after 140 ms (or later) and is predicted by motion energy (in MT, EBA and LOC), arousal (in MT), spatial expanse (in LOC, pSTS and aSTS), agent distance (in LOC, pSTS and aSTS) and communication (in aSTS). No effects of facingness, object-directedness, joint action, or valence are found in any ROI.

Two observations are highlighted: features important for recognizing social interactions (i.e., agent distance) are encoded with a similar latency in the mid-level LOC and in STS; the communication content is selectively encoded in anterior STS.

It is concluded that social interactions are processed in a hierarchical way, with more abstract information (i.e., communication) encoded in more anterior regions. Moreover, because this higher-level information is encoded at similar or even shorter latencies than mid-level features in mid- and high-level ROIs, these findings are taken to challenge a strictly feedforward hierarchy within the third visual pathway.

Strengths:

This is a timely manuscript concerning a newly discovered and now extensively investigated brain network, purportedly dedicated to social perception. In the manuscript, the selectivity of this network for the perception of social interactions is assumed rather than demonstrated. Instead, the study addresses two related questions that are critical for understanding how the network (or pathway) is organized and functions: (i) whether representations are hierarchically organized, progressing from more specific information (lower-level visual features) to increasingly abstract information (interaction category, e.g., communication), and (ii) whether processing follows a feedforward mechanism with prior and rapid encoding of lower-level features of social interactions, followed by encoding of mid-level and higher-level features.

Addressing these questions is important because it goes beyond the knowledge of where social information is represented, toward understanding what kinds of visual input and transformations support the recognition of social interactions, and how the processing stages are organized (e.g., in a feedforward manner or with recurrent interactions). Establishing both aspects would define the particular computational architecture supporting the human ability to understand visual social interactions.

The manuscript is also methodologically valuable, as it introduces a variation of more conventional fMRI-EEG fusion analyses. This method establishes fMRI and EEG correspondences, thus revealing spatiotemporal dynamics of brain activity (i.e., where activity occurs in the brain, at which time point), and further links these dynamics to stimulus features (i.e., which aspects of the stimuli predict activity in a given brain area at a given time). The description and discussion of the method in the manuscript are clear and transparent. This facilitates replication and allows appreciation of the rationale of the approach and its potential for future research.

Weaknesses:

The method introduced in the manuscript appears to be a valuable addition to the range of non-invasive approaches to study the spatiotemporal dynamics of representations in the human brain. The results presented in the manuscript, however, do not make a strong case for the sensitivity of the method.

The study fails to replicate several effects that have demonstrated reliability across previous studies. In particular, motion energy in the stimuli only poorly predicts - relatively late -activity in motion-sensitive MT. Moreover, unlike several previous studies, including work by McMahon, Isik, and colleagues, the study finds no effects of facingness (or any other mid-level features) in EBA, object-directedness in LOC, joint actions in pSTS, and valence in pSTS. The manuscript does not discuss discrepancies, raising concerns about the sensitivity and validity of the method. In particular, the absence of previously reported effects is difficult to interpret without considering whether the method is sufficiently sensitive to detect the relevant stimulus dimensions.

Another related and unexplored issue concerns the potentially limited correspondence between EEG and fMRI signals, and between stimuli used to derive the stimulus features for the analyses and stimuli used in the actual EEG experiment. EEG and fMRI data were collected in different samples, and different individuals may assign different social value to the same events, potentially affecting neural processing in regions such as the pSTS (see e.g., work by Varrier, Finn et al.). Furthermore, the stimuli used in the EEG experiment were derived from, but differed from those used in the fMRI experiment. Although this is not entirely clear from the manuscript, it appears that stimulus features were measured for the original stimuli used in the fMRI experiment and assumed to be valid also for stimuli in the EEG experiment. The manuscript does not acknowledge these issues, or other potential sources of noise arising from them, either on their own right or in relation to the above null effects. This makes it difficult to determine whether the absence of several expected effects reflects genuine characteristics of neural representations or limitations of the measurement and analysis pipeline.

Beyond the methodological concerns, the results themselves are less straightforward than the manuscript suggests. Evidence for hierarchical representation emerges from the effect of low-level visual features in EVC and the encoding of communication in aSTS. Activity in all ROIs 'in the middle', from MT, EBA and LOC to pSTS, is predicted by a mixture of low- and mid-level stimulus features, whose effects in some cases come online after the encoding of the interaction category in aSTS (at 195 ms). This raises important questions concerning how a higher-level representation of an interaction category can emerge before the encoding of the lower- or mid-level information that is presumably diagnostic of that category. The manuscript would benefit from a more thorough discussion of this issue, particularly because the temporal ordering of these effects is central to the authors' claims about hierarchical and feedforward processing.

Reviewer #2 (Public review):

Summary:

The authors seek to understand the hierarchical temporal representation of social communicative actions in the posterior superior temporal sulcus and other regions of the lateral pathway. They apply an analysis framework in which they predict fMRI responses from EEG responses to the same stimuli. (The EEG stimuli were shorter versions of the same clips shown as fMRI stimuli.) They examine which temporal segments of EEG data are most predictive of responses in which part of the brain. They find that early (< 100 ms post-stimulus) EEG activity is predictive of responses in early visual cortex, and that later (~ 150 ms post-stimulus) EEG activity is predictive of responses in the lateral visual stream. They then model the fMRI responses predicted by the EEG signals - which they term the "EEG-relevant" portion of the fMRI signal - as a function of various social and visual factors. They conclude that social interactions elicit responses in aSTS-SI later than low-level visual features elicit responses in early visual cortex, and they provide some evidence that mid-level social primitives elicit responses in mid-level areas somewhere in between.

Strengths:

I find the question and the framework interesting. The extraction of "EEG-relevant" fMRI signal as a target for visual and social feature-based encoding models has potential, though the approach in this instance suffers from low signal-to-noise. The evidence that the anterior portion of the superior temporal sulcus responds to communicative actions is clear and supports prior work in this regard.

Weaknesses:

I think there are several important points that the authors need to address to strengthen their case, and I think that many points could use more explicit description, explanation, and discussion.

Major concerns:

First, the basis of the EEG predictions of fMRI activity is not clear. The paradigm relies on predicting fMRI responses at each voxel from a weighted combination of EEG responses measured anywhere over the scalp. However, the authors never show any kind of weight map to visualize which electrodes on the scalp were weighted to predict responses in each fMRI voxel or ROI. I see that the use of PCA to orthogonalize the EEG responses creates problems with localizing the predictive signal in EEG to particular electrodes. Is the PCA step necessary? In any case, some form of analysis to estimate where in the brain the predictive EEG signal is coming from seems important. It's fairly clear that the best signal quality in the EEG recordings is in the early visual cortex. Thus, it could be the case that electrodes over EVC provide the main basis for predictions across the entire brain, with different temporal phases of EVC activity associated with activity in different regions in the fMRI data. It could also be the case (tho less likely in my opinion) that activity in regions away from EVC on the scalp provides the basis for predicting more anterior brain regions in the fMRI data. These two possibilities each suggest a substantially different interpretation of the present results: in the former case, there may be some sort of feedback loop between early visual areas and lateral areas; in the latter case, the hierarchical interpretation the authors have laid out seems more plausible. It seems important to draw a distinction between these possibilities.

In any case, the authors should say more in the discussion about the sources of EEG signals and fMRI signals and the relationship between the two.

On a similar note: the authors show that their EEG data was reliable based on a split-half analysis of the data. However, the authors only show average split-half correlation across electrodes and subjects. It stands to reason that some electrodes were much more reliable than others. The authors should show a map of electrode reliability across the scalp. One relevant question is: are there reliable electrode locations throughout the visual cortex? Or are the EEG signals only, or by far most reliable, in EVC? (How reliable is the EEG signal in regions that are at least somewhat near to the location of the lateral regions that is the principal object of investigation here?)

The use of predicted fMRI signals (predicted by EEG) as a target for image & social feature encoding models is only valid to the extent that the EEG signals accurately predict the fMRI signals. The prediction accuracy going from EEG to fMRI is not very high - a correlation of less than 0.2 in most regions besides EVC and the anterior extent of the activity along the ventral temporal lobe. In the best-predicted regions, the activity only reaches a correlation of 0.3. The majority of the signal (> 90% of the variance) is unmodeled in this analysis. Thus, I would hesitate to characterize the evidence for activation by particular stimulus features at particular times in this analysis as "strong". The authors themselves are somewhat ambivalent on this point, as well. While they write in the conclusion: "Finally, we find strong evidence that social interactions are computed via hierarchical computation of low- and mid-level social visual features in the lateral visual pathway." However, This statement conflicts somewhat with other statements the authors make in the discussion section: "Together, these results suggest that lateral visual regions may not be organized in a strict feedforward hierarchy as has previously been suggested" Later in the discussion there is more ambivalence about the hierarchical nature of the representation: "Our findings suggest that the processing of social action features from posterior to anterior brain regions does not adhere to a strict feedforward hierarchy." The authors should present a more consistent interpretation of their findings.

Finally, the low performance of most of the mid-level models in mid-level regions is a concern; it points to poor signal overall. The common threads in the models that do predict responses above chance seem to be space (spatial expanse and agent distance) and salience (to the extent that arousal may constitute salience). This raises a question: Were the social model features correlated with either of the low-level model features?

I think that the overriding issue here is poor signal quality in the EEG data. The estimate of repeatability from the split-half analysis is not high, even on average for the whole brain; it is likely to be very poor for areas outside EVC. I think this matters. An important null hypothesis to consider is that there may be a few fairly low-level, or at least simple and straightforward, elements of the response that are reflected in the noisy EEG signal. Chief among these is likely to be that the most salient or high-contrast retinotopic regions in the image, which both AlexNet and motion energy models would likely pick up on, are likely to elicit responses in EVC. This is likely why EVC is by far the best-predicted region, both based on the raw EEG data and in the two-step regression analysis. There could be elements of spatial priority at all levels of the visual system. This could be what the authors are observing: there is some relatively small (in terms of variance explained) component of the signal that is common to many regions, and which emerges somewhat later in later regions due to conduction delays.

I think an important exception to this signal problem is the finding of communicative actions eliciting responses in aSTS-SI. That seems very robust and clear.

Medium comments:

There should be a section in the methods on how the ROIs were defined. (There is a brief description in the results section, but this is not sufficient). The ROI outlines should also be displayed somewhere in the paper (not just circles showing the general region as in Figure 1). Figure 3 would be a nice place for them, as would Figure S3.

It's also worth noting in the methods and discussion that the use of template ROIs for hMT in particular is potentially sub-optimal. hMT lies in anatomically variable locations across subjects. One reason for the seemingly weak signal in hMT may be that the voxels being averaged do not fall within the true hMT for each subject.

In the results (section 3.2.1), the authors write, "We scored the regression by predicting the EEG-predicted fMRI activity (estimated fMRItest) to the held-out test videos and then correlating the actual fMRI response (fMRItest) in the test set to this prediction at each time point (estimated fMRItest)." Unless I am missing something, I think the authors can safely say that they predicted responses to the withheld stimuli without specifying the intended prediction target (whether it is the fMRI data or the estimated fMRI data), because the prediction is simply based on weights, labeled or estimated image features, and stimuli. So, the prediction might as well be a direct prediction of the fMRI test data. It seems to me that mentioning the estimated fMRI test in this sentence only muddies the waters.

In the two-stage regression analysis, were all the visual and social features fit to the EEG-predicted fMRI data in a single regression? Was this regression regularized? How many parameters total were fit? (This is clear for the AlexNet and motion energy models, but not clear for the other features - were they all single-feature models?)

Author response:

We thank the reviewers and editors for their constructive feedback and are in the process of preparing a detailed point-by-point response. In the meantime, we would like to underscore our main conclusions and highlight the consistency with this study and the prior fMRI literature.

First, while the reviewers are correct to highlight the difficulty in interpreting null effects, the claims of our paper are primarily concerned with the positive findings. To our knowledge, this is the first evidence that social interaction perception is extracted in a fast hierarchy, which progresses from early visual features to mid-level social primitives to high-level information about communicative actions in the STS. This is an exciting and novel finding that sheds new light on the neural computations that support this core human ability. Importantly, the main null finding is the lack of difference in the onset latency between mid-level and anterior ROIs, rather than an inability to decode information in these regions (see below) and does not undercut this central claim.

Second, we would like to highlight that, overwhelmingly, our results are consistent with the prior fMRI literature. For example:

• Using our two-step regression procedure, we recover representations of motion energy in MT validating the use of this method

• Further, we find representations of mid-level features in mid-level regions ~150 ms after stimulus onset, primarily the distance between people.

This consistency, and the decoding of high-level features like communication exclusively in STS regions (as highlighted by reviewer 1), provides support for our method and shows that the EEG-fMRI correspondence extends beyond early visual regions.

The reviewers also raise questions about potential inconsistencies with our prior fMRI study (McMahon et al., 2023). We would like to clarify those findings here, because the current manuscript relies on the same fMRI data, and thus we would not expect to see effects here that are not reported in that fMRI data. In our original fMRI study, we did not find an effect of facingness, joint action, or valence in any ROI. Further, in that study, the effect for communication was most robust in aSTS, rather than pSTS. The primary effect from our original fMRI study that we do not find with EEG-fMRI fusion is information about object-directed actions in LOC. We will address these in a more detailed point-by-point response and revision soon.

Beyond the practical limitations of what we could find in the current fMRI dataset, we would also like to discuss the theoretical reasons for differences between our findings and the broader literature. First, our lack of finding an effect of valence likely reflects our stimuli, not neural representations. The Moments in Time computer vision dataset that our videos are derived from does not sample strongly negative videos. This restricted valence range is also evident in the behavioral ratings that participants provided.

Further, there are many differences between how we have looked for an effect of facing direction compared to prior work in this area. First, we do not use a binary facing/not facing but a graded scale from facing to not facing, and, to our knowledge, ours is the only study that has investigated a facing effect in naturalistic videos. It is possible that due to the graded scale or the natural stimuli that the facing effect is too subtle for us to detect. Another possibility is that while facingness is a social cue that is encoded in the brain, it is not a dominant cue in naturalistic videos where other cues are present (e.g., the distance between people, physical contact, and contingent motion).

Finally, on joint action, a type of physical social interaction between people, there is now convergent evidence from multiple studies that this may not be the primary type of social interaction represented in the STS. McMahon et al. (2023) found that communicative, not joint interactions, predicted STS responses. This was replicated and extended in (Zhao et al., 2025), where we find that joint action representations are more represented in lateral occipital regions, while communicative interactions are represented in the STS. From ongoing work in controlled stimuli (McMahon & Kanwisher, 2026 VSS), we have found that regions in the STS respond more to communicative interactions than joint actions. This present study provides new insight into the computations that support this, suggesting that information about communicative interactions is extracted along a spatiotemporal hierarchy, consistent with the fast timescale of visual processing, along the newly proposed lateral stream.

References:

McMahon, E., Bonner, M. F. & Isik, L. Hierarchical organization of social action features along the lateral visual pathway. Current Biology 33, 5035-5047.e8 (2023).

Zhao, Y., McMahon, E. & Isik, L. Separate Neural Representations for Physical and Communicative Social Interactions: Evidence from Data-driven Voxel Decomposition. Preprint at https://doi.org/10.31234/osf.io/fmnq9_v1 (2025).

McMahon, E. & Nancy Kanwisher. The STS Responds Preferentially to Communicative Signals. Vision Science Society; May 15–19, 2026; St. Petersburg, FL.