Data and method overview

A) One example video depicting two person actions (top). Participants viewed the videos while their neural responses were recoded with EEG (middle-left) or fMRI (middle-middle), and participants online provided behavioral ratings for eight different features of the visual and social scene. The EEG data is an example of activity in 10 channels of the 64 channels used in the regression, the brain depicts a schematic of the lateral stream ROIs that were predicted from EEG, and the behavioral ratings are example ratings for one video. B) Two-step regression schematic. Regression 1 is fit between the EEG and fMRI data at each time point using 5-fold cross-validation (pink arrow in A). The color bands in the fMRI and EEG data represent the cross-validation folds. The result of this step is the predicted fMRI response for all stimuli in the training data. Regression 2 is fit between the EEG-predicted fMRI response and the behavioral ratings of the stimuli (green arrow in A). Because of license restrictions, the image in (A) is only representative of a video in the stimulus set. The image is “Parents and kids learn together” by DFID - UK Department for International Development licensed under CC BY 3.0.

ROI decoding from EEG.

Decodability of responses lateral stream ROIs from the EEG signals across time. Top: early regions EVC and MT. Middle: mid-level regions LOC and EBA. Bottom: high-level regions pSTS-SI and aSTS-SI. Shaded regions are the 95% confidence intervals from bootstrapped variance distributions. Bold horizontal lines indicate significant decoding (permutation testing, cluster-corrected p < 0.05). The time leading significant lines indicates significance onset. The vertical dashed lines indicate the stimulus presentation period. For visualization purposes alone, the prediction time courses were further smoothed using a 25 ms sliding window. Significant time points are independent of this smoothing kernel.

Whole-brain fMRI decoding from EEG.

The prediction of voxelwise activity in the whole brain from EEG signals across time. Shown are the lateral (top) and ventral (bottom) views of the right hemisphere in the native space of one representative fMRI subject. This is shown as a snapshot of encoding performance in 50 ms increments from 0 to 300 ms. Other fMRI participants are visible in Figure S3.

Joint EEG-feature fMRI encoding.

Joint EEG-feature encoding in three select ROIs in (A) EVC, (B) LOC, and (C) aSTS-SI for a low-level feature (Alexnet-conv2, yellow), mid-level feature (agent distance, red), and high-level feature (communication, purple). All other plotting conventions follow those of Figure 2. See Figures SS4–SS9 for the time course of feature encoding in each ROI.

EEG data reliability.

Data reliability in the test set averaged across participants and channels. Shaded areas are the bootstrapped 95% confidence intervals and solid horizontal lines represent time windows of significant reliability after cluster correction at a level of p < 0.05. Horizontal lines mark time clusters of significant reliability (permutation testing, cluster-corrected p < 0.05). As in Figure 2A data are smoothed in a 25 ms sliding window for visualization alone.

Latency of all ROI prediction from EEG decoding.

Decodability of ROI activity recorded in time bins extended from Figure 2B.

Whole brain prediction in other fMRI participants.

Plotting conventions are all the same as Figure 2C, but for (A) sub-01, (B) sub-03, and (C) sub-04 from McMahon et al. (6).

Time course of joint EEG-feature fMRI encoding of each feature in EVC

Extended to all features from Figure 4A. For visibility, all features are shown on separate subplots.

Time course of joint EEG-feature fMRI encoding of each feature in MT

Related to main text Figure 4.

Time course of joint EEG-feature fMRI encoding of each feature in LOC

Extended from main text Figure 4B for al features.

Time course of joint EEG-feature fMRI encoding of each feature in EBA

Related to main text Figure 4.

Time course of joint EEG-feature fMRI encoding of each feature in pSTS-SI

Related to main text Figure 4.

Time course of joint EEG-feature fMRI encoding of each feature in aSTS-SI

Extended from main text Figure 4C for all features.

Latency of feature prediction in ROIs.

Extended from main text Figure 4D–F for all features and all ROIs.