Social-affective features drive human representations of observed actions

  1. Diana C Dima  Is a corresponding author
  2. Tyler M Tomita
  3. Christopher J Honey
  4. Leyla Isik
  1. Department of Cognitive Science, Johns Hopkins University, United States
  2. Department of Psychological and Brain Sciences, Johns Hopkins University, United States

Abstract

Humans observe actions performed by others in many different visual and social settings. What features do we extract and attend when we view such complex scenes, and how are they processed in the brain? To answer these questions, we curated two large-scale sets of naturalistic videos of everyday actions and estimated their perceived similarity in two behavioral experiments. We normed and quantified a large range of visual, action-related, and social-affective features across the stimulus sets. Using a cross-validated variance partitioning analysis, we found that social-affective features predicted similarity judgments better than, and independently of, visual and action features in both behavioral experiments. Next, we conducted an electroencephalography experiment, which revealed a sustained correlation between neural responses to videos and their behavioral similarity. Visual, action, and social-affective features predicted neural patterns at early, intermediate, and late stages, respectively, during this behaviorally relevant time window. Together, these findings show that social-affective features are important for perceiving naturalistic actions and are extracted at the final stage of a temporal gradient in the brain.

Editor's evaluation

This study investigates and characterizes the representations of visual actions in video stimuli. The combination of the analytical techniques and stimulus domain makes the article likely to be of broad interest to scientists interested in action representation amidst complex sequences. This article enhances our understanding of visual action representation and the extraction of such information in natural settings.

https://doi.org/10.7554/eLife.75027.sa0

Introduction

In daily life, we rely on our ability to recognize a range of actions performed by others in a variety of different contexts. Our perception of others’ actions is both efficient and flexible, enabling us to rapidly understand new actions no matter where they occur or who is performing them. This understanding plays a part in complex social computations about the mental states and intentions of others (Jamali et al., 2021; Spunt et al., 2011; Thornton et al., 2019; Thornton and Tamir, 2021; Weaverdyck et al., 2021). Visual action recognition also interacts cross-modally with language-based action understanding (Bedny and Caramazza, 2011; Humphreys et al., 2013). However, there are two important gaps in our understanding of action perception in realistic settings. First, we still do not know which features of the visual world underlie our representations of observed actions. Second, we do not know how different types of action-relevant features, ranging from visual to social, are processed in the brain, and especially how they unfold over time. Answering these questions can shed light on the computational mechanisms that support action perception. For example, are different semantic and social features extracted in parallel or sequentially?

Relatively few studies have investigated the temporal dynamics of neural responses to actions. During action observation, a distributed network of brain areas extracts action-related features ranging from visual to abstract, with viewpoint-invariant responses emerging as early as 200 ms (Isik et al., 2018). Visual features include the spatial scale of an action (i.e., fine-scale manipulations like knitting vs. full-body movements like running) represented throughout visual cortex (Tarhan and Konkle, 2020), and information about biological motion, thought to be extracted within 200 ms in superior temporal cortex (Giese and Poggio, 2003; Hirai et al., 2003; Hirai and Hiraki, 2006; Johansson, 1973; Jokisch et al., 2005; Vangeneugden et al., 2014). Responses in occipito-temporal areas have been shown to reflect semantic features like invariant action category (Hafri et al., 2017; Lingnau and Downing, 2015; Tucciarelli et al., 2019; Tucciarelli et al., 2015; Wurm and Caramazza, 2019; Wurm and Lingnau, 2015), as well as social features like the number of agents and sociality of actions (Tarhan and Konkle, 2020; Wurm et al., 2017; Wurm and Caramazza, 2019).

Among the visual, semantic, and social features thought to be processed during action observation, it is unclear which underlie our everyday perception in naturalistic settings. Mounting evidence suggests that naturalistic datasets are key to improving ecological validity and reliability in visual and social neuroscience (Haxby et al., 2020; Nastase et al., 2020; Redcay and Moraczewski, 2020). Most action recognition studies to date have used controlled images and videos showing actions in simple contexts (Isik et al., 2018; Wurm and Caramazza, 2019). However, presenting actions in natural contexts is critical as stimulus–context interactions have been shown to modulate neural activity (Willems and Peelen, 2021). Recent attempts to understand naturalistic action perception, however, have yielded mixed results, particularly with regard to the role of social features. For example, one recent study concluded that sociality (i.e., presence of a social interaction) was the primary organizing dimension of action representations in the human brain (Tarhan and Konkle, 2020). Another, however, found that semantic action category explained the most variance in fMRI data, with little contribution from social features (Tucciarelli et al., 2019).

Here, we combined a new large-scale dataset of everyday actions with a priori feature labels to comprehensively sample the hypothesis space defined by previous work. This is essential in light of the conflicting results from previous studies, as it allowed us to disentangle the contributions of distinct but correlated feature spaces. We used three-second videos of everyday actions from the “Moments in Time” dataset (Monfort et al., 2020) and replicated our results across two different stimulus sets. Action videos were sampled from different categories based on the American Time Use Survey (ATUS, 2019) and were highly diverse, depicting a variety of contexts and people. We quantified a wide range of visual, action-related, and social-affective features in the videos and, through careful curation, ensured that they were minimally confounded across our dataset.

We used this dataset to probe the behavioral and neural representational space of human action perception. To understand the features that support natural action viewing, we predicted behavioral similarity judgments using the visual, action-related, and social-affective feature sets. Next, to investigate the neural dynamics of action perception, we recorded electroencephalography (EEG) data while participants viewed the stimuli, and we used the three sets of features to predict time-resolved neural patterns.

We found that social-affective features predict action similarity judgments better than, and independently of, visual and action-related features. Visual and action-related features explained less variance in behavior, even though they included fundamental features such as the scene setting and the semantic category of each action. Neural patterns revealed that behaviorally relevant features are automatically extracted by the brain in a progression from visual to action to social-affective features. Together, our results reveal the importance of social-affective features in how we represent other people’s actions, and show that these representations emerge in the brain along a temporal gradient.

Results

Disentangling visual, action, and social-affective features in natural videos

We curated two sets of naturalistic three-second videos of everyday actions from the Moments in Time dataset (Monfort et al., 2020). The videos were selected from a larger set, ensuring that features of interest were minimally correlated. 18 common activities based on the National Bureau of Labor Statistics’ American Time Use Survey (ATUS, 2019) were represented (Table 1; see section ‘Behavior: Stimuli’). The two stimulus sets contained 152 videos (eight videos per activity and eight additional videos with no agents, included to add variation in the dataset, see section ‘Behavior: Stimuli’) and 65 videos (three or four videos per activity), respectively. The second set was used to replicate behavioral results in a separate experiment with different stimuli and participants.

Table 1
Activities from the American Time Use Survey (ATUS) included in each of the two stimulus sets, with the amount of daily hours spent performing each activity and the corresponding verb labels from the Moments in Time dataset.

Note that control videos were only included in the first dataset. Fighting and hiking were added for variation in valence and action setting.

ActivityHoursVerb labels (Moments in Time)
Childcare/taking care of children0.37Crying, cuddling, feeding, giggling, socializing
Driving1.17Driving, socializing
Eating1.06Chewing, eating
FightingFighting
Gardening0.17Gardening, mowing, planting, shoveling, weeding
Grooming0.68Bathing, brushing, combing, trimming, washing
HikingHiking
Housework0.53Cleaning, dusting, repairing, scrubbing, vacuuming
Instructing and attending class0.22Instructing, teaching
Playing games0.26Gambling, playing+fun, playing+videogames, socializing
Preparing food0.60Barbecuing, boiling, chopping, cooking, frying, grilling, rinsing, stirring
Reading0.46Reading
Religious activities0.14Praying, preaching
Sleeping8.84Resting, sleeping
Socializing and social events0.64Celebrating, dancing, marrying, singing, socializing, talking
Sports0.34Exercising, playing+sports, swimming, throwing
Telephoning0.16Calling, telephoning
Working3.26Working
Control videosBlowing, floating, raining, shaking

Naturalistic videos of actions can vary along numerous axes, including visual features (e.g., the setting in which the action takes place or objects in the scene), action-specific features (e.g., semantic action category), and social-affective features (e.g., the number of agents involved or perceived arousal). For example, an action like ‘eating’ may vary in terms of context (in the kitchen vs. at a park), object (eating an apple vs. a sandwich), and number of agents (eating alone vs. together). Drawing these distinctions is crucial to disambiguate between context, actions, and agents in natural events. To evaluate these different axes, we quantified 17 visual, action-related, and social-affective features using image properties, labels assigned by experimenters, and behavioral ratings collected in online experiments (Figure 1a). Visual features ranged from low-level (e.g., pixel values) to high-level features related to scenes and objects (e.g., activations from the final layer of a pretrained neural network). Action-related features included transitivity (object-relatedness), activity (the amount of activity in a video), effectors (body parts involved), and action category based on the ATUS (ATUS, 2019). Finally, social-affective features included sociality, valence, arousal, and number of agents (see section ‘Representational similarity analysis’). Representational dissimilarity matrices (RDMs) were created for each feature by calculating pairwise Euclidean distances between all videos.

Quantifying visual, social-affective, and action features in the two stimulus sets.

(a) Correlations between feature representational dissimilarity matrices (RDMs). Note the low correlations between visual features and action/social-affective features (white rectangle). (b) Behavioral rating distributions in the two stimulus sets. The z-scored ratings were visualized as raincloud plots showing the individual data points, as well as probability density estimates computed using MATLAB’s ksdensity function (Allen et al., 2019).

In both video sets, there were only weak correlations between visual features and the higher-level action/social-affective features (Figure 1a). The highest correlations were those within each of the three sets of features, including visual features (Experiment 1: Conv1 and image saturation/gist, τA = 0.29; Experiment 2: Conv1 and image hue, τA = 0.32), action features (Experiment 1: arousal and activity, τA = 0.31; Experiment 2: activity and effectors, τA = 0.33), and social features (sociality and number of agents; Experiment 1: τA = 0.31, Experiment 2: τA = 0.3).

The distributions of action and social-affective features were not significantly different between the two stimulus sets (all Mann–Whitney z < 1.08, p>0.28). The width of these distributions suggests that the stimuli spanned a wide range along each feature (Figure 1b). In both experiments, transitivity was notable through its bimodal distribution, likely reflecting the presence or absence of objects in scenes, while other features had largely unimodal distributions.

Behaviorally rated features differed in reliability in Experiment 1 (F(4,819) = 22.35, p<0.001), with sociality being the most reliable and arousal the least reliable (Figure 3—figure supplement 1). In Experiment 2, however, there was no difference in reliability (F(4,619) = 0.76, p=0.55). Differences in reliability were mitigated by our use of feature averages to generate feature RDMs.

Individual feature contributions to behavioral similarity

To characterize human action representations, we measured behavioral similarity for all pairs of videos in each set in two multiple arrangement experiments (see section ‘Multiple arrangement’). Participants arranged videos according to their similarity inside a circular arena (Figure 2). The task involved arranging different subsets of 3–8 videos until sufficiently reliable distance estimates were reached for all pairs of videos. Videos would play on hover, and participants had to play and move each video to proceed to the next trial. In Experiment 1, participants arranged different subsets of 30 videos out of the total 152, while in Experiment 2, participants arranged all 65 videos. To emphasize natural behavior, participants were not given specific criteria to use when judging similarity. Behavioral RDMs containing the Euclidean distances between all pairs of stimuli were reconstructed from each participant’s multiple arrangement data using inverse MDS (Kriegeskorte and Mur, 2012).

Experimental and analysis pipeline for evaluating the contribution of different features to action representations.

Above: a multiple arrangement task was used to generate behavioral representational dissimilarity matrices (RDMs) in the two behavioral experiments. Below: electroencephalography (EEG) data was recorded during a one-back task, and time-resolved neural RDMs were generated using pairwise decoding accuracies. Cross-validated variance partitioning was used to assess the unique contributions of visual, social-affective, and action features to the behavioral and neural RDMs, quantified as the predicted squared Kendall’s τA . The stimuli in this figure are public domain images similar to the types of videos used in the experiments.

The multiple arrangement task was unconstrained, which meant that participants could use different criteria. Although this may have introduced some variability, the adaptive algorithm used in the multiple arrangement task enabled us to capture a multidimensional representation of how actions are intuitively organized in the mind, while at the same time ensuring sufficient data quality. Data reliability was quantified using leave-one-subject-out correlations of the dissimilarity estimates and was above chance in both experiments (Kendall’s τA = 0.13 ± 0.08 and 0.18 ± 0.08 respectively, both p<0.001, permutation testing; Figure 3—figure supplement 1a). Reliability was significantly higher in Experiment 2 than in Experiment 1 (Mann–Whitney z = 3.21, p=0.0013), potentially reflecting differences in both participant pools and sampling methods (subsets of videos in Experiment 1 vs. full video dataset in Experiment 2; see section ‘Multiple arrangement’).

We assessed the contribution of 17 different visual, social, and action features to behavior in both experiments by correlating each feature RDM to each participant’s behavioral RDM (Supplementary file 1b). In Experiment 1 (Figure 3), only two visual features were significantly correlated with the behavioral RDMs (environment and activations from the final fully connected layer FC8 of AlexNet). However, there were significant correlations between behavioral RDMs and all action-related RDMs (action category, effectors, transitivity, and activity), as well as all social-affective RDMs (valence, arousal, sociality, and number of agents).

Figure 3 with 1 supplement see all
Feature contributions to behavioral similarity.

Feature-behavior correlations are plotted against the noise ceiling (gray). Each dot is the correlation between an individual participant’s behavioral representational dissimilarity matrix (RDM) and each feature RDM. Asterisks denote significance (p<0.005, sign permutation testing). The reliability of the data and feature ratings is presented in Figure 3—figure supplement 1.

In Experiment 2, the only visual feature that moderately correlated with behavior was the final fully connected layer of AlexNet (p=0.006, below our threshold for significance). Among action features, only effectors and activity were significantly correlated with the behavioral RDMs. However, we found significant correlations with all social-affective features. The results thus converge across both experiments in suggesting that social-affective and, to a lesser extent, action-related features, rather than visual properties, explain behavioral similarity.

Social-affective features explain the most unique variance in behavioral representations

We performed a cross-validated variance partitioning analysis (Groen et al., 2018; Lescroart et al., 2015; Tarhan et al., 2021) to determine which features contributed the most unique variance to behavior (see section ‘Variance partitioning’). We selected the 10 features that contributed significantly to behavior in either experiment, that is, two visual features (environment and layer FC8 of AlexNet) and all action and social-affective features. To keep the analysis tractable and understand the contribution of each type of information, we grouped these features according to their type (visual, action, and social-affective) and used them as predictors in a cross-validated hierarchical regression (Figure 4). Note that there was no collinearity among the 10 predictors, with an average variance inflation factor of 1.34 (Experiment 1) and 1.37 (Experiment 2).

Figure 4 with 5 supplements see all
Social-affective features explain behavior better than visual and action features.

The unique variance explained by visual, action, and social-affective features is plotted against the split-half reliability of the data (gray). Significant differences are marked with asterisks (all p<0.001, Wilcoxon signed-rank tests). For control analyses on how individual features (e.g., action category and the number of agents) and their assignment to groups affect the results, see Figure 4—figure supplements 15.

Together, the 10 predictors explained most of the systematic variance in behavior. In Experiment 1, the predicted squared Kendall’s τA of the full model (τA2 = 0.06 ± 0.001) was higher on average than the true split-half squared correlation (τA2 = 0.04 ± 0.002). This is likely to be due to the lower reliability of the behavioral similarity data in this experiment and suggests that the 10 predictors are able to explain the data well despite the overall lower prediction accuracy. In Experiment 2, the full model achieved a predicted τA2 of 0.18 ± 0.1 on average compared to a true squared correlation of 0.25 ± 0.1, suggesting that the 10 predictors explain most of the variance (73.21%) in the behavioral data.

In both experiments, social-affective features contributed significantly more unique variance to behavior than visual or action features (Figure 4, all Wilcoxon z > 5.5, all p<0.001). While all three groups of features contributed unique variance to behavior in Experiment 1 (all p<0.001, randomization testing), in Experiment 2, only social-affective features contributed significantly to behavior (p<0.001), while visual and action features did not (p=0.06 and 0.47, respectively). Shared variance between feature groups was not a significant contributor in either dataset. Although the effect sizes were relatively low, social-affective features explained more than twice as much unique variance as either the visual or action features in Experiment 1, and six times as much in Experiment 2. Furthermore, given the limits placed on predictivity by the reliability of the behavioral data, affective features predicted a large portion of the explainable variance in both experiments.

The semantic RDM included among the action features was a categorical model based on activity categories (ATUS, 2019). To assess whether a more detailed semantic model would explain more variance in behavior, we generated a feature RDM using WordNet similarities between the verb labels corresponding to the videos in the Moments in Time dataset. However, replacing the action category RDM with the WordNet RDM did not increase the variance explained by action features (Figure 4—figure supplement 1).

Similarly, our decision to quantify motion and image properties separately by using an optic flow model may have reduced the explanatory power of motion features in our data. Indeed, a motion energy model (Adelson and Bergen, 1985; Nunez-Elizalde et al., 2021) significantly correlated with behavior in Experiment 1, but not in Experiment 2. However, the addition of this model did not change the pattern of unique feature contributions (Figure 4—figure supplement 2).

Although the assignment of features to domains was not always straightforward, our results were robust to alternative assignment schemes. For example, high-level visual features can be seen as bordering the semantic domain, while features like the number of agents or the amount of activity can be seen as visual. However, feature assignment was not the main factor driving our results, which stayed the same even when the activity feature was assigned to the visual group. More strikingly, the social-affective feature group explained significantly more variance than all other features grouped together in both experiments (Figure 4—figure supplement 3). This is a particularly stringent test as it pits the unique and shared contributions of all visual, semantic, and action features against the four social-affective features. In Experiment 1, the combined contribution of visual and action features approached that of social-affective features, while in Experiment 2 the difference was larger. Together with the larger contribution of the number of agents in Experiment 2 (Figure 4—figure supplement 4, Figure 4—figure supplement 5), this suggests that Experiment 2 may have captured more social information, potentially thanks to the exhaustive sampling of the stimuli that allowed each participant to arrange the videos according to different criteria.

Among the social-affective features we tested, the number of agents could be seen as straddling the visual and social domains. To assess whether our results were driven by this feature, we performed a control variance partitioning analysis pitting the number of agents against the other, higher-level social-affective features (Figure 4—figure supplement 3). In both experiments, the higher-level features (sociality, valence, and arousal) contributed more unique variance than the number of agents, suggesting that our results are not explained by purely visual factors.

Furthermore, an additional analysis looking at the separate contributions of the number of agents, sociality, and affective features (valence and arousal) found that the affective features contributed the greatest variance in both experiments (Figure 4—figure supplement 5). For technical reasons, this analysis compared the joint contribution of both affective features to each single social feature and did not discount the impact of variance shared with visual or action-related features. Despite these limitations, the results suggest that the contribution of the social-affective feature group is not driven by the number of agents or the variance it shares with sociality, and highlight the role of affective features (valence and arousal) in explaining behavior.

EEG patterns reflect behavioral similarity

We performed an EEG experiment to investigate how action-relevant features are processed over time. Participants viewed 500 ms segments of the 152 videos from Experiment 1 and performed a one-back action task in which they detected repetitions of the action category (see section ‘EEG: Experimental procedure’). To relate neural patterns to behavioral and feature RDMs, we computed time-resolved neural RDMs for each participant using decoding accuracies between all pairs of videos (Figures 2 and 5a). The time course of decoding performance was similar to that observed in previous E/MEG studies using still visual stimuli (Carlson et al., 2013; Cichy et al., 2014; Dima et al., 2018; Greene and Hansen, 2018; Isik et al., 2014). Decoding accuracy rose above chance at 50 ms after video onset, reached its maximum at 98 ms (63.88 ± 6.82% accuracy), and remained above chance until 852 ms after video onset (cluster-corrected p<0.05, sign permutation testing).

Figure 5 with 2 supplements see all
The features that explain behavioral action representations also contribute to neural representations.

(a) Time course of video decoding accuracy, averaged across all pairs of videos and participants (in gray: SEM across participants). The horizontal line marks above-chance performance (sign permutation testing, cluster-corrected p<0.05). (b) Behavioral similarity correlates with the neural representational dissimilarity matrices (RDMs). The noise ceiling is shown in light blue (leave-one-subject-out correlation, mean ± SEM). Horizontal lines mark significant time windows (sign permutation testing, cluster-corrected p<0.05). (c) The distribution of significant correlation onsets for each feature model across 1000 bootstrapping iterations (sign permutation testing, cluster-corrected p<0.05). Color rectangles show 90% confidence intervals. The time courses of all feature correlations are shown in Figure 5—figure supplement 1. The average electroencephalography (EEG) evoked response is visualized in Figure 5—figure supplement 2.

To assess brain–behavior correlations, we related the average behavioral RDM obtained in Experiment 1 to the time-resolved neural RDMs (Kendall’s τA). The behavioral RDM correlated significantly with neural patterns during a cluster between 62 and 766 ms after video onset (Figure 5b), suggesting that the features guiding the intuitive categorization of naturalistic actions also underlie their neural organization.

Neural timescale of individual feature representations

We assessed the correlations between EEG patterns and the 10 feature RDMs found to contribute to behavior in Experiment 1. We also included an additional feature RDM based on the first convolutional layer of AlexNet, which best captures early visual neural responses (Figure 5—figure supplement 1; see section ‘Multivariate analysis’). The feature RDMs that contributed to behavioral similarity also correlated with the EEG patterns (Figure 5—figure supplement 1), with a single exception (sociality).

A bootstrapping analysis of the cluster onsets of these correlations (Figure 5c) suggests a progression from visual to action and social-affective features. Visual predictors correlated with the neural patterns between 65 ± 15ms (mean ± SD, Conv1) and 84 ± 62 ms (Environment), while action category also had an early onset (58 ± 9 ms). Other action-related features, however, emerged later (transitivity: 170 ± 67 ms, effectors: 192 ± 94 ms, activity: 345 ± 133 ms). Among social-affective features, the number of agents had the earliest correlation onset (178 ± 81 ms), while valence and arousal emerged later (395 ± 81 and 404 ± 91 ms, respectively). Importantly, these features are spontaneously extracted in the brain, as none of them, with the exception of action category, were directly probed in the one-back task performed by participants. In addition, all features were extracted during behaviorally relevant time windows (Figure 5b).

A temporal hierarchy in action perception

A cross-validated variance partitioning analysis revealed different stages in the processing of naturalistic actions (Figure 6). Visual features dominated the early time windows (66–138 ms after video onset). Action features also contributed a significant amount of unique variance (162–598 ms), as well as variance shared with social-affective features (354–598 ms; Figure 6—figure supplement 1). Finally, social-affective features independently predicted late neural responses (446–782 ms). Importantly, visual features did not share a significant amount of variance with either action or social-affective features.

Figure 6 with 3 supplements see all
Hierarchical processing of visual, action, and social-affective features.

(a) Unique variance explained by each group of features over time. The split-half reliability of the data is shown in gray (shaded area; see also Figure 5b). Horizontal lines mark significant time windows (sign permutation testing, cluster-corrected p<0.05). The time course of shared variance is displayed in Figure 6—figure supplement 1. See Figure 6—figure supplement 2 for the results of a fixed-effects analysis. Figure 6—figure supplement 3 shows how the addition of a motion energy model affects these results. (b) The distribution of effect onsets across 100 split-half iterations (sign permutation testing, cluster-corrected p<0.05). Color rectangles show 90% confidence intervals.

An analysis of effect onsets across 100 split-half iterations points to the hierarchical processing of these features, with a progression from visual to action to social-affective features. Social-affective features (mean onset 418 ± 89 ms) contributed unique variance significantly later than other feature sets, while action features (245 ± 104 ms) came online later than visual features (65 ± 8ms; all Wilcoxon z > 7.27, p<0.001; Figure 6b). A fixed-effects analysis revealed the same order of feature information with larger effect sizes (Figure 6—figure supplement 2).

Motion has been shown to drive the response of visual areas to naturalistic stimuli (Russ and Leopold, 2015; Nishimoto et al., 2011). To better assess the effect of motion on EEG responses, we performed an additional analysis including the motion energy model. There was a sustained correlation between motion energy and EEG patterns beginning at 62 ms (Figure 6—figure supplement 3). In the variance partitioning analysis, the addition of motion energy increased the unique contribution of visual features and decreased that of action features, indicating that the action features share variance with motion energy. However, the three stages of temporal processing were preserved in the fixed-effects analysis even with the addition of motion energy, suggesting that the three feature groups made distinct contributions to the neural patterns. Importantly, the unique contribution of social-affective features was unchanged in both analyses by the addition of the motion energy model.

Discussion

Here, we used a large-scale naturalistic stimulus set to disentangle the roles of different features in action perception. Two novel findings emerge from our study. First, our behavioral results suggest that social-affective features play the most important role in how we organize naturalistic everyday actions, above and beyond fundamental visual and action features like scene setting or action category. Second, these behaviorally relevant features are spontaneously extracted in the brain and follow a hierarchical sequence from visual to action-related and culminating with social-affective features. These results offer an account of how internal representations of everyday actions emerge in the mind and brain.

Behavioral representations: What features support action perception?

Across two separate multiple arrangement experiments with large-scale naturalistic stimulus sets, we found that social-affective features predicted similarity judgments better than, and independently of, visual and action-related features. By sampling a comprehensive feature space ranging from low-level to conceptual, we were able to distinguish between components that often covary, such as scene setting and action category or sociality and transitivity. Previous studies have operationalized features in different ways, and an exhaustive investigation is thus difficult; however, our approach of including several important features from each group mitigated this, as suggested by the high amount of variance in behavior collectively explained by our features.

Our work adds to a growing body of evidence for the importance of social-affective features in action perception and extends it by disentangling the contributions of specific social and semantic features. Previous work has highlighted sociality as an essential feature in neural action representations (Tarhan and Konkle, 2020; Wurm et al., 2017; Wurm and Caramazza, 2019), and a recent study (Tarhan et al., 2021) found that behavioral action similarity judgments were better explained by similarity in actors’ goals than by visual similarity. In line with this work, we found a minimal contribution of visual features to action similarity judgments. In contrast, all of our social-affective features – the number of agents, sociality, valence, and arousal – were significantly correlated with behavioral similarity. Furthermore, only two individual action-related features replicated across the two experiments: the amount of activity and the effector (body part) feature, the latter of which is highly relevant to the actors’ goals. This could be interpreted as further evidence for the importance of socially relevant features in our internal representations of actions, and identifies specific social and goal-related features that are important for action understanding.

A hypothesis-driven approach will always pose challenges due to practical limitations in the number of feature spaces one can feasibly test. Our approach of grouping predictors together based on theoretical distinctions made it possible to rigorously evaluate the unique contributions of different types of features, which is an essential first step in understanding naturalistic action representations. This analysis revealed that social-affective features contributed the most unique variance in both experiments, suggesting that they robustly predict behavioral similarity judgments, while visual and action features explained little unique variance in either experiment (Figure 4). An exploratory follow-up analysis showed that this effect was primarily driven by affective features (valence and arousal), with the number of agents as a secondary contributor. Recent work found that affective features drive the perceived similarity of memories of real-life events (Tomita et al., 2021), suggesting that these features bridge the action, event, and memory domains in organizing mental representations.

Among our social-affective features, the number of agents could be construed as a perceptual precursor to sociality. Indeed, previous fMRI work has suggested that neural representations of actions in the visual system reflect perceptual precursors of social features rather than higher-level social features (Wurm and Caramazza, 2019). Here, we found that high-level social-affective features (particularly valence and arousal) contributed significantly to behavior independently of the number of agents. Further, affective features explained significantly more unique variance in behavior than the number of agents in both experiments (Figure 4—figure supplements 4 and 5). Our findings suggest that high-level social-affective features, and in particular valence and arousal, uniquely drive human action representations.

Neural representations: How does action perception unfold over time?

Using EEG, we tracked the temporal dynamics of naturalistic action perception. Using naturalistic stimuli and a rich feature space enabled us to disentangle the contributions of different features and investigate their relative timing. Visual, action, and social-affective features made unique contributions to the EEG patterns at different processing stages, revealing a representational hierarchy of spontaneously extracted features.

Almost all behaviorally relevant features correlated with the EEG patterns, with action-related and social-affective features emerging later than visual features (Figure 5c). Most action-related features emerged within 200 ms, on the timescale of feedforward processing, which is consistent with prior work showing invariant responses to actions as early as 200 ms (Isik et al., 2018; Tucciarelli et al., 2015), and action transitivity processing as early as 250 ms (Wamain et al., 2014). Among social-affective features, the number of agents emerged earliest (162 ms), pointing to the role of this feature as a perceptual precursor in social perception (Papeo, 2020; Wurm and Caramazza, 2019). Valence and arousal emerged later, around 400 ms after video onset. Interestingly, sociality, which has been highlighted as an important dimension in previous fMRI work on action perception (Tarhan and Konkle, 2020; Wurm et al., 2017), did not correlate with the EEG patterns. This effect was not likely to be driven by a lower reliability in the measurement of this feature, as sociality was more reliable than all other behaviorally rated features in Experiment 1 (Figure 3—figure supplement 1). While the absence of an effect does not preclude the possibility that this feature is being processed, it is possible that prior work has confounded sociality with other correlated social-affective features (such as the number of agents or arousal). Alternatively, our operationalization of this feature (which was broader than in some previous studies, e.g., Tucciarelli et al., 2019; Wurm et al., 2017) may have led to differences in the information captured. Note that this finding is mirrored in our behavioral results, where we observed larger unique contributions from valence, arousal, and the number of agents than sociality (Figure 4—figure supplement 5).

Importantly, these features emerged spontaneously as the one-back task performed during the EEG recordings only related to action category. However, the semantic processing required to perform the task may have contributed to these computations. The emergence of features irrelevant to the task at hand (action category is not correlated with any other features in the dataset) suggests that this temporal hierarchy would also emerge in the absence of a task; however, future work can more directly test the impact of implicit and explicit (e.g., social-affective) processing on these neural dynamics.

Variance partitioning revealed a clear temporal progression from visual features (~100 ms) to action features (~150–600 ms) to social-affective features (~400–00 ms). Importantly, these processing stages emerged after partialling out the contributions of other groups of predictors in a cross-validated analysis, validating our a priori distinctions between feature classes. These findings suggest that the extraction of visual features occurs rapidly, within 200 ms, and is likely supported by feedforward computations. The social-affective features that support behavioral representations, however, were extracted last. This is consistent with theories suggesting that internal visual experience reverses the stages of perceptual processing (Dijkstra et al., 2020; Hochstein and Ahissar, 2002). Specifically, it was the final, social-affective stage of neural processing that was reflected in the intuitive behavioral representations, and not the initially extracted visual features. Furthermore, action-related features were extracted significantly before social-affective features, suggesting the two are not extracted in parallel, but instead pointing to a hierarchy in which both visual and action-related features may contribute to socially relevant computations. Given the short duration of our videos and the relatively long timescale of neural feature processing, it is possible that social-affective features are the result of ongoing processing relying on temporal integration of the previously extracted features. However, more research is needed to understand how these temporal dynamics change with continuous visual input (e.g., a natural movie), and whether social-affective features rely on previously extracted information.

Our results add temporal characterization to previous fMRI findings, suggesting that the seemingly conflicting features revealed by previous studies, like the number of agents (Wurm and Caramazza, 2019), sociality (Tarhan and Konkle, 2020), or semantic action category (Tucciarelli et al., 2019), emerge at different stages during action observation. Thus, the existence of different organizing dimensions can be explained not just through spatial segregation within and across brain areas, but also through a temporal gradient starting with visual features and concluding with behaviorally relevant social and affective representations. More work is needed to understand where these dynamic representations emerge in the brain, and whether they are supported by overlapping or distinct networks. Future research could test this using EEG-fMRI fusion to track the spatiotemporal dynamics of action representations.

Actions in context

As real-world actions tend to occur in a rich social context, studies of action perception should consider socially relevant features and the interactions between different systems for perceiving actions, agents, and their mental states (Quadflieg and Koldewyn, 2017). Recent work suggests that social perception enhances visual processing (Bellot et al., 2021; Papeo, 2020) and recruits dedicated neural circuits (Isik et al., 2017; Pitcher and Ungerleider, 2021). Our findings open exciting new avenues for connecting these areas of research. For example, future studies could more explicitly disentangle the perceptual and conceptual building blocks of social and affective features, such as body posture or facial expression, and their roles in action and interaction perception.

One fundamental question that lies at the root of this work is how actions should be defined and studied. Here, we adopted a broad definition of the term, focusing on activities as described in the ATUS (ATUS, 2019). Although our stimuli were selected to clearly depict short, continuous actions performed by visible agents, their naturalistic and context-rich nature means that they could be understood as ‘events,’ encompassing elements that are not singularly specific to actions. A wealth of evidence has shown that context changes visual processing in a nonadditive way (Bar, 2004; Willems and Peelen, 2021), and emerging evidence suggests that the same is true for actions (Wurm et al., 2012). Studying actions in context holds promise for understanding how semantically rich representations emerge in naturalistic vision. This, in turn, will pave the way towards a computational understanding of the neural processes that link perception and cognition.

Materials and methods

Behavior: Stimuli

Request a detailed protocol

We curated two stimulus sets containing three-second videos of everyday actions from the Moments in Time dataset (Monfort et al., 2020). To broadly sample the space of everyday actions, we first identified the most common activities from the National Bureau of Labor Statistics’ American Time Use Survey (ATUS, 2019). Our final dataset included 18 social and nonsocial activities that lend themselves to visual representation (Table 1), to ensure a diverse and balanced stimulus set representative of the human everyday action space. We note that the ATUS distinctions are based on performed rather than observed actions. While imperfect, they provide ecologically relevant and objective criteria with which to define our action space.

Action categories were selected from the second-level activities identified in the ATUS. We used a minimum cutoff of 0.14 hr/day to select common actions (Table 1). To diversify our dataset, we added a ‘hiking’ category (to increase variability in scene setting) and a ‘fighting’ category (for variability along affective dimensions). In addition, ‘driving’ was selected as a more specific instance of the ‘travel’ categories in ATUS as it is the most common form of transportation in the United States. Some adjustments were also made to the ‘relaxing and leisure’ category by selecting two specific activities that were easy to represent and distinguish visually, as well as above threshold (‘reading’ and ‘playing games’). In addition, our ‘reading’ category included both ‘reading for personal interest’ and ‘homework and research’ as this distinction is difficult to convey visually. We omitted three leisure categories that were difficult to represent in brief videos (‘watching TV,’ ‘relaxing and thinking,’ and ‘computer use for leisure, excluding games’), as well as the ‘consumer goods purchases’ category.

We curated an initial set of 544 videos from the Moments in Time dataset by identifying the verb labels relevant to our chosen activities. Videos were chosen that were horizontally oriented (landscape), of reasonable image quality, clearly represented the activity in question, clearly depicted an agent performing the activity, and varied in terms of number of agents, gender and ethnicity of agents, and scene setting. Although some categories were represented by fewer verb labels than others in the final set, our curation procedure aimed to balance important features within and across action categories. We also curated a set of control videos depicting natural and indoors scenes.

We then selected two subsets of videos (1) that sampled all activities in a balanced manner and (2) where sociality (as assessed through behavioral ratings, see section ‘Behavioral ratings’) was minimally correlated to the number of agents (experimenter-labeled). This was done by randomly drawing 10,000 subsets of videos that sampled all activities equally and selecting the video set with the lowest correlation between sociality and the number of agents. These two features are difficult to disentangle in naturalistic stimulus sets, and we were able to minimize, though not fully eliminate, this correlation (Figure 1a).

The first stimulus set contained 152 videos (eight videos per activity and eight additional videos with no agents) and was used in Experiment 1. The videos with no agents were included to provide variation in the dataset along visual properties that did not pertain to actions or agents, as well as variation in the overall dataset in terms of number of agents per video. From the remaining videos, a second set of 76 videos was sampled and manually adjusted to remove videos without agents (in the interest of experimental time) and any videos that were too similar visually to other videos in the same category (e.g., involving a similar number of agents in similar postures). The second stimulus set thus contained 65 videos (three or four videos per activity) and was used in Experiment 2. The videos were preprocessed to a frame rate of 24 frames per second using the videoWriter object in MATLAB and resized to 600 × 400 pixels. This was done by first resizing the videos in order to meet the dimension criteria (using MATLAB’s imresize function with bicubic interpolation). The videos were then cropped to the correct aspect ratio either centrally or using manually determined coordinates to make sure the action remained clear after cropping.

Behavior: Participants

Behavioral ratings

Request a detailed protocol

A total of 256 workers (202 after exclusions, located in the United States, worker age and gender not recorded) from the online platform Amazon Mechanical Turk provided sociality, valence, arousal, and activity ratings of the video stimuli, and 43 workers (35 after exclusions) provided transitivity ratings.

Multiple arrangement

Request a detailed protocol

Two separate online multiple arrangement experiments were performed on each of the two stimulus sets. A total of 374 workers from Amazon Mechanical Turk took part in Experiment 1 (300 after exclusions, located in the United States, worker age and gender not recorded). Experiment 2 involved 58 participants (53 after exclusions, 31 female, 20 male, 1 non-binary, 1 not reported, mean age 19.38 ± 1.09) recruited through the Department of Psychological and Brain Sciences Research Portal at Johns Hopkins University.

All procedures for online data collection were approved by the Johns Hopkins University Institutional Review Board (protocol number HIRB00009730), and informed consent was obtained from all participants.

Behavior: Experimental procedure

Behavioral ratings

Request a detailed protocol

Participants viewed subsets of 30–60 videos from the initially curated large-scale set and rated the events depicted on a five-point scale. In a first set of experiments, the dimensions rated were sociality (how social the events were, from 1 – not at all to 5 – very social); valence (how pleasant the events were, from 1 – very unpleasant to 5 – very pleasant); arousal (how intense the events were, from 1 – very calm to 5 – very intense); and activity (how active they were, from 1 – no action to 5 – very active). In separate experiments, participants provided transitivity ratings for the two final stimulus sets (i.e., to what extent the actions involved a person or people interacting with an object, from 1 – not at all to 5 – very much). Participants were excluded if they responded incorrectly to catch trials (approximately 10% of trials) requiring them to label the action shown in the prior video, or if they provided overly repetitive ratings (e.g., using only two unique values or fewer out of five possible ratings throughout the entire experiment). This amounted to an average of 17.46 ± 2.14 ratings per video (Experiment 1) and 18.22 ± 2.09 ratings per video (Experiment 2). The experiments were implemented in JavaScript using the jsPsych library (de Leeuw, 2015).

Multiple arrangement

Request a detailed protocol

To characterize human action representations, we measured behavioral similarity using two multiple arrangement experiments. The experiments were conducted on the Meadows platform (https://meadows-research.com/) and required participants to arrange the videos according to their similarity inside a circular arena. Participants were free to use their own criteria to determine similarity, so as to encourage natural behavior.

Each trial started with the videos arranged around the circular arena. The videos would start playing on hover, and the trial would not end until all videos were played and dragged-and-dropped inside the arena (Figure 2). Different sets of videos were presented in different trials. An adaptive ‘lift-the-weakest’ algorithm was used to resample the video pairs placed closest together, so as to gather sufficient evidence (or improve the signal-to-noise ratio) for each pair. This procedure was repeated until an evidence criterion of 0.5 was reached for each pair or until the experiment timed out (Experiment 1: 90 min; Experiment 2: 120 min). By asking participants to zoom into the subsets previously judged as similar, the task required the use of different contexts and criteria to judge relative similarities. Compared to other methods of measuring similarity, multiple arrangement thus combines efficient sampling of a large stimulus set with adaptive behavior that can recover a multi-dimensional similarity structure (Kriegeskorte and Mur, 2012).

In Experiment 1, participants arranged different subsets of 30 videos from the 152-video set, with a maximum of 7 videos shown in any one trial. The stimuli were sampled in a balanced manner across participants. The task took on average 32 ± 14.4 min and 86.8 ± 22.6 trials.

In Experiment 2, all participants arranged the same 65 videos (entire 65-video set), with a maximum of 8 videos shown in any one trial. The task took on average 87.5 ± 24.6 min, including breaks, and 289.7 ± 57.3 trials.

The experiments included a training trial in which participants arranged the same seven videos (in Experiment 1) or eight videos (in Experiment 2) before beginning the main task. In both experiments, these videos were hand-selected to represent clear examples from four categories. Participants were excluded from further analysis if there was a low correlation between their training data and the average of all other participants’ data (over 2 SDs below the mean). They were also excluded if they responded incorrectly to a catch trial requiring them to label the action in previously seen videos.

Inverse MDS was used to construct behavioral dissimilarity matrices containing normalized Euclidean distances between all pairs of videos (Kriegeskorte and Mur, 2012). In Experiment 1, the behavioral RDM contained 11,476 pairs with an average of 11.37 ± 3.08 estimates per pair; in Experiment 2, there were 2080 pairs arranged by all 53 participants.

Behavior: Data analysis

Representational similarity analysis

Request a detailed protocol

Everyday actions can be differentiated along numerous axes. Perceptually, they can differ in terms of visual properties, like the setting in which they take place. They can also be characterized through action-related features, like semantic action category, or through social features, like the number of agents involved. Understanding how these features contribute to natural behavior can shed light on how naturalistic action representations are organized. Here, we used representational similarity analysis (RSA) to assess the contribution of visual, action, and social-affective features to the behavioral similarity data.

We quantified features of interest using image properties, labels assigned by experimenters (Supplementary file 1a), and behavioral ratings (provided by participants, see section ‘Behavioral ratings’). We calculated the Euclidean distances between all pairs of stimuli in each feature space, thus generating 17 feature RDMs.

To quantify visual features, image properties were extracted separately for each frame of each video and averaged across frames. These included pixel value (luminance), hue, saturation, optic flow (the magnitude of the optic flow estimated using the Horn–Schunck method), and the spatial envelope of each image quantified using GIST (Oliva and Torralba, 2001). We also extracted activations from the first convolutional layer and last fully-connected layer of a pretrained feedforward convolutional neural network (AlexNet; Krizhevsky et al., 2012). These features were vectorized prior to computing Euclidean distances between them (see Supplementary file 1b for the dimensionality of each feature). Two additional experimenter-labeled features were included: scene setting (indoors/outdoors) and the presence of a watermark. To assess whether a motion energy model (Adelson and Bergen, 1985; Nishimoto et al., 2011; Watson and Ahumada, 1985) would better capture the impact of motion, we performed control analyses by computing motion energy features for each video using a pyramid of spatio-temporal Gabor filters with the pymoten package (Nunez-Elizalde et al., 2021).

Action feature RDMs were based on transitivity and activity ratings (provided by participants, see above), as well as action category (a binary RDM clustering the stimuli into activity categories based on the initial dataset designations) and effectors (experimenter-labeled). The latter consisted of binary vectors indicating the involvement of body parts in each action (face/head, hands, arms, legs, and torso). To assess whether a more detailed semantic model would capture more information, we also performed a control analysis using a feature RDM based on WordNet similarities between the verb labels in the ‘Moments in Time’ dataset (Figure 4—figure supplement 1).

Social-affective feature RDMs were based on sociality, valence, and arousal ratings (all provided by participants, see section ‘Behavioral ratings’ above) and the number of agents in each video, which was labeled by experimenters on a four-point scale (from 0, no agent present, to 3, three or more agents present).

Each participant’s behavioral RDM was correlated to the feature RDMs, and the resulting Kendall’s τA values were tested against chance using one-tailed sign permutation testing (5000 iterations). P-values were omnibus-corrected for multiple comparisons using a maximum correlation threshold across all models (Nichols and Holmes, 2002).

A noise ceiling was calculated by correlating each subject’s RDM to the average RDM (upper bound), as well as to the average RDM excluding the left-out subject (lower bound; Nili et al., 2014).

Variance partitioning

Request a detailed protocol

Despite low correlations between features of interest in both stimulus sets (Figure 1a), shared variance could still contribute to the RSA results. To estimate the unique contributions of the three primary groups of features, we performed a cross-validated variance partitioning analysis, excluding individual features that did not correlate with the behavioral data in the above RSA analysis. The three groups included visual features (scene setting and the last fully connected layer of AlexNet), action features (action category, effectors, transitivity, action), and social-affective features (number of agents, sociality, valence, arousal).

The behavioral data were randomly split into training and test sets (100 iterations) by leaving out half of the individual similarity estimates for each pair of videos in Experiment 1 (since different participants saw different subsets of videos) or half of the participants in Experiment 2. We fit seven different regression models using the average training RDM (with every possible combination of the three groups of features), and we calculated the squared Kendall’s τA between the predicted responses and the average test RDM. These values were then used to calculate the unique and shared portions of variance contributed by the predictors (Groen et al., 2018; Lescroart et al., 2015; Tarhan et al., 2021).

The resulting values were tested against chance using one-tailed sign permutation testing (5000 iterations, omnibus-corrected for multiple comparisons). Differences between groups of features were assessed with two-sided Wilcoxon signed-rank tests.

EEG: Stimuli

Request a detailed protocol

The stimulus set from behavioral Experiment 1 was used in the EEG experiment, containing 152 videos from 18 categories, as well as control videos. The three-second stimuli were trimmed to a duration of 0.5 s centered around the action as determined by visual inspection, to ensure that the shorter videos were easily understandable. This helped improve time-locking to the EEG signals and allowed for a condition-rich experimental design. An additional 50 videos were included as catch stimuli (25 easily identifiable pairs depicting the same action, manually chosen from the larger stimulus set).

EEG: Participants

Request a detailed protocol

Fifteen participants (six female, nine male, mean age 25.13 ± 6.81) took part in the EEG experiment. All participants were right-handed and had normal or corrected-to-normal vision. Informed consent was obtained in accordance with the Declaration of Helsinki, and all procedures were approved by the Johns Hopkins University Institutional Review Board (protocol number HIRB00009835).

EEG: Experimental procedure

Request a detailed protocol

Continuous EEG recordings with a sampling rate of 1000 Hz were made with a 64-channel Brain Products ActiCHamp system using actiCAP electrode caps in a Faraday chamber. Electrode impedances were kept below 25 kΩ when possible, and the Cz electrode was used as an online reference.

Participants were seated upright while viewing the videos on a back-projector screen situated approximately 45 cm away. The 152 videos were shown in pseudorandom order in each of 10 blocks with no consecutive repetition allowed. In addition, four repetitions of the 25 catch video pairs were presented at random times during the experiment. The video pairs were presented in different orders to minimize learning effects, so that for each video pair (V1, V2), half of the presentations were in the order V1-V2 and half of them were in the order V2-V1. Participants performed a one-back task and were asked to press a button on a Logitech game controller when they detected two consecutive videos showing the same action. Participants were not instructed on what constituted an action, beyond being given ‘eating’ as a simple example. There was a break every 150 trials, and participants could continue the experiment by pressing a button. In total, the experiment consisted of 1720 trials (1520 experimental trials and 200 catch trials) and took approximately 45 min.

The stimuli were presented using an Epson PowerLite Home Cinema 3000 projector with a 60 Hz refresh rate. Each trial started with a black fixation cross presented on a gray screen for a duration chosen from a uniform distribution between 1 and 1.5 s, followed by a 0.5 s video. The stimuli were presented on the same gray background and subtended approximately 15 × 13 degrees of visual angle. The fixation cross remained on screen, and participants were asked to fixate throughout the experiment. A photodiode was used to accurately track on-screen stimulus presentation times and account for projector lag. The paradigm was implemented in MATLAB R2019a using the Psychophysics Toolbox (Brainard, 1997; Kleiner et al., 2007; Pelli, 1997).

EEG: Data analysis

Preprocessing

Request a detailed protocol

EEG data preprocessing was performed using MATLAB R2020b and the FieldTrip toolbox (Oostenveld et al., 2011). First, the EEG data were aligned to stimulus onset using the photodiode data to correct for any lag between stimulus triggers and on-screen presentation. The aligned data were segmented into 1.2 s epochs (0.2 s pre-stimulus to 1 s post-stimulus onset), baseline-corrected using the 0.2 s prior to stimulus onset, and high-pass filtered at 0.1 Hz.

Artifact rejection was performed using a semi-automated pipeline. First, the data were filtered between 110 and 140 Hz and Hilbert-transformed to detect muscle artifacts; segments with a z-value cutoff above 15 were removed. Next, channels and trials with high variance were manually rejected based on visual inspection of a summary plot generated using the ft_rejectvisual function in FieldTrip. Finally, independent component analysis (ICA) was performed to identify and remove eye movement components from the data.

Catch trials were removed from the data together with any trials that elicited a button response (13.74 ± 1.82% of all trials). Of the remaining trials, 8.36 ± 5.01% (ranging between 25 and 275 trials) were removed during the artifact rejection procedure. A maximum of two noisy electrodes were removed from eight participants’ datasets.

Prior to further analysis, the data were re-referenced to the median across all electrodes, low-pass filtered at 30 Hz to investigate evoked responses, and downsampled to 500 Hz.

Multivariate analysis

Request a detailed protocol

We performed multivariate analyses to investigate (1) whether EEG patterns reflected behavioral similarity and (2) whether different visual, action, and social-affective features explained variance in the neural data.

First, time-resolved decoding of every pair of videos was performed using a linear support vector machine classifier as implemented in the LibSVM library (Chang and Lin, 2011). Split-half cross-validation was used to classify each pair of videos in each participant’s data. To do this, the single-trial data was divided into two halves for training and testing, whilst ensuring that each condition was represented equally. To improve SNR, we combined multiple trials corresponding to the same video into pseudotrials via averaging. The creation of pseudotrials was performed separately within the training and test sets. As each video was shown 10 times, this resulted in a maximum of five trials being averaged to create a pseudotrial. Multivariate noise normalization was performed using the covariance matrix of the training data (Guggenmos et al., 2018). Classification between all pairs of videos was performed separately for each time point. Data were sampled at 500 Hz, and so each time point corresponded to nonoverlapping 2 ms of data. Voltage values from all EEG channels were entered as features to the classification model.

The entire procedure, from dataset splitting to classification, was repeated 10 times with different data splits. The average decoding accuracies between all pairs of videos were then used to generate a neural RDM at each time point for each participant. To generate the RDM, the dissimilarity between each pair of videos was determined by their decoding accuracy (increased accuracy representing increased dissimilarity at that time point).

Next, we evaluated the correlations between each participant’s time-resolved neural RDM and the feature RDMs found to correlate with behavioral similarity (Experiment 1). To investigate the link between behavioral and neural representations, we also correlated neural RDMs with the average behavioral RDM obtained from the multiple arrangement task in Experiment 1. This analysis was performed using 10 ms sliding windows with an overlap of 6 ms. The resulting Kendall’s τA values were tested against chance using one-tailed sign permutation testing (5000 iterations, cluster-corrected for multiple comparisons across time using the maximum cluster sum, α = 0.05, cluster setting α = 0.05). A noise ceiling was calculated using the same procedure as in the behavioral RSA (see ‘Representational similarity analysis’). Effect latencies were assessed by bootstrapping the individual correlations 1000 times with replacement to calculate 90% confidence intervals around effect onsets.

To quantify the contributions of visual, social-affective, and action features to the neural RDMs, a time-resolved cross-validated variance partitioning procedure was performed. Using 100 split-half cross-validation iterations, the neural RDM was entered as a response variable in a hierarchical regression with three groups of feature RDMs (visual, social-affective, and action) as predictors. This analysis employed the same 10 feature RDMs used in the behavioral variance partitioning (see section ‘Variance partitioning’), with the addition of activations from the first convolutional layer of AlexNet (Conv1). As Conv1 best captures early visual responses (Figure 5—figure supplement 1), its inclusion ensured that we did not underestimate the role of visual features in explaining neural variance. We did not use frame-wise RDMs to model these visual features; however, our approach of averaging features across video frames was justified by the short duration of our videos and the high correlation of CNN features across frames (Conv1: Pearson’s ρ=0.89±0.09; FC8: ρ=0.98±0.03).

The analysis was carried out using 10 ms sliding windows with an overlap of 6 ms. The resulting predicted Kendall’s τA values were tested against chance using one-tailed sign permutation testing (5000 iterations, cluster-corrected for multiple comparisons using the maximum cluster sum across time windows and regressions performed, α = 0.05, cluster-setting α = 0.05). The distributions of effect onsets across the 100 split-half iterations were compared using two-sided Wilcoxon signed-rank tests.

Data availability

Request a detailed protocol

Behavioral and EEG data and results have been archived as an Open Science Framework repository (https://osf.io/hrmxn/). Analysis code is available on GitHub (https://github.com/dianadima/mot_action; Dima, 2021).

Data availability

Behavioral and EEG data and results have been archived as an Open Science Framework repository (https://osf.io/hrmxn/). Analysis code is available on GitHub (https://github.com/dianadima/mot_action, copy archived at swh:1:rev:af9eede56f27215ca38ddd32564017f1f90417d0).

The following data sets were generated
    1. Dima DC
    2. Tomita TM
    3. Honey CJ
    4. Isik L
    (2021) Open Science Framework
    ID hrmxn. Social-affective features drive human representations of observed actions.

References

    1. Adelson EH
    2. Bergen JR
    (1985) Spatiotemporal energy models for the perception of motion
    Journal of the Optical Society of America. A, Optics and Image Science 2:284–299.
    https://doi.org/10.1364/josaa.2.000284
  1. Report
    1. ATUS
    (2019)
    American Time Use Survey, United States Department of Labor
    Bureau of Labor Statistics.
    1. Brainard DH
    (1997)
    The Psychophysics Toolbox
    Spatial Vision 10:433–436.
    1. Chang CC
    2. Lin CJ
    (2011) LIBSVM: A Library for support vector machines
    ACM Transactions on Intelligent Systems and Technology 2:1961199.
    https://doi.org/10.1145/1961189.1961199
    1. Kleiner M
    2. Brainard DH
    3. Pelli DG
    4. Broussard C
    5. Wolf T
    6. Niehorster D
    (2007)
    What’s new in Psychtoolbox-3?
    Perception 36:1–16.
    1. Papeo L
    (2020) Twos in human visual perception
    Cortex; a Journal Devoted to the Study of the Nervous System and Behavior 132:473–478.
    https://doi.org/10.1016/j.cortex.2020.06.005
    1. Watson AB
    2. Ahumada AJ
    (1985) Model of human visual-motion sensing
    Journal of the Optical Society of America. A, Optics and Image Science 2:322–341.
    https://doi.org/10.1364/josaa.2.000322

Decision letter

  1. Chris I Baker
    Senior and Reviewing Editor; National Institute of Mental Health, National Institutes of Health, United States
  2. Angelika Lingnau
    Reviewer; Institute of Psychology, University of Regensburg, Germany
  3. Mark Lescroart
    Reviewer; University of Nevada at Reno, United States

Our editorial process produces two outputs: i) public reviews designed to be posted alongside the preprint for the benefit of readers; ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.

Decision letter after peer review:

Thank you for submitting your article "Social-affective features drive human representations of observed actions" for consideration by eLife. Your article has been reviewed by 2 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Chris Baker as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Angelika Lingnau (Reviewer #1); Mark Lescroart (Reviewer #2).

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

1) Reviewer 1 (Angelika Lingnau) notes and I agree that there are some needed qualifications/clarifications needed about the strength of the distinctions between visual, action, and social features. Given the correlations between these features, some steps need to be taken analytically to strengthen the dissociation or the claims should be attenuated. Note also her thorough smaller comments that should all be directly addressed to strengthen the paper.

2) Relatedly, there are a number of clarifications requested by Reviewer 2 (Mark Lescroart) about the construction of the features and the procedure used in the information analyses. These concerns should be carefully addressed as they will strengthen the paper.

Reviewer #1 (Recommendations for the authors):

Below please find a number of comments that might help strengthen the science and the presentation of this manuscript.

(1) Assignment of features to domains (e.g. page 4, line 92ff): As mentioned above, in the view of this reviewer some of the assignments between features to domains were less clear. As an example why is the setting in which the action takes place considered to be a visual feature (in contrast to, say, a semantic feature, such as indoors or outdoors)? Likewise, why are activations from the final layer of a pretrained neural network considered a high-level visual feature rather than a conceptual/semantic feature? Why is 'activity' (the amount of activity) in a video considered an action-related feature instead of a visual feature? Why is the number of agents considered a social-affective feature (instead of a visual feature; see also my next comment)? Regarding the latter, the authors admit that this feature might be considered either visual or social-affective.

(2) I liked the attempts to try to minimize the correlations between features, which is a known problem when working with naturalistic stimuli. However, in light of the significant correlations between some of the features (in particular between sociality and the number of agents), I am concerned about biases in the estimation of the β weights, which in turn may have impacted the results of the variance partitioning analysis. Moreover, if the features sociality and number of agents are correlated, and if we accept that the number of agents might be a visual rather than a social feature, how do we know that the results obtained for the feature sociality really are based on a social-affective feature?

(3) Methods (page 4, line 89): What was the reason to choose 8 additional videos with no agents?

(4) Figure 1: It might be helpful for the reader to indicate in the figure which of the features are considered to be visual, action-related or socio-affective.

(5) Figure 1B: Please provide some more details on how these plots were generated, and what they show.

(6) Multi-arrangement task (page 7, line 130f): At several points in the manuscript, the authors write that they collected behavioural similarity ratings when referring to the multi-arrangement task. This wording might be confusing since the multi-arrangement task did not require an explicit rating (in contrast to the ratings the authors collected from a separate set of participants for the features sociality, valence, arousal, activity, and transitivity). The authors may want to use a different wording when referring to the results of the multi-arrangement task.

(7) Related to the previous point, I'm somewhat concerned regarding the fact that participants were not provided with specific instructions as to the criteria they should follow when performing the multi-arrangement task. Whereas the authors argue that this was done to emphasize natural behaviour, in the view of this reviewer this approach comes with the risk of not knowing what the participants did in the end. To give concrete examples, it is possible that some participants focused on the context of the actions, while other participants focused on the valence, while yet another set of participants focused on some low level visual features. Moreover, participants might have changed their strategies throughout the experiment if they figured out that a criterion they first considered useful no longer worked for some of the configurations. Whereas the reported leave-one-subject-out correlations were above chance in both experiments, they were overall quite low, which might be related to the issue described above. How can the authors rule out such behaviour (and if not, how could this impact the interpretation of the results)?

(8) Figure 4: The overall differences between the three feature types appear to be quite small. What is the contribution of the number of agents for the social features (see also my corresponding comment on Figure 1a/ correlations between these two features)?

(9) On page 10, 2nd paragraph, the authors report the variance explained by the ten predictors in Experiment 2. What about the corresponding value for Experiment 1?

(10) Task EEG experiment (e.g. page 11, line 212f): participants had to perform a one-back action task in which they detected repetitions of the action category. Did they know what the action categories were (see also related question on the contents of Table 1)?

(11) Figure 5B: maybe add 'noise ceiling' to the legend?

(12) Discussion: the authors write that sociality did not correlate with the EEG patterns. Could it be that different participants used different strategies when performing this rating? (see corresponding comment on instructions for rating of sociality, page 22).

(12) Discussion: page 18: The authors argue for a hierarchy from visual to action-related to socially relevant computations. How might the one-back task (focusing on repetitions of the same action category) have contributed to this result?

(13): Table 1 (page 20): The mapping between the verb labels provided in Table 1 and the 8 exemplars used for each activity did not become quite clear to me. It seems from Table 1 that some of the categories consisted of a range of different actions (e.g. crying, cuddling, feeding, giggling, socializing for the activity 'childcare'), while some of them consisted of one single action only (e.g. reading), which makes the categories hard to compare, both in terms of the underlying perceptual variability and the task difficulty associated with judging a repetition of that category. Please comment.

(14) Table 1 (page 20): Unless I misunderstood, the action 'socializing' occurred within four of the 18 activities (in addition to being one of the 18 activities). What was the reason for this? Could this have impacted the results, showing a stronger contribution of socio-affective features in comparison to action-related and visual features? Likewise, could the results obtained in this study be affected by the curation of the two stimulus datasets (adjusted to include both social and non-social activities)?

(15) Page 20: What was the reason to use these specific common activities? Was there a specific cutoff?

(16) Page 21, line 404f: What was the purpose of the control videos depicting natural and indoors scenes?

(17) Page 21, line 415f: Please explain what 'repetitive' videos mean in this context.

(18) Page 22, behavioural ratings: What were the exclusion criteria for the behavioural ratings and the multiple arrangement task?

(19) Page 22: Participants were instructed to rate 'how social the events were'. In the view of this reviewer, this leaves a lot of room for interpretation (and also differs from the instructions given in previous studies). Please comment.

(20) Page 22: How consistent were the behavioural ratings across participants? Was variability across participants higher for some features than for others?

(21) Page 23 line 454f: How was the subset of videos presented in trial 1 selected?

(22) Page 28 line 541f: According to which criteria did the authors center the 0.5 seconds window around the action?

(23) Page 28 line 557f: What do the authors mean by 'shuffled pairs'?

(24) Page 29, 2nd paragraph: Please provide details on the projector.

Reviewer #2 (Recommendations for the authors):

If I have read the paper (and code) correctly, it seems that a single RDM was used to model all timepoints in the EEG responses, and that single RDM was correlated with the RDMs generated by the pairwise decoding analysis for each timepoint. For purposes of modeling behavioral data – in which the subjects are making a judgment based on the entire video clip – a single RDM per video clip seems reasonable. However, for purposes of modeling time-varying neural responses, it may be less than ideal to use a single RDM to summarize each video. The usefulness of a single temporal summary RDM will depend on how homogenous the features are across the video clip. Having looked over a few of the video clips in Moments in Time, the clips appear to be generally slow-moving – i.e., no large changes in background elements or actors across the 3 seconds. This seems like a reasonable justification for the use of a single temporal summary RDM; I would encourage the authors to clarify their rationale for using one RDM for the whole timecourse instead of framewise RDMs. Quantitative arguments would be useful.

[Editors’ note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Social-affective features drive human representations of observed actions" for further consideration by eLife. Your revised article has been evaluated by Chris Baker (Senior Editor) in consultation with the original reviewers.

We all agree the manuscript has been improved, the issues raised by the reviewers have been addressed well, and the manuscript will make a nice addition to the literature. But one of the reviewers highlighted a remaining concern that I wanted to draw your attention to. This concern could be addressed with additional analyses or additional discussion and motivation of the approach you used (I will leave it up to you which avenue you want to pursue). I'll quote from the reviewer:

"Among the visual features the authors investigate is motion. To me, this is a visual feature that would seem to have a strong potential to explain the EEG data in particular, since motion contrast strongly drives many visual areas. I am not completely satisfied with the way the authors have quantified motion, and I have a medium concern that their motion model is a bit of a straw man. First, they refer to the model as a "motion energy" model, which is not technically correct. Motion energy (i.e. Adelson and Bergen, 1985; Nishimoto et al., 2011) reflects image contrast, whereas optic flow does not (at least to the same degree). For example, a gray square moving 2 degrees on a slightly less gray background will generate the same magnitudes of optic flow vectors as a starkly black square on a white background moving the same distance. Motion energy will be substantially greater for higher-contrast stimuli. As such, it's likely that motion energy would be a better model for visual responses (at least in early visual areas) than optic flow. They also choose to compute optic flow densely, with one value per pixel. Thus, if actors in the various videos are in slightly different locations, the optic flow metric could be quite different, and the RDM may characterize arguably similar stimuli as distinct. I think that a multi-scale pyramid of optic flow (or better still, motion energy) would have a better chance of capturing motion selectivity in the way that the brain does.

There is one more choice that the authors make that I'm not entirely comfortable with, which relates to this same issue. They choose to use only the models that can be related to behavioral data to model the EEG data. This choice is motivated by the desire to relate brain activity to behavior, which is a worthwhile endeavor. However, I think there is an assumption underlying this choice: the authors assume that the signal that they measure with EEG must reflect neural representations that guide behavior and not low-level perceptual processes. I do not think that this is guaranteed to be the case. EEG measures a spatially diffuse signal, which may well be dominated by activity from quite low-level areas (e.g. V1). I think the question of whether the EEG signal reflects relatively high-level cognitive processing or relatively low-level perceptual processing – for any given experiment – is still open. For most of the models they exclude, I don't think this is a big issue. For example, I think it's reasonable to test luminance basically as a control (to show that there aren't huge differences in luminance that can explain behavior) and then to exclude it from the EEG modeling. However, I'm less happy with exclusion of motion, based on a somewhat sub-optimal motion model not explaining behavior.

The combination of these two issues has left me wanting a better quantification of motion as a model for the EEG data. My bet would be that motion would be a decent model for the EEG signal at an earlier stage than they see the effects of action class and social factors in the signal, so I don't necessarily think that modeling motion would be likely to eliminate the effects they do see; to my mind, it would just present a fairer picture of what's going on with the EEG signal."

https://doi.org/10.7554/eLife.75027.sa1

Author response

Reviewer #1 (Recommendations for the authors):

Below please find a number of comments that might help strengthen the science and the presentation of this manuscript.

(1) Assignment of features to domains (e.g. page 4, line 92ff): As mentioned above, in the view of this reviewer some of the assignments between features to domains were less clear. As an example why is the setting in which the action takes place considered to be a visual feature (in contrast to, say, a semantic feature, such as indoors or outdoors)? Likewise, why are activations from the final layer of a pretrained neural network considered a high-level visual feature rather than a conceptual/semantic feature? Why is 'activity' (the amount of activity) in a video considered an action-related feature instead of a visual feature? Why is the number of agents considered a social-affective feature (instead of a visual feature; see also my next comment)? Regarding the latter, the authors admit that this feature might be considered either visual or social-affective.

This is an important point, and we agree the distinction between feature domains is not always clear cut. To answer the question about why the setting and final DNN layer are considered visual: since this work focused on actions, we considered action-related features to pertain to actions and their semantics. Thus, we considered the setting of an action to be visual and not semantic, since it only pertains to the action’s context. Similarly, the neural network we used was pretrained on ImageNet, and as such the features extracted by it would tend to describe the environment and objects involved rather than the action.

The word ‘visual’, particularly when contrasted with ‘action’ and ‘social-affective’, may indeed suggest only low-level information; however, we believe the distinction between context, actions, and agents to be relevant as it captures the variability of naturalistic actions. For example, an action like ‘eating’ would be characterized by action features (action category, transitivity, effectors involved and amount of activity) that would stay largely consistent across exemplars. However, exemplars may vary in terms of context (eating in the kitchen vs at a park), object (eating an apple vs a sandwich), and agents (eating alone vs together). We clarified these points in the paper, when introducing the visual features :

“Naturalistic videos of actions can vary along numerous axes, including visual features (e.g. the setting in which the action takes place or objects in the scene), action-specific features (e.g. semantic action category), and social-affective features (e.g. the number of agents involved or perceived arousal). For example, an action like ‘eating’ may vary in terms of context (in the kitchen vs at a park), object (eating an apple vs a sandwich), and number of agents (eating alone vs together). Drawing these distinctions is crucial to disambiguate between context, actions, and agents in natural events. […] Visual features ranged from low-level (e.g. pixel values) to high-level features related to scenes and objects (e.g. activations from the final layer of a pretrained neural network).”

Combining these scene-related features with action-related features does, indeed, reach an explanatory power almost as high as that of social-affective features in Experiment 1 (but not in Experiment 2). This can also be inferred from our original Figure 4 by looking at the effect sizes.

In our view, this strengthens our conclusions. Considering that this analysis discounts any variance shared with high-level visual, semantic, and action features, the significantly greater unique contribution of social-affective features is striking. However, the difference in effect size between the two experiments becomes more apparent than in our previous analysis (Figure 4), and we now discuss this in the manuscript :

“Although the assignment of features to domains was not always straightforward, our results were robust to alternative assignment schemes. For example, high-level visual features can be seen as bordering the semantic domain, while features like the number of agents or the amount of activity can be seen as visual. However, feature assignment was not the main factor driving our results, which stayed the same even when the activity feature was assigned to the visual group. More strikingly, the social-affective feature group explained significantly more variance than all other features grouped together in both experiments (Figure 4 —figure supplement 2). This is a particularly stringent test as it pits the unique and shared contributions of all visual, semantic, and action features against the four social-affective features. In Experiment 1, the combined contribution of visual and action features approached that of social-affective features, while in Experiment 2 the difference was larger. Together with the larger contribution of the number of agents in Experiment 2, this suggests that Experiment 2 may have captured more social information, potentially thanks to the exhaustive sampling of the stimuli which allowed each participant to arrange the videos according to different criteria.”

We also agree that the number of agents straddles the boundary between visual and social features. However, our control analysis showed that the number of agents did not explain the unique contribution of social-affective features to behavior (Figure 4 —figure supplement 3 ).

Finally, we took “activity” to pertain to actions (is it a static action like reading, or a high-movement action like running?) We agree that it could also be considered a visual feature along the lines of motion energy. However, the assignment of this feature to the visual group does not impact our conclusions, which we now note :

“However, feature assignment was not the main factor driving our results, which stayed the same even when the activity feature was assigned to the visual group.”

Author response image 1
Unique contributions of visual features (environment, FC8, activity), action features (action category, effectors, transitivity) and social-affective features (number of agents, sociality, valence and arousal).
Author response image 2
Social-affective features (sociality, valence, and arousal) still contribute more variance than visual features (environment, FC8, activity, and number of agents) and action features (action category, transitivity, and effectors).

(2) I liked the attempts to try to minimize the correlations between features, which is a known problem when working with naturalistic stimuli. However, in light of the significant correlations between some of the features (in particular between sociality and the number of agents), I am concerned about biases in the estimation of the β weights, which in turn may have impacted the results of the variance partitioning analysis. Moreover, if the features sociality and number of agents are correlated, and if we accept that the number of agents might be a visual rather than a social feature, how do we know that the results obtained for the feature sociality really are based on a social-affective feature?

Indeed, despite our efforts, there were still moderate correlations between certain related features. However, the variance inflation factors were low (1.34 in Exp 1 and 1.37 in Experiment 2), suggesting that the regressions did not suffer from collinearity. Furthermore, different assignments of features to groups (see above) did not impact the overall pattern of results.

It is true that these analyses do not allow us to draw conclusions about individual features, as variance partitioning with more than three sets of features becomes intractable. Specifically, the role of sociality and number of agents is unclear.

We performed an additional analysis looking at the unique contributions of the number of agents, sociality, and the grouped affective features (valence and arousal): Figure 4 —figure supplement 4. This has the advantage of removing any shared variance between the number of agents and sociality. Based on this analysis, it appears that valence and arousal best accounted for behavioral similarity estimates, with the number of agents also contributing a significant amount, particularly in Experiment 2. However, this analysis does not partial out the contributions of other features (visual or action-related). Grouping the two affective features together may have also advantaged them over the single social features. Given these limitations and the post-hoc, exploratory nature of this new analysis, we prefer to focus our interpretation on feature domains and their contributions. Nonetheless, we have adjusted our claims in light of this new finding, as outlined below.

We discuss this additional analysis in the manuscript :

“Furthermore, an additional analysis looking at the separate contributions of the number of agents, sociality, and affective features (valence and arousal) found that the affective features contributed the greatest variance in both experiments (Figure 4 —figure supplement 4). For technical reasons, this analysis compared the joint contribution of both affective features to each single social feature and did not discount the impact of variance shared with visual or action-related features. Despite these limitations, the results suggest that the contribution of the social-affective feature group is not driven by the number of agents or the variance it shares with sociality, and highlight the role of affective features (valence and arousal) in explaining behavior.”

We also discuss these findings in the Discussion :

“An exploratory follow-up analysis showed that this effect was primarily driven by affective features (valence and arousal), with the number of agents as a secondary contributor. Recent work found that affective features drive the perceived similarity of memories of real-life events (Tomita et al., 2021), suggesting that these features bridge the action, event and memory domains in organizing mental representations.”

Interestingly, this new finding is in line with our EEG results, where the number of agents, valence and arousal correlated with the neural responses, while sociality did not. Although this is an exploratory finding, we briefly discuss it :

“Note that this finding is mirrored in our behavioral results, where we observed larger unique contributions from valence, arousal and the number of agents than sociality (Figure 4 —figure supplement 4).”

(3) Methods (page 4, line 89): What was the reason to choose 8 additional videos with no agents?

We added this explanation to page 4 , as well as the Methods :

“and 8 additional videos with no agents, included to add variation in the dataset, see Materials and methods, section (Behavior: Stimuli)”

“The videos with no agents were included to provide variation in the dataset along visual properties that did not pertain to actions or agents, as well as variation in the overall dataset in terms of number of agents per video.”

(4) Figure 1: It might be helpful for the reader to indicate in the figure which of the features are considered to be visual, action-related or socio-affective.

We added a color-coded legend to the figure to make it clearer at a glance how features were assigned to groups .

(5) Figure 1B: Please provide some more details on how these plots were generated, and what they show.

We added the following caption to this figure panel :

“Behavioral rating distributions in the two stimulus sets. The z-scored ratings were visualized as raincloud plots showing the individual data points, as well as probability density estimates computed using Matlab’s ksdensity function (Allen et al., 2019).”

This panel shows the distribution of labeled features across our two stimulus sets, in particular that the ratings were consistent and stimuli spanned a wide range along each feature.

(6) Multi-arrangement task (page 7, line 130f): At several points in the manuscript, the authors write that they collected behavioural similarity ratings when referring to the multi-arrangement task. This wording might be confusing since the multi-arrangement task did not require an explicit rating (in contrast to the ratings the authors collected from a separate set of participants for the features sociality, valence, arousal, activity, and transitivity). The authors may want to use a different wording when referring to the results of the multi-arrangement task.

Thank you for bringing this to our attention. Indeed, these were not ratings in the sense that the other features were behaviorally rated. We replaced such wording throughout the manuscript with either ‘similarity estimates’ or ‘similarity judgments’. (This latter wording is more accurate in our view, since participants were instructed to judge the similarity of the videos in order to place them on screen.) See pages 7, 8, 27, 31, 43 etc.

(7) Related to the previous point, I'm somewhat concerned regarding the fact that participants were not provided with specific instructions as to the criteria they should follow when performing the multi-arrangement task. Whereas the authors argue that this was done to emphasize natural behaviour, in the view of this reviewer this approach comes with the risk of not knowing what the participants did in the end. To give concrete examples, it is possible that some participants focused on the context of the actions, while other participants focused on the valence, while yet another set of participants focused on some low level visual features. Moreover, participants might have changed their strategies throughout the experiment if they figured out that a criterion they first considered useful no longer worked for some of the configurations. Whereas the reported leave-one-subject-out correlations were above chance in both experiments, they were overall quite low, which might be related to the issue described above. How can the authors rule out such behaviour (and if not, how could this impact the interpretation of the results)?

In this work, we specifically focused on intuitive (and thus unconstrained) similarity. Indeed, this means that participants could have used different criteria or changed their strategies over time; however, we argue that this is a strength of our approach. We used the iterative multiple arrangement task as implemented in Meadows specifically because by varying the groups of stimuli shown in each trial it is able to recover a multidimensional representation, with many potential features coming into play. Although this method is not as exhaustive as, for example, an odd-one-out task (e.g. Hebart et al., 2021), it was a good compromise in terms of handling a large stimulus set and recovering a multidimensional structure. While the leave-one-subject-out correlations were only moderate, they reached a similar level as seen in related prior studies (e.g., Tarhan et al., Neuropsychologia, 2021) with smaller, more homogenous datasets. Finally, we note that all reported measures in the paper are in terms of Kendall’s Tau-A, which is a more conservative metric than the more commonly reported Spearman’s correlation.

We also argue that our ability to find and replicate the main organizing axes of this space, despite the many sources of potential variability you outline above, suggests that there is a common element to participants’ multiple arrangement strategies. Recent work on intuitive action understanding supports this by suggesting that intuitive arrangements may be guided by judgments of action goals (Tarhan et al., Neuropsychologia, 2021). We have added the following lines to the Results :

“The multiple arrangement task was unconstrained, which meant that participants could use different criteria. Although this may have introduced some variability, the adaptive algorithm used in the multiple arrangement task enabled us to capture a multidimensional representation of how actions are intuitively organized in the mind, while at the same time ensuring sufficient data quality.”

(8) Figure 4: The overall differences between the three feature types appear to be quite small. What is the contribution of the number of agents for the social features (see also my corresponding comment on Figure 1a/ correlations between these two features)?

Although the differences are small, they are significant relative to each other and as a percentage of the true correlation. In Experiment 1, social-affective features explain more than twice the unique variance explained by visual and action features (although note that the lower reliability of this data makes the true correlation more difficult to trust as a measure of ‘best possible fit’). In Experiment 2, social-affective features uniquely contribute 58% of the true correlation, with only 9% contributed by action features and approximately 0.5% contributed by visual features. We added more information on this to the Results:

“Although the effect sizes were relatively low, social-affective features explained more than twice as much unique variance as either the visual or action features in Experiment 1, and six times as much in Experiment 2. Furthermore, given the limits placed on predictivity by the reliability of the behavioral data, affective features predicted a large portion of the explainable variance in both experiments.”

We had not analyzed the unique contributions of each feature within each group, as this is difficult to assess in the context of a variance partitioning analysis given the large number of features (please see our response to point 2). Our above analyses (see Figure 4 – supplements 3 and 4) suggest that the social-affective dominance is not driven by the number of agents.

(9) On page 10, 2nd paragraph, the authors report the variance explained by the ten predictors in Experiment 2. What about the corresponding value for Experiment 1?

We have edited the text to make this clearer. The section now reads as follows :

“In Experiment 1, the predicted squared Kendall’s τA of the full model (τA2=0.06±0.001 ) was higher on average than the true split-half squared correlation (τA2=0.04±0.002). This is likely to be due to the lower reliability of the behavioral similarity data in this experiment, and suggests that the ten predictors are able to explain the data well despite the overall lower prediction accuracy.”

Due to the lower reliability of this dataset , we did not want to over-interpret the prediction accuracy obtained here.

(10) Task EEG experiment (e.g. page 11, line 212f): participants had to perform a one-back action task in which they detected repetitions of the action category. Did they know what the action categories were (see also related question on the contents of Table 1)?

The task was not constrained by the action categories used in our dataset curation. Participants were simply instructed to press a button when they identified the same action in consecutive videos, and given ‘eating’ as a simple example. However, the catch video pairs contained more easily identifiable repeated actions (e.g. people doing ballet rather than people doing different sports, which might have been more difficult to label as a repeat). This ensured that participants could perform the task without biasing their semantic categorization. We have clarified this in the text :

“Participants were not instructed on what constituted an action, beyond being given “eating” as a simple example.”

(11) Figure 5B: maybe add 'noise ceiling' to the legend?

Thank you. We have added this .

(12) Discussion: the authors write that sociality did not correlate with the EEG patterns. Could it be that different participants used different strategies when performing this rating? (see corresponding comment on instructions for rating of sociality, page 22).

This is an interesting point. We assessed the reliability of the ratings using leave-one-subject-out correlations, and did not find sociality to be less consistent than other ratings (Figure 1 —figure supplement 1). In Experiment 1, sociality was more reliable than all other ratings, and significantly more reliable than valence, arousal and activity ratings. In Experiment 2, there was no significant difference between sociality and the other ratings, except transitivity, which was more reliable. The fact that we do see EEG correlations with other features like valence and arousal seems to suggest that the lack of an effect is not due to differences in reliability. We now mention this point :

“This effect was not likely to be driven by a lower reliability in the measurement of this feature, as sociality was more reliable than all other behaviorally rated features in Experiment 1 (Figure 1 —figure supplement 1).”

We also now discuss the reliability of the behavioral ratings (see also below comment 20).

(12) Discussion: page 18: The authors argue for a hierarchy from visual to action-related to socially relevant computations. How might the one-back task (focusing on repetitions of the same action category) have contributed to this result?

We thank the Reviewer for this interesting point. Although the task was not related to most of the features, it required a degree of explicit, semantic processing that may have contributed to the neural dynamics. Although we cannot fully answer this question with the current data, we added the following paragraph to the discussion :

“Importantly, these features emerged spontaneously, as the one-back task performed during the EEG recordings only related to action category. However, the semantic processing required to perform the task may have contributed to these computations. The emergence of features irrelevant to the task at hand (action category is not correlated with any other features in our dataset) suggests that this temporal hierarchy would also emerge in the absence of a task; however, future work can more directly test the impact of implicit and explicit (e.g. social-affective) processing on these neural dynamics.”

(13): Table 1 (page 20): The mapping between the verb labels provided in Table 1 and the 8 exemplars used for each activity did not become quite clear to me. It seems from Table 1 that some of the categories consisted of a range of different actions (e.g. crying, cuddling, feeding, giggling, socializing for the activity 'childcare'), while some of them consisted of one single action only (e.g. reading), which makes the categories hard to compare, both in terms of the underlying perceptual variability and the task difficulty associated with judging a repetition of that category. Please comment.

The reviewer raises an important point. The reason for this partially stems from the imperfect mapping between our broader, ATUS-based categories and the verb labels in the Moments in Time (MiT) dataset. This is related to broader issues with category labels in computer vision datasets, which are often criticized for not being meaningful distinctions for humans (e.g., the dozens of different dog breeds in Imagenet). We sought to divide the MiT videos into more meaningful categories as defined by ATUS, but due to the somewhat arbitrary nature of the verb labels in MiT, there is not a perfect one-to-one mapping. Some of the categories mapped directly onto the labels, while other categories were broader and represented by several labels. Furthermore, many videos could have been represented by several other labels in addition to the ones they were listed under (hence why some ‘driving’ videos were found under ‘socializing’, for example).

Our original curation procedure involved going through the MiT verb labels and selecting videos from any label that seemed relevant for each ATUS activity category. This led to an initial selection of approximately 500 videos, from which the two datasets were selected solely based on the ATUS activity categories without regard to the MiT verb labels (as described in the Methods). Although the verb labels may seem heterogeneous, we attempted to enhance and match perceptual variability across categories by selecting videos for each ATUS category that varied along different criteria such as environment and number of agents. For example, the ‘reading’ category, though represented by a single label in the Moments in Time dataset, contains videos depicting one or two agents of different ages, genders and ethnicities reading books or magazines both indoors (in a living room, in bed, in a library, on the subway etc.) and outdoors (in a park, by a river, in a hammock etc.). We have added this to the manuscript :

“Videos were chosen that were horizontally-oriented (landscape), of reasonable image quality, that clearly represented the activity in question, that clearly depicted an agent performing the activity, and that varied in terms of number of agents, gender and ethnicity of agents, and scene setting. Although some categories were represented by fewer verb labels than others in the final set, our curation procedure aimed to balance important features within and across action categories.”

Finally, any remaining heterogeneity across action categories was addressed through our feature-driven analysis approach, which was intended to capture any perceptual components that may affect how participants arrange the videos.

(14) Table 1 (page 20): Unless I misunderstood, the action 'socializing' occurred within four of the 18 activities (in addition to being one of the 18 activities). What was the reason for this? Could this have impacted the results, showing a stronger contribution of socio-affective features in comparison to action-related and visual features? Likewise, could the results obtained in this study be affected by the curation of the two stimulus datasets (adjusted to include both social and non-social activities)?

Our aim was to sample actions that involved both single and multiple agents in a balanced way, as this has been rarely done in previous work using controlled videos of actions, usually performed by one agent. Thus, some of the actions were found under the label ‘socializing’ and involved multiple agents performing the action (whether interacting or not). This is also due to the way the Moments in Time labels were assigned, as described above. The presence of a socializing category, however, was driven by the prominence assigned to this category in the 2019 American Time Use Survey under the broader ‘leisure’ umbrella. We acknowledge that the distinctions from ATUS are not perfect, but provided the best objective measure to curate our dataset. We have added a brief description of these shortcomings .

“We note that the ATUS distinctions are based on performed rather than observed actions. While imperfect, they provide an ecologically-relevant and objective criteria with which to define our action space.”

To your main concern, we do not see the presence of both social and non-social actions as a source of bias. First, most action categories were represented by both single-agent and multiple-agent videos. Although some categories inherently involved multiple people (socializing, instructing), others tended to involve single agents and no social aspect (sleeping, housework). Second, all features (including sociality and number of agents) were rated independently of action category, and feature distributions suggest that the stimuli varied along many axes (Figure 1b). Finally, each feature’s contribution to behavior was assessed independently of category labels.

Ultimately, we believe that an action space that includes both social and non-social activities is more representative of human actions in the real world, since many actions are directed towards others. On the other hand, no stimulus set is definitive, and it would be great to replicate these results with different types of stimuli. The question of how action categories should be defined is still an open one that we did not attempt to address here – although we argue that our feature-based approach mitigates this somewhat.

We have clarified our reasoning in text :

“Our final dataset included 18 social and non-social activities that lend themselves to visual representation (Table 1), to ensure a diverse and balanced stimulus set representative of the human everyday action space.”

(15) Page 20: What was the reason to use these specific common activities? Was there a specific cutoff?

We have clarified our selection of ATUS categories in the manuscript . The selected activities were second-level ATUS activities with a cutoff of at least 0.14 hours/day, with few exceptions (additions and omissions) that are now explicitly described in text :

“Action categories were selected from the second-level activities identified in the ATUS. We used a minimum cutoff of 0.14 hours/day to select common actions (Table 1). To diversify our dataset, we added a ‘hiking’ category (to increase variability in scene setting) and a “fighting” category (for variability along affective dimensions). In addition, “driving” was selected as a more specific instance of the “travel” categories in ATUS, as it is the most common form of transportation in the US. Some adjustments were also made to the “relaxing and leisure” category, by selecting two specific activities that were easy to represent and distinguish visually, as well as above threshold (“reading” and “playing games”). In addition, our “reading” category included both “reading for personal interest” and “homework and research”, as this distinction is difficult to convey visually. We omitted three leisure categories that were difficult to represent in brief videos (“watching TV”, “relaxing and thinking”, and “computer use for leisure, excluding games”), as well as the ”consumer goods purchases” category.”

We have also added the number of hours per day spent on each activity according to ATUS to Table 1.

(16) Page 21, line 404f: What was the purpose of the control videos depicting natural and indoors scenes?

We added this explanation to the Methods. Please see response to point 3 above.

(17) Page 21, line 415f: Please explain what 'repetitive' videos mean in this context.

After running our randomization procedure, there were a few videos within the same action category that shared a high degree of visual similarity (e.g. involving a similar number of agents in similar postures). The issue of visually similar stimuli was not apparent in the larger Experiment 1 set. We clarified our wording in text:

“From the remaining videos, a second set of 76 videos was sampled and manually adjusted to remove videos without agents (in the interest of experimental time) and any videos that were too similar visually to other videos in the same category (e.g., involving a similar number of agents in similar postures).”

(18) Page 22, behavioural ratings: What were the exclusion criteria for the behavioural ratings and the multiple arrangement task?

We thank the reviewer for spotting this oversight. We added this information in text:

“Participants were excluded if they responded incorrectly to catch trials (approximately 10% of trials) requiring them to label the action shown in the prior video, or if they provided overly repetitive ratings (e.g. using only two unique values or fewer out of five possible ratings throughout the experiment).”

(19) Page 22: Participants were instructed to rate 'how social the events were'. In the view of this reviewer, this leaves a lot of room for interpretation (and also differs from the instructions given in previous studies). Please comment.

We agree that our instructions did not much constrain participants’ judgments of sociality. Given the diversity of our natural videos (which involved interactions as well as people simply acting alongside each other), we wanted to capture the full spectrum of what sociality means to people, rather than focusing on a yes/no detection of social interaction. Previous studies are also quite heterogeneous in their measurement of sociality, with some focusing on interactions (e.g. Tucciarelli et al., 2019) and others defining it as person-directedness (Tarhan and Konkle, 2020) or relevance of one agents’ actions to another (Wurm and Caramazza 2019, Wurm, Caramazza and Lingnau 2017). In our view, the validity of our instructions is supported by the fact that sociality ratings were not less reliable than other ratings and among the most reliable of our social ratings. On the other hand, the different operationalization means that this feature may have captured different information than in previous studies, and may explain our lack of correlation between sociality and EEG data. We now mention this in the discussion:

“Alternatively, our operationalization of this feature (which was broader than in some previous studies, e.g. Tucciarelli et al., 2019; Wurm et al., 2017) may have led to differences in the information captured.”

(20) Page 22: How consistent were the behavioural ratings across participants? Was variability across participants higher for some features than for others?

This information is summarized in Figure 3 —figure supplement 1 and its caption. We now also discuss this in text:

“Behaviorally rated features differed in reliability in Experiment 1 (F(4,819) = 22.35, P<0.001), with sociality being the most reliable and arousal the least reliable (Figure 3 —figure supplement 1). In Experiment 2, however, there was no difference in reliability (F(4,619) = 0.76, P=0.55). Differences in reliability were mitigated by our use of feature averages to generate feature RDMs.”

(21) Page 23 line 454f: How was the subset of videos presented in trial 1 selected?

We added some more information on this procedure in text:

“The experiments included a training trial in which participants arranged the same seven videos (in Experiment 1) or eight videos (in Experiment 2) before beginning the main task. In both experiments, these videos were hand selected to represent clear examples from four categories.”

(22) Page 28 line 541f: According to which criteria did the authors center the 0.5 seconds window around the action?

The procedure was based on visual inspection of the videos. In most cases, the first 0.5 seconds were selected; however, a different segment was selected in cases where this made the action clearer (e.g. a video that began with someone catching a ball to then throw it was cropped around the throwing action, which could then be shown in its entirety). This was especially important given the brief nature of the 0.5 s videos. We have clarified this in text:

“The three-second stimuli were trimmed to a duration of 0.5 seconds centered around the action as determined by visual inspection, to ensure that the shorter videos were easily understandable. This helped improve time-locking to the EEG signals and allowed for a condition-rich experimental design.”

(23) Page 28 line 557f: What do the authors mean by ‘shuffled pairs’?

We clarified what we meant:

“The video pairs were presented in different orders to minimize learning effects, so that for each video pair (V1, V2), half of the presentations were in the order V1-V2 and half of them were in the order V2-V1.”

(24) Page 29, 2nd paragraph: Please provide details on the projector.

We added this information:

“The stimuli were presented using an Epson PowerLite Home Cinema 3000 projector with a 60Hz refresh rate.”

Reviewer #2 (Recommendations for the authors):

If I have read the paper (and code) correctly, it seems that a single RDM was used to model all timepoints in the EEG responses, and that single RDM was correlated with the RDMs generated by the pairwise decoding analysis for each timepoint. For purposes of modelling behavioral data – in which the subjects are making a judgment based on the entire video clip – a single RDM per video clip seems reasonable. However, for purposes of modelling time-varying neural responses, it may be less than ideal to use a single RDM to summarize each video. The usefulness of a single temporal summary RDM will depend on how homogenous the features are across the video clip. Having looked over a few of the video clips in Moments in Time, the clips appear to be generally slow-moving – i.e., no large changes in background elements or actors across the 3 seconds. This seems like a reasonable justification for the use of a single temporal summary RDM; I would encourage the authors to clarify their rationale for using one RDM for the whole timecourse instead of framewise RDMs. Quantitative arguments would be useful.

We agree that a time-resolved approach to modelling the video features could offer more information. Here, however, the videos used in the EEG paradigm were very brief – only 0.5 seconds long. Furthermore, we made efforts to ensure we selected videos where actions were clear and continuous, without sudden camera or actor changes. The shorter 0.5s duration of EEG stimuli likely increased visual homogeneity throughout each clip.

Finally, this would only apply to the visual features (specifically, the layer activations from AlexNet), as the other features we used for our EEG analysis did not vary across the frames (environment, number of agents, and behavioral judgments based on the entire clips). Still, to ensure that we maximized the amount of information present in our RDMs, our procedure for generating AlexNet features entailed extracting these features from each frame and averaging them across the 12 frames that the 0.5 s clips consisted of. Importantly, we did not see significant changes in AlexNet features across these frames.

In Author response image 3, we plotted the average correlation across frames between Conv1 and FC8 features extracted from AlexNet. For Conv1, the correlation was on average r = 0.89, SD = 0.09 ; for FC8, it was r = 0.98, SD = 0.03. Although there is more variability in correlations for Conv1, they are still high enough to justify our decision to select the average layer activation across frames. Furthermore, this variability diminishes in the FC8 features, which are expected to capture more high-level information, thus supporting our idea that the videos are sufficiently consistent over time to justify our approach:

Author response image 3
Average correlation of CNN features across frames.

Each dot in the scatterplots is a video, with the distribution of the correlations shown above.

We added the following to the manuscript:

“We did not use frame-wise RDMs to model these visual features; however, our approach of averaging features across video frames was justified by the short duration of our videos and the high correlation of CNN features across frames (Conv1: Pearson’s ρ= 0.89±0.09; FC8: ρ= 0.98±0.03).”

[Editors’ note: further revisions were suggested prior to acceptance, as described below.]

We all agree the manuscript has been improved, the issues raised by the reviewers have been addressed well, and the manuscript will make a nice addition to the literature. But one of the reviewers highlighted a remaining concern that I wanted to draw your attention to. This concern could be addressed with additional analyses or additional discussion and motivation of the approach you used (I will leave it up to you which avenue you want to pursue). I’ll quote from the reviewer:

“Among the visual features the authors investigate is motion. To me, this is a visual feature that would seem to have a strong potential to explain the EEG data in particular, since motion contrast strongly drives many visual areas. I am not completely satisfied with the way the authors have quantified motion, and I have a medium concern that their motion model is a bit of a straw man. First, they refer to the model as a “motion energy” model, which is not technically correct. Motion energy (i.e. Adelson and Bergen, 1985; Nishimoto et al., 2011) reflects image contrast, whereas optic flow does not (at least to the same degree). For example, a gray square moving 2 degrees on a slightly less gray background will generate the same magnitudes of optic flow vectors as a starkly black square on a white background moving the same distance. Motion energy will be substantially greater for higher-contrast stimuli. As such, it's likely that motion energy would be a better model for visual responses (at least in early visual areas) than optic flow. They also choose to compute optic flow densely, with one value per pixel. Thus, if actors in the various videos are in slightly different locations, the optic flow metric could be quite different, and the RDM may characterize arguably similar stimuli as distinct. I think that a multi-scale pyramid of optic flow (or better still, motion energy) would have a better chance of capturing motion selectivity in the way that the brain does.

We have removed our reference to the optic flow model as a motion energy model, and added a new motion energy model:

“To assess whether whether a motion energy model (Adelson and Bergen, 1985; Nishimoto et al., 2011; Watson and Ahumada, 1985) would better capture the impact of motion, we performed control analyses by computing motion energy features for each video using a pyramid of spatio-temporal Gabor filters with the pymoten package (Nunez-Elizalde et al., 2021).”

As the Reviewer predicted, the motion energy model correlated better with behavior than our optic flow model. This correlation was significant in Experiment 1, but not in Experiment 2, in line with our previous results regarding the role of visual features across the two experiments. In the variance partitioning analysis, the addition of the motion energy model to the visual feature group did not change the pattern of results in either experiment. We describe these findings in text:

“Similarly, our decision to quantify motion and image properties separately by using an optic flow model may have reduced the explanatory power of motion features in our data. Indeed, a motion energy model (Adelson and Bergen, 1985; Nunez-Elizalde et al., 2021) significantly correlated with behavior in Experiment 1, but not in Experiment 2. However, the addition of this model did not change the pattern of unique feature contributions (Figure 4 —figure supplement 2).”

There is one more choice that the authors make that I'm not entirely comfortable with, which relates to this same issue. They choose to use only the models that can be related to behavioral data to model the EEG data. This choice is motivated by the desire to relate brain activity to behavior, which is a worthwhile endeavor. However, I think there is an assumption underlying this choice: the authors assume that the signal that they measure with EEG must reflect neural representations that guide behavior and not low-level perceptual processes. I do not think that this is guaranteed to be the case. EEG measures a spatially diffuse signal, which may well be dominated by activity from quite low-level areas (e.g. V1). I think the question of whether the EEG signal reflects relatively high-level cognitive processing or relatively low-level perceptual processing – for any given experiment – is still open. For most of the models they exclude, I don't think this is a big issue. For example, I think it's reasonable to test luminance basically as a control (to show that there aren't huge differences in luminance that can explain behavior) and then to exclude it from the EEG modeling. However, I'm less happy with exclusion of motion, based on a somewhat sub-optimal motion model not explaining behavior.

The combination of these two issues has left me wanting a better quantification of motion as a model for the EEG data. My bet would be that motion would be a decent model for the EEG signal at an earlier stage than they see the effects of action class and social factors in the signal, so I don't necessarily think that modeling motion would be likely to eliminate the effects they do see; to my mind, it would just present a fairer picture of what's going on with the EEG signal."

In selecting only models that explained behavior for our EEG analysis, we did not intend to assume that EEG activity only reflects behaviorally relevant features. By adding Conv1 to the behaviorally relevant visual models (FC8 and environment), we hoped we would capture low-level visual processing while at the same time keeping the number of features tractable and linking brain and behavior (as noted by the Reviewer). However, we agree that this approach neglects the role of motion, and we have performed new analyses with the motion energy model.

Indeed, the motion energy model correlated with our EEG data during a sustained time window, with a peak during early time windows. The addition of motion energy to the group of visual features in the variance partitioning analysis increased the unique contribution of visual features and decreased the contribution of action features, suggesting that the ‘action’ portion of the signal we originally detected included some shared variance with motion. However, the temporal hierarchy was not changed in the fixed-effects analysis, which leads us to believe that the three feature groups explain distinct portions of the signal. Most importantly, the portion of variance uniquely explained by social-affective features was unchanged by the addition of the motion energy model.

We describe these results in text:

“Motion has been shown to drive the response of visual areas to naturalistic stimuli (Russ and Leopold, 2015). To better assess the effect of motion on EEG responses, we performed an additional analysis including the motion energy model. There was a sustained correlation between motion energy and EEG patterns beginning at 62 ms (Figure 6 —figure supplement 3). In the variance partitioning analysis, the addition of motion energy increased the unique contribution of visual features and decreased that of action features, indicating that the action features share variance with motion energy. However, the three stages of temporal processing were preserved in the fixed-effects analysis even with the addition of motion energy, suggesting that the three feature groups made distinct contributions to the neural patterns. Importantly, the unique contribution of social-affective features was unchanged in both analyses by the addition of the motion energy model.”

https://doi.org/10.7554/eLife.75027.sa2

Article and author information

Author details

  1. Diana C Dima

    Department of Cognitive Science, Johns Hopkins University, Baltimore, United States
    Contribution
    Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review and editing
    For correspondence
    ddima@jhu.edu
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-9612-5574
  2. Tyler M Tomita

    Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, United States
    Contribution
    Conceptualization, Data curation, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
  3. Christopher J Honey

    Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, United States
    Contribution
    Conceptualization, Methodology, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-0745-5089
  4. Leyla Isik

    Department of Cognitive Science, Johns Hopkins University, Baltimore, United States
    Contribution
    Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Writing – review and editing
    Competing interests
    No competing interests declared

Funding

National Science Foundation (CCF-1231216)

  • Leyla Isik

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. The authors wish to thank Tara Ghazi, Seah Chang, Alyssandra Valenzuela, Melody Lee, Cora Mentor Roy, Haemy Lee Masson, and Lucy Chang for their help with the EEG data collection, Dimitrios Pantazis for pairwise decoding code, and Emalie McMahon for comments on the manuscript.

Ethics

Human subjects: All procedures for data collection were approved by the Johns Hopkins University Institutional Review Board, with protocol numbers HIRB00009730 for the behavioral experiments and HIRB00009835 for the EEG experiment. Informed consent was obtained from all participants.

Senior and Reviewing Editor

  1. Chris I Baker, National Institute of Mental Health, National Institutes of Health, United States

Reviewers

  1. Angelika Lingnau, Institute of Psychology, University of Regensburg, Germany
  2. Mark Lescroart, University of Nevada at Reno, United States

Publication history

  1. Preprint posted: October 26, 2021 (view preprint)
  2. Received: October 26, 2021
  3. Accepted: May 24, 2022
  4. Accepted Manuscript published: May 24, 2022 (version 1)
  5. Version of Record published: June 1, 2022 (version 2)

Copyright

© 2022, Dima et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 411
    Page views
  • 147
    Downloads
  • 0
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Diana C Dima
  2. Tyler M Tomita
  3. Christopher J Honey
  4. Leyla Isik
(2022)
Social-affective features drive human representations of observed actions
eLife 11:e75027.
https://doi.org/10.7554/eLife.75027

Further reading

    1. Neuroscience
    Arefeh Sherafati et al.
    Research Article Updated

    Cochlear implants are neuroprosthetic devices that can restore hearing in people with severe to profound hearing loss by electrically stimulating the auditory nerve. Because of physical limitations on the precision of this stimulation, the acoustic information delivered by a cochlear implant does not convey the same level of acoustic detail as that conveyed by normal hearing. As a result, speech understanding in listeners with cochlear implants is typically poorer and more effortful than in listeners with normal hearing. The brain networks supporting speech understanding in listeners with cochlear implants are not well understood, partly due to difficulties obtaining functional neuroimaging data in this population. In the current study, we assessed the brain regions supporting spoken word understanding in adult listeners with right unilateral cochlear implants (n=20) and matched controls (n=18) using high-density diffuse optical tomography (HD-DOT), a quiet and non-invasive imaging modality with spatial resolution comparable to that of functional MRI. We found that while listening to spoken words in quiet, listeners with cochlear implants showed greater activity in the left prefrontal cortex than listeners with normal hearing, specifically in a region engaged in a separate spatial working memory task. These results suggest that listeners with cochlear implants require greater cognitive processing during speech understanding than listeners with normal hearing, supported by compensatory recruitment of the left prefrontal cortex.

    1. Neuroscience
    Mohammad Ali Salehinejad et al.
    Research Article Updated

    Sleep strongly affects synaptic strength, making it critical for cognition, especially learning and memory formation. Whether and how sleep deprivation modulates human brain physiology and cognition is not well understood. Here we examined how overnight sleep deprivation vs overnight sufficient sleep affects (a) cortical excitability, measured by transcranial magnetic stimulation, (b) inducibility of long-term potentiation (LTP)- and long-term depression (LTD)-like plasticity via transcranial direct current stimulation (tDCS), and (c) learning, memory, and attention. The results suggest that sleep deprivation upscales cortical excitability due to enhanced glutamate-related cortical facilitation and decreases and/or reverses GABAergic cortical inhibition. Furthermore, tDCS-induced LTP-like plasticity (anodal) abolishes while the inhibitory LTD-like plasticity (cathodal) converts to excitatory LTP-like plasticity under sleep deprivation. This is associated with increased EEG theta oscillations due to sleep pressure. Finally, we show that learning and memory formation, behavioral counterparts of plasticity, and working memory and attention, which rely on cortical excitability, are impaired during sleep deprivation. Our data indicate that upscaled brain excitability and altered plasticity, due to sleep deprivation, are associated with impaired cognitive performance. Besides showing how brain physiology and cognition undergo changes (from neurophysiology to higher-order cognition) under sleep pressure, the findings have implications for variability and optimal application of noninvasive brain stimulation.