Action recognition is central for navigating social environments, as it provides the basis for understanding others’ intentions, predicting future events, and social interaction. Many actions aim to induce a change in the world, often targeting inanimate objects (e.g. opening or closing a door) or persons (e.g. kissing or hitting someone). Recognizing such goal-directed actions is a computationally challenging task, as it requires not only the temporospatial processing of body movements, but also processing of how the body interacts with, and thereby induces an effect on, the object targeted by the action, for example a change in location, shape, or state. While a large body of work has investigated the neural processing of observed body movements as such (Grossman et al., 2000; Giese and Poggio, 2003; Puce and Perrett, 2003; Peuskens et al., 2005; Peelen et al., 2006), the neural mechanisms underlying the analysis of action effects, and how the representations of body movements and action effects differ from each other, remain unexplored.

The recognition of action effects builds on a complex analysis of spatial and temporal relations between entities. For example, recognizing a given action as ‘opening a door’ requires the analysis of how different objects or object parts (e.g. door and doorframe) spatially relate to each other and how these spatial relations change over time. The specific interplay of temporospatial relations is usually characteristic for an action type (e.g. opening, as opposed to closing), independent of the concrete target object (e.g. door or trash bin), and is referred to here as action effect structure (Figure 1A). In addition, action effects are often independent of specific body movements – for example, we can open a door by pushing or by pulling the handle, depending on which side of the door we are standing on. This suggests that body movements and the effects they induce might be at least partially processed independently from each other. In this study, we define action effects as induced by intentional agents, but the notion of action effect structures might be generalizable to physical changes as such (e.g. an object’s change of location or configuration, independently of whether the change is induced by an agent or not). Moreover, we argue that representations of action effect structures are distinct of conceptual action representations: The former capture the temporospatial structure of an object change (e.g. the separation of a closing object element), the latter capture the meaning of an action (e.g. bringing an object into an opened state to make something accessible) and can also be activated via language (e.g. by reading ‘she opens the box’). Previous research suggests that conceptual action knowledge is represented in left anterior lateral occipitotemporal cortex (LOTC) (Watson et al., 2013; Lingnau and Downing, 2015; Wurm and Caramazza, 2022) whereas structural representations of action effects have not been investigated yet.

Figure 1

Download asset Open asset

We argue that object- and movement-general representations of action effect structures are necessary for the recognition of goal-directed actions as they allow for inferring the induced effect (e.g. that something is opened) independently of specific, including novel, objects. Strikingly, humans recognize actions even in the absence of any object- and body-related information: In the animations of Heider and Simmel, 1944, the only available cues are abstract geometrical shapes and how these shapes move relative to each other and to scene elements. Yet, humans automatically and effortlessly attribute actions to these animations (e.g. opening, chasing, hiding, kissing), which argues against an inferential process and rather points toward an evolutionary optimized mechanism in the service of action recognition. Here, we test for the existence of a processing stage in the action recognition hierarchy that encode action effect representations independently from representations of body movements. We argue that both the recognition of body movements and the effects they induce rely critically on distinct but complementary subregions in parietal cortex, which is associated with visuospatial processing (Goodale and Milner, 1992; Kravitz et al., 2011), action recognition (Caspers et al., 2010), and mechanical reasoning about manipulable objects (Binkofski and Buxbaum, 2013; Leshinskaya et al., 2020) and physical events (Fischer et al., 2016; Fischer and Mahon, 2021). Specifically, we hypothesize that the neural analysis of action effects relies on anterior inferior parietal lobe (aIPL), whereas the analysis of body movements relies on superior parietal lobe (SPL). aIPL shows a representational profile that seems ideal for the processing of action effect structures at a high level of generality: Action representations in bilateral aIPL generalize across perceptually variable action exemplars, such as opening a bottle or a box (Wurm and Lingnau, 2015; Hafri et al., 2017; Vannuscorps et al., 2019), as well as structurally similar actions and object events, for example, a girl kicking a chair and a ball bouncing against a chair (Karakose-Akbiyik et al., 2023). Moreover, aIPL is critical for understanding how tools can be used to manipulate objects (Goldenberg and Spatt, 2009; Reynaud et al., 2016). More generally, aIPL belongs to a network important for physical inferences of how objects move and impact each other (Fischer et al., 2016).

Also the recognition of body movements builds on visuospatial and temporal processing, but their representation should be more specific for certain movement trajectories (e.g. pulling the arm toward the body, regardless of the movement’s intent to open or close a door). The visual processing of body movements has been shown to rely on posterior superior temporal sulcus (Grossman et al., 2000; Giese and Poggio, 2003; Puce and Perrett, 2003; Peuskens et al., 2005; Peelen et al., 2006). However, recent research found that also SPL, but less so aIPL, encodes observed body movements: SPL is more sensitive in discriminating actions (e.g. a girl kicking a chair) than structurally similar object events (e.g. a ball bouncing against a chair) (Karakose-Akbiyik et al., 2023). Moreover, point-light-displays (PLDs) of actions, which convey only motion-related action information but not the interactions between the body and other entities, can be decoded with higher accuracy in SPL compared to aIPL (Yargholi et al., 2023). Together, these findings support the hypothesis of distinct neural systems for the processing of observed body movements in SPL and the effect they induce in aIPL.

Using an fMRI-based cross-decoding approach (Figure 1B), we isolated the neural substrates for the recognition of action effects and body movements in parietal cortex. Specifically, we demonstrate that aIPL encodes abstract representations of action effect structures independently of motion and object identity, whereas SPL is more tuned to body movements irrespective of visible effects on objects. Moreover, cross-decoding between pantomimes and animations revealed that right aIPL represents action effects even in response to implied object interactions. These findings elucidate the neural basis of understanding the physics of actions, which is a key stage in the processing hierarchy of action recognition.

To isolate neural representations of action effect structures and body movements from observed actions, we used a cross-decoding approach: In four separate fMRI sessions, right-handed participants observed videos of actions (e.g. breaking a stick, squashing a plastic bottle) along with corresponding PLD stick figures, pantomimes, and abstract animations of agent–object interactions (Figure 2) while performing a simple catch-trial-detection task (see Methods for details).

Figure 2

Download asset Open asset

To identify neural representations of action effect structures, we trained a classifier to discriminate the neural activation patterns associated with the action videos, and then tested the classifier on its ability to discriminate the neural activation patterns associated with the animations (and vice versa). We thereby isolated the component that is shared between the naturalistic actions and the animations – the perceptually invariant action effect structure – irrespective of other action features, such as motion, object identity, and action-specific semantic information (e.g. the specific meaning of ‘breaking a stick’).

Likewise, to isolate representations of body movements independently of the effect they have on target objects, we trained a classifier on action videos and tested it on the PLD stick figures (and vice versa). We thereby isolated the component that is shared between the naturalistic actions and the PLD stick figures – the coarse body movement patterns – irrespective of action features related to the target object, such as the way they are grasped and manipulated, and the effect induced by the action.

Additionally, we used pantomimes of the actions, which are perceptually richer than the PLD stick figures and provide more fine-grained information about hand posture and movements. Thus, pantomimes allow inferring how an object is grasped and manipulated. Using cross-decoding between pantomimes and animations, we tested whether action effect representations are sensitive to implied hand–object interactions or require a visible object change.

Cross-decoding of action effect structures and body movements

We first tested whether aIPL is more sensitive in discriminating abstract representations of action effect structures, whereas SPL is more sensitive to body movements. Action-animation cross-decoding revealed significant decoding accuracies above chance in left aIPL but not left SPL, as well as in right aIPL and, to a lesser extent, in right SPL (Figure 3A). Action-PLD cross-decoding revealed the opposite pattern of results, that is, significant accuracies in SPL and, to a lesser extent, in aIPL. A repeated measures ANOVA with the factors region of interest (ROI; aIPL, SPL), TEST (action-animation, action-PLD), and HEMISPHERE (left, right) revealed a significant interaction between ROI and TEST (F(1,23) = 35.03, p = 4.9E−06), confirming the hypothesis that aIPL is more sensitive to effect structures of actions, whereas SPL is more sensitive to body movements. Post hoc t-tests revealed that, for action-animation cross-decoding in both left and right hemispheres, decoding accuracies were higher in aIPL than in SPL (left: t(23) = 1.81, p = 0.042, right: 4.01, p = 0.0003, one-tailed), whereas the opposite effects were found for action-PLD cross-decoding (left: t(23) = −4.17, p = 0.0002, right: −2.93, p = 0.0038, one-tailed). Moreover, we found ANOVA main effects of TEST (F(1,23) = 33.08, p = 7.4E−06), indicating stronger decoding for action-PLD versus action-animation cross-decoding, and of HEMISPHERE (F(1,23) = 12.75, p = 0.0016), indicating stronger decoding for right versus left ROIs. An interaction between TEST and HEMISPHERE indicated that action-animation cross-decoding was disproportionally stronger in the right versus left hemisphere (F(1,23) = 9.94, p = 0.0044).

Figure 3 with 1 supplement see all

Download asset Open asset

These findings were corroborated by the results of a searchlight analysis (Figure 3B): Within the parietal cortex, action-animation cross-decoding revealed a cluster peaking in right aIPL, whereas the parietal clusters for the action-PLD cross-decoding peaked in bilateral SPL. The whole-brain results further demonstrated overall stronger decoding for action-PLD throughout the action observation network, and in particular in LOTC extending into pSTS (see also Figure 3—figure supplement 1 for ROI results in LOTC and pSTS), which was expected because of the similarity of movement kinematics between the naturalistic actions and the PLDs. Note that we were not interested in the representation of movement kinematics in the action observation network as such, but in testing the specific hypothesis that SPL is disproportionally sensitive to movement kinematics as opposed to aIPL. Interestingly, the action-animation cross-decoding searchlight analysis revealed an additional prominent cluster in right LOTC (and to a lesser extent in left LOTC), suggesting that not only aIPL is critical for the representation of effect structures, but also right LOTC. We therefore include LOTC in the following analyses and discussion. In addition, we observed subtle but significant above chance decoding for action-animation in bilateral early visual cortex (EVC; see also Figure 3—figure supplement 1). This was surprising because the different stimulus types (action videos and animations) should not share low-level visual features. However, it is possible that there were coincidental similarities between action videos and animations that were picked up by the classifier. To assess this possibility, we tested whether the five actions can be cross-decoded using motion energy features extracted from the action videos and animations (Appendix 1—figure 1). This analysis revealed significant above chance decoding accuracy (30%), suggesting that actions and animations indeed contain coincidental visual similarities. To test whether these similarities can explain the effects observed in V1, we used the motion energy decoding matrix as a model for a representational similarity analysis (RSA; see Results section ‘Representational geometry of action-structure- and body-motion-related representations’).

Representation of implied versus visible action effects

Humans can recognize many goal-directed actions from mere body movements, as in pantomime. This demonstrates that the brain is capable of inferring the effect that an action has on objects based on the analysis of movement kinematics without the analysis of a visible interaction with an object. Inferring action effects from body movements is easier via pantomimes than with PLD stick figures, because the former provide richer and more fine-grained body information, and in the case of object manipulations, object information (e.g. shape) implied by the pantomimed grasp. Hence, neither pantomimes nor PLDs contain visible information about objects and action effects, but this information is more easily accessible in pantomimes than in PLDs. This difference between pantomimes and PLDs allows testing whether there are brain regions that represent effect structures in the absence of visual information about objects and action effects. We focused on brain regions that revealed the most robust decoding of action effect structures, that is, aIPL and LOTC (see Figure 4—figure supplement 1 for results in SPL). We first tested for sensitivity to implied effect structures by comparing the cross-decoding of actions and pantomimes (strongly implied hand-object interaction) with the cross-decoding of actions and PLDs (less implied hand-object interaction). This was the case in both aIPL and LOTC: action-pantomime cross-decoding revealed higher decoding accuracies than action-PLD cross-decoding (Figure 4A; all t(23) > 3.54, all p < 0.0009; one-tailed). The same pattern should be observed in the comparison of action-pantomime and pantomime-PLD cross-decoding, which was indeed the case (Figure 3A; all t(23) > 2.96, all p < 0.0035; one-tailed). These findings suggest that the representation of action effect structures in aIPL does not require a visible interaction with an object. However, the higher decoding across actions and pantomimes might also be explained by the higher visual and kinematic similarity between actions and pantomimes, in particular the shared information about hand posture and hand movements, which are not present in the PLDs. A more selective test is therefore the comparison of animation-pantomime and animation-PLD cross-decoding: as the animations do not provide any body-related information, a difference can only be explained by the stronger matching of effect structures between animations and pantomimes. We found higher cross-decoding for animation-pantomime versus animation-PLD in right aIPL and bilateral LOTC (all t(23) > 3.09, all p < 0.0025; one-tailed), but not in left aIPL (t(23) = 0.73, p = 0.23, one-tailed). However, a repeated measures ANOVA revealed no significant interaction between TEST (animation-pantomime, animation-PLD) and ROI (left aIPL, right aIPL; F(1,23) = 3.66, p = 0.068), precluding strong conclusions about differential sensitivity of right vs. left aIPL in representing implied action effects.

Figure 4 with 1 supplement see all

Download asset Open asset

Together, these results suggest that right aIPL and bilateral LOTC are sensitive to implied action effects. This finding was also obtained in a whole-brain conjunction analysis, which revealed effects in right aIPL and bilateral LOTC but not in other brain regions (Figure 4B).

Representational content of brain regions sensitive to action effect structures and body motion

To explore in more detail what types of information were isolated by the action-animation and action-PLD cross-decoding, we performed an RSA.

We first focus on the representations identified by the action-animation decoding. To characterize the representational content in the ROIs, we extracted the classification matrices of the action-animation decoding from the ROIs (Figure 5A) and compared them with different similarity models (Figure 5B) using multiple regression. Specifically, we aimed at testing at which level of granularity action effect structures are represented in aIPL and LOTC: Do these regions encode the broad type of action effects (change of shape, change of location, ingestion) or do they encode specific action effects (compression, division, etc.)? In addition, we aimed at testing whether the effects observed in EVC can be explained by a motion energy model that captures the similarities between actions and animations that we observed in the stimulus-based action-animation decoding using motion energy features. We therefore included V1 in the ROI analysis. We found clear evidence that the representational content in right aIPL and bilateral LOTC can be explained by the effect type model but not by the action-specific model (Figure 5C; all two-sided paired t-tests between models p < 0.005). In left V1, we found that the motion energy model could indeed explain some representational variance; however, in both left and right V1 we also found effects for the effect type model. We assume that there were additional visual similarities between the broad types of actions and animations that were not captured by the motion energy model (or other visual models; see Appendix 1—figure 1). A searchlight RSA revealed converging results, and additionally found effects for the effect type model in the ventral part of left aIPL and for the action-specific model in the left anterior temporal lobe, left dorsal central gyrus, and right EVC (Figure 5D). The latter findings were unexpected and should be interpreted with caution, as these regions (except right EVC) were not found in the action-animation cross-decoding and therefore should not be considered reliable (Ritchie et al., 2017). The motion energy model did not reveal effects that survived the correction for multiple comparison, but a more lenient uncorrected threshold of p = 0.005 revealed clusters in left EVC and bilateral posterior SPL.

Figure 5 with 1 supplement see all

Download asset Open asset

To characterize the representations identified by the action-PLD cross-decoding, we used a model of manuality that captures whether the actions are unimanual or bimanual, an action-specific model as used in the action-animation RSA above, and a kinematics model that was based on the three-dimensional (3D) kinematic marker positions of the PLDs (Figure 6B). Since pSTS is a key region for biological motion perception, we included this region in the ROI analysis. The manuality model explained the representational variance in the parietal ROIs, pSTS, and LOTC, but not in V1 (Figure 6C; all two-sided paired t-tests between V1 and other ROIs p < 0.002). By contrast, the action-specific model revealed significant effects in V1 and LOTC, but not in pSTS and parietal ROIs (but note that effects in V1 and pSTS did not differ significantly from each other; all other two-sided paired t-tests between mentioned ROIs were significant at p < 0.0005). The kinematics model explained the representational variance in all ROIs. A searchlight RSA revealed converging results, and additionally found effects for the manuality model in bilateral dorsal/medial prefrontal cortex and in right ventral prefrontal cortex and insula (Figure 6D).

Figure 6

Download asset Open asset

Representational similarity between brain regions

Finally, we investigated how similar the ROIs were with regard to the representational structure obtained by the action-animation and action-PLD cross-decoding. To this end, we correlated the classification matrices for both decoding schemes and all ROIs with each other (Figure 7A) and displayed the similarities between them using multidimensional scaling (Figure 7B) and dendrograms (Figure 7C).

Figure 7

Download asset Open asset

The aim for this analysis was twofold: First, we wanted to assess for the action-animation decoding how the representational structure in V1 relates to LOTC and aIPL. If V1 is representationally similar to LOTC and aIPL, this might point toward potential visual factors that drove the cross-decoding in visual cortex but potentially also in higher-level LOTC and aIPL. However, this was not the case: The V1 ROIs formed a separate cluster that was distinct from a cluster formed by aIPL and LOTC. This suggests that the V1 represents different information than aIPL and LOTC.

Second, we aimed at testing whether the effects in aIPL for the action-PLD decoding reflect the representation of action effect structures or rather representations related to body motion. In the former case, the representational organization in aIPL should be similar for the action-animation and action-PLD cross-decoding. In the latter case, the representational organization for action-PLD should be similar between aIPL and the other ROIs. We found that for the action-PLD decoding, all ROIs were clustered relatively closely together, and aIPL did not show similarity to the action-animation ROIs, specifically to aIPL. This finding argues against the interpretation that the effects in aIPL for the action-PLD cross-decoding were driven by action effect structures. Rather, it suggests that aIPL also encodes body motion, which is in line with the RSA results reported in the previous section.

We provide evidence for neural representations of action effect structures in aIPL and LOTC that generalize between perceptually highly distinct stimulus types – naturalistic actions and abstract animations. The representation of effect structures in aIPL is distinct from parietal representations of body movements, which were predominantly located in SPL. While body movement representations are generally bilateral, action effect structure representations are lateralized to the right aIPL and LOTC. In right aIPL and bilateral LOTC, action effect structure representations do not require a visible interaction with objects but also respond to action effects implied by pantomime.

Recognizing goal-directed actions requires a processing stage that captures the effect an action has on a target entity. Using cross-decoding between actions and animations, we found that aIPL and LOTC encode representations that are sensitive to the core action effect structure, that is, the type of change induced by the action. As the animations did not contain biological motion or specific object information matching the information in the action videos, these representations are independent of specific motion characteristics and object identity. This suggests an abstract level of representation of visuospatial and temporal relations between entities and their parts that may support the identification of object change independently of specific objects (e.g. dividing, compressing, ingesting, or moving something). Object-generality is an important feature as it enables the recognition of action effects on novel, unfamiliar objects. This type of representation fits the idea of a more general neural mechanism supporting mechanical reasoning about how entities interact with, and have effects on, each other (Fischer et al., 2016; Karakose-Akbiyik et al., 2023). Using multiple regression RSA, we showed that action effect structure representations in these regions capture the broad action effect type, that is, a change of shape/configuration, a change of location, and ingestion. However, since this analysis was based on only five actions, a more comprehensive investigation is needed to understand the organization of a broader range of action effect types (see also Worgotter et al., 2013).

In right aIPL and bilateral LOTC, the representation of action effect structures did not depend on a visible interaction with objects but could also be activated by pantomime, that is, an implied interaction with objects. This suggests that right aIPL and LOTC do not merely represent temporospatial relations of entities in a perceived scene. Rather, the effects in these regions might reflect a more inferential mechanism critical for understanding hypothetical effects of an interaction on a target object.

Interestingly, action effect structures appear lateralized to the right hemisphere. This is in line with the finding that perception of cause–effect relations, for example, estimating the effects of colliding balls, activates right aIPL (Fugelsang et al., 2005; Straube and Chatterjee, 2010). However, in the context of action recognition, the involvement of aIPL is usually bilateral or sometimes left-lateralized, in particular for actions involving an interaction with objects (Caspers et al., 2010). Also, mechanical reasoning about tools – the ability to infer the effects of tools on target objects based on the physical properties of tools and objects, such as shape and weight – is usually associated with left rather than right aIPL (Goldenberg and Spatt, 2009; Reynaud et al., 2016; Leshinskaya et al., 2020). Thus, left and right aIPL appear to be disproportionally sensitive to different structural aspects of actions and events: Left aIPL appears to be more sensitive to the type of interaction between entities, that is, how a body part or an object exerts a force onto a target object (e.g. how a hand makes contact with an object to push it), whereas right aIPL appears to be more sensitive to the effect induced by that interaction (the displacement of the object following the push). In our study, the animations contained interactions, but they did not show precisely how a force was exerted onto the target object that led to the specific effects: In all animations, the causer made contact with the target object in the same manner. Thus, the interaction could not drive the cross-decoding between actions and animations. Only the effects – the object changes – differed and could therefore be discriminated by the classification. Two questions arise from this interpretation: Would similar effects be observed in right aIPL (and LOTC) if the causer were removed, so that only the object change were shown in the animation? And would effects be observed in the left aIPL for distinguishable interactions (e.g. a triangle hitting a target object with the sharp or the flat side), perhaps even in the absence of the induced effect (dividing or compressing object, respectively)?

Action effect representations were found not only in aIPL but also LOTC. Interestingly, the RSA did not reveal substantially different representational content – both regions are equally sensitive to the effect type and their representational organization in response to the five action effects used in this experiment is highly similar. As it appears unlikely that aIPL and LOTC represent identical information, this raises the question of what different functions these regions provide in the context of action effect representation. Right LOTC is associated with the representation of socially relevant information, such as faces, body parts, and their movements (Chao et al., 1999; Pitcher and Ungerleider, 2021). Our findings suggest that right LOTC is not only sensitive to the perception of body-related information but also to body-independent information important for action recognition, such as object change. It remains to be investigated whether there is a dissociation between the action-independent representation of mere object change (e.g. in shape or location) and a higher-level representation of object change as an effect of an action. Left LOTC is sensitive to tools and effectors (Bracci and Peelen, 2013), which might point toward a role in representing putative causes of the observed object changes. Moreover, action representations in left LOTC are perceptually more invariant, as they can be activated by action verbs (Watson et al., 2013), generalize across vision and language (Wurm and Caramazza, 2019b), and more generally show signatures of conceptual representation (Lingnau and Downing, 2015; Wurm and Caramazza, 2022). Thus, left LOTC might have generalized across actions and animations at a conceptual, possibly propositional level (e.g. the meaning of dividing, compressing, etc.), rather than at a structural level. Notably, conceptual action representations are typically associated with left anterior LOTC, but not right LOTC and aIPL, which argues against the interpretation that the action-animation cross-decoding in right LOTC and aIPL captured conceptual action representations rather than structural representations of the temporospatial object change type. From a more general perspective, cross-decoding between different stimulus types and formats might be a promising approach to address the fundamental question of whether the format of certain representations is propositional (Pylyshyn, 2003) or depictive (Kosslyn et al., 2006; Martin, 2016).

In contrast to the perceptually general representation of action effect structures in aIPL, the representation of body movements is more specific in terms of visuospatial relations between involved elements, that is, body parts. These representations were predominantly found in bilateral SPL, rather than aIPL, as well as in LOTC. This is in line with previous studies demonstrating stronger decoding of PLD actions in SPL than in aIPL (Yargholi et al., 2023) and stronger decoding of human actions as opposed to object events in SPL (Karakose-Akbiyik et al., 2023). The RSA revealed that SPL, as well as adjacent regions in parietal cortex including aIPL and LOTC, are particularly sensitive to manuality of the action (uni- vs. bimanual) and the movement kinematics specific to an action. An interesting question for future research is whether movement representation in SPL is particularly tuned to biological motion or equally to similarly complex nonbiological movements. LOTC was not only sensitive to manuality and kinematics, but particularly in discriminating the five actions from each other, which suggests that LOTC is most sensitive to capture even subtle movement differences.

The action-PLD cross-decoding revealed widespread effects in LOTC and parietal cortex, including aIPL. What type of representation drove the decoding in aIPL? One possible interpretation is that aIPL encodes both body movements (isolated by the action-PLD cross-decoding) and action effect structures (isolated by the action-animation cross-decoding). Alternatively, aIPL selectively encodes action effect structures, which have been activated by the PLDs. A behavioral test showed that PLDs at least weakly allow for recognition of the specific actions (Supplementary file 2), which might have activated corresponding action effect structure representations. In addition, the finding that aIPL revealed effects for the cross-decoding between animations and PLDs further supports the interpretation that PLDs have activated, at least to some extent, action effect structure representations. On the other hand, if aIPL encodes only action effect structures, we would expect that the representational similarity patterns in aIPL are similar for the action-PLD and action-animation cross-decoding. However, this was not the case; rather, the representational similarity pattern in aIPL was more similar to SPL for the action-PLD decoding, which argues against substantially distinct representational content in aIPL versus SPL for the action-PLD decoding. In addition, the RSA revealed sensitivity to manuality and kinematics also in aIPL, which suggests that the action-PLD decoding in aIPL was at least partially driven by representations related to body movements. Taken together, these findings suggest that aIPL encodes not only action effect structures, but also representations related to body movements. Likewise, also SPL shows some sensitivity to action effect structures, as demonstrated by effects in SPL for the action-animation and pantomime-animation cross-decoding. Thus, our results suggest that aIPL and SPL are not selectively but disproportionally sensitive to action effects and body movements, respectively.

The action effect structure and body movement representations in aIPL and SPL identified here may not only play a role in the recognition of others' actions but also in the execution of goal-directed actions, which requires visuospatial processing of own body movements and of the changes in the world induced by them (Fischer and Mahon, 2021). This view is compatible with the proposal that the dorsal ‘where/how’ stream is subdivided into sub-streams for the visuomotor coordination of body movements in SPL and the manipulation of objects in aIPL (Rizzolatti and Matelli, 2003; Binkofski and Buxbaum, 2013).

In conclusion, our study dissociated important stages in the visual processing of actions: the representation of body movements and the effects they induce in the world. These stages draw on subregions in parietal cortex – SPL and aIPL – as well as LOTC. These results help to clarify the roles of these regions in action understanding and more generally in understanding the physics of dynamic events. The identification of action effect structure representations in aIPL and LOTC has implications for theories of action understanding: Current theories (see for review e.g. Zentgraf et al., 2011; Kemmerer, 2021; Lingnau and Downing, 2024) largely ignore the fact that the recognition of many goal-directed actions requires a physical analysis of the action-induced effect, that is, a state change of the action target. Moreover, premotor and inferior parietal cortex are usually associated with motor- or body-related processing during action observation. Our results, together with the finding that premotor and inferior parietal cortex are similarly sensitive to actions and inanimate object events (Karakose-Akbiyik et al., 2023), suggest that large parts of the ‘action observation network’ are less specific for body-related processing in action perception than usually thought. Rather, this network might provide a substrate for the physical analysis and predictive simulation of dynamic events in general (Schubotz, 2007; Fischer, 2024). In addition, our finding that the (body-independent) representation of action effects substantially draws on right LOTC contradicts strong formulations of a ‘social perception’ pathway in LOTC that is selectively tuned to the processing of moving faces and bodies (Pitcher and Ungerleider, 2021). The finding of action effect representation in right LOTC/pSTS might also offer a novel interpretation of a right pSTS subregion thought to specialized for social interaction recognition: Right pSTS shows increased activation for the observation of contingent action-reaction pairs (e.g. agent A points toward object; agent B picks up object) as compared to two independent actions (i.e. the action of agent A has no effect on the action of agent B) (Isik et al., 2017). Perhaps the activation reflects the representation of a social action effect – the change of an agent’s state induced by someone else’s action. Thus, the representation of action effects might not be limited to physical object changes but might also comprise social effects not induced by a physical interaction between entities. Finally, not all actions induce an observable change in the world. It remains to be tested whether the recognition of, for example, communication (e.g. speaking, gesturing) and perception actions (e.g. observing, smelling) similarly relies on structural action representations in aIPL and LOTC.

Stimuli, MRI data, and code are deposited at the Open Science Framework (https://osf.io/am346/).

1. 1. Wurm MF
   2. Erigüç DY

   (2024) Open Science Framework

   ID am346. Decoding the physics of observed actions in the human brain.

   https://osf.io/am346/

1. 1. Benjamini Y
   2. Yekutieli D

   (2001) The control of the false discovery rate in multiple testing under dependency

   The Annals of Statistics 29:1165–1188.

   https://doi.org/10.1214/aos/1013699998

   • Google Scholar
2. 1. Binkofski F
   2. Buxbaum LJ

   (2013) Two action systems in the human brain

   Brain and Language 127:222–229.

   https://doi.org/10.1016/j.bandl.2012.07.007

   • PubMed
   • Google Scholar
3. 1. Bracci S
   2. Peelen MV

   (2013) Body and object effectors: the organization of object representations in high-level visual cortex reflects body-object interactions

   The Journal of Neuroscience 33:18247–18258.

   https://doi.org/10.1523/JNEUROSCI.1322-13.2013

   • PubMed
   • Google Scholar
4. 1. Brainard DH

   (1997)

   The psychophysics toolbox

   Spatial Vision 10:433–436.

   • PubMed
   • Google Scholar
5. 1. Caspers S
   2. Zilles K
   3. Laird AR
   4. Eickhoff SB

   (2010) ALE meta-analysis of action observation and imitation in the human brain

   NeuroImage 50:1148–1167.

   https://doi.org/10.1016/j.neuroimage.2009.12.112

   • PubMed
   • Google Scholar
6. 1. Chao LL
   2. Haxby JV
   3. Martin A

   (1999) Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects

   Nature Neuroscience 2:913–919.

   https://doi.org/10.1038/13217

   • PubMed
   • Google Scholar
7. 1. Coutanche MN
   2. Thompson-Schill SL

   (2013) Informational connectivity: identifying synchronized discriminability of multi-voxel patterns across the brain

   Frontiers in Human Neuroscience 7:15.

   https://doi.org/10.3389/fnhum.2013.00015

   • PubMed
   • Google Scholar
8. 1. Fischer J
   2. Mikhael JG
   3. Tenenbaum JB
   4. Kanwisher N

   (2016) Functional neuroanatomy of intuitive physical inference

   PNAS 113:E5072–E5081.

   https://doi.org/10.1073/pnas.1610344113

   • PubMed
   • Google Scholar
9. 1. Fischer J
   2. Mahon BZ

   (2021) What tool representation, intuitive physics, and action have in common: the brain’s first-person physics engine

   Cognitive Neuropsychology 38:455–467.

   https://doi.org/10.1080/02643294.2022.2106126

   • PubMed
   • Google Scholar
10. 1. Fischer J

    (2024) Physical reasoning is the missing link between action goals and kinematics: a comment on “An active inference model of hierarchical action understanding, learning, and imitation” by Proietti et al

    Physics of Life Reviews 48:198–200.

    https://doi.org/10.1016/j.plrev.2023.08.017

    • PubMed
    • Google Scholar
11. 1. Fugelsang JA
    2. Roser ME
    3. Corballis PM
    4. Gazzaniga MS
    5. Dunbar KN

    (2005) Brain mechanisms underlying perceptual causality

    Brain Research. Cognitive Brain Research 24:41–47.

    https://doi.org/10.1016/j.cogbrainres.2004.12.001

    • PubMed
    • Google Scholar
12. 1. Giese MA
    2. Poggio T

    (2003) Neural mechanisms for the recognition of biological movements

    Nature Reviews. Neuroscience 4:179–192.

    https://doi.org/10.1038/nrn1057

    • PubMed
    • Google Scholar
13. 1. Goldenberg G
    2. Spatt J

    (2009) The neural basis of tool use

    Brain 132:1645–1655.

    https://doi.org/10.1093/brain/awp080

    • PubMed
    • Google Scholar
14. 1. Goodale MA
    2. Milner AD

    (1992) Separate visual pathways for perception and action

    Trends in Neurosciences 15:20–25.

    https://doi.org/10.1016/0166-2236(92)90344-8

    • PubMed
    • Google Scholar
15. 1. Grosbras MH
    2. Beaton S
    3. Eickhoff SB

    (2012) Brain regions involved in human movement perception: a quantitative voxel-based meta-analysis

    Human Brain Mapping 33:431–454.

    https://doi.org/10.1002/hbm.21222

    • PubMed
    • Google Scholar
16. 1. Grossman E
    2. Donnelly M
    3. Price R
    4. Pickens D
    5. Morgan V
    6. Neighbor G
    7. Blake R

    (2000) Brain areas involved in perception of biological motion

    Journal of Cognitive Neuroscience 12:711–720.

    https://doi.org/10.1162/089892900562417

    • PubMed
    • Google Scholar
17. 1. Hafri A
    2. Trueswell JC
    3. Epstein RA

    (2017) Neural representations of observed actions generalize across static and dynamic visual input

    The Journal of Neuroscience 37:3056–3071.

    https://doi.org/10.1523/JNEUROSCI.2496-16.2017

    • PubMed
    • Google Scholar
18. 1. Heider F
    2. Simmel M

    (1944) An experimental study of apparent behavior

    The American Journal of Psychology 57:243.

    https://doi.org/10.2307/1416950

    • Google Scholar
19. 1. Isik L
    2. Koldewyn K
    3. Beeler D
    4. Kanwisher N

    (2017) Perceiving social interactions in the posterior superior temporal sulcus

    PNAS 114:E9145–E9152.

    https://doi.org/10.1073/pnas.1714471114

    • PubMed
    • Google Scholar
20. 1. Karakose-Akbiyik S
    2. Caramazza A
    3. Wurm MF

    (2023) A shared neural code for the physics of actions and object events

    Nature Communications 14:3316.

    https://doi.org/10.1038/s41467-023-39062-8

    • PubMed
    • Google Scholar
21. 1. Kemmerer D

    (2021) What modulates the mirror neuron system during action observation?: multiple factors involving the action, the actor, the observer, the relationship between actor and observer, and the context

    Progress in Neurobiology 205:102128.

    https://doi.org/10.1016/j.pneurobio.2021.102128

    • PubMed
    • Google Scholar
22. Book

    1. Kosslyn SM
    2. Thompson WL
    3. Ganis G

    (2006) The Case for Mental Imagery

    Oxford University Press.

    https://doi.org/10.1093/acprof:oso/9780195179088.001.0001

    • Google Scholar
23. 1. Kravitz DJ
    2. Saleem KS
    3. Baker CI
    4. Mishkin M

    (2011) A new neural framework for visuospatial processing

    Nature Reviews. Neuroscience 12:217–230.

    https://doi.org/10.1038/nrn3008

    • PubMed
    • Google Scholar
24. Book

    1. Leshinskaya A
    2. Wurm MF
    3. Caramazza A

    (2020) Concepts of actions and their objects

    In: Gazzaniga M, Mangun GR, Poepped D, editors. The Cognitive Neurosciences. MIT Press. pp. 757–765.

    https://doi.org/10.7551/mitpress/11442.001.0001

    • Google Scholar
25. 1. Lingnau A
    2. Downing PE

    (2015) The lateral occipitotemporal cortex in action

    Trends in Cognitive Sciences 19:268–277.

    https://doi.org/10.1016/j.tics.2015.03.006

    • PubMed
    • Google Scholar
26. Book

    1. Lingnau A
    2. Downing P

    (2024) Action Understanding

    Cambridge University Press.

    https://doi.org/10.1017/9781009386630

    • Google Scholar
27. 1. Martin A

    (2016) GRAPES—Grounding representations in action, perception, and emotion systems: How object properties and categories are represented in the human brain

    Psychonomic Bulletin & Review 23:979–990.

    https://doi.org/10.3758/s13423-015-0842-3

    • Google Scholar
28. 1. Nishimoto S
    2. Vu AT
    3. Naselaris T
    4. Benjamini Y
    5. Yu B
    6. Gallant JL

    (2011) Reconstructing visual experiences from brain activity evoked by natural movies

    Current Biology 21:1641–1646.

    https://doi.org/10.1016/j.cub.2011.08.031

    • PubMed
    • Google Scholar
29. Software

    1. Nishimoto S
    2. Lescroart M

    (2018) Motion_energy_matlab, version 7848623

    GitHub.

    https://github.com/gallantlab/motion_energy_matlab
30. 1. Oosterhof NN
    2. Connolly AC
    3. Haxby JV

    (2016) CoSMoMVPA: multi-modal multivariate pattern analysis of neuroimaging data in matlab/gnu octave

    Frontiers in Neuroinformatics 10:27.

    https://doi.org/10.3389/fninf.2016.00027

    • PubMed
    • Google Scholar
31. 1. Peelen MV
    2. Wiggett AJ
    3. Downing PE

    (2006) Patterns of fMRI activity dissociate overlapping functional brain areas that respond to biological motion

    Neuron 49:815–822.

    https://doi.org/10.1016/j.neuron.2006.02.004

    • PubMed
    • Google Scholar
32. 1. Peuskens H
    2. Vanrie J
    3. Verfaillie K
    4. Orban GA

    (2005) Specificity of regions processing biological motion

    The European Journal of Neuroscience 21:2864–2875.

    https://doi.org/10.1111/j.1460-9568.2005.04106.x

    • PubMed
    • Google Scholar
33. 1. Pitcher D
    2. Ungerleider LG

    (2021) Evidence for a third visual pathway specialized for social perception

    Trends in Cognitive Sciences 25:100–110.

    https://doi.org/10.1016/j.tics.2020.11.006

    • PubMed
    • Google Scholar
34. 1. Puce A
    2. Perrett D

    (2003) Electrophysiology and brain imaging of biological motion

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 358:435–445.

    https://doi.org/10.1098/rstb.2002.1221

    • PubMed
    • Google Scholar
35. 1. Pylyshyn Z

    (2003) Return of the mental image: are there really pictures in the brain?

    Trends in Cognitive Sciences 7:113–118.

    https://doi.org/10.1016/S1364-6613(03)00003-2

    • Google Scholar
36. 1. Reynaud E
    2. Lesourd M
    3. Navarro J
    4. Osiurak F

    (2016) On the neurocognitive origins of human tool use : a critical review of neuroimaging data

    Neuroscience and Biobehavioral Reviews 64:421–437.

    https://doi.org/10.1016/j.neubiorev.2016.03.009

    • PubMed
    • Google Scholar
37. 1. Ritchie JB
    2. Bracci S
    3. Op de Beeck H

    (2017) Avoiding illusory effects in representational similarity analysis: what (not) to do with the diagonal

    NeuroImage 148:197–200.

    https://doi.org/10.1016/j.neuroimage.2016.12.079

    • PubMed
    • Google Scholar
38. 1. Rizzolatti G
    2. Matelli M

    (2003) Two different streams form the dorsal visual system: anatomy and functions

    Experimental Brain Research 153:146–157.

    https://doi.org/10.1007/s00221-003-1588-0

    • PubMed
    • Google Scholar
39. 1. Schubotz RI

    (2007) Prediction of external events with our motor system: towards a new framework

    Trends in Cognitive Sciences 11:211–218.

    https://doi.org/10.1016/j.tics.2007.02.006

    • PubMed
    • Google Scholar
40. 1. Straube B
    2. Chatterjee A

    (2010) Space and time in perceptual causality

    Frontiers in Human Neuroscience 4:28.

    https://doi.org/10.3389/fnhum.2010.00028

    • PubMed
    • Google Scholar
41. 1. van Boxtel JJA
    2. Lu H

    (2013) A biological motion toolbox for reading, displaying, and manipulating motion capture data in research settings

    Journal of Vision 13:7.

    https://doi.org/10.1167/13.12.7

    • PubMed
    • Google Scholar
42. Preprint

    1. van den Hurk J
    2. Op de Beeck HP

    (2019) Generalization asymmetry in multivariate cross86 classification: when representation A generalizes better to representation B than B to A

    bioRxiv.

    https://doi.org/10.1101/592410

    • Google Scholar
43. 1. Vannuscorps G
    2. F Wurm M
    3. Striem-Amit E
    4. Caramazza A

    (2019) Large-scale organization of the hand action observation network in individuals born without hands

    Cerebral Cortex 29:3434–3444.

    https://doi.org/10.1093/cercor/bhy212

    • PubMed
    • Google Scholar
44. 1. Walther A
    2. Nili H
    3. Ejaz N
    4. Alink A
    5. Kriegeskorte N
    6. Diedrichsen J

    (2016) Reliability of dissimilarity measures for multi-voxel pattern analysis

    NeuroImage 137:188–200.

    https://doi.org/10.1016/j.neuroimage.2015.12.012

    • PubMed
    • Google Scholar
45. 1. Watson CE
    2. Cardillo ER
    3. Ianni GR
    4. Chatterjee A

    (2013) Action concepts in the brain: an activation likelihood estimation meta-analysis

    Journal of Cognitive Neuroscience 25:1191–1205.

    https://doi.org/10.1162/jocn_a_00401

    • PubMed
    • Google Scholar
46. 1. Worgotter F
    2. Aksoy EE
    3. Kruger N
    4. Piater J
    5. Ude A
    6. Tamosiunaite M

    (2013) A simple ontology of manipulation actions based on hand-object relations

    IEEE Transactions on Autonomous Mental Development 5:117–134.

    https://doi.org/10.1109/TAMD.2012.2232291

    • Google Scholar
47. 1. Wurm MF
    2. Lingnau A

    (2015) Decoding actions at different levels of abstraction

    The Journal of Neuroscience 35:7727–7735.

    https://doi.org/10.1523/JNEUROSCI.0188-15.2015

    • PubMed
    • Google Scholar
48. 1. Wurm MF
    2. Caramazza A

    (2019a) Distinct roles of temporal and frontoparietal cortex in representing actions across vision and language

    Nature Communications 10:289.

    https://doi.org/10.1038/s41467-018-08084-y

    • PubMed
    • Google Scholar
49. 1. Wurm MF
    2. Caramazza A

    (2019b) Lateral occipitotemporal cortex encodes perceptual components of social actions rather than abstract representations of sociality

    NeuroImage 202:116153.

    https://doi.org/10.1016/j.neuroimage.2019.116153

    • PubMed
    • Google Scholar
50. 1. Wurm MF
    2. Caramazza A

    (2022) Two “what” pathways for action and object recognition

    Trends in Cognitive Sciences 26:103–116.

    https://doi.org/10.1016/j.tics.2021.10.003

    • PubMed
    • Google Scholar
51. 1. Yargholi E
    2. Hossein-Zadeh GA
    3. Vaziri-Pashkam M

    (2023) Two distinct networks containing position-tolerant representations of actions in the human brain

    Cerebral Cortex 33:1462–1475.

    https://doi.org/10.1093/cercor/bhac149

    • PubMed
    • Google Scholar
52. 1. Zentgraf K
    2. Munzert J
    3. Bischoff M
    4. Newman-Norlund RD

    (2011) Simulation during observation of human actions--theories, empirical studies, applications

    Vision Research 51:827–835.

    https://doi.org/10.1016/j.visres.2011.01.007

    • PubMed
    • Google Scholar

Author details

1. Moritz F Wurm

   CIMeC – Center for Mind/Brain Sciences, University of Trento, Rovereto, Italy

   Contribution

   Conceptualization, Resources, Data curation, Software, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing

   For correspondence

   moritz.wurm@unitn.it

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4358-9815

2. Doruk Yiğit Erigüç

   1. CIMeC – Center for Mind/Brain Sciences, University of Trento, Rovereto, Italy
   2. Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany

   Contribution

   Data curation, Formal analysis, Investigation, Visualization, Methodology, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0009-0003-8073-8135

• 491

  views

• 4

  downloads

• 0

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.