(A) Simplified schematic illustration of the action effect structure of “opening”. Action effect structures encode the specific interplay of temporospatial object relations that are characteristic for an action type independently of the concrete object (e.g. a state change from closed to open). (B) Cross-decoding approach to isolate representations of action effect structures and body movements. Action effect structure representations were isolated by training a classifier to discriminate neural activation patterns associated with actions (e.g. “breaking a stick”) and testing the classifier on its ability to discriminate activation patterns associated with corresponding abstract action animations (e.g. “dividing”). Body movement representations were isolated by testing the classifier trained with actions on activation patterns of corresponding PLD stick figures.

Experimental design.

In 4 fMRI sessions, participants observed 2-second-long videos of 5 actions and corresponding animations, PLO stick figures, and pantomimes. For each stimulus type, 8 perceptually variable exemplars were used (e.g. different geometric shapes, persons, viewing angles, and left-right flipped versions of the videos). A fixed order of sessions from abstract animations to naturalistic actions was used to minimize memory and imagery effects.

Cross-decoding of action effect structures (action-animation) and body movements (action-PLD).

(A) ROI analysis in left and right aIPL and SPL (Brodmann Areas 40 and 7, respectively; see Methods for details). Decoding of action effect structures (action-animation cross-decoding) is stronger in aIPL than in SPL, whereas decoding of body movements (action-PLD cross-decoding) is stronger in SPL than in aIPL. Asterisks indicate FDR-corrected significant decoding accuracies above chance (* p<0.05, ** p<0.01, *** p<0.001, **** p<0.0001). Error bars indicate SEM. (B) Mean accuracy whole-brain maps thresholded using Monte Carlo correction for multiple comparisons (voxel threshold p=0.001, corrected cluster threshold p=0.05). Action-animation cross-decoding is stronger in the right hemisphere and reveals additional representations of action effect structures in LOTC.

Cross-decoding of implied action effect structures.

(A) ROI analysis. Cross-decoding schemes involving pantomimes but not PLDs (action-pantomime, animation-pantomime) reveal stronger effects in right aIPL than cross-decoding schemes involving PLDs (action-PLD, pantomime-PLD, animation-PLD), suggesting that action effect structure representations in right aIPL respond to implied object manipulations in pantomime irrespective of visuospatial processing of observable object state changes. Same conventions as in Fig. 3. (B) Conjunction of the contrasts action-pantomime vs. action-PLD, action-pantomime vs. pantomime-PLD, and animation-pantomime vs. animation-PLD. Uncorrected t-map thresholded at p=0.01; yellow outlines indicate clusters surviving Monte-Carlo-correction for multiple comparisons (voxel threshold p=0.001, corrected cluster threshold p=0.05).

RSA for the action-animation representations.

(A) Classification matrices of ROIs. (B) Similarity models used in the RSA. (C) Multiple regression RSA ROI analysis. Asterisks indicate FDR-corrected significant decoding accuracies above chance (* p<0.05, ** p<0.01, *** p<0.001, **** p<0.0001). Error bars indicate SEM. (D) Multiple regression RSA searchlight analysis. T-maps are thresholded using Monte Carlo correction for multiple comparisons (voxel threshold p=0.001, corrected cluster threshold p=0.05) except for the motion energy model (p = 0.005, uncorrected).

RSA for the action-PLD representations.

(A) Classification matrices of ROIs. (B) Similarity models used in the RSA. (C) Multiple regression RSA ROI analysis. (D) Multiple regression RSA searchlight analysis. Same figure conventions as in Fig. 5.

ROI similarity for action-animation and action-PLD representations.

(A) Correlation matrix for the action-animation and action-PLD decoding and all ROIs. (B) Multidimensional scaling. (C) Dendrogram plot.

Cross-decoding of action effect structures (action-animation) and body movements (action-PLD) in left and right LOTC, pSTS, and V1 (respectively; see Methods for details on ROI definition).

Dark tones show effects in left ROIs, light tones show effects in right ROIs. Asterisks indicate FDR-corrected significant decoding accuracies above chance (* p<0.05, ** p<0.01, *** p<0.001, **** p<0.0001). Error bars indicate SEM.

Stimulus-based cross-decoding.

To investigate to which extent the action-animation and action-PLD decoding can be explained by visual features, we carried out a stimulus-based cross-decoding analysis. Specifically, the aim was two-fold: First, we aimed at testing what could have caused the action-animation cross-decoding in early visual cortex. Second, we aimed at testing whether the action-PLD cross-decoding in SPL (and other regions thought to be sensitive to body motion) could be explained by lower-level motion features.

Cross-decoding of implied action effect structures in SPL.

Implied action effect structures should reveal higher decoding accuracies in cross-decoding schemes involving pantomimes but not PLDs (action-pantomime, animation-pantomime) as compared to cross-decoding schemes involving PLDs (action-PLD, pantomime-PLD, animation-PLD). In both left and right SPL, there were no differences for the comparisons of Action-Pant with Action-PLD and Pant-PLD, whereas there was stronger decoding for Anim-Pant vs. Anim-PLD.

(A) Univariate activation maps for each session (all 5 actions vs. Baseline; FDR-corrected at p = 0.05) and (B) within-session decoding maps (Monte-Carlo-corrected for multiple comparisons; voxel threshold p=0.001, corrected cluster threshold p=0.05).

Direction-specific cross-decoding effects.

To test whether there were differences between the two directions in the cross-decoding analyses, we ran, for each of the 6 across-session decoding schemes, two-tailed paired samples t-tests between the decoding maps of one direction (e.g. action animation) vs. the other direction (animation action). Direction effects were observed in left early visual cortex for the directions action animation, PLD animation, and pantomime PLD, as well in right middle temporal gyrus and dorsal premotor cortex for action PLD. These effects might be due to noise differences between stimulus types (van den Hurk and Op de Beeck, 2019) and do not affect the interpretation of direction-averaged cross-decoding effects in the main text. Monte-Carlo-corrected for multiple comparisons; voxel threshold p=0.001, corrected cluster threshold p=0.05.

(A) Cross-decoding maps for Action-Pantomime, Pantomime-PLD, Animation-Pantomime, and Animation-PLD (Monte-Carlo-corrected for multiple comparisons; voxel threshold p=0.001, corrected cluster threshold p=0.05). (B) Classification matrices extracted from the cross-decoding maps.

ROIs used in the study.

Spherical ROIs were in volume space (12 mm radius); here we projected them on the cortical surface for a better comparison with the whole brain maps in the main article.

Results of behavioral pilot experiment for abstract animations.

Verbal descriptions of each participant and mean confidence ratings (from 1 = not at all to 10 = very much) ± standard deviations).

Results of behavioral pilot experiment for PLD stick figures.

Same conventions as in S1.

Results of behavioral pilot experiment for pantomimes.

Same conventions as in S1.