Figures and data
(A) Simplified schematic illustration of the action effect structure of “opening”. Action effect structures encode the specific interplay of temporospatial object relations that are characteristic for an action type independently of the concrete object (e.g. a state change from closed to open). (B) Cross-decoding approach to isolate representations of action effect structures and body movements. Action effect structure representations were isolated by training a classifier to discriminate neural activation patterns associated with actions and testing the classifier on its ability to discriminate activation patterns associated with corresponding abstract action animations. Body movement representations were isolated by testing the classifier trained with actions on activation patterns of corresponding PLD stick figures.
Experimental design. In 4 fMRI sessions, participants observed 2-second-long videos of 5 actions and corresponding animations, PLD stick figures, and pantomimes. For each stimulus type, 8 perceptually variable exemplars were used (e.g. different geometric shapes, persons, viewing angles, and left-right flipped versions of the videos). A fixed order of sessions from abstract animations to naturalistic actions was used to minimize memory and imagery effects.
Cross-decoding of action effect structures (action-animation) and body movements (action-PLD). (A) ROI analysis in left and right aIPL and SPL (Brodmann Areas 40 and 7, respectively; see Methods for details). Decoding of action effect structures (action-animation cross-decoding) is stronger in aIPL than in SPL, whereas decoding of body movements (action-PLD cross-decoding) is stronger in SPL than in aIPL. Asterisks indicate FDR-corrected significant decoding accuracies above chance (* p<0.05, ** p<0.01, *** p<0.001, **** p<0.0001). Error bars indicate SEM. (B) Mean accuracy whole-brain maps thresholded using Monte Carlo correction for multiple comparisons (voxel threshold p=0.001, corrected cluster threshold p=0.05). Action-animation cross-decoding is stronger in the right hemisphere and reveals additional representations of action effect structures in LOTC.
Cross-decoding of implied action effect structures. (A) ROI analysis. Cross-decoding schemes involving pantomimes but not PLDs (action-pantomime, animation-pantomime) reveal stronger effects in right aIPL than cross-decoding schemes involving PLDs (action-PLD, pantomime-PLD, animation-PLD), suggesting that action effect structure representations in right aIPL respond to implied object manipulations in pantomime irrespective of visuospatial processing of observable object state changes. Same conventions as in Fig. 3. (B) Conjunction of the contrasts action-pantomime vs. action-PLD, action-pantomime vs. pantomime-PLD, and animation-pantomime vs. animation-PLD. Uncorrected t-map thresholded at p=0.01; yellow outlines indicate clusters surviving Monte-Carlo-correction for multiple comparisons (voxel threshold p=0.001, corrected cluster threshold p=0.05).
Representational similarity of action effect structures and body movements. Pairwise classifications of the 5 actions from the action-animation cross-decoding (A) and the action-PLD cross-decoding (C) were extracted from each ROI, averaged across voxels, and entered into a cluster analysis using average distance. The resulting hierarchical cluster trees are displayed as dendrograms (B, D). In aIPL and LOTC, action effect structure representations formed meaningful clusters reflecting the 3 broad categories of change types: object shape/configuration changes (break, squash), location changes (hit, place), and ingestion (drink), supporting the interpretation that the cross-decoding between actions and animations isolated the coarse type of action effect. The cluster analysis in SPL and LOTC for body movements revealed similar representational clusters, which probably reflect categories of body movements, that is, bimanual actions (break, squash), unimanual actions (hit, place), and drinking as a mouth-directed action.
(A) Univariate activation maps for each session (all 5 actions vs. Baseline; FDR-corrected at p = 0.05) and (B) within-session decoding maps (Monte-Carlo-corrected for multiple comparisons; voxel threshold p=0.001, corrected cluster threshold p=0.05).
Direction-specific cross-decoding effects. To test whether there were differences between the two directions in the cross-decoding analyses, we ran, for each of the 6 across-session decoding schemes, two-tailed paired samples t-tests between the decoding maps of one direction (e.g. action ⟶ animation) vs. the other direction (animation ⟶ action). Direction effects were observed in left early visual cortex for the directions action ⟶ animation, PLD ⟶ animation, and pantomime ⟶ PLD, as well in right middle temporal gyrus and dorsal premotor cortex for action ⟶ PLD. These effects do not appear to affect the interpretation of direction-averaged cross-decoding effects in the main text. Monte-Carlo-corrected for multiple comparisons; voxel threshold p=0.001, corrected cluster threshold p=0.05.
