Recording locations and stimulus selectivity

(A). Targeted patches (red circles) shown in the native space of M1 (left) and M2 (right) on 2 mm coronal slices. Brown vertical lines illustrate penetrations. Monkey patches were defined by the contrast monkeys > monkey control objects (M-OM; p < 0.05 FEW). (B). Normalized net responses in the Selectivity test (Experiment 1) in aSTS (left) and mSTS (right). Rows represent neurons from M1 and M2 (separated by a dotted line). Columns correspond to the stimuli, color-coded per category with a bar and one example image. Below, bars in corresponding colors show the mean normalized net response to each stimulus (95% confidence intervals (1000 resamplings)). n indicates the number of neurons.

Monkey avatar stimuli for the Head-body orientation test (Experiment 1).

(A). 64 combinations of head and body orientations for the sitting (P1, left) and the standing pose (P2, right). Body orientation varies by row and head orientation varies by column. Different colors group the 8 avatar orientations of four head-body angles. (B). Example stimuli of P1 and P2 in the monkey-centered (MC, left) and the head-centered (HC, right) conditions, with zero angle between head and body. (C). Headless bodies (B) and heads (H) for the zero and straight angle head-body configurations, showing the frontal, back, and two lateral views for P1 and P2. Images are shown for MC (left) and HC (right) conditions and were positioned to match their corresponding avatars in (A). The location of the red fixation target is indicated in (B) and (C).

Neuronal responses in the Head-body orientation test (Experiment 1).

(A). Distribution of MSI computed for the responses for all orientation, pose (P1 and P2), and centering (MC (blue) and HC (red)) combinations in aSTS (upper panel) and mSTS (lower panel). Data for M1 and M2, and medians (triangles) are shown in different hues. (B). Significant main effects and interactions of the factors body and head orientation, for P1 and P2 in MC (blue) and HC (red) conditions in corresponding colors) in aSTS (left) and mSTS (right). Pooled data of both subjects. Significant interactions in both MC and HC are highlighted with darker blue and red (number of cases (n) in black). (C). Distribution of the correlation coefficients between responses to the head-body configurations in MC and HC conditions in aSTS (left) and mSTS (right). Data from both subjects and poses were pooled. Correlation coefficients for all cases are shown above the x-axis. Below the x-axis, the light green bars represent the correlation coefficients for cases with a significant interaction (either MC or HC), while the dark green bars indicate the coefficients for cases with a significant interaction in both MC and HC. N and medians (triangles) are shown in corresponding colors. (D). Normalized net responses of example neurons to the 64 combinations of head and body orientations in aSTS (left) and mSTS (right) for MC and HC conditions. Body orientation (ORT) varies by row and head orientation varies by column, from 0° to 315°. (E). Net responses of example neurons to bodies, heads, and the corresponding monkey configurations with MSI of 1, −1, or the median. Data for P1 and P2, MC and HC, and aSTS and mSTS (smoothed with the Matlab function interp1).

Decoding of head-body angles from neuronal responses in the Head-body orientation test (Experiment 1).

(A). Head-body configurations with zero and straight angles (black arrows in (B)) and leftward and rightward right angles of head and body (grey arrows in (B)), of P1 and P2, at eight avatar orientations. (B). Mean decoding accuracies for the head-body configurations with zero, straight, leftward, and rightward right angles versus other head-body angles as indicated, in aSTS (top two panels) and mSTS (bottom two panels), for poses P1 and P2. N indicates the number of neurons. Results are for MC training and HC testing. Responses to the 8 orientations of a configuration (rows in (A)) were pooled when training the classifier. Error bars show standard deviations across resamplings. The shaded area indicates null distribution (stimulus labels permutation). (C). Decoding accuracies for different head-body angle pairs in aSTS, for P1 (top) and P2 (bottom). Decoded pairs are connected with lines, with decoding accuracies indicated by color. Training was performed after pooling the 8 orientations of a head-body configuration defined by the angle of the head and body. Results are from MC training and HC testing. Avatars of one orientation of the head-body configuration angles are shown.

Decoding of head-body angles across orientations (Experiment 1).

(A). Head-body configurations with zero and straight angles, and leftward and rightward right angles for P1 and P2, split into four orientations for training and testing. (B). Decoding accuracies for the head-body configurations with zero, straight, leftward, and rightward right angles in aSTS, for P1 (top) and P2 (bottom); n indicates the number of neurons. Results are for MC training and HC testing, using four orientations for training and testing. Panels one and three: classifier trained on ‘+’ orientations and tested on ‘x’ orientations; panels two and four: classifier trained on ‘x’ orientations and tested on ‘+’ orientations. Same conventions as Figure 4.

Decoding of head-body angles in the Sum vs. Configuration test (Experiment 2).

(A). Stimuli of Experiment 2: heads and bodies presented at four orientations (frontal, back, and the two lateral ones; B+H), along with all head-body configurations (M). Images were monkey-centered and the isolated heads and bodies were positioned to match their corresponding avatars. The red square indicates the fixation target. (B). Decoding accuracies for head-body configurations (M) and head-body sums (B+H) with zero, straight, leftward, and rightward right angles of head and body (trained pooling the four orientations (A)), for aSTS and mSTS, n indicates the number of units. Same conventions as Figure 4.

Effect of head-body configuration inversion (Experiment 3)

(A). Stimuli of the head-body configurations of P1 with zero and straight angles, and leftward and rightward right angles, at eight orientations, upright and inverted. (B). Decoding accuracies for the head-body configurations with zero, straight, leftward, and rightward right angles in aSTS, for poses P1 (top panels) and P2 (bottom panels), n indicates the number of units. Solid and dotted lines represent data for upright and inverted configurations, respectively. Same conventions as Figure 4. (C). Responses to the preferred orientation of upright or inverted head-body configurations for aSTS and mSTS and P1 and P2 (M1 (dark green); M2 (light green)), for the monkey-responsive units. Medians (diamonds) and the number of units per subject are shown in corresponding colors. P values of Wilcoxon signed-rank tests (data pooled across subjects) testing the difference in response between upright and inverted are shown (red: significant).