Conceptual approach. A. We hypothesized that the condition-dependent instantaneous subspace of PM MN activity shifts progressively throughout the time course of behavioral trials both during execution (orange) and during observation (green). Such shifting can be examined by calculating the principal angles between a selected instantaneous subspace and every other subspace in the time series, t. B. Segments clipped from the neural trajectories of two different movements (magenta, purple) in a high dimensional space, I, show varying distance between them when projected into a time series (t = i, ii, iii) of shifting, low-dimensional instantaneous subspaces (gray). This varying distance indicative of the progressive shifting of the instantaneous subspace can be followed by decoding the different movements from the trajectory segments projected into the time series of instantaneous subspaces. C. Neural trajectory segments from the four RGM movements (magenta, purple, cyan, and yellow) during execution and during observation originate in the same high-dimensional space (a), but project into distinct low-dimensional execution (orange, b1) and observation (green, b2) subspaces. Nevertheless, canonical correlation analysis (CCA) may identify another subspace (pale blue, c) where the projected magenta, purple, cyan, and yellow segments from both execution and observation show a similar spatial relationship to one another, with the two segments of each color projecting close to one another. Such correlation between the two sets of trajectory segments projected into the same subspace would indicate similar latent dynamic relationships among the four movements during execution and observation.

The reach-grasp-manipulate (RGM) task. A. In separate blocks of trials monkeys reached to, grasped, and manipulated four different objects themselves (Exe), and then observed a human performing the same task (Obs). B. The times of eight behavioral events from Start-of-trial to End-of-trial divided each trial into seven epochs from Initial hold to Reward. For analyses the data were aligned separately on, and trajectories were sampled for 100 msec following, the times of four selected events—Instruction onset (I), Go cue (G), Movement onset (M), and the beginning of the final Hold (H). C. Recording array locations in PMv (green) and PMd (orange) for each monkey have been redrawn from intraoperative photographs. PCD – precentral dimple; AS – arcuate sulcus; CS – central sulcus; r – rostral; m – medial. Scale bars, representing 4 mm, apply to all three monkeys.

Numbers of trials in each session.

For each of the three sessions from each of the three monkeys, numbers of trials involving each of the four objects (sphere, button, coaxial cylinder, perpendicular cylinder) are given in parentheses separately for execution and for observation.

Numbers of PM units in each session.

For each of the three sessions from each of the three monkeys (R, T, and F), numbers of PM units are given for each of four classes in the format of Total (PMv, PMd). MN – mirror neurons, modulated significantly during action execution and during action observation. AE – action execution neurons, modulated during execution but not during observation. AO – action observation neurons, modulated during observation but not execution. NS – not significant, units not modulated significantly during either execution or observation.

Neural trajectories of condition-independent versus condition-dependent activity. A. Neural trajectories of PM MN firing rates averaged across multiple execution trials involving each of the four objects (Sphere – purple, Button – cyan, Coaxial cylinder [Coax]– magenta, Perpendicular cylinder [Perp]– yellow) have been projected into the PC1 vs PC2 plane of the Total neural activity. Averaging these four trajectories gives their common, condition-independent (CI) trajectory (black). Time proceeds clockwise from left, with data separately aligned at four selected times: triangle – instruction onset (I); circle – go cue (G); square – movement onset (M); diamond – beginning of final hold (H). B. Condition-dependent trajectories obtained by subtracting the CI trajectory (black) from each of the four single-object trajectories (colors) in A, and then projected into the PC1 vs PC2 plane of their common, condition dependent (CD) subspace across the entire time course of trials. Data from monkey R, session 2.

Time course of the first principal angle between instantaneous subspaces. A – D: MN populations during execution trials; E – H: MN populations during observation trials; I – L: AE neuron populations during execution trials. Each frame shows the time course of the first principal angle between the time series of instantaneous subspaces and that present at one of four selected times— A, E, I: instruction onset; B, F, J: go cue; C, G, K: movement onset; or D, H, L: the beginning of the final hold. Results in 1 ms steps have been aligned separately at the times of the instruction onset (I), go cue (G), movement onset (M), and hold (H)—each indicated by a vertical line as labeled in the frame at upper left. Red, green, and blue traces represent sessions 1, 2, and 3, respectively, from monkey R. Solid traces represent means and shaded areas represent ±1 standard deviation across 10-fold cross validation as described in the Methods. Horizontal black lines indicate the average (solid) and the average minus 3 standard deviations (dashed) of the first principal angle between a fixed 3D space and other 3D spaces chosen randomly within a N-dimensional space (see Figure 4 – figure supplement 2 and related Methods). Here N = 37, the number of MNs in session 3. Horizontal purple bars in the left column (A, E, I) indicate 500 ms, which applies to the entire row.

First, second, and third principal angles as a function of time. An example in which the three principal angles, θ1, θ2, θ3, between the instantaneous subspace at time M (movement onset) and the entire time series of instantaneous subspaces have been plotted as a function of time for PM MNs. (Data from monkey R, session 1.) Note that all three principal angles go to 0°at time M when the current instantaneous subspace is, by definition, the subspace at time M.

First principal angles between a fixed 3D subspace and 5000 other 3D subspaces randomly chosen from spaces of dimensionality, N, varying from 5 to 500. Error bars indicate ± 1 standard deviation from the mean. Note that as the dimensionality of the parent space decreases, the random principal angle also decreases.

Time course of the first principal angle of instantaneous subspaces for AE neurons during observation trials. As would be expected given that AE neurons were not modulated significantly during observation trials, in the observation context AE populations had no gradual changes in principal angle, showing only relatively sharp troughs dipping toward 0° at each of the four selected times when the current instantaneous subspace, by definition, approached that at times I, G, M, or H. Formatting is the same as in Figure 4.

MN trajectory segments projected into instantaneous subspaces. A. Using execution data from an example session (monkey T, session 3), trajectory segments averaged across trials involving each of the four objects (sphere – purple, button – cyan, coaxial cylinder [coax] – magenta, perpendicular cylinder [perp] – yellow) were clipped for 100 ms immediately following each of four behavioral events (rows: Instruction onset, Go cue, Movement onset, Hold). Each set of these four segments then was projected into the PC1 vs PC2 plane of the instantaneous 3D subspace present at four different times (columns: I, G, M, H). B. The same process was performed using observation data from the same session. The PC1 vs PC2 scales at lower left in B apply to all frames in both A and B.

Cumulative separation. To summarize the changes in trajectory separation illustrated in Figure 5, we calculated the 3-dimensional cumulative separation (CS, the summed pointwise Euclidean distance between all pairwise combinations of the four object-specific trajectory segments, see Methods) for each set of four segments projected into each of the four instantaneous subspaces at times I, G, M, or H. CS values, which we use only to characterize the phenomenon of trajectory separation, are illustrated for execution from the example session of Figure 5 as a color matrix in A, and for observation in B. For both execution and observation, the highest CS values lie on the main diagonal, increasing in temporal order from Instruction to Go to Movement to Hold, with the exception that for execution, CS for Hold was less than for Movement. C and D show CS matrices averaged across all three sessions from all three monkeys for execution and observation, respectively, demonstrating that the features seen in the example session of Figure 5 were relatively consistent across sessions. Across all nine sessions two-way ANOVA showed significant main effects on CS values of both segment and subspace as well as a significant interaction effect during both execution and observation (p < 0.05). In both of these contexts, as the instantaneous subspace of the PM MN population shifted progressively over the time course of RGM trials, the separation of condition-dependent neural trajectories also increased.

Decodable information as a function of time. A – D: Classification accuracy for mirror-neuron execution trajectory segments projected into instantaneous execution subspaces; E – H: for mirror-neuron observation trajectory segments projected into their instantaneous observation subspaces; I-L: for action-execution neuron trajectory segment projected into their instantaneous execution subspaces. A, E, I: Instruction trajectory segments; B, F, J: Go segments; C, G, K: Movement segments; D, H, L: Hold segments. Red, green, and blue traces represent sessions 1, 2, and 3, respectively, from monkey R. Results in 50 ms steps have been aligned separately at the times of the instruction onset (I), go cue (G), movement onset (M), and hold (H)—each indicated by a vertical line as labeled in the frame at upper left. In each frame, the short horizontal orange flag at the top of the vertical lines indicates the 100 ms during which each set of trajectory segments was clipped; the horizontal purple bar at lower left represents 500 ms. Solid curves indicate mean classification accuracy across 10-fold cross validation as a function of time, with the shaded areas indicating 1 standard deviation. Horizontal black lines indicate the mean (solid) ± 3 standard deviations (dashed) classification accuracy obtained by projecting each set of trajectory segments into 500 randomly selected 3D spaces.

Decodable information as a function of time in PM AE neuron populations. Formatting is the same as in Figure 6. As might have been expected, AE populations showed little if any decodable information during observation.

Time course of the first principal angle cross-calculated between instantaneous execution and observation subspaces of PM MNs as a function of time. First principal angles between the instantaneous execution subspace at selected times I, G, M, or H and the entire time series of instantaneous observation subspaces are shown above (A-D); between the instantaneous observation subspace at selected times I, G, M, or H and the entire time series of instantaneous execution subspaces below (E-H). Formatting is the same as in Figure 4.

Classification accuracy of trajectory segments cross-projected between instantaneous execution and observation subspaces of PM MNs as a function of time. On top, Instruction, Go, Movement, and Hold execution trajectory segments (A, B, C, D, respectively) from individual trials have projected into the time series of instantaneous observation subspaces and classified with a separate LSTM decoder at each time point; below, Instruction, Go, Movement, and Hold observation trajectory segments (E, F, G, H, respectively) have been projected into the time series of instantaneous execution subspaces and classified. Neither of these cross-projections showed gradual progression to peaks of classification accuracy. Nor did the classification accuracy in either cross-projection exceed that expected from chance alone (horizontal dashed lines). These results confirm that little if any overlap between instantaneous, condition-dependent execution and observation subspaces was present in monkey R. Findings were similar in monkey F. Formatting is the same as in Figure 6.

Partial overlap of execution and observation subspaces in monkey T. A-D. The first principal angles between the instantaneous execution subspace at times I, G, or M (though not H) and the time series of instantaneous observation subspaces showed an abrupt drop beginning at the time of instruction onset (I) and continuing until the time of movement onset (M). This drop, which reflects partial overlap of the execution and observation subspaces, was marked during session 1 (red), but less so during sessions 2 and 3 (green and blue, respectively). A’-D’. Likewise, Instruction, Go, or Movement, execution trajectory segments projected into the time series of instantaneous observation subspaces showed a rise in decodable information, also indicative of some degree of overlap, beginning at the time of instruction onset (I). A-D are formatted as Figure 4; A’-D’ as in Figure 6. Overlap like that seen here in monkey T was not found in monkeys R or F.

Alignment of trajectory segments by canonical correlation. A. For an example session (monkey F, session 2), mirror neuron Hold trajectory segments from execution trials have been projected into their original instantaneous execution subspace at time H (left), and from observation trials into their original instantaneous observation subspace also at time H (right). B. The same execution (left) and observation (right) trajectory segments all have been projected into another, common subspace identified with canonical correlation. Colors indicate trajectory segments from trials involving the sphere – purple, coaxial cylinder (coax)– magenta, perpendicular cylinder (perp) – yellow, and button – cyan. C. The three correlation coefficients resulting from CCA (CC1, CC2, and CC3) have been averaged across comparisons from all sessions from the three monkeys. Thick bars representing the standard deviations of the three coefficients cross at their means, with a thin line dropped vertically from that point to the CC1 vs CC2 plane. CCA of MN trajectory segments from execution trials recorded in two different sessions from the same monkey (black, MN:1/2) is used as a point of reference with which to compare alignment of MN execution versus observation trials collected in the same session (red, MN:E/O), and MN versus AE neuron execution segments from the same session (blue, MN/AE). D. Correlation coefficients from within-group CCA alignment for MN execution segments (gray, MN:E/E), MN observation trajectory segments (orange, MN:O/O) and AE execution segments (light blue, AE:E/E). See text for further description.