Many, but not all, mirror neurons (MNs) discharged dependent on the type of action being observed. This dependence of the discharge rate on the observed action often existed only at certain times of the action. Moreover, for some neurons, the relationship between the discharge rate and the type of action observed, that is, the code for observed actions, also depended on time. This is exemplified by the neuron shown in Figure 2A. Focusing on the preferred action as a simplified description of the action code, we see that the neuron preferred a lift around the time of touch of the object but a twist and a shift during the later hold epoch. A dependence of action code on time bin is also evident for the neuron in Figure 2B. In this neuron, the discharge rate during action observation was highest for lift during the manipulation epoch, but highest for twist during the early hold epoch. Compared to the action code during observation, the action code during execution was temporarily the same only in a few neurons. For example, the neuron in Figure 2A preferred the same action during execution and observation except for the time bin before the onset of the hold epoch. However, in most neurons, time bins in which the action code during observation differed from that during execution predominated (as for example in the neuron in Figure 2C or in the time bin before touch in the neuron in Figure 2B) or in which actions were distinguished only during observation but not during execution (as for example in the time bin after hold in the neuron in Figure 2B).

Figure 2

Download asset Open asset

To investigate the existence of an action code that is shared between execution and observation in a systematic manner we focused on the action period from release of the start-button to reward, which consists of 12 relative time bins. We first tested for each time bin and neuron whether actions were encoded at all when observed and when executed as indicated by the fact that the discharge rate depended on the type of action. The percentage of MNs encoding observed actions ranged from 0 to 45% depending on the time bin (green in Figure 3A). Starting with the second bin after release (RLS in Figure 3A), the proportion increased, reached its maximum of 45% in the first bin after reaching the target object position (HLD in Figure 3A), and decreased to about 20% shortly before the reward wasgiven. Of the total 2124 bins tested (12 bins * 177 MNs), 22% (n=458, Figure 3A and B) encoded observed actions.

Figure 3

Download asset Open asset

In 60% of the n=458 bins that encoded the actions on observation, the actions were also encoded on execution (n=274, Exe & Obs, yellow in Figure 3). This means, conversely, that 40% of the bins with action coding during observation did not exhibit any action coding during execution and must therefore be eliminated as carrier of a shared action code (Obs only, gray in Figure 3).

The bins in which observed actions were discriminated were distributed across all 177 MNs examined such that 72% of MNs (n=128, 53% of F5 neurons examined) encoded the observed actions in at least one bin. In both observation and execution, the actions were encoded in at least one bin by 49% of MNs (n=87, 36% of F5 neurons). Another 19% of MNs (n=33, 14% of F5 neurons) encoded the actions on observation in at least one bin and also on execution in at least one bin, but without containing a bin in which the actions were encoded during both observation and execution. Only 5% of MNs (n=8, 3% of F5 neurons), encoded the actions on observation in at least one bin but not on execution in any of the 12 bins of the action period. In summary, more than two thirds of the MNs encoded the observed actions in at least one bin and more than 90% of them encoded the actions on execution in at least one bin, although the bins did not have to be the same.

For the bins in which actions were encoded when observed and when executed (n=274, indicated yellow in Figure 3B), we tested if the action code was shared. To this end, we used three alternative approaches. The first approach (referred to as ‘same discharge method’, Dis in Figure 3) aimed at testing for a ‘perfect’ shared code. The shared code is ‘perfect’ if the discharge rates associated with each action are the same during execution and observation (for more details see Material and methods). Of the bins examined (n=274), about 10% (n=28) exhibited such a shared code (Figure 3B). In relation to all bins in which the observed actions were encoded (n=458), 6% had a shared code, but 94% did not. The proportion of bins with a shared code in relation to all bins examined (n=2124) was 1.3% (Figure 3C). To assess whether this proportion is large enough to infer physiological significance of these shared bins, we compared it with the proportion of bins with shared code that would be expected by chance if the coding of actions in the population of MNs at observation were independent of the coding at execution (for more details see Material and methods). Testing against this null hypothesis was first proposed by Csibra, 2005. It turned out that the proportion found does not differ significantly from the proportion predicted by this null hypothesis (Figure 3C). The qualitatively same result was obtained when considering the proportion of MNs that had a shared code in at least one bin. This proportion was 13% (n=23, 10% of F5 neurons), which is also in a range to be expected by chance (Figure 3D).

The second approach (referred to as ‘same preference method’, Pref in Figure 3) uses a less stringent criterion for same action coding. Here we considered only whether the action that had the highest or lowest discharge rate when executed also had the highest or lowest discharge rate when observed and concluded on a shared code if this was the case (for more details see Materials and methods). This criterion leans on the idea that a neuron responds to only one action. Using this method, of the bins examined (n=274), 31% (n=84) had a shared code (Figure 3B). In relation to all bins in which the actions were encoded during observation (n=458), 18% had a shared code, but 82% did not. The proportion of bins with a shared code in relation to all bins examined (n=2124) was 4.0% (Figure 3C). This proportion is higher than to be expected by chance according to the aforementioned null hypothesis. The proportion of MNs with a shared code in at least one bin was 21% (n=38, 16% of the F5 neurons), which is also above the range to be expected by chance (Figure 3D).

The third approach relies on linear discriminant analysis (referred to as ‘cross-task classification method’, LDA in Figure 3). We tested whether the observed actions could be predicted significantly better than chance by a linear classifier trained on trials during execution (for further details see Materials and methods). In contrast to the same preference method, the discharge rate is used here without any information about whether an action is being executed or observed (since neither offset correction nor rescaling or ranking is used). In contrast to the same discharge method, the discharge rate can vary in the mean and in the distribution (across trials) between the actions. Of the bins examined (n=274), 23% (n=64) turned out to have a shared code (Figure 3C). In relation to all bins in which the actions were encoded during observation (n=458), 14% had a shared code, but 86% did not. The proportion of bins with a shared code in relation to all bins examined (n=2124) was 3.0% (Figure 3D). This proportion is higher than would be expected by chance according to the aforementioned null hypothesis. The proportion of MNs with a shared code in at least one bin was 18% (n=32, 13% of the F5 neurons), which is also above the range to be expected by chance (Figure 3E).

Hence, all three methods document that the overwhelming majority (about 80–95%) of bins in which action-related information is encoded when observed do not have a shared code. However, both the ‘same preference’ method and the ‘cross-task classification’ method also show that the proportion of bins that exhibit a shared code is higher than would be expected if the encoding of actions in the population of MNs during observation were independent of the encoding during execution. This suggests that at least certain aspects associated with the observed action are coded in some MNs in the same way as when the monkey performs the action himself.

In the cross-task classification method, the criterion used to decide whether a shared code exists is that observed actions can be predicted significantly better than chance by a classifier trained on trials during execution. However, this criterion is too weak to test the representation hypothesis. In fact, to accept the existence of a shared code where in F5 MNs observed actions are encoded according to their motor representation, it is required that the classifier trained during execution is the best one to read out observed actions. In other words, the accuracy of a classifier trained on observation should not be better than the accuracy of the classifier trained on execution. To compare the two accuracies for all bins (n=2124), we plotted the accuracies of predicting the observed actions by classifiers trained on execution (execution-to-observation, e2o classifier) against that classifiers trained on observation (observation-to-observation, o2o classifier, Figure 4C). This comparison revealed a small but clearly distinguishable cluster of bins in which the accuracy was nearly the same for the two types of classifiers (4B-E). Almost all the previously mentioned bins that had a significant accuracy of observed actions with a e2o classifier (n=64) fell into this group (Figure 4F). The demonstration of this definable cluster of bins supports the representation hypothesis for F5 MNs that contain such time bins.

Figure 4

Download asset Open asset

Following the idea that single neurons can be ascribed the property of representing observed and executed action aspects in the same way, we examined whether there are ‘pure mirror neurons’, defined as neurons that encode observed actions exclusively with the code used during execution. To this end, we considered the distribution of bins with a shared code and bins in which observed actions were encoded with a non-shared code or not encoded at all during execution (Figure 5A). The bins with a shared code (n=84 for the same preference method, n=64 for the cross-task classification method) were contributed by 38 and 32 neurons, respectively. Only three (Pref method) or two (LDA method) of these neurons exhibited exclusively shared bins, and the neurons differed depending on the method (IDs 44, 154, 96 for Pref, IDs 9, 84 for LDA). This means that almost all MNs that encode observed actions with a shared code do so only at certain times, but encode the actions differently at other times. A ‘pure mirror neuron’ is an exception. The property of encoding observed actions with a shared code can thus be described as a dynamic property of a subset of MNs (21% and 18% of MNs, 16% and 13% of F5 neurons, for same preference and cross-task classification method, respectively).

Figure 5

Download asset Open asset

If there are aspects of action that are coded in the same way, then the question arises as to what these are and whether they differ from those that are coded in a different way. To address this question, we compared the frequency distribution of action segments with shared code with the frequency distribution of action segments with non-shared code (Figure 5B–E). –We defined an action segment as the set of consecutive bins with the same type of code, shared or non-shared, adding to the non-shared code segments the bins in which the action was encoded only when observed but not when executed (‘obs only’ in Figure 5A). An action segment starts with a bin in which the observed action is encoded (shared or non-shared), and ends before a bin in which the observed action is encoded either by the other code type or not at all. With 12 time bins, a segment can have a duration between 1 and 12 bins and a starting position between 1 and 12. This results in 78 possible segments that can be coded (Figure 5B, bottom). For both the same preference and the cross-task classification method, the absolute frequencies showed that for almost all occurring action segments, there were more often those with non-shared code than with shared code (Figure 5B). This result is not unexpected in view of the overall predominance of bins with non-shared code described above. For both types of segments, shared and non-shared, it can be noted in Figure 5B that (1) segments of shorter duration predominated, (2) segments with a duration of only one bin after reaching the target position of the object were particularly frequent, (3) there were hardly any action segments with a duration of 7 or more bins. There were two MNs that appeared to encode the entire action, from a bin before reaching the object to the reward (IDs 64 and 19). However, inspection of these two neurons revealed that although the neurons distinguished the observed actions throughout the action, the action code changed. Notably, both in these two neurons and in the neurons with segments consisting of 7 bins, the encoding changed once the object had reached its target position.

The comparison of the relative frequencies of action segments showed only minor differences between the segments with shared and non-shared codes (Figure 5C). One of these differences was the dominance of shared segments with a duration of just one bin in the period starting with the hand having reached the object. Note that shared segments with only one bin should not be considered random, as they were more frequent than would be expected according to the aforementioned null hypothesis (Figure 5D). The onset of segments, regardless of their duration, differed between shared and non-shared mainly in that non-shared segments occurred earlier in the approach epoch than shared segments (Figure 5E, left). The relative frequency of segment duration did not differ between shared and non-shared segments (Figure 5E, right). In summary, this descriptive analysis showed that (1) not the entire observed action, but action segments are encoded, (2) reaching the target position plays a special role in this, (3) shared and non-shared coded segments hardly differ with respect to starting point and duration, with the exceptions that (4) non-shared segments occur earlier in the approach epoch, and (5) shared segments occur more often after the hand has reached the object.

We may safely assume that populations of cortical neurons, rather than single neurons, determine an animal’s behavior or perception. Hence, we asked whether the whole population of MNs or a subpopulation thereof codes observed actions with a shared code. To address this question, we first considered the whole population of all MNs studied (n=177). The mean accuracies of different classifiers based on the single neuron analysis described earlier are shown in Figure 6A. It is confirmed that (1) observed actions were discriminated later by execution-trained classifiers (red in Figure 6) than by observation-trained classifiers (green), and (2) observed actions were discriminated best by observation-trained classifiers after reaching the target position of the object. A result not yet described was that observed actions were discriminated as well as executed actions at this time bin, as evidenced by the fact that the mean accuracies of the e2e classifier (blue) and the o2o classifier did not differ. The mean accuracy of the classifiers, each based on the discharge rate of single neurons, was comparatively low with a maximum of 43% (chance 33%). This is explained by the fact that a large proportion of neurons did not discriminate actions at all in a given time bin (see Figure 3A) and that neurons that did discriminate actions did not discriminate them completely (see Figure 4A). If, in contrast, one trains classifiers not on the discharge rate of single neurons but on the discharge rate vector formed by the discharge rate of all 177 MNs, one bypasses the limitation of single neurons and suppresses the effect of those neurons that do not discriminate actions. Accordingly, for both types of within-task classifiers, e2e and o2o, higher accuracies were achieved with an optimal value of 100% in the bin where the target object position was reached (Figure 6B). As expected, the accuracies of the e2o (and also o2e) classifiers were also higher than the mean accuracy of these cross-task classifiers trained on single neurons. However, with maximum values of 50–60%, the accuracy was still lower than the accuracy of the within-task classifiers. If one applies the criterion for a shared code, as required by the representation hypothesis, that observed actions may not be better discriminated with classifiers trained by observation than with classifiers trained by execution, then actions in the whole population are not shared coded.

Figure 6

Download asset Open asset

This leaves open the question whether a subpopulation of MNs exists that has accuracies for e2o classifiers that are not worse than the accuracies of the o2o classifiers of this subpopulation and that are also not worse than the accuracies of the o2o classifiers of the whole population of F5 MNs. Neurons eligible for such a subpopulation are those classified as MNs with at least one bin with a shared code using the cross-task classification method at the single-cell level. However, these MNs often had non-shared codes in other bins that could counteract a shared population code in those bins. Accordingly, the population analysis of such a selected subpopulation (n=32, neurons with bins with a shared code in Figure 5A, right) showed a higher e2o accuracy than the total population (~65%), but in most bins a worse e2o accuracy compared to the o2o accuracy (Figure 6C). Alternatively, one might assume that only MNs that ‘clearly’ encode the observed actions constitute a subpopulation with shared code. An artificial threshold for o2o accuracy can be set, whose crossing is considered a clear encoding. We set a threshold of 10% above chance level for o2o accuracy (see the threshold plotted Figure 4A, bins were included without any statistical test on single-cell level). Depending on whether the e2o accuracy was above or below the average boundary angle (see Figure 4F), we classified a bin as shared or non-shared (referred to as ‘threshold method’). Restricting the consideration to neurons that contained only shared bins (n=17) avoided the inclusion of neurons that contained relevant non-shared bins. As a result, the population analysis showed, as expected, a good agreement of the e2o and o2o accuracies in the action period (Figure 6D), that is, a shared code. However, these accuracies were in the range of 50–60%, which corresponded to the e2o accuracy of the whole population. Thus, although observed actions in this subpopulation were not discriminated worse with execution-trained classifiers than with observation-trained classifiers, they were discriminated considerably worse than with observation-trained classifiers in the whole population.

The question notwithstanding how this could be achieved, one might assume that there are target structures, such as populations of downstream neurons, that read out action-related information only at specific times from specific MNs with shared codes. To estimate what accuracies could be achieved under this assumption we selected neurons bin-wise for both selection procedures described before. A neuron was selected for population analysis in a time bin if it had a shared code in that bin (6 E, F, top). We found e2o population accuracies of up to 80% (Figure 6E and F). The accuracies were higher for the second selection procedure, which could be due to the higher number of included neurons. For both procedures, the e2o accuracies were slightly below the o2o accuracies, and the o2o accuracies were below the o2o accuracies of the whole population (Figure 6B). This indicates that even in subpopulations of neurons selected in the best possible way, the observed actions are not coded as well with a shared code as with a non-shared code in the whole population.

Despite the low number of neurons with a shared action code, they might still serve a significant function. One indication of such a distinct functionality could be an anatomical separation between neurons consisting of only shared bins and those consisting of only non-shared bins. As shown in Figure 7, the anatomical locations of these two subpopulations were largely overlapped. Hence, we did not find convincing evidence that neurons with only shared bins are anatomically separated from neurons with only non-shared bins.

Figure 7

Download asset Open asset

Our finding is evidence against the representation hypothesis, which is based on a strictly defined mirror mechanism. That is to say, the activity of F5 MNs, as defined by Gallese et al., 1996, during action observation is not a supramodal motor representation of the observed action, as if the observer were performing it in place of the actor. At best, the representation hypothesis might hold for a subset of F5 MNs, yet only for restricted segments of an action (see below). Therefore, the prerequisite for the understanding hypothesis is not met for the whole population of F5 MNs. The objection could be raised that it is sufficient for the representation hypothesis to hold that observed actions could be decoded from the whole population with execution-trained classifiers with an accuracy that is above chance level (see Figure 6B). This finding indeed indicates that there is a similarity in the action-specific activity states of the whole population between execution and observation, a similarity which in principle could be exploited by a biologically plausible readout. However, since the observed actions were much better distinguished with observation-trained classifiers, the action-specific activity during observation must be more than a supramodal motor representation of the observed action. Therefore, the representation hypothesis for the whole population of F5 MNs must be rejected.

In contrast to the whole population of F5 MNs, about 20% of the MNs in our study that contained time bins with shared code met the requirement for a motor representation according to the strictly defined mirror mechanism in these time bins. If this information could be selectively read out from these MNs, taking time into account, these MNs could in principle be the substrate of sparse coding of observed actions according to the mirror mechanism (Földiak, 2002; Vinje and Gallant, 2000). That F5 MNs may comprise functionally different subsets of MNs with some serving shared coding at particular times may not be implausible given that they differ in their anatomic location and connections (Kraskov et al., 2009) as well as their cellular identity (Ferroni et al., 2021). Consistent with this, F5 MNs differ in how they modulate during the execution or observation of actions and whether or not certain factors influence them (Caggiano et al., 2011; Maranesi et al., 2013). Whether some MNs at some time points indeed function to provide motor information about the observed action according to the strictly defined mirror mechanism cannot be conclusively answered at present. However, the following three findings rather argue against it. First, simulating decoders that time-dependently read from the population of MNs with shared code did not achieve the performance of decoders that read from the whole population. Second, the comparison of the duration and starting position of the segments with shared and non-shared codes did not provide convincing evidence that segments with shared code represent specific action aspects. Third, there is as yet no evidence that the neurons of such subpopulations act together on a target. For example, we could not find convincing evidence for a specific anatomical location of the neurons of a subpopulation with only shared codes compared to those with only non-shared codes.

The fact is that the overwhelming majority of F5 MNs encoded observed actions in a non-shared manner at most times. And there are reasonable doubts whether the shared coding that occurs at times in some neurons really has a function according to the strictly defined mirror mechanism. Therefore, one may wonder what other purpose the observation-related responses of the F5 MNs might serve. In what follows, we distinguish between four alternative concepts and critically compare them with our results and previous findings.

The first alternative concept posits that F5 MNs’ activity is a sensory representation of aspects of the environment that could be relevant for the subject’s own action control, that is, for action selection, planning, preparation, and execution. This can be, for example, the appearance of an object or a change in the own finger configuration caused by the execution of the own action. But also the action of the other is a change in the environment. A sensory representation of expedient information is based on sensory input and/or predictions. The crucial difference we make here compared to a motor representation is that the information about the environment is exclusively of the kind that exteroceptors, proprioceptors and interoceptors can provide. Thus, it does not include information that we consider unique to action control, such as information about a goal, here defined as an intended state of the environment, or information about motor commands, here defined as which muscles to excite or inhibit, how much, and for how long. According to this concept, the predominance of non-shared codes in our study may be a consequence of the fact that for discrimination of actions only somatosensory input was available during execution, whereas only visual input was available during observation. This explanation assumes a multimodal sensory representation. However, if we consider a supramodal sensory representation, then non-shared codes can be explained by, from the monkey’s viewpoint, different target positions of the object during execution and observation. For example, if the actor shifts the object to the right from his point of view, the observer sitting opposite of him observes a shift of the object to the left. The occurrence of shared codes could be led back to similarities between the represented aspects of the actions during execution and observation such as the upward position of the target object relative to the table. Thus, our results can in principle be explained by the concept of sensory representation. However, evidence from previous studies suggests otherwise. In terms of action execution, neural responses of F5 neurons to a visual stimulus such as the presentation of an object have been shown to represent not the sensory properties of the stimulus itself, but the possible or upcoming action, for example, the way of grasping the object afforded by the object properties (Schaffelhofer and Scherberger, 2016). With respect to action observation, findings that many F5 MNs are influenced by the observer’s reachability to the observed action (Caggiano et al., 2009) and by the expected value of the goal for the observer (Pomper et al., 2020) show that the activity of many F5 MNs cannot be explained by this ‘sensory concept’.

The second alternative concept posits that observation-related activity of F5 MNs is a motor representation, yet not one of the observed action according to a mirror mechanism. Instead, the concept posits that the activity reflects possible actions by the observer in response to the observed action that might be performed under natural conditions (Pomper et al., 2015). Following Gibson, these action options can be referred to as ‘social affordances’ (Gibson, 1979; Orban et al., 2021), and according to the ‘affordance competition hypothesis’, multiple action options could be represented simultaneously, competing with each other for execution, influenced by factors like the expected reward associated with each action option (Cisek, 2007). This concept is supported by the findings that the activity of F5 MNs depends on whether the observer can reach the observed action (Caggiano et al., 2009) and how much reward the observer can expect after the action (Caggiano et al., 2012; Pomper et al., 2020). It is to be expected that possible responses to the observed action will hardly ever be pantomimes of the observed action, which might explain our finding that non-shared codes prevailed. The occurrence of shared codes, albeit rare, remains hard to explain. An even bigger challenge for this concept arises if we consider the fact that there was no way for the observing monkey in our experiment to execute any of the action options. Therefore, it seems ecologically inefficient to represent specific response options for each of the three observed actions, which, moreover, would have to be assumed to change due to the modulation of the neurons in the course of an observed action. Hence, this ‘affordance concept’ does not offer a good explanation either.

The third alternative concept posits that the activity of F5 MNs during observation is a motor representation of an ‘action’ by the observer, with the observer himself pursuing a goal. By ‘goal’ we refer to an intended state of the environment. A state is intended because its occurrence is associated with a value for the subject – the ‘goal value’. In our experiment, the state that has direct value for the observer is the receipt of water following the observation of the action of the other. Therefore, we consider the receipt of water to be the primary goal of the observer. For the human actor, we deem the primary goal to be hearing the click sound associated with the delivery of water to the observer, signaling the successful execution of the action, which has value for the human actor. This click sound, or the receipt of water, is the last event in a series of events that can be conceptualized as goal pursuit. The pursuit of a goal is a process that begins with the setting of a goal and, if successful, ends with the achievement of the goal. Goal pursuit consists of a sequence of intermediate goals, some of which usually require specific muscle activation patterns controlled by motor commands. In our experiment, we consider releasing the hand from the start button, touching the object, reaching the target position, and holding the object in the target position as intermediate goals. We call the last of these intermediate goals, which precedes the primary goal and which, in the case of one’s own action execution, determines possible motor command sequences that bring about the last intermediate goal, the action goal. Although the primary goal differs between the human actor and the observer, the action goal does not. The action goal of the observer, and of the human actor, is that the object is held in a certain position. For the human actor, motor commands are required to achieve this action goal. The observer, however, executes an ‘action’ in the sense of a goal pursuit without motor commands. Since the other one is bringing about the action goal, there is no need for the observer to act himself. However, it may be relevant to monitor the goal pursuit, that is, the successful achievement of the intermediate goals. Under natural condition, this enables the observer to recognize a deviation from the expected sequence of intermediate goals, and to intervene in order to ensure the achievement of the action goal. Besides, even such a goal pursuit without own action allows the observer to have an expectation of the action goal value and to update this action goal value as the action goal is reached. This may be important for the later selection of own action goals. From the observer’s point of view, the action goal of such an ‘action’ can be different from that of the actor depending on the subject’s position in relation to the object. When the action goal is described in a world-centered frame of reference, it is the same.

According to this goal-pursuit-by-observer concept, as with the sensory concept, the predominance of non-shared codes and the rare occurrence of shared codes in our study can be explained by differences and similarities in the target position of the object during execution and observation from the monkeys' point of view. Depending on this target position, there are differences and similarities between the intermediate goals including the action goal. Shared codes would also be explained if intermediate goals were represented in a viewpoint-invariant manner. A number of previous findings can be explained by this concept. First, viewpoint-invariant encoding of an observed action has been shown in a subset of F5 MNs (Caggiano et al., 2011). Second, since the temporal sequence of intermediate goals is the same during execution and observation, the similarity of state transitions between execution and observation found by Mazurek et al., 2018 can be explained if a state transition is interpreted as achievement of an intermediate goal and the activation of a subsequent intermediate goal. Notably, the finding of state transitions during action observation is similar to the distribution of the starting position of action segments in our study (see Figure 5): segments begin more often after the actor reaches the object and after the object is put to its target position, time points, which may correspond to state transitions. Third, that the monkey really pursues his own goal when observing is supported by the finding that the expected goal value determines how well he observes the action (Pomper et al., 2020). Fourth, the preparedness to act on one’s own to achieve the goal in the case of the other’s failure explains why some MNs are influenced by the reachability and spatial configuration of the observed action (Caggiano et al., 2011; Caggiano et al., 2009; Maranesi et al., 2017). Fifth, it is consistent with the representation of a goal pursuit during observation that information about the expected goal value for the observer is contained in F5 MNs (Pomper et al., 2020). Sixth, and most importantly, this information about the expected goal value explains the action goal selection deficit of monkeys with bilateral lesions of the subset of F5 neurons that project to the medial prefrontal cortex (Ninomiya et al., 2020). Monkeys with such lesions were less able to detect ‘choice errors’ of the observed actor. To detect a ‘choice error’, it is necessary to map the visual input of the observed action to an observer’s own action goal and its expected value. Only if there is an expectation about the action goal value of an observed action can an error in action goal selection by the actor be detected and an incorrect updating of the action goal value, which influences a subsequent own action goal selection, be avoided. Thus, the study of Ninomyia et al. clearly shows that F5 neurons are neccessary for this mapping and/or the updating of the action goal value based on this mapping.

The fourth alternative concept posits that the activity of F5 MNs during observation is a motor representation of the observed action, resorting to our broad definition of the mirror mechanism. The broad definition posits that, as with the sensory concept, the observed action is represented, but unlike the sensory concept, the representation contains information that we consider ‘motor’, that is, unique to action control. This information includes, but is not limited to, the action goal as an intended state, the action goal value, and the successful achievement of intermediate goals including the action goal. In this way, sensory input representing environmental changes is mapped onto a representation of a goal pursuit by the actor. This could allow the observer to perceive (understand) the process in the environment as goal pursuit and/or updating of own action goals or action goal values. In contrast to the goal-pursuit-by-observer concept, this goal-pursuit-by-actor concept assumes that the goal pursuit of the actor, but not that of the observer, is represented. We consider two possible distinctions from the strictly defined mirror mechanism. First, the observed action could be represented from the observer’s point of view. In that case, the action representation in the observer would be predominantly different from that in the actor. Second, the observed action could, after all, be represented as if the observer were performing it in place of the actor, but not supramodally, as postulated by the strictly defined mirror mechanism, yet multimodally. That is, information of different sensory modalities is combined without reaching an abstract supramodal level. This account seems to be supported by the finding that the activity of some F5 MNs during execution depends on whether the action is performed in darkness or in light (Maranesi et al., 2015). However, this account does not explain the result of the Ninomyia et al. study that the value of an own action goal is updated by observing an action of the other, where the action goal from the observer’s point of view is different from that from the actor’s point of view. Therefore, it is more likely that the observed action is represented from the observer’s point of view.

Both concepts of a goal pursuit, that by the actor and that by the observer, explain our results and most of the findings discussed so far to a similar extent. They also have in common to emphasize a representation of action control components upstream of motor commands, for which there is good evidence (Ferrari et al., 2005; Umiltà et al., 2008). Hence, we currently do not consider either of them to be the better concept. But a critical experiment can be suggested for which the two concepts make different predictions. The experiment is similar to that of the Ninomyia et al. study (2020) with the difference that when observed, only the actor, not the observer, is rewarded differently, but in a way that is predictable to the observer. If the goal-pursuit-by-observer concept holds, the prediction is that F5 MNs of the observer represent only the expectation of the own reward and that a ‘choice error’ of the actor is not detected by the observer. However, if the goal-pursuit-by-actor concept holds, the prediction is that F5 MNs of the observer represent the expectation of the reward for the actor in the same way as the expectation of the own reward and that a ‘choice error’ of the actor is detected.

Finally, the question must be addressed whether it is justified to hold on to the term ‘mirror neuron’. The term suggests that these neurons are a motor representation of the observed action resulting from a mirror mechanism. As mentioned above, the term ‘mirror mechanism’ can be understood in different ways. Depending on the interpretation, the assessment of the evidence for whether MNs ‘mirror’ or not will differ. Moreover, several of the concepts discussed here, that offer an explanation for the activity of MNs during observation, contradict even a broader understanding of a mirror mechanism. An alternative, as some already practice, would be to speak of self-other-action-related neurons, which is a descriptive term that allows for different conceptual interpretations. However, if the focus is to be on neurons related to a mirror mechanism, then the mirror mechanism should be included in the definition of these neurons. For example, the cross-task classification could be used to define a ‘mirror neuron’, provided that the mirror mechanism is understood according to our strict definition.

We conclude that our finding of a rare but more than random occurrence of time periods with shared action code does not rule out a motor representation of the observed action according to the strictly defined mirror mechanism in some F5 MNs at some time points. However, the predominance of non-shared codes with good population representation of observed actions renders alternative concepts more likely. Although it is certainly possible that several of the concepts discussed apply at different time points in different neurons, the current evidences suggest that many F5 MNs represent a goal pursuit from the observer’s point of view, in which either the observer aims at the same action goal as the actor, or in which only the actor aims at a goal. Such a goal pursuit may allow the observer to use an expectation about the action goal value to optimize the updating of action goal values through observation alone, which is beneficial for subsequent own action goal selection. Which of the two possible variants of goal pursuits might be represented could be decided by the suggested future experiment.

The experiments were performed on two adult male rhesus monkeys (Macaca mulatta, age 8 and 9 years, respectively, median weight 10.6 kg and 8.4 kg, respectively). The animals lived in socially compatible groups with regular access to an open-air enclosure. Both animals had been obtained from the DPZ (Deutsches Primatenzentrum, Göttingen, Germany). All experiments were approved and controlled by the regional veterinary administration (Regierungspräsidium Tübingen and Landratsamt Tübingen, Permit Number: N4/14) and conducted in accordance with German and European law and the National Institutes of Health’s Guide for the Care and Use of Laboratory Animals, and regularly and carefully monitored by the veterinary service of the University of Tübingen, the latter also providing care in case of medical problems.

Three objects to be manipulated by either the monkey or a human actor were fixed on a table and aligned parallel to the ground. The table rotated twice between trials (in the same or in opposite directions and for variable durations) to place randomly one of the three objects in front of the respective actor. The objects were metallic discs identical in shape and size (diameter 3.5 cm, height 0.5 cm). In the execution task, the center of the object at stake had a distance of 23 cm from the monkey’s eyes. In the observation task, in which a human standing opposite to the monkey acted on the relevant object, the distance to the monkey was 49 cm. Each object accommodated only one specific type of action: it could be either lifted (2.7 cm up), twisted (50° clockwise), or shifted (1.9 cm to the right). Note that the identical appearance of the three objects denied any clue on the type of action they accommodated.

In the execution task, the rotation of the table and the execution of the action on the target object by the monkey took place in darkness. In the observation task, a light was switched on when the object was rotated in place so that the monkey could observe the experimenter standing opposite to him (light was turned off with object release or if a trial was aborted).

A trial started when the object was rotated in place and the actor (monkey in the execution task, human in the observation task) had pressed the start button (start button could be pressed before, during or after the rotation). A variable time (1–1.5 s) after trial start, an LED next to the positioned object that had halted in front of the respective actor, was turned on for 0.1–0.3 s. The LED color indicated the manipulation to be performed: green or yellow for a lift, white or blue for a twist, red for a shift. To evaluate whether the LED color was, indeed, used as a cue for action selection in the execution task, we randomly interleaved trials with a different LED, non-informative for the type of action, as a control in 5% of the trials: this LED was in a slightly different position and had a yellowish color that differed from the yellow of the informative cue. In these trials, the action was feasible by trying different manipulations on the object and using the resulting somatosensory feedback. The actor was allowed to release immediately the start-button to approach, manipulate, and finally hold the object against a resistance (against gravity for lift, against a spring for twist and shift). After a variable time after object hold start (0.3–0.8 s), the monkey — no matter if he had performed the action or had observed the human action — received water (or occasionally also units of a banana flavor high caloric drink) as reward, followed by the actor releasing the object. The amount of fluid per trial varied between 0.15 and 0.5 ml across sessions but was the same across the six actions (3 for each task) within a session.

The task events ‘start-button press’, ‘start-button release’, ‘LED onset’, ‘touch of object’, ‘start moving the object’, ‘reaching the hold position’, ‘reward’, and ‘withdraw from the object’ by releasing it, were registered by mechanoelectrical sensors. A trial was aborted (and a beep was given as auditory feedback to the monkey) if the start-button was released within 100 ms after LED onset (to motivate the monkey to use the LED cue), if the object was released before reward delivery or if a timeout was reached 10 s after trial start. To motivate the monkey even more to use the LED cue in the execution task, another timeout was active in 30% of trials (session-dependent rarely up to 100%) for the time period between ‘touch of object’ to ‘start moving the object’: 0.15 s (rarely 0.1 s) for a twist and a shift, 0.35 s (rarely 0.3 s) for a lift. The selection of time constraints and the proportion of trials in which they were applied, was a pragmatic compromise between a time limit, at which the LED signal had to be used as an informative cue for action selection in order to comply with the task, and a time span that allowed the task to be completed even when overall motivation was low. In the control trials with a non-informative LED signal no timeout was active. In the observation task, a trial was also aborted (and a beep was given as feedback to the monkey) if the monkey did not attend to the action as indicated by gaze not staying inside a given fixation window (see below). For experiment control and the recording of behavioral data, we deployed in-house open-source software (nrec: https://nrec.neurologie.uni-tuebingen.de, developed and maintained by F. Bunjes, J. Gukelberger, V. Gueler) running under Debian Linux on a standard PC.

Eye position was recorded using an in-house video eye-tracker based on pupil detection in infrared light, operating at a sampling rate of 50 Hz, in one monkey and by permanently implanted search coils in the other monkey. Eye position recordings were calibrated at the beginning of each experimental session by having the monkey fixate a dot displayed on a monitor, seen by the monkey in the table plane after redirection by a horizontally oriented mirror placed on top of the grasping table in front of the monkey. In the table plane, the target dot (red color, 0.1° radius) appeared within the range the human actor performed his action, allowing the reliable association of object position and eye movement records. As the monkey´s head was painlessly fixed (see below) during the experiment, eye position corresponded to gaze position. In the action observation task, attention to the action was ensured by making the delivery of reward dependent on gaze staying within a fixation window located horizontally at the center of the object, vertically 2.4° above the center of the front top edge of the object (from the monkey’s viewpoint) and extending ± 7° vertically and ± 5° horizontally.

The animals were implanted with a titanium post accommodating the painless fixation of the head and a titanium recording chamber overlying area F5 of the left hemisphere. The correct position of the chamber was determined using information from a pre-surgical anatomical MRI scan. All surgical procedures were conducted under strict aseptic conditions deploying combination anesthesia with isoflurane (0.8%) and remifentanil (1–2 microgram/kg·min) with full control of relevant physiological parameters such as body temperature, heart rate, blood pressure, PO₂, and PCO₂. Postoperatively, buprenorphine was given until signs of pain were gone. Animals were allowed to recover fully before starting the experiments.

Extracellular action potentials were recorded using glass-coated tungsten electrodes (0.5- to 2 MΩ impedance; Alpha Omega) using a multielectrode system equipped with up to eight probes (Alpha Omega Engineering). Action potentials of individual neurons were discriminated online resorting to template matching provided by Alpha Omega´s Multi Spike Detector.

The mirror neurons reported in this study were recorded from area F5 of the left hemispheres of the two experimental animals. Area F5 was targeted, guided by presurgical MRI, information on the location of the arcuate sulcus provided by electrode penetrations and a consideration of the response properties of neurons in F5 and in FEF as well as characteristic behavioral reactions to microstimulation with saccades evoked from the FEF and arm, hand, face, or mouth movements elicited when stimulating F5.

Unless mentioned otherwise, we used software based on MATLAB (2019a). For the analysis of discharge patterns, only well-isolated single units from area F5 were considered for which at least eight valid trials per condition were available. A trial was classified ‘valid’ if the aforementioned sequence of task events (‘start-button press’ to ‘withdraw from the object’) was registered by the sensors without technical malfunction. A neuron was classified as an F5 mirror neuron resorting to established criteria (e.g., Pomper et al., 2020). To this end, a task was subdivided into 5 epochs: baseline (–750 ms to –250 ms from LED onset), approach phase (from start-button release until touching the object), manipulation phase I (from touching the object until moving the object), manipulation phase II (from moving the object until holding the object in its target position), hold phase (from holding the object in its target position until 150ms later). A Friedman test with the factor ‘epoch’ was performed for each action (lift, twist or shift) separately. The neuron’s response was classified as ‘motor’ or ‘visual’, if the Friedman test was significant for at least one action (Bonferroni correction, alpha = .05/3) in execution or observation, respectively. The classification as ‘mirror neuron’ was made, if a neuron had a motor and a visual response. Note that in accordance with standard procedures this classification did not require that the modulation affected the same epoch or same direction (that is discharge rate increase vs. decrease).

We used a linear discriminant classifier (Fisher, 1936) by employing the ‘fitcdiscr’ function (discriminant type: diaglinear) available in MATLAB 2021a, in order to explore if the information offered by either single mirror neurons or the population discharge, predicted specific actions (lift, twist, shift). Single neuron classifications were performed on a neuron’s discharge rate in a given time bin. For the population classification, these discharge rates were used as a feature of the classifier. The number of trials for each of the six conditions (two tasks times three actions) considered for single neuron classification was neuron-dependent and corresponded to the minimum number of trials per condition across both tasks and all the three actions. We performed an eight-fold cross validation method, which required that the minimum number of trials had to be adjusted to a multiple of eight. For population classification, the number of trials for each task condition was eight, since only neurons with at least eight trials per condition were considered (this also met the eight-fold cross-validation requirement). To obtain a confidence interval of mean accuracy for each classification, a bootstrapping procedure was applied (n=1,000 resamples). For each resample, as many trials were randomly selected with replacement from all trials of a condition as had been determined as the minimum number of trials before. The mean over eight accuracies obtained from eightfold cross validation was calculated and constitutes the accuracy assigned to a resample. For single neuron classification, we used the mean of the obtained distribution of accuracies as the neuron’s classification accuracy in a certain time bin (Figure 4 and Figure 6A). We conducted four types of classifications. The classifiers were trained on execution or observation trials and tested on the same task (within-task classification: e2e, o2o) or the other task (cross-task classification: e2o, o2e) respectively.

To test whether actions (lift vs. twist vs. shift) were encoded in a neuron at a certain time bin when observed and when executed, we deployed the Kruskal-Wallis test for each neuron and bin during execution (alpha <0.05, Benjamini-Hochberg-corrected for 177*24 tests) and during observation (alpha <0.05, Benjamini-Hochberg-corrected for 177*24 tests). For the following three methods to test for shared action code we considered time bins during the action period (from start-button release to reward). In each method, a test for shared action code was performed in single neurons for bins encoding actions during both execution and observation determined by the Kruskal-Wallis test described above.

For each of the three actions a Wilcoxon rank sum test was applied bin-wise to test whether the discharge rate differs between the two tasks, execution and observation. The significance level of 5% was Benjamini-Hochberg-corrected for the number of tests (number of all bins in which actions were encoded times 3 for the three actions). If there was no significant difference for any of the three actions in a bin, the action-specific discharge rate was considered to be the same in both tasks.

For each bin, a positive action preference similarity index was determined according to the following four steps: (1) The mean discharge rate across trials was calculated for each of the three actions and for each of the two tasks resulting in two three-dimensional vectors, one for execution, one for observation. (2) Each vector was transformd so that the action with the highest discharge rate was given the value 1 and the other two were given the value 0. This was achieved by dividing by the highest discharge rate and then by rounding down. The case where the discharge rate was exactly the same or zero for all three actions was excluded by the preceding Kruskal-Wallis test, which had to be significant. (3) Each vector was normalized (by dividing by the length of the vector). (4) The dot product of the transformed and normalized vectors corresponds to the positive action preference similarity index. This index is 1 if the action preference is the same. It is 0 if different actions are preferred. The index is between 0 and 1 if two actions are preferred in one task but either only one of the two actions in the other task, or two actions in the other task with only one of the two being the same. To account for the case where one action had a lower discharge rate than the other two actions in both tasks, we also determined a negative action preference similarity index. This index was determined using the 4 steps described, with a different transformation in the second step. In order to give a value of 1 to the action with the lowest discharge rate, and a value of 0 to the others, the highest discharge rate was subtracted and the resulting vector was divided by its lowest value (with sign). Then, it was rounded down. We would like to point out that steps 3 and 4 of the determination of both indices are equal to the procedure described by Mazurek et al., 2018 for the determination of a similarity index. Our index, however, is different, as it allows to unambiguously infer a same action preference from an index value of 1 due to step 2. In addition, we excluded by the preceding statistical test that a value of 1 results from the unlikely but possible case that all discharge rates are exactly the same. To test for significance of same action preference we resorted to a bootstrapping procedure (n=1000 resamples). The bootstrap distribution and p-value were determined separately for the positive and negative action preference similarity index. The p-value was calculated by the relative number of resamples with an index below 1. The significance level of 5% was Benjamini-Hochberg-corrected for the number of tests (number of all bins in which actions were encoded times 2 for testing of positive and negative similarity index). If the positive or negative action preference similarity index of a bin had a significant value of 1 the same action preference was inferred.

To test for significance of cross-task classification in a time bin, we used the bootstrap distribution of accuracy of the e2o classifier in this bin (see Classification analyses). A p-value was calculated by the relative number of resamples with an accuracy smaller or equal to chance level (1/3*100%). The significance level of 5% was Benjamini-Hochberg-corrected for the number of tests (number of all bins in which actions were encoded).

To test against the null hypothesis that the coding of actions in the population of MNs at observation is independent of the coding at execution, which we believe is consistent with the null hypothesis suggested by Gergely Csibra, who formulated it as follows: "... motor and visual properties are randomly distributed across neurons, with no systematic relationship between them." (Csibra, 2005), we performed the following permutation test (n=250 resamples). For each resample, each neuron’s discharge rates of all trials of the three actions for execution were randomly assigned to a different neuron’s discharge rates of all trials of the three actions for observation. It was ensured that there were not two resamples with the same remapping. The test for action encoding in single neurons and the three methods for shared action code were applied to the data of each resample to obtain method-specific null distributions for the number of neurons with shared action code in each time bin of the action period (from start-button release to reward).

We compared the accuracies for predicting the observed actions of classifiers trained on execution (e2o classifiers) against that of classifiers trained on observation (o2o classifiers) for all bins of the action period (n=12*177=2,124 bins given n=177 MNs and n=12 bins of action period). Rather than using the absolute accuracies, we chose the difference relative to chance level, and plotted relative e2o accuracies vs. relative o2o accuracies (see Figure 4). In this way, data points indicating the same accuracies for both classifiers (e2o and o2o) fall on the unit line. Unequal accuracies lead to data points distributed around the zero line, which corresponds to the random level of e2o classifiers. In order to assess how data points are distributed with respect to these reference lines, we transformed each data point to polar coordinates and created a distribution of angles. Data points with an angle of 45 deg fall on the unity line and indicate a shared code, data points with 0 deg fall on the zero line. An obvious caveat is that meaningless data points are contained, characterized by the lack of action discrimination in the reference case, that is the o2o classification. As there is no a priori criterion that would allow one to distinguish good accuracy from insufficient accuracy, we compared the influence of increasingly restrictive, filtering thresholds on the o2o accuracy (starting from 5% above chance level to 15% in steps of 1%). For each threshold a one-Gaussian and a two-Gaussian fit was calculated for the histogram of the angles of all dots above this threshold using the function ‘fitoptions’ in MATLAB 2021a (method: non-linear least squares). The adjusted R-squared values and the residuals were calculated for both fits and are shown in Figure 4. For the two-Gaussian fit the peak values (‘lower peak’ and ‘upper peak’) and the trough value between them (‘boundary’) were determined. A 95%-confidence interval was calculated for adjusted R-squared, lower peak, upper peak and boundary by bootstrapping (original data, n=2124,, were resampled with replacement 1000 times). Whenever the algorithm failed to find a fit with two distinct modes, the procedure was repeated with new initial guesses. This reduced the number of failed fits from 3.25% to 1.8%. The average boundary angle was calculated across the boundaries of the 11 thresholds.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

We acknowledge the contribution of Dan Arnstein in the design of the experiment and Dr. Peter Dicke for the presurgical MRT scans and implant design, assistance in surgeries and postsurgical care of animals as well as for his invaluable support in developing the experimental setup. We acknowledge support from the Open Access Publication Fund of the University of Tübingen.

All experiments were approved and controlled by the regional veterinary administration (Regierungspräsidium Tübingen and Landratsamt Tübingen, Permit Number: N4/14) and conducted in accordance with German and European law and the National Institutes of Health's Guide for the Care and Use of Laboratory Animals, and regularly and carefully monitored by the veterinary service of the University of Tübingen, the latter also providing care in case of medical problems.

© 2023, Pomper, Shams, Wen et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.