Higher social tolerance is associated with more complex facial behavior in macaques

Reviewer #1 (Public Review):

This study investigates the context-specificity of facial expressions in three species of macaques to test predictions for the 'social complexity hypothesis for communicative complexity'. This hypothesis has garnered much attention in recent years. A proper test of this hypothesis requires clear definitions of 'communicative complexity' and 'social complexity'. Importantly, these two facets of a society must not be derived from the same data because otherwise, any link between the two would be trivial. For instance, if social complexity is derived from the types of interactions individuals have, and different types of signals accompany these interactions, we would not learn anything from a correlation between social and communicative complexity, as both stem from the same data.

The authors of the present paper make a big step forward in operationalising communicative complexity. They used the Facial Action Coding System to code a large number of facial expressions in macaques. This system allows decomposing facial expressions into different action units, such as 'upper lid raiser', 'upper lip raiser' etc.; these units are closely linked to activating specific muscles or muscle groups. Based on these data, the authors calculated three measures derived from information theory: entropy, specificity and prediction error. These parts of the analysis will be useful for future studies.

The three species of macaque varied in these three dimensions. In terms of entropy, there were differences with regard to context (and if there are these context-specific differences, then why pool the data?). Barbary and Tonkean macaques showed lower specificity than rhesus macaques. Regarding predicting context from the facial signals, a random forest classifier yielded the highest prediction values for rhesus monkeys. These results align with an earlier study by Preuschoft and van Schaik (2000), who found that less despotic species have greater variability in facial expressions and usage.

Crucially, the three species under study are also known to vary in terms of their social tolerance. According to the highly influential framework proposed by Bernard Thierry, the members of the genus Macaca fall along a graded continuum from despotic (grade 1) to highly tolerant (grade 4). The three species chosen for the present study represent grade 1 (rhesus monkeys), grade 3 (Barbary macaques), and grade 4 (Tonkean macaques).

The authors of the present paper define social complexity as equivalent to social tolerance - but how is social tolerance defined? Thierry used aggression and conflict resolution patterns to classify the different macaque species, with the steepness of the rank hierarchy and the degree of nepotism (kin bias) being essential. However, aggression and conflict resolution are accompanied by facial gestures. Thus, the authors are looking at two sides of the same coin when investigating the link between social complexity (as defined by the authors) and communicative complexity. Therefore, I am not convinced that this study makes a significant advance in testing the social complexity for communicative complexity hypothesis. A further weakness is that - despite the careful analysis - only three species were considered; thus, the effective sample size is very small.

Reviewer #2 (Public Review):

This is a well-written manuscript about a strong comparative study of diversity of facial movements in three macaque species to test arguments about social complexity influencing communicative complexity. My major criticism has to do with the lack of any reporting of inter-observer reliability statistics - see comment below. Reporting high levels of inter-observer reliability is crucial for making clear the authors have minimized chances of possible observer biases in a study like this, where it is not possible to code the data blind with regard to comparison group. My other comments and questions follow by line number:

38-40. Whereas I am an advocate of this hypothesis and have tested it myself, the authors should probably comment here, or later in the discussion, about the reverse argument - greater communicative complexity (driven by other selection pressures) could make more complicated social structures possible. This latter view was the one advocated by McComb & Semple in their foundational 2005 Biology Letters comparative study of relationships between vocal repertoire size and typical group size in non-human primate species.

72-84 and 95-96. In the paragraph here, the authors outline an argument about increasing uncertainty / entropy mapping on to increasing complexity in a system (social or communicative). In lines 95-96, though, they fall back on the standard argument about complex systems having intermediate levels of uncertainty (complete uncertainty roughly = random and complete certainty roughly = simple). Various authors have put forward what I think are useful ways of thinking about complexity in groups - from the perspective of an insider (i.e., a group member, where greater randomness is, in fact, greater complexity) vs from the perspective of an outside (i.e., a researcher trying to quantify the complexity of the system where is it relatively easy to explain a completely predictable or completely random system but harder to do so for an intermediately ordered or random system). This sort of argument (Andrew Whiten had an early paper that made this argument) might be worth raising here or later in the discussion? (I'm also curious where the authors sentiments lie for this question - they seem to touch on it in lines 285-287, but I think it's worth unpacking a little more here!)

115-129. See also:
Maestripieri, D. (2005). "Gestural communication in three species of macaques (Macaca mulatta, M. nemestrina, M. arctoides): use of signals in relation to dominance and social context." Gesture 5: 57-73.
Maestripieri, D. and K. Wallen (1997). "Affiliative and submissive communication in rhesus macaques." Primates 38(2): 127-138.
On that note, it is probably worth discussing in this paragraph and probably later in the discussion exactly how this study differs from these earlier studies of Maestripieri. I think the fact that machine learning approaches had the most difficulty assigning crested data to context is an important methodological advance for addressing these sorts of questions - there are probably other important differences between the authors' study here and these older publications that are worth bringing up.

220-222. What is known about visual perception in these species? Recent arguments suggest that more socially complex species should have more sensitive perceptual processing abilities for other individuals' signals and cues (see Freeberg et al. 2019 Animal Behaviour). Are there any published empirical data to this effect, ideally from the visual domain but perhaps from any domain?

274-277. I am not sure I follow this - could not different social and non-social contexts produce variation in different affective states such that "emotion"-based signals could be as flexible / uncertain as seemingly volitional / information-based / referential-like signals? This issue is probably too far away from the main points of this paper, but I suspect the authors' argument in this sentence is too simplified or overstated with regard to more affect-based signals.

288 on. Given there are only three species in this study, the chances of one of the species being the 'most complex' in any measure is 0.33. Although I do not believe this argument I am making here, can the authors rule out the possibility that their findings related to crested macaques are all related to chance, statistically speaking?

329-330. The fact that only one male rhesus macaque was assessed here seems problematic, given the balance of sexes in the other two species. Can the authors comment more on this - are the gestures they are studying here identical across the sexes?

354-371. Inter-observer reliability statistics are required here - one of the authors who did not code the original data set, or a trained observer who is not an author, could easily code a subset of the video files to obtain inter-observer reliability data. This is important for ruling out potential unconscious observer biases in coding the data.