1. Ecology
  2. Evolutionary Biology
Download icon

Quorums enable optimal pooling of independent judgements in biological systems

  1. James AR Marshall  Is a corresponding author
  2. Ralf HJM Kurvers
  3. Jens Krause
  4. Max Wolf  Is a corresponding author
  1. University of Sheffield, United Kingdom
  2. Max Planck Institute for Human Development, Germany
  3. Leibniz-Institute of Freshwater Ecology and Inland Fisheries, Germany
Research Communication
  • Cited 1
  • Views 616
  • Annotations
Cite this article as: eLife 2019;8:e40368 doi: 10.7554/eLife.40368

Abstract

Collective decision-making is ubiquitous, and majority-voting and the Condorcet Jury Theorem pervade thinking about collective decision-making. Thus, it is typically assumed that majority-voting is the best possible decision mechanism, and that scenarios exist where individually-weak decision-makers should not pool information. Condorcet and its applications implicitly assume that only one kind of error can be made, yet signal detection theory shows two kinds of errors exist, ‘false positives’ and ‘false negatives’. We apply signal detection theory to collective decision-making to show that majority voting is frequently sub-optimal, and can be optimally replaced by quorum decision-making. While quorums have been proposed to resolve within-group conflicts, or manage speed-accuracy trade-offs, our analysis applies to groups with aligned interests undertaking single-shot decisions. Our results help explain the ubiquity of quorum decision-making in nature, relate the use of sub- and super-majority quorums to decision ecology, and may inform the design of artificial decision-making systems.

Editorial note: This article has been through an editorial process in which the authors decide how to respond to the issues raised during peer review. The Reviewing Editor's assessment is that all the issues have been addressed (see decision letter).

https://doi.org/10.7554/eLife.40368.001

Introduction

Effective decision-making is essential in all areas of human society and, more generally, for all organisms. A fundamental question in this context is when a group of decision-makers is superior to individual decision-makers and vice-versa (Galton, 1907; Surowiecki, 2005; Bahrami et al., 2010; Lorenz et al., 2011; Koriat, 2012; Kurvers et al., 2016). Both in human and animal collective decision-making, the Condorcet Jury Theorem is one of the key principles guiding our thinking about this question (List, 2004; Hastie and Kameda, 2005; King and Cowlishaw, 2007; Sumpter et al., 2008; Austen-Smith and Feddersen, 2009; Conradt and List, 2009; Kao and Couzin, 2014a; Marshall et al., 2017). In a nutshell, for pairwise decision problems (e.g. disease-, lie- or predator detection), Condorcet’s Jury Theorem states that a group of decision-makers employing the majority rule is superior to individual decision-makers in contexts where individuals are relatively accurate (i.e. accuracy >50%); conversely, individual decision-makers are superior to majority voting groups in contexts where individuals are relatively inaccurate (i.e. accuracy <50%). Consequently, across diverse fields ranging from organismal behaviour and human psychology to political sciences, two heuristics are commonly employed (List, 2004; Hastie and Kameda, 2005; King and Cowlishaw, 2007; Sumpter et al., 2008; Conradt and List, 2009; Kao and Couzin, 2014a): (i) groups of decision-makers outperform individuals only in contexts where individuals are relatively accurate and (ii) the majority rule is a powerful mechanism to reap the benefits of collective decision-making. We here show that both statements are not true, and in doing so explain the ubiquity of quorum decision rules in the collective behaviour of humans and other social organisms (Seeley and Visscher, 2004; Sumpter and Pratt, 2009; Ward et al., 2012; Pratt et al., 2002; Ross-Gillespie and Kümmerli, 2014; Walker et al., 2017; Bousquet et al., 2011).

Over the past few decades, substantial research effort has focussed on two key explicit assumptions underlying Condorcet’s Jury Theorem, independence (i.e. judgments/votes by different members of the group are assumed to be independent from each other) and homogeneity (i.e. all decision-makers within a group are assumed to be identical, both in competence and in goals) (Kao and Couzin, 2014a; Sumpter and Pratt, 2009; Boland, 1989; Ladha, 1992; Berg, 1993; Marshall et al., 2017). We here focus on a third, implicit, assumption of Condorcet’s Jury Theorem, namely that decision-makers make only one type of error. This assumption stands in contradiction to the well-known fact that, when confronted with a pairwise decision problem like the one studied in Condorcet’s Jury Theorem, two different types of error are possible (i.e. false positive and false negative). Surprisingly, up to now, this basic and well-known feature of pairwise decision problems has not been fully taken into account when investigating Condorcet’s Jury Theorem.

We start by providing a brief summary of the basic model considered in Condorcet’s Jury Theorem. We then introduce an extended model that takes into account the fact that decision-makers can make two types of errors. Based on this extended model, we then show that Condorcet’s Jury Theorem makes several important predictive errors, which apply to the majority of decision scenarios. Moreover, majority voting is frequently suboptimal, whereas quorum-based voting with an appropriate quorum is always optimal, in that it enables groups to simultaneously maximise true positive rate and minimise false positive rate (Wolf et al., 2013). While an analytical solution for the optimal quorum threshold has been derived before (Ben-Yashar and Nitzan, 1997), dependent on true and false positive rates and key ecological characteristics (i.e. classification error cost, prior probabilities), this analysis treated true and false positive rates as independent of these ecological characteristics, whereas in reality the former depend on the latter. In contrast, here we also make use of signal detection theory to optimise individual decision-makers, thereby delineating precisely where non-majority quorums should be used, depending on parameters of the decision ecology. Thus, the simple majority threshold is only a special case of the more general quorum decision mechanism, in which optimal sub-majority or super-majority quorums are the rule rather than the exception.

Methods

Condorcet’s Jury Theorem: the basic model

Request a detailed protocol

Condorcet’s Jury Theorem considers a binary (pairwise) choice situation, in which a decision-maker can choose between two actions, labelled +1 and –1. Each decision-maker is characterised by a single parameter a, corresponding to the probability of making a correct decision, or decision accuracy. Importantly, the decision accuracy a of each decision-maker is assumed to be conditionally-independent of the realised decisions of all other decision-makers.

Condorcet’s Jury Theorem now considers a group of identical decision-makers of size N that performs a majority vote. A simple combinatorial argument shows that if the accuracy of decision-makers is above 50% (i.e. a > 0.5), then the probability of making a correct choice increases with increasing group size and asymptotically approaches 1 (Boland, 1989). Conversely, if the accuracy of individual decision-makers is below 50% (i.e. a < 0.5), then the probability of making a correct choice decreases with increasing group size and asymptotically approaches 0 (King and Cowlishaw, 2007). This is because, as group size increases, the probability of the more probable decision (+ or –) also being the majority decision rapidly increases towards one.

These results have led to three key interpretations: first, pooling independent judgements is beneficial (i.e. improves decision accuracy) whenever individuals are relatively good decision-makers (a > 0.5) (List, 2004; King and Cowlishaw, 2007; Novaes Tump et al., 2018). Second, pooling judgements is detrimental (i.e. decreases decision accuracy) whenever individuals are poor decision-makers (a < 0.5) (List, 2004; King and Cowlishaw, 2007; Novaes Tump et al., 2018). Third, the majority rule is the appropriate mechanism to reap the benefits of collective decision-making (List, 2004; Hastie and Kameda, 2005).

In the following, we show that each of these interpretations is incorrect. In particular, we show that (i) in cases where decision-makers are good (a > 0.5) majority decisions may decrease decision accuracy, (ii) in cases where decision-makers are poor (a < 0.5) majority decisions may increase decision accuracy, and (iii) pooling independent decisions is always beneficial as long as an appropriate quorum-based decision rule is used. Taken together, these results show that, for a large proportion of decision scenarios, the simple majority decision rule performs poorly and gives incorrect predictions about group decision accuracy.

Condorcet’s Jury Theorem: an extended model

Request a detailed protocol

Condorcet’s Jury Theorem implicitly assumes that decision-makers can make only one error, that is, the probability of making an incorrect decision is 1 – a. This is in contrast to the well-known fact that – in pairwise decision problems – decision-makers can make two types of errors, false positives and false negatives (Green and Swets, 1966; Swets, 1988). For example, an animal under predation risk may run away in the absence of a predator (false positive) or it may not run away in the presence of a predator (false negative) (Trimmer et al., 2008). Similarly, a doctor screening for a disease may diagnose in the absence of a disease (false positive) or not diagnose in the presence of a disease (false negative) (Wolf et al., 2013). As we discuss below, the implicit assumption of Condorcet’s Jury Theorem is equivalent to assuming that decision-makers have an identical probability of committing the two errors – this is an important assumption that does not reflect the vast majority of real world decisions.

We here consider an extension of the above-described model, which takes into account the fact that decision-makers can make two types of errors. Again, decision-makers can choose between two actions, +1 and –1. Unlike in the basic model above, however, and consistent with standard decision theory for pairwise decisions, we now assume that the world can be in two states, state + and state –, corresponding to, for example, the presence and absence of a predator or the presence and the absence of a disease; state + holds with probability p and state – thus holds with probability 1 – p. Action +1 is the better choice in state + (i.e. it achieves a higher payoff), while action –1 is the better choice in state –; for example, running away is better than staying in the presence of a predator, while staying is better than running away in the absence of a predator. Consequently, and in contrast to the basic Condorcet model above, each decision-maker is now characterised by two parameters a+ and a, corresponding to the probabilities of making correct decisions in state + and state –, respectively; this inevitably implies that individuals make two types of errors. As in the simple model above, where decision-makers have equal accuracies a and are independent, for any individual in a group of N individuals the probability of making a correct decision based on the true state of the world is conditionally-independent of the probabilities of other group members making the correct decision, given that same state of the world.

To relate our analysis to predictions made by applying Condorcet, we must define expected individual accuracy a, as used in Condorcet, in terms of our state probability p and state-wise accuracy parameters a+ and a. The expected individual accuracy is thus

(1) a=pa++(1p)a.

From Equation 1 we can see that Condorcet implicitly assumes that accuracies in the two states of the world are equal since then p disappears from the equation; as shown in Supplementary Information, this occurs when both states of the world are equally likely, and the costs and benefits of classifications in the two states of the world are equal, although asymmetric decision problems can also result in equal accuracies (as can be confirmed with reference to Supplementary Information for Figure 1). Later, we explain how signal detection theory determines optimal values of a+ and a for a given decision scenario where these assumptions are violated.

Receiver Operating Characteristic (ROC) curve plot showing the optimal compromise between true positive rate (TPR = a+) and false positive rate (FPR = 1 – a).

The shape of the ROC curve is determined by the difficulty of the decision problem (e.g. Shettleworth, 2010), with hard decision curves lying close to the dashed diagonal, and easy decision curves approaching the left and top edge of the plot. According to prior probability of states, and relative cost of errors, an optimal decision-maker then selects a point on the ROC curve that gives the best possible expected decision performance; points that do not prioritise accuracy in either state of the world (a+=a ⇔ TPR = 1 – FPR; dotted diagonal, and filled circle) are implicitly assumed by the Condorcet Jury Theorem, as discussed in the main text and Figure 2. The white square represents the region of ROC space in which simple majority decisions are best, and Condorcet predictions are fulfilled (note the equal accuracy dotted line described above always occupies this region). Light grey triangles represent regions of ROC-space which, if selected by optimal individual decision-makers, lead to sub-optimal collective decisions when combined by simple majority decision rules; in this region Condorcet predictions are systematically incorrect. The dark grey lower-right triangle represents combinations of TPR and FPR that should not be observed, since in these decision-makers are systematically wrong and could simultaneously improve both their TPR and FPR by simply inverting their predictions to move above the dashed diagonal. ROC parameters: μ– = 0, μ+ = 1, σ = 1, prior = ½, ratio = 1.

https://doi.org/10.7554/eLife.40368.002

Results

We are now ready to formalise when Condorcet gives incorrect predictions, and when simple majority voting is suboptimal. We begin by determining optimal values of a+ and a. Given the prior probability p, the cost and benefits associated with the two states of the world, and the state-dependent characteristics of the cue(s) decision-makers base their decisions on, the optimal realised individual accuracies a+ and a are derived by solving a signal detection problem (Figure 1). Signal detection theory has been applied repeatedly to hypothetical and real world decision problems, for example predator detection by foraging animals (Trimmer et al., 2008; Trimmer et al., 2017); diagnostic decision-making (Kurvers et al., 2016; Wolf et al., 2015); and lie detection (Klein and Epley, 2015). As described in the appendix, for simplicity and tractability our analysis is conducted for the simplest signal detection problem, determining which of two normal distributions a single scalar random variable is drawn from; this could, in the example of a predator detection problem, be the instantaneous volume of a sound heard by a forager. In this case signal detection theory enables us to find a Receiver Operating Characteristic (ROC) curve of optimal a+, a value pairs, dependent on decision difficulty (Figure 1; Green and Swets, 1966; Shettleworth, 2010); the optimal point of this curve gives us a unique pair of a+ and a values dependent on costs and benefits of different decision outcomes, and the prior probability of the + state, p.

We next make the observation that, to simultaneously improve both true positive and false positive rates, a group must choose a quorum q that lies between these two values (Wolf et al., 2013); that is, an optimal group must choose q such that

(2) 1a<q<a+

The intuition behind this result is that group accuracy will converge on the appropriate accuracy for the true state of the world, as group size increases; thus, by setting a quorum between these two accuracies the true state of the world can be determined with high probability for sufficiently large groups. For further details see (Wolf et al., 2013). Since the simple majority quorum q = 1/2, assumed by Condorcet, only satisfies this inequality when both a+>1/2 and a>1/2, when either of these conditions are violated then both simple majority decisions and Condorcet-based reasoning will be deficient. Furthermore, we should never see true positive rate (a+) less than false positive rate (1a) (dark grey region of ROC space in Figure 1), since such a decision-maker could simultaneously improve both their true and false positive rates simply by inverting their decisions and moving above the diagonal. Therefore, the ROC space is divided into two meaningful regions: in the first a+>1/2 and a<1/2 (white region in Figure 1), so simple majority voting is asymptotically-optimal as group size increases, and Condorcet-based predictions are correct. In the second, a+<1/2 and a>1/2, or a+>1/2 and a<1/2, while ensuring a+>1a (light grey regions in Figure 1); in these regions simple majority decisions will be sub-optimal, and Condorcet-based reasoning will be erroneous.

We now describe the systematic errors Condorcet leads to when faced with these decision scenarios. These errors can also be described in terms of false positive (frequently referred to as type I) and false negative (type II) errors, in predicting the performance of the Condorcet majority rule, hence we label the errors in such terms.

Error Ia: Condorcet predicts group accuracy approaches 1, but majority groups do not

This error occurs when majority voting is sub-optimal (a+<1/2 and a>1/2, or a+>1/2 and a<1/2; light grey regions in Figure 1, as described above), and when expected individual accuracy a>1/2, since under this last condition individuals on average make more correct decisions than incorrect decisions, and Condorcet thus predicts that group accuracy approaches one as group size increases (King and Cowlishaw, 2007; Boland, 1989). From Equation 1 this requirement is thus that

p(a+a)+a>1/2

While Condorcet predicts group accuracy (denoted a-) approaches 1, that is

(4) limna-=1

in fact the majority quorum q = 1/2 is either below 1 – a-, or above a+, contra inequality 2; thus the group converges to making completely correct choices in one state of the world, and completely incorrect choices in the other state of the world (Wolf et al., 2013). Hence, group accuracy converges to

limna={pif a+>12and a<12 1pif a+<12and a>12

The wide range of decision scenarios in which Condorcet makes this predictive error are illustrated in Figure 2a. An example of this error is presented in Figure 3a; Figure 3b illustrates how the error can be avoided by choosing an appropriate quorum, in this case a super-majority quorum.

Signal detection theory shows when majority-based reasoning is incorrect.

Decision scenarios in which Condorcet will make different kinds of inaccurate predictions about groups of individually-optimal (see Figure 1) decision-makers (a–c), and in which majority voting is sub-optimal (d); prior probability of state + is on x-axis, ratio of cost of false positive to cost of false negative is on y-axis, so values larger than one indicate classifying – as + is relatively worse than classifying + as –, and vice versa. Grey regions show decision scenarios in which Condorcet leads to at least one predictive error: (a) As group size increases Condorcet incorrectly predicts that majority group accuracy will asymptotically increase towards 1, whereas it actually does not (Error Ia, main text); (b) As group size increases Condorcet incorrectly predicts that majority group accuracy will asymptotically increase towards 1, whereas groups actually make worse decisions than individuals (Error Ib, main text); (c) As group size increases Condorcet incorrectly predicts that majority group accuracy will decrease towards zero, whereas it actually converges to a positive level (Error II, main text); (d) Condorcet makes at least one of the errors just described; this plot also corresponds to decision scenarios in which majority decision-making is suboptimal, and should be replaced by a sub- (upper-left area) or super- (lower-right area) majority quorum rule as described in Results. Signal detection analysis for individual decision-makers is described in the Appendix; parameters for the analysis are μ-=0,μ+=σ=1, CTP=CTN=0, CFN=1, CFP=ratio.

https://doi.org/10.7554/eLife.40368.003
How majority voting can be optimally replaced by quorum decisions.

Illustrative failures of simple majority voting and Condorcet predictions (left column) and their remediation through appropriate use of sub- or super-majority decision quorums (right column); group size increases on x-axis, while group accuracy increases on y-axis; red-dashed line indicates individual decision accuracy, black dot-dash line represents prior probability of state + (panels a and c) or of state – (panel e), solid blue line represents group accuracy. (a) Error Ia: Condorcet predicts that increasing group size will result in group accuracy converging to 1, but it converges to p which is above the individual accuracy level. (b) Choosing a super-majority quorum leads group accuracy to converge to 1 with increasing group size. (c) Error Ib: Condorcet predicts that increasing group size will result in group accuracy converging to 1, but it converges to p which is below the individual accuracy level, hence groups perform worse than individuals. (d) Choosing a super-majority quorum leads group accuracy to converge to 1 with increasing group size. (e) Error II: Condorcet predicts that increasing group size will result in group accuracy converging to 0, but it converges to 1 – p which is below the individual accuracy level. (f) Choosing a sub-majority quorum leads group accuracy to converge to 1 with increasing group size. Simulations comprise 10,000 replicates with individual accuracies derived from signal detection analysis with parameters μ-=0,μ+=σ=1, CTP=CTN=0, CFN=1 and: (aCFP=4,p=0.9,q=0.5, (bCFP=4,p=0.9,q=0.7, (cCFP=1,p=0.7,q=0.5, (dCFP=1,p=0.7,q=0.75, (eCFP=10,p=0.7,q=0.5, (fCFP=10,p=0.7,q=0.035.

https://doi.org/10.7554/eLife.40368.004

Error Ib: Condorcet predicts group accuracy approaches 1, but majority groups are worse than individuals

In error Ia, while group accuracy does not converge to one with increasing group size, nothing is said about whether or not groups are better than individuals. Error Ib refines error Ia, by showing that there are cases where Condorcet predicts group accuracy approaching 1, but groups actually have lower accuracy than individuals. These cases are found by refining the conditions given in error Ia to include the additional condition that the group accuracy converged to (Equation 5 above) is less than individual expected accuracy (Equation 1). This gives the conditions

(6) p<a1a++a ifa+>12and a<12,and
p>1a1+a+a ifa+<12and a>12

The wide range of decision scenarios in which Condorcet makes this predictive error are illustrated in Figure 2b. An example of this error is presented in Figure 3c; Figure 3d illustrates how the error can be avoided by choosing an appropriate quorum, in this case a super-majority quorum.

Error II: Condorcet predicts group accuracy approaches 0, but majority groups do not

In error II individual expected accuracy is below 1/2, thus Condorcet predicts that group accuracy should converge to 0; while groups using the majority decision rule do decrease in accuracy, they converge to a non-zero group accuracy given by Equation 5 above. To find cases where this occurs we simply solve for when individual expected accuracy a < 1/2. From Equation 1 this gives us the conditions

(8) p<1/2aa+a ifa+>12and a<12,and
(9) p>1/2aa+a ifa+<12and a>12

Decision scenarios in which Condorcet makes this predictive error are illustrated in Figure 2c. An example of this error is presented in Figure 3e; Figure 3f illustrates how the error can be avoided by choosing an appropriate quorum, in this case a sub-majority quorum.

Note that it is not possible to find an ‘Error IIb’ case that parallels Error Ib; that is if Condorcet predicts that group accuracy approaches 0, majority groups will always be worse than individuals and never better, even if their group accuracy remains positive. This is because it is not possible simultaneously to satisfy the conditions just given (inequalities 8 and 9), and the opposite of the conditions (inequalities 6 and 7) given in Error Ib (i.e. the conditions that group accuracy converged to is greater than individual expected accuracy), as can be confirmed by algebra.

Majority voting is usually suboptimal

Combining the cases in which Condorcet makes one of the above described predictive errors, Figure 2d illustrates when Condorcet will make at least one error in predicting the performance of decision-making groups using majority decision-making. This set also corresponds to the set of decision scenarios in which majority decision-making is sub-optimal, and is outperformed by an appropriately set sub- or super-majority quorum. Figure 2d shows that Condorcet is optimal in far fewer decision scenarios than those in which it is outperformed by an appropriate quorum rule. Furthermore, our analysis also relates decision-ecology to the requirement for sub- or super-majority quorums. From inequality (2), sub-majority quorums are required whenever a< 1/2, which corresponds to the upper-left area of Figure 2d; in contrast, super-majority quorums are required whenever 1 – a> 1/2, which corresponds with the lower-right area of Figure 2d (see Supplementary Information). Thus, whenever the positive state of the world + is rarer, and/or false positives are relatively expensive compared to false negatives, then a sub-majority quorum should typically be employed, while the converse holds for super-majority quorums.

Discussion

We have shown that simple majority-based collective decisions are often suboptimal, and that consequently sub- or super-majority quorums should frequently be employed by groups of ‘like-minded’ individuals combining independent decisions. Our results are important for two reasons. First, the majority-based decision rule, and Condorcet-based reasoning, is widespread in several major branches of collective decision theory. In the animal behaviour community the Condorcet prescription on individual accuracy exceeding ½ has been used to recommend when opinions should be pooled (King and Cowlishaw, 2007), and when experts should be favoured over group opinions (Katsikopoulos and King, 2010). Other authors have also invoked Condorcet and majority voting as the gold-standard for collective decision-making (Hastie and Kameda, 2005; Kao and Couzin, 2014a; Kao et al., 2014b; Miller et al., 2013). These results have been useful in highlighting the benefits of information pooling in collective decisions, but such studies implicitly neglect the reality that most decisions have two types of errors. Second, while quorums have been widely studied in collective decision theory, we here present a comprehensive theory that may help explain their prevalence. Sub- and super-majority quorums have been considered theoretically (Sumpter and Pratt, 2009); Sumpter and Pratt implicitly assume only one type of error need be considered, and proceed from that point with their analysis, referring to quorum functions such as managing speed-accuracy trade-offs (Marshall et al., 2009). Ward et al. (2008) consider quorum use for facilitating information transfer in shoaling fish, yet ignore the possibility of different error types despite their great asymmetry in the scenario studied, predator detection. Conradt and Roper (2005) in their review refer to true and false positives, but in explaining quorum usage refer back to earlier analysis as a mechanism to avoid extreme group decisions where individual fitness interests do not completely align (Conradt and Roper, 2003). However List (2004) as well as Conradt and List (2009) noted the effect of cost and prior asymmetry on quorum usage, referring back to earlier political science results (Ben-Yashar and Nitzan, 1997) discussed below. We also note that convergence on an intermediate group accuracy between 0 and 1 has previously been observed, without consideration of signal detection theory; the result presented in Figure 1b of Kao and Couzin (2014a) occurs for a similar reason to our error Ia, in that inappropriate use of a majority decision rule leads increasing group size to result in group accuracy converging on a parameter of the decision problem, in their case the reliability of a low correlation cue.

In contrast to these previous analyses here we have shown the fundamental importance of quorums in one of the simplest possible collective behaviour scenarios, single-shot collective decisions in homogenous groups, where individuals’ interests are aligned, and decision-making abilities do not differ. Thus, we might expect the use of quorums to be widespread in the natural world, even in the simplest of decisions. The widespread use of quorum sensing in bacteria provides evidence of this (Gross, 2017), and may prove a particularly good testbed for our theory given its binary nature and asymmetric state priors and error costs, although evolutionary conflicts of interests within bacterial communities may result in confounds. Moreover, as humans have been shown to employ quorum rules and adaptively adjust the associated quorum thresholds, human decision-making experiments may also provide a powerful approach to test our predictions (Kurvers et al., 2014; Clément et al., 2015).

It is surprising that signal detection theory has seen relatively scant application to collective decision-making. Wolf et al. (2013) noted the potential for different error types in identifying how quorums can improve collective decision accuracy; while motivated by signal detection theory they did not directly apply it to optimise the individual decision-makers and relate this back to group behaviour. Kirstein and von Wangenheim, 2010 also noted the possibility for independent error types, again with reference to signal detection theory; they noted the potential for Condorcet to make the same qualitatively incorrect predictions that we note here, however they did not apply the relevant theory to delineate the situations under which Condorcet reasoning is incorrect, nor did they consider the possibility for quorum-based decision rules to rescue these situations. Sorkin et al. (2001) applied signal detection theory to Condorcet-like models with varying super-majority quorums, but did not find the mechanism by which group decisions can be optimised (Wolf et al., 2013). Laan et al. (2017) noted that Condorcet and majority voting can be suboptimal, and considered ways in which voting can be improved; while they discussed signal detection theory and voting mechanisms they neither explicitly considered error types, nor quorum thresholds, focussing mainly on correlations between decision-makers and the impact of the cost function used, as well as suggesting a data-driven machine learning approach to improving collective decision-making rules. The results that most closely anticipate ours are those of Ben-Yashar and Nitzan (1997), who analytically solved for the general optimal decision rule by recognising both error cost and prior asymmetry, as well as simultaneously considering the case of variable individual decision-ability (e.g. Marshall et al., 2017). These results describe the relationship between prior asymmetry and optimal quorum threshold, and error cost and optimal quorum threshold. However, because they did not apply signal detection theory to optimise individual agents’ decisions (i.e. they treated true and false positive rates of individuals as independent from the ecological parameters error cost and prior asymmetry), they were unable quantitatively to uncover the complex nonlinear relationship between these three quantities (Figure 2).

In contrast to earlier work, by applying signal detection theory we have simultaneously shown here both how fragile Condorcet and majority-voting are, and how the use of sub- or super-majority quorums should relate to decision ecology. Although simple collective behaviour models have been well studied and highly influential, our results, and others relaxing other assumptions of such models (Marshall et al., 2017), indicate the subtlety that may be revealed in collective decision-making by a richer consideration of individual decision theory. Other approaches to such problems should be pursued in the future. For example, it is possible to optimise individual quorums (Ben-Yashar and Nitzan, 1997) rather than simply set them within a suitable interval as we do here, thereby giving greater benefits to smaller groups. Similarly, rather than apply signal detection theory one could apply Bayesian decision-theory (e.g. Pérez-Escudero and de Polavieja, 2011; Arganda et al., 2012; Pérez-Escudero and de Polavieja, 2017), thereby attempting to deal with further complexities such as non-independence of individual decisions. In many scenarios errors and correct decisions may be correlated although even when multiple individuals observe the same stimulus their information can be considered independent due to sensory noise (Marshall et al., 2017). We believe that our simple approach has, however, the benefit of tractability while still revealing the complexity of collective decision-making even in the simplified scenario considered.

Our work takes inspiration from political science and decision theory to address questions in behavioural ecology, but may additionally have the potential to inform work in the design of artificial decision-making systems, machine learning and robotics. For example, in the field of ensemble learning, in which predictions from multiple weak classifiers such as neural networks are combined to improve decision accuracy, variable quorums, referred to as ‘threshold shift’, are used (e.g. Dmochowski et al., 2010). However majority voting is still among the simplest and most ubiquitous vote fusion rules discussed (Sagi and Rokach, 2018; Krawczyk et al., 2017). Hence, we suggest that the simple perspective on how to combine votes presented here may also yield technological insight.

Appendix 1

Signal detection theory reveals cases where Condorcet Predictions are incorrect

To determine if a group of optimal decision-makers could exist such that simple application of Condorcet would lead to erroneous predictions, but a quorum rule would allow optimal opinion pooling, we consider groups of identically-capable individual decision-makers (as assumed in Condorcet), modelled as making optimal decisions using signal detection theory in order to classify continuous signals arising from one of two possible normal signal distributions of equal variance (Green and Swets, 1966). That is, decision-makers are faced with a signal

(A.1) s{N(μ+,σ)withprobabilitypN(μ,σ)withprobability1p

and must choose an optimal signal threshold, x, in order to classify signals as being drawn from either of the two possible normal distributions. Each distribution has a different mean (μ+ versus μ-) but the same standard deviation (σ). In a natural setting the signal could represent information as to whether a predator is present or not, for example, with the two states of the world, predator present versus predator absent, having different distributions for this signal.

The optimal decision threshold x is chosen to minimise the expected loss for the decision-maker (or maximise the expected gain). The expected loss from a decision is

(A.2) ELoss=pa+CTP+1-a+CFN+1-pa-CTN+1-a-CFP,

where CTP, CFN, CTN and CFP are respectively the costs of true positives (correctly classifying state +), false negatives (incorrectly classifying state –), true negatives (correctly classifying state –) and false positives (incorrectly classifying state +). Thus, an optimal decision-maker should minimise (equation A.2) by appropriately choosing the decision threshold x. Since, given x,

(A.3) a+=1Φ(μ+,σ,x),and
(A.4) a-=Φμ-,σ,x,

where Φ is the cumulative distribution function for the normal distribution, the optimal x can be found by substituting (equation A.3) and (equation A.4) into (equation A.2), differentiating the resulting equation and solving for zero (Green and Swets, 1966). Note that the optimal threshold x, and thus the optimal accuracies under state + and state – of the world, a+and a- respectively, are affected both by class imbalance (p1/2), and by asymmetric error costs (CTP-CFNCTN-CFP).

References

  1. 1
  2. 2
    Information aggregation and communication in committees
    1. D Austen-Smith
    2. TJ Feddersen
    (2009)
    Philosophical Transactions of the Royal Society B: Biological Sciences 364:763–769.
    https://doi.org/10.1098/rstb.2008.0256
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
    Group decisions in humans and animals: a survey
    1. L Conradt
    2. C List
    (2009)
    Philosophical Transactions of the Royal Society B: Biological Sciences 364:719–742.
    https://doi.org/10.1098/rstb.2008.0276
  10. 10
  11. 11
  12. 12
    Maximum likelihood in cost-sensitive learning: model specification, approximations, and upper bounds
    1. JP Dmochowski
    2. P Sajda
    3. LC Parra
    (2010)
    Journal of Machine Learning Research 11:3313–3332.
  13. 13
  14. 14
    Signal Detection Theory and Psychophysics
    1. DM Green
    2. JA Swets
    (1966)
    New York, United States: Wiley.
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
    MAGKS Papers on Economics
    1. R Kirstein
    2. G von Wangenheim
    (2010)
    A Generalized Condorcet Jury Theorem with Two Independent Probabilities of Error, MAGKS Papers on Economics, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics.
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
    Rescuing collective wisdom when the average group opinion is wrong
    1. A Laan
    2. G Madirolas
    3. GG de Polavieja
    (2017)
    Frontiers in Robotics and AI, 4, 10.3389/frobt.2017.00056.
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
    Collective decision-making in microbes
    1. A Ross-Gillespie
    2. R Kümmerli
    (2014)
    Frontiers in Microbiology, 5, 10.3389/fmicb.2014.00054, 24624121.
  39. 39
    Ensemble learning: a survey
    1. O Sagi
    2. L Rokach
    (2018)
    Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8:e1249.
    https://doi.org/10.1002/widm.1249
  40. 40
  41. 41
    Cognition, Evolution, and Behavior
    1. SJ Shettleworth
    (2010)
    Oxford University Press.
  42. 42
  43. 43
  44. 44
    Quorum responses and consensus decision making
    1. DJT Sumpter
    2. SC Pratt
    (2009)
    Philosophical Transactions of the Royal Society B: Biological Sciences 364:743–753.
    https://doi.org/10.1098/rstb.2008.0204
  45. 45
    The Wisdom of Crowds
    1. J Surowiecki
    (2005)
    Random House LLC.
  46. 46
  47. 47
  48. 48
  49. 49
  50. 50
  51. 51
  52. 52
  53. 53

Decision letter

  1. Andrew James King
    Reviewing Editor; Swansea University, United Kingdom
  2. Joshua I Gold
    Senior Editor; University of Pennsylvania, United States

In the interests of transparency, eLife includes the editorial decision letter, peer reviews, and accompanying author responses.

[Editorial note: This article has been through an editorial process in which the authors decide how to respond to the issues raised during peer review. The Reviewing Editor's assessment is that all the issues have been addressed.]

Thank you for submitting your article "Quorums enable optimal pooling of independent judgements" for consideration by eLife. Your article has been reviewed by two peer reviewers, and the evaluation has been overseen by a guest Reviewing Editor and Joshua Gold as the Senior Editor. One of the two reviewers has agreed to reveal his identity: Gonzalo G de Polavieja (Reviewer #2).

The Reviewing Editor has highlighted the concerns that require revision and/or responses, and we have included the separate reviews below for your consideration. If you have any questions, please do not hesitate to contact us.

Summary:

This is a well written and interesting contribution to our understanding of the mechanisms underlying collective decisions. The manuscript is set out to explain the error of our ways when using Condorcet's theory to inform our understanding, and present refinements of the model to broaden its applicability and use in biological and human sciences. I (and the two reviewers) support the publication of this work, but would all like to see the authors consider a number of points during the revision process. Below, I outline these main issues that have arisen both as a result of the initial reviews and our interactive discussion.

Concerns:

1) To "explain the error of our ways" is fine, expect some (if not most) of the theory and empirical work cited acknowledge that the Condorcet's theory represents a very simplified view of the world, and more complex models/mechanisms can be employed. We would like to see the authors 'soften' the critiques of earlier work so to better represent the advance being made here (e.g. paragraph 1 and 2, Discussion section) and ensure researchers in the field take forward and use the results presented.

2) Regarding earlier work, reviewer #1 points to an earlier model that presents a similar model and conclusion to that presented here (Ben-Yashar and Nitzan, 1997). As a behavioural ecologist who has interest in these models, I was unaware of it, and therefore many in the field may be in the same position. As the work is presented in a narrative form,… "first we did this, next we tested, etc." it would be useful to weave into the narrative how this work is similar or different to what is being presented, as explained nicely by reviewer #1. Moreover, the 1970s through 1990s saw a lot of theoretical extensions to Condorcet, and today, there's a lot of work studying the effect of social influence, correlations, heterogeneity, emergent sensing, etc. on collective wisdom, which are all deviations from Condorcet. This should be more explicitly acknowledged early in the manuscript.

3) Related to points 1 and 2, the paragraph setting out the three key interpretations (Introduction section) requires citations to back up each of the statements. I can think of one of my own papers that would be appropriate to cite, but how many more? The authors are suggesting we have all been misled, so we need to have a good number of citations here to back up this claim.

4) The definition of 'optimal' and its relation to accuracy in decision-making (here, related to probability of correct decision) needs to be explained properly in the Introduction, or else a different more representative term used (accurate/accuracy?).

5) Non-independence of errors is probably the rule rather than the exception in nature, as well as in lab experiments with animals or humans. Both reviewers query if and how this will effect either the construction of "state-of-the-art" models like this one, or their interpretation and generality – if not explicitly incorporated. At very least, this should be discussed.

6) Reviewer #2 thinks a Bayesian analysis would be as or more insightful than the signal detection theory, then it is likely so will others. Therefore, I would encourage Authors to present an argument against this in the discussion (where the use of signal detection theory is highlighted: paragraph three, Discussion section), and/or consider their recommendation to work out a general solution/expression using Bayesian analysis.

7) The Abstract should begin with one or two sentences giving a broader background. At present, it begins with very specific background regarding the Condorcet Jury Theorem. Also, the biological system should be indicated in the title and/or Abstract: please revise the title and/or Abstract with this advice in mind. A simple solution would be to could add "in biological systems" (or similar) to the title and then explain in the Abstract that the theory/models apply to all sorts of 'social' organisms.

Separate reviews (please respond to each point):

Reviewer #1:

In this manuscript, the authors draw attention to two ideas that are under-appreciated in the collective decision-making making literature: 1) that there are two errors that can be made in a binary decision (false positives and false negatives), rather than the single error (correct/incorrect), that is the focus of many previous studies, and 2) that there exists a family of collective decision rules, within which simple majority rule is a special case. The authors generalize Condorcet's jury theorem to allow for different false positive and false negative rates and show that for many scenarios, simple majority rule is not the optimal method to combine individual opinions. They identify three types of mostly different errors that can arise when naively applying the predictions of Condorcet's jury theorem, and show that an error will arise in the majority of parameter space. They demonstrate that a 'quorum' decision rule, defined as a threshold for making a decision is a fraction different from 50%, is optimal for these scenarios.

I agree with the authors that these two ideas need to be of greater theoretical and empirical research, and this manuscript does a good job of highlighting the importance of these ideas. The authors rightly show that in nature, the prior probabilities, and the costs/benefits, of different possible outcomes are generally not equal (there may rarely be a predator, but when there is one, the cost of being eaten is very high). They link the large literature on quorum decision-making to optimal decisions, demonstrating how quorum decisions may be optimal within the ecological scenarios in which species evolve, which is important.

My main concern with this manuscript is its novelty. In particular, there are two previously published papers (one which the authors cited and one which the authors did not), which together contain all of the main ideas and much of the results of the present manuscript. I describe these papers below, and their intersection with this manuscript, and then describe what I think are the novel sections remaining in this manuscript.

The first paper, which the authors did not cite, is: Ben-Yashar RC and Nitzan SI (1997) The optimal decision rule for fixed-size committees in dichotomous choice situations: the general result. International Economic Review, 38(1):175-186. The model in this paper incorporates the main ideas of the present manuscript, namely that the costs/benefits ('payoffs') may be different for different outcomes; the prior probability of different outcomes may be different; and the probabilities of false positives and false negatives may be different. The model described in Ben-Yashar and Nitzan, 1997, is essentially identical to the extended model described in the current manuscript. That paper also showed that 'quorum' rules are generally optimal, but rather than denote the region of parameter space in which this is the case, those authors went further and mathematically proved the exact value of the quorum threshold for each set of parameter values (they also further considered heterogeneous individuals, which was not considered in the current manuscript). In short, the optimal solution for the model appears to have been solved already.

The second paper, which the authors did cite, is: Kao and Couzin, 2014. This paper used a slightly different model but with some key similarities. In particular, the model described two cues in the environment, one which is independently perceived by each individual, and one which is globally perceived by all individuals. Although the authors did not describe the global cue as such, one interpretation of a global cue is as a prior probability of an outcome. By mapping the model in the current manuscript to the Kao and Couzin, 2014 model, one can find that the condition for "Error Ia" is the same as the boundary shown in Figure 1B of the Kao and Couzin paper. In the latter paper, the authors show that in this region of parameter space, the collective accuracy for very large groups approaches the accuracy of the global cue, which is the same result as Equation 5 and the left column of Figure 3 in the present manuscript.

To summarize, the model described in the present manuscript appears nearly identical to the one described in the Ben-Yashar and Nitzan paper, while the older paper gave a more general solution to than the present manuscript. Furthermore, one of the three errors presented in the present manuscript has been essentially described in the Kao and Couzin paper.

Despite all of this, I do think that there remain some novel results in the present manuscript. To my knowledge, Error Ib and Error II have not been previously described, although Error Ib is a subset of Error Ia. Furthermore, the way the authors parsed out the different errors and illustrated its prevalence in parameter space is useful. I also think that despite this model being solved already, the Ben-Yashar and Nitzan paper is not broadly known to the ecology/behavior community and drawing attention to it and its results is important. However, I think that the authors need to revise their manuscript in light of these two papers to highlight the aspects of their work that are truly novel.

Minor Comments:

I would suggest that the authors move most, or all, of the appendix to the main text, as it's not possible to understand the model or the results without reading this. For example, it's not clear why there is a trade-off between a+ and a- without knowing the details of the model (one may wonder why one can't simply set a+ and a- both to 1).

Reviewer #2:

The authors extend Condorcet analysis by taking into account false and negative errors instead of collapsing them into a single error. They work out the consequences of their analysis. I completely agree with the authors that the role of Condorcet theorem has been misinterpreted in the literature. It is a nice analytical model full of assumptions and we can learn a lot by realizing what is the role of these assumptions.

The author's key insight is that the world, for the agent, can be in different states, say + and -, with probabilities p and 1-p, and the error in each world state is different. In contrast, Condorcet's treatment is for a single world state.

There are two ideas I would consider:

1) I think a Bayesian analysis is actually as or more insightful than the signal detection theory. It naturally takes into account all types of errors and even correlations. You can also see in this treatment that you can do very well in supermajority or below majority cases.

2) In this respect, the analysis in the paper assumes errors to be independent. This is fine, Condorcet assumes independent voters. But this assumption does not hold in practice. So, again, I think the more general treatment is a Bayesian one including all types of errors and correlations (see Perez-Escudero and Polavieja, 2011 and more recently in Interface).

I think it would be nice to work it out from Bayes (maybe even more general using costs), obtain the general expression, and work out the signal-detection-theory case as a particular case.

As you are in this new experimental reviewing mode, you might try my suggestion (or not), and if it gets messy, leave it for another time (or not).

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Quorums enable optimal pooling of independent judgements in biological systems" for further consideration at eLife. Your revised article has been favorably evaluated by Joshua Gold (Senior Editor), a guest Reviewing Editor, and one reviewer.

Thank you very much for the time taken to carefully consider the reviewers and my own comments and revise your manuscript. I sent your revised manuscript to reviewer 1 to consider the changes you have made. Reviewer 1 is still not convinced about the novelty of your manuscript, and has suggested further revisions in relation to the Ben-Yashar and Nitzan paper, which we have been discussing previously. In particular, they point out that an analytical solution for the optimal quorum threshold is provided in this study, and therefore, provide a quantitative rather than qualitative solution. I tend to agree, and think that it is important that the value of this work is not dismissed out-of-hand (especially since many will not take the time to go back and read it). Reviewer 1 suggests the Ben-Yashar and Nitzan paper deserves a more prominent role. You can and should follow this suggestion whilst emphasising your own advances (e.g., using signal detection theory).

Reviewer 1 also provides further details of a misunderstanding with respect to the Kao and Couzin model which you will need to address, since the description of this model provided by Kao and Couzin may well be closer to your own work than is immediately obvious from reading the original work (indeed, it was not clear to me). Please ensure that this aspect of the Kao and Couzin is cited and discussed properly.

Reviewer 1 comments:

1) Regarding the Ben-Yashar and Nitzan, 1997, paper, the authors write in their response letter: "…while they are able to make qualitative discussions of the relationships between asymmetries and the resulting changes in quorum threshold, they cannot quantify this, or reason about interactions," and in their revised manuscript, "These results describe a qualitative relationship between prior asymmetry and optimal quorum threshold, and cost asymmetry and optimal quorum threshold. However, because they did not apply signal detection theory to optimise individual agents' decisions, they were unable quantitatively to uncover the complex nonlinear relationship between these three quantities".

In fact, theorem 3.1 of the Ben-Yashar and Nitzan, 1997, paper gives the exact, analytical result of what the optimal quorum threshold is, as a function of the type I and type II error probabilities (p_i^1 and p_i^2 in that equation), the prior probabilities (α in that equation), and the costs and benefits of the possible outcomes (B in that equation). Rather than a qualitative result, this is an exact, analytical solution that tells you the optimal quorum threshold as a function of all of the variables that the authors of the present manuscript are interested in (it even goes further to allow for individual differences in the probabilities of committing type I and II errors, which was not considered in the present manuscript).

Indeed, it seems that the results in the present manuscript are actually the more qualitative of the two, since most of their results are in the form of inequalities, compared to the exact solution given by Ben-Yashar and Nitzan. The authors themselves suggest as much when they write "…it is possible to optimise individual quorums [Ben-Yashar and Nitzan, 1997] rather than simply set them within a suitable interval as we do here…".

The characterization that the Ben-Yashar and Nitzan paper provided only a qualitative solution, and the present manuscript a quantitative solution, therefore seems incorrect. One could in fact make an argument that the older paper is more precise, and goes further than the present one in several ways. It seems unfair to me for that paper to be mentioned for the first time only in paragraph one of the Discussion section (briefly), and then discussed in paragraph three in a couple of sentences. I feel that paper deserves a much more prominent role in the current manuscript, with its contributions, and the present manuscript's contributions (e.g., using signal detection theory, emphasizing certain errors), accurately characterized and compared.

2) The point regarding the Kao and Couzin, 2014 paper is of much lower importance than the above point, but since the authors disagreed with my point, I'd just like to clarify it, as I do think that there's a mapping between the two models. First, the authors mistakenly refer to one of the cues as the low 'accuracy' cue (and also refer to it as such in the manuscript in paragraph one of the Discussion), when it is actually a low 'correlation' cue -- this should be corrected. Furthermore, in the Kao and Couzin model, the probability rH that the high correlation cue gives correct information can be mapped to the prior probability p of being in state + in the present manuscript, since the state of the high correlation cue is a global one that affects all of the individuals equally. Then, if the high correlation cue gives 'correct' information, the probability that an individual selects the correct option (a+ in the current notation) is given by q*rL + (1-q) (in the notation of the Kao and Couzin paper, where I've changed p to q for clarity). On the other hand, if the high correlation cue gives incorrect information (a- in the current notation), then an individual selects the correct option with probability q*rL. Plugging these quantities into Equation 3 of the present manuscript gives rH*(1-q) + q*rL > 1/2, which must be true if rL > 1/2 and rH > 1/2, as was assumed in the Kao and Couzin paper (but q can vary from 0 to 1). The condition a+ > 1/2 then maps to q*(1-rL) < 1/2, which again must be true if rL > 1/2. Finally, the condition a- < 1/2 maps to q < 1/(2*rL), which is the condition shown in Figure 1B of the Kao and Couzin paper.

So the Error Ia condition does in fact map onto the boundary shown in Figure 1B of that paper. I concede that it's not a very obvious mapping, but it is there nonetheless. Indeed, the 'voting strategy' described in the Kao and Couzin paper could be interpreted as a 'soft,' probabilistic, quorum threshold, performed at the individual level, in contrast to the hard quorum threshold performed at a group level in the present manuscript.

https://doi.org/10.7554/eLife.40368.011

Author response

Concerns:

1) To "explain the error of our ways" is fine, expect some (if not most) of the theory and empirical work cited acknowledge that the Condorcet's theory represents a very simplified view of the world, and more complex models/mechanisms can be employed. We would like to see the authors 'soften' the critiques of earlier work so to better represent the advance being made here (e.g. paragraph one and two, Discussion section) and ensure researchers in the field take forward and use the results presented.

We have attempted to soften the language in some places in the passage highlighted by the editor, but how this reads will of course be subjective. However we have been able to acknowledge a positive instance of considering error asymmetry in the behavioural ecology literature, thanks to reviewer 1’s comments as discussed in the editor’s point 2 below, as well as a positive instance of observing group accuracy converging to intermediate values, thanks also to reviewer 1’s observations.

2) Regarding earlier work, reviewer #1 points to an earlier model that presents a similar model and conclusion to that presented here (Ben-Yashar and Nitzan, 1997). As a behavioural ecologist who has interest in these models, I was unaware of it, and therefore many in the field may be in the same position. As the work is presented in a narrative form,… "first we did this, next we tested, etc." it would be useful to weave into the narrative how this work is similar or different to what is being presented, as explained nicely by reviewer #1. Moreover, the 1970s through 1990s saw a lot of theoretical extensions to Condorcet, and today, there's a lot of work studying the effect of social influence, correlations, heterogeneity, emergent sensing, etc. on collective wisdom, which are all deviations from Condorcet. This should be more explicitly acknowledged early in the manuscript.

We agree on the importance of the paper the reviewer has very helpfully pointed us towards, which we were unaware of. We have rewritten portions of the paper to clarify the relationship of our results to those of Ben-Yashar & Nitzan. Briefly, while Ben-Yashar & Nitzan indeed present the general solution and note the relationship between asymmetries of payoffs and priors, and interestingly unify confidence weighting with quorum rules, they do not apply signal detection theory to model individual decision-makers (beyond assuming the basic relationship between true positive and false-positive rates). Because of this, while they are able to make qualitative discussions of the relationships between asymmetries and the resulting changes in quorum threshold, they cannot quantify this, or reason about interactions. Our quantitative approach does enable this and reveals such interactions to be non-linear (Figure 2), however, and is, based on our application of the mathematics of signal detection theory, our primary contribution. Awareness of this paper has also helped us appreciate where existing behavioural ecology research has realised the importance of quorums in this situation, and acknowledge it appropriately, enabling us better to respond to the editor’s point 1, as described above.

We do however feel we explicitly acknowledge existing work on generalising Condorcet’s result, citing a number of relevant papers in the following passage:

“Over the past few decades, substantial research effort has focussed on two key explicit assumptions underlying Condorcet’s Jury Theorem, independence (i.e. judgments/votes by different members of the group are assumed to be independent from each other) and homogeneity (i.e. all decision-makers within a group are assumed to be identical, both in competence and in goals) [12, 16, 22-25].”

3) Related to points 1 and 2, the paragraph setting out the three key interpretations (Introduction section) requires citations to back up each of the statements. I can think of one of my own papers that would be appropriate to cite, but how many more? The authors are suggesting we have all been misled, so we need to have a good number of citations here to back up this claim.

We have included these references as requested.

4) The definition of 'optimal' and its relation to accuracy in decision-making (here, related to probability of correct decision) needs to be explained properly in the Introduction, or else a different more representative term used (accurate/accuracy?).

In the first occurrence of optimal outside the Abstract we have clarified our definition. In the Discussion we have also clarified that it is possible to optimise the quorum threshold used, and explained why that is not our approach here.

5) Non-independence of errors is probably the rule rather than the exception in nature, as well as in lab experiments with animals or humans. Both reviewers query if and how this will effect either the construction of "state-of-the-art" models like this one, or their interpretation and generality – if not explicitly incorporated. At very least, this should be discussed.

We have addressed this in the closing Discussion, in also introducing Bayesianism as an approach (editor’s point 6 and reviewer 2’s points below).

6) Reviewer #2 thinks a Bayesian analysis would be as or more insightful than the signal detection theory, then it is likely so will others. Therefore, I would encourage Authors to present an argument against this in the discussion (where the use of signal detection theory is highlighted: paragraph three, Discussion section), and/or consider their recommendation to work out a general solution/expression using Bayesian analysis.

We now cite such approaches in the Discussion and argue that our approach has the merit of simplicity, and revealing interesting structure of collective decisions even with very stringent assumptions. We feel that conducting such an analysis in this revision would result in a substantially different manuscript, so are grateful for the editor’s and reviewer 2’s willingness for us to justify not doing this. We are, however, committed Bayesians and hope future work may apply this approach further to the scenario we have considered here.

7) The Abstract should begin with one or two sentences giving a broader background. At present, it begins with very specific background regarding the Condorcet Jury Theorem. Also, the biological system should be indicated in the title and/or Abstract: please revise the title and/or Abstract with this advice in mind. A simple solution would be to could add "in biological systems" (or similar) to the title and then explain in the Abstract that the theory/models apply to all sorts of 'social' organisms.

Unfortunately the existing Abstract gives us only 6 more words to work with; feeling unable to jettison any of the existing Abstract text we have added “Collective decision-making is ubiquitous” to the beginning of the Abstract – we have also added ‘in biological systems’ to the title, as suggested.

Separate reviews (please respond to each point):

Reviewer #1:

[…] I agree with the authors that these two ideas need to be of greater theoretical and empirical research, and this manuscript does a good job of highlighting the importance of these ideas. The authors rightly show that in nature, the prior probabilities, and the costs/benefits, of different possible outcomes are generally not equal (there may rarely be a predator, but when there is one, the cost of being eaten is very high). They link the large literature on quorum decision-making to optimal decisions, demonstrating how quorum decisions may be optimal within the ecological scenarios in which species evolve, which is important.

My main concern with this manuscript is its novelty. In particular, there are two previously published papers (one which the authors cited and one which the authors did not), which together contain all of the main ideas and much of the results of the present manuscript. I describe these papers below, and their intersection with this manuscript, and then describe what I think are the novel sections remaining in this manuscript.

The first paper, which the authors did not cite, is: Ben-Yashar RC and Nitzan SI (1997) The optimal decision rule for fixed-size committees in dichotomous choice situations: the general result. International Economic Review, 38(1):175-186. The model in this paper incorporates the main ideas of the present manuscript, namely that the costs/benefits ('payoffs') may be different for different outcomes; the prior probability of different outcomes may be different; and the probabilities of false positives and false negatives may be different. The model described in Ben-Yashar and Nitzan, 1997, is essentially identical to the extended model described in the current manuscript. That paper also showed that 'quorum' rules are generally optimal, but rather than denote the region of parameter space in which this is the case, those authors went further and mathematically proved the exact value of the quorum threshold for each set of parameter values (they also further considered heterogeneous individuals, which was not considered in the current manuscript). In short, the optimal solution for the model appears to have been solved already.

We thank the reviewer for drawing this very important paper to our attention, which we now cite and discuss as described in our response to the editor’s point 2 above; in that response we also discuss how our results are still novel in this revised context.

The second paper, which the authors did cite, is: Kao and Couzin, 2014. This paper used a slightly different model but with some key similarities. In particular, the model described two cues in the environment, one which is independently perceived by each individual, and one which is globally perceived by all individuals. Although the authors did not describe the global cue as such, one interpretation of a global cue is as a prior probability of an outcome. By mapping the model in the current manuscript to the Kao and Couzin, 2014 model, one can find that the condition for "Error Ia" is the same as the boundary shown in Figure 1B of the Kao and Couzin paper. In the latter paper, the authors show that in this region of parameter space, the collective accuracy for very large groups approaches the accuracy of the global cue, which is the same result as Equation 5 and the left column of Figure 3 in the present manuscript.

Again, we thank the reviewer for drawing this aspect of the paper we cited to our attention; however we respectfully disagree that the result maps onto our Error Ia; in Kao and Couzin’s model individuals attend to one cue or another with a probability determined by their individual strategy — we do not see a formal analogy with our model in this regard, and feel that the nature of Figure 1B compared to our Figure 2A, in which plot axes represent different quantities, highlights the difference between the results — we also do not see that the low accuracy cue in Kao and Couzin is analogous to the prior in our model; in Kao and Couzin individuals have a propensity to attend to the low accuracy cue, whereas in our model the prior leads to a shift in the optimal quorum threshold. However we do agree that the reason for Kao and Couzin’s result in their Figure 1B, and our error Ia, are the same, and now cite their paper in that additional context as well as comparing the two models in our revised Discussion.

To summarize, the model described in the present manuscript appears nearly identical to the one described in the Ben-Yashar and Nitzan paper, while the older paper gave a more general solution to than the present manuscript. Furthermore, one of the three errors presented in the present manuscript has been essentially described in the Kao and Couzin paper.

Despite all of this, I do think that there remain some novel results in the present manuscript. To my knowledge, Error Ib and Error II have not been previously described, although Error Ib is a subset of Error Ia. Furthermore, the way the authors parsed out the different errors and illustrated its prevalence in parameter space is useful. I also think that despite this model being solved already, the Ben-Yashar and Nitzan paper is not broadly known to the ecology/behavior community and drawing attention to it and its results is important. However, I think that the authors need to revise their manuscript in light of these two papers to highlight the aspects of their work that are truly novel.

Once again we thank the reviewer for pointing out these links to pre-existing work, and for the reasons discussed above we still consider that our results are substantially novel, in which we are in agreement with the reviewer we believe.

Minor Comments:

I would suggest that the authors move most, or all, of the appendix to the main text, as it's not possible to understand the model or the results without reading this. For example, it's not clear why there is a trade-off between a+ and a- without knowing the details of the model (one may wonder why one can't simply set a+ and a- both to 1).

We didn’t feel this would aid the flow of the manuscript, as it would represent a substantial digression at that point in the manuscript; when the appendix is first discussed (before Equation 2) it is made clear that the appendix explains how pairs of a+, a- values are determined. Additionally, for some readers the signal detection theory appendix may be redundant, as signal detection theory is a staple of many behavioural ecology textbooks, such as the one we cite in our manuscript.

Reviewer #2:

The authors extend Condorcet analysis by taking into account false and negative errors instead of collapsing them into a single error. They work out the consequences of their analysis. I completely agree with the authors that the role of Condorcet theorem has been misinterpreted in the literature. It is a nice analytical model full of assumptions and we can learn a lot by realizing what is the role of these assumptions.

The author's key insight is that the world, for the agent, can be in different states, say + and -, with probabilities p and 1-p, and the error in each world state is different. In contrast, Condorcet's treatment is for a single world state.

There are two ideas I would consider:

1) I think a Bayesian analysis is actually as or more insightful than the signal detection theory. It naturally takes into account all types of errors and even correlations. You can also see in this treatment that you can do very well in supermajority or below majority cases.

We simultaneously agree and disagree — in our revised Discussion we now justify the utility of our simpler analysis, with more assumptions, as described in our response to the editor’s point 6.

2) In this respect, the analysis in the paper assumes errors to be independent. This is fine, Condorcet assumes independent voters. But this assumption does not hold in practise. So, again, I think the more general treatment is a Bayesian one including all types of errors and correlations (see Perez-Escudero and Polavieja, 2011 and more recently in Interface).

I think it would be nice to work it out from Bayes (maybe even more general using costs), obtain the general expression, and work out the signal-detection-theory case as a particular case.

As you are in this new experimental reviewing mode, you might try my suggestion (or not), and if it gets messy, leave it for another time (or not).

We thank the reviewer for noting that this may be more work than is appropriate for a manuscript revision. In our opinion this would substantially change the nature of our manuscript, and therefore now highlight the interest in following a Bayesian approach in future work, as well as explaining the interest in our simpler approach.

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Reviewer 1 comments:

1) Regarding the Ben-Yashar and Nitzan, 1997, paper, the authors write in their response letter: "…while they are able to make qualitative discussions of the relationships between asymmetries and the resulting changes in quorum threshold, they cannot quantify this, or reason about interactions," and in their revised manuscript, "These results describe a qualitative relationship between prior asymmetry and optimal quorum threshold, and cost asymmetry and optimal quorum threshold. However, because they did not apply signal detection theory to optimise individual agents' decisions, they were unable quantitatively to uncover the complex nonlinear relationship between these three quantities".

In fact, theorem 3.1 of the Ben-Yashar and Nitzan, 1997, paper gives the exact, analytical result of what the optimal quorum threshold is, as a function of the type I and type II error probabilities (p_i^1 and p_i^2 in that equation), the prior probabilities (α in that equation), and the costs and benefits of the possible outcomes (B in that equation). Rather than a qualitative result, this is an exact, analytical solution that tells you the optimal quorum threshold as a function of all of the variables that the authors of the present manuscript are interested in (it even goes further to allow for individual differences in the probabilities of committing type I and II errors, which was not considered in the present manuscript).

Indeed, it seems that the results in the present manuscript are actually the more qualitative of the two, since most of their results are in the form of inequalities, compared to the exact solution given by Ben-Yashar and Nitzan. The authors themselves suggest as much when they write "…it is possible to optimise individual quorums [Ben-Yashar and Nitzan, 1997] rather than simply set them within a suitable interval as we do here…".

The characterization that the Ben-Yashar and Nitzan paper provided only a qualitative solution, and the present manuscript a quantitative solution, therefore seems incorrect. One could in fact make an argument that the older paper is more precise, and goes further than the present one in several ways. It seems unfair to me for that paper to be mentioned for the first time only in paragraph one of the Discussion section (briefly), and then discussed in paragraph three in a couple of sentences. I feel that paper deserves a much more prominent role in the current manuscript, with its contributions, and the present manuscript's contributions (e.g., using signal detection theory, emphasizing certain errors), accurately characterized and compared.

We thank the reviewer for their further comments on the relationship between Ben-Yashar and Nitzan’s work, and our own, and thank them again for highlighting this very important paper in their initial review. We fully agree with the reviewer’s suggestion that the work be cited and discussed earlier in the text given its relevance and we now do exactly this in the Introduction of our paper. Also, as a response to the reviewer’s comment, we further adjusted our discussion of Ben-Yashar and Nitzan in the Discussion section of our paper – in particular, we followed the reviewers suggestion and again highlight the importance of Ben-Yashar and Nitzan and do not refer to their results as qualitative any more. To summarise our argument here, since Ben-Yashar and Nitzan did not relate true positive rate and false positive rate to prior and cost asymmetries, they were unable to quantitatively relate optimal quorum threshold to use of a super or sub-majority quorum, as we do here for the first time.

2) The point regarding the Kao and Couzin, 2014 paper is of much lower importance than the above point, but since the authors disagreed with my point, I'd just like to clarify it, as I do think that there's a mapping between the two models. First, the authors mistakenly refer to one of the cues as the low 'accuracy' cue (and also refer to it as such in the manuscript in paragraph one of the Discussion), when it is actually a low 'correlation' cue -- this should be corrected. Furthermore, in the Kao and Couzin model, the probability rH that the high correlation cue gives correct information can be mapped to the prior probability p of being in state + in the present manuscript, since the state of the high correlation cue is a global one that affects all of the individuals equally. Then, if the high correlation cue gives 'correct' information, the probability that an individual selects the correct option (a+ in the current notation) is given by q*rL + (1-q) (in the notation of the Kao and Couzin paper, where I've changed p to q for clarity). On the other hand, if the high correlation cue gives incorrect information (a- in the current notation), then an individual selects the correct option with probability q*rL. Plugging these quantities into Equation 3 of the present manuscript gives rH*(1-q) + q*rL > 1/2, which must be true if rL > 1/2 and rH > 1/2, as was assumed in the Kao and Couzin paper (but q can vary from 0 to 1). The condition a+ > 1/2 then maps to q*(1-rL) < 1/2, which again must be true if rL > 1/2. Finally, the condition a- < 1/2 maps to q < 1/(2*rL), which is the condition shown in Figure 1B of the Kao and Couzin paper.

So the Error Ia condition does in fact map onto the boundary shown in Figure 1B of that paper. I concede that it's not a very obvious mapping, but it is there nonetheless. Indeed, the 'voting strategy' described in the Kao and Couzin paper could be interpreted as a 'soft,' probabilistic, quorum threshold, performed at the individual level, in contrast to the hard quorum threshold performed at a group level in the present manuscript.

We thank the reviewer for pointing out our mislabelling of Kao and Couzin’s variables and corrected this error, that is, the cue that we mistakenly labeled as “low accuracy cue” is now labeled as “low correlation cue”. We also thank the reviewer for further explaining their reasoning about analogies between our analysis, and that of Kao and Couzin. However we maintain our position that while the results have the same mechanism at their heart, they are not equivalent. The fundamental question is (from the initial review) whether “by mapping the model in the current manuscript to the Kao and Couzin, 2014, model, one can find that the condition for 'Error Ia' is the same as the boundary shown in Figure 1B of the Kao and Couzin paper". As we argued before, a high correlation cue, despite being accessible to all individuals in Kao and Couzin’s model, is not equivalent to a prior – individuals do not directly choose to attend to a prior or not, whereas they do in Kao and Couzin. In response, the reviewer claims to have shown how our inequality 3 can explain their (Kao and Couzin’s) boundary in Figure 1B, through deriving our quantities a_+ and a_- in terms of the their variables. That the two results are not the same can be seen most easily by observing that a_+ and a_- are related to each other by a simple linear relationship according to the reviewer’s derivation, which is a function of the individual agent’s strategy and a simple accuracy (i.e. neglecting the possibility of two error types, which is fundamental in our analysis); in contrast, in the result presented in the manuscript these quantities are independent, bar the simple relationship a_+ > 1-a_-. Therefore Kao and Couzin’s condition might best be seen as an instantiation of our condition in a special case. However, even then we dispute the equivalence since, as discussed above, the two decision problems are quite different. While we do not disagree with the reviewer’s symbol manipulation, it is not surprising if models of different decision problems give comparable mathematical expressions when dealing with expectations, and sharing a few similarities such as majority rule. We therefore maintain that the reason for Kao and Couzin’s result is the same as the reason for ours, as we already wrote in the revised manuscript, and as the reviewer writes in their subsequent comments. However we feel that providing further argumentation about perceived similarities between the results is not illuminating and in fact risks confusion. Therefore we have not revised our manuscript further in this regard, except to correct the aforementioned error in labelling Kao and Couzin’s variables.

https://doi.org/10.7554/eLife.40368.012

Article and author information

Author details

  1. James AR Marshall

    Department of Computer Science, University of Sheffield, Sheffield, United Kingdom
    Contribution
    Conceptualization, Formal analysis, Writing—original draft, Writing—review and editing
    For correspondence
    james.marshall@sheffield.ac.uk
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-1506-167X
  2. Ralf HJM Kurvers

    Centre for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
    Contribution
    Conceptualization, Writing—review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-3460-0392
  3. Jens Krause

    Department of Fish Behavior and Ecology, Leibniz-Institute of Freshwater Ecology and Inland Fisheries, Berlin, Germany
    Contribution
    Conceptualization, Writing—review and editing
    Competing interests
    No competing interests declared
  4. Max Wolf

    Department of Fish Behavior and Ecology, Leibniz-Institute of Freshwater Ecology and Inland Fisheries, Berlin, Germany
    Contribution
    Conceptualization, Formal analysis, Writing—original draft, Writing—review and editing
    For correspondence
    m.wolf@igb-berlin.de
    Competing interests
    No competing interests declared

Funding

H2020 European Research Council (647704)

  • James AR Marshall

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

We thank Gavin Brown and Nikolaos Nikolaou for helpful discussions on ensemble learning theory, and Andrew King, Gonzalo de Polavieja and an anonymous reviewer for helpful comments during the review process. JARM was funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement number 647704).

Senior Editor

  1. Joshua I Gold, University of Pennsylvania, United States

Reviewing Editor

  1. Andrew James King, Swansea University, United Kingdom

Publication history

  1. Received: July 24, 2018
  2. Accepted: January 8, 2019
  3. Version of Record published: February 13, 2019 (version 1)

Copyright

© 2019, Marshall et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 616
    Page views
  • 85
    Downloads
  • 1
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)