Coarse population coding has been widely explored in motor systems, where neurons show broad activity profiles and are thought to ‘vote’ for the movement typically associated with their peak activity (e.g., Georgopoulos et al., 1986; Lee et al., 1988). However, individual motor systems only generate one movement at a time. Such a coarse coding/population voting scheme cannot work in sensory systems where there are generally many stimuli to be represented rather than a single (e.g., arm or eye) movement to be specified. It has been assumed that sensory receptive fields are small enough that coarse coding does not apply, but this seems questionable. For example, the letters on the page you are reading now are probably <0.25° apart, but foveal V1 receptive fields are approximately 0.5–2° in diameter (Dow et al., 1981; Alonso and Chen, 2009; Xing et al., 2009; Dubey and Ray, 2016; Keliris et al., 2019). Receptive fields get even larger at later stages along the visual processing stream (e.g., Alonso and Chen, 2009). In the auditory system, mammalian neurons may be responsive to nearly any location in space (e.g., Woods et al., 2001; Groh et al., 2003; McAlpine and Grothe, 2003; Werner-Reiss and Groh, 2008; Grothe et al., 2010; Higgins et al., 2010) and even frequency tuning is broad at conversational sound levels (Bulkin and Groh, 2011; Willett and Groh, 2022). Such breadth of tuning means that there can be overlap in the population of neurons activated by individual stimuli, making it unclear how information about multiple objects is preserved.

Logic suggests that information about each distinct stimulus must be segregated within the neural code in some fashion, either into exclusive neural subpopulations, different epochs of time, or some combination of both. We have recently presented the hypothesis that the nervous system might employ a form of neural turn-taking (time division multiplexing) in which individual neurons fluctuate between responding to each of the items in or near their receptive fields across various epochs of time (Caruso et al., 2018; Mohl et al., 2020). Such a coding scheme could preserve information about each stimulus across time and/or across neural subpopulations.

This theory raises three key open questions. First, is multiplexing a general phenomenon that occurs across a range of different brain areas? Our original study tested one subcortical auditory area (the inferior colliculus) and one extrastriate visual cortical area (area MF of the inferotemporal [IT] face patch system). More areas need to be tested to understand how such a coding scheme might operate. Second, in brain areas that exhibit such coding fluctuations, do neurons fluctuate together, and if so, how? Pairs of neurons might show positive, negative, or no correlations with each other. The pattern of such correlations across the population can reveal whether the population as a whole retains information about both stimuli and whether there is bias favoring one stimulus over another.

Third, does the pattern of fluctuations depend on the parsing of the scene into separate objects? Individual stimuli can fuse into one object or be perceived as distinct from each other. Stimuli that segregate into separate objects may be more likely to be associated with fluctuations in neural activity and their attendant correlations across neurons (Milner, 1974; Gray and Singer, 1989; Von Der Malsburg, 1994; Singer and Gray, 1995; Gray, 1999) (but see Palanca and DeAngelis, 2005), whereas stimuli that fuse into a single distinct object may cause activity patterns that are akin to those observed when only one stimulus is present. Such a pattern would specifically implicate activity fluctuations in playing a role in the perceptual process of object segregation.

To address these questions, we turned to the primary visual cortex (V1). V1 allows for a strong test of these hypotheses since V1 neurons have comparatively small receptive fields and are therefore less subject to the multiple-stimulus-overlap problem than the more broadly tuned areas such as the inferior colliculus or inferotemporal (IT) cortex that were assessed in our previous report (Caruso et al., 2018). Even though V1 neurons themselves have comparatively small receptive fields, V1 contributes to processing in higher cortical areas where spatial tuning is coarser. V1 could therefore also exhibit fluctuating activity patterns so as to facilitate preservation of information about multiple stimuli at higher stages.

We evaluated activity in V1 while monkeys viewed either individual stimuli (gratings) or two different types of combined stimuli (superimposed vs. adjacent gratings). When the two gratings were superimposed, they presumably appeared as one fused object, or plaid (Adelson and Movshon, 1982; Rodman and Albright, 1989; Heeger et al., 1996; Busse et al., 2009; Lima et al., 2010), whereas when they were adjacent they appeared as two distinct objects. We found evidence for coding fluctuations when two gratings were present at separate locations (two objects) but not when the gratings were superimposed at the same location and appeared as one fused object. We then evaluated the degree and sign of the spike count correlations (commonly referred to as ‘noise’ correlations; Cohen and Kohn, 2011) observed between pairs of simultaneously recorded units in response to presentations of particular stimulus conditions. We found that the pattern of correlations varied dramatically depending on whether the stimuli were presented either individually or superimposed (single stimuli or one fused object) vs. when they were presented side-by-side (two separate objects). In the two-object case, the distribution of spike count correlations was markedly different from previous reports involving individual stimuli (Table 1, Cohen and Kohn, 2011), and encompassed a range spanning many negative correlations in addition to positive ones. Whether the correlations tended to be positive vs. negative depended on whether the two neurons in the pair preferred the same stimulus (median correlation + 0.25) or preferred different stimuli (median correlation –0.05). The distribution of spike count correlation values was even more widely spread among pairs of neurons that showed demonstrably fluctuating activity across stimulus presentations (same preference: +0.49 and different preference: –0.14). In contrast, in the single stimuli and fused object (superimposed gratings) cases, positive correlations predominated (single stimuli: median value 0.15–0.19; fused object: median value +0.15). Distinct tuning-preference-related distributions of spike count correlations for adjacent stimuli but not for superimposed/fused stimuli were also seen in a smaller additional dataset in V4.

Overall, this pattern of results is consistent with the possibility that when two visual objects are presented in close proximity, a subpopulation of visual cortical neurons fluctuates in a coordinated fashion, generally retaining information about segregated objects and suggesting an account for why they can be perceived at once.

The central observations in this article are twofold. First, we identified fluctuating activity patterns in V1, evoked only by combinations of stimuli that are parsed as separate objects. These fluctuations were formally identified using a statistical analysis method benchmarked to the response patterns evoked by each of the stimuli independently (Caruso et al., 2018; Mohl et al., 2020; Glynn et al., 2021). This finding suggests not only that multiplexing of information may be a general characteristic of sensory signals in the brain, but also implicates it in the process of separating vs. grouping of stimuli into objects.

These findings laid the groundwork for our second major question, how fluctuating activity patterns are coordinated across neurons and the implications for coding of stimuli at the population level. We found patterns of spike count correlations that differed substantially from those observed previously, but only when two objects were presented. Single objects (whether individual gratings or two superimposed gratings) yielded correlation patterns very similar to previous reports in the literature (Cohen and Kohn, 2011; Ruff et al., 2016; Ruff and Cohen, 2016), and the correlations did not greatly depend on whether the two units in the pair preferred the same individual stimulus or different ones. In contrast, when two stimuli were presented adjacent to one another other, two distinct distributions emerged based on whether the two units in the pair preferred the same (congruent) individual stimulus (associated with generally positive spike count correlations) vs. different (incongruent) individual stimuli (associated with generally negative spike count correlations in V1 or simply less positive spike count correlations in V4). This pattern was observed in the population as a whole, but was especially pronounced in the subset of units that exhibited ‘mixture’-type response patterns indicating fluctuating across stimulus presentations between the response distributions associated with each of the individual stimuli.

We interpret these observations under the conceptual framework of the challenge that visual cortex faces when representing a visual scene that contains either individual stimuli, combinations of stimuli that bind to form one object, or combinations of stimuli that remain perceptually distinct from each other. The pattern of positive and negative (or less positive) correlations exhibited between pairs of such units is consistent with a population code biased toward one of the two stimuli on any given stimulus presentation, but that preserves information about the other stimulus as well.

It is interesting to note that we observed evidence of multiplexing each stimulus even in V1 where receptive fields are small and the stimuli we used did not themselves typically span more than one receptive field. Put another way, for most of these V1 ‘mixtures,’ the observed fluctuations involved responding vs. not responding rather than fluctuating between two different levels of responding. Thus, the coarseness of tuning did not necessarily pose a problem for the encoding of these particular stimuli in this particular brain area, and yet fluctuations were observed. Thus, the precision of V1’s spatial code may not be the limiting factor. Multiplexing is likely to have some as yet unknown characteristic spatial scale that may be determined by the coarsest tuning evident at any stage in the sensory pathway. Future work in which stimuli are systematically varied to manipulate the amount of overlap in the activity patterns evoked in different brain areas by each stimulus alone is needed to answer this question.

The constellation of our findings cannot be easily explained by any obvious alternative explanations. For example, could our focus on the activity patterns of multiunit clusters have impacted the results? If anything, this would be expected to work against the sensitivity of the analyses, if such clusters consisted of individual neurons who were behaving differently from one another. In fact, our findings were broadly similar in the subset of the data that involved well-isolated single units as compared to the full dataset involving multiunit activity. Furthermore, our previous study identifying coding fluctuations in the inferior colliculus and the MF face patch of inferotemporal cortex was conducted on well-isolated single units (Caruso et al., 2018). Thus, it seems unlikely that single vs. multiunit isolation significantly impacted our findings.

Could either microsaccades or small differences in the fixation position across trials have impacted the results? As noted earlier, we excluded trials with microsaccades, so such small eye movements are unlikely to have affected the findings, and fixational scatter did not vary by stimulus conditions (Figure 1—figure supplement 2). Thus, it is unlikely that variation in fixation position contributed to the difference we observed between two-object vs. fused-object response patterns or the differences between congruent-preference and incongruent-preference pairs of units. Furthermore, as noted above, we observed similar coding fluctuations in two brain areas (the IC and MF face patch) for auditory and large visual stimuli – that is, stimulus conditions that are thought to involve less sensitivity to differences in fixation position than V1 and V4 (Bremmer, 2000; Groh et al., 2001; Porter et al., 2006; Porter et al., 2007; Lehky et al., 2008; Maier and Groh, 2010; Bulkin and Groh, 2012a; Bulkin and Groh, 2012b; Merriam et al., 2013; Caruso et al., 2018).

Finally, there is precedent in the literature for differences in the spike count correlation patterns of congruent vs. incongruent-preference pairs: a previous study in V4 (Verhoef and Maunsell, 2017) also reported such correlation differences. The general effect size of our V4 results seems to be similar to theirs, particularly when considering the most comparable conditions. The Verhoef and Maunsell study was not designed to identify coding fluctuations, so the most comparable point of comparison in our study would be the pooled results across all model categories (Figure 8a). Our analysis focused on units with well-separated responses to the individual stimuli, which is most comparable to the right side of their Fig. 2C; our unit pairs were classified categorically as congruent- or incongruent-preferring rather than on a sliding scale, so it is not immediately apparent how to relate the two studies in that dimension (y axis in their Fig. 2C). Nevertheless, the approximate difference between the median or mean congruent-preferring vs. incongruent-preferring correlations is reassuringly similar at about 0.06 in our study and a maximum of about 0.12 in theirs. This suggests that the same-preference vs. different-preference correlation patterns observed in the two studies are likely to generalize across different experimental designs.

There has been a rich literature concerning the implications of correlated activity between visually responsive neurons in recent decades. One school of thought considers correlations in the context of the variability of neural firing. Under this ‘noise correlation’ view, positive correlations have historically been seen as detrimental for encoding information at the population level (Shadlen and Newsome, 1994; Zohary et al., 1994). Such views have also seen notable refinement and qualification since these early studies (Romo et al., 2003; Averbeck and Lee, 2004; Averbeck et al., 2006; Moreno-Bote et al., 2014; Kanitscheider et al., 2015; Kohn et al., 2016; Nogueira et al., 2020; Kafashan et al., 2021), including recent work noting that neural variability could be a signal reflecting stimulus uncertainty (Hénaff et al., 2020). Arguably closer to the current work is a different school of thought, the temporal correlation hypothesis (Milner, 1974; Gray and Singer, 1989; Von Der Malsburg, 1994; Singer and Gray, 1995; Gray, 1999). This theory focused on the need to connect the brain’s representation of different attributes of a given object together, and proposed that such binding might be mediated through precise synchrony of spikes among neurons responding to the same object. This view, then, sees correlated activity as both useful and specifically relevant to object vision. Studies exploring this hypothesis have, however, primarily focused on within-trial temporal synchrony of spikes on the order of milliseconds, whereas the noise correlation literature has focused on spike counts in the domain of hundreds of milliseconds and analysis at the level of the ensemble of trials or stimulus presentations. By evaluating spike count variation at the level of stimulus presentations and comparing the results as a function of the number of stimuli/objects, the present work forges a bridge between these two areas of the literature.

Our findings also suggest reconsideration of two other key processes in visual neuroscience: selective attention and normalization. Selective attention refers to the fact that perceptual awareness is not equal across all stimuli present in a sensory scene. Selective attention can be controlled through ‘top-down’ means, such as via tasks in which participants are cued to focus on one stimulus and ignore others. Indeed, the monkeys were performing just such a task in our adjacent V1 dataset, and thus were in theory ignoring the stimuli whose responses we studied here. But even with correct task performance, top-down control of attention is imperfect. Might the fluctuating responses we observed be due to covert shifts of attention from one of the supposedly unattended stimuli to the other (which could contribute to the observed activity patterns; Ecker et al., 2016; Engel et al., 2016; Denfield et al., 2018)? We think not. If this were the case, then the neurons should have fluctuated in more perfect harmony with one another. As shown quantitatively in Figure 6 and qualitatively in Figure 9, although there is a bias in which stimulus is ‘capturing’ the response patterns on individual trials, there remains a substantial portion of the neural population that is responding to the other stimulus. And the observed pattern of fluctuating activity is very different from a simulation of covert attention (Figure 9a vs. Figure 9d). While further work on this question is needed, we think it is worth noting that the patterns of activity that we observed can in principle support preservation of information about all stimuli in the scene. Processes involved in selective attention might contribute to the creation of biases within this representation or could act at later stages on the information preserved within these representations to enhance awareness of one or more of the represented stimuli.

Previous findings from the existing literature on attention and related areas are consistent with this new view, and could easily be evaluated anew using the approach we described here. Trial-averaged neural responses to attended and unattended stimuli can often be modeled as a weighted combination of the responses to those stimuli when presented alone (Boynton, 2009; Lee and Maunsell, 2009; Reynolds and Heeger, 2009; Ni et al., 2012; Ni and Maunsell, 2017; Verhoef and Maunsell, 2017; Ni and Maunsell, 2019; Lee and Maunsell, 2009; Ni et al., 2012; Ni and Maunsell, 2017; Verhoef and Maunsell, 2017; Ni and Maunsell, 2019). Such averaging responses are seen not only in attention paradigms but in other contexts as well (e.g., see also Xiao et al., 2014; Xiao and Huang, 2015) and are generally referred to as normalization. Importantly, these reports have generally concerned responses pooled across trials. Trial-wise spike count distribution models such as those used here and/or faster subtrial analyses such as those we have introduced in previous work (Caruso et al., 2018; Glynn et al., 2021) might indicate that such apparently averaging responses actually indicate fluctuations occurring on either the stimulus presentation or sub-stimulus presentation timescales, and not a true stable average (e.g., Figure 9a vs. Figure 9e).

That ‘normalization’ may not involve a fixed, stable operation that is constant across trials has recently garnered considerable interest. For example, several important recent studies have begun to explore how recurrent circuit mechanisms might implement dynamic fluctuations in neural activity (Heeger and Mackey, 2019; Heeger and Zemlianova, 2020) and have postulated that shifts in the balanced excitation and inhibition that is thought to underlie normalized average responses when two stimuli are presented might contribute to sizeable positive or negative spike count correlations (Verhoef and Maunsell, 2017). Finally, recent work by Coen-Cagli and colleagues proposes a method of assessing normalization strength on individual trials and demonstrated a connection between the neural responses that are well-described under a normalization model and the level of variability of firing that they show (Coen-Cagli and Solomon, 2019; Weiss et al., 2022).

Returning to the topic of attention, recent work suggests a perceptual tie to the findings we report here. The likelihood of detecting a brief near-threshold visual stimulus varies with the phase of the brain wave oscillations at the time the stimulus is presented (Busch et al., 2009; Busch and VanRullen, 2010; Vanrullen et al., 2011; Fiebelkorn et al., 2013; Fiebelkorn et al., 2018; Helfrich et al., 2018; Fiebelkorn and Kastner, 2019; see also Engel et al., 2016). This might reflect a perceptual consequence of a brain mechanism in which neurons are slightly biased toward representing some stimuli in the visual scene over others in a naturally occurring oscillatory fashion. Such bias was evident in the responses observed here, although we did not deploy a task to assess any potential connection to behavior. In our previous study (Caruso et al., 2018), we found that the LFP signal prior to stimulus onset was predictive of whether neurons would ‘pick’ A vs. B on a given trial. Future work will be needed to ascertain whether a similar phenomenon occurs in V1 or V4.

Finally, it is worth noting here that considering how the brain preserves information about two visual stimuli presented is still a far cry from understanding how the myriad details present in a natural scene are encoded. When the number of objects gets too great, it is unlikely that neurons can fluctuate between all of them, and this is likely to have consequences for perception, perhaps accounting for well-known limits on the number of objects we can perceive, attend to, and remember (e.g., Miller, 1956; Whitney and Levi, 2011; Henry and Kohn, 2020). Future studies incorporating many stimuli and investigating how this changes the pattern of fluctuating activity and correlations between units are needed to shed light on how our brains operate outside the rarefied environment of the laboratory.

Source data is provided in a zip file; the individual files have informative names that indicate what they contain and thus to what figures they are linked.

1. 1. Adelson EH
   2. Movshon JA

   (1982) Phenomenal coherence of moving gratings

   Nature 300:523–525.

   https://doi.org/10.1038/300523a0

   • PubMed
   • Google Scholar
2. 1. Alonso JM
   2. Chen Y

   (2009) Receptive field

   Scholarpedia 4:5393.

   https://doi.org/10.4249/scholarpedia.5393

   • Google Scholar
3. 1. Averbeck BB
   2. Lee D

   (2004) Coding and transmission of information by neural ensembles

   Trends in Neurosciences 27:225–230.

   https://doi.org/10.1016/j.tins.2004.02.006

   • PubMed
   • Google Scholar
4. 1. Averbeck BB
   2. Latham PE
   3. Pouget A

   (2006) Neural correlations, population coding and computation

   Nature Reviews. Neuroscience 7:358–366.

   https://doi.org/10.1038/nrn1888

   • PubMed
   • Google Scholar
5. 1. Berger JO
   2. Pericchi LR

   (1996) The intrinsic bayes factor for model selection and prediction

   Journal of the American Statistical Association 91:109–122.

   https://doi.org/10.1080/01621459.1996.10476668

   • Google Scholar
6. 1. Boynton GM

   (2009) A framework for describing the effects of attention on visual responses

   Vision Research 49:1129–1143.

   https://doi.org/10.1016/j.visres.2008.11.001

   • PubMed
   • Google Scholar
7. 1. Bremmer F

   (2000) Eye position effects in macaque area V4

   Neuroreport 11:1277–1283.

   https://doi.org/10.1097/00001756-200004270-00027

   • PubMed
   • Google Scholar
8. 1. Bulkin DA
   2. Groh JM

   (2011) Systematic mapping of the monkey inferior colliculus reveals enhanced low frequency sound representation

   Journal of Neurophysiology 105:1785–1797.

   https://doi.org/10.1152/jn.00857.2010

   • PubMed
   • Google Scholar
9. 1. Bulkin DA
   2. Groh JM

   (2012a) Distribution of eye position information in the monkey inferior colliculus

   Journal of Neurophysiology 107:785–795.

   https://doi.org/10.1152/jn.00662.2011

   • PubMed
   • Google Scholar
10. 1. Bulkin DA
    2. Groh JM

    (2012b) Distribution of visual and saccade related information in the monkey inferior colliculus

    Frontiers in Neural Circuits 6:61.

    https://doi.org/10.3389/fncir.2012.00061

    • PubMed
    • Google Scholar
11. 1. Busch NA
    2. Dubois J
    3. VanRullen R

    (2009) The phase of ongoing EEG oscillations predicts visual perception

    The Journal of Neuroscience 29:7869–7876.

    https://doi.org/10.1523/JNEUROSCI.0113-09.2009

    • PubMed
    • Google Scholar
12. 1. Busch NA
    2. VanRullen R

    (2010) Spontaneous EEG oscillations reveal periodic sampling of visual attention

    PNAS 107:16048–16053.

    https://doi.org/10.1073/pnas.1004801107

    • PubMed
    • Google Scholar
13. 1. Busse L
    2. Wade AR
    3. Carandini M

    (2009) Representation of concurrent stimuli by population activity in visual cortex

    Neuron 64:931–942.

    https://doi.org/10.1016/j.neuron.2009.11.004

    • PubMed
    • Google Scholar
14. 1. Caruso VC
    2. Mohl JT
    3. Glynn C
    4. Lee J
    5. Willett SM
    6. Zaman A
    7. Ebihara AF
    8. Estrada R
    9. Freiwald WA
    10. Tokdar ST
    11. Groh JM

    (2018) Single neurons may encode simultaneous stimuli by switching between activity patterns

    Nature Communications 9:2715.

    https://doi.org/10.1038/s41467-018-05121-8

    • PubMed
    • Google Scholar
15. 1. Coen-Cagli R
    2. Solomon SS

    (2019) Relating divisive normalization to neuronal response variability

    The Journal of Neuroscience 39:7344–7356.

    https://doi.org/10.1523/JNEUROSCI.0126-19.2019

    • PubMed
    • Google Scholar
16. 1. Cohen MR
    2. Kohn A

    (2011) Measuring and interpreting neuronal correlations

    Nature Neuroscience 14:811–819.

    https://doi.org/10.1038/nn.2842

    • PubMed
    • Google Scholar
17. 1. Denfield GH
    2. Ecker AS
    3. Shinn TJ
    4. Bethge M
    5. Tolias AS

    (2018) Attentional fluctuations induce shared variability in macaque primary visual cortex

    Nature Communications 9:2654.

    https://doi.org/10.1038/s41467-018-05123-6

    • PubMed
    • Google Scholar
18. 1. Dow BM
    2. Snyder AZ
    3. Vautin RG
    4. Bauer R

    (1981) Magnification factor and receptive field size in foveal striate cortex of the monkey

    Experimental Brain Research 44:213–228.

    https://doi.org/10.1007/BF00237343

    • PubMed
    • Google Scholar
19. 1. Dubey A
    2. Ray S

    (2016) Spatial spread of local field potential is band-pass in the primary visual cortex

    Journal of Neurophysiology 116:1986–1999.

    https://doi.org/10.1152/jn.00443.2016

    • PubMed
    • Google Scholar
20. 1. Ecker AS
    2. Denfield GH
    3. Bethge M
    4. Tolias AS

    (2016) On the structure of neuronal population activity under fluctuations in attentional state

    The Journal of Neuroscience 36:1775–1789.

    https://doi.org/10.1523/JNEUROSCI.2044-15.2016

    • PubMed
    • Google Scholar
21. 1. Engbert R
    2. Kliegl R

    (2003) Microsaccades uncover the orientation of covert attention

    Vision Research 43:1035–1045.

    https://doi.org/10.1016/s0042-6989(03)00084-1

    • PubMed
    • Google Scholar
22. 1. Engel TA
    2. Steinmetz NA
    3. Gieselmann MA
    4. Thiele A
    5. Moore T
    6. Boahen K

    (2016) Selective modulation of cortical state during spatial attention

    Science 354:1140–1144.

    https://doi.org/10.1126/science.aag1420

    • PubMed
    • Google Scholar
23. 1. Fiebelkorn IC
    2. Saalmann YB
    3. Kastner S

    (2013) Rhythmic sampling within and between objects despite sustained attention at a cued location

    Current Biology 23:2553–2558.

    https://doi.org/10.1016/j.cub.2013.10.063

    • PubMed
    • Google Scholar
24. 1. Fiebelkorn IC
    2. Pinsk MA
    3. Kastner S

    (2018) A dynamic interplay within the frontoparietal network underlies rhythmic spatial attention

    Neuron 99:842–853.

    https://doi.org/10.1016/j.neuron.2018.07.038

    • PubMed
    • Google Scholar
25. 1. Fiebelkorn IC
    2. Kastner S

    (2019) A rhythmic theory of attention

    Trends in Cognitive Sciences 23:87–101.

    https://doi.org/10.1016/j.tics.2018.11.009

    • PubMed
    • Google Scholar
26. 1. Georgopoulos AP
    2. Schwartz AB
    3. Kettner RE

    (1986) Neuronal population coding of movement direction

    Science 233:1416–1419.

    https://doi.org/10.1126/science.3749885

    • PubMed
    • Google Scholar
27. 1. Glynn C
    2. Tokdar ST
    3. Zaman A
    4. Caruso VC
    5. Mohl JT
    6. Willett SM
    7. Groh JM

    (2021) Analyzing second order stochasticity of neural spiking under stimuli-bundle exposure

    The Annals of Applied Statistics 15:41–63.

    https://doi.org/10.1214/20-AOAS1383

    • PubMed
    • Google Scholar
28. 1. Gray CM
    2. Singer W

    (1989) Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex

    PNAS 86:1698–1702.

    https://doi.org/10.1073/pnas.86.5.1698

    • PubMed
    • Google Scholar
29. 1. Gray CM

    (1999) The temporal correlation hypothesis of visual feature integration: still alive and well

    Neuron 24:31–47.

    https://doi.org/10.1016/s0896-6273(00)80820-x

    • PubMed
    • Google Scholar
30. 1. Groh JM
    2. Trause AS
    3. Underhill AM
    4. Clark KR
    5. Inati S

    (2001) Eye position influences auditory responses in primate inferior colliculus

    Neuron 29:509–518.

    https://doi.org/10.1016/s0896-6273(01)00222-7

    • PubMed
    • Google Scholar
31. 1. Groh JM
    2. Kelly KA
    3. Underhill AM

    (2003) A monotonic code for sound azimuth in primate inferior colliculus

    Journal of Cognitive Neuroscience 15:1217–1231.

    https://doi.org/10.1162/089892903322598166

    • PubMed
    • Google Scholar
32. 1. Grothe B
    2. Pecka M
    3. McAlpine D

    (2010) Mechanisms of sound localization in mammals

    Physiological Reviews 90:983–1012.

    https://doi.org/10.1152/physrev.00026.2009

    • PubMed
    • Google Scholar
33. 1. Heeger DJ
    2. Simoncelli EP
    3. Movshon JA

    (1996) Computational models of cortical visual processing

    PNAS 93:623–627.

    https://doi.org/10.1073/pnas.93.2.623

    • PubMed
    • Google Scholar
34. 1. Heeger DJ
    2. Mackey WE

    (2019) Oscillatory recurrent gated neural integrator circuits (organics), a unifying theoretical framework for neural dynamics

    PNAS 116:22783–22794.

    https://doi.org/10.1073/pnas.1911633116

    • PubMed
    • Google Scholar
35. 1. Heeger DJ
    2. Zemlianova KO

    (2020) A recurrent circuit implements normalization, simulating the dynamics of V1 activity

    PNAS 117:22494–22505.

    https://doi.org/10.1073/pnas.2005417117

    • PubMed
    • Google Scholar
36. 1. Helfrich RF
    2. Fiebelkorn IC
    3. Szczepanski SM
    4. Lin JJ
    5. Parvizi J
    6. Knight RT
    7. Kastner S

    (2018) Neural mechanisms of sustained attention are rhythmic

    Neuron 99:854–865.

    https://doi.org/10.1016/j.neuron.2018.07.032

    • PubMed
    • Google Scholar
37. 1. Hénaff OJ
    2. Boundy-Singer ZM
    3. Meding K
    4. Ziemba CM
    5. Goris RLT

    (2020) Representation of visual uncertainty through neural gain variability

    Nature Communications 11:2513.

    https://doi.org/10.1038/s41467-020-15533-0

    • PubMed
    • Google Scholar
38. 1. Henry CA
    2. Kohn A

    (2020) Spatial contextual effects in primary visual cortex limit feature representation under crowding

    Nature Communications 11:1687.

    https://doi.org/10.1038/s41467-020-15386-7

    • PubMed
    • Google Scholar
39. 1. Higgins NC
    2. Storace DA
    3. Escabí MA
    4. Read HL

    (2010) Specialization of binaural responses in ventral auditory cortices

    The Journal of Neuroscience 30:14522–14532.

    https://doi.org/10.1523/JNEUROSCI.2561-10.2010

    • PubMed
    • Google Scholar
40. 1. Kafashan M
    2. Jaffe AW
    3. Chettih SN
    4. Nogueira R
    5. Arandia-Romero I
    6. Harvey CD
    7. Moreno-Bote R
    8. Drugowitsch J

    (2021) Scaling of sensory information in large neural populations shows signatures of information-limiting correlations

    Nature Communications 12:473.

    https://doi.org/10.1038/s41467-020-20722-y

    • PubMed
    • Google Scholar
41. 1. Kanitscheider I
    2. Coen-Cagli R
    3. Pouget A

    (2015) Origin of information-limiting noise correlations

    PNAS 112:E6973–E6982.

    https://doi.org/10.1073/pnas.1508738112

    • PubMed
    • Google Scholar
42. 1. Keliris GA
    2. Li Q
    3. Papanikolaou A
    4. Logothetis NK
    5. Smirnakis SM

    (2019) Estimating average single-neuron visual receptive field sizes by fmri

    PNAS 116:6425–6434.

    https://doi.org/10.1073/pnas.1809612116

    • PubMed
    • Google Scholar
43. 1. Kohn A
    2. Coen-Cagli R
    3. Kanitscheider I
    4. Pouget A

    (2016) Correlations and neuronal population information

    Annual Review of Neuroscience 39:237–256.

    https://doi.org/10.1146/annurev-neuro-070815-013851

    • PubMed
    • Google Scholar
44. 1. Lee C
    2. Rohrer WH
    3. Sparks DL

    (1988) Population coding of saccadic eye movements by neurons in the superior colliculus

    Nature 332:357–360.

    https://doi.org/10.1038/332357a0

    • PubMed
    • Google Scholar
45. 1. Lee J
    2. Maunsell JHR

    (2009) A normalization model of attentional modulation of single unit responses

    PLOS ONE 4:e4651.

    https://doi.org/10.1371/journal.pone.0004651

    • PubMed
    • Google Scholar
46. 1. Lehky SR
    2. Peng X
    3. McAdams CJ
    4. Sereno AB

    (2008) Spatial modulation of primate inferotemporal responses by eye position

    PLOS ONE 3:e3492.

    https://doi.org/10.1371/journal.pone.0003492

    • PubMed
    • Google Scholar
47. 1. Lima B
    2. Singer W
    3. Chen NH
    4. Neuenschwander S

    (2010) Synchronization dynamics in response to plaid stimuli in monkey V1

    Cerebral Cortex 20:1556–1573.

    https://doi.org/10.1093/cercor/bhp218

    • PubMed
    • Google Scholar
48. 1. Long B
    2. Yu CP
    3. Konkle T

    (2018) Mid-level visual features underlie the high-level categorical organization of the ventral stream

    PNAS 115:E9015–E9024.

    https://doi.org/10.1073/pnas.1719616115

    • PubMed
    • Google Scholar
49. 1. Maier JX
    2. Groh JM

    (2010) Comparison of gain-like properties of eye position signals in inferior colliculus versus auditory cortex of primates

    Frontiers in Integrative Neuroscience 4:121–132.

    https://doi.org/10.3389/fnint.2010.00121

    • PubMed
    • Google Scholar
50. 1. McAlpine D
    2. Grothe B

    (2003) Sound localization and delay lines -- do mammals fit the model?

    Trends in Neurosciences 26:347–350.

    https://doi.org/10.1016/S0166-2236(03)00140-1

    • PubMed
    • Google Scholar
51. 1. Merriam EP
    2. Gardner JL
    3. Movshon JA
    4. Heeger DJ

    (2013) Modulation of visual responses by gaze direction in human visual cortex

    The Journal of Neuroscience 33:9879–9889.

    https://doi.org/10.1523/JNEUROSCI.0500-12.2013

    • PubMed
    • Google Scholar
52. 1. Miller GA

    (1956) The magical number seven plus or minus two: some limits on our capacity for processing information

    Psychological Review 63:81–97.

    https://doi.org/10.1037/h0043158

    • PubMed
    • Google Scholar
53. 1. Milner PM

    (1974) A model for visual shape recognition

    Psychological Review 81:521–535.

    https://doi.org/10.1037/h0037149

    • PubMed
    • Google Scholar
54. Preprint

    1. Mohl JT
    2. Caruso VC
    3. Tokdar ST
    4. Groh JM

    (2020) Sensitivity and Specificity of a Bayesian Single Trial Analysis for Time Varying Neural Signals

    bioRxiv.

    https://doi.org/10.1101/690958

    • Google Scholar
55. 1. Moreno-Bote R
    2. Beck J
    3. Kanitscheider I
    4. Pitkow X
    5. Latham P
    6. Pouget A

    (2014) Information-limiting correlations

    Nature Neuroscience 17:1410–1417.

    https://doi.org/10.1038/nn.3807

    • PubMed
    • Google Scholar
56. 1. Ni AM
    2. Ray S
    3. Maunsell JHR

    (2012) Tuned normalization explains the size of attention modulations

    Neuron 73:803–813.

    https://doi.org/10.1016/j.neuron.2012.01.006

    • PubMed
    • Google Scholar
57. 1. Ni AM
    2. Maunsell JHR

    (2017) Spatially tuned normalization explains attention modulation variance within neurons

    Journal of Neurophysiology 118:1903–1913.

    https://doi.org/10.1152/jn.00218.2017

    • PubMed
    • Google Scholar
58. 1. Ni A.M
    2. Maunsell JHR

    (2019) Neuronal effects of spatial and feature attention differ due to normalization

    The Journal of Neuroscience 39:5493–5505.

    https://doi.org/10.1523/JNEUROSCI.2106-18.2019

    • PubMed
    • Google Scholar
59. 1. Nogueira R
    2. Peltier NE
    3. Anzai A
    4. DeAngelis GC
    5. Martínez-Trujillo J
    6. Moreno-Bote R

    (2020) The effects of population tuning and trial-by-trial variability on information encoding and behavior

    The Journal of Neuroscience 40:1066–1083.

    https://doi.org/10.1523/JNEUROSCI.0859-19.2019

    • PubMed
    • Google Scholar
60. 1. Palanca BJA
    2. DeAngelis GC

    (2005) Does neuronal synchrony underlie visual feature grouping?

    Neuron 46:333–346.

    https://doi.org/10.1016/j.neuron.2005.03.002

    • PubMed
    • Google Scholar
61. 1. Porter KK
    2. Metzger RR
    3. Groh JM

    (2006) Representation of eye position in primate inferior colliculus

    Journal of Neurophysiology 95:1826–1842.

    https://doi.org/10.1152/jn.00857.2005

    • PubMed
    • Google Scholar
62. 1. Porter KK
    2. Metzger RR
    3. Groh JM

    (2007) Visual- and saccade-related signals in the primate inferior colliculus

    PNAS 104:17855–17860.

    https://doi.org/10.1073/pnas.0706249104

    • PubMed
    • Google Scholar
63. 1. Reynolds JH
    2. Heeger DJ

    (2009) The normalization model of attention

    Neuron 61:168–185.

    https://doi.org/10.1016/j.neuron.2009.01.002

    • PubMed
    • Google Scholar
64. 1. Rodman HR
    2. Albright TD

    (1989) Single-Unit analysis of pattern-motion selective properties in the middle temporal visual area (MT)

    Experimental Brain Research 75:53–64.

    https://doi.org/10.1007/BF00248530

    • PubMed
    • Google Scholar
65. 1. Romo R
    2. Hernández A
    3. Zainos A
    4. Salinas E

    (2003) Correlated neuronal discharges that increase coding efficiency during perceptual discrimination

    Neuron 38:649–657.

    https://doi.org/10.1016/s0896-6273(03)00287-3

    • PubMed
    • Google Scholar
66. 1. Ruff DA
    2. Alberts JJ
    3. Cohen MR

    (2016) Relating normalization to neuronal populations across cortical areas

    Journal of Neurophysiology 116:1375–1386.

    https://doi.org/10.1152/jn.00017.2016

    • PubMed
    • Google Scholar
67. 1. Ruff DA
    2. Cohen MR

    (2016) Attention increases spike count correlations between visual cortical areas

    The Journal of Neuroscience 36:7523–7534.

    https://doi.org/10.1523/JNEUROSCI.0610-16.2016

    • PubMed
    • Google Scholar
68. 1. Shadlen MN
    2. Newsome WT

    (1994) Noise, neural codes and cortical organization

    Current Opinion in Neurobiology 4:569–579.

    https://doi.org/10.1016/0959-4388(94)90059-0

    • PubMed
    • Google Scholar
69. 1. Singer W
    2. Gray CM

    (1995) Visual feature integration and the temporal correlation hypothesis

    Annual Review of Neuroscience 18:555–586.

    https://doi.org/10.1146/annurev.ne.18.030195.003011

    • PubMed
    • Google Scholar
70. 1. Vanrullen R
    2. Busch NA
    3. Drewes J
    4. Dubois J

    (2011) Ongoing EEG phase as a trial-by-trial predictor of perceptual and attentional variability

    Frontiers in Psychology 2:60.

    https://doi.org/10.3389/fpsyg.2011.00060

    • PubMed
    • Google Scholar
71. 1. Verhoef BE
    2. Maunsell JHR

    (2017) Attention-related changes in correlated neuronal activity arise from normalization mechanisms

    Nature Neuroscience 20:969–977.

    https://doi.org/10.1038/nn.4572

    • PubMed
    • Google Scholar
72. 1. Von Der Malsburg C

    (1994)

    Models of Neural Networks

    95–119, The correlation theory of brain function, Models of Neural Networks, Springer.

    • Google Scholar
73. Preprint

    1. Weiss O
    2. Bounds HA
    3. Adesnik H
    4. Coen-Cagli R

    (2022) Modeling the Diverse Effects of Divisive Normalization on Noise Correlations

    bioRxiv.

    https://doi.org/10.1101/2022.06.08.495145

    • Google Scholar
74. 1. Werner-Reiss U
    2. Groh JM

    (2008) A rate code for sound azimuth in monkey auditory cortex: implications for human neuroimaging studies

    The Journal of Neuroscience 28:3747–3758.

    https://doi.org/10.1523/JNEUROSCI.5044-07.2008

    • PubMed
    • Google Scholar
75. 1. Whitney D
    2. Levi DM

    (2011) Visual crowding: a fundamental limit on conscious perception and object recognition

    Trends in Cognitive Sciences 15:160–168.

    https://doi.org/10.1016/j.tics.2011.02.005

    • PubMed
    • Google Scholar
76. 1. Willett SM
    2. Groh JM

    (2022) Multiple sounds degrade the frequency representation in monkey inferior colliculus

    The European Journal of Neuroscience 55:528–548.

    https://doi.org/10.1111/ejn.15545

    • PubMed
    • Google Scholar
77. 1. Woods TM
    2. Su TK
    3. Recanzone GH

    (2001) Spatial tuning as a function of stimulus intensity of single neurons in awake macaque monkey auditory cortex

    Soc Neurosci Abstr 27:512–517.

    https://doi.org/10.1152/jn.2000.83.4.2315

    • Google Scholar
78. 1. Xiao J
    2. Niu YQ
    3. Wiesner S
    4. Huang X

    (2014) Normalization of neuronal responses in cortical area MT across signal strengths and motion directions

    Journal of Neurophysiology 112:1291–1306.

    https://doi.org/10.1152/jn.00700.2013

    • PubMed
    • Google Scholar
79. 1. Xiao J
    2. Huang X

    (2015) Distributed and dynamic neural encoding of multiple motion directions of transparently moving stimuli in cortical area MT

    The Journal of Neuroscience 35:16180–16198.

    https://doi.org/10.1523/JNEUROSCI.2175-15.2015

    • PubMed
    • Google Scholar
80. 1. Xing D
    2. Yeh CI
    3. Shapley RM

    (2009) Spatial spread of the local field potential and its laminar variation in visual cortex

    The Journal of Neuroscience 29:11540–11549.

    https://doi.org/10.1523/JNEUROSCI.2573-09.2009

    • PubMed
    • Google Scholar
81. 1. Zohary E
    2. Shadlen MN
    3. Newsome WT

    (1994) Correlated neuronal discharge rate and its implications for psychophysical performance

    Nature 370:140–143.

    https://doi.org/10.1038/370140a0

    • PubMed
    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "Coordinated multiplexing of information about separate objects in visual cortex" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by Joshua Gold as the Senior Editor. The following individual involved in the review of your submission has agreed to reveal their identity: Ruben Coen-Cagli (Reviewer #3).

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

While this paper shows some intriguing results, we feel that there are a lot of open questions that need to be addressed before convincing evidence of multiplexing can be established. These points are discussed below:

1. The best spike count model shown in Figure 2C is confusing. It seems that the number of "conditions" is a small fraction of the total number of conditions (and neurons?) that were tested. Supplementary Figure 1 provides more details (for example, the "mixture" corresponds to only 14% of total cases), but it is still confusing (for example, what does WinProb>Min mean?). From what I understood, the total number of neurons recorded for the Adjacent case in V1 is 1604, out of which 935 are Poisson-like with substantially separated means. Each one has 2 conditions (for the two directions), leading to 1870 conditions (perhaps a few less in case both conditions were not available). I think the authors should show 5 bar plots – the first one showing the fraction for which none of the models won by 2/3 probability, and then the remaining 4 ones. That way it is clear how many of the total cases show the "multiplexing" effect. I also think that it would be good to only consider neurons/conditions for which at least some minimum number of trials are available (a cutoff of say ~15) since the whole point is about finding a bimodal distribution for which enough trials are needed.

2. More RF details need to be provided. What was the size of the V1 RFs? What was the eccentricity? Typically, the RF diameter in V1 at an eccentricity of ~3 degrees is no more than 1 degree. It is not enough to put 2 Gabors of size 1 degree each to fit inside the RF. How close were the Gabors? We are confused about the statement in the second paragraph of page 9 "typically only one of the two adjacent gratings was located within the RF" – we thought the whole point of multiplexing is that when both stimuli (A and B) are within the RF, the neuron nonetheless fires like A or B? The analysis should only be conducted for neurons for which both stimuli are inside the RF. When studying noise correlations, only pairs that have overlapping RFs such as both A and B and within the RFs of both neurons should be considered. The cortical magnification factor at ~3-degree eccentricity is 2-2.5mm/degree, so we expect the RF center to shift by at least 2 degrees from one end of the array to the other.

3. Eye data analysis: the reviewers are concerned that this could potentially be a big confound. Removing trials that had microsaccades is not enough. Typically, in these tasks the fixation window is 1.5-2 degrees, so that if the monkey fixates on one corner in some trials and another corner in other trials (without making any microsaccades in either), the stimuli may nonetheless fall inside or away from the RFs, leading to differences in responses. This needs to be ruled out. We do not find the argument presented on pages 18 or 23 completely convincing, since the eye positions could be different for a single stimulus versus when both stimuli are presented. It is important to show that the eye positions are similar in "AB" trials for which the responses are "A" like versus "B" like, and these, in turn, are similar to when "A" and "B" are presented alone.

Relatedly, more details on the detection of microsaccades and threshold values for inclusion (relative to stimulus and RF sizes) should be provided, given how central a role they might play. In particular, the authors state in Discussion that small residual eye movements would inflate response variability in all stimulus conditions. This is correct, but because of the stimulus design, it is possible (likely?) that the effects are quite different for segregated stimuli versus superimposed and single-stimulus conditions. Furthermore, the difference might be precisely in the direction of the effects reported. That is because segregated stimuli are spatially separate, and each stimulus only covers some receptive fields in the recording, it is possible that eye movements would bring inside the RF a different stimulus in each trial. In addition to producing bimodal response distributions for individual neurons, this would also induce positive correlations for pairs with the same preference and negative correlations for pairs with opposite preferences. On the other hand, in the superimposed condition where the stimulus is large enough to cover all RFs, at most eye movements would bring (part of) the stimulus inside versus outside the RF across trials, therefore contributing to positive noise correlations for all pairs (i.e., to shifts from stimulus-driven to spontaneous activity, in the extreme case). In our opinion, the main concern raised in the public review deserves more in-depth analysis, of the data and/or simulations, for us to be convinced that eye movements truly do not play a role. Conversely, if they do, it may be worth discussing the possibility that they are part of the mechanism underlying the proposed coding scheme.

4. Figures 5 and 6 show that the difference in noise correlations between the same preference and different preference neurons remains even for non-mixture type neurons. So, although the reason for the particular type of noise correlation was given for multiplexing neurons (Figure 3 and 4), it seems that the same pattern holds even for non-multiplexers. Although the absolute values are somewhat different across categories, one confound that still remains is that the noise correlations are typically dependent on signal correlation, but here the signal correlation is not computed (only responses to 2 stimuli are available). If there is any tuning data available for these recordings, it would be great to look at the noise correlations as a function of signal correlations for these different pairs. Another analysis of interest would be to check whether the difference in the noise correlation for simply "A"/"B" versus "AB" varies according to neuron pair category. Finally, since the authors mention in the Discussion that "correlations did not depend on whether the two units preferred the same stimulus or different", it would be nice to explicitly show that in figure 5C by showing the orange trace ("A" alone or "B" alone) for both same (green) and different (brown) pairs separately.

5.We are confused about the nature of Poisson models. If we are correct, the Poisson(a+b) is the sum of the two Poisson(a) and Poisson(b), that is, Poisson(a+b) = Poisson(a) + Poisson(b). Then, the mixture and intermediate models are very similar, identical if a*λ_A and (1-a)*λ_B happen to be integer numbers.

6. It is unclear why the 'outside' model predicts responses outside the range if neurons were to linearly sum the A and B responses.

7. It is also unclear why the 'single' hypothesis would indicate a winner-take-all response. If we understand correctly, under this model, the response to A+B is either the rate A or B, but not the max between λ_A and λ_B. Also, this model could have given an extra free parameter to modulate its amplitude to the stimulus A+B.

8. The concept of "coarse population coding" can be misleading, as actual population coding can represent stimulus with quite good precision. The authors refer to the broad tuning of single cells, but this does not readily correspond to coarse population coding. This could be clarified.

9. As a complement to the correlation analysis, one could check whether, on a trial-by-trial basis, the neuronal response of a single neuron is closer to the A+B response average, or to either the A or B responses. This would clearly indicate that the response fluctuates between representing A or B, or simultaneously represents A+B. I am trying to understand why this is not one of the main analyses of the paper instead of the correlation analysis, which involves two neurons instead of one.

10. In the discussion about noise correlations, the recent papers Nogueira et al., J Neuroscience, 2020 and Kafashan et al., Nat Comm, 2021 could be cited. Also, noise correlations can also be made time-dependent, so the distinction between the temporal correlation hypotheses and noise correlations might not be fundamental.

11. It would be interesting to study the effect of contrast on the mixed responses. Is it reasonable to predict that with higher contrast the mixture responses would be more dominant than the single ones? This could be the case if the selection mechanism has a harder time suppressing one of the object responses. This would also predict that noise correlations will go down with higher contrast.

12. What is the time bin size used for the analysis? Would the results be the same if one focuses on the early time responses or on the late time responses? At least from the units shown in Figure 2, it looks that there is always an object response that is delayed respect to the other, so it would seem interesting to test noise correlations in those two temporal windows.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Coordinated multiplexing of information about separate objects in visual cortex" for further consideration by eLife. Your revised article has been evaluated by Joshua Gold (Senior Editor) and a Reviewing Editor.

The manuscript has been improved and many of the reviewers' concerns have been addressed. However, some major issues remain. Although we typically avoid repeated revise/resubmit cycles, we believe that it is important for these issues to be addressed in a new revision. Specifically, upon discussion the reviewers unanimously remained concerned about the possibility that at least some of your results could be accounted for by subtle differences in eye movements. The new analyses related to that issue were appreciated but considered inadequate.

As detailed below, we would like you to provide:

1. More information about whether the electrodes that show evidence of multiplexing are the ones whose RF straddles the two stimuli, because in that case, small eye movements will bring one of the two stimuli inside the RF.

2. Further analyses of eye position to rule out the possibility described above. In our discussions, it was noted that the new Figure 1 – Supplementary Figure 1 appears to show that numerous V1 RFs straddle the two stimuli, and under those conditions, we really want to know if the "multiplexed" responses are because of small fixational differences/microsaccades that affect which of the two stimuli takes a more dominant position in the RF. To test for this kind of effect, just the STD of eye position per trial for one-stimulus vs two-stimuli conditions does not seem to be sufficient. Instead, it seems important to know whether, in the two-stimuli condition, responses were more "A-like" when gaze put the A stimulus closer to the RF center, and were more "B-like" when gaze put the B stimulus closer to the RF center. So something like a linear regression of spiking response versus "eye position along the axis defined by the centers of the two stimuli, increasing towards the stimulus that alone elicited the larger response" could be useful.

Reviewer #1 (Recommendations for the authors):

The authors have made several changes in the manuscript to address previous concerns. However, the fact that typically only one stimulus spanned the RF makes it difficult to make a case for multiplexing, since the other stimulus is outside the classical RF. The arguments made by the authors in response to the previous RF-related question (point 2) are based on their own hypothesis about some underlying "spatial scale" of multiplexing which is coarser than the RF size of V1, which I do not find particularly convincing and would be extremely difficult to implement in V1. Further, while the authors showed more results related to eye position analysis, they do not show the key comparison that was requested previously.

To better appreciate the spatial scales involved, I refer to figures from the following two studies in V1 where very small stimuli (0.1 – 0.2 degrees) were used to map the RFs: Figure 2 of Xing et al., 2009, JNS, and Figure 2 of Dubey and Ray, 2016, JNP. Typically, the SD of fitted Gaussian is typically no more than 0.25 degrees at an eccentricity of 2-3 degrees (if you consider the radius as 2SD (0.5 degrees), the diameter is about a degree). For such RFs, there is no response if a stimulus is more than a degree away from the RF. For Gabor patches used in this paper, only the "size" is mentioned. Does size refer to the radius or SD? In either case, there is no way to fit two Gabors within the RF. What is the separation between two Gabors? Figure 1 should highlight all these details, including the radius (not just the center) of the V1 units.

Given the concern that one of the stimuli is always outside the RF, how do we explain the findings? One possible answer is in small differences in eye position. Multiplexing is anyway observed in only ~100 out of 1389 units. I suspect these are the units whose RF center is between the two stimuli. For the AB condition (i.e., both stimuli are presented), small jitters in eye position would bring one of the two stimuli in the RF, and therefore the unit would respond like either A or B. To address this, the authors should show the RF centers of the units that show evidence of multiplexing, along with the stimuli.

In addition, it is important to check for possible differences in eye position within AB conditions for trials for which responses were "A like" versus "B like". The authors have compared the AB condition to A and B conditions presented alone, but that is not enough. The argument that the stimuli were presented for a short duration and in pseudorandom order and therefore it is not possible for the animal to have systematically different eye positions for A/B versus AB conditions is obviously true, but that is not the point. The point is that the eye positions have small variations from trial to trial (as shown in Figure 1 – Supp 2) even before stimulus onset, and AB trials in which the position happens to be in one location get classified as "A like" and another location gets classified as "B like". It is in fact very hard to rule out this possibility given the instrument noise which affects the precision of eye positions, but it is crucial to rule this out.

Reviewer #2 (Recommendations for the authors):

The authors have appropriately addressed all comments. In particular, the controls on eye movements are sound.

Reviewer #3 (Recommendations for the authors):

The authors have addressed my concerns and all other concerns raised in the editor's summary. My main concern was whether uncontrolled fixational eye movements (microsaccades) could account in part for the observed multiplexing. I understand now that concern was largely because of missing details about eye position, RF sizes, stimulus sizes, etc, which are now reported. The possibility remains that trial-by-changes in eye position (within the fixation window) would inflate the proportion of single-neuron "mixture" cases for adjacent two-object stimuli (by effectively changing which object is inside the RF in any given trial). But, importantly, this could not explain the observed patterns of noise correlations.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Coordinated multiplexing of information about separate objects in visual cortex" for further consideration by eLife. Your revised article has been evaluated by Joshua Gold (Senior Editor) and a Reviewing Editor.

The manuscript has been improved but there are some remaining issues that need to be addressed. We realized we were not as specific as we should have been in the last round of feedback and have tried to clarify here exactly what the reviewers are looking for.

In particular, because the authors claim that ~100 electrodes have mixture responses, the possible influence of eye movements should be tested for all the mixture units, not only 9/100 as done in the previous review. We would therefore like the analyses done for all the mixtures. To avoid ambiguity, we are listing the requested analyses in more detail below:

1. Show the RF centers of ALL the units labeled as mixtures (~100). The best way to do this would be to make a line passing through the centers of the two stimuli, and make five groups of units depending on their RF centers – (i) left of the left stimulus, (ii) on the left stimulus, (iii), between left and right stimulus, (iv) on the right stimulus and (v) right of the right stimulus. Then show what proportion of units in each category are mixtures. The eye movement hypothesis predicts that mixtures will be predominant in (iii). Actually, since 91/100 units have one of the two modes at zero (as far as I could understand the previous analysis), these units could even be at (i) or (v).

2. Do the eye position analysis for ALL mixture units. Since the stimuli are mainly separated along the x-axis, all we are asking is to make a scatter plot of spike counts and average eyeX position for each trial and then check for correlation between the two. This analysis is valid even for units that only respond to one stimulus (i.e. the remaining 91 units). We expect to find a significant correlation (either positive or negative) if the results are due to small eye movements, but not if the firing is due to their multiplexing hypothesis. Or, if you find significant correlations since the RF sizes are comparable to the stimulus sizes, you need to show that these correlations are insufficient to explain the bimodality.

3. In addition, I think the authors should at least show the typical RF radius of the units (make a circle of radius of 0.5 degrees on a few of the units, perhaps the ones shown in Figure 4). We think it is important to show that the RFs do not encompass both stimuli, which is not clear from the plots.

4. The fact that one mode of the bimodal distribution is zero in 91/100 cases should also be made clearer, perhaps in the methods section. Essentially the bimodal response in most cases is not A versus B, but A/B versus zero.

Essential revisions:

While this paper shows some intriguing results, we feel that there are a lot of open questions that need to be addressed before convincing evidence of multiplexing can be established. These points are discussed below:

1. The best spike count model shown in Figure 2C is confusing. It seems that the number of "conditions" is a small fraction of the total number of conditions (and neurons?) that were tested. Supplementary Figure 1 provides more details (for example, the "mixture" corresponds to only 14% of total cases), but it is still confusing (for example, what does WinProb>Min mean?). From what I understood, the total number of neurons recorded for the Adjacent case in V1 is 1604, out of which 935 are Poisson-like with substantially separated means. Each one has 2 conditions (for the two directions), leading to 1870 conditions (perhaps a few less in case both conditions were not available). I think the authors should show 5 bar plots – the first one showing the fraction for which none of the models won by 2/3 probability, and then the remaining 4 ones. That way it is clear how many of the total cases show the "multiplexing" effect. I also think that it would be good to only consider neurons/conditions for which at least some minimum number of trials are available (a cutoff of say ~15) since the whole point is about finding a bimodal distribution for which enough trials are needed.

We thank you for catching this confusion. We have revised Figure 2C as requested, modified Supplementary Figure 1 (now Figure 2 – Supp Figure 1) to be more clear, and improved Table 1 and its legend to more succinctly walk the reader through the numbers of included conditions and how many units and distinct sessions they concern. We have also added a Table 2 to provide the trial counts, which exceed the criteria we have previously benchmarked and used (Mohl et al 2020; Caruso et al., 2018).

Details:

We first explain the changes to Table 1. The most important column is the last one; this column provides the total numbers of conditions (“Triplets”) that passed the Poisson and good response-separation criteria for inclusion in the analyses. The remaining columns illustrate how many units and sessions these included triplets came from. For example, in the adjacent data set in V1, there were 16 sessions; all 16 sessions yielded data that was included. There were 1604 units of which 935 yielded data that was included. We’ve re-written the legend to better explain this.

Note: if the reviewers feel it would be beneficial to simplify, we could easily delete the “Available sessions” and “Available units” columns, and just report the number of sessions and units that were actually used.

Regarding “the total number of neurons recorded for the Adjacent case in V1 is 1604, out of which 935 are Poisson-like with substantially separated means. Each one has 2 conditions (for the two directions), leading to 1870 conditions” – this is not quite correct, but it was a very helpful explanation of how we had confused the matter. To summarize the Adjacent case in V1, 935 units provided triplets, 1-2 each, yielding a total of 1389.

For Figure 2C (now 2C and D), “I think the authors should show 5 bar plots - the first one showing the fraction for which none of the models won by 2/3 probability, and then the remaining 4 ones.”, we have included a pie chart to complement the bar chart to illustrate the total number of conditions involved.

For Supplementary Figure 1 “what does WinProb>Min mean”, that refers to the lowest possible winning probability. We used a flat prior, i.e. each model starts off with a 0.25 probability of being the best model. We have modified this figure (now Figure 2 – Supplementary Figure 1) to replace “WinProb>Min” with “WinProb>0.25”.

Regarding “I also think that it would be good to only consider neurons/conditions for which at least some minimum number of trials are available (a cutoff of say ~15) since the whole point is about finding a bimodal distribution for which enough trials are needed”, we think we are in good shape on this front, but we agree we had not documented this. Accordingly, we now include Table 2 which is accompanied by following text in the Methods section: “The numbers of trials involved for the different data sets are provided in Table 2. The trial counts were adequate to provide accurate model identification according to our previous simulations. Depending on the separation between λA and λB, we previously found that model identification accuracy in simulations is high for trial counts as low as 5 “AB” trials, and plateaus near ceiling around “AB” trial counts of about 10 trials and above – i.e. below the mean trial counts available here for all datasets (see Figure 4 of Mohl et al., 2020).”

The relevant figure supporting this point is Figure 4 from Mohl et al. (2020).

Mohl JT, Caruso VC, Tokdar S, Groh JM (2020) Sensitivity and specificity of a Bayesian single trial analysis for time varying neural signals. Neurons, Theory, Data Analysis, and Behavior.

2. More RF details need to be provided. What was the size of the V1 RFs? What was the eccentricity? Typically, the RF diameter in V1 at an eccentricity of ~3 degrees is no more than 1 degree. It is not enough to put 2 Gabors of size 1 degree each to fit inside the RF. How close were the Gabors? We are confused about the statement in the second paragraph of page 9 "typically only one of the two adjacent gratings was located within the RF" – we thought the whole point of multiplexing is that when both stimuli (A and B) are within the RF, the neuron nonetheless fires like A or B? The analysis should only be conducted for neurons for which both stimuli are inside the RF. When studying noise correlations, only pairs that have overlapping RFs such as both A and B and within the RFs of both neurons should be considered. The cortical magnification factor at ~3-degree eccentricity is 2-2.5mm/degree, so we expect the RF center to shift by at least 2 degrees from one end of the array to the other.

We now provide the requested details regarding stimulus size and receptive field locations in the Methods. But before summarizing those results, we would like to clarify a conceptual point: while our study is motivated by the observation that tuning in sensory structures is coarse relative to our perceptual capabilities, we did not mean to limit this consideration to the spatial scale of V1 neurons specifically. Rather, we see this as an overall problem across sensory pathways. For example, even if two particular stimuli activate completely separate populations in V1, this may not be the case at later stages of processing such as IT cortex where receptive fields are quite large. We believe the brain has to solve this problem at the scale of the entire sensory processing stream, and It must do so without prior knowledge of the particular spacing of particular individual stimuli.

That said, we agree that it could well have turned out that fluctuating activity might only be evident when two stimuli were within the same receptive fields of individual neurons in a particular brain area. However, this did not turn out to be the case. Many of the units in the present study only had one stimulus in the receptive field, yet a subset showed evidence of fluctuating response patterns.

Perhaps we would have observed more evidence for multiplexing if both stimuli were in the receptive field. Indeed, in our previous study involving the inferior colliculus and inferotemporal cortex (Caruso et al 2018), we observed a higher incidence of multiplexing than in the current study. In the IC, spatial tuning is very broad (sound location is rate coded in the mammalian brain), so in general most neurons responded to both sounds. We are currently working on a followup study in which we are parametrically varying the frequency of the two sounds to smoothly vary the response differences between the A and B stimuli. In this new study, we find that multiplexing is indeed more prevalent when the two stimuli are both in the frequency response area of the neuron than when only one stimulus is. So the fact that in general only one stimulus drove the units in the current study may well have limited the evidence for multiplexing in the current dataset.

Changes made: We have modified the language in the introduction to describe the rationale more carefully, so as not to imply that multiplexing need only occur in cases where the two stimuli are within the same receptive field of the neurons under study in a particular brain area. Rather, multiplexing is likely to have some as yet unknown characteristic spatial scale that may be determined by the coarsest tuning evident at any stage in the sensory pathway. We also now return to this topic in the Discussion to specifically clarify our interpretation regarding this issue. We have future experiments planned in which we will systematically vary the locations of the stimuli involved, so we expect to have data to address this issue in the future.

Key changes in the introduction: The sentence “Such breadth of tuning means that individual neurons will commonly experience more than one stimulus to which they could in principle respond, making it unclear how information about multiple objects is preserved.” has been rewritten as: “Such breadth of tuning means that there will be overlap in the population of neurons activated by individual stimuli, making it unclear how information about multiple objects is preserved.” This shifts the meaning to focus on the (implied) population overlap rather than the receptive fields of the particular individual neurons we studied.

Key changes in the Methods: The requested information regarding receptive fields and stimulus locations is provided in the Methods section under “Electrophysiological recordings and visual stimuli”: “…we analyzed trials in which attention was directed to a Gabor patch located in the hemisphere ipsilateral to the recorded V1 neurons (i.e. well away from those neurons’ receptive fields, see below). On these trials, two unattended Gabor patches were presented in close proximity to each other within the area covered by the receptive fields of the recorded V1 neurons – these receptive fields were approximately 3 degrees eccentric and had classical receptive field diameters typically estimated to be <1 degree of visual angle. These patches were centered 2.5-3.5 degrees eccentrically and each stimulus typically subtended 1 degree of visual angle (see Ruff and Cohen 2016 Figure 1B for a sketch, reproduced here in Figure 1 – Supplementary Figure 1)…”

Key changes in the Discussion: We inserted a new paragraph that reads “It is interesting to note that we observed evidence of multiplexing each stimulus even in V1 where there receptive fields are small and the stimuli we used did not themselves typically span more than one receptive field – thus the coarseness of tuning did not necessarily pose a problem for the encoding of these particular stimuli in this particular brain area. However, the precision of V1’s spatial code may not be the limiting factor.

Multiplexing is likely to have some as yet unknown characteristic spatial scale that may be determined by the coarsest tuning evident at any stage in the sensory pathway. Future work in which stimuli are systematically varied to manipulate the amount of overlap in the activity patterns evoked in different brain areas by each stimulus alone are needed to answer this question. “

3. Eye data analysis: the reviewers are concerned that this could potentially be a big confound. Removing trials that had microsaccades is not enough. Typically, in these tasks the fixation window is 1.5-2 degrees, so that if the monkey fixates on one corner in some trials and another corner in other trials (without making any microsaccades in either), the stimuli may nonetheless fall inside or away from the RFs, leading to differences in responses. This needs to be ruled out. We do not find the argument presented on pages 18 or 23 completely convincing, since the eye positions could be different for a single stimulus versus when both stimuli are presented. It is important to show that the eye positions are similar in "AB" trials for which the responses are "A" like versus "B" like, and these, in turn, are similar to when "A" and "B" are presented alone.

Relatedly, more details on the detection of microsaccades and threshold values for inclusion (relative to stimulus and RF sizes) should be provided, given how central a role they might play. In particular, the authors state in Discussion that small residual eye movements would inflate response variability in all stimulus conditions. This is correct, but because of the stimulus design, it is possible (likely?) that the effects are quite different for segregated stimuli versus superimposed and single-stimulus conditions. Furthermore, the difference might be precisely in the direction of the effects reported. That is because segregated stimuli are spatially separate, and each stimulus only covers some receptive fields in the recording, it is possible that eye movements would bring inside the RF a different stimulus in each trial. In addition to producing bimodal response distributions for individual neurons, this would also induce positive correlations for pairs with the same preference and negative correlations for pairs with opposite preferences. On the other hand, in the superimposed condition where the stimulus is large enough to cover all RFs, at most eye movements would bring (part of) the stimulus inside versus outside the RF across trials, therefore contributing to positive noise correlations for all pairs (i.e., to shifts from stimulus-driven to spontaneous activity, in the extreme case). In our opinion, the main concern raised in the public review deserves more in-depth analysis, of the data and/or simulations, for us to be convinced that eye movements truly do not play a role. Conversely, if they do, it may be worth discussing the possibility that they are part of the mechanism underlying the proposed coding scheme.

We appreciate the reviewers raising these concerns, which we now address more fully in the manuscript with a combination of new data analysis, additional methodological details, and more fully fleshed-out reasoning. The key points are as follows:

(a) Eye position did not differ during the A/B vs. AB stimulus presentations. See new Figure 1 -Supplementary Figure 2 in the manuscript, and included below. This is the simplest answer to the above questions, but lest any concerns remain, read on:

(b) The timing doesn’t work for variation in eye position to pose a substantial confound. The A/B/AB stimuli are delivered in pseudorandom order, so it was not possible for the animals to anticipate which stimulus would appear; thus pre-stimulus eye position effects are not a possibility. And there isn’t time for post-stimulus eye position effects to have a meaningful influence during the short spike counting window we used – 200 ms after stimulus onset. It typically takes 150-350 ms for visual stimuli to elicit a macrosaccade; a similar latency is likely involved for any potential stimulus-influenced modulation of fixational eye movements (Engbert and Kliegl, 2003). Because we only analyzed a 200 ms spike counting temporal epoch, even if stimuli were to differentially modulate the position of the eyes within the fixation window, there is very little time for such modulation to affect the spikes we counted.

(Note – any saccades directed to one or the other targets would of course be excluded as incorrect trials, but we understood the concern to relate to any more subtle tendencies to fixate differently based on the stimulus conditions.)

(c) Given that there was no difference in eye position between conditions in the adjacent stimuli data sets, we believe it is beyond the scope of this paper to conduct this analysis for the superimposed stimuli datasets as the lack of an effect in the adjacent dataset is sufficient to eliminate this as a candidate explanation for our findings.

(d) Details regarding microsaccade detection, RF sizes, stimulus sizes, and fixation windows are now included as described above. We apologize for omitting these details. To recap for the adjacent V1 dataset:

Fixation windows were +/- 0.5 degrees.

RF sizes: typically less than 1 degree

Stimulus sizes: typically 1 degree, about 2.5-3.5 eccentric

Microsaccade detection criteria: stimulus presentations in which eye velocity ever exceed six standard deviations from the mean eye velocity during fixation were excluded from further analysis, following the method of Engbert and Kliegl, 2003.

(f) Regarding “this would also induce positive correlations for pairs with the same preference and negative correlations for pairs with opposite preferences”, we respectfully disagree. This would only be the case if the receptive fields of the neurons with the same preferences were truly precisely aligned in space, as opposed to merely overlapping. As shown in Ruff and Cohen 2016 Figure 1B, and now illustrated in our Figure 1 – Supplementary Figure 1, this was not the case. The logic is most easily conveyed in Author response image 1:

Author response image 1

Download asset Open asset

4. Figures 5 and 6 show that the difference in noise correlations between the same preference and different preference neurons remains even for non-mixture type neurons. So, although the reason for the particular type of noise correlation was given for multiplexing neurons (Figure 3 and 4), it seems that the same pattern holds even for non-multiplexers. Although the absolute values are somewhat different across categories, one confound that still remains is that the noise correlations are typically dependent on signal correlation, but here the signal correlation is not computed (only responses to 2 stimuli are available). If there is any tuning data available for these recordings, it would be great to look at the noise correlations as a function of signal correlations for these different pairs. Another analysis of interest would be to check whether the difference in the noise correlation for simply "A"/"B" versus "AB" varies according to neuron pair category. Finally, since the authors mention in the Discussion that "correlations did not depend on whether the two units preferred the same stimulus or different", it would be nice to explicitly show that in figure 5C by showing the orange trace ("A" alone or "B" alone) for both same (green) and different (brown) pairs separately.

We thank you for this interesting suggestion. We have conducted the desired analysis (see Author response table 1). The basic conclusions have not changed as described below.

Details: This analysis required one additional complexity, namely that the concepts of “congruent” and “incongruent” don’t translate cleanly to the single stimulus case. Consider an “incongruent” pair in which unit 1 prefers “A” and unit 2 prefers “B”. When “A” is presented, unit 1 will be strongly driven but unit 2 will not. When “B” is presented, the pattern will be the opposite. These two cases are thus more or less equivalent. However, for “congruent” pairs, when they both prefer “A” and “A” is the stimulus that is presented, the correlations could be quite different than when “B” is presented. Accordingly, for the “congruent” pairs, we subdivided according to whether the stimuli were the better (“driven”) or worse (“not driven”) of the two stimuli, and we focused the comparison on the difference between the congruent-driven and the incongruent spike count correlations. The bottom two rows of the Supplementary Table show the results for the single stimulus (A or B) conditions compared to the dual stimulus (AB) conditions.

Results: The basic pattern of a larger difference in the correlation patterns between the Congruent and Incongruent units for dual (AB) stimuli than for individual stimuli (A or B) holds across all the different categories of tested pairs. Compare the bottom two rows:

We have revised the wording in the Discussion to be consistent with the data as presented.

5. We are confused about the nature of Poisson models. If we are correct, the Poisson(a+b) is the sum of the two Poisson(a) and Poisson(b), that is, Poisson(a+b) = Poisson(a) + Poisson(b). Then, the mixture and intermediate models are very similar, identical if a*λ_A and (1-a)*λ_B happen to be integer numbers.

We thank the reviewers for bringing this confusion to our attention. We have modified Figure 2 to be more clear. The key distinction involves the rate parameter (λ) vs the draws from the Poisson distribution involving that rate parameter. This is most easily explained with an expanded illustration in Author response image 2:

Author response image 2

Download asset Open asset

These patterns (and more) are illustrated in Figure 2B, and we have clarified the associated equations to be readable in plain language – see the section now entitled “Spike count drawn from:” on the right hand side:

6. It is unclear why the 'outside' model predicts responses outside the range if neurons were to linearly sum the A and B responses.

We hope this is now more clear with the improved explanation above. Suppose a neuron’s average response to a stimulus “A” is 5 spikes and a stimulus “B” is 10 spikes. If “A” and “B” are both presented together and the neuron responds with 15 spikes, this is linear summation. We would define it as an “outside” response because 15 is not between 5 and 10. In short, linear summation is very rare in the V1 adjacent dataset, since we almost never observed “outside” classifications.

7. It is also unclear why the 'single' hypothesis would indicate a winner-take-all response. If we understand correctly, under this model, the response to A+B is either the rate A or B, but not the max between λ_A and λ_B. Also, this model could have given an extra free parameter to modulate its amplitude to the stimulus A+B.

Correct – “single” can also be “loser-take-all”. This is now stated on page 9 (“indicating a winner (or loser) -take-all response patterns”). Regarding an extra free parameter related to amplitude (of response?), we are not sure we follow. The distributions of spike counts observed over the set of trials involving the A, B, and AB stimuli are fully incorporated into the model comparison. Thus, response “amplitude”, meaning spike count, is already evaluated by the model.

8. The concept of "coarse population coding" can be misleading, as actual population coding can represent stimulus with quite good precision. The authors refer to the broad tuning of single cells, but this does not readily correspond to coarse population coding. This could be clarified.

We thank the reviewers for noting that clarification is needed on this issue. Our key point is that while codes of coarsely tuned units can produce very precise estimates of the value being encoded - c.f. excellent modeling work by Baldi and Heiligenberg (1988) – this has only been shown in circumstances where only one value is being encoded, such as in motor systems. We can only make one movement with a given body part at a given moment in time, whereas sensory systems must at least initially encode a broad array of stimuli.

We have revised the wording of the introductory paragraph to provide more emphasis on the “one movement” limitation of previous work involving coarse coding in motor systems (Introduction, paragraph 1: “However, individual motor systems only generate one movement at a time.”).Baldi P, Heiligenberg W (1988) How sensory maps could enhance resolution through ordered arrangements of broadly tuned receivers. Biological Cybernetics 59:313-318.

9. As a complement to the correlation analysis, one could check whether, on a trial-by-trial basis, the neuronal response of a single neuron is closer to the A+B response average, or to either the A or B responses. This would clearly indicate that the response fluctuates between representing A or B, or simultaneously represents A+B. I am trying to understand why this is not one of the main analyses of the paper instead of the correlation analysis, which involves two neurons instead of one.

We thank the reviewers for this suggestion and have added a new figure (Figure 9) to the manuscript. Details: At the individual cell level, the analysis suggested above is accomplished formally by the Bayesian model comparison presented in Figure 2 and subsequent figures – units classified as “mixtures” have responses on a trial-by-trial basis that alternate between A-like and B-like patterns.

Then, the coordinations in these fluctuations are assessed using the pair-wise correlation analysis.

What we now add is an example recording session showing the full pattern of fluctuations across a set of simultaneously recorded “mixture”-responding units. This example shows all of the main findings - individual units alternating between A- and B-like responses across trials, positive correlations between some pairs of units, negative correlation between others, and lack of correlation between yet others.

There is a bias in the overall representation – in this case, “A” is over represented compared to “B”.

However, on any given individual trial, there are some units responding in an “A-like” fashion and others in a “B-like” fashion (red and blue squares are present in every column).

10. In the discussion about noise correlations, the recent papers Nogueira et al., J Neuroscience, 2020 and Kafashan et al., Nat Comm, 2021 could be cited. Also, noise correlations can also be made time-dependent, so the distinction between the temporal correlation hypotheses and noise correlations might not be fundamental.

We appreciate having these excellent studies brought to our attention, and have included them in the list of citations. We concur that the temporal correlation and noise correlation distinctions may be a matter of techniques employed to date.

11. It would be interesting to study the effect of contrast on the mixed responses. Is it reasonable to predict that with higher contrast the mixture responses would be more dominant than the single ones? This could be the case if the selection mechanism has a harder time suppressing one of the object responses. This would also predict that noise correlations will go down with higher contrast.

This would be interesting but we do not have the data to address this question for this particular study. In a study that is currently in progress in the Groh lab, we have evidence in the inferior colliculus that when a visual stimulus is paired with one of two sounds, the neural responses are biased to the visually-paired sound in comparison with the non-visually-paired sound. This will be its own separate publication when we have completed the data collection in a second animal.

12. What is the time bin size used for the analysis? Would the results be the same if one focuses on the early time responses or on the late time responses? At least from the units shown in Figure 2, it looks that there is always an object response that is delayed respect to the other, so it would seem interesting to test noise correlations in those two temporal windows.

We apologize for the confusion, but the only time bin used in the current study is the 200-ms spike counting window. The examples shown in Figure 2 are spike count distribution plots, not PSTHs - the x axis is spike counts, not time. We have revised the title of Figure 2A to make this more clear.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

The manuscript has been improved and many of the reviewers' concerns have been addressed. However, some major issues remain. Although we typically avoid repeated revise/resubmit cycles, we believe that it is important for these issues to be addressed in a new revision. Specifically, upon discussion the reviewers unanimously remained concerned about the possibility that at least some of your results could be accounted for by subtle differences in eye movements. The new analyses related to that issue were appreciated but considered inadequate.<break />As detailed below, we would like you to provide:<break />1. More information about whether the electrodes that show evidence of multiplexing are the ones whose RF straddles the two stimuli, because in that case, small eye movements will bring one of the two stimuli inside the RF.

Among the 100 triplets classified as showing “mixture” response patterns with a winning probability of at least 0.67 (i.e. “mixture” is 2X as likely as all other possible classifications combined), we identified only 9 triplets exhibiting responses to both the A and B stimuli alone, as defined as a response greater than 1 SD above baseline.

As context, recall that one of the critical exclusionary screens for all the analyses in this paper is that a unit’s responses to the A and B stimuli must be sufficiently different that the Bayesian spike count distribution classification can be conducted. If the responses are too similar, it is not possible to evaluate whether a given spike count is more “A-like” or “B-like” etc. So, it is perhaps not very surprising that so few of our “mixtures” were responsive to both “A” and “B” stimuli.

Nevertheless, we proceeded with the additional analysis suggested below.

2. Further analyses of eye position to rule out the possibility described above. In our discussions, it was noted that the new Figure 1 – Supplementary Figure 1 appears to show that numerous V1 RFs straddle the two stimuli, and under those conditions, we really want to know if the "multiplexed" responses are because of small fixational differences/microsaccades that affect which of the two stimuli takes a more dominant position in the RF. To test for this kind of effect, just the STD of eye position per trial for one-stimulus vs two-stimuli conditions does not seem to be sufficient. Instead, it seems important to know whether, in the two-stimuli condition, responses were more "A-like" when gaze put the A stimulus closer to the RF center, and were more "B-like" when gaze put the B stimulus closer to the RF center. So something like a linear regression of spiking response versus "eye position along the axis defined by the centers of the two stimuli, increasing towards the stimulus that alone elicited the larger response" could be useful.

Of these 9 doubly-responsive triplet conditions identified above, only 1 showed a strongly statistically significant effect of fixational variation on the response patterns (linear regression as requested above, i.e. spiking response as a function of eye position along the axis defined by the centers of the two stimuli, p<0.01; two others showed borderline effects 0.05>p>0.025 whose potential significance did not survive when checked in a second multiple linear regression with both parallel and orthogonal dimensions of eye position included).

In short, the number of triplets involving any provable impact of eye position on the response patterns is a very small proportion of the total and does not impact the overall pattern of the results.

The outcome of these control experiments is summarized here:

Author response image 3

Download asset Open asset

Reviewer #1 (Recommendations for the authors):

The authors have made several changes in the manuscript to address previous concerns. However, the fact that typically only one stimulus spanned the RF makes it difficult to make a case for multiplexing, since the other stimulus is outside the classical RF. The arguments made by the authors in response to the previous RF-related question (point 2) are based on their own hypothesis about some underlying "spatial scale" of multiplexing which is coarser than the RF size of V1, which I do not find particularly convincing and would be extremely difficult to implement in V1. Further, while the authors showed more results related to eye position analysis, they do not show the key comparison that was requested previously.

It is true that the hypothesis about a coarser-than-V1-RF spatial scale is only a hypothesis, but it is consistent with the observations of our paper so we feel it is a reasonable speculation to offer. We hope future studies will be able to answer this question. The underlying mechanism(s) that might create these multiplexed signals – whether in V1 or elsewhere – likewise remains unknown.

To better appreciate the spatial scales involved, I refer to figures from the following two studies in V1 where very small stimuli (0.1 – 0.2 degrees) were used to map the RFs: Figure 2 of Xing et al., 2009, JNS, and Figure 2 of Dubey and Ray, 2016, JNP. Typically, the SD of fitted Gaussian is typically no more than 0.25 degrees at an eccentricity of 2-3 degrees (if you consider the radius as 2SD (0.5 degrees), the diameter is about a degree). For such RFs, there is no response if a stimulus is more than a degree away from the RF. For Gabor patches used in this paper, only the "size" is mentioned. Does size refer to the radius or SD? In either case, there is no way to fit two Gabors within the RF. What is the separation between two Gabors? Figure 1 should highlight all these details, including the radius (not just the center) of the V1 units.

1. We thank the reviewer for pointing out these omissions. We have clarified the text associated with Figure 1, and we now include the stimulus location details in an expanded Figure 1 Supplementary Figure 1, which has a new panel B:

In the main text, the legend to Figure 1 now clarifies the size/diameter issue and points explicitly to Figure 1 Supplementary Figure 1 for the full description:

“C. In the “superimposed” dataset, gratings were presented either individually or in combination at a consistent location and were large enough to cover the V1 and V4 receptive fields (stimulus diameter range: 2.5-7o,). The combined gratings appeared as a plaid (rightmost panel). Monkeys maintained fixation throughout stimulus presentation and performed no other task. D. In the V1 “adjacent” dataset, Gabor patches were smaller (typically ~1o,see Figure 1 Supplementary Figure 1).”

As for the receptive fields, these were mapped with stimuli of fixed sizes, permitting identification of the centers as shown in Supplementary Figure 1A, but the precise boundaries were not systematically evaluated as this would require varying the sizes of the mapping stimuli. This limitation reflects the original experimental needs of choosing satisfactory stimulus locations for performance of the behavioral task and optimizing the stimulus positions for the simultaneously recorded MT units (Ruff and Cohen 2016; MT data not analyzed here). Hence, we must rely on a rough sense inferred from a combination of the stimulus positions and the inclusion criteria: i.e. only units responsive to at least one of these stimuli were included for analysis.

2. We appreciate the elegant work of Xing et al., 2009 and Dubey and Ray, 2016 on receptive field mapping and have included citations to these studies.

Given the concern that one of the stimuli is always outside the RF, how do we explain the findings? One possible answer is in small differences in eye position. Multiplexing is anyway observed in only ~100 out of 1389 units. I suspect these are the units whose RF center is between the two stimuli. For the AB condition (i.e., both stimuli are presented), small jitters in eye position would bring one of the two stimuli in the RF, and therefore the unit would respond like either A or B. To address this, the authors should show the RF centers of the units that show evidence of multiplexing, along with the stimuli.

See analysis above. Only 9 (9%) of the “mixture”-classified units are receptive field straddlers. Note that to be included for analysis, units had to exhibit different responses to the A and B stimuli individually; this tended to limit the number of units responsive to both that were included in the final tally.

In addition, it is important to check for possible differences in eye position within AB conditions for trials for which responses were "A like" versus "B like". The authors have compared the AB condition to A and B conditions presented alone, but that is not enough. The argument that the stimuli were presented for a short duration and in pseudorandom order and therefore it is not possible for the animal to have systematically different eye positions for A/B versus AB conditions is obviously true, but that is not the point. The point is that the eye positions have small variations from trial to trial (as shown in Figure 1 – Supp 2) even before stimulus onset, and AB trials in which the position happens to be in one location get classified as "A like" and another location gets classified as "B like". It is in fact very hard to rule out this possibility given the instrument noise which affects the precision of eye positions, but it is crucial to rule this out.

We apologized for the previous confusion: we understood the previous concern to involve the very distinct patterns of spike count correlation exhibited on dual stimulus trials rather than the mixture classification itself. We hope the reviewer’s concern is now satisfactorily addressed by the analysis described above.

Reviewer #3 (Recommendations for the authors):

The authors have addressed my concerns and all other concerns raised in the editor's summary. My main concern was whether uncontrolled fixational eye movements (microsaccades) could account in part for the observed multiplexing. I understand now that concern was largely because of missing details about eye position, RF sizes, stimulus sizes, etc, which are now reported. The possibility remains that trial-by-changes in eye position (within the fixation window) would inflate the proportion of single-neuron "mixture" cases for adjacent two-object stimuli (by effectively changing which object is inside the RF in any given trial). But, importantly, this could not explain the observed patterns of noise correlations.

We thank you, and see above analyses for reassurance on the mixture-inflation question.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

The manuscript has been improved but there are some remaining issues that need to be addressed. We realized we were not as specific as we should have been in the last round of feedback and have tried to clarify here exactly what the reviewers are looking for.

We appreciate the efforts of the reviewers to help us improve our paper and make it as strong and clear as possible. Before we delve into the detailed results of the newly requested analyses – which we think the reviewers will find reassuring – we think it is worth re-capping the evidence as a whole regarding the main topic at issue, which we understand to be eye position/eye movement as a possible factor in the observations reported in this paper and the inferences we draw from those observations.

The observations in this paper fall into two overarching categories: (1) a “multiplexing” analysis, first introduced in Caruso et al. 2018 (involving recordings in an auditory brain area (the inferior colliculus) and a visual brain area (the face patch system of IT cortex); and (2) patterns of spike count (“noise”) correlations between pairs of simultaneously recorded neurons.

Our present paper shows that the pattern of spike count correlations in V1 is very different when two stimuli are presented vs when only one stimulus is presented – there are stronger correlations, in both positive and negative directions, than observed when only one stimulus is presented (Figure 4). Regarding contributions of eye position to this pattern, as discussed in earlier rounds or review, the single- and dual-stimulus conditions were randomly interleaved and trials with microsaccades excluded, so variation in eye position cannot account for this aspect of our findings (Figure 1 Supplementary Figure 2). Our understanding is that the reviewers are satisfied as to this point.

In the immediately prior and current rounds of review, concerns have centered further on the potential role of eye position in the “mixture” classification itself. For this to undermine our conclusions, such an effect would need to be sufficiently prevalent that any remaining results after excluding potentially confounded ones are insufficient to support the overall interpretations. Put simply, for correlation-witheye-position to be a confound, it would have to be some combination of big, common, systematic, and related not only to the mixture classification but also the spike count correlation. We think correlation with-eye-position fails as an explanation due to shortcomings in all of these realms.

First: how common is correlation with eye position? In the previous round, we were asked to consider a subset of the “mixtures”, those that were doubly-responsive to both stimuli. We showed that only about 11% of this subset had a significant effect of eye position evident in the response patterns. In the current round, reviewers requested that we extend this analysis to the full population of mixtures, which we have now done (detailed below). Again, only about 9% of the conditions labeled “mixture” exhibited a significant correlation with eye position. This compares with about 5% of the conditions not labeled as “mixtures” showing such an effect. The prevalence of eye position correlation in “mixtures” vs “non mixtures” is not significantly different by chi-square analysis (p=0.17).

Second: is correlation with eye position systematically related to stimulus location/receptive field location? The analyses requested by the reviewers (and presented in full below) answer this question definitively: the incidence of a correlation with eye position is not related to the relationship between RF location and stimulus location in any particularly obvious way.

Third: If we were to exclude any of the modest number of conditions that showed a correlation with eye position, the remaining observations in the manuscript would not change in any meaningful way. Indeed, our manuscript already shows that while the bimodal spike count correlation pattern is especially strong when both units in a pair are classified as “mixtures” (Figure 4) it is also seen other combinations of response patterns (Figure 5). This supports the interpretation that even if variation in eye position were to erroneously contribute to labeling of some conditions as “mixtures” when they shouldn’t be, this error would not meaningfully undermine the overall observations regarding spike count correlations differing on single vs dual stimulus conditions.

Zooming out to a larger context, as noted above, we have previously identified “mixture” response patterns in an auditory brain region and a visual brain region with much larger receptive fields than those in V1 (Caruso et al. 2018). Eye position within a small fixation window is unlikely to have affected the classification of response patterns in these areas. We also have numerous additional datasets (not yet published) that support “mixtures” as a general phenomenon. For example, we have seen “mixture” response pattern in datasets involving visual and auditory stimuli and recordings in the superior colliculus and prefrontal cortex, as well as additional data involving visual stimuli and area MT. Thus, we are quite confident that “mixtures” do not reflect some weird artifact or exceptional sensitivity to eye position that might hold true only for the present study.

We turn below to the specific analyses requested:

In particular, because the authors claim that ~100 electrodes have mixture responses, the possible influence of eye movements should be tested for all the mixture units, not only 9/100 as done in the previous review. We would therefore like the analyses done for all the mixtures. To avoid ambiguity, we are listing the requested analyses in more detail below:

1. Show the RF centers of ALL the units labeled as mixtures (~100). The best way to do this would be to make a line passing through the centers of the two stimuli, and make five groups of units depending on their RF centers – (i) left of the left stimulus, (ii) on the left stimulus, (iii), between left and right stimulus, (iv) on the right stimulus and (v) right of the right stimulus. Then show what proportion of units in each category are mixtures. The eye movement hypothesis predicts that mixtures will be predominant in (iii). Actually, since 91/100 units have one of the two modes at zero (as far as I could understand the previous analysis), these units could even be at (i) or (v).

2. Do the eye position analysis for ALL mixture units. Since the stimuli are mainly separated along the x-axis, all we are asking is to make a scatter plot of spike counts and average eyeX position for each trial and then check for correlation between the two. This analysis is valid even for units that only respond to one stimulus (i.e. the remaining 91 units). We expect to find a significant correlation (either positive or negative) if the results are due to small eye movements, but not if the firing is due to their multiplexing hypothesis. Or, if you find significant correlations since the RF sizes are comparable to the stimulus sizes, you need to show that these correlations are insufficient to explain the bimodality.

As requested in point #1, Figure 1—figure supplement 3 shows the RF centers of all* the units, aligned on one of the two stimuli for clarity (since the relationship between the two stimuli was relatively consistent, this is a reasonable simplification that allows for visualization). The data points indicate the RF centers of conditions labeled as mixtures (pink dots) and those labeled as non-mixtures (green dots); conditions that passed criterion for inclusion but did not achieve a high-confidence label in the spike-count distribution analysis are shown in yellow-orange.

Breaking these down into the groups as requested in the second part of point #1 above, we get:

Note that following the method requested by the reviewers in point #1, the “in” groups also include units with RF centers above or below the corresponding stimuli. We think it can be seen from the figure that the conclusion would not differ if we were to adjust these category boundaries to define “in” as inside the estimated circumference of the gabor patch stimuli.

Completing the requested analysis as described in point #2, a statistically significant relationship between spike counts and variation in eye position is observed in ~9% of units labeled “mixtures” (9/102), and ~5% of units not labeled mixtures (10/206). These proportions do not differ significantly (chi square test, p=0.17).

Altogether, we conclude that the correlation with eye position is insufficient to account for the bimodality in ~91% of the cases we labeled as “mixtures”. This is based on the following:

1. Correlation with eye position is rare (9%) and only somewhat more prevalent among “mixtures” vs.non mixtures (5%)

2. Mixtures-with-eye-position-correlation appear quite randomly distributed in space in relationship to RF center and stimulus location.

Methodological details: Note that there were no included units with RF centers to the right of the rightmost stimulus, because it was usually placed on or near the vertical meridian. Note also that some units were excluded from this analysis due to a failure to identify an RF center with high confidence for that electrode location. Finally, since the electrode arrays were fixed in position, RF centers did not move much on a day to day basis. Accordingly, for this analysis we used one estimate of the RF centers for each monkey and applied it across all sessions for that monkey.

Changes to manuscript: The analysis is described in the methods “Correlations between firing rates and scatter in fixation position were assessed for the dual stimulus trials using the component of eye position that lay along a line connecting the two stimulus locations chosen for the recording session (see Figure 1 – Supplementary Figure 3 for results). “ and text has been added to the second paragraph of the Results: “Finally, we assessed the responses of individual units to ascertain what proportion of units showed a correlation between firing rate and fixational scatter; this proportion was small overall (4-9%) and did not co-vary with the outcomes of the main analyses of the study (see Figure 1 – Supplementary Figure 3 for details) “

The larger pink and green bull’s eye circles indicate the conditions showing a correlation with eye position (p<0.01) among the mixtures (pink) and non mixtures (green).

3. In addition, I think the authors should at least show the typical RF radius of the units (make a circle of radius of 0.5 degrees on a few of the units, perhaps the ones shown in Figure 4). We think it is important to show that the RFs do not encompass both stimuli, which is not clear from the plots.

4. The fact that one mode of the bimodal distribution is zero in 91/100 cases should also be made clearer, perhaps in the methods section. Essentially the bimodal response in most cases is not A versus B, but A/B versus zero.

We agree.

Changes:

In the discussion, we have added the underlined sentence to paragraph 4 (with a little re-writing around it):

“It is interesting to note that we observed evidence of multiplexing each stimulus even in V1 where receptive fields are small and the stimuli we used did not themselves typically span more than one receptive field. Put another way, for most of these V1 “mixtures”, the observed fluctuations involved responding vs not responding rather than fluctuating between two different levels of responding. Thus the coarseness of tuning did not necessarily pose a problem for the encoding of these particular stimuli in this particular brain area, and yet fluctuations were observed. Thus, the precision of V1’s spatial code may not be the limiting factor. Multiplexing is likely to have some as yet unknown characteristic spatial scale that may be determined by the coarsest tuning evident at any stage in the sensory pathway. Future work in which stimuli are systematically varied to manipulate the amount of overlap in the activity patterns evoked in different brain areas by each stimulus alone are needed to answer this question. “

The numeric results are now presented in the legend to Figure 1 Supplementary Figure 3: “Finally, about 9% of “mixtures” (n=9) were responsive to both stimuli (i.e. RF centers were intermediate between the two stimuli and responses exceeded baseline firing by at least one standard deviation for both stimuli alone); among these 9 only 1 showed sensitivity to eye position (11%).”.

And we have added a sample RF size (0.5 degrees radius) to Figure 1 Supplementary Figure 1.

Author details

1. Na Young Jun

   1. Department of Neurobiology, Duke University, Durham, United States
   2. Center for Cognitive Neuroscience, Duke University, Durham, United States
   3. Duke Institute for Brain Sciences, Durham, United States

   Contribution

   Data curation, Software, Formal analysis, Validation, Visualization, Methodology, Writing – original draft, Writing – review and editing

   For correspondence

   nayoung.jun@duke.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8841-3947

2. Douglas A Ruff

   1. Department of Neuroscience, University of Pittsburgh, Pittsburgh, United States
   2. Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, United States

   Contribution

   Conceptualization, Resources, Data curation, Software, Validation, Investigation, Methodology, Writing – review and editing

   Competing interests

   No competing interests declared

3. Lily E Kramer

   1. Department of Neuroscience, University of Pittsburgh, Pittsburgh, United States
   2. Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, United States

   Contribution

   Data curation, Investigation

   Competing interests

   No competing interests declared

4. Brittany Bowes

   1. Department of Neuroscience, University of Pittsburgh, Pittsburgh, United States
   2. Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, United States

   Contribution

   Data curation, Investigation

   Competing interests

   No competing interests declared

5. Surya T Tokdar

   Department of Statistical Science, Duke University, Durham, United States

   Contribution

   Conceptualization, Resources, Software, Formal analysis, Supervision, Funding acquisition, Validation, Visualization, Project administration, Writing – review and editing

   Competing interests

   No competing interests declared

6. Marlene R Cohen

   1. Department of Neuroscience, University of Pittsburgh, Pittsburgh, United States
   2. Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, United States

   Contribution

   Conceptualization, Resources, Data curation, Software, Supervision, Funding acquisition, Investigation, Methodology, Project administration, Writing – review and editing

   Competing interests

   No competing interests declared

7. Jennifer M Groh

   1. Department of Neurobiology, Duke University, Durham, United States
   2. Center for Cognitive Neuroscience, Duke University, Durham, United States
   3. Duke Institute for Brain Sciences, Durham, United States
   4. Department of Psychology and Neuroscience, Duke University, Durham, United States
   5. Department of Biomedical Engineering, Duke University, Durham, United States
   6. Department of Computer Science, Duke University, Durham, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Supervision, Funding acquisition, Validation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing

   For correspondence

   jmgroh@duke.edu

   Competing interests

   Reviewing editor, eLife

   "This ORCID iD identifies the author of this article:" 0000-0002-6435-3935

• 1,034

  Page views

• 156

  Downloads

• 1

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.