Color depends on the spectral composition and spatial organization of lights (Brown and MacLeod, 1997; Kraft and Brainard, 1999; Monnier and Shevell, 2003; Shevell and Monnier, 2005; Xian and Shevell, 2004). The joint spectral and spatial statistics of lights in natural scenes are critical for object recognition, visual memory, and scene segmentation (Fine et al., 2003; Spence et al., 2006; Wurm et al., 1993). Despite its importance for these critical functions, however, the spatial integration of spectral signals by neurons in the visual system is poorly understood.

Some V1 neurons combine signals from all cone photoreceptor types (from which they receive any input at all) with the same sign. Many of these cone non-opponent neurons have receptive fields (RFs) consisting of side-by-side ON and OFF subfields, rendering them sensitive to luminance contrast of a particular orientation at a particular location in their RFs. Other V1 neurons combine signals from multiple cone types with opposite sign and are therefore cone-opponent. A subset of V1 cone-opponent neurons, called double-opponent (DO) cells, are both cone-opponent and spatially opponent, rendering them sensitive to chromatic edges of particular orientations at particular locations in their RFs (Conway, 2001; De and Horwitz, 2021a; Johnson et al., 2008). V1 is the first stage of the primate visual system at which DO cells exist (Hubel and Wiesel, 1968; Livingstone and Hubel, 1984).

V1 is also the first stage of the primate visual system that contains simple cells (Hubel and Wiesel, 1968). These cells, by definition, respond to a drifting achromatic sinusoidal grating of appropriate spatial- and temporal-frequency with spike rate modulation that exceeds the unmodulated response (Skottun et al., 1991). Applied to macaque V1 neurons, this definition likely includes many DO cells (simple cells were originally characterized in cat, an animal with few if any cone-opponent V1 neurons). We use the term ‘simple cell’ in this report to refer to phase-sensitive, orientation-tuned macaque V1 neurons that are not cone-opponent, so as to distinguish them from DO cells. Cone non-opponent simple cells have much in common with DO cells (De and Horwitz, 2021a). A major goal of the current study was to determine whether these two cell types integrate visual information across their RFs similarly.

Classically defined simple cells combine light information across their RFs approximately as a weighted sum (Carandini et al., 1997; DeAngelis et al., 1993; Hubel and Wiesel, 1959; Movshon et al., 1978b). These neurons can therefore be thought of as linear filters that operate on the retinal image. This insight propelled scientific progress in at least three ways. First, it facilitated the construction of image-computable models of achromatic image representation (Adelson and Bergen, 1991; Marr and Hildreth, 1980; Mehrotra et al., 1992). Second, it shed light on cortical circuitry: simple cells receive excitation (push) and inhibition (pull) with opposite spatial tuning, and this appears to be a critical step for establishing linearity in the face of nonlinear input from the lateral geniculate nucleus (LGN) (Ferster, 1988; Ferster and Miller, 2000; Hirsch et al., 1998; McLaughlin, 2000). Third, it provided an essential building block for quantitative models of V1 complex cells (Adelson and Bergen, 1985; Movshon et al., 1978a), neurons in higher-order cortical areas (Cadieu et al., 2007; Freeman et al., 2013; Okazawa et al., 2015; Rust et al., 2006; Simoncelli and Heeger, 1998; Willmore et al., 2010), and a variety of psychophysical phenomena (Adelson and Bergen, 1985; Beaudot and Mullen, 2006; Graham, 2011; Malik and Perona, 1990; Moreland and Boynton, 2017; Wilson et al., 1997). Currently, all of these advances have been restricted to the achromatic domain. A similar quantitative understanding of image representation in the chromatic domain is lacking and is necessary to extend these advances to natural, full-color images.

The nature of the spatial antagonism implemented by DO cells has important implications for their contributions to vision. For example, consider a hypothetical DO cell that compares cone-opponent signals between the left and right halves of its RF (Figure 1A). If the cell integrates signals linearly and with equal weight, then it is well suited for extracting vertical chromatic edges (Figure 1B–D). In this case, the excitation produced by a preferred light in one half of the RF can be canceled by the same light in the neighboring half. On the other hand, if the cell integrates signals nonlinearly, for example, by weighting contrast increments more heavily than contrast decrements, it would encode both edges and surfaces (Figure 1E and F). In this case, a light in one half of the RF would fail to cancel the same light appearing in the neighboring half.

Figure 1

Download asset Open asset

In this study, we measured the spatial integration of visual signals by individual V1 neurons in awake, fixating monkeys. Neurons can combine signals in many ways, but we focus on linear combinations because of their theoretical significance. Linearity is a mathematical ideal that is never fully realized by neurons, so we do not ask whether neurons integrate visual information across their RFs linearly but instead quantify how well a linear model describes spatial integration relative to a more flexible model. Models were fit to data collected using a closed-loop stimulus generation technique (Bölinger and Gollisch, 2012; Horwitz and Hass, 2012). The closed-loop technique involved the real-time construction of optimally oriented and positioned edges that varied in color but drove identical spike count responses. An advantage of this approach is that it is robust to static output nonlinearities.

Using this technique, we found that many DO cells responded linearly to differences in cone-opponent signals across their RFs, in quantitive similarity to how other V1 neurons responded to spatial differences in luminance. Nevertheless, a significant proportion of spatially opponent neurons combined signals across their RFs nonlinearly and in ways that could neither be attributed to a static output nonlinearity nor to a nonlinear combination of signals from linear subfields. These results contribute to our understanding of cone signal combination in V1 and reveal a population of DO cells that combine signals linearly across their RFs. We speculate that the linear DO cells provide input to complex cells, a proposal that explains several previous results and provides new insight into the functional role of color-sensitive V1 complex cells.

We analyzed the spiking responses of 98 V1 neurons from two awake, fixating male macaque monkeys (69 from Monkey 1, 29 from Monkey 2). Each neuron was visually stimulated with pixel white noise: a 10×10 grid of square, 0.2° pixels, each of which changed randomly and independently on every screen refresh. Data were analyzed by spike-triggered averaging to identify a pair of functionally distinct subfields within the classical RF. Visual stimulation was then targeted to these subfields to characterize their individual and joint contributions to the neuron’s firing rate.

RF characterization

Spike-triggered averages (STAs) of some neurons resembled uniform blobs, indicating invariant spectral sensitivity across the RF. Other STAs were spatially structured, for example, consisting of a set of yellow pixels adjacent to a set of blue pixels (Figure 2A). These structured STAs are a signature of neurons that compare spectral information across space. If an STA did not reveal at least two distinct subfields, the neuron was passed over for data collection.

Figure 2 with 5 supplements see all

Download asset Open asset

A major goal of these experiments was to characterize spatial integration within the RFs of DO cells. The pixel white noise stimulus is ill-suited for achieving this goal because it modulates V1 neurons weakly, and nonlinear spatial integration might appear linear in response to small perturbations. To drive the neurons more effectively, we customized the stimulus for each neuron studied. From the pixel white noise STA, we identified two contiguous groups of pixels, each covering an RF subfield, and yoked each group into a ‘hyperpixel’ (Figure 2B). White noise modulation of the two hyperpixels stimulated the two subfields strongly and independently (Figure 2—figure supplement 1), driving a wide range of firing rates (Figure 2—figure supplement 2).

On the basis of responses to the hyperpixel white noise stimulus, neurons with spatially opponent RFs were classified into three categories. The largest category consisted of neurons with a significant excitatory stimulus feature that was distinct from the STA (n=35, see Material and methods). Some of these neurons may have been complex cells, a classification that is inconsistent with the classical definitions of simple cells and DO cells. These neurons were therefore given the designation ‘other spatially opponent‘ (OSO). For the 63 cells not classified as OSO by this criterion, cone weights were computed and analyzed. An additional 12 neurons with cone weights insufficient for a decisive cone-opponent or non-opponent classification were also classified as OSO. The remaining 51 neurons were divided into DO cells and simple cells on their basis of cone-opponency. Altogether, 26 neurons were classified as simple cells, 25 were classified as DO cells, and 47 were classified as OSO cells.

The two sides of the hyperpixel STAs from simple cells and DO cells were complementary, or nearly so. This complementarity was evident via a comparison between the background-subtracted red, green, and blue phosphor intensities at the two hyperpixels (Pearson’s r=–0.94±0.11 (mean ± SD) for simple cells and –0.76±0.23 for DO cells), and between cone weights derived from these RGB values (Figure 2—figure supplement 3). The results of these analyses are color space-dependent, but the fact that similar results were obtained in the physiologically relevant space of cone contrast and the device-specific space in which sampling errors are independent across color channels, attests to the spatial opponency of simple cells and DO cells in our data set.

Measuring spatial integration using the isoresponse method

For each neuron, we found a collection of stimuli that evoked the same response. Each stimulus was spatially identical to the hyperpixel STA, but the contrast of the two hyperpixels was adjusted according to the algorithm described in Materials and methods: Contrast staircase procedure. Negative contrasts were allowed.

To appreciate the necessity of this technique, consider a classical alternative. A classic test of linearity is to present one stimulus at the RF of a recorded neuron, then another, and then both together. If the response to the combined stimulus does not equal the sum of responses to the two components, the neuron is not linear. However, this test is sensitive to nonlinearities that are logically distinct from the linearity of spatial summation and are present in otherwise linear neurons (e.g., spike firing thresholds and saturating contrast-response functions). An alternative approach is to find a collection of stimuli that evoke the same response from an isolated neuron and analyze these stimuli to identify the features they share. This approach has been used previously to study signal integration in the salamander retina and locust auditory receptor cells (Bölinger and Gollisch, 2012; Gollisch et al., 2002). It has also been used previously to analyze the linearity of signal integration across cone types by individual macaque V1 neurons (Horwitz and Hass, 2012), but it has not been used previously to analyze the linearity of signal integration across a V1 RF.

If a neuron combines cone-contrast signals linearly across its RF, then stimuli that drive the same response will lie on lines when represented in the stimulus space shown in Figure 2C or any other stimulus space that is a linear transformation away from it (e.g., cone excitation difference or opponent modulation spaces [Brainard, 1996]). If the stimuli lie on a curve instead of a line, the hypothesis of linear spatial summation can be rejected. This approach makes no assumptions about static nonlinearities downstream of spatial integration whereas the GLM and GQM assumed a logistic function. It also does not assume linearity of cone signal integration within individual RF subfields. This assumption was used in the GLM and GQM analyses to reduce RGB values from 15-frame-long stimulus movie fragments to two stimulus projections.

In Figure 3A, each point represents a stimulus, distance from the origin represents contrast relative to the background, and angle represents contrast between the two sides of the stimulus. Within this plane, a search was performed to find physically distinct stimuli that evoked the same neuronal response. Angles were selected pseudo-randomly, and distances were titrated by a staircase procedure until a target firing rate was achieved (Figure 3—figure supplement 1). To mitigate the impact of spontaneous activity on the staircase procedure, the target firing rate for each neuron exceeded the 90th percentile of the baseline firing rate distribution, and for 95/98 neurons, the target firing rate exceeded the 95th percentile (Figure 3—figure supplement 2). Target firing rates did not differ across cell types (p=0.63, Kruskal-Wallis test).

Figure 3 with 5 supplements see all

Download asset Open asset

For some neurons, staircase termination points lay close to a line when plotted in the stimulus space (Figure 3A–B & Figure 3—figure supplement 3). This result shows that the excitation produced by a preferred light in one part of the RF can be canceled by an anti-preferred light in a neighboring part with a fixed constant of proportionality over the entire gamut of our video display. This cancelation is consistent with linearity of spatial integration (Figure 1A) and not with differential sensitivity to contrast increments and decrements (Figure 1B). However, not all neurons behaved this way. For some neurons, staircase termination points lay on a curve (Figure 3C), consistent with nonlinear spatial integration.

To determine quantitatively whether a line or a curve provided the better description of the staircase termination points, we compared linear and quadratic models fits for each neuron (see Materials and methods: Evaluating model fits to staircase termination points; Figure 3—figure supplement 4). We defined an isoresponse nonlinearity index (isoresponse NLI) similarly to the white noise NLI defined previously (see Materials and methods: Evaluating model fits to staircase termination points). An isoresponse NLI of 0 indicates that the linear and quadratic models made equally accurate response predictions, NLI<0 indicates that the linear model predicted responses more accurately than the quadratic model, and NLI>0 indicates that the quadratic model predicted responses more accurately than the linear model. Cross-validation ensured that the quadratic model did not achieve greater prediction accuracy simply by virtue of having more parameters.

Isoresponse NLIs of DO cells and simple cells were close to zero and did not differ significantly (median isoresponse NLI for DO cells=0.1007, median isoresponse NLI for simple cells=–0.0097; p=0.14, Mann-Whitney U-test; Figure 3D). In contrast, NLIs were greater for OSO neurons (median isoresponse NLI=0.2822, p=0.02, Kruskal-Wallis test). We conclude that neurons that we classified as simple cells and DO cells are similarly linear over the range that we were able to test given the limits of our display and that they are more linear than other neurons in V1.

Isoresponse NLIs (from Phase 3 of the experiment) were positively correlated with white noise NLIs (from Phase 2 of the experiment) (r=0.30, p=0.001, Spearman’s correlation; Figure 3E). This correlation was driven primarily by OSO cells (r=0.41, p=0.004,) and not by DO (r=0.01, p=0.95,) or simple cells (r=–0.19, p=0.34). Even for the OSO cells, however, this correlation was far from perfect. Many neurons had white noise NLIs near 0 but isoresponse NLIs>>0. The reason for this discrepancy is unclear but is consistent with the higher contrasts used in Phase 3 engaging nonlinear mechanisms that were not engaged in Phase 2. However, we cannot rule out the possibility that isoresponse NLIs were inflated by nonstationarity in firing rate combined with the sequential measurement procedure used in Phase 3, or by other differences in the data collection or analysis procedures.

Measuring signal integration within individual subfields

An appreciable fraction of neurons in our data set, principally those in the OSO category, were poorly described by the linear model. We asked whether the nonlinear model shown in Figure 1E described these neurons more accurately. Under this model, signals from the three types of cone photoreceptors combine linearly within each RF subfield and are then transformed nonlinearly prior to spatial integration.

We tested the assumption of linear combination within a subfield by regressing spikes recorded during Phase 2 onto linear and nonlinear combinations of the three display primaries within each subfield. If signals from multiple cone types combine linearly within each RF subfield, the influence of that subfield on the spiking response can be summarized by a weighted sum of red, green, and blue primary intensities.

In this analysis, we treat the influence of each subfield as additive noise on the signal produced by the other. This approach is justified by the facts that the two hyperpixels in Phase 2 modulated independently, and under the model, each subfield makes an additive contribution to the firing rate (Figure 4A).

Figure 4

Download asset Open asset

For each neuron, we fit four models (two models × two hyperpixels) to the hyperpixel white noise data: a GLM and a GQM that predicted spiking responses as a function of the background-subtracted RGBs at each hyperpixel (Figure 4). We computed model performance using ROC analysis and quantified the difference in performance of the GLM and GQM with an NLI (see Materials and methods: Hyperpixel white noise analysis of signal combination within subfields). Model performance was averaged across hyperpixels to provide a single ‘within-subfield’ NLI per neuron.

Within-subfield NLIs differed across the three cell types (p<0.0001, Kruskal-Wallis test; Figure 5). On average, within-subfield NLIs were higher for OSO cells than for simple or DO cells (median within-subfield NLI for simple cells=–0.0016, for DO cells=–0.0003, for OSO cells=0.0047, p<0.0001, Kruskal-Wallis test). This result extends to individual RF subfields the observation that simple cells and DO cells combine signals with a similar degree of linearity and shows that the nonlinear signal integration exhibited by many OSO cells is poorly described as resulting from a nonlinear combination of two linear RF subfields.

Figure 5

Download asset Open asset

Relationship between S-cone input and signal integration across and within subfields

Many V1 neurons that are driven strongly by modulations of the S-cones are poorly described by a linear model (Horwitz et al., 2005). Consistent with this observation, we found that the absolute value of the normalized S-cone weight derived from the hyperpixel STA was correlated with isoresponse NLI (r=0.23, p=0.02, Spearman’s correlation; Figure 6). However, S-cone weight did not correlate significantly with the other two NLIs (white noise and within-subfield, both r<0.2, p>0.05).

Figure 6

Download asset Open asset

A fundamental goal of visual neuroscience is to characterize the transformation from lights to the spiking responses of individual neurons. In this study, we characterized the spatial integration of light by neurons in area V1 of macaques with two types of visual white noise stimulation, closed-loop isoresponse measurements, and statistical model comparisons. We found that some DO cells integrate signals across their RFs linearly, like cone non-opponent simple cells do and in contrast to other V1 neurons. We did not find strong evidence for V1 neurons that pool information linearly across cone photoreceptor types within a subfield and then nonlinearly across subfields.

Below, we compare our results to those from previous studies and discuss how our results are affected by cell classification criteria and the monkeys’ eye movements. We then discuss the implications of our results for the circuitry that underlies DO and simple cells and how these two cell types may contribute to downstream image processing. We conclude with speculations on parallels between the processing of color and other stimulus features in V1 by complex cells.

Contributions to downstream processing

Simple cells are thought to provide the dominant input to complex cells (Alonso and Martinez, 1998; Hubel and Wiesel, 1962). Under a standard model, complex cells pool signals from simple cells with overlapping RFs and shared preferred orientation (Figure 7A). One possibility is that some complex cells also receive input from DO cells (Figure 7B). We speculate that complex cells receiving (cone non-opponent) simple cell input only are luminance-sensitive (Figure 7C) whereas those that receive input from simple and DO cells are both color- and luminance-sensitive (Figure 7D). This conjecture is consistent with the observation that the preferred orientation of color-sensitive complex cells is maintained across color directions (Johnson et al., 2001). It is also consistent with the observation that color-sensitive complex cells have multiple preferred color directions by STC analysis (Horwitz et al., 2007) and no null directions in cone-contrast space (Horwitz and Hass, 2012). Complex cells that are insensitive to chromatic modulation abound (e.g., Figure 7C), whereas those that are insensitive to luminance modulation are exceedingly rare, if present (Horwitz et al., 2007).

Figure 7

Download asset Open asset

Alternatively, the chromatic sensitivity of color-sensitive complex cells could arise entirely from DO cell inputs (Michael, 1978), which have been reported to be similarly responsive to chromatic and luminance contrast (Johnson et al., 2001; Johnson et al., 2004; Johnson et al., 2008). Yet another possibility is that it arises directly from LGN inputs. This explanation is consistent with the direct koniocellular projections to the cytochrome oxidase blobs in area V1 (Xiao, 2014), but is difficult to reconcile with the sharp orientation tuning and spatial phase-invariance of some color-sensitive complex cells across color directions. These visual tuning properties would seem to require a phase- and orientation-selective upstream intermediate or else elaborate dendritic computations.

All data associated with this study are available at https://github.com/horwitzlab/Chromatic_spatial_contrast (copy archived at swh:1:rev:fe8d51dc732fdc0336532ecc47cb7b18c01dc8cf).

1. 1. De and Horwitz

   (2021) Github

   ID fe8d51d. Chromatic spatial contrast.

   https://github.com/horwitzlab/Chromatic_spatial_contrast

1. 1. Adelson EH
   2. Bergen JR

   (1985) Spatiotemporal energy models for the perception of motion

   Journal of the Optical Society of America. A, Optics and Image Science 2:284–299.

   https://doi.org/10.1364/josaa.2.000284

   • PubMed
   • Google Scholar
2. Book

   1. Adelson EH
   2. Bergen JR

   (1991)

   The Plenoptic Function and the Elements of Early Vision

   In: Landy MS, Movshon JA, editors. Computational Models of Visual Processing. The MIT Press. pp. 3–20.

   • Google Scholar
3. 1. Alonso JM
   2. Martinez LM

   (1998) Functional connectivity between simple cells and complex cells in cat striate cortex

   Nature Neuroscience 1:395–403.

   https://doi.org/10.1038/1609

   • PubMed
   • Google Scholar
4. 1. Anzai A
   2. Ohzawa I
   3. Freeman RD

   (1999) Neural mechanisms for processing binocular information I. Simple cells

   Journal of Neurophysiology 82:891–908.

   https://doi.org/10.1152/jn.1999.82.2.891

   • PubMed
   • Google Scholar
5. 1. Barnett MA
   2. Aguirre GK
   3. Brainard DH

   (2021) A quadratic model captures the human V1 response to variations in chromatic direction and contrast

   eLife 10:e65590.

   https://doi.org/10.7554/eLife.65590

   • PubMed
   • Google Scholar
6. 1. Beaudot WHA
   2. Mullen KT

   (2006) Orientation discrimination in human vision: psychophysics and modeling

   Vision Research 46:26–46.

   https://doi.org/10.1016/j.visres.2005.10.016

   • PubMed
   • Google Scholar
7. 1. Bölinger D
   2. Gollisch T

   (2012) Closed-loop measurements of iso-response stimuli reveal dynamic nonlinear stimulus integration in the retina

   Neuron 73:333–346.

   https://doi.org/10.1016/j.neuron.2011.10.039

   • PubMed
   • Google Scholar
8. Book

   1. Brainard DH

   (1996)

   Cone contrast and opponent modulation color spaces

   In: Kaiser PK, Boynton RM, editors. Human Color Vision. Washington, DC: Optical Society of America. pp. 563–578.

   • Google Scholar
9. 1. Brainard DH

   (1997)

   The Psychophysics Toolbox

   Spatial Vision 10:433–436.

   • PubMed
   • Google Scholar
10. 1. Brown RO
    2. MacLeod DI

    (1997) Color appearance depends on the variance of surround colors

    Current Biology 7:844–849.

    https://doi.org/10.1016/s0960-9822(06)00372-1

    • PubMed
    • Google Scholar
11. 1. Browne MW

    (2000) Cross-validation methods

    Journal of Mathematical Psychology 44:108–132.

    https://doi.org/10.1006/jmps.1999.1279

    • PubMed
    • Google Scholar
12. 1. Cadieu C
    2. Kouh M
    3. Pasupathy A
    4. Connor CE
    5. Riesenhuber M
    6. Poggio T

    (2007) A model of V4 shape selectivity and invariance

    Journal of Neurophysiology 98:1733–1750.

    https://doi.org/10.1152/jn.01265.2006

    • PubMed
    • Google Scholar
13. 1. Carandini M
    2. Heeger DJ
    3. Movshon JA

    (1997)

    Linearity and normalization in simple cells of the macaque primary visual cortex

    The Journal of Neuroscience 17:8621–8644.

    • PubMed
    • Google Scholar
14. 1. Carandini M
    2. Demb JB
    3. Mante V
    4. Tolhurst DJ
    5. Dan Y
    6. Olshausen BA
    7. Gallant JL
    8. Rust NC

    (2005) Do we know what the early visual system does?

    The Journal of Neuroscience 25:10577–10597.

    https://doi.org/10.1523/JNEUROSCI.3726-05.2005

    • PubMed
    • Google Scholar
15. 1. Carandini M

    (2006) What simple and complex cells compute

    The Journal of Physiology 577:463–466.

    https://doi.org/10.1113/jphysiol.2006.118976

    • PubMed
    • Google Scholar
16. Book

    1. Cavanagh P

    (1991)

    Vision at Equiluminance

    In: Kulikowski JJ, Murray IJ, Walsh V, editors. ViIision and Visual Dysfunction: Limits of Vision, Vol. 5. Boca Raton, FL: CLC Press. pp. 234–250.

    • Google Scholar
17. 1. Chichilnisky EJ

    (2001)

    A simple white noise analysis of neuronal light responses

    Network (Bristol, England) 12:199–213.

    • PubMed
    • Google Scholar
18. 1. Chichilnisky EJ
    2. Kalmar RS

    (2002)

    Functional asymmetries in ON and OFF ganglion cells of primate retina

    The Journal of Neuroscience 22:2737–2747.

    • Google Scholar
19. 1. Conway BR

    (2001) Spatial structure of cone inputs to color cells in alert macaque primary visual cortex (V-1)

    The Journal of Neuroscience 21:2768–2783.

    https://doi.org/10.1523/JNEUROSCI.21-08-02768.2001

    • PubMed
    • Google Scholar
20. 1. Conway BR
    2. Hubel DH
    3. Livingstone MS

    (2002) Color contrast in macaque V1

    Cerebral Cortex 12:915–925.

    https://doi.org/10.1093/cercor/12.9.915

    • PubMed
    • Google Scholar
21. 1. Conway BR
    2. Livingstone MS

    (2006) Spatial and temporal properties of cone signals in alert macaque primary visual cortex

    The Journal of Neuroscience 26:10826–10846.

    https://doi.org/10.1523/JNEUROSCI.2091-06.2006

    • Google Scholar
22. 1. Cumming BG
    2. Parker AJ

    (1997) Responses of primary visual cortical neurons to binocular disparity without depth perception

    Nature 389:280–283.

    https://doi.org/10.1038/38487

    • Google Scholar
23. 1. De A
    2. Horwitz GD

    (2021a) Spatial receptive field structure of double-opponent cells in macaque V1

    Journal of Neurophysiology 125:843–857.

    https://doi.org/10.1152/jn.00547.2020

    • PubMed
    • Google Scholar
24. Software

    1. De A
    2. Horwitz GD

    (2021b) Chromatic spatial contrast, version swh:1:rev:fe8d51dc732fdc0336532ecc47cb7b18c01dc8cf

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:acecbbaeedd2a4a3c87d1144318daf0b48253a1c;origin=https://github.com/horwitzlab/Chromatic_spatial_contrast;visit=swh:1:snp:96e3b7ea8274702a72e3d11c40164c8dd46f55bd;anchor=swh:1:rev:fe8d51dc732fdc0336532ecc47cb7b18c01dc8cf
25. 1. DeAngelis GC
    2. Ohzawa I
    3. Freeman RD

    (1993) Spatiotemporal organization of simple-cell receptive fields in the cat’s striate cortex. II. Linearity of temporal and spatial summation

    Journal of Neurophysiology 69:1118–1135.

    https://doi.org/10.1152/jn.1993.69.4.1118

    • PubMed
    • Google Scholar
26. 1. Eckart C
    2. Young G

    (1936) The approximation of one matrix by another of lower rank

    Psychometrika 1:211–218.

    https://doi.org/10.1007/BF02288367

    • Google Scholar
27. Book

    1. Efron B
    2. Tibshirani RJ

    (1994) An Introduction to the Bootstrap

    Boca Raton: CRC Press.

    https://doi.org/10.1201/9780429246593

    • Google Scholar
28. 1. Ferster D

    (1988) Spatially opponent excitation and inhibition in simple cells of the cat visual cortex

    The Journal of Neuroscience 8:1172–1180.

    https://doi.org/10.1523/JNEUROSCI.08-04-01172.1988

    • PubMed
    • Google Scholar
29. 1. Ferster D

    (1994) Linearity of synaptic interactions in the assembly of receptive fields in cat visual cortex

    Current Opinion in Neurobiology 4:563–568.

    https://doi.org/10.1016/0959-4388(94)90058-2

    • PubMed
    • Google Scholar
30. 1. Ferster D
    2. Miller KD

    (2000) Neural mechanisms of orientation selectivity in the visual cortex

    Annual Review of Neuroscience 23:441–471.

    https://doi.org/10.1146/annurev.neuro.23.1.441

    • PubMed
    • Google Scholar
31. 1. Fine I
    2. MacLeod DIA
    3. Boynton GM

    (2003) Surface segmentation based on the luminance and color statistics of natural scenes

    Journal of the Optical Society of America. A, Optics, Image Science, and Vision 20:1283–1291.

    https://doi.org/10.1364/josaa.20.001283

    • PubMed
    • Google Scholar
32. Book

    1. Fox J

    (2002)

    Robust Regression: An R and S-Plus Companion to Applied Regression

    Thousand Oaks, CA: Sage Publications.

    • Google Scholar
33. 1. Freeman J
    2. Ziemba CM
    3. Heeger DJ
    4. Simoncelli EP
    5. Movshon JA

    (2013) A functional and perceptual signature of the second visual area in primates

    Nature Neuroscience 16:974–981.

    https://doi.org/10.1038/nn.3402

    • PubMed
    • Google Scholar
34. 1. Gollisch T
    2. Schütze H
    3. Benda J
    4. Herz AVM

    (2002) Energy integration describes sound-intensity coding in an insect auditory system

    The Journal of Neuroscience 22:10434–10448.

    https://doi.org/10.1523/JNEUROSCI.22-23-10434.2002

    • PubMed
    • Google Scholar
35. 1. Graham NV

    (2011) Beyond multiple pattern analyzers modeled as linear filters (as classical V1 simple cells): useful additions of the last 25 years

    Vision Research 51:1397–1430.

    https://doi.org/10.1016/j.visres.2011.02.007

    • PubMed
    • Google Scholar
36. Book

    1. Green DM
    2. Swets JA

    (1966)

    Signal Detection Theory and Psychophysics.

    New York: Wiley.

    • Google Scholar
37. 1. Haefner RM
    2. Cumming BG

    (2008) Adaptation to natural binocular disparities in primate V1 explained by a generalized energy model

    Neuron 57:147–158.

    https://doi.org/10.1016/j.neuron.2007.10.042

    • PubMed
    • Google Scholar
38. 1. Hirsch JA
    2. Alonso JM
    3. Reid RC
    4. Martinez LM

    (1998) Synaptic integration in striate cortical simple cells

    The Journal of Neuroscience 18:9517–9528.

    https://doi.org/10.1523/JNEUROSCI.18-22-09517.1998

    • PubMed
    • Google Scholar
39. 1. Horwitz GD
    2. Albright TD

    (2005) Paucity of chromatic linear motion detectors in macaque V1

    Journal of Vision 5:525–533.

    https://doi.org/10.1167/5.6.4

    • PubMed
    • Google Scholar
40. 1. Horwitz GD
    2. Chichilnisky EJ
    3. Albright TD

    (2005) Blue-yellow signals are enhanced by spatiotemporal luminance contrast in macaque V1

    Journal of Neurophysiology 93:2263–2278.

    https://doi.org/10.1152/jn.00743.2004

    • PubMed
    • Google Scholar
41. 1. Horwitz GD
    2. Chichilnisky EJ
    3. Albright TD

    (2007) Cone inputs to simple and complex cells in V1 of awake macaque

    Journal of Neurophysiology 97:3070–3081.

    https://doi.org/10.1152/jn.00965.2006

    • PubMed
    • Google Scholar
42. 1. Horwitz GD
    2. Hass CA

    (2012) Nonlinear analysis of macaque V1 color tuning reveals cardinal directions for cortical color processing

    Nature Neuroscience 15:913–919.

    https://doi.org/10.1038/nn.3105

    • PubMed
    • Google Scholar
43. 1. Hubel DH
    2. Wiesel TN

    (1959) Receptive fields of single neurones in the cat’s striate cortex

    The Journal of Physiology 148:574–591.

    https://doi.org/10.1113/jphysiol.1959.sp006308

    • PubMed
    • Google Scholar
44. 1. Hubel DH
    2. Wiesel TN

    (1962) Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex

    The Journal of Physiology 160:106–154.

    https://doi.org/10.1113/jphysiol.1962.sp006837

    • PubMed
    • Google Scholar
45. 1. Hubel DH
    2. Wiesel TN

    (1968) Receptive fields and functional architecture of monkey striate cortex

    The Journal of Physiology 195:215–243.

    https://doi.org/10.1113/jphysiol.1968.sp008455

    • PubMed
    • Google Scholar
46. 1. Johnson EN
    2. Hawken MJ
    3. Shapley R

    (2001) The spatial transformation of color in the primary visual cortex of the macaque monkey

    Nature Neuroscience 4:409–416.

    https://doi.org/10.1038/86061

    • PubMed
    • Google Scholar
47. 1. Johnson EN
    2. Hawken MJ
    3. Shapley R

    (2004) Cone inputs in macaque primary visual cortex

    Journal of Neurophysiology 91:2501–2514.

    https://doi.org/10.1152/jn.01043.2003

    • PubMed
    • Google Scholar
48. 1. Johnson EN
    2. Hawken MJ
    3. Shapley R

    (2008) The orientation selectivity of color-responsive neurons in macaque V1

    The Journal of Neuroscience 28:8096–8106.

    https://doi.org/10.1523/JNEUROSCI.1404-08.2008

    • PubMed
    • Google Scholar
49. 1. Kingdom FAA

    (2003) Color brings relief to human vision

    Nature Neuroscience 6:641–644.

    https://doi.org/10.1038/nn1060

    • Google Scholar
50. Conference

    1. Kleiner M
    2. Brainard DH
    3. Pelli DG

    (2007)

    What’s new in Psychtoolbox-3?

    Perception 36 ECVP Abstract Supplement.

    • Google Scholar
51. 1. Kraft JM
    2. Brainard DH

    (1999) Mechanisms of color constancy under nearly natural viewing

    PNAS 96:307–312.

    https://doi.org/10.1073/pnas.96.1.307

    • PubMed
    • Google Scholar
52. 1. Kunsberg B
    2. Holtmann-Rice D
    3. Alexander E
    4. Cholewiak S
    5. Fleming R
    6. Zucker SW

    (2018) Colour, contours, shading and shape: flow interactions reveal anchor neighbourhoods

    Interface Focus 8:20180019.

    https://doi.org/10.1098/rsfs.2018.0019

    • PubMed
    • Google Scholar
53. 1. Livingstone MS
    2. Hubel DH

    (1984)

    Anatomy and physiology of a color system in the primate visual cortex

    The Journal of Neuroscience 4:309–356.

    • PubMed
    • Google Scholar
54. 1. Malik J
    2. Perona P

    (1990) Preattentive texture discrimination with early vision mechanisms

    Journal of the Optical Society of America. A, Optics and Image Science 7:923–932.

    https://doi.org/10.1364/josaa.7.000923

    • PubMed
    • Google Scholar
55. 1. Marr D
    2. Hildreth E

    (1980) Theory of edge detection

    PNAS 207:187–217.

    https://doi.org/10.1098/rspb.1980.0020

    • PubMed
    • Google Scholar
56. 1. McLaughlin D

    (2000)

    A neuronal network model of sharpening and dynamics of orientation tuning in an input layer of macaque primary visual cortex

    PNAS 97:8087–8092.

    • Google Scholar
57. 1. Mehrotra R
    2. Namuduri KR
    3. Ranganathan N

    (1992) Gabor filter-based edge detection

    Pattern Recognition 25:1479–1494.

    https://doi.org/10.1016/0031-3203(92)90121-X

    • Google Scholar
58. 1. Michael CR

    (1978) Color vision mechanisms in monkey striate cortex: dual-opponent cells with concentric receptive fields

    Journal of Neurophysiology 41:572–588.

    https://doi.org/10.1152/jn.1978.41.3.572

    • PubMed
    • Google Scholar
59. 1. Monnier P
    2. Shevell SK

    (2003) Large shifts in color appearance from patterned chromatic backgrounds

    Nature Neuroscience 6:801–802.

    https://doi.org/10.1038/nn1099

    • PubMed
    • Google Scholar
60. 1. Moreland JC
    2. Boynton GM

    (2017) A neurophysiological explanation for biases in visual localization

    Attention, Perception, & Psychophysics 79:553–562.

    https://doi.org/10.3758/s13414-016-1251-z

    • PubMed
    • Google Scholar
61. 1. Movshon JA
    2. Thompson ID
    3. Tolhurst DJ

    (1978a) Receptive field organization of complex cells in the cat’s striate cortex

    The Journal of Physiology 283:79–99.

    https://doi.org/10.1113/jphysiol.1978.sp012489

    • PubMed
    • Google Scholar
62. 1. Movshon JA
    2. Thompson ID
    3. Tolhurst DJ

    (1978b) Spatial summation in the receptive fields of simple cells in the cat’s striate cortex

    The Journal of Physiology 283:53–77.

    https://doi.org/10.1113/jphysiol.1978.sp012488

    • PubMed
    • Google Scholar
63. 1. Ohzawa I
    2. Freeman RD

    (1986) The binocular organization of simple cells in the cat’s visual cortex

    Journal of Neurophysiology 56:221–242.

    https://doi.org/10.1152/jn.1986.56.1.221

    • PubMed
    • Google Scholar
64. 1. Okazawa G
    2. Tajima S
    3. Komatsu H

    (2015) Image statistics underlying natural texture selectivity of neurons in macaque V4

    PNAS 112:E351–E360.

    https://doi.org/10.1073/pnas.1415146112

    • PubMed
    • Google Scholar
65. Conference

    1. Paninski L

    (2003)

    Convergence properties of some spike-triggered analysis techniques

    Advances in Neural Information Processing Systems. pp. 189–196.

    • Google Scholar
66. 1. Pelli DG

    (1997)

    The VideoToolbox software for visual psychophysics: transforming numbers into movies

    Spatial Vision 10:437–442.

    • PubMed
    • Google Scholar
67. 1. Read JCA
    2. Cumming BG

    (2003) Testing quantitative models of binocular disparity selectivity in primary visual cortex

    Journal of Neurophysiology 90:2795–2817.

    https://doi.org/10.1152/jn.01110.2002

    • PubMed
    • Google Scholar
68. 1. Rust NC
    2. Schwartz O
    3. Movshon JA
    4. Simoncelli EP

    (2005) Spatiotemporal elements of macaque v1 receptive fields

    Neuron 46:945–956.

    https://doi.org/10.1016/j.neuron.2005.05.021

    • PubMed
    • Google Scholar
69. 1. Rust NC
    2. Mante V
    3. Simoncelli EP
    4. Movshon JA

    (2006) How MT cells analyze the motion of visual patterns

    Nature Neuroscience 9:1421–1431.

    https://doi.org/10.1038/nn1786

    • PubMed
    • Google Scholar
70. 1. Shapley R

    (2009) Linear and nonlinear systems analysis of the visual system: why does it seem so linear?: A review dedicated to the memory of Henk Spekreijse

    Vision Research 49:907–921.

    https://doi.org/10.1016/j.visres.2008.09.026

    • PubMed
    • Google Scholar
71. 1. Shevell SK
    2. Monnier P

    (2005) Color shifts from S-cone patterned backgrounds: contrast sensitivity and spatial frequency selectivity

    Vision Research 45:1147–1154.

    https://doi.org/10.1016/j.visres.2004.11.013

    • PubMed
    • Google Scholar
72. 1. Simoncelli EP
    2. Heeger DJ

    (1998) A model of neuronal responses in visual area MT

    Vision Research 38:743–761.

    https://doi.org/10.1016/s0042-6989(97)00183-1

    • PubMed
    • Google Scholar
73. 1. Skottun BC
    2. De Valois RL
    3. Grosof DH
    4. Movshon JA
    5. Albrecht DG
    6. Bonds AB

    (1991) Classifying simple and complex cells on the basis of response modulation

    Vision Research 31:1079–1086.

    https://doi.org/10.1016/0042-6989(91)90033-2

    • PubMed
    • Google Scholar
74. 1. Spence I
    2. Wong P
    3. Rusan M
    4. Rastegar N

    (2006) How color enhances visual memory for natural scenes

    Psychological Science 17:1–6.

    https://doi.org/10.1111/j.1467-9280.2005.01656.x

    • PubMed
    • Google Scholar
75. 1. Stockman A
    2. MacLeod DI
    3. Johnson NE

    (1993) Spectral sensitivities of the human cones

    Journal of the Optical Society of America. A, Optics, Image Science, and Vision 10:2491–2521.

    https://doi.org/10.1364/josaa.10.002491

    • PubMed
    • Google Scholar
76. 1. Tailby C
    2. Solomon SG
    3. Lennie P

    (2008) Functional asymmetries in visual pathways carrying S-cone signals in macaque

    The Journal of Neuroscience 28:4078–4087.

    https://doi.org/10.1523/JNEUROSCI.5338-07.2008

    • PubMed
    • Google Scholar
77. 1. Tanabe S
    2. Cumming BG

    (2008) Mechanisms underlying the transformation of disparity signals from V1 to V2 in the macaque

    The Journal of Neuroscience 28:11304–11314.

    https://doi.org/10.1523/JNEUROSCI.3477-08.2008

    • PubMed
    • Google Scholar
78. 1. Tolhurst DJ
    2. Dean AF

    (1990) The effects of contrast on the linearity of spatial summation of simple cells in the cat’s striate cortex

    Experimental Brain Research 79:582–588.

    https://doi.org/10.1007/BF00229326

    • PubMed
    • Google Scholar
79. 1. Touryan J
    2. Lau B
    3. Dan Y

    (2002)

    Isolation of relevant visual features from random stimuli for cortical complex cells

    The Journal of Neuroscience 22:10811–10818.

    • PubMed
    • Google Scholar
80. 1. Vos JJ

    (1978) Colorimetric and photometric properties of a 2° fundamental observer

    Color Research & Application 3:125–128.

    https://doi.org/10.1002/col.5080030309

    • Google Scholar
81. 1. Weller JP
    2. Horwitz GD

    (2018) Measurements of neuronal color tuning: procedures, pitfalls, and alternatives

    Vision Research 151:53–60.

    https://doi.org/10.1016/j.visres.2017.08.005

    • PubMed
    • Google Scholar
82. 1. Willmore BDB
    2. Prenger RJ
    3. Gallant JL

    (2010) Neural representation of natural images in visual area V2

    The Journal of Neuroscience 30:2102–2114.

    https://doi.org/10.1523/JNEUROSCI.4099-09.2010

    • PubMed
    • Google Scholar
83. 1. Wilson HR
    2. Wilkinson F
    3. Asaad W

    (1997) Concentric orientation summation in human form vision

    Vision Research 37:2325–2330.

    https://doi.org/10.1016/s0042-6989(97)00104-1

    • PubMed
    • Google Scholar
84. 1. Wool LE
    2. Packer OS
    3. Zaidi Q
    4. Dacey DM

    (2019) Connectomic identification and three-dimensional color tuning of S-OFF midget ganglion cells in the primate retina

    The Journal of Neuroscience 39:7893–7909.

    https://doi.org/10.1523/JNEUROSCI.0778-19.2019

    • PubMed
    • Google Scholar
85. 1. Wurm LH
    2. Legge GE
    3. Isenberg LM
    4. Luebker A

    (1993) Color improves object recognition in normal and low vision

    Journal of Experimental Psychology: Human Perception and Performance 19:899–911.

    https://doi.org/10.1037//0096-1523.19.4.899

    • PubMed
    • Google Scholar
86. 1. Xian SX
    2. Shevell SK

    (2004) Changes in color appearance caused by perceptual grouping

    Visual Neuroscience 21:383–388.

    https://doi.org/10.1017/s0952523804213062

    • PubMed
    • Google Scholar
87. 1. Xiao Y

    (2014) Processing of the S-cone signals in the early visual cortex of primates

    Visual Neuroscience 31:189–195.

    https://doi.org/10.1017/S0952523813000278

    • PubMed
    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "Coding of chromatic spatial contrast by macaque V1 neurons" for consideration by eLife. Your article has been reviewed by 2 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Tirin Moore as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Bevil R Conway (Reviewer #1); David Brainard (Reviewer #2).

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Your manuscript was well received by the reviewers. They felt that your work fills a major gap in the understanding of cortical processing of color signals and is likely to be of broad interest. They also felt that the experiments are well executed and paper well-written. However, they raised a number of concerns which need to be addressed for the work to be publishable in eLife. Below are the topics that will require substantial revision.

1. The extent to which the results are affected by the cell classification criteria. The reviewers raise several related points and provide some possible ways of addressing the issue (which will potentially require further analysis).

2. The implications of the results for models, including the issue of how stimuli were defined and the implications of the stimulus space that was used. It will be essential that enough information is provided in the paper that others could convert the display RGB data into device-independent stimulus spaces such as cone-contrast space.

3. The color properties of the subfields.

4. The impact of eye movements on the results.

5. Potential implications of your choice of stimulus space

6. Provide information about your display's RGB data that would allow their conversion into a cone contrast space

In addition to the main concerns listed above, the two reviews contain detailed comments and suggestions which you are likely to find useful, as you revise the manuscript.

Reviewer #1 (Recommendations for the authors):

Many studies of the chromatic spatial response properties of V1 cells have used heterochromatic gratings where the colors of the grating are fixed by some assumed theory (typically cone opponency through, e.g., DKL color space). I am aware of only three studies that have addressed how V1 cells sum responses across receptive field subunits: one showed responses for only two cells, one clearly linear and one clearly non-linear (Conway, 2001); a second study focused on the integration of color signals in time, rather than in space (Conway et al., 2002); and a third measured responses to pairs of cone-isolating spots and found that L-M cells were linear while S-dominated cells were often non-linear (Conway and Livingstone, 2006). The approach in the present report, combining the white-noise analysis and the "hyperpixel" contrast approach are very powerful (and clever), overcoming the limited set of colors used in the previous reports and adding substantially more quantitative resolution. But the analysis of the results do not seem to fully explore the richness of the data, and leave us with an unsurprising conclusion that confirms what we already know-double-opponent cells include a population of linear neurons. As the authors point out, previously published spatial mapping studies of cone-opponent neurons have recovered spatial structure, and the technique used in those studies only reveal spatial structure if the neurons show some degree of linearity. It is well established that the majority of V1 cells in primates are non-linear (complex) and the minority of simple cells in primate are linear (by definition). So it is not surprising that double-opponent cells in the present work are more likely to be linear compared to the somewhat random sample of non-DO/non-simple cells, which were, by definition, classified as nonlinear (discussion, last paragraph, pg. 14).

The value of the present study would seem to rest on the distinction between simple and DO cells and the selection criteria for the NSNDO population. I appreciate that there is some arbitrariness in how cells are categorized, and the authors have attempted to show that the main conclusion is maintained if category assignments are made using two different criteria. But I wonder if a simpler approach, which I think cleaves nature more naturally along the joints, would be to define cells based on significance of S cone input, and then on whether the L and M inputs have the same sign or different sign. This approach would leave the categorization of "simple" and "double opponent" to the discussion, and it would preclude an objection of the results/conclusions on the grounds of miss-categorization. The anatomy suggests that neurons carrying S cone signals are distinct, and probably have different patterns of wiring; and cells that respond with the same sign to L increments and M increments (without responding to S cone stimuli) would seem to be functionally different from neurons that respond in an L-vs-M opponent fashion. Moreover, there is considerable data (from at least two groups, Horwitz and Conway), that V1 cells with strong S-cone input show striking nonlinearities. (Are the greater number of NLI cells in the DO population and the simple cell population those with stronger S cone responses?) The physiological responses documented in the present report would be more valuable, in my opinion, if they provided clues that link with the anatomy, rather than being constrained by categorization based on potentially arbitrary physiological criteria (such as "simple" vs. "double opponent"). Of course, this is only a suggestion, and the authors should be free to analyze and present their data as they see best.

The analysis side-steps the important temporal dynamics of the cell responses. As with simple cells and double-opponent cells, responses are often push-pull in space and in time. It would seem important to test the extent to which the assigned linearity score of a neuron in space is predictive of its linearity score in time.

The hyperpixel paradigm involves presenting the stimuli at a fixed spatial location while the monkeys maintain fixation. The validity of the approach depends on the extent to which a given receptive field subregion is stimulated reliability and exclusively despite small eye movements. Some assurance (ideally with data) that this is the case would be helpful. My worry is that the interpretability of the results (given the eye movements) depends on the neurons being linear.

Finally, the paper does not explore the color tuning of the subunits, except to state that "the colors on the two sides of the hyperpixel STA were complementary or nearly so". The Pearson's r analysis is insufficient: it tells us nothing about the color tuning, and high correlation values can be obtained for trivial reasons. The paradigm illustrated in Figure 2B provides a terrific chance to determine, directly, what the color tuning is of the separate subunits, and the extent to which the color tuning of one subunit of a given cell can predict the color tuning of the other subunits. This is important because color tuning inferred from cone weights suggests a bias (M+S vs L) that the paper is poised to directly test. Such an analysis could also help tease apart the distinction between simple and DO cells.

Other comments

Methods. The procedure by which different proportions of the two images were combined to create the family of stimuli is not entirely clear to me. The proportionality needs to be done by quantifying the stimuli in some way. The authors state that this was done by defining the 45 degree direction as the projection of the RGB values onto one half of the STA. But here's where I get tripped up: the goal of defining the family of stimuli is to then assess whether the neuron responses map on a line in the space or on a curve. If the space is arbitrarily defined, then a given neural response can be captured either by a line or a space depending on how the space is defined. I need some way out of this (apparent) circularity.

Intro. Given the relevance to simple cells, it might be useful to have a reference to the experimental evidence of simple cells in NHPs (there was a time, not too long ago, when dogma was that NHPs did not have simple cells).

Results. Given the validity of the results hinges on the distinction between simple, double-opponent, and NSNDO cells, I think it would be helpful to present the categorization criteria in the Results section, rather than relegating this information to the methods.

Results, pg. 6. The "hyperpixel" approach is analogous to the two-bar approach used in earlier studies. Except that in the earlier study the pair of bars were presented across the RF, which allowed quantification of more than two subunits. In the present work, did the stimulus always comprise just two halves? What if the cell had more than two subunits, as is not uncommon for simple and DO cells (and appears to be true for the example cell in Figure 2A)?

Results, pg. 6. I was a bit tripped up by the flow of the results: the STA analysis (2A) is what is used to recover the NLI histograms (2D-2F); the panels in 2B and 2C show a separate experiment. Perhaps regrouping the figures accordingly would help avoid the confusion that Figure 2D-2F quantify what is shown in Figure 2C (which is what would be logically assumed given the current sequence of panels and how they are described in the text).

Results, pg. 7. Is it not true that the extra parameters of the nonlinear model will also make it more likely that cross-validated data will be better fit by it? In other words, using cross validation doesn't get over the unfairness of comparing the GQM and GLM models, does it?

Results, pg. 8. The Kruskal-Wallis test outcome is driven entirely by the NSNDO population, which likely includes complex cells. It is well established that complex cells and simple cells differ in linearity, so it's not clear to me what this NLI analysis recovers. As the authors point out, the reverse-correlation mapping studies published previously only recover spatial structure if the cells have a linear component. So we already know that DO cells are more linear than the average V1 cell.

Results, pg. 10. The discussion of the necessity of the technique is helpful. Is this is the same logic underlying the use of responses to L-decrements to measure suppression by L (for green-ON cells) and M-decrements to measure suppression by M (for red-ON cells) by others?

Results, pg. 10. It would be helpful to state what conclusion (curved or linear) the authors interpret Figure 2C as showing. I'm assuming curved because the pattern of gray shading forms an arc around the 0,0 point. But then what are the dashed white lines? If they are intended to show model fits of stimuli that elicit the same firing rate, then I think something is wrong with that algorithm because the data do not, to my eye, align with the dashed lines. If they are intended to show theoretically which stimuli would have the same firing rate if the neuron were linear, then I'm confused why the lines are tilted.

Results, pg. 10. It seems odd to set the target firing rate to be lower than significantly (substantially) above baseline firing rate for any cells (even three). It would be useful to know the range, mean, and confidence limits around the mean, for the target firing rates (in terms of % above background firing rate).

Discussion, pg. 12. I appreciate the point the authors are trying to make, but as framed here, it seems weak ("we used a visual stimulation paradigm that only works if the cells are linear to test if the cells are linear").

Discussion, pg. 13. The prior study discussed by the authors concluded that the responses to pairs of bars was linear. Another earlier study (Conway, 2001) showed two-bar interaction maps for two cells, one of which was clearly non-linear (and the non-linear cell was the one with strong S-cone input).

Discussion, pg. 13. The authors state that the prior work made an assumption that excitatory responses to an increment were the same size but opposite sign to suppression of a contrast decrement. It's not clear to me that this was an assumption rather than a test. The reverse-correlation procedure used in the earlier work used as a baseline the responses to stimuli presented well outside the receptive field, which allows the assumption to be directly tested.

Discussion, pg. 16 "In either case, DO cells are a likely basis for the chromatic sensitivity of color-sensitive complex cells (Michael, 1978)." This isn't clear to me. It seems equally plausible (arguably more plausible) that color-sensitive complex cells could derive color tuning by direct sampling of geniculate inputs.

Figure 3. It would be helpful to show the STA for each of the example cells. The distribution of data points in Figure 3A is stated in the results as being linear, but it looks curved to me (the two outliers at the far left notwithstanding). None of the example cells in Figure 3 are L-vs.-M cells, which is odd since the majority of "classical" double-opponent cells are defined by L-vs.-M opponency. By contrast, all the cells in Figure 3A could plausibly be simple cells. I'm a bit confused how meaningful the responses of a cell are if they fall on a line that goes through the origin. Responses at the origin are to a null stimulus (blank gray), which indicates that the target firing rate is nota much different from baseline.

Reviewer #2 (Recommendations for the authors):

General. I think this paper would benefit from a sharpening of terminology and usage throughout. For example, always being explicit about what sort of non-linearity is being considered in any given sentence, and about which of the two types of white noise experiments is being referred to in each place where a reference is made. I could almost always figure out it what was intended from context, but that was extra work and left some uncertainty, as well distracted from following the flow of ideas. Specific examples given in some of the comments below.

Starting on page 3 and throughout. A key to understanding this paper is to understand the taxonomy the authors have applied to the neurons they recorded from. I struggled with the simple cell terminology, though. The criteria the authors use are different from more conventional ways used distinguish simple and complex cells. I think that if one applied F1/F0 criteria using luminance grating stimulation, one might well call at least some of the DO cells simple, as well as possibly some of the cells not studied. The criteria here are that the hyperpixel white noise analysis shows that cone contrast inputs are combined linearly and that the RFs are not cone opponent. I would have an easier time if the term simple cell were not used, to avoid confusion with the conventional taxonomy. Perhaps single opponent cells, or spatially-opponent non-cone-opponent cells. Thesse suggestion don't capture the linearity aspect of the criteria, but nor does the term double opponent and the authors are content with that. Maybe both terms should more clearly denote the linearity criteria. With a different choice of terminology, the relation to the conventional simple/complex taxonomy could be handled entirely in the discussion, with some care devoted to treating possible differences in classification. Avoiding the simple cell terminology might also allow ease exposition on the possible relation between NSNDO cells in the current study and chromatically sensitive complex cells, where I think more elaboration would be helpful.

Page 3. Introduction: "The first stage of the primate visual system at which spatial comparison of spectral signals is implemented by individual neurons is area V1." – This statement is mysterious to me. Spatial comparisons are made at the first synapse, as signals from cones interact with signals from horizontal cells, and these interactions can involve spectral differences as signals from different classes of cones participate. Perhaps the authors mean something very specific when the qualify with "by individual neurons" but it is not clear what this is.

Page 4. I think more should be said about the 134 neurons that were recorded from and not studied further. These don't have a name in the current study and it is not entirely clear to me what caused them to be excluded. Perhaps only that they did not show any spatial opponency in the initial pixel white noise analysis (page 6), but I'm not sure that's the only reason so more detail on the exclusion criteria would be helpful.

Page 5. I agree with the authors that equating stimulus contrast across chromatic directions is a difficult and subtle matter, and that whenever possible it is useful to look for ways to frame questions that are robust to such equation. I diverge in that I don't see that their method avoids this difficulty. Just as one example, the frame selection criteria (pp. 21-22) are based on deciding which STA frame differs most from the background. That calculation is using some metric across color directions. Similarly with the criterion used to decide that the chromatic preferences of the two subfields of a DO neuron differ from each other.

Page 12. While we're on this point, the authors have used device primary RGB as their metric. The justify the reasonableness of this choice by writing "The fact that DO cells and simple cells combined light intensities across space approximately linearly in this representation shows that it is a reasonable one for analyzing V1 neurons." This seems odd, first because it assumes the answer to the question being studied in this paper. Second, because if the combination is revealed to be linear in one choice of color representation, it will also be linear in another choice that is a linear transformation of the first. (Indeed, the authors themselves say this in the sentence after the one quoted.) So I don't see how the linearity of the result tells us much about whether the representational choice was good. If nothing else, I would ask that the authors provide enough information about their device so that others can transform their data into a device independent representation such as LMS contrast with respect to a well-defined choice of cone fundamentals.

Page 6. Figure 2B. I assumed the noise is independent across color channels in RGB, but would be good to say so explicitly.

Page 8. Figure 3. What should we make of the high indices for some DO cells, and the low indices for many NSNDO cells? Both distributions suggest at least that possibility that neither group is a homogeneous population with respect to non-linear spatial summation. More discussion of this seems worthwhile.

Page 10. How were the target firing rates chosen?

Page 13. "Nearly all quantitative studies of V1 RFs have used achromatic stimuli". Perhaps worth listing the studies of this sort that have used chromatic stimuli. Also, I think that by "V1 RFs" the authors mean "V1 spatial RF properties" and this might be worth spelling out.

Page 13. "This maximal response exceeded the response to either bar in isolation, consistent with linearity as well as with other models." Might make explicit that the linearity in this sentence is the linearity of spatial summation, as well as that the reason the result could be consistent with summation linearity is that there could be a response non-linearity after spatial summation.

Page 14. "Reclassifying cells …. " There's a lot of cell classification in this paper; perhaps sharpen here to emphasize that it is the simple/DO dichotomy classification being referred to here.

Page 14. "Most other recent studies of DO cells used cone-isolating stimuli, which cannot reveal interactions among cone types (Conway, 2001; Johnson et al., 2001, 2004; Conway and Livingstone, 2006; Johnson et al., 2008)." Johnson et al., used mixtures in some of their reported measurements.

Page 14. "We classified neurons with nonlinear responses to white noise as NSNDO. This criterion was necessary to satisfy the assumptions underlying the conversion of the STA to cone weights." (A) In the first sentence, "nonlinear" seems too general since the paper believes it worthwhile to check for a specific type of nonlinearity in the simple and DO cells. (B) I don't understand this because the authors go on to apply the conversion of the STA to cone weights with the NSNDO cells, in that such conversion is used for the test in Phase 2, as shown in Figure 3F. (C) Here is an other places, it would be helpful to say "hyperpixel white noise" rather than just "white noise"

Page 15. "Pooling LGN afferents with the same sign (ON or OFF) creates non-opponent spectral sensitivity." I'm not following what is meant here, need more detail on what is being assumed about chromatic properties of the afferents as well as exactly what is being pooled.

Page 17 and Figure 4. If the authors are going to show this data can they say anything about how the results for the experiments that are the focus of the current paper would come out for these two cells? That would connect these preliminary data better to the rest of the paper. And, did the authors find any cells like the one shown in 4C, but with the responses to luminance and chromatic gratings flipped?

Page 22. Please provide criteria used to decide whether the two subfields had distinct chromatic preferences, and how this related to cell classification. On page 7, correlations (computed how?) between the two halves of the STA are given, and it sounds like this is result obtained before selecting or classifying cells. One page 24 a percent variance explained number is given, again independent of cell classification. But I'm not entirely sure that this property wasn't required for cells to be included for further study.

Page 22. The authors note that using small pixels and a fast frame rate in the white noise experiments leads to the neurons being weakly driven. They might comment/think about whether the choices they made for these parameters are optimal – if they were to do the work again, could they consider using larger white noise pixels and a slower refresh rate to obtain better responses? That is, small/fast is at one end of a continuum, and maybe some other choice would be better for future work of this sort. Similarly, any requirement for use of the Gaussian distribution on the noise? Maybe a uniform or some other distribution would produce larger drive.

Page 22, last paragraph and into p. 23. The paragraph starts off talking about frames as six-dimensional vectors, but by the end it sounds like two separate 3-dimensional projections are done, yielding the two scalars one for each subfield. Would be clearer, I think, if only one of the two ways of talking about it is used here. Also might remind reader that this is the hyperpixel white noise STA, not the pixel white noise STA.

Page 23. We visualized a firing rate map from these projections by computing the ratio of spike-triggered stimuli to the total stimuli." I don't understand what metric of stimuli is being used in the ratio, which visualization in the paper this refers to, nor how the ratio of stimuli as described provides a map of firing rate.

Page 23. "Direction represents the overall contrast between …" Worth noting that what these axes represent in stimulus space is cell-specific, in that the axes for each cell are derived from that cell's hyperpixel STA. Also, although this is a matter of convention and up to the authors, I would find it more intuitive if the STA axes were the x and y axes of the plot, not the 45 and 135 directions.

Page 24. 232 neurons were recorded from. 98 are reported on here. What can we say about the rest, and why they were not included here. The one clear exclusion criterion is that some neurons had only a single spatial region revealed by the pixel white noise STA. Was that the only reason? On this point, I don't think NSNDO is the best term for the set of neurons out of the 98 that were not classified as simple or DO, because of all the other neurons that were not included and that also weren't classified as simple or DO.

Page 24. Is it a theorem that if you take the first row singular vector and the first column singular vector, that this is the optimal color-space separable accounting of the STA? Or could a joint analysis do better?

Page 24-25. On page 25, the classification of cells into simple, DO, and NSNDO is described. Because the term "white noise" is used to describe the protocol used for the classification and because this applies to both Phase 1 (pixel white noise) and Phase 2 (hyperpixel white noise) of the experiment, it would be good to remind the reader that it is (I think) the hyperpixel white noise that's relevant here. Indeed, I would try to be explicit each time in the paper the term white noise or STA is used, which of the two it refers to. Also, why is a substantial S cone input a requirement to be a DO cell? This seems like it would exclude cells with perfectly good spatially opponent L-M inputs. And, I'm not clear what happens to such cells, if there were any. I think from the next sentence that like neurons with a significant PC1, they get called NSNDO. But because the flow of this section had me thinking at the start that it was just those neurons with signifiant PC1 that were NSNDO, I'm also not clear on this. If they are NSNDO, then it is worth being explicit elsewhere that heterogeneous criteria were used to place neurons in this category, and perhaps looking at NLI separately for cells that arrived as NSNDO viat different paths.

Page 26. I'm surprised that Equation 1 doesn't include an amplitude parameter that captures the overall responsivity of the cell.

Figure 2 —figure supplement 1. I am not sure what is in panel D. Are these from Phase 2? And how does the 5th and 95th percentile information mentioned in the caption show up in the figure? More generally, I think the point of this figure is to provide a sense of how much harder the Phase 3 stimuli drive the cells than the Phase 1 and 2 stimuli, but I am not seeing this in the figure itself. Where are the Phase 3 data?

Figure 2 —figure supplement 2. Can you clarify the temporal parameters that go into this analysis?

Figure 3 —figure supplement 3. The staircase shown suggests that using the last stimulus presented does not give as good an estimate of stimulus that produces the criterion firing rate as would an analysis of the whole staircase. Do I understand that the last stimulus magnitude is what was used? If so, please elaborate why. If not, please clarify.

Reviewer #1 (Recommendations for the authors):

Many studies of the chromatic spatial response properties of V1 cells have used heterochromatic gratings where the colors of the grating are fixed by some assumed theory (typically cone opponency through, e.g., DKL color space). I am aware of only three studies that have addressed how V1 cells sum responses across receptive field subunits: one showed responses for only two cells, one clearly linear and one clearly non-linear (Conway, 2001); a second study focused on the integration of color signals in time, rather than in space (Conway et al., 2002); and a third measured responses to pairs of cone-isolating spots and found that L-M cells were linear while S-dominated cells were often non-linear (Conway and Livingstone, 2006). The approach in the present report, combining the white-noise analysis and the "hyperpixel" contrast approach are very powerful (and clever), overcoming the limited set of colors used in the previous reports and adding substantially more quantitative resolution. But the analysis of the results do not seem to fully explore the richness of the data, and leave us with an unsurprising conclusion that confirms what we already know-double-opponent cells include a population of linear neurons. As the authors point out, previously published spatial mapping studies of cone-opponent neurons have recovered spatial structure, and the technique used in those studies only reveal spatial structure if the neurons show some degree of linearity. It is well established that the majority of V1 cells in primates are non-linear (complex) and the minority of simple cells in primate are linear (by definition). So it is not surprising that double-opponent cells in the present work are more likely to be linear compared to the somewhat random sample of non-DO/non-simple cells, which were, by definition, classified as nonlinear (discussion, last paragraph, pg. 14).

The value of the present study would seem to rest on the distinction between simple and DO cells and the selection criteria for the NSNDO population. I appreciate that there is some arbitrariness in how cells are categorized, and the authors have attempted to show that the main conclusion is maintained if category assignments are made using two different criteria. But I wonder if a simpler approach, which I think cleaves nature more naturally along the joints, would be to define cells based on significance of S cone input, and then on whether the L and M inputs have the same sign or different sign. This approach would leave the categorization of "simple" and "double opponent" to the discussion, and it would preclude an objection of the results/conclusions on the grounds of miss-categorization. The anatomy suggests that neurons carrying S cone signals are distinct, and probably have different patterns of wiring; and cells that respond with the same sign to L increments and M increments (without responding to S cone stimuli) would seem to be functionally different from neurons that respond in an L-vs-M opponent fashion. Moreover, there is considerable data (from at least two groups, Horwitz and Conway), that V1 cells with strong S-cone input show striking nonlinearities. (Are the greater number of NLI cells in the DO population and the simple cell population those with stronger S cone responses?) The physiological responses documented in the present report would be more valuable, in my opinion, if they provided clues that link with the anatomy, rather than being constrained by categorization based on potentially arbitrary physiological criteria (such as "simple" vs. "double opponent"). Of course, this is only a suggestion, and the authors should be free to analyze and present their data as they see best.

Thank you for these suggestions. We have added a new analysis to the manuscript showing the relationship between the magnitude of S-cone weight and NLI. As the Reviewer predicted, neurons with large S-cone weights tend to be nonlinear. However, some neurons in our data set appeared to receive little S-cone input but nonetheless were highly non-linear. Defining cone weights for these neurons is difficult. Our estimates of how much input a neuron receives from the three cone types depends critically on how linearly these inputs are combined whereas our estimates of linearity are not trivially influenced by the strength of input from the three cone types. For these reasons, we have maintained the original criteria (screening first on the basis of linearity and second on the basis of cone weights) for defining the three groups of cells. However, we have changed the Introduction to more clearly explain these criteria.

The analysis side-steps the important temporal dynamics of the cell responses. As with simple cells and double-opponent cells, responses are often push-pull in space and in time. It would seem important to test the extent to which the assigned linearity score of a neuron in space is predictive of its linearity score in time.

We agree that the integration of visual information over time by V1 neurons is an important issue, but the data we collected are ill-suited for investigating this issue with the care it deserves. Ideally, to measure temporal linearity, we would have probed neurons with pairs of stimuli that are identical in space but separated in time (as in Conway and Livingstone, 2006). Instead, we probed neurons with pairs of stimuli that were separated in space but coincident in time, which is more appropriate for measuring spatial than temporal linearity.

Nevertheless, inspired by the Reviewer's suggestion, we analyzed our data for signs of push-pull excitation-inhibition, which manifests in OFF responses following the removal of a suppressive stimulus. For each neuron, we examined responses following the disappearance of a stimulus and measured the correlation of these responses with the isoresponse NLI. We would like to have have restricted this analysis to the removal of suppressive stimuli, but few of the stimuli presented were suppressive. As a proxy, we considered only trials in which no spikes were generated during the stimulus presentation period (consistent with the stimulus having been suppressive). We counted spikes from the disappearance of the stimulus (plus an additional latency derived from the hyperpixel STA) until 150 ms after stimulus disappearance. Fixation breaks during this epoch aborted trials. We found no significant relationship between the average OFF response and the isoresponse NLI (r=-0.01, p=0.89, Spearman’s rank correlation).

We also considered a stricter criterion, considering only neurons for which the average firing rate before the stimulus presentation (the baseline firing rate) was > 0 spikes per second (71/96 neurons satisfied the criteria). This allowed us to more easily identify suppressive stimuli. Even within this subset of neurons, we still did not find a significant correlation (r=0.01, p=0.84, Spearman’s rank correlation). The fact that we did not present many suppressive stimuli may have prevented us from observing strong OFF responses.

In interpreting these data, we note that linearity can be dissociated from OFFresponses; a linear neuron with a biphasic temporal impulse response produces an OFF response, but a linear neuron with a monophasic temporal impulse response does not (Author response image 1) .

Author response image 1

Download asset Open asset

The hyperpixel paradigm involves presenting the stimuli at a fixed spatial location while the monkeys maintain fixation. The validity of the approach depends on the extent to which a given receptive field subregion is stimulated reliability and exclusively despite small eye movements. Some assurance (ideally with data) that this is the case would be helpful. My worry is that the interpretability of the results (given the eye movements) depends on the neurons being linear.

We have included new analyses of fixational eye movements to the revised manuscript. Differences in fixation eye movements across neurons of different type were subtle, if present. This is expected: fixational eye movements shift the stimulus on the RF, but these shifts tend to be small with respect to the hyperpixels. Eye movements can make a two-subfield RF appear to have a single subfield (if the two hyperpixels spend an equal amount of time stimulating each subfield), but they cannot make a one-subfield RF present as an STA with multiple spectrally distinct subregions. All of the neurons described in this report had STAs with multiple spectrally distinct subregions. Moreover, we never observed a neuron that had two RF subregions in response to the pixel white noise but only one in response to hyperpixel white noise.

Importantly, in Phase 3 of the experiment, images were flashed briefly, spikes were counted, and stimuli were not averaged together. The fact that we were able to cancel an increase in preferred color on one side of the RF with a similar change in light on the other side attests to the stability of the image on the retina (opponent regions of the RF were successfully targeted). We are unsure how eye movements might affect the interpretation of the data differentially for linear and nonlinear cells. Perhaps the reviewer is alluding to the fact that eye movements could cause a linear neuron to have a significant PC1 if these movements bring individual hyperpixels onto multiple subfields of different spectral sensitivity. Through this mechanism, eye movements would be expected to increase the significance of the PC1 of linear neurons. In this sense, our PC1 criterion is conservative.

Finally, the paper does not explore the color tuning of the subunits, except to state that "the colors on the two sides of the hyperpixel STA were complementary or nearly so". The Pearson's r analysis is insufficient: it tells us nothing about the color tuning, and high correlation values can be obtained for trivial reasons. The paradigm illustrated in Figure 2B provides a terrific chance to determine, directly, what the color tuning is of the separate subunits, and the extent to which the color tuning of one subunit of a given cell can predict the color tuning of the other subunits. This is important because color tuning inferred from cone weights suggests a bias (M+S vs L) that the paper is poised to directly test. Such an analysis could also help tease apart the distinction between simple and DO cells.

Thank you for this suggestion. We have added a new analysis of cone weights from the two hyperpixels separately. This analysis more clearly shows the distinction between simple cells and DO cells, documents the fact that the RF subfields of neurons in both groups have approximately opposite spectral tuning, and confirms the M+S vs L bias that has been documented before.

Other comments

Methods. The procedure by which different proportions of the two images were combined to create the family of stimuli is not entirely clear to me. The proportionality needs to be done by quantifying the stimuli in some way. The authors state that this was done by defining the 45 degree direction as the projection of the RGB values onto one half of the STA. But here's where I get tripped up: the goal of defining the family of stimuli is to then assess whether the neuron responses map on a line in the space or on a curve. If the space is arbitrarily defined, then a given neural response can be captured either by a line or a space depending on how the space is defined. I need some way out of this (apparent) circularity.

Thank you for pointing out this confusing part of the manuscript. The reviewer is correct that isoresponse contours have different shapes in different color spaces. However, lines in one color space remain lines in all color spaces that are related by a linear transformation. The color space we used is related to many conventional color spaces by a linear transformation (e.g. cone-contrast and DKL). This is clarified in the revised manuscript.

At each of the two hyperpixels, the STA reveals the neuron's relative sensitivity to the red, green, and blue primaries (and, by extension, the relative sensitivity to modulations of the L-, M-, and S-cone photoreceptors). We describe this sensitivity with a threeelement, unit vector. For example, if the STA indicated maximal sensitivity to increments of the red primary, half as much sensitivity to decrements of the green primary, and zero sensitivity to the blue primary, this vector would be [0.89 -0.46 0]. If the background was [0.5 0.5 0.5] (where "0" means completely off and "1" means maximum intensity), then possible stimuli at this location included [0.589 0.454 0.5] (reddish), [0.411 0.546 0.5] (greenish), and [0.5 0.5 0.5] (invisible). To express the contrast of each stimulus with a single number, we subtracted the background and then projected onto the unit vector: ([0.589 0.454 0.5]-[0.5 0.5 0.5])*[0.89 -0.46 0]' = 0.1. This calculation provides one of the two coordinates needed to represent the stimulus as a point on a plane. The other coordinate was calculated by performing the identical procedure on the other half of the STA. We have added the above explanation to the revised text.

Intro. Given the relevance to simple cells, it might be useful to have a reference to the experimental evidence of simple cells in NHPs (there was a time, not too long ago, when dogma was that NHPs did not have simple cells).

References to the existence of simple cells in NHPs have been added to the Introduction.

Results. Given the validity of the results hinges on the distinction between simple, double-opponent, and NSNDO cells, I think it would be helpful to present the categorization criteria in the Results section, rather than relegating this information to the methods.

We have expanded the explanation of the cell classification criteria in the Results section.

Results, pg. 6. The "hyperpixel" approach is analogous to the two-bar approach used in earlier studies. Except that in the earlier study the pair of bars were presented across the RF, which allowed quantification of more than two subunits. In the present work, did the stimulus always comprise just two halves? What if the cell had more than two subunits, as is not uncommon for simple and DO cells (and appears to be true for the example cell in Figure 2A)?

We did occasionally observe simple cells (and less frequently, DO cells) with > 2 subfields. For cells with more than 2 subfields, we selected for visual stimulation the most prominent pair of subfields (the three-subfield neuron in Figure 2A is a good example). We wholeheartedly agree that measuring spatial integration across multiple (> 2) subfields is important, but the three-phase, closed-loop technique implemented in the current study does not scale well beyond 2 subfields.

Results, pg. 6. I was a bit tripped up by the flow of the results: the STA analysis (2A) is what is used to recover the NLI histograms (2D-2F); the panels in 2B and 2C show a separate experiment. Perhaps regrouping the figures accordingly would help avoid the confusion that Figure 2D-2F quantify what is shown in Figure 2C (which is what would be logically assumed given the current sequence of panels and how they are described in the text).

Thank you for alerting us to this confusing issue. The NLI histograms (Figures 2D-2F) are computed from responses to the hyperpixel white noise (Figure 2B), and not the pixel white noise (Figure 2A). We have modified each mention of white noise or STA in the manuscript to specify whether we are referring to the pixel or the hyperpixel stimulus.

Results, pg. 7. Is it not true that the extra parameters of the nonlinear model will also make it more likely that cross-validated data will be better fit by it? In other words, using cross validation doesn't get over the unfairness of comparing the GQM and GLM models, does it?

Yes, we believe that cross-validation gets over the unfairness of comparing the GQM and GLM models. A non-cross-validated comparison would be unfair because the GQM makes the identical predictions to the GLM if some of the parameters of the GQM are set to zero. If these parameters are allowed to be something other than zero, the GQM will fit the data better unless the best-fitting values of these parameters happen to be zero, in which case there will be a tie.

When the data used to fit the model differ from the data used to test the model, as is the case in cross-validation, the GQM no longer enjoys this advantage over the GLM. The GQM will continue to fit the training data better than the GLM does, and this is for one of two reasons: The GQM may fit genuine structure in the data (patterns that are consistent across datasets produced by the same data generating mechanism) that the GLM does not fit. In this case, we expect the GQM to fit a new data set (or, equivalently, a held-out data set) better than the GLM does. Alternatively, the GQM may fit noise (patterns that are specific to the particular dataset being fit) that the GLM does not fit. In this case, we expect the GQM to fit a new data set worse than the GLM does.

Results, pg. 8. The Kruskal-Wallis test outcome is driven entirely by the NSNDO population, which likely includes complex cells. It is well established that complex cells and simple cells differ in linearity, so it's not clear to me what this NLI analysis recovers. As the authors point out, the reverse-correlation mapping studies published previously only recover spatial structure if the cells have a linear component. So we already know that DO cells are more linear than the average V1 cell.

Yes, the Kruskal-Wallis test outcome is driven by the OSO (previously called “NSNDO”) cells. However, a spatially structured STA was a requirement for inclusion in this study. Pure complex cells, which respond identically to contrast increments and decrements, were not studied. It seems likely that some neurons in the OSO category responded to contrast increments and decrements in the same RF locations, but with different sensitivity to each, and therefore had STAs with some spatial structure. We suspect that many of these cells would have been classified as complex cells by other procedures, but we cannot know for sure.

The white noise NLI is related to the spike-triggered covariance criterion that was used to exclude complex cells from the simple and DO cell categories but is not redundant with it. Please see our response to Reviewer 2, comment 20 for a more complete explanation of how these criteria differ.

Results, pg. 10. The discussion of the necessity of the technique is helpful. Is this is the same logic underlying the use of responses to L-decrements to measure suppression by L (for green-ON cells) and M-decrements to measure suppression by M (for red-ON cells) by others?

There may be a connection between the advantages we lay out for the isoresponse method and the use of responses to L-decrements to measure suppression by L and the use of M-decrements to measure suppression by M, but this connection is not obvious to us.

In this section, we focus on responses to pairs of stimuli and to each member of a pair separately. The key piece of logic is that two stimuli, when presented together, may drive a response that is not equal to the sum of the responses to the individual members of the pair but may still be predictable from these individual responses. The use of L-decrements to measure suppression by L and the use of M-decrements to measure suppression by M requires an assumption that two types of stimuli exert equal and opposite effects on the firing rate, which is not required in the above explanation.

Results, pg. 10. It would be helpful to state what conclusion (curved or linear) the authors interpret Figure 2C as showing. I'm assuming curved because the pattern of gray shading forms an arc around the 0,0 point. But then what are the dashed white lines? If they are intended to show model fits of stimuli that elicit the same firing rate, then I think something is wrong with that algorithm because the data do not, to my eye, align with the dashed lines. If they are intended to show theoretically which stimuli would have the same firing rate if the neuron were linear, then I'm confused why the lines are tilted.

The reviewer is correct that the neuron in Figure 2C was fit slightly better by the GQM than the GLM. The GQM vs. GLM comparison for this example cell (quantified by the hyperpixel white noise NLI) is shown in Figure 2D as a red vertical tick mark. This neuron was selected as an example because its NLI lies near the mode of the distribution.

The data shown in Figure 2C are binned for the purposes of visualization. This binning was necessary because stimulus projections onto the two halves of the hyperpixel STA varied continuously. Please see Figure 2 —figure supplement 1 for a scatterplot of the stimulus projections (panel A, red points) and the corresponding binned histogram (panel C). Bins that are far from the center of the box (0,0) contain few stimuli and thus estimates of spiking probability for these bins are noisy. Also, estimates of the probability of spiking are rendered in uncalibrated grayscale, making it difficult to compare differences in gray level by eye. The GLM and GQM were fit to the unbinned data.

We have revised the text to clarify these points and have updated Figure 2C to show firing rate predictions from the GQM fit, which are similar to those from the GLM fit. We note that tilted isoresponse lines are consistent with linearity and indicate that one subfield was dominant over the other.

Results, pg. 10. It seems odd to set the target firing rate to be lower than significantly (substantially) above baseline firing rate for any cells (even three). It would be useful to know the range, mean, and confidence limits around the mean, for the target firing rates (in terms of % above background firing rate).

We have summarized the information about baseline firing rates and target firing rates in Figure 3—figure supplement 2. On average, the target firing rate was 15.65 standard deviations above the mean baseline firing rate. The target firing rate of three neurons fell between the 90^th and 95^th percentiles (none were lower). Removing these neurons from the data set did not change the main conclusions of this study.

Discussion, pg. 12. I appreciate the point the authors are trying to make, but as framed here, it seems weak ("we used a visual stimulation paradigm that only works if the cells are linear to test if the cells are linear").

We believe that the reviewer is referring to our statement that we "characterized the spatial integration by individual V1 neurons using… white noise RF mapping". White noise stimulation can be used to characterize both linear and nonlinear neurons, but we agree that the concept of an RF map assumes linearity. We have changed the text from "white noise RF mapping" to "white noise visual stimulation".

Discussion, pg. 13. The prior study discussed by the authors concluded that the responses to pairs of bars was linear. Another earlier study (Conway, 2001) showed two-bar interaction maps for two cells, one of which was clearly non-linear (and the non-linear cell was the one with strong S-cone input).

These studies are cited in the revised Discussion.

Discussion, pg. 13. The authors state that the prior work made an assumption that excitatory responses to an increment were the same size but opposite sign to suppression of a contrast decrement. It's not clear to me that this was an assumption rather than a test. The reverse-correlation procedure used in the earlier work used as a baseline the responses to stimuli presented well outside the receptive field, which allows the assumption to be directly tested.

The idea that the excitatory response to a stimulus is the same magnitude, but opposite sign, as the suppression to a contrast-inverted stimulus is described by Conway and Livingstone (2006) as an assumption:

“The suppression shown by many [V1] neurons to some stimuli resulted in a near complete reduction in background firing rate, leading one to suspect that the stimulus was capable of more suppression than could by measured extracellularly. To overcome this rectification nonlinearity in simple cells, the full extent of the response is often inferred by subtracting the responses to opposite-contrast stimuli (Ferster, 1994). The assumption that the excitatory response to a stimulus is the same magnitude, but opposite sign, as the suppression to a stimulus has been shown to be valid for LGN cells in the cat (Martinez et al., 2005). We made this assumption, too, to represent the responses succinctly by difference maps, which allow a direct quantitative calculation of the cone weights.” (emphasis added)

Our intention is simply to paraphrase this description.

Discussion, pg. 16 "In either case, DO cells are a likely basis for the chromatic sensitivity of color-sensitive complex cells (Michael, 1978)." This isn't clear to me. It seems equally plausible (arguably more plausible) that color-sensitive complex cells could derive color tuning by direct sampling of geniculate inputs.

We consider this possibility in the revised Discussion.

Figure 3. It would be helpful to show the STA for each of the example cells. The distribution of data points in Figure 3A is stated in the results as being linear, but it looks curved to me (the two outliers at the far left notwithstanding). None of the example cells in Figure 3 are L-vs.-M cells, which is odd since the majority of "classical" double-opponent cells are defined by L-vs.-M opponency. By contrast, all the cells in Figure 3A could plausibly be simple cells. I'm a bit confused how meaningful the responses of a cell are if they fall on a line that goes through the origin. Responses at the origin are to a null stimulus (blank gray), which indicates that the target firing rate is nota much different from baseline.

The hyperpixel STA for each neuron is presented in Figures 3A–C as the image at the top of each Figure. We have added the pixel white noise STA to these figures as well as bar plots of normalized cone weights (which the color rendering in the figure captures inadequately).

The Reviewer is correct that a point at the origin would indicate a response to a blank screen. None of the data points are at the origin, but some come very close to it and so does the fit. We have added a new supplementary figure (Figure 3 —figure supplement 3) that shows a zoomed in representation of the origin from Figures 3A and 3B.

Reviewer #2 (Recommendations for the authors):

General. I think this paper would benefit from a sharpening of terminology and usage throughout. For example, always being explicit about what sort of non-linearity is being considered in any given sentence, and about which of thet two types of white noise experiments is being referred to in each place where a reference is made. I could almost always figure out it what was intended from context, but that was extra work and left some uncertainty, as well distracted from following the flow of ideas. Specific examples given in some of the comments below.

We have edited the manuscript to better indicate which type of non-linearity and which type of white noise are being considered throughout.

Starting on page 3 and throughout. A key to understanding this paper is to understand the taxonomy the authors have applied to the neurons they recorded from. I struggled with the simple cell terminology, though. The criteria the authors use are different from more conventional ways used distinguish simple and complex cells. I think that if one applied F1/F0 criteria using luminance grating stimulation, one might well call at least some of the DO cells simple, as well as possibly some of the cells not studied. The criteria here are that the hyperpixel white noise analysis shows that cone contrast inputs are combined linearly and that the RFs are not cone opponent. I would have an easier time if the term simple cell were not used, to avoid confusion with the conventional taxonomy. Perhaps single opponent cells, or spatially-opponent non-cone-opponent cells. Thesse suggestion don't capture the linearity aspect of the criteria, but nor does the term double opponent and the authors are content with that. Maybe both terms should more clearly denote the linearity criteria. With a different choice of terminology, the relation to the conventional simple/complex taxonomy could be handled entirely in the discussion, with some care devoted to treating possible differences in classification. Avoiding the simple cell terminology might also allow ease exposition on the possible relation between NSNDO cells in the current study and chromatically sensitive complex cells, where I think more elaboration would be helpful.

As pointed out by the reviewer, several criteria had to be met for a neuron to qualify as simple or DO in this study: spatial opponency and the lack of a significant PC1(for both cell types), and cone opponency or cone non-opponency (for DO cells and simple cells, respectively). Capturing all of these criteria with concise labels is difficult. The term "single-opponent" has already been used to describe neurons that have uniform coneopponency throughout their RFs (Johnson et al., 2008; Livingstone 1984; Shapley 2011).

We agree that many of the DO cells that we studied would likely have been classified as simple cells by the standard F1/F0 criterion, which is almost always measured with isochromatic stimuli. Simple cells were first described in the cat, an animal with few cone-opponent V1 cells, and the distinction between cone-opponency and nonopponency is more important in the macaque. Double-opponent cells in macaques are, by definition, cone-opponent, and we know of no widely accepted term for doubleopponent cells that lack cone-opponency. Such cells have RFs consisting of spatially distinct, oriented ON and OFF RF subfields, and are almost certain to be categorized as simple cells by standard criteria. As pointed out by the reviewer, however, so are others.

To maximize the accessibility of our manuscript to the broad readership of eLife, we have elected to retain the “simple cell" terminology in our revision. We believe that this label captures the key characteristics of the approximately linear, spatially opponent, cone non-opponent, orientation-tuned cells that we studied. Nevertheless, we have added text explaining that our definition of simple cells is unconventional because is excludes cone-opponent cells and cells with certain types of nonlinearities. We have also revised the manuscript to clarify the categorization criteria, their rationale, and their relationships to classical criteria.

Page 3. Introduction: "The first stage of the primate visual system at which spatial comparison of spectral signals is implemented by individual neurons is area V1." – This statement is mysterious to me. Spatial comparisons are made at the first synapse, as signals from cones interact with signals from horizontal cells, and these interactions can involve spectral differences as signals from different classes of cones participate. Perhaps the authors mean something very specific when the qualify with "by individual neurons" but it is not clear what this is.

Thank you for pointing this out. V1 is the first stage of the primate visual system that contains DO cells. This has been clarified in the revised manuscript.

Page 4. I think more should be said about the 134 neurons that were recorded from and not studied further. These don't have a name in the current study and it is not entirely clear to me what caused them to be excluded. Perhaps only that they did not show any spatial opponency in the initial pixel white noise analysis (page 6), but I'm not sure that's the only reason so more detail on the exclusion criteria would be helpful.

We have expanded discussion of these neurons in the revised manuscript. There are three reasons that neurons were excluded from the current study:

1. The STA was indistinguishable from noise.

2. The STA did not reveal spatial opponency (e.g. it resembled a uniform blob).

3. Electrical isolation was lost early during or prior to Phase 3 (the isoresponse contour measurement).

Many neurons in the first group (and possibly some in the second group) may have been complex cells. Others may have had contrast thresholds that were too high, RFs that were too small to be adequately stimulated by the pixel white noise, or had noise levels that were too high to produce a clear STA. Neurons in the second group appeared to have spatially homogeneous RFs and were therefore unsuitable for the current study. We were unable to collect sufficient data from neurons in the third group.

Page 5. I agree with the authors that equating stimulus contrast across chromatic directions is a difficult and subtle matter, and that whenever possible it is useful to look for ways to frame questions that are robust to such equation. I diverge in that I don't see that their method avoids this difficulty. Just as one example, the frame selection criteria (pp. 21-22) are based on deciding which STA frame differs most from the background. That calculation is using some metric across color directions. Similarly with the criterion used to decide that the chromatic preferences of the two subfields of a DO neuron differ from each other.

We have edited the text to be clearer about the color space-specificity of the analyses.

As pointed out by the reviewer, the STA frame that differed most from the background was selected on the basis of the (background subtracted) R²+G²+B², which is a color space-specific statistic. Under the null hypothesis, this statistic has a scaled chi-squared distribution, which simplified the analysis. Had the analysis been done in, for example, (background subtracted) LMS space, correlations between the three color channels would have complicated the distribution (of L²+M²+S²). Correcting for these correlations is equivalent to doing the analysis in RGB space.

For each neuron, there were only 3–6 STA frames that were clearly different from noise. Across these 3–6 frames, R:G:B ratios changed little, so the light selected would have been similar under any reasonable selection rule (e.g. see Figure 2B). The frame selection procedure was used only to select lights on the two sides of the RF to stimulate the cell in Phase 3, and we have no reason to think that small changes in these lights would have had a large effect on the results.

As pointed out by the reviewer, Pearson’s correlation coefficient depends on the color space in which the analysis is performed. In the revised manuscript, we present a new analysis of cone weights estimated separately from the two subfields of each STA. This analysis (which is also color space-specific) confirms that chromatic preferences of the simple and DO cells are approximately opposite in this more-physiological color space.

Page 12. While we're on this point, the authors have used device primary RGB as their metric. The justify the reasonableness of this choice by writing "The fact that DO cells and simple cells combined light intensities across space approximately linearly in this representation shows that it is a reasonable one for analyzing V1 neurons." This seems odd, first because it assumes the answer to the question being studied in this paper. Second, because if the combination is revealed to be linear in one choice of color representation, it will also be linear in another choice that is a linear transformation of the first. (Indeed, the authors themselves say this in the sentence after the one quoted.) So I don't see how the linearity of the result tells us much about whether the representational choice was good. If nothing else, I would ask that the authors provide enough information about their device so that others can transform their data into a device independent representation such as LMS contrast with respect to a well-defined choice of cone fundamentals.

We agree and have removed the statement about the reasonableness of RGB space from the text. We provide tabulated primary emission spectra and background RGB levels (γ-corrected) at https://github.com/horwitzlab/Chromatic_spatial_contrast.

Page 6. Figure 2B. I assumed the noise is independent across color channels in RGB, but would be good to say so explicitly.

This information is now stated explicitly.

Page 8. Figure 3. What should we make of the high indices for some DO cells, and the low indices for many NSNDO cells? Both distributions suggest at least that possibility that neither group is a homogeneous population with respect to non-linear spatial summation. More discussion of this seems worthwhile.

We agree and have amended the Discussion to clarify our perspective that DO cells lie on a continuum with some NSNDO cells with regard to the linearity of spatial summation.

Page 10. How were the target firing rates chosen?

Baseline firing rates for each neuron were monitored online during Phases 1 and 2 of the experiment. Target firing rates were chosen such that they were substantially higher than the mean baseline firing rate (on average, 15.65 SDs higher). We have added a supplementary figure (Figure 3 —figure supplement 2) showing target firing rates and baseline firing rates for each neuron studied.

Page 13. "Nearly all quantitative studies of V1 RFs have used achromatic stimuli". Perhaps worth listing the studies of this sort that have used chromatic stimuli. Also, I think that by "V1 RFs" the authors mean "V1 spatial RF properties" and this might be worth spelling out.

We have revised the text as per the reviewer’s suggestions.

Page 13. "This maximal response exceeded the response to either bar in isolation, consistent with linearity as well as with other models." Might make explicit that the linearity in this sentence is the linearity of spatial summation, as well as that the reason the result could be consistent with summation linearity is that there could be a response non-linearity after spatial summation.

We have revised the text as per the reviewer’s suggestion.

Page 14. "Reclassifying cells …. " There's a lot of cell classification in this paper; perhaps sharpen here to emphasize that it is the simple/DO dichotomy classification being referred to here.

We have revised the text as per the reviewer’s suggestion.

Page 14. "Most other recent studies of DO cells used cone-isolating stimuli, which cannot reveal interactions among cone types (Conway, 2001; Johnson et al., 2001, 2004; Conway and Livingstone, 2006; Johnson et al., 2008)." Johnson et al., used mixtures in some of their reported measurements.

We have revised the sentence.

Page 14. "We classified neurons with nonlinear responses to white noise as NSNDO. This criterion was necessary to satisfy the assumptions underlying the conversion of the STA to cone weights." (A) In the first sentence, "nonlinear" seems too general since the paper believes it worthwhile to check for a specific type of nonlinearity in the simple and DO cells. (B) I don't understand this because the authors go on to apply the conversion of the STA to cone weights with the NSNDO cells, in that such conversion is used for the test in Phase 2, as shown in Figure 3F. (C) Here is an other places, it would be helpful to say "hyperpixel white noise" rather than just "white noise"

A. We have revised this part of the text to clarify that we mean specifically nonlinearities that manifest in a PC1 that is significantly larger than expected by chance (e.g. the contrast-polarity invariance exhibited by complex cells).

B. We use the STA of the NSNDO cells only to identify lights on the two sides of the RF that are likely to drive a strong response. We agree that calculating cone weights from the NSNDO cells is unlikely to be a useful conversion. Nevertheless, we do this in Figure 6 for the sake of completeness.

C. We have specified which type of white noise we are referring to throughout the revised manuscript.

Page 15. "Pooling LGN afferents with the same sign (ON or OFF) creates non-opponent spectral sensitivity." I'm not following what is meant here, need more detail on what is being assumed about chromatic properties of the afferents as well as exactly what is being pooled.

Thank you for pointing out this confusing section. The intended meaning is that pooling L- and M-cone-dominated ON afferents from one part of the visual space produces sensitivity to cone non-opponent increments, the precise details of which depends on the net spectral sensitivity of the afferents (e.g. L-cone dominated, M-cone dominated, or balanced). Surrounds are assumed to contribute little, spatially, relative to the center, and this is the case in the LGN under conditions similar to those examined in this study (e.g. (Horwitz 2020)). On the other hand, S-ON and S-OFF afferents carry strong, asymmetric L- and M-cone signals and therefore do not fit into this simple framework. This is explained more completely in the revised manuscript.

Page 17 and Figure 4. If the authors are going to show this data can they say anything about how the results for the experiments that are the focus of the current paper would come out for these two cells? That would connect these preliminary data better to the rest of the paper. And, did the authors find any cells like the one shown in 4C, but with the responses to luminance and chromatic gratings flipped?

Thank you for this question. We suspect, but do not know, that the neurons in Figure 7C (formerly Figure 4C) would have been either passed over in this study or would have been classified as NSNDO given their insensitivity to the phase of drifting gratings.

No, we have never seen a complex cell with a vigorous response to chromatic (L-M) modulation that did not also have a vigorous response to luminance (L+M) modulation (see also Horwitz, Chichilnisky, and Albright, 2007).

Page 22. Please provide criteria used to decide whether the two subfields had distinct chromatic preferences, and how this related to cell classification. On page 7, correlations (computed how?) between the two halves of the STA are given, and it sounds like this is result obtained before selecting or classifying cells. One page 24 a percent variance explained number is given, again independent of cell classification. But I'm not entirely sure that this property wasn't required for cells to be included for further study.

The criteria used to decide whether the two subfields had distinct chromatic preferences online were qualitative. The pixel STA was rendered on the display of the data acquisition computer. If the experimenter could identify two visually distinct regions of the STA, these regions were manually segmented for the hyperpixel white noise stimulus (Phase 2). As soon as the two subfields of the hyperpixel STA (again, rendered online) became visually distinct and clearly different from the background gray, the experiment advanced to Phase 3.

The % variance explained refers to how well the hyperpixel STA is recovered by combining the color and spatial weighting functions computed by SVD. All cells were subjected to this analysis. Every neuron in the data set had a spatial weighting function consisting of one positive and one negative weight, consistent with opposite spectral sensitivity in the adjacent subfields.

Cells that survived this filter were classified into simple, DO, and OSO categories. We computed Pearson’s correlation between the 2 sets of background-subtracted RGB values (one for each subfield) from the hyperpixel STA of each neuron. We have revised the text to explain this procedure more clearly.

Page 22. The authors note that using small pixels and a fast frame rate in the white noise experiments leads to the neurons being weakly driven. They might comment/think about whether the choices they made for these parameters are optimal – if they were to do the work again, could they consider using larger white noise pixels and a slower refresh rate to obtain better responses? That is, small/fast is at one end of a continuum, and maybe some other choice would be better for future work of this sort. Similarly, any requirement for use of the Gaussian distribution on the noise? Maybe a uniform or some other distribution would produce larger drive.

The pixel size was matched to the size of the average fixational saccade (0.2°) made by monkeys viewing similar displays (Hass and Horwitz 2011). Longer frame durations (or, really, repeated frames, since the stimulus was shown on a CRT) might have been preferable for this study because temporal aspects of the response were of secondary interest. Gaussian distributions facilitate the interpretation of the spike-triggered covariance analysis relative to uniform distributions, which we agree would likely drive the cells harder (Paninski 2003).

Page 22, last paragraph and into p. 23. The paragraph starts off talking about frames as six-dimensional vectors, but by the end it sounds like two separate 3-dimensional projections are done, yielding the two scalars one for each subfield. Would be clearer, I think, if only one of the two ways of talking about it is used here. Also might remind reader that this is the hyperpixel white noise STA, not the pixel white noise STA.

Thank you for pointing out this confusing passage. We have simplified the explanation and emphasized that the analysis was performed on two separate 3-dimensional projections onto the hyperpixel STA.

Page 23. We visualized a firing rate map from these projections by computing the ratio of spike-triggered stimuli to the total stimuli." I don't understand what metric of stimuli is being used in the ratio, which visualization in the paper this refers to, nor how the ratio of stimuli as described provides a map of firing rate.

Thank you for alerting us to this confusing section. The stimuli were represented as linearized, background-subtracted RGB values, and these stimuli were projected onto the two halves of the STA. Each stimulus therefore produced a pair of scalars that define a location in a plane (Figure 2C). Because the elements of the white noise stimulus are jointly Gaussian, this distribution of projections is a bivariate Gaussian.

This procedure was performed on every stimulus that was shown in the experiment. Some of these stimuli preceded spikes, and most did not. We gridded the plane into small, square bins and, within each bin, counted the number of stimuli that preceded a spike and the number that did not. We estimate the probability of spiking, conditional on a stimulus being in a particular bin, as the number of stimuli in that bin that preceded a spike divided by the total number of stimuli in that bin. We refer to the probability of spiking as a function of two stimulus projections as a firing rate map. This has been clarified in the revised text.

Page 23. "Direction represents the overall contrast between …" Worth noting that what these axes represent in stimulus space is cell-specific, in that the axes for each cell are derived from that cell's hyperpixel STA. Also, although this is a matter of convention and up to the authors, I would find it more intuitive if the STA axes were the x and y axes of the plot, not the 45 and 135 directions.

We have revised the text based on reviewer’s suggestions.

The rationale for the unusual plotting convention is that most of the stimuli that were sampled in Phase 3 provided positive contrast to both subfields. If we were to align the STA axes with the abscissa and ordinate, the figure would be rotated by 45°, and approximately half of the figure would be unused white space. In an effort to use the figure space efficiently, we have elected to retain the current rotation of the axes.

Page 24. 232 neurons were recorded from. 98 are reported on here. What can we say about the rest, and why they were not included here. The one clear exclusion criterion is that some neurons had only a single spatial region revealed by the pixel white noise STA. Was that the only reason? On this point, I don't think NSNDO is the best term for the set of neurons out of the 98 that were not classified as simple or DO, because of all the other neurons that were not included and that also weren't classified as simple or DO.

Please see our response to comment #26 for the answer to this question. We refer to the NSNDO cells as “other spatially opponent” (OSO) cells in the revised manuscript.

Page 24. Is it a theorem that if you take the first row singular vector and the first column singular vector, that this is the optimal color-space separable accounting of the STA? Or could a joint analysis do better?

Yes, the Eckart-Young-Mirsky theorem (Eckart 1936) states that the outer-product of the first column singular vector and the first row singular vector (multiplied by the first singular value) is the separable (rank 1) matrix that comes as close as possible to the original matrix in the sense of Frobenius norm. We have added a sentence to the text to inform the reader of the optimality of this procedure.

Page 24-25. On page 25, the classification of cells into simple, DO, and NSNDO is described. Because the term "white noise" is used to describe the protocol used for the classification and because this applies to both Phase 1 (pixel white noise) and Phase 2 (hyperpixel white noise) of the experiment, it would be good to remind the reader that it is (I think) the hyperpixel white noise that's relevant here. Indeed, I would try to be explicit each time in the paper the term white noise or STA is used, which of the two it refers to. Also, why is a substantial S cone input a requirement to be a DO cell? This seems like it would exclude cells with perfectly good spatially opponent L-M inputs. And, I'm not clear what happens to such cells, if there were any. I think from the next sentence that like neurons with a significant PC1, they get called NSNDO. But because the flow of this section had me thinking at the start that it was just those neurons with signifiant PC1 that were NSNDO, I'm also not clear on this. If they are NSNDO, then it is worth being explicit elsewhere that heterogeneous criteria were used to place neurons in this category, and perhaps looking at NLI separately for cells that arrived as NSNDO viat different paths.

We apologize for the ambiguity and explicitly mention which white noise (pixel or hyperpixel) was used throughout the revised text.

Our criteria for double-opponency did admit neurons with spatially opponent L-M inputs and small (or even non-existent) S-cone inputs. If a neuron had oppositely signed L- and M-cone weights, each of which accounted for at least 20% of the total, the neuron was classified as double-opponent, irrespective of the magnitude of the S-cone weight. The threshold on weight magnitude prevented neurons with non-opponent cone weights from being misclassified (small, independent noise in RGB produces small, anticorrelated noise in LM). Additionally, neurons were classified as DO if they had oppositely signed cone weights and an S-cone weight that accounted for at least 20% total. The former criteria selected the opponent L-M DO cells whereas the latter criteria selected the S-M+L , S+M-L and S-M-L DO cells (irrespective of the magnitudes of their L- and M-cone weights). We have rewritten the text to clarify the DO cell classification better.

The revised text is more explicit about the multiple criteria that were used to place neurons in the OSO (formerly NSNDO) category. Large differences across neurons that arrived in this category via different routes were looked for in response to this comment but not observed. (75% of the cells in the OSO category arrived via the PC1 criterion.)

Page 26. I'm surprised that Equation 1 doesn't include an amplitude parameter that captures the overall responsivity of the cell.

The amplitude parameter in Equation 1 is exp(c), where "c" is a fitted parameter. “Pred resp” in Equation 1 represents the probability that a stimulus will evoke a spike, and therefore has an upper asymptote of 1.

Figure 2 —figure supplement 1. I am not sure what is in panel D. Are these from Phase 2? And how does the 5th and 95th percentile information mentioned in the caption show up in the figure? More generally, I think the point of this figure is to provide a sense of how much harder the Phase 3 stimuli drive the cells than the Phase 1 and 2 stimuli, but I am not seeing this in the figure itself. Where are the Phase 3 data?

Thank you for alerting us to this confusing figure. Panel D shows data from Phase 2 in the same format as panel A. The square firing rate map in the center of panel D shows, in grayscale, the proportion of stimuli (the red dots in panel A) that drove spikes. The 5th and 95th percentiles relate to the size of this square; had we included stimuli from the 1st to 99th percentiles, the box would have subsumed nearly all of the red points in panel A and thus been larger, but our estimate of the proportion of stimuli that drove a spike in bins near the edges would have been noisier because counts in these bins would have been low.

The previously submitted manuscript did not include an analysis of how much harder the Phase 3 stimuli drove the cells than the Phase 1 and 2 stimuli. We include this new analysis in Figure 2 —figure supplement 2 of the revised manuscript.

Figure 2 —figure supplement 2. Can you clarify the temporal parameters that go into this analysis?

We considered 15 frames of the hyperpixel STA prior to when a spike occurred, which is equivalent to 200 ms (frame rate = 75 Hz)(Figure 2B). We then convolved the hyperpixel white noise video with the two halves of the hyperpixel STA to obtain the projection values for each time bin. We have clarified these temporal parameters in the revised text.

Figure 3 —figure supplement 3. The staircase shown suggests that using the last stimulus presented does not give as good an estimate of stimulus that produces the criterion firing rate as would an analysis of the whole staircase. Do I understand that the last stimulus magnitude is what was used? If so, please elaborate why. If not, please clarify.

Yes, the final stimulus contrast magnitude was used to estimate the isoresponse point. Inspired by this comment, we repeated the analysis, defining isoresponse points in two new ways: first, as the geometric mean of the last 5 staircase points (geometric mean because the contrast was scaled multiplicatively in the staircase), and second, interpolation using a Naka-Rushton fit to the entire staircase assuming Poisson error. As shows in Author response image 2 and Author response image 3 the results from these three procedures are almost identical. For simplicity, we have decided to stay with the original definition.

Author response image 2

Download asset Open asset

Author response image 3

Download asset Open asset

References

Eckart C, and Young, G. The approximation of one matrix by another of lower rank. Psychometrika 1: 211-218, 1936.

Hass CA, and Horwitz GD. Effects of microsaccades on contrast detection and V1 responses in macaques. J Vis 11: 1-17, 2011.

Horwitz GD. Temporal information loss in the macaque early visual system. PLoS Biology 18: e3000570, 2020.

Horwitz GD, and Hass CA. Nonlinear analysis of macaque V1 color tuning reveals cardinal directions for cortical color processing. Nature Neuroscience 15: 913-919, 2012.

Johnson EN, Hawken MJ, and Shapley R. The orientation selectivity of color-responsive neurons in macaque V1. The Journal of Neuroscience 28: 8096-8106, 2008.

Livingstone MS, Hubel, D. H. Anatomy and physiology of a color system in the primate visual cortex. J Neurosci 4: 309-356, 1984.

Paninski L. Convergence properties of some spike-triggered analysis techniques. Advances in neural information processing systems 189-196, 2003.

Shapley R, and Hawken, M. J. Color in the cortex: single-and double-opponent cells. Vision research 51: 701-717, 2011.

Author details

1. Abhishek De

   1. Graduate Program in Neuroscience, University of Washington, Seattle, United States
   2. Department of Physiology and Biophysics, Washington National Primate Research Center, University of Washington, Seattle, United States

   Contribution

   Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-2978-473X

2. Gregory D Horwitz

   Department of Physiology and Biophysics, Washington National Primate Research Center, University of Washington, Seattle, United States

   Contribution

   Conceptualization, Methodology, Writing – original draft, Writing – review and editing, Funding acquisition, Supervision

   For correspondence

   ghorwitz@uw.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5130-5259

• 743

  Page views

• 89

  Downloads

• 0

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.