Having conscious experience is arguably the most important reason why it matters to us whether we are alive or dead. The question which signals in the brain reflect this conscious experience and which reflect obligatory processing of input regardless of conscious experience is a central puzzle of neuroscience. For example, activations in the retina may correlate with the conscious percept of flashing light but are arguably entirely driven by physical input, much of which never evolves into a conscious percept. Another driver of neural activity that can be confounded with signals related to conscious perception is report. Recently, it has been suggested that brain regions may correlate with conscious perception simply because they are driven by the active report of it (Aru et al., 2012; Block, 2019; Block, 2020; Boly et al., 2017; Frässle et al., 2014; Koch et al., 2016; Overgaard and Fazekas, 2016; Panagiotaropoulos et al., 2020; Safavi et al., 2014; Tsuchiya et al., 2016; Tsuchiya et al., 2015).

A paradigm known as binocular rivalry is useful for distinguishing responses related to conscious perception from those driven by obligatory processing of physical input (Blake et al., 2014; Tong et al., 2006): when two incompatible stimuli such as a face and an object are shown to the left and right eyes, respectively, one does not perceive a constant superimposition of the two but rather an alternation between face and object, even though the physical input is fixed (Figure 1a). Since these alternations are internally generated, they cannot be attributed to pure feedforward processing of external input.

Figure 1

Download asset Open asset

In previous studies, researchers trained monkeys to report their percept during binocular rivalry by releasing a lever. They found that the proportion of cells modulated by the reported percept increases along the visual hierarchy, with 20% of cells showing modulations in V1 (Leopold and Logothetis, 1996) compared to 90% of cells showing modulations in inferotemporal (IT) cortex (Sheinberg and Logothetis, 1997). Using functional magnetic resonance imaging (fMRI), Tong et al., 1998 found that the human fusiform face area responds to reported perceptual switches. Using single-unit recording, Gelbard-Sagiv et al., 2018 found that the activity of neurons in the human medial temporal lobe and frontal cortex is also modulated by the reported percept.

Although binocular rivalry dissociates the conscious percept from physical input, an important confounding factor remains. In all studies cited above, the monkey or human subject always actively reported their percept by a motor response. Thus it is possible that the observed neuronal activations were due to the act of report itself, including introspection, decision making, and motor action accompanying report, rather than a switch in conscious percept. This concern was emphasized in an fMRI experiment by Frässle et al., 2014 who compared modulations in the brain with and without active report. Many of the modulations observed in higher-level brain regions such as the frontal lobe disappeared when subjects did not actively report perceptual switches.

To infer the subject’s percept in the absence of report, Frässle et al. used two no-report paradigms that depended on pupil size and optokinetic nystagmus, respectively. To exploit pupil size, they presented stimuli with different brightness in the two eyes, causing the subject’s pupil size to vary according to the dominant percept’s brightness. To exploit optokinetic nystagmus, they presented gratings moving in opposite directions in the two eyes, causing the subject’s eye position to reflexively follow the direction of the dominant grating. Therefore, the conscious percept could be inferred by reading out pupil size and drift of eye position, respectively.

These no-report paradigms allow accurate prediction of the subject’s percept but are not free of confounds themselves (Overgaard and Fazekas, 2016). First, pupil size is known to correlate with arousal, surprise, attention, and other confounding factors (Bradley et al., 2008; Hoeks and Levelt, 1993; Preuschoff et al., 2011). Second, when optokinetic nystagmus is applied to moving non-grating stimuli such as natural objects that drive IT cortex, there will be confounding physical stimulus differences. For example, the dominant stimulus that is smoothly pursued by the subject’s eyes will tend to be stationary on the subject’s fovea and optimally modulate IT areas with foveal biases, while the non-dominant stimulus will be more eccentric and have increased motion velocity. Moreover, optokinetic nystagmus is still present in monkeys in which the conscious percept is diminished due to anesthesia with low doses of ketamine (Leopold et al., 2002).

Here, we introduce a new no-report paradigm that relies on active tracking of a fixation spot, unlike the reflex-based paradigms mentioned above. In this fixation-based paradigm the subject is required to maintain fixation on a jumping spot, a task that many animals in vision research are already trained to perform. While following the fixation spot, subjects view either unambiguous, monocular stimuli physically switching between a face and an object, or a binocular rivalry stimulus that switches only perceptually. For the binocular rivalry stimulus, a fixation spot is shown to each eye at different positions on the screen. Thus, when perceiving a face in the left eye, the subject will generally perceive only the fixation spot in the left eye and saccade to it, ignoring the fixation spot in the right eye. In this way, the subject’s percept can be inferred from eye movement patterns without active report.

In a second innovation, we performed electrophysiological recordings using a novel 128-electrode site Neuropixels-like probe that allowed us to measure responses from large numbers of cells simultaneously. This allowed us to address for the first time the extent to which neural activity is modulated by conscious perception in single trials. Sheinberg and Logothetis, 1997 reported that 90% of IT cells are modulated by conscious perception. However, a fact that has been largely overlooked is that the response modulations found in that study during the rivalry condition were clearly smaller than those in the physical condition. It is possible that the decrease arose due to incorrect reporting of the percept by the monkey on some trials, and cells were modulated just as strongly by perceptual as by physical alternations. However, the decrease could also have been due to a more interesting possibility: mixed selectivity of cells for the conscious percept and the suppressed stimulus on single trials in the rivalry condition. In other words, it is possible that single cells encode both the conscious percept and the suppressed stimulus during rivalry. Inter-trial averaging confounds these two possibilities. To distinguish them, it is critical to compare perceptual vs. physical response modulations for single trials.

To explore correlates of conscious perception, we targeted recordings to macaque face patches ML and AM. The macaque face patch system constitutes an anatomically connected network of regions in IT cortex dedicated to face processing (Chang and Tsao, 2017; Grimaldi et al., 2016; Hesse and Tsao, 2020; Tsao et al., 2006) and has served as an archetypal system for understanding object recognition in IT in general (Bao et al., 2020). To date, most response properties of cells in the face patch network can be explained using a feedforward framework without invoking conscious perception. For example, the functional hierarchy of this network, with increasing view invariance as one moves anterior from ML to AM (Freiwald and Tsao, 2010), can be explained by simple feedforward pooling mechanisms (Leibo et al., 2017). The representation of facial identity by cells in face patches through projection onto specific preferred axes can also be explained by feedforward mechanisms (Chang and Tsao, 2017).

Here, we explore activity in the face patch network using a binocular rivalry paradigm in which neural activity modulation is difficult to explain by feedforward filtering processes, since the stimulus remains unchanged. The hierarchical and feedback-rich organization of the face patch network (Freiwald and Tsao, 2010; Grimaldi et al., 2016) makes it a ripe testbed to examine the neural circuits underlying construction of conscious visual experience beyond feedforward filtering of visual input. It has been postulated that the fundamental architecture of the cortex is a predictive loop in which inference guided by internal priors plays a key role in determining what we see (Rao and Ballard, 1999). One explanation for binocular rivalry is that it arises as a consequence of such predictive coding, reflecting a high-level prior that two objects cannot occupy the same space (Hohwy et al., 2008).

We recorded from fMRI-identified face patches ML and AM in two monkeys using high channel-count electrodes, while we inferred the animals’ conscious percept through the no-report paradigm described above. We found that large proportions of cells in both face patches (57% in ML and 73% in AM) encoded the conscious percept even without active report. Population activity of perceptually modulated cells was modulated more weakly during rivalry than during physical stimulus transitions in single trials. Nevertheless, we could reliably decode the dynamically changing conscious percept from activity in single trials. Surprisingly, we could also decode suppressed stimuli using activity from the same cells, indicating that single cells multiplex information about the conscious percept and the suppressed stimulus. These findings suggest that the neural correlate of consciousness within IT cortex resides in a population code rather than a subset of cells perfectly reflecting consciousness, and different linear readouts can decode either the consciously perceived or the suppressed stimulus from the same population.

We first confirmed that it is possible to correctly infer a subject’s conscious percept using a fixation-based no-report paradigm through a behavioral experiment in humans. We presented binocular rivalry stimuli consisting of a face (e.g., Obama) in the right eye and a non-face object (e.g., a taco) in the left eye, causing the percept to stochastically alternate between the two (Figure 1a). Each of the stimuli contained a fixation spot that jumped to one of four possible locations every trial. Trials were 2000 ms long and contained no blank period, that is, stimuli were presented continuously. If subjects fixated at the fixation spot presented in the right eye on a given trial, we inferred that they perceived the face and vice versa for the object. To verify that the percept of face or object could be inferred from fixations, we instructed six naïve human subjects to perform the fixation task while simultaneously reporting their conscious percept with button presses. On trials where the percept switched, subjects also switched the fixation spot they were following (Figure 1b). We were able to infer which image the subjects were consciously perceiving with accuracies ranging from 86% to 98% across subjects (average: 93%, Figure 1c).

We next used the same method in monkeys to infer their conscious percept while recording from face patches ML and AM in IT. Importantly, the two monkeys in this study had never been trained to report their percept. They had previously been trained to maintain fixation on a spot (presented binocularly) and learned to perform the new task within 1 or 2 days, respectively (maintaining fixation on a spot for at least 80% of all trials). Since the monkeys were so adept at the task, we set the trial length to 800 ms (compared to 2000 ms in humans); this allowed higher temporal fidelity in determining the animal’s percept. We presented two types of stimuli: in the ‘physical’ condition, unambiguous monocular stimuli were physically switched between face and object. In the ‘perceptual’ (binocular rivalry) condition, the same face and object were continuously presented to the right and left eyes, respectively, so any changes in percept were internally generated. To account for individuals’ eye dominance, we balanced the contrasts of the stimuli in the two eyes so that the monkey followed both fixation spots equally often in the rivalry condition. After balancing, median dominance durations were 7.2 s for faces and 7.2 s for objects across the two monkeys. Similarly, in human subjects, median dominance durations were 8 s for faces and 10 s for objects as estimated from fixation patterns, and 8.1 s for faces and 8.3 s for objects as estimated from reports. We inferred switches during rivalry when monkeys behaviorally switched from following the fixation spot in one eye to following the fixation spot in the other eye (example eye traces, Figure 2a, top). Spike rasters from an example ML cell showed a stronger response after switches from face to object compared to switches from object to face (Figure 2a, bottom; rasters aligned to onset of trials in which a switch occurred). Figure 2b compares average response time courses to physical vs. perceptual switches in two example cells, one from ML and one from AM. Both cells responded more strongly to a physically presented face than object. Importantly, in the binocular rivalry condition the response of both cells was also higher when the monkey perceived a face (as inferred by its eye movement) than when the monkey perceived an object. Since the physical stimulus was constant in this condition, the response reflected the monkey’s conscious percept of a face and not just the physical input.

Figure 2

Download asset Open asset

We recorded a total of 348 cells in ML and 210 cells in AM that were selective, i.e., they showed a significant difference between face and object in the physical switch condition (${\displaystyle \text{p}<0.05}$, two-sided two-sample t-test). Since we recorded from face patches, most cells showed stronger responses to the physically presented face stimulus. Importantly, most cells kept their preference in the binocular rivalry condition (Figure 3). In face patch ML, 57% (200/348) of cells were significantly modulated by the conscious percept in the binocular rivalry condition and showed preference consistent with the physical switch condition (${\displaystyle \text{p}<0.05}$, two-sided t-test), while 10% (34/348) of cells were significantly but inconsistently modulated. In AM, a face patch that receives input from ML (Grimaldi et al., 2016) and is the highest patch in the face patch hierarchy within IT (Freiwald and Tsao, 2010), the percentage of significant consistent modulation increased to 73% (153/210), with only 2% (5/210) showing significant inconsistent modulation. For both patches there was a clear correlation between modulation by the physical stimulus and modulation by the percept in binocular rivalry (${\displaystyle \text{p}=2\times {10}^{-83}}$, Pearson’s $r=0.70$, ${\displaystyle \text{N}=558}$ cells). Thus, in a no-report paradigm, cells in IT exhibit modulations by the conscious percept that reflect their response selectivity to physically unambiguous inputs.

Figure 3 with 1 supplement see all

Download asset Open asset

After eliminating the report confound, two important potential confounds remain. First, cells could be selective for the eye-of-origin of the fixation point that the animal is following (e.g., a cell could respond selectively to a fixation spot in the fovea of the left eye). Second, since we presented binocular stimuli using red/cyan anaglyph goggles, a confound could arise if cells were selective for the color of the fixation spot that is in the fovea. To control for these two potential confounds, we switched the colors and eye-of-origin of the face and object stimuli, that is, where the face and its corresponding fixation spot was previously presented in red in one eye, it was now presented in cyan in the other eye and vice versa for the object (Figure 3—figure supplement 1). If cells followed color or eye-of-origin, then all the dots in the upper right quadrant in Figure 3—figure supplement 1a should move to the lower left quadrant in Figure 3—figure supplement 1b. Instead, the majority of cells followed the object identity rather than color or eye-of-origin for both the physical and perceptual conditions (${\displaystyle \text{p}=9\times {10}^{-42}}$ for physical condition and ${\displaystyle \text{p}={10}^{-19}}$ for perceptual condition, one-sided t-test, ${\displaystyle \text{N}=313}$ cells, alternative hypothesis: modulation indices for switched condition are greater than 0). This confirms that cells in IT cortex indeed represent the conscious percept rather than the color or eye-of-origin of the fixation spot.

The strong modulation by conscious percept in single cells suggests that we should be able to decode the percept on single trials from population activity. To test this, we performed recordings from multiple neurons simultaneously using S-probes with 32 electrode sites and passive Neuropixels-like probes with 128 electrode sites (see Materials and methods for details). Figure 4 shows the recordings from face patch ML in one session using the Neuropixels probe. In this session, we recorded 81 cells simultaneously, of which 63 were face-selective (Figure 4a). An example population time course snippet of cells recorded simultaneously in the perceptual switch condition showed clearly stronger activity across the recorded population during perception of face compared to object (Figure 4b). The average population response across cells to perceptual switches is shown in Figure 4c. We found above chance decoding of the perceptual condition in all 12 sessions (in all but one session, responses were recorded in both ML and AM, and cells were pooled across the two patches). Cross-validated accuracies of linear classifiers across different sessions are shown in Figure 4d (see Materials and methods). Decoding accuracy was 99% for the best session and 95% on average for the physical condition. For the perceptual condition, decoding accuracy was 88% on the best session and 78% on average.

Figure 4

Download asset Open asset

Looking at the population time course, we noticed bursts of activity that appeared to be triggered by saccades, which occurred even when an object was perceived (blue epochs in Figure 4b; small black dots on top indicate detected saccades). This suggested to us that cells modulated by perception might still carry information about the physical stimulus: the bursts may have been caused by responses to the suppressed face stimulus. To investigate this further, we selected cells that (i) showed both significant physical and perceptual modulation and (ii) consistently preferred the face over the object. We then averaged responses across these cells and computed response time courses triggered by individual saccades, grouped by whether a saccade occurred during a trial inferred to be face or object, respectively (Figure 5). We observed response modulations for both physical and perceptual conditions starting around 130 ms after saccade onset (Figure 5a). In the physical condition, a saccade during an object epoch led to response suppression, while a saccade during a face epoch led to response increase. In striking contrast, in the rivalry condition saccades led to response increase in both object and face epochs. As a consequence, during rivalry the response difference to a saccade between face and object, though significant (${\displaystyle \text{p}=6\times {10}^{-23}}$, two-sample t-test, ${\displaystyle \text{N}=701}$ saccades for object, ${\displaystyle \text{N}=703}$ saccades for face), was weaker than during the physical condition. Computing histograms of responses averaged across neurons for individual saccades shows that responses in the rivalry condition were less bimodal and spanned a smaller range compared to the physical condition (Figure 5b). Importantly, this difference in response profiles between physical and perceptual conditions was apparent even when pooling across both face and object trials (Figure 5b, middle), and hence cannot be explained by mistakes in inferring the percept from eye movements. We computed the absolute value of these responses and found the difference in response distributions to be significant (Figure 5b, right, ${\displaystyle \text{p}=6\times {10}^{-35}}$, two-sample t-test on absolute value distributions, ${\displaystyle \text{N}=229}$ saccades for physical condition, ${\displaystyle \text{N}=1404}$ saccades for perceptual condition).

Figure 5 with 1 supplement see all

Download asset Open asset

The observation of different response profiles for physical and perceptual conditions was not specific for saccades: histograms were also less bimodal and spanned a smaller range for the rivalry condition when triggering responses on trial onsets rather than saccades in both ML (Figure 5—figure supplement 1a, ${\displaystyle \text{p}=9\times {10}^{-15}}$, two-sample t-test on absolute value distributions, ${\displaystyle \text{N}=150}$ trials for physical condition, ${\displaystyle \text{N}=571}$ trials for perceptual condition) and AM (Figure 5—figure supplement 1b, ${\displaystyle \text{p}=0.0014}$, two-sample t-test on absolute value distributions, ${\displaystyle \text{N}=120}$ trials for physical condition, ${\displaystyle \text{N}=480}$ trials for perceptual condition). Therefore, it appears that throughout rivalry, for perceptually modulated cells, response differences to face and object are less pronounced than in the physical condition, and this is true in both ML and AM. One tantalizing explanation for this phenomenon is that perceptually modulated cells may be multiplexing information about both the physical stimulus and the perceptual state during single trials, allowing both to be simultaneously represented across the face patch hierarchy.

Is it possible that the apparent responses to the suppressed face were due to incomplete suppression, leading to piecemeal percepts on some trials? We performed simulations of the worst-case effect of mixture, in which the percept would be exactly half-face and half-object, by taking the responses of the physical condition and averaging responses to face and body on a specific proportion of trials. The simulated distributions only became statistically indistinguishable from the observed binocular rivalry condition if 50–70% of trials were mixed percepts of half-face and half-body. This is markedly inconsistent with reports from every human subject that on most trials they did not perceive any mixture. We of course cannot be absolutely sure that monkeys do not experience mixed percepts significantly more often than humans. Yet, under the reasonable assumption that percepts were similar in the two species, trials with mixed or piecemeal percepts cannot account for the difference in response distributions between physical and perceptual conditions.

To directly test the hypothesis that cells multiplex information about the perceptually dominant and suppressed stimulus, we performed a new experiment in which we varied the identity of the suppressed stimulus. In this experiment, instead of having only two rivalling stimuli, we used three images A, B, and C to create two different binocular rivalry stimuli, (A,B) and (A,C), presented in separate blocks (Figure 6a). This allowed us to keep the dominant percept fixed as image A, and compare responses to trial types (A,B) and (A,C) to test whether neural responses could discriminate the suppressed stimulus (bold font indicates the dominant image, as inferred by eye movements). We trained a linear decoder to distinguish between trial types (A,B) and (A,C). Remarkably, the decoding accuracy for distinguishing the two trial types was 74% (Figure 6b). For comparison, the decoding accuracy for distinguishing (A,B) vs. (A,C) from the same cell population was 88%. Thus, while the conscious percept can be decoded better than the suppressed stimulus, face cells do encode significant information about the latter. Potential mislabeling of trials by the no-report paradigm could not account for this decoding accuracy (see Supplementary text).

Figure 6

Download asset Open asset

Do the same cells multiplex information about both the conscious and suppressed stimuli, or are there two distinct subpopulations, one encoding the conscious stimulus and another encoding the suppressed stimulus? To address this question, we compared modulation indices for the dominant stimulus with modulation indices for the suppressed stimulus for each cell. For the former, we fixed the suppressed stimulus while varying the dominant stimulus, that is, we compared responses to trial type (A,B) with responses to trial type (A,C) to compute the modulation index ${\displaystyle M{I}_{\text{dominant}}=\left(R_{\mathrm{A}\mathbf{B}}-R_{\mathrm{A}\mathbf{C}}\right)/\left(R_{\mathrm{A}\mathbf{B}}+R_{\mathrm{A}\mathbf{C}}\right)}$. For the latter, we fixed the dominant stimulus while varying the suppressed stimulus, that is, we compared responses to trial type (A,B) with responses to trial type (A,C) to compute the modulation index ${\displaystyle M{I}_{\text{suppressed}}=\left(R_{\mathbf{A}\mathrm{B}}-R_{\mathbf{A}\mathrm{C}}\right)/\left(R_{\mathbf{A}\mathrm{B}}+R_{\mathbf{A}\mathrm{C}}\right)}$ (Figure 6c). We found a positive correlation between dominant stimulus modulation indices and suppressed stimulus modulation indices (${\displaystyle \text{p}=1.4\times {10}^{-6}}$, Pearson’s $r=0.55$, ${\displaystyle \text{N}=66}$ physically selective cells). This suggests that cells strongly modulated by the dominant stimulus tend to be similarly modulated by the suppressed stimulus. Thus there are not two separate populations of cells that encode conscious and unconscious stimuli.

In summary, our findings indicate that the neural correlate of consciousness in IT does not reside in a subset of cells perfectly reflecting consciousness but rather in a population code. This is supported by the findings that (i) modulation by the conscious percept is weaker than modulation by the physical stimulus (Figures 3 and 5), (ii) both consciously perceived and suppressed stimuli can be decoded from the same population (Figure 6b), and (iii) modulation indices for consciously perceived and suppressed stimuli are correlated in single cells (Figure 6c).

In this study, we developed a new no-report paradigm for tracking conscious state and used it to investigate the neural correlate of consciousness in face patches within macaque IT cortex. We made two new findings. First, we found that face patches ML and AM are modulated by conscious perception and do not merely encode the physical input. Importantly, monkeys in this study had never been trained to actively report their percept. Instead, we were able to infer their percept from eye movements using a new no-report paradigm. Thus activity modulations attributed to switches in conscious perception in IT cannot be explained simply by active report. Second, we found that cells in face patches are modulated by the identity of both the consciously perceived stimulus and the suppressed stimulus, such that both stimuli can be read out from the same population using different linear decoders. This finding challenges the widely held notion that in IT cortex almost all neurons respond only to the consciously perceived stimulus.

Previous single-unit recordings in IT cortex using active report to infer the percept found 90% of cells represent the conscious percept (Sheinberg and Logothetis, 1997). Here, we found proportions of 57% in ML and 73% in the more anterior patch AM. The quantitative difference may be due to several factors including different recording sites (Sheinberg and Logothetis recorded from both upper and lower banks of the superior temporal sulcus in a less specifically targeted manner), imperfect accuracy of the no-report paradigm, and differences in stimuli and analysis methods. Importantly, our results confirm that the majority of cells in IT cortex do represent conscious perception. Furthermore, this new paradigm makes studies of consciousness in monkeys more accessible, by replacing the need to train the animal to signal its conscious percept (which can be a laborious process) with a simple task that only requires animals to follow a fixation spot.

Our results show that for cells that are modulated by conscious perception, the modulation is not ‘all-or-none’. Instead, the average response modulation during the perceptual condition was weaker than during the physical condition (Figure 3). This was also observed in a previous study of rivalry (Sheinberg and Logothetis, 1997), but somehow, this fact has been forgotten in popular lore surrounding the neural correlates of consciousness. For example, the Wikipedia entry for ‘neural correlates of consciousness’ states that “in [inferior temporal cortex] almost all neurons responded only to the perceptually dominant stimulus, so that a ‘face’ cell only fired when the animal indicated that it saw the face and not the pattern presented to the other eye”. We think the reason this fact - the decreased average modulation of IT cells by switches in conscious percept compared to switches in physical stimulus - has not garnered much attention up to now is that it could, at least up to now, be simply explained by imperfect labeling of the animal’s perceptual state.

The key question is: what happens during single trials? In the rivalry condition, do responses in single trials look like those to either physically presented faces or objects? By recording from a large number of face cells simultaneously using a novel 128-electrode site probe specifically designed for use in primates, we could address this question for the first time. Surprisingly, we found a dramatically different response profile on single trials between the perceptual and physical conditions (Figure 5). Although in the physical condition responses clustered into two groups, in the rivalry condition responses appeared unimodal, lying in between the two clusters for the physical condition. This suggests that single cells are multiplexing the conscious percept and the veridical physical stimulus during single trials. To directly test this hypothesis, we presented more than one binocular rivalry stimulus, created from pairs of three images, and found that the subconscious stimulus could indeed by decoded from face patch activity. Moreover, the same cells that were strongly modulated by the conscious percept also tended to be strongly modulated by the suppressed stimulus, ruling out the existence of a subpopulation of cells in IT purely reflecting consciousness. These findings strongly suggest that rivalry is not fully resolved before IT. It remains an open question where and how the conscious percept is ultimately isolated from the suppressed stimulus to produce conscious awareness of the former and not the latter.

In Figure 6d, we sketch three models for how perceptually modulated cells in IT cortex could encode stimuli during binocular rivalry. In Model I, cells exactly reflect the conscious percept, encoding it the same way they would encode an unambiguous stimulus. In Model II, the response during binocular rivalry is in between the responses to the two unambiguous stimuli, with the contributions of the two stimuli weighted differently depending on which stimulus is dominant and which stimulus is suppressed. Thus, both the consciously perceived stimulus and suppressed stimulus can be decoded using two different decoders. For Model II, one can further distinguish between two different sub-models depending on whether consciously perceived and suppressed stimuli are encoded by different subsets of spikes or not: in Model IIa, spikes are stochastically generated from the average firing rate on a trial, which is determined by a linearly weighted sum of consciously perceived and suppressed stimuli. Alternatively, in Model IIb, there are two different types of spikes that encode the conscious percept or physical stimulus, respectively. The type of a spike may depend on the phase of a high-frequency oscillation at which the spike occurs (the oscillation would need to be faster than alternations in perceptual dominance), or on whether the spike occurs synchronously with spikes from other neurons. Unlike Model IIa, Model IIb harbors an explicit neural correlate of the conscious percept within a subset of spikes. Importantly, our result that the suppressed stimulus can be decoded rules out the cartoon picture of Model I. Our findings are compatible with both Models IIa and IIb, and future experiments may be able to distinguish between the two.

Compared to previous approaches that attempted to isolate representations of the conscious percept, our new no-report binocular rivalry paradigm has several advantages. For flash suppression, where a stimulus flashed in one eye suppresses the stimulus in the other eye, report is also not required (Tsuchiya and Koch, 2005; Wilke et al., 2003; Wolfe, 1984). However, in that case, the physical input when the target is perceived is not identical to that when it is suppressed, and thus any modulation observed may be driven entirely externally. Indeed, it is known that if a distractor stimulus is presented simultaneously with a preferred stimulus, the response can be reduced compared to when the preferred stimulus is presented alone, due to simple normalization mechanisms (Bao and Tsao, 2018). Another paradigm that has been widely used to study the neural correlates of consciousness is backward masking. Here, the stimulus is presented for such a short time before being masked that sometimes it enters consciousness and sometimes not (Breitmeyer et al., 1984). So far, backward masking has always relied on report. Also, it is more susceptible to modulations arising from bottom-up withdrawal of attention or low-level (e.g., retinal) noise, whereas in binocular rivalry perceptual switches appear to be internally generated. One potential confound described by Block as the ‘bored monkey problem’ is that the monkey may still be thinking about whether it is perceiving object or face and internally report it even if it is not required to actively report it (Block, 2020). It is methodologically very difficult to entirely remove this confound, but the fact that monkeys had to simultaneously perform a very challenging unrelated task of saccading to jumping fixation points should at least alleviate this concern.

Alternative approaches to the no-report paradigms of Frässle et al., 2014 have been developed in which the monkey or human subject is unaware of when a perceptual switch is happening and hence cannot report it, either due to anesthesia or due to the difference in stimuli being too subtle to report. Brascamp et al., 2015 reported that fMRI responses to binocular rivalry switches in fronto-parietal regions disappear when the difference between the percepts is made so subtle that subjects cannot report it; however, it is possible that the difference between the percepts was just too small to be picked up by the fMRI signal. Zou et al., 2016 created rivalry stimuli from orthogonal gratings where the grating in one eye was flickered fast enough that it was perceived as uniform gray and only produced fMRI activations in early visual cortex. These stimuli evoked rivalry according to behavioral reports whereas physically uniform stimuli do not, indicating that competition occurred in early visual cortex. In another study consistent with competition in early visual cortex, Xu et al., 2016 performed optical imaging in V1 while monkeys were anesthetized. They found that during binocular rivalry activations clearly alternated in counter-phase between left eye and right eye dominance columns. We note that competition occurring in V1 is not incompatible with our findings, although our findings suggest that rivalry is unlikely to be fully resolved in early areas, given our ability to decode the suppressed stimulus from cells in IT. It should also be noted as a caveat that hemodynamic signals, as measured in the above studies by fMRI or optical imaging, only indirectly reflect neural activity and have previously shown discrepancies with single-unit responses (Leopold and Logothetis, 1996; Tong and Engel, 2001). Overall, to the best of our knowledge, the current study describes the representation of conscious and subconscious stimuli in IT cells in the most confound-free way to date. Our study complements a study conducted in parallel by Kapoor et al., 2020 that found modulations by conscious percept in prefrontal cortex using a different no-report paradigm based on optokinetic nystagmus.

The existence of two directly connected functional modules with a hierarchical relationship (ML and AM) that both encode the conscious percept of a particular type of object opens the possibility for future studies to investigate how changes in conscious percept are coordinated across the brain. Recordings and perturbations in multiple face patches simultaneously using high-channel count recordings may reveal whether switches occur in a feedforward or feedback wave, and thus yield insight into how our interpretation of the world can be rendered consistent across different levels of representation.

All animal procedures in this study complied with local and National Institute of Health guidelines including the US National Institutes of Health Guide for Care and Use of Laboratory Animals. All experiments were performed with the approval of the Caltech Institutional Animal Care and Use Committee (IACUC). The behavioral experiment with human subjects for the human psychophysics experiment complied with a protocol approved by the Caltech Institutional Review Board (IRB).

All data generated or analysed during this study are included in the manuscript and supporting files.

1. 1. Aru J
   2. Bachmann T
   3. Singer W
   4. Melloni L

   (2012) Distilling the neural correlates of consciousness

   Neuroscience & Biobehavioral Reviews 36:737–746.

   https://doi.org/10.1016/j.neubiorev.2011.12.003

   • PubMed
   • Google Scholar
2. 1. Bao P
   2. She L
   3. McGill M
   4. Tsao DY

   (2020) A map of object space in primate inferotemporal cortex

   Nature 583:103–108.

   https://doi.org/10.1038/s41586-020-2350-5

   • PubMed
   • Google Scholar
3. 1. Bao P
   2. Tsao DY

   (2018) Representation of multiple objects in macaque category-selective Areas

   Nature Communications 9:1774.

   https://doi.org/10.1038/s41467-018-04126-7

   • PubMed
   • Google Scholar
4. 1. Blake R
   2. Brascamp J
   3. Heeger DJ

   (2014) Can binocular rivalry reveal neural correlates of consciousness?

   Philosophical Transactions of the Royal Society B: Biological Sciences 369:20130211.

   https://doi.org/10.1098/rstb.2013.0211

   • Google Scholar
5. 1. Block N

   (2019) What is wrong with the No-Report paradigm and how to fix it

   Trends in Cognitive Sciences 23:1003–1013.

   https://doi.org/10.1016/j.tics.2019.10.001

   • PubMed
   • Google Scholar
6. 1. Block N

   (2020) Finessing the bored monkey problem

   Trends in Cognitive Sciences 24:167–168.

   https://doi.org/10.1016/j.tics.2019.12.012

   • PubMed
   • Google Scholar
7. 1. Boly M
   2. Massimini M
   3. Tsuchiya N
   4. Postle BR
   5. Koch C
   6. Tononi G

   (2017) Are the neural correlates of consciousness in the front or in the back of the cerebral cortex? clinical and neuroimaging evidence

   The Journal of Neuroscience 37:9603–9613.

   https://doi.org/10.1523/JNEUROSCI.3218-16.2017

   • PubMed
   • Google Scholar
8. 1. Bradley MM
   2. Miccoli L
   3. Escrig MA
   4. Lang PJ

   (2008) The pupil as a measure of emotional arousal and autonomic activation

   Psychophysiology 45:602–607.

   https://doi.org/10.1111/j.1469-8986.2008.00654.x

   • PubMed
   • Google Scholar
9. 1. Brascamp J
   2. Blake R
   3. Knapen T

   (2015) Negligible fronto-parietal BOLD activity accompanying unreportable switches in bistable perception

   Nature Neuroscience 18:1672–1678.

   https://doi.org/10.1038/nn.4130

   • Google Scholar
10. Book

    1. Breitmeyer BG
    2. Hoar WS
    3. Randall D
    4. Conte FP

    (1984)

    Visual Masking: An Integrative Approach

    Clarendon Press.

    • Google Scholar
11. 1. Chang L
    2. Tsao DY

    (2017) The code for facial identity in the primate brain

    Cell 169:1013–1028.

    https://doi.org/10.1016/j.cell.2017.05.011

    • PubMed
    • Google Scholar
12. Conference

    1. Dutta B
    2. Andrei A
    3. Harris T
    4. Lopez C
    5. O’Callahan J
    6. Putzeys J
    7. Raducanu B
    8. Severi S
    9. Stavisky S
    10. Trautmann E

    (2019) The neuropixels probe: a CMOS based integrated microsystems platform for neuroscience and brain-computer interfaces. 2019

    IEEE International Electron Devices Meeting (IEDM).

    https://doi.org/10.1109/IEDM19573.2019.8993611

    • Google Scholar
13. 1. Frässle S
    2. Sommer J
    3. Jansen A
    4. Naber M
    5. Einhauser W

    (2014) Binocular rivalry: frontal activity relates to introspection and action but not to perception

    Journal of Neuroscience 34:1738–1747.

    https://doi.org/10.1523/JNEUROSCI.4403-13.2014

    • Google Scholar
14. 1. Freiwald WA
    2. Tsao DY

    (2010) Functional compartmentalization and viewpoint generalization within the macaque face-processing system

    Science 330:845–851.

    https://doi.org/10.1126/science.1194908

    • PubMed
    • Google Scholar
15. 1. Gelbard-Sagiv H
    2. Mudrik L
    3. Hill MR
    4. Koch C
    5. Fried I

    (2018) Human single neuron activity precedes emergence of conscious perception

    Nature Communications 9:2057.

    https://doi.org/10.1038/s41467-018-03749-0

    • Google Scholar
16. 1. Grimaldi P
    2. Saleem KS
    3. Tsao D

    (2016) Anatomical connections of the functionally defined "Face Patches" in the Macaque Monkey

    Neuron 90:1325–1342.

    https://doi.org/10.1016/j.neuron.2016.05.009

    • PubMed
    • Google Scholar
17. 1. Hesse JK
    2. Tsao DY

    (2020) The macaque face patch system: a turtle’s underbelly for the brain

    Nature Reviews Neuroscience 147:695–716.

    https://doi.org/10.1038/s41583-020-00393-w

    • Google Scholar
18. 1. Hoeks B
    2. Levelt WJM

    (1993) Pupillary dilation as a measure of attention: a quantitative system analysis

    Behavior Research Methods, Instruments, & Computers 25:16–26.

    https://doi.org/10.3758/BF03204445

    • Google Scholar
19. 1. Hohwy J
    2. Roepstorff A
    3. Friston K

    (2008) Predictive coding explains binocular rivalry: an epistemological review

    Cognition 108:687–701.

    https://doi.org/10.1016/j.cognition.2008.05.010

    • PubMed
    • Google Scholar
20. 1. Jun JJ
    2. Steinmetz NA
    3. Siegle JH
    4. Denman DJ
    5. Bauza M
    6. Barbarits B
    7. Lee AK
    8. Anastassiou CA
    9. Andrei A
    10. Aydın Ç
    11. Barbic M
    12. Blanche TJ
    13. Bonin V
    14. Couto J
    15. Dutta B
    16. Gratiy SL
    17. Gutnisky DA
    18. Häusser M
    19. Karsh B
    20. Ledochowitsch P
    21. Lopez CM
    22. Mitelut C
    23. Musa S
    24. Okun M
    25. Pachitariu M
    26. Putzeys J
    27. Rich PD
    28. Rossant C
    29. Sun WL
    30. Svoboda K
    31. Carandini M
    32. Harris KD
    33. Koch C
    34. O'Keefe J
    35. Harris TD

    (2017) Fully integrated silicon probes for high-density recording of neural activity

    Nature 551:232–236.

    https://doi.org/10.1038/nature24636

    • PubMed
    • Google Scholar
21. Preprint

    1. Kapoor V
    2. Dwarakanath A
    3. Safavi S
    4. Werner J
    5. Besserve M
    6. Panagiotaropoulos TI
    7. Logothetis NK

    (2020) Decoding the contents of consciousness from prefrontal ensembles

    bioRxiv.

    https://doi.org/10.1101/2020.01.28.921841

    • Google Scholar
22. 1. Koch C
    2. Massimini M
    3. Boly M
    4. Tononi G

    (2016) Neural correlates of consciousness: progress and problems

    Nature Reviews Neuroscience 17:307–321.

    https://doi.org/10.1038/nrn.2016.22

    • PubMed
    • Google Scholar
23. 1. Leibo JZ
    2. Liao Q
    3. Anselmi F
    4. Freiwald WA
    5. Poggio T

    (2017) View-Tolerant face recognition and hebbian learning imply Mirror-Symmetric neural tuning to head orientation

    Current Biology 27:62–67.

    https://doi.org/10.1016/j.cub.2016.10.015

    • PubMed
    • Google Scholar
24. 1. Leopold DA
    2. Plettenberg HK
    3. Logothetis NK

    (2002) Visual processing in the ketamine-anesthetized monkey

    Experimental Brain Research 143:359–372.

    https://doi.org/10.1007/s00221-001-0998-0

    • Google Scholar
25. 1. Leopold DA
    2. Logothetis NK

    (1996) Activity changes in early visual cortex reflect monkeys' percepts during binocular rivalry

    Nature 379:549–553.

    https://doi.org/10.1038/379549a0

    • Google Scholar
26. 1. Ohayon S
    2. Tsao DY

    (2012) MR-guided stereotactic navigation

    Journal of Neuroscience Methods 204:389–397.

    https://doi.org/10.1016/j.jneumeth.2011.11.031

    • PubMed
    • Google Scholar
27. 1. Overgaard M
    2. Fazekas P

    (2016) Can No-Report paradigms extract true correlates of consciousness?

    Trends in Cognitive Sciences 20:241–242.

    https://doi.org/10.1016/j.tics.2016.01.004

    • PubMed
    • Google Scholar
28. Conference

    1. Pachitariu M
    2. Steinmetz NA
    3. Kadir SN
    4. Carandini M
    5. Harris KD

    (2016)

    Fast and accurate spike sorting of high-channel count probes with KiloSort

    Advances in Neural Information Processing Systems.

    • Google Scholar
29. 1. Panagiotaropoulos TI
    2. Dwarakanath A
    3. Kapoor V

    (2020) Prefrontal Cortex and Consciousness: Beware of the Signals

    Trends in Cognitive Sciences 24:343–344.

    https://doi.org/10.1016/j.tics.2020.02.005

    • Google Scholar
30. 1. Preuschoff K
    2. 't Hart BM
    3. Einhäuser W

    (2011) Pupil dilation signals surprise: evidence for noradrenaline's Role in Decision Making

    Frontiers in Neuroscience 5:115.

    https://doi.org/10.3389/fnins.2011.00115

    • PubMed
    • Google Scholar
31. 1. Rao RP
    2. Ballard DH

    (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects

    Nature Neuroscience 2:79–87.

    https://doi.org/10.1038/4580

    • PubMed
    • Google Scholar
32. 1. Safavi S
    2. Kapoor V
    3. Logothetis NK
    4. Panagiotaropoulos TI

    (2014) Is the frontal lobe involved in conscious perception?

    Frontiers in Psychology 5:1063.

    https://doi.org/10.3389/fpsyg.2014.01063

    • PubMed
    • Google Scholar
33. 1. Sheinberg DL
    2. Logothetis NK

    (1997) The role of temporal cortical Areas in perceptual organization

    PNAS 94:3408–3413.

    https://doi.org/10.1073/pnas.94.7.3408

    • PubMed
    • Google Scholar
34. 1. Tong F
    2. Nakayama K
    3. Vaughan JT
    4. Kanwisher N

    (1998) Binocular rivalry and visual awareness in human extrastriate cortex

    Neuron 21:753–759.

    https://doi.org/10.1016/S0896-6273(00)80592-9

    • PubMed
    • Google Scholar
35. 1. Tong F
    2. Meng M
    3. Blake R

    (2006) Neural bases of binocular rivalry

    Trends in Cognitive Sciences 10:502–511.

    https://doi.org/10.1016/j.tics.2006.09.003

    • Google Scholar
36. 1. Tong F
    2. Engel SA

    (2001) Interocular rivalry revealed in the human cortical blind-spot representation

    Nature 411:195–199.

    https://doi.org/10.1038/35075583

    • PubMed
    • Google Scholar
37. 1. Trautmann EM
    2. Stavisky SD
    3. Lahiri S
    4. Ames KC
    5. Kaufman MT
    6. O'Shea DJ
    7. Vyas S
    8. Sun X
    9. Ryu SI
    10. Ganguli S
    11. Shenoy KV

    (2019) Accurate estimation of neural population dynamics without spike sorting

    Neuron 103:292–308.

    https://doi.org/10.1016/j.neuron.2019.05.003

    • PubMed
    • Google Scholar
38. 1. Tsao DY
    2. Freiwald WA
    3. Tootell RB
    4. Livingstone MS

    (2006) A cortical region consisting entirely of face-selective cells

    Science 311:670–674.

    https://doi.org/10.1126/science.1119983

    • PubMed
    • Google Scholar
39. 1. Tsuchiya N
    2. Wilke M
    3. Frässle S
    4. Lamme VAF

    (2015) No-Report paradigms: extracting the true neural correlates of consciousness

    Trends in Cognitive Sciences 19:757–770.

    https://doi.org/10.1016/j.tics.2015.10.002

    • PubMed
    • Google Scholar
40. 1. Tsuchiya N
    2. Frässle S
    3. Wilke M
    4. Lamme V

    (2016) No-report and report-based paradigms jointly unravel the NCC: response to Overgaard and fazekas

    Trends in Cognitive Sciences 20:242–243.

    https://doi.org/10.1016/j.tics.2016.01.006

    • PubMed
    • Google Scholar
41. 1. Tsuchiya N
    2. Koch C

    (2005) Continuous flash suppression reduces negative afterimages

    Nature Neuroscience 8:1096–1101.

    https://doi.org/10.1038/nn1500

    • PubMed
    • Google Scholar
42. 1. Wilke M
    2. Logothetis NK
    3. Leopold DA

    (2003) Generalized flash suppression of salient visual targets

    Neuron 39:1043–1052.

    https://doi.org/10.1016/j.neuron.2003.08.003

    • PubMed
    • Google Scholar
43. 1. Wolfe JM

    (1984) Reversing ocular dominance and suppression in a single flash

    Vision Research 24:471–478.

    https://doi.org/10.1016/0042-6989(84)90044-0

    • PubMed
    • Google Scholar
44. 1. Xu H
    2. Han C
    3. Chen M
    4. Li P
    5. Zhu S
    6. Fang Y
    7. Hu J
    8. Ma H
    9. Lu HD

    (2016) Rivalry-Like neural activity in primary visual cortex in anesthetized monkeys

    Journal of Neuroscience 36:3231–3242.

    https://doi.org/10.1523/JNEUROSCI.3660-15.2016

    • PubMed
    • Google Scholar
45. 1. Zou J
    2. He S
    3. Zhang P

    (2016) Binocular rivalry from invisible patterns

    PNAS 113:8408–8413.

    https://doi.org/10.1073/pnas.1604816113

    • PubMed
    • Google Scholar

Author details

1. Janis Karan Hesse

   1. Division of Biology and Biological Engineering, Computation and Neural Systems, Pasadena, United States
   2. Howard Hughes Medical Institute, Pasadena, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   jhesse@caltech.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-0405-8632

2. Doris Y Tsao

   1. Division of Biology and Biological Engineering, Computation and Neural Systems, Pasadena, United States
   2. Howard Hughes Medical Institute, Pasadena, United States

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Methodology, Writing - original draft, Project administration, Writing - review and editing

   For correspondence

   dortsao@caltech.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-1083-1919

• 4,345

  views

• 637

  downloads

• 37

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.