Introduction

A conscious state means that a person has some kind of subjective experiences, such as seeing a painting, hearing a sound, thinking about a problem, or feeling an emotion ¹. The neural mechanism of consciousness is one of the most fundamental, exciting and challenging pursuits in the 21st-century brain science ². Among the various aspects of consciousness, visual awareness, as a core of consciousness, has attracted enormous attention in consciousness research. Visual awareness seems quite simple: we see an object and seem to immediately know its shape, color and other properties. However, the neural processes behind the emergence of visual awareness are quite sophisticated and largely unknown ³.

The typical method to study the neural mechanism of visual awareness is to present external stimuli with similar physical properties but to elicit different conscious experiences, such as using near perceptual threshold stimuli, visual masking, or attentional blinks ^4,5. Then, the aware and unaware states are determined through the subjective reports of participants, and the awareness-related neural activity is dissociated by the brain activation difference between the two states. Using this approach, some studies have found that activities in broad brain regions are correlated with visual awareness. For example, an fMRI study of lexical visual masking found that only consciously perceived and reported words can induce activation of higher-order cortical regions such as the frontal and parietal lobes ⁶. Another study using a visual masking task found changes in the amplitude of intracranial electroencephalography (iEEG) signals and high-gamma power in occipital-temporal regions, which indicated that these regions played an important role in the emergence of awareness ⁷.

However, there is still much debate about the origin of conscious awareness in the brain. Some hypotheses (global neuronal workspace theory, higher-order theory, etc.) argued that visual awareness arose from the frontoparietal network, in which the prefrontal cortex played a key role, and they argued that the earliest biomarker of awareness in EEG is the P3b wave onset approximately 300 ms after stimulus onset ^8,9. For example, based on the electrophysiological study of a visual awareness-guided saccade selection paradigm in macaques, by comparing the action potential firing characteristics of neurons in the dorsolateral prefrontal cortex (dlPFC), V4 and V1, only dlPFC neurons were closely associated with visual awareness states. Therefore, the authors argued that stimuli must elicit a threshold level of prefrontal activity for conscious detection to occur ¹⁰. In contrast, other hypotheses (integrated information theory, recurrent processing theory, etc.) proposed that the “hot zone” of the posterior cortex of the brain, including the parietal lobe, occipital lobe, and temporal lobe, is sufficient to generate visual awareness, and its biomarkers on EEG are the relatively earlier visual awareness negativity (VAN) at approximately 200 ms after stimulus onset ^1,11–13, whereas the PFC is not necessary for the emergence of conscious experience.

This controversy is partially caused by the confounding effect of report-related activity, such as motor planning and execution, on awareness-related activity. The reason is that the study of awareness usually requires subjective reports to determine different awareness states, while the prefrontal cortex is closely related to such report-related functions ¹⁴. To solve this problem, some researchers use the no-report paradigm to rule out report-related impacts. For instance, a functional magnetic resonance imaging (fMRI) study employing human binocular rivalry paradigms found that when subjects need to manually report the changing of their awareness between conflict visual stimuli, the frontal, parietal, and occipital lobes all exhibited awareness-related activity. However, when report was not required, awareness-related activation was largely diminished in the frontal lobe but remained in the occipital and parietal lobes¹⁵. Nevertheless, the no-report paradigm may overestimate the neural correlates of awareness by including unconscious processing, because it infers the awareness state through other relevant physiological indicators, such as optokinetic nystagmus and pupil size¹⁶. In the absence of subjective reports, it remains controversial regarding whether the presented stimuli are truly seen or not. Therefore, neither the typical report paradigm nor the no-report paradigm is ideal for visual awareness research, thus, it is difficult to solve the problem of whether the PFC participates in conscious perception.

In addition, when studying the role of the PFC in visual awareness, the noninvasive technologies mostly used in previous studies, i.e., Functional magnetic resonance imaging (fMRI) and M/EEG have limitations. Although fMRI research can detect the whole brain activation pattern, it is difficult to describe the dynamic changes and sequences of different brain regions in the rapid process of visual awareness due to its limited time resolution (approximately several seconds). For example, even though a recent fMRI study using the no-reported paradigm found that there are still visual awareness-related activities in the PFC ¹⁷, it remains unclear when these activities begin and whether PFC activity occurs earlier or later than other brain regions. At the same time, although EEG and MEG have a very high temporal resolution, their low spatial resolution and low signal-to-noise ratio limit their interpretation of awareness-related activities, especially high-frequency activities (highly correlated with local population neuronal activities). For example, although increasing evidence shows that the earliest biomarker of visual awareness in EEG is more likely to be the VAN, which starts at approximately 180-300 ms (may be delayed to 300-460 ms for low-visible stimulus) and peaks at the parieto-occipital region ^18–20, rather than the P3b wave onset at approximately 300-500 ms which peaks in the central frontoparietal region, it is still unclear whether there are local visual awareness-related activities in the PFC during the VAN period. In addition, a few recent studies have suggested that awareness may be related to the activities of brain networks, including the PFC, rather than that of a single brain region. However, most of these brain network studies used noninvasive research methods ^17,21, and due to the limitations mentioned above, it is difficult to detect the dynamic changes in brain networks in a more precise spatial-temporal profile during the generation of visual awareness.

To address the questions mentioned above, we designed a novel visual awareness task (Figure 1) that can minimize motor-related confounding, while retaining the explicit subjective report. The contrast of the cue stimulus (grating) was roughly divided into three levels: well above the visual perceptual threshold, near the visual perceptual threshold and zero contrast (no stimulus). During each experiment, in nearly 80% of trials, the contrast of the grating was close to the subject’s visual perceptual threshold so that the subjects could sometimes see the grating and sometimes could not. Subjects were required to choose a saccade direction according to the visual awareness state (seeing the grating or not seeing the grating) and the color of the fixation point. Notably, subjects were unable to choose the saccade direction or effectively prepare for saccades until the color of the fixation point was changed. Moreover, the subjects needed to choose the saccade direction, both in the awareness and unawareness states, according to the color of the fixation point so that the report behaviors were matched between the two awareness states. Such a paradigm can effectively dissociate awareness-related activity from motor-related activity in terms of time (the fixation point changes color 650 ms after the grating onset) and report behavior, thereby being report-independent of the awareness states ²², thus minimizing the impact of motor-related factors on visual awareness. In addition, saccade parameters (such as reaction time, etc.) under different awareness states can be compared to explore the influence of visual awareness on saccade behavior. Then, we applied the above behavioral paradigm to 6 clinical patients with implanted electrodes (stereoelectroencephalography, sEEG) in prefrontal and other cortices and 10 healthy subjects. The behavior and local field potential (LFP) data of 6 patients were recorded while they performed the task, and we also collected the behavioral data of 10 healthy subjects for comparison with the patients’ behavioral data.

Figure 1.

Results

Behavioral Results

To detect the effect of cue stimulus (grating) contrast on the emergence of visual awareness, we first analyzed the proportion of participants being aware of the grating as a function of its contrast. Figure 2A shows exemplified sessions from a patient and healthy subject, and Figure 2B shows the population results of the two participant groups. The results showed that with increasing grating contrast, the ratio of reporting ‘awareness’ of the grating gradually increased for all participants (including patients and healthy subjects), showing a classical psychometric curve. Such results showed that the grating contrast level had a direct impact on the emergence of visual awareness and thus is a reliable way to study visual awareness. In addition, in trials with no grating and grating contrast well above the perceptual threshold, subjects showed high accuracy (patients 94.75% ± 2.65; healthy subjects, 96.80% ± 0.71, mean ± SEM) and sensitivity (d’=1.81±0.27 for patients and 2.12±0.37 for healthy subjects), indicating that the subjects understood and performed the task well following the task rules.

Figure 2.

To explore whether visual awareness affects behavioral performance, we divided the trials into 4 groups according to the level of grating contrast and the participants’ reported awareness state, i.e., high-contrast aware (HA), near-threshold-contrast aware (NA), near-threshold-contrast unaware (NU), and low-contrast unaware (LU) (Figure 2A, also see Methods). We compared the saccadic parameters between the awareness and unawareness states in near-threshold contrast levels (NA vs. NU), as well as in high and low contrast levels (HA vs. LU), as a control. While saccadic reaction time showed a significant difference between the two awareness states (Figure 2B), the other parameters did not. Figure 2C shows the mean saccadic reaction time of patients and healthy subjects under different awareness states and contrast levels. The results showed that the average saccadic reaction time was significantly shorter under the awareness state than under the unawareness state at both the near-threshold contrast level and the high-low contrast level (p < 0.05, Wilcoxon signed rank test), and it was also explicit and consistent at the individual level. The results showed that the saccadic reaction time in the aware trials was systematically shorter than that in the unaware trials. Such results demonstrate that visual awareness significantly affects the speed of information processing in the brain.

Local Field Potential Results

While the patients were performing our visual awareness task, we recorded the local field potential (LFP) from 901 recording sites (Figure 2D, projected on Montreal Neurological Institute (MNI) brain template ICBM152), in which 245 sites were located in the PFC. The table in Figure 2D shows a summary of the number of recording sites and patients in different lobes and structures.

Early visual awareness-related iERP activity in the PFC

While looking through the event-related response of the LFP data, a striking phenomenon in all patients was that the amplitude of LFP showed remarkably different patterns between awareness and unawareness states. We show the grand average (left) and single-trial data (right) of LFP activity of recording sites in the PFC from each patient in Figure 3A, which represents the typical awareness-related neural activity of the PFC. There was a vigorous visual evoked response under the HA condition (light red line) compared to the LU (light blue line) condition. However, in the near-threshold condition, although the contrast of the grating was similar (see Results below), the LFP amplitude of the NA (red line) and NU (blue line) trials started to differ significantly at approximately 200-300 ms (thick black line in panels, p < 0.01 corrected, by independent t-test, details see Methods). This difference is also very explicit and consistent at the single-trial (>180 trials for NA and NU conditions in individual participants, see Methods) and single-subject level. The divergence onset time (DOT, see Methods) of different recording sites in the PFC is consistently within the period of 200∼300 ms after grating onset, i.e., the time window of VAN in scalp EEG mentioned above.

Figure 3.

As proposed above, the different activities between NA and NU are regarded as awareness-related activities. We found 89 sites with awareness-related iERP activity in the PFC (323 for all recording sites). Figure 3B shows the spatial-temporal dynamics of visual awareness-related activities in all recording sites during the 180-650 ms period (sampled at 8 time points that equally divided this period, for a clearer display, projected to the cortex, see the Supplementary Materials for the complete video). Visual awareness-related activities clearly appeared in the PFC during 200-300 ms, and these early activities were mainly concentrated in the middle lateral prefrontal cortex (LPFC) and then spread to other brain areas of the PFC. We further calculated the start time of awareness-related activity, i.e., DOT, of all recording sites. Figure 3C shows the spatial distribution of all recording sites DOT (standardized and convenient for visual display), and it can be found that the DOT of the PFC is not explicitly later than those in posterior brain regions, including the posterior parietal lobe, temporal lobe, occipital lobe, etc., and the earliest region in PFC is still located in the middle lateral prefrontal cortex.

Considering the possibility that the brain region with earlier awareness-related activity might be more important for visual awareness, we tried to detect awareness-related activity in the PFC during the ‘early’ period, i.e., the VAN time window mentioned above. We focused on the sites in PFC with earlier DOTs. Because of the low-visible stimulus used in this experiment, the onset time of VAN may be delayed to 300-350 ms, and we took 350 ms as the cutoff value and divided DOT into early (n = 43, 48.31%) and late (n = 46, 51.69%) phases. We focused on the early DOT sites and calculated the normalized average iERP response of the early DOT sites (Figure 3D). At the population level, visual awareness-related activities in the PFC were also significant and started at approximately 200-300 ms, which is located in the VAN time window (Figure 3D, left). Importantly, there were sites with DOT < 350 ms from each patient and these sites from different patients were uniformly concentrated in the middle lateral prefrontal cortex (Figure 3D, right). That is, the early visual awareness-related activities in the prefrontal cortex are robust and consistent not only at the group level but also at the individual level.

Early visual awareness-related high-gamma activity in the PFC

Although the above iERP results indicate that there are early visual awareness-related neural activities in the prefrontal cortex, it is difficult to explain whether this response is the result of local processing in the PFC or transmission from other brain regions. Therefore, we calculated the LFP response in the high-gamma (HG) band, which is more representative of the local processing, of the prefrontal cortex. Figure 4A shows the exemplified high-gamma activity in the prefrontal cortex from each patient at the grand average (left), spectrogram (middle), and single-trial level (right). It is similar to the iERP results (Figure 3) except the data are the HG power and there was a vigorous visual evoked power increase under the HA condition compared to the LU condition. However, in the near-threshold condition, while the contrast of the grating was similar, the high-gamma power of the NA and NU trials started to differ significantly after ∼ 230 ms (thick black line in panels, p < 0.01 corrected by independent t-test; for details, see Methods). This difference is also very explicit and consistent at the single-trial and single-subject levels. The HG activity divergence onset time (DOT) of different recording sites in the PFC was also consistent within the early stage, which was similar to the above iERP result.

Figure 4.

We found 31 sites with awareness-related HG activity in the PFC (74 for all recording sites). Figure 4B shows the spatial-temporal dynamics of visual awareness-related HG activities in all recording sites during the 0-650 ms period (for a clearer display, projected to the cortex, see the Supplementary Materials for the complete video). Visual awareness-related HG activities also clearly appeared in the PFC during 200-300 ms, while they were rarely seen in lower-level visual areas (occipital and temporal lobe), and these early activities were mainly concentrated in the middle lateral prefrontal cortex (LPFC) and then spread to other brain areas of the PFC. We further calculated the start time of awareness-related HG activity, i.e., DOT, of all recording sites. Figure 4C shows the spatial distribution of all recording sites DOT (standardized and convenient for visual display), and it can be found that the DOT of the PFC is not explicitly later than those in posterior brain regions, and the earliest region is still located in the middle lateral prefrontal cortex.

We continue to take 350 ms as the cutoff value and divide the HG DOT into early (n=18, 58.06%) and late (13, 41.94%). We focused on the early DOT sites and calculated the normalized mean response (Figure 4D) and spectrum of the early DOT sites. At the population level, early visual awareness-related HG activities of the prefrontal sites still exist, and the start time is approximately 200 ms, which is within the VAN time window. Importantly, there are sites with HG DOT < 350 ms for each patient except one, and these sites are also uniformly concentrated in the middle lateral prefrontal cortex. In other words, this effect is significant and consistent at both the group and individual levels. It is worth noting that in the PFC recording sites, the early sites with a high-gamma visual response (7.35%) were explicitly less than those with ERP visual response (17.55%), and this is also evident in the temporal dynamics (Figure 4E), which may indicate that in the process of visual awareness generation, there may also be a variety of other bands activity for information interaction between multiple brain regions.

Decoding awareness state through ERP and HG activity in the PFC

In addition to the univariate analysis above, we also adopted the machine learning decoding method based on multivariate pattern analysis (MVPA) to test the ability of broadband and high-gamma activities in the PFC to predict the awareness state at the single-trial level. We applied a linear discriminant analysis (LDA) to classify aware versus unaware trials based on broadband and high-gamma LFP activities in the PFC.

Figure 5A shows the decoding performance through the ERP activity in the PFC. We found that the ERP activity of the PFC can accurately predict the state of visual awareness at the single-trial level in the delayed period after grating onset, and this result is also relatively consistent at the individual level. Moreover, this decoding accuracy has begun to be much higher than the chance level in 200-300 ms, consistent with our above univariate analysis results. Furthermore, we applied the temporal generalization method to test the predictive ability across time. The results (Figure 5B) show that the decoding performance of population ERP is very limited in temporal generalization and cannot be well generalized over a long period. Importantly, this phenomenon is also very consistent at the individual level, which indicates that the representation of visual awareness via broadband activities in the PFC is relatively dynamic rather than static. Moreover, we also applied temporal generalization across the control conditions, i.e., train on NA/NU trials but test on HA/LU trials to test the predictive ability across conditions. The classifier trained on the NA/NU trials can be well generalized to the HA/LU trials, and the generalization across time is also very consistent. These results further indicated the robustness of the above decoding results.

Figure 5.

Figure 5C-D shows the results of a similar analysis of HG activity in the prefrontal cortex. Figure 5C shows the decoding performance through HG activity in the PFC. We found that the HG activity of the PFC can accurately predict the state of visual awareness at the single-trial level in the delayed period, and this result is also relatively consistent at the individual level. Moreover, this decoding accuracy began to be significantly higher than chance at approximately 200-300 ms, which is also consistent with our above univariate analysis results. Furthermore, we applied the temporal generalization method to the HG activity. Although the group results still show that the decoding performance of HG activity is limited in time generalization, the generalization results are not consistent at the individual level, which may be attributed to the variability of electrode location in different patients. This may suggest that the representation of visual awareness-related information by population neurons at different regions in the PFC is inconsistent, i.e., some are relatively stable, and others are dynamic. Similarly, we applied temporal generalization across the HA/LU conditions, and the result was also very consistent.

Dynamic functional connectivity changes in the PFC associated with visual awareness

We further explored the dynamic functional connectivity between the prefrontal cortex and other brain regions in the generation of visual awareness. First, we calculated the time-across phase-locking value (PLV) ²³ in NA and NU conditions at the sensor level (baseline removed, see Methods for details). Figure 6A-B shows the example results of two exemplified patients (see SM for other patients). We found that in the low-frequency band (1-8 Hz), overall, there was a significant difference in the PLV between the NA and NU trials. In the first 150 ms after the appearance of the grating, there was no significant difference in the PLV between these two conditions. However, in the period of approximately 200-300 ms, the PLV between many recoding sites in the NA trials began to increase explicitly and lasted until after 600 ms. In contrast, there was no such phenomenon in the NU trials. Importantly, this phenomenon is strikingly explicit and consistent in each patient.

Figure 6.

Furthermore, we grouped the recording sites by brain regions according to the DKT brain atlas ²⁴ and averaged the PLV between different sites and different subjects according to brain regions (Figure 6C). The functional connectivity between the prefrontal cortex and subcortical structures, such as the thalamus, and the posterior cortex, such as the posterior parietal cortex, was significantly enhanced in the NA trials compared with the PLV in NU trials. In addition, the functional connectivity between the brain regions within the PFC was also significantly enhanced. For clearer visualization, we show the dynamics of PLV in the NA and NU trials on a topographic map (Figure 6D).

Interestingly, we found that at the single-subject level, the sites in the prefrontal cortex that showed early awareness-related PLV activity coincided with the above sites with early awareness-related HG activity. Figure 6E shows the averaged PLV between these sites (n=18) with early visual awareness-related HG activity in the PFC and other sites in the NA and NU trials. The averaged PLV also showed significant visual awareness-related activities. Moreover, at the population level, the start time of awareness-related PLV activity at these sites also started at approximately 200 ms, which is very similar to that of HG activity.

The different neural activity between NA and NU trials is not caused by the grating contrast

Since there may be a slight difference in the grating contrast under the NA and NU trials, the aforementioned different activity between the NA and NU conditions may be caused by the different contrast of grating. To rule out this possibility, we compared the distribution of grating contrast in the NA and NU conditions for each patient (Figure 7A). Overall, we find that the distributions of grating contrasts largely overlap with each other (red and blue curves). The Kolmogorov‒Smirnov test resulted in no significant difference (p>0.05) between the two contrast distributions in all patients except patient 2. Although intuitively the trials with higher contrast contributed more to the NA condition and the trials with lower contrast contributed more to the NU condition, the contrast difference between the NA and NU conditions were not significantly different perhaps because the grating contrast was designed to be near the perceptual threshold (the ‘aware’ percent is 50%) in most trials. Moreover, for patient 2, the difference between the mean grating contrast in the NA and NU trials was also subtle (0.15%), much less than the difference between NA and HA (17.35%) or NU and LU (0.84%). However, it is clear in the results above that the different activities between NA and NU were much stronger than those between the same awareness states, i.e., NA vs. HA or NU vs. LU. Therefore, the difference in neural activity between aware and unaware trials cannot be explained by the contrast differences in the grating.

Figure 7.

Discussion

We employed a novel visual awareness task and found that, first, the saccadic reaction time of all subjects was shorter when they were aware of grating than when they were not aware. Second, there are ERP and HG activities related to visual awareness in the prefrontal LFP, and these activities begin at the early stage (200-300 ms). Then, we found that the awareness state can be reliably decoded from the broadband and HG activities of the PFC at approximately 200-300 ms after stimulus onset. Finally, the functional connectivity between the PFC and other brain regions in the low-frequency band (1-8 Hz) also appeared awareness-related changes, and the initial time and location of this change are consistent with the above HG results. The results above are not caused by the difference in the physical properties of external stimuli. Therefore, we propose that the prefrontal cortex also plays a key role in the early stage of visual awareness generation even if motor effects are excluded.

Advantages and limitations of the behavioral paradigm used in the present study

Compared with the regular report-dependent visual awareness task, we employed a task that effectively dissociated the activity of visual awareness from the activity of saccadic decision and execution in terms of time (the fixation point changes color 650 ms after the grating onset) and reporting behavior. Since the choice of reporting behavior depends on the visual awareness state and the color of the fixation point, the subjects were unable to effectively prepare the subsequent saccadic reports during the delay period of the task; thus, the visual awareness-related activity evoked by the grating stimulus and report-preparation-related activity were temporally dissociated. Furthermore, both in the awareness and unawareness states, the subjects needed to choose the saccade direction after the fixation point changes its color so that the saccade directions in the two visual awareness states were fully matched to exclude the effects of saccade preparation- and execution-related activity on visual awareness-related activity. Compared with the no-report paradigms, our novel paradigm required subjects to make explicit reports of their awareness states so that it can be more accurate regarding the judgment of the awareness status.

Moreover, the intervals for the initial fixation period and delay period were both fixed at 600 ms, instead of randomly varied, in order to keep subjects’ anticipation being similar in all trials. Also, the location of grating remained same in all trials for the same purpose.

It was also worth noting that our task does not remove the confound of report entirely, i.e. the post-perceptual confound might relate to planning to report perception, which is different for perceived and not perceived stimuli. Nevertheless, our task did minimize the motor-related confound to awareness-related activity.

Visual awareness states affect saccadic reaction time

Many factors affect saccadic reaction time, including the characteristics of external stimuli and the internal state of the brain. The physical properties of external visual stimuli have a direct impact on the performance of saccades; for example, saccadic reaction time decreases with increasing visual stimulus contrast ^25,26. A widely accepted view is that the greater contrast of visual stimuli will automatically attract more bottom-up attention and speed up information processing in the brain, thereby shortening the saccadic reaction time ^27,28. Other studies found that endogenous factors, such as top-down attention, can also significantly shorten the saccadic reaction time ^28,29.

One study briefly reported that the average reaction time (button press) for word semantic classification in the awareness state was approximately 300 ms faster than in the unawareness state, but the statistical test did not reach a significant level ³⁰. In present study, we found that the saccadic reaction time of all subjects in the aware trials was significantly smaller than that in the unaware trials (Figure 2C). More critically, the grating contrast near the threshold was not significantly different between the aware and unaware trials (Figure 7A); thus, such results cannot be merely caused by the contrast difference in the grating. Here, we showed that awareness states significantly affect saccadic reaction time, which supports our previous hypothesis.

An alternative interpretation for RT difference between aware and unaware condition in our study is that the difference in task-strategies used by subjects/patients to remember the response mapping rules between the perception and the color cue (e.g., if the YES+GREEN=RIGHT and YES+RED=LEFT rules were held in memory, while the NO mappings were inferred secondarily rather than being actively held in memory).

Another possibility is that the reaction time is strongly modulated by the confident level, which has been described in previous studies^31,32. However, in previous studies, the confident levels were usually induced by presenting stimulus with different physical property, such as spatial frequency, eccentricity and contrast. However, the dependence of visual process on the salience of visual stimulus confounds with the effect of visual awareness on the reaction time of responsive movements, which is hard to attribute the shorter reaction time in more salient condition purely to visual awareness. In contrast, we create a condition (near aware threshold) in the present study, in which the saliency (contrast) of visual stimulus is very similar in both aware and unaware conditions in order to eliminate the influence of stimulus saliency in reaction time. We think that the difference in reaction time in our study is mainly due to the modulation of awareness state, which was not reported previously.

Visual awareness-related activities appeared in the PFC at an early stage

The long debate about the neural mechanism of consciousness focuses on the location, i.e., ‘front’ vs. ‘back’, and the time, i.e., ‘early’ vs. ‘late’, of its origin in the brain ¹⁴. Concerning the dispute of location, our results show that the prefrontal cortex still displays visual awareness-related activities even after minimizing the influence of the motor-related confounding variables related to subjective reports such as motion preparation, which indicates that the prefrontal cortex does participate in the information processing of visual awareness. Regarding the time dispute, our results show that at 200-300 ms after stimulus onset, the VAN time window, the prefrontal cortex has activities related to visual awareness, and these activities include not only ERP activities but also HG activities, which are highly related to the firing of local population neurons⁷. Notably, the recording sites with early ERP and HG visual awareness-related activities are mostly concentrated in the middle lateral prefrontal cortex. This shows that the prefrontal cortex, especially the middle lateral prefrontal cortex, started visual awareness-related activities, including local activities, at a relatively early stage.

Several iEEG studies have shown the early prefrontal cortical involvement in visual perception, including ERP and HFA^33–35. However, in these studies, the differential activity between conscious and unconscious conditions was not investigated, thus, the activity in prefrontal cortex might be correlated with unconscious processing, rather than conscious processing. In present study, we compared the neural activity in PFC between conscious and unconscious trials, and found the correlation between PFC activity and conscious perception.

Although one iEEG study³⁰ reported awareness-specific PFC activation, the awareness-related activity started 300 ms after the onset of visual stimuli, which was ∼100 ms later than the early awareness related activity in our study. Also, due to the limited number of electrodes in the previous study (2 patients with 19 recording sites mostly in mesiofrontal and peri-insular regions), it was restricted while exploring the awareness-related activity in PFC. In the present study, the number of recording sites (245) were much more than previous study and covered multiple areas in PFC. Our results further show earlier awareness-related activity (∼200 ms after visual stimuli onset), including ERP, HFA and PLV. These awareness-related activity in PFC occurred even earlier (∼150 ms after stimulus onset) for the salient stimulus trials (Fig. 3A\D and Fig. 4A\D, HA condition)’

However, the proportions are much smaller than that reported by Gaillard et al, which peaked at ∼60%. We think that one possibility for the difference may be due to the more sampled PFC subregions in present study and the uneven distribution of awareness-related activity in PFC. Meanwhile, we noticed that the peri-insula regions and middle frontal gyrus (MFG), which were similar with the regions reported by Gaillard et al, seemed to show more fraction of awareness-related sites than other subregions during the delay period (0 - 650 ms after stimulus onset). To test such possibility and make comparison with the study of Gaillard et al. we calculated the proportion of awareness-related site in peri-insula and MFG regions. We found although the proportion of awareness-related site was larger in peri-insula and MFG than in other subregions, it was much lower than the report of Gaillard et al. One alternative possibility for the difference between these two studies might be due to the more complex task in Gaillard et al. Nevertheless, we think these new results would contribute to our understanding of the neural mechanism underlying conscious perception, especially for the role of PFC.

Some recent studies utilizing the no-report paradigm supported our results. For example, a recent electrophysiological study of rhesus monkeys found that even if monkeys do not need to report their conscious content under the binocular competition paradigm, there were also visual awareness-related spiking activities in the prefrontal cortex ³⁶. In addition, some previous human intracranial studies also reported ERP, HG, and single-neuron activities related to visual perception in the higher visual areas of the temporal lobe, such as the superior temporal gyrus ^7,37. However, due to the limitations of the experimental paradigm, these studies cannot completely exclude the impact of subjective reports. Moreover, most of the electrodes in previous studies were located in the temporal lobe, and electrodes in the prefrontal cortex are rare. As our data show that the proportion of sites with earlier ERP and HG responses in the prefrontal cortex is low and relatively concentrated, it is not hard to explain that earlier visual awareness-related activities in the prefrontal cortex were not found in previous iEEG studies, especially in the HG band. Our results, on the one hand, support that the prefrontal cortex may play an important role in the emergence of visual awareness and thus support the ‘front of the brain’ theories of consciousness in the origin brain regions. However, our data also support the prediction of the early emergence of consciousness proposed by the ‘back of the brain’ theories. This means that our result also challenges some predictions of these mainstream theories.

Awareness state can be effectively decoded from PFC activities from an early stage

Compared with traditional univariate analysis, MVPA-based machine learning decoding may be more sensitive to the information contained in brain activities ³⁸, so it has gradually become a powerful tool for the study of neural correlates of consciousness (NCCs). Therefore, we used linear discriminant analysis (LDA) to decode the awareness states at the single-trial level through broadband and HG activities in the prefrontal cortex. Our decoding results show that whether through broadband or HG activities, the decoding performance began well above the chance level at approximately 200-300 ms. This shows that the prefrontal cortex has begun to encode visual awareness-related information in 200-300 ms, which is consistent with the above univariate analysis results. In addition, a recent electrophysiological study of rhesus monkeys using the no-report paradigm also found that even without the requirement of explicit reporting, awareness-related neurons can still be found in the lateral prefrontal cortex of rhesus monkeys, and conscious content can be accurately decoded by the MVPA method ³⁶. Another human fMRI study also found similar results ¹⁷. These results supported our experimental results.

In addition, we also applied the temporal generalization method during decoding. Temporal generalization analysis shows how different brain activity patterns change and transform over time³⁹. Our results show that the representation pattern of neural activity in the prefrontal cortex for visual awareness-related information is dynamic, not static, and this result is very consistent at the individual level and across the control conditions. Such a result may be consistent with the nature of conscious perception. The process of conscious perception (visual awareness) includes a series of processes from phenomenal consciousness (‘qulia’) to conscious access. Therefore, at the level of large, high-order brain regions, the representation of this complex process in the brain should be continuously transmitted and transformed, that is, dynamically changed.

Evaluate the role of the prefrontal cortex in visual awareness at the brain network level

There is growing evidence showing that conscious experience may be represented at the brain network level ^17,21. Thanks to the simultaneously recorded LFP data in multiple brain regions with a high signal-to-noise ratio and high spatial-temporal resolution, we can analyze the dynamic functional connectivity changes under the aware and unaware conditions at the millisecond scale. Since the signals in the low-frequency band are more involved in the information interaction between different brain regions and the activities in the high-frequency band are more representative of the local neural activities, it is easy to understand the functional connectivity changes in the low-frequency band between relatively distant brain regions. Some previous studies have found that the phase synchronization of neural activities in beta and gamma bands is related to visual awareness, but because most of these studies are based on fMRI, it is difficult to describe the dynamic changes in such brain networks on a more precise time scale. The only human iEEG study ³⁰ reported that the phase synchronization of the beta band in the aware condition also occurred relatively late (>300 ms) and mainly confined to posterior zones but not PFC. Moreover, most of these studies did not rule out the influence of motor-related processing, and these processes are often associated with changes in the functional connectivity of multiple brain regions. To the best of our knowledge, we are the first to report dynamic changes in functional connectivity on the millisecond time scale in the low-frequency band (1-8 Hz) associated with visual awareness, and this change is not caused by motor-related confounding factors, such as motion preparation. Moreover, the early sites of PLV and HG overlapped each other to a large extent. Such converging results may further suggest that the middle lateral prefrontal cortex, the brain regions where these sites are located, plays a crucial role in the emergence of visual awareness.

Our results may contribute to some theoretical concepts related to consciousness

Compared with the unaware trials, the high-frequency activity in the PFC and the functional connectivity between many brain regions in the aware trials increased. These results may confirm some theoretical concepts, such as ‘ignition’ and conscious access. Global neuronal workspace theory (GNWT) emphasizes ignition and conscious access ^5,8. Ignition means nonlinear brain activation to small stimulus changes (such as perceptual threshold stimulation). For the near-threshold stimulus used in the current study, while the sensory information in the brain passes through a certain threshold in a specific brain area, such as the lateral prefrontal cortex, the population of neurons discharges (HG activity of local field potential), that is, “ignition”. This is consistent with many previous human intracranial studies ^7,37. However, as discussed above, in contrast with previous studies, our study detected earlier awareness-specific ‘ignition’ in the human PFC, while minimizing the motor-related confounding.

Conscious access means that conscious contents can be shared by many other local information processors in the brain ^5,8. For example, when you see the grating, you can either report it orally or draw it down. Such a function may require the participation and coordination of many related brain regions or networks, but there is no clear biomarker for how these brain regions coordinate. The explicit increase in the PLV, specifically appearing in the aware trials, in the low-frequency band found in the current study may provide potential biological meaning for conscious access; that is, many brain regions interact with each other through phase synchronization in the low-frequency band to accomplish information sharing of conscious content.

Interestingly, the beginning time of prefrontal visual awareness-related activities found in ERP and HG activities is very consistent with the onset time of dynamic functional connectivity enhancement, both of which start at 200-300 ms. In addition, the dynamic functional connectivity of the sites with early HG visual awareness-related activity is usually the earliest to enhance compared to other sites, and the two have good consistency. And this consistency suggested that conscious access and phenomenal awareness may be closely coupled, occurring initially in the lateral PFC at the early stage, and indicate a potential process of conscious perception: subtle change in the external stimulus or internal brain state - population neuron discharge in specific brain areas (ignition, phenomenal consciousness appeared) - phase synchronization of various brain areas induced by specific brain areas (conscious access). Further research is needed to regulate the activity of the middle lateral prefrontal cortex in the high-gamma band or the phase synchronization in the low-frequency band to explore whether the prefrontal cortex plays a causal role in the process of conscious perception.

Methods

Data acquisition

Six adult patients with drug-resistant epilepsy (6 males, 32.33±4.75 years old, mean±SEM) and 10 healthy subjects (4 males, 6 females, age 27.80 ± 3.92, mean ± SEM) participated in this study. All subjects had normal or corrected-to-normal vision. Electrophysiological signals (LFPs) were obtained from 6 patients. Stereotaxic EEG depth electrodes (SINOVATION MEDICAL TECHNOLOGY CO., LTD. Ltd., Beijing, China) containing 8 to 20 sites were implanted in patients in the Department of Neurosurgery, PLA General Hospital. Each site was 0.8 mm in diameter and 2 mm in length, with 1.5 mm spacing between adjacent sites. A few electrodes were segmented, and the distance between the two segments was 10 mm. Electrode placement was based on clinical requirements only. Recordings were referenced to a site in white matter. The sEEG signal was sampled at a rate of 1 kHz, filtered between 1 and 250 Hz, and notched at 50 Hz (NEURACLE Technology Co., Ltd., Beijing, China). Stimulus-triggered electrical pulses were recorded simultaneously with sEEG signals to precisely synchronize the stimulus with electrophysiological signals.

Ten healthy subjects performed the experiment in a quiet dark room. Stimuli were presented on a 27-inch LED screen (BenQ, refresh rate of 120 Hz, resolution of 1920 x 1080), and eye movement data were recorded by an infrared eye tracker (EyeLink 1000 Desktop; SR Research) with a sampling rate of 1 kHz. Patients performed the experiment in a quiet dark environment. The stimuli were presented on a 24-inch LED screen (Admiral Overseas Corporation, refresh rate of 144 Hz, resolution of 1920 x 1080), and eye position data were obtained by an infrared eye tracker (Jsmz EM2000C, Beijing Jasmine Science & Technology, Co. Ltd), sampling at 1 kHz. The experimental paradigm was presented by MATLAB (The MathWorks) and the toolbox Psychtoolbox-3 (PTB-3; Brainard and Pelli). Patients were regular recipients of standard medication for epilepsy treatment during the experimental period.

All subjects provided informed consent to participate in this study. The Ethics Committee of Beijing Normal University approved the experimental procedures.

Electrode localization

Electrode locations were determined by registering postoperative computed tomography (CT) scans and preoperative T1 MRI using a standardized mutual information method in SPM12 ⁴⁰. Cortical reconstruction and parcellation (Desikan-Killiany Atlas) were conducted by FreeSurfer ²⁴. To demonstrate the MNI coordinates of recording sites, a nonlinear surface registration was performed using a nonlinear registration in SPM12, aligning to the MNI ICBM152 template.

Experimental task

A trial started when a fixation point (0.5° x 0.5°, white cross) appeared at the center of the screen (radius of eye position check window is 4°, the dotted circle). After the subject fixated on the fixation point for 600 ms, a cue stimulus (Gabor grating, 2 x 2° circle) was presented for 50 ms at a fixed position (7°) on the left (or right) side of the screen for all participants. The side where the grating appears (left or right) is opposite to the hemisphere where the patients’ electrodes are implanted (left or right). For example, if the patient’s electrodes are implanted on the right side, the grating would appear on the left side of the screen. For patients 1, 5, and 6, the electrodes were implanted in both hemispheres, and the grating is set on the right side. In 70% of the trials, the grating contrast (weber contrast, see ¹⁰) was maintained near the subject’s perceptual threshold by a 1 up/1 down staircase method, and the step was either 0.39% or 0.78% for each participant. In 10% of the trials, the stimulus contrast was well above the threshold, and in the other 20% of the trials, the stimulus contrast was 0, namely, no stimulus appeared. After another 600 ms delay, the color of the fixation point turned red or green, and two saccade targets (1° x 1°, white square) appeared at fixed positions (10°) on the left and right sides of the screen. If the grating was seen, the green fixation point was required to make a saccade to the right target, while the red fixation point was required to make a saccade to the left target. If the grating was not seen, the rule of saccadic direction was inverted. Gratings with high contrast (well above the perceptual threshold) and zero contrast (grating absent) served as control conditions to evaluate the understanding and performance of the task. Before starting to collect data, subjects accepted 1-2 training sessions, and the contrast perceptual threshold was determined for each subject. In the formal experiments, each session consisted of 180 trials, and the intertrial interval (ITI) was 800 ms.

Data analysis

Each healthy subject completed 2 sessions, and each patient completed 5-7 sessions. Each session contained 180 trials in total. We excluded trials in which subjects broke fixation during the fixation period (eye position exceeded 4 x 4° check window by more than 40 ms) or the saccadic latency exceeded max response duration (2000 ms for most patients, 5000 ms for patient 6 according to the patient’s performance) after the target appeared. For all subjects, more than 85% of the trials were available for further analysis.

In each session, according to the subjects’ reports of being aware and unaware of the existence of grating, we divided the trials into awareness and unawareness states. According to the percentage of subjects reporting ‘awareness’, we further divided grating contrast into 3 levels: high contrast (aware percentage > 75%), near-threshold contrast (25% <= aware percentage <= 75%), and low contrast (aware ratio < 25%). Due to the few trials of low-contrast-aware and high-contrast-unaware trials, we next analyzed only trials under four conditions of awareness: low-contrast-unaware (LU), near-threshold-unaware (NU), near-threshold-aware (NA) and high contrast-awareness (HA). In total, under LU/NU/NA/HA conditions, there were 243/353/343/160 trials (22.1/32.1/31.2/15.6%, averaged across patients). Since we were more concerned with the differences in subjects’ behavior and electrophysiological responses during the physical stimulus, we focused on the results under NA and NU conditions in the subsequent analysis. The results under LU and HA conditions were classified as the control group and were only used to verify and check the results during calculation.

Behavioral data analysis

Psychophysical curve fitting uses the following formula:

AP, ratio of reported awareness; SC, stimulus contrast; a-d, individual fitting parameters; For population psychometric curve fitting, adjacent contrast levels were averaged, resulting in 8 contrast levels in each session.

Saccadic latency calculations

The saccadic latency was calculated according to a custom program. Eye position and saccade velocity were smoothed with 20 ms sliding windows, and each step was 1 ms. Saccade onset time was calculated as the time when the saccade velocity first exceeded 30°/s and lasted more than 20 ms. Saccadic latency was defined as the time interval between the onset of the saccadic target and the onset of the saccade.

LFP data analysis

Data preprocessing

Data preprocessing was performed by the software package EEGLAB ⁴¹ in MATLAB. Data were filtered between 1-250 Hz and notch-filtered at 50, 100, 150, 200, and 250 Hz with an FIR filter. Epochs were extracted from −490 ms before grating onset to 1299 ms after grating onset. Bad recording sites were discarded for analysis based on visual inspection and power spectral density (PSD) analysis. each recording site was applied a bipolar reference, i.e., rereferenced to its direct neighbor.

Spectral analyses

Time-frequency decomposition in high-gamma bands (60-150 Hz) was achieved by Morlet wavelets in the Brainstorm toolbox ⁴². Before computation, the mean ERP in each condition was removed. For visualization, we used a prestimulus baseline correction over −200 to 0 ms, and the results in Figure 4 show the HG activity increases or decreases relative to this baseline. To avoid windowing artifacts, we only reported results of −200 to 800 ms.

Quantitative definition of visual awareness-related activity in recording sites

We first identified visual awareness-related sites. For all recording sites, we compared the broadband amplitude (for ERP analysis) or high-gamma magnitude (for HG analysis) of LFP between the NA and NU conditions within 650 ms after grating onset by statistical tests for each time point. We defined the site where there was a significant difference (p < 0.01 FDR corrected for time points and channels, independent t-test) in the LFP between NA and NU conditions and persisted for more than 20 ms as an awareness-related site. We also defined the start time of divergence as the divergence onset time (DOT) and defined the mean of the LFP amplitude difference in the first divergence period as the divergence amplitude (DA).

Normalized DOT:

DOT, divergence onset time; T_target, time from target onset to grating onset, 650 ms; T_d, time from DOT to T_target; T_dmin, the minimum value of T_d in all awareness-related sites within one patient; _Tdmax, the maximum value of T_d of all awareness-related sites within one patient; normDOT, normalized DOT value, a larger normDOT value means a shorter DOT;

For population analysis of recording sites, we conducted baseline normalization for each site.

Baseline normalization for ERP amplitude:

x_norm, normalized value; x, LFP amplitude; μ, mean LFP amplitude in the baseline period (−200 to 0 ms).

Since it is not clearly understood so far regarding the biological characteristics of LFP polarity⁴³, to simplify such complex issue, we consider the change in magnitude of LFP during delay period in our task is awareness related activity, regardless its actual value being positive or negative. Therefore, while analyzing the awareness related population activity, we first calculate the absolute value of activity difference between aware and unaware trials in individual recording site, then pool the data of 43 recording sites together and calculate the mean and standard error of mean (SEM) (Fig. 3D). As in Fig. 3A, the activity difference between aware (red) and unaware (blue) trials lasts until/after the end of delay period. Thus, the awareness related population activity in Fig 3D extends out to 600 ms.

Baseline normalization for HG magnitude:

x_norm, normalized value; x, high-gamma magnitude or power; μ, mean of high-gamma magnitude or power in the baseline period (−200 to 0 ms); σ, standard error of high-gamma magnitude or power in the baseline period (−200 to 0 ms).

Baseline normalization for spectrogram:

x_norm, normalized value; x, power value; μ, mean power in the baseline period (−200 to 0 ms).

Topographic display of ERP and HG data

To visualize the topography of the ERP and HG activities, we used the Brainstorm ⁴² package of MATLAB to map the results on the ICBM152 cortical surface. It is a simple 3D interpolation with Shepard’s method (weights decreasing with the square of the distance), similar to what is done when projecting subject sources to an anatomy template, where vertices of the cortical surface that are further than 15 mm from the center of the sEEG contact are ignored.

For ERP visualization, we normalized the activity in absolute value because it is not clearly understood so far regarding the biological characteristics of LFP polarity. To simplify such complex issue, we consider the change in magnitude of LFP during delay period in our task represents awareness related activity, regardless its actual value being positive or negative. Therefore, we first calculated the absolute value of activity difference between aware and unaware trials in individual recording site, then used Shepard’s method to calculate the activity in each vertex and projected on the surface of brain template as shown in Fig. 3B.

Decoding Analysis

We implemented a machine learning framework for trial-by-trial classification using broadband amplitude and high-gamma magnitude of LFP activities in the PFC via the MVPA-Light toolbox ⁴⁴. We applied the LDA classification and temporal generalization on each participant separately, and all trials from each class were undersampled to balance the unequal trial number between different conditions. Then, the trials were randomly assigned into 10 folds, and all trials within each fold were subaveraged across every 4 trials to increase the signal-to-noise ratio. Decoding was then followed with a leave-one-out cross-validation procedure on the subaveraged trials. This procedure was repeated 10 times so that each fold in the dataset was used exactly once for testing, and at least once for training, but never at the same time. And the fold assignment was repeated 5 times, which resulted in total 50 iterations for the decoding analysis of each patient.

Functional Connectivity Analysis

The time-across phase-locking value (PLV) was calculated in the brainstorm toolbox. Time-frequency transformation was conducted via the Hilbert transform method. The time-across PLV measures if the phase difference between two signals is consistent across trials with respect to the event that defines the trials (i.e., the output is a mean across trials per time point)⁴⁵. PLV values range between 0 (for totally random) and 1 (for perfect phase locking). For clear demonstration here, the mean PLV in the baseline period (−200 to 0 ms) of each pair of recording sites was subtracted. The group analysis of PLV was performed according to ^46,47. The topographic display of the connectivity was conducted by the BrainNet Viewer toolbox ⁴⁸.

Data and software availability

Electrophysiological data were analyzed using MATLAB in conjunction with the toolboxes mentioned above. Data and customized MATLAB analysis scripts are available upon reasonable request from Ming-sha Zhang (mingsha.zhang@bnu.edu.cn).

Acknowledgements

We thank the participants for volunteering to take part in the study. This study was funded by STI2030-Major Projects+2021ZD0204300. National Natural Science Foundation of China (32061143004, 32030045). Funded by Open Research Fund of the State Key Laboratory of Cognitive Neuroscience and Learning (CNLZD2202).

Supplementary Information

SI Figures and Videos

Figure S1.

Figure S2.

Video S1. The spatial-temporal dynamics of visual awareness-related activities (event-related potential (ERP)) in all recording sites during the 180-650 ms period (for a clearer display, projected to the cortex). The curves in the upper left panel represent the significant (p<0.01 corrected, see Methods) difference (normalized) in averaged amplitude of local field potential in NA and NU trials at all visual awareness-related ERP sites. The brain image showed the difference projected on the surface at a specific time point after the grating appeared (top, left and right views were showed at upper right, lower left and lower right panels, respectively). The color represents the normalized voltage difference (see Methods).

Video S2. The spatial-temporal dynamics of visual awareness-related activities (high-gamma (HG) activity) in all recording sites during the 180-650 ms period (for a clearer display, projected to the cortex). The same as Video S1 but for high-gamma activity. The curves in the upper left panel represent the significant (p<0.01 corrected, see Methods) difference (normalized) in averaged magnitude of high-gamma activity in NA and NU trials at all visual awareness-related HG sites. The brain image showed the difference projected on the surface at a specific time point after the grating appeared (top, left and right views were showed at upper right, lower left and lower right panels, respectively). The color represents the normalized magnitude difference (see Methods).