Introduction

To predict the activity of neurons in the visual cortex, multiple studies have focused on correlating external stimuli with trial-averaged responses (Hubel and Wiesel, 1962; Pasupathy et al., 2020). Between the stimulus and cortical neurons, there is a complex signal processing cascade involving multiple processing stages. Therefore, computational models of visual processing typically gloss over most of the relevant biological machinery in an attempt to fit average firing rates from images (Serre et al., 2007a; Yamins et al., 2014). A mechanistic understanding of the factors that govern firing in the visual cortex requires models that can capture the trial-by-trial transformations across those processing stages. Moreover, neurons throughout the cortex fire “spontaneously” in the absence of any visual input. Thus, by definition, any model predicting neuronal activity that is exclusively dependent on visual stimulation does not account for such fluctuations. Previous studies in mice have revealed significant non-visual influences in neuronal activity in cortex, even in V1, partly accounted for by movement (Stringer et al., 2019b; Avitan and Stringer, 2022; Polack et al., 2013; Niell and Stryker, 2010; Dadarlat and Stryker, 2017). These observations contrast with a recent macaque study which did not find the same motor-related spontaneous activity in either V1, V2, or V3 (Talluri et al., 2023). Nevertheless, variables that are not related to movement, such as attention, expectation, and arousal, also modulate stimulusand non-stimulus driven neuronal activity in monkeys (Reynolds and Chelazzi, 2004; Gazzaley et al., 2007; Okazaki et al., 2008; Gilbert and Li, 2013), potentially adding to the response variability across stimulus repeats and to neuronal activity in the absence of visual stimuli.

Neuronal interactions between visual areas occur in the presence and absence of visual stimuli (Chen et al., 2022; Stringer et al., 2019b; Wosniack et al., 2021; Ringach, 2009; Avitan and Stringer, 2022). Therefore, such interactions can and should be studied both as a function of sensory inputs and contextual cues but also in the absence of external stimulation or task demands (Chacron et al., 2003; Hsu et al., 2004; Ringach, 2009). A paradigmatic example of inter-area interactions is the series of synaptically-connected laminar (e.g. layer 4 → layer 2/3) and cortical areas (e.g. V1→V2→V4→IT) within the ventral visual stream (Lee et al., 2016; Felleman and Van Essen, 1991; Markov et al., 2014; Douglas and Martin, 2004; Wang and Burkhalter, 2007; Consortium et al., 2021). Due to feedforward, feedback, and horizontal connections in the ventral visual stream, the inter-areal interactions could represent shared visual and non-visual reliable information. Several studies examined in vivo interactions between visual areas in mice and macaques, focusing on the entire population level (Semedo et al., 2019, 2022; Tang et al., 2023; Morales-Gregorio et al., 2024), trial-averaged responses removing transient fluctuations (Semedo et al., 2019), neuronal activity in response to only one image presentation (Semedo et al., 2019, 2022), or in the absence of any stimulus (Morales-Gregorio et al., 2024). Here we investigated interactions between areas in single trials at the level of cortical layers or brain areas across different stimulus types or in the absence of visual stimulation, across different species, and across different recording techniques and temporal resolutions. We focused on multiple simultaneously recorded areas of the ventral visual stream to assess the stimulus- and non-stimulus-driven variability shared between cortical subnetworks. We found that it is possible to reciprocally predict neuronal activity, both during visual stimulation but also during spontaneous activity, and that this predictability depends on the intrinsic properties of each neuron, the degree of receptive field overlap, and the relative timing of activity across areas.

Results

Layer 4 activity predicts layer 2/3 activity and V1 activity predicts V4 activity in single trials

We studied neuronal activity from two open datasets: mouse neurons in V1 layer 4 and layers 2/3 (L4 and L2/3; calcium imaging; Figure 1A) (Stringer et al., 2019a), and macaque multiunit sites in areas V1 and V4 (extracellular electrophysiology; Figure 1B) (Chen et al., 2022). The mouse neuronal recordings we used for this experiment were based on approx. 5,500 per mouse (n=4, Table 1) responding to visual stimuli (drifting gratings or static natural black and white images; total of 7 recording days), in addition to “spontaneous” activity during approximately 30 minutes of gray/black screen presentation on 6 of the 7 recording days. The macaque recordings were based on 688 out of the 1,024 channels (n=1, Table 2) responding to visual stimuli (full-size static checkerboard image, small and thin bar slow-moving in a small clockwise square direction; large and thick bar fast-moving in a big clockwise square direction; total of 5 recording days) in addition to spontaneous activity during gray screen presentation in all recording days. There was also a lights-off condition, where the head-fixed monkey was free to open or close eyes for approximately 25 min-utes in 3 our of the 5 recording days. We omitted some of the 1,024 channels with signal-to-noise ratio of less than 2, or that were considered “spurious” by the authors of the ope dataset (Chen et al., 2022).

Figure 1.

Table 1.

Table 2.

We examined two types of interactions between areas: inter-laminar (Figure 1C; mouse V1) and inter-cortical (Figure 1D; macaque). We used linear ridge regression to predict neuronal activity in one area from activity in the other area in single trials (Figure 1E) (Semedo et al., 2019). Performance was evaluated using cross-validation over trials and quantified as squared Pearson’s r (hereafter, “explained variance” or EV, Methods). Figure 2A shows sample neuronal activity from three example mouse V1 L2/3 cells during image presentation (black traces). Overlaid, the figure also shows the predicted neuronal activity (red). The predicted neuronal activity is shown as a function of the actual activity in response to every image presentation for the same example cells in Figure 2C. The top cell illustrates a case where the predicted activity closely matches the actual actual activity (EV = 0.67), the middle cell shows a typical case (EV = 0.39), and the bottom cell illustrates a case where the predictions deviated from the actual neuronal activity (EV = 0.07). We focused on neurons deemed “visually responsive” (∼17% of total L2/3 neurons; Table 1, Methods, see results for all neurons in Figure Supplement 1). The ridge regression model predicted singletrial L2/3 activity from L4 activity across both types of visual stimuli with an average EV of 0.28 ± 0.16 (mean ± stdev. across neurons, Figure 2E) whereas the shuffle control mean EV was 0.004 ± 0.002 (see results for individual mice in Figure Supplement 1).

Figure 2.

In the macaque, trial-to-trial variations in V4 activity were predicted from V1 activity across the three types of visual stimuli. Example recording sites are shown in Figure 2B, D. The ridge regression model predicted the single-trial responses in V4 activity from V1 activity with an average EV of 0.34 ± 0.15 (Figure 2F whereas the shuffle control mean EV was 0.005 ± 0.005. There were few sites that were not visually responsive in macaques; EV results for all sites are shown in Figure Supplement 1).

In sum, it was possible to provide estimates of neuronal activity in single trials in both species, across different layers within primary visual cortex in mice and across different visual cortical areas in monkeys.

Inter-cortical predictions are asymmetrical

In the previous section, we demonstrated the possibility of predicting L2/3 activity from L4 activity and V4 from V1. We asked whether we could also predict neuronal responses in the opposite direction. To directly compare predictability between directions in mouse and macaque, we matched the number of predictors (i.e., number of neurons/sites used to predict activity) and the degree of self-consistency (split-half r values) by randomly subsampling in each layer or cortical region prior to computing the predictability metrics (Figure 3A, C, Methods).

Figure 3.

In mice, it was possible to predict L4 neuronal activity from the activity of populations of neurons in L2/3 and there was no statistically significant difference between the two directions (p > 0.05, hierarchical permutation test, Figure 3B). When using the entire layer populations to predict each other’s neural activity (923-2,369 cells in L4, 5,420-7,980 cells in L2/3), L2/3 could predict L4 better than the reverse direction (p < 0.05, Figure Supplement 2G).

In macaques, while we could also predict V1 activity from the activity of a population of neurons in V4, when controlling for neuron number and split-half correlation values, the EV fraction in the V1→V4 direction was higher than in the V4→V1 direction (p < 0.001, Figure 3D). Even without controlling for the number of predictors or their respective split-half correlation values (627-688 sites in V1, 86-115 sites in V4), we found better predictability in the V1 to V4 direction than the reverse (p < 0.001, Figure Supplement 2I).

Predictability of neuronal activity is dependent on the visual stimulus

We evaluated whether the predictability of neuronal activity varied with the type of visual stimulus presented to the animal. In mice, we compared the inter-laminar prediction of neuronal activity of visually responsive neurons in response to drifting gratings versus natural images (Figure 4A). We could predict mouse L4 and L2/3 activity under both stimulus conditions (p < 0.001, paired permutation test of prediction vs. shuffled frames prediction). Predictability was higher for drifting gratings than natural images in the L4→L2/3 direction (Figure 4B; p < 0.001, hierarchical permutation test).

Figure 4.

In macaques, we compared inter-cortical predictability of visually responsive site recordings in responses to a slow-moving small thin bar, fast-moving large thick bar, and a full-size checkerboard image (Figure 4C). We could predict V1 and V4 activity across all stimulus types (p < 0.001, paired permutation test of prediction vs. shuffled frames prediction). The predictability was the highest in both directions for neuronal activity in response to a full field checkerboard images (Figure 4D). In the V1→V4 direction, the EV fraction was higher when predicting a slow moving small thin bar compared to a fast moving large thick bar (Figure 4D, left), where as the opposite was true for the V4→V1 direction (Figure 4D, right).

Neuronal activity could be predicted even during spontaneous activity

Given the dependence on the visual stimulus, we next asked whether it would be possible to predict neuronal responses in the absence of any visual stimulus, during “spontaneous activity”. We compared the predictability of stimulus-evoked activity in mice (drifting gratings and natural images) versus the predictability of activity recorded during gray screen presentation. This comparison was conducted in both visually (SNR >2 & split-half r >0.8) and non-visually (SNR <2 & split-half r <0.8) responsive neurons (n=3 mice; mouse MP027 did not undergo 30 min. of gray screen presentation). In visually responsive neurons, there was a significant reduction in EV during gray screen compared to visual stimulus presentation (Figure 5A left, p < 0.001, hierarchical paired permutation test). In contrast, for non-visually responsive neurons, predictability was higher during the gray screen condition (Figure 5A right, p < 0.001, hierarchical paired permutation test). Additionally, there was no correlation between neuronal predictability in the responses to visual stimulus presentations and in the response to gray screen presentations in visually responsive neurons (Figure 5B) but there was a strong correlation for non-visually responsive neurons (Figure 5C). The difference in predictability in the absence of a stimulus could in principle change according to the directionality in inter-laminar interactions. There was no statistically significant difference in the EV fraction between laminar directions (L4→L2/3 vs. L2/3→L4) using the same control population as in Figure 3B (Figure 5A-C and Figure Supplement 2H).

Figure 5.

In macaques, we focused on visually responsive sites since the majority of the neuronal population was visually responsive (Figure Supplement 1D). Additionally, an SNR of less than 2 (one of the requirements to define non-visual neurons in the mouse data) most likely reflects artefactual issues with the electrode recording the multiunit site (Chen et al., 2022). We compared inter-areal prediction of stimulus presentation activity (checkerboard images and moving bars), gray screen presentation, and during lights-off. Similar to the conclusions drawn from the mouse data, the predictability of neuronal activity was higher in response to stimulus presentation than to gray screen presentations (Figure 5D for checkerboard presentations, Figure Supplement 4D for moving bars; p < 0.001, paired permutation test). However, the EV fraction in the lights-off condition was significantly higher than during the stimulus presentations in both directions. Eye closure and sleep can induce global oscillations (Hohaia et al., 2022) and therefore may correlate neuronal activity, causing an increase in predictability. To test this idea, we further separated the lights-off neuronal activity into periods where the macaque’s eyes were open or closed. The EV was higher than stimulus presentation activity only during the eyes-closed period (Figure 5D). Unlike the mouse, macaque correlation of visual predictability between stimulus presentation and spontaneous activity was high across all types of spontaneous conditions (Figure 5E, Figure Supplement 4C). When assessing the inter-cortical prediction directionality during spontaneous conditions, we found the same asymmetrical relationship as in Figure 3, where V1→V4 EV fraction was significantly higher than V4→V1 prediction in both gray screen (p < 0.01, permutation test) and lights-off (p < 0.001, permutation test) conditions (Figure Supplement 2G).

Receptive field overlap and neuronal response properties impact predictability

We investigated which neuronal properties are related to the ability to predict responses by comparing EV and key indicators of neuronal response reliability and receptive field properties, in both visuallyand non-visually responsive neurons, during either visual presentations or spontaneous conditions. First, we considered the following properties: (i) max r² value (i.e., maximum squared correlation between each neuron in the predictor population and the predicted neuron), (ii) 1-vsrest r² (the squared correlation of one neuron’s activity across all stimuli with the mean neuron activity of the rest of the population), (iii) SNR of the predicted neuron, (iv) variance across stimuli (computed for mice only, given that there were 32 different stimuli presented in all stimulus recordings in mice; macaque stimulus recordings included repeating the same checkerboard image), and (v) split-half r (Methods). We plotted EV against each of these variables (mouse: Figure 6B, macaque: Figure 6E) and report the correlation coefficient between EV and each variable in the yaxis in Figure 6A (mouse) and Figure 6D (macaque).

Figure 6.

In mice, during both stimulus presentation and gray screen presentation, the most correlated property with a neuron’s inter-areal predictability was the max r² (Figure 6A). For the other 4 properties, there was a strong distinction between stimulus presentation (dark bars) and gray screen presentation (light bars): All 4 properties were positively correlated with the neural activity predictability EV fraction during stimulus presentation but they were slightly anticorrelated with their predictability EV fraction during gray screen presentation. Because the split-half correlation calculation averages out the non-stimulus-dependent variability in both halves of the trials, it showed a weaker correlation with EV, which depends on trial-by-trial modulation. The one-vs-rest r² metric, which also examines trial-by-trial modulation and does not average split-half trials, yielded a stronger correlation with EV.

When examining the relationship between 1-vs-rest self-consistency and inter-laminar prediction EV in mice, we observed a bimodal distribution of neurons: one group of neurons showed high EV despite having low self-consistency and in the other group EV correlated well with selfconsistency (Figure 6B third column). The responses of neurons with low self-consistency also showed high EV during gray screen presentation. This bimodality was present in two out of the three mice we tested (MP031 and MP032; self-consistency and EV fraction relationships across all mice can be seen in Figure Supplement 5A). To better understand the responses of neurons with low self-consistency we projected out the “non-visual ongoing neuronal activity” from the neuronal responses (Stringer et al., 2019a) (Methods). This non-visual ongoing activity is deemed to be influenced by spontaneous behavior (Stringer et al., 2019b). Projecting out this non-visual activity largely led to a unimodal distribution (Figure 6C). Removing the non-visual ongoing activity also increased the correlation between self-consistency and inter-laminar predictability (Figure 6C). This observation could be because the responses of neurons with low self-consistency can no longer be predicted, or because the responses of those neurons became more reliable and therefore were highly predicted. To distinguish between these two possibilities, we compared both the 1-vs-rest self consistency and the prediction EV before and after removing the non-visual activity. Removing the non-visual ongoing activity increased the self-consistency value across the three mice (Figure Supplement 5C; p < 0.001, paired permutation test). Interestingly, the inter-laminar EV fraction decreased in MP031 and MP032 mice, yet increased in MP033 (Figure Supplement 5D p < 0.001, paired permutation test). When examining individual pairwise relationships in a fraction of highly predictable neurons, we found that some of the highly predictable neurons remained predictable after removing the non-visual activity whereas other highly predictable neurons dropped EV fraction dramatically.

In macaques, one of the highest correlated property with inter-areal prediction EV across all conditions was also the max correlation value (Figure 6D, E first column). Other neuron properties like SNR, split-half correlation and one-vs-rest correlation were also highly correlated with inter-cortical predictability EV(Figure 6D). Unlike the mouse, the split-half correlation was highly correlated with EV fraction, although the relationship was highly non-linear (Figure 6D, middle column). Using the one-vs-rest squared correlation removed some of this non-linearity and further increased the correlation between it and the EV fraction (Figure 6D, third column). In addition, there was no bimodal distribution of neurons when relating one-vs-rest correlation and EV fraction.

We conjectured that neurons that have overlapping receptive fields (RFs) should share more information, and therefore their responses would better predict each other than neurons with non-overlapping RFs. In addition, even when all neurons are exposed to the same stimulus (full field symmetrical checkerboard image, gray screen, darkness, etc), neurons with overlapping RFs may be more synaptically connected, resulting in better inter-cortical predictions. To test this hypothesis, we compared inter-cortical predictions in different ensemble of neurons with RFs that differed in their degree of overlap. This hypothesis was only tested in the macaque data because we did not have access to RF estimates in the mouse data. For each V4 site whose responses we predicted, we separated the predictors into two size controlled groups: one where all the V1 predictor sites had <10% RF overlap (sample of one V4 site, Figure 6F, top), and one where all the V1 predictor sites had >80% RF overlap (sample of one V4 site, Figure 6F, bottom). A similar procedure was followed when predicting the activity of V1 neurons from V4 predictor neurons (Figure 6H). Inter-areal prediction was higher in the >80% RF overlap condition compared to the <10% RF overlap ensembles in both directions and across all stimulus conditions (Figure 6G,I, n = 110 total V4 site recordings across all conditiosn, n = 970 total V1 site recordings across all conditions). In most cases, predicting the >80% RF overlap ensembles was still lower than ceiling performance (when using all predictors with all types of overlap percentages; Figure 6E,F)

Inter-areal predictability is both stimulus and non-stimulus driven

The results in Figure 5 and Figure 6 pointed to components of the predictable responses that are stimulus driven and other components that are non-stimulus driven. To further examine the non-stimulus driven component, we reasoned that if the shared information between areas were strictly driven by the visual stimulus, then using the activity of a stimulus presentation repeat to one specific image could be used to predict the responses to any other stimulus repeat of the same image. On the other hand, if the shared activity does not have any stimulus response information, then the prediction model would not work when considering responses across repeated presentation of identical stimuli in different trials. To test these two opposing ideas, we compared the inter-areal prediction EV fractions using unshuffled versus shuffled trials. Shuffling was done across repeat trials of the same images (mice: Figure 7A, macaques: Figure 7D). In mice, one stimulus presentation was either a drifting grating or a natural static image. In macaques, one stimulus presentation was either the one checkerboard image, a large thick fast moving bar, or a small thin slow moving bar. In both species and in both directions, inter-areal prediction EV fraction persisted (p < 0.001, paired permutation test of shuffled trials prediction vs.shuffled frames prediction), yet the EV fraction decreased after shuffling stimulus repeats compared to before shuffling (Figure 7B,E). In mice, neurons showed a bimodal distribution in terms of their response predictability in shuffled and unshuffled trials. For a subset of neurons, the EV fraction was still high in the shuffled condition, albeit their EV was still higher in the unshuffled case (Figure 7C; points below but near the diagonal line). For another subset of neurons the EV fraction during shuffled trials was much lower or even near zero. The responses of the latter group had the highest predictability during gray screen activity. In the macaque, there was no bimodal distribution, yet neurons farther away from the diagonal line also had a higher EV fraction during gray screen activity (Figure 7F). In addition, we examined whether shuffling repeat presentations of gray screen images (simulating spontaneous activity) would result in any prediction at all. We found a more profund decrease in inter-cortical performance (Figure Supplement 6B) with no neurons that remained as predictable during shuffled repeats compared to unshuffled repeats (Figure Supplement 6C,D).

Figure 7.

Accounting for latency differences improves inter-areal activity predictions in macaque visual area sub-populations

Given the latency differences in neuronal responses between V1 and V4 Schmolesky et al., 1998, we asked whether accounting for this latency could result in better inter-area prediction. To test this hypothesis, we offset the neuronal activity using different lags for each area (Figure 7G, H) and recalculated the ridge regression predictions. For each offset level, we calculated the percentage of neurons where the EV fraction peaked at that offset. For the checkerboard image, in the macaque V1→V4 predictions, the biggest percentage of neurons had a peak performance when there was no time offset between areas (Figure 7I, left). A substantial proportion of neurons had a peak performance for 25 ms or 50 ms offsets in the negative direction (i.e., V1 activity preceding V4 activity). This distribution of peak EV values was only present during early visual responses (first 275 ms of stimulus onset). In the macaque V4→V1 direction, there was a large proportion of neurons with peak EV when considering 25 ms to 50 ms offsets in the positive direction (i.e., V4 after V1, Figure 7I, right). These differences were apparent in the early part of the visual response, before 250 ms. When offsetting the neuronal responses to gray screen presentations, across all times and areas, the highest percentage of neurons with peak EV was when there was no time offset (Figure Supplement 6E,F).

Discussion

Neuronal activity in one brain region or layer within the visual cortex can be used to predict neuronal activity in another nearby and anatomically connected region or layer in single trials (Figure 2). In monkeys, predictability was asymmetric: V1 activity better accounted for V4 activity than vice versa (Figure 3, Figure 7). This inter-areal prediction persisted across different stimuli (Figure 4) but also in the absence of a visual stimulus, during gray-screen and lights-off periods (Figure 5). The degree of predictability increased with signal-to-noise ratio, response variance, and the degree of overlap between receptive fields (Figure 6).

In line with other studies in mice (Stringer et al., 2019b; Niell and Stryker, 2008; Andermann et al., 2011), we observed an approximately bimodal distribution of neuronal responses, with a large subset of neurons that do not show reliable responses to visual stimuli both in L4 and L2/3. Yet, even if these neurons are “non-visual”, at least within the set of stimuli and conditions examined here, their activity remains highly predictable. This bimodal distribution dissipates when projecting out potential non-sensory ongoing activity (Stringer et al., 2019b, 2021). At the population level, neuronal encoding subspaces in mouse visual cortex have been shown to have little overlap between visual sensory and non-sensory (behavioral) information, with only one shared dimension (Stringer et al., 2019b). The visually unreliable, yet highly predictable, subset of neurons described here could be the neuronal group driving this orthogonality. As expected, the activity of “visual” neurons can be better predicted during visual presentation and is predicted almost at chance levels during gray screen presentation. In stark contrast, the activity of non-visual neurons can be predicted even better during gray screen presentation than during visual stimulation. There was no such bimodal distribution in the data from monkeys. One possibility is that there may be no (or very few) non-visual neurons in macaque V1 or V4. Indeed the overwhelming majority of neurons in V1 and V4 responded strongly to visual stimulation. Yet, the comparisons between the results in mice and monkeys reported here need to be interpreted with caution because the two datasets differ in terms of recording techniques (electrophysiology versus two-photon imaging), consequently also the temporal resolution (one millisecond versus hundreds of milliseconds), and the type of interaction studied (across areas versus across layers), in addition to any differences between species.

In macaques, sites where the receptive fields (RF) of V1 and V4 overlap can better predict each other compared to other sites showing little RF overlap. This observation could reflect RFdependent intrinsic connectivity between areas, but also RF-dependent shared inputs from other areas like the thalamus. In the latter case, those putative shared inputs cannot be strictly dependent on visual inputs given that the effect of RF overlap persists even during gray screen conditions.

Many computational models that aim to explain neuronal activity in visual cortex are based on feedforward signal propagation, with increased receptive field sizes, selectivity, and feature invariance along the visual hierarchy (Serre et al., 2007b; Kreiman, 2021; Connor et al., 2007). Consistent with this idea, we described an asymmetry in the degree of predictability, with V1 neurons explaining V4 responses better than the other way around. This observation persisted after controlling for neuronal count and split-half correlation values and was also apparent during the lights-off condition. In contrast, there was no asymmetry when comparing inter-laminar prediction directions in mice. The lack of asymmetry in inter-laminar prediction directions in mice could be due to the slow dynamics in calcium imaging, the lack of a clear inter-areal hierarchy, or differences between species.

The asymmetry in directionality is also observed when implementing temporal delays to interareal prediction, consistent with processing delays across areas (Semedo et al., 2022; Gokcen et al., 2022; Schmolesky et al., 1998). A substantial proportion of neurons increased their inter-areal predictability when offsetting the times between areas, specifically in the direction that aligns their neuronal activities In contrast to the temporal delays associated with processing visual stimulation, during gray screen presentation, the majority of neurons was best predicted in the absence of any time offsets, suggesting that the internally generated neuronal activity during spontaneous conditions may be largely driven in a non-feedforward manner.

Further evidence supporting the distinction between visually-driven and non-visually-driven interactions comes from the observation that trial repeat shuffling reduced, but did not eliminate, predictability in both mice and monkeys. In mice, when plotting shuffled vs. unshuffled activity, we encountered again a bimodal distribution, where a group of neurons was closer to the diagonal line (their responses were predicted as well during the shuffled compared to the non-shuffled condition), and another group of neurons which were closer to the x-axis (their responses could not be predicted during the shuffled condition). The responses of the latter group were best predicted during gray screen activity, suggesting that they mostly shared non-visual information. The predictive power in mouse V1 from layer 4 to layer 2/3 during spontaneous conditions has been recently shown in (Papadopouli et al., 2024), consistent with our findings. The overall area population decrease in predictability after shuffling may be due to the influence of non-visual activity such as movement (Stringer et al., 2019b), especially since these non-visual stimulus effects have been shown to occur in the one-second timescale as in our study. In the macaque, context-dependent effects are likely not due to movement, since the monkey maintained fixation during the stimulus task, and visually-evoked activity is not driven by movement (Talluri et al., 2023).

The results on the prediction of neuronal responses constitute a lower bound. First, we focus on linear predictability but other (non-linear) models could better capture neuronal activity. Second, and critically, the experimental data provide only a fraction of the inputs to a given neuron– excluding most (in macaque dataset) if not all (in mouse dataset) inhibitory inputs that are crucial for the organization of circuit and microcircuits in visual cortex (Jiang et al., 2015; Shen et al., 2020; Ibrahim et al., 2020; Schuman et al., 2021). Third, biophysical realistic models of the transformation between inputs and outputs of a given neuron should include their dendritic locations and specific synaptic potentials (Park et al., 2019; Petousakis et al., 2023).

We introduce a unifying method to evaluate inter-areal interactions in different types of neuronal recordings, timescales, and species. These interactions can be assessed in single trials, separating visually-driven and non-visual contributions, and accounting for the directionality and dynamics of neuronal responses. These efforts constitute an initial step towards systematically building computational models that can account for the transformations from sensory inputs to their encoding in the cortex.

Methods

Datasets

We used the mouse dataset from (Stringer et al., 2019a) containing calcium-imaging activity measurements from 43,630 neurons in layer 4 (L4) and 12,060 neurons in layers 2/3 (L2/3) in V1 of 4 mice during 32 different randomly interleaved presentations of either drifting gratings or gray-scale natural images (each one repeated more than 90 times), along with spontaneous activity during 30 minutes of exposure to a gray/black screen (Figure 1A, data acquisition details in (Stringer et al., 2019a,b)). Calcium imaging activity was recorded during stimulus presentations at a scan rate of 2.5 Hz or 3 Hz (each frame was acquired every 400 ms or 333 ms). The computed stimulus responses per stimulus presentation were averaged based on two frames immediately post stimulus onset. Cortical layers were determined using the 10-12 planar z-positions retrieved during the multi-plane calcium activity acquisition. For stimulus-response and spontaneous recordings, neuronal activity of each neuron was z-scored using its 30-minute gray screen spontaneous activity (mean gray-screen activity subtracted and divided by gray-screen activity standard deviation).

We used the macaque monkey dataset from (Chen et al., 2022). This dataset consists of envelope multiunit activity (MUAe) from 1,024 recording sites in one monkey in response to either multiple-day recordings of more than 60 repetitions of a full-size checkerboard image, moving small-thin or large-thick bars in 4 directions, gray screen presentations, or more than 30 minutes of baseline activity where the monkey was in a room with the lights off (Figure 1B). Neuronal activity was averaged over 25 ms non-overlapping bins. Activity duration was 300 ms, 400 ms, and 1 s for gray screen, checkerboard, and moving bar presentations, respectively. For the recordings during visual stimulation, the neuronal activity was normalized by subtracting the mean activity during the gray screen presentations separately for each site.

Visual responsiveness

A neuron or site was defined to be visually responsive if its signal-to-noise ratio (SNR) was 2 or higher and its split-half correlation value was 0.8 or higher. Due to the high number of repetitions of visual stimuli, the split-half correlation was skewed toward high values, which is why we used a higher split-half correlation threshold than commonly used in other studies. In mice, the SNR for each neuron was calculated as:

where <> denotes mean, std denotes the standard deviation, r_stim is the average activity in response to stimuli, and r_spont indicates the average activity over the 30-minute gray screen presentation activity.

In monkeys, we followed the definition in (Chen et al., 2022) and calculated the SNR for each site as:

using the peak activity during the checkerboard presentation for the signal, and the average gray screen neuronal activity as background (denoted as r_spont).

In mice, the split-half consistency was calculated by correlating the average activity for the 32 stimuli in a randomly chosen half of the trials, with the average activity in the other half of the trials, followed by Spearman-Brown correction (used to correct for the division of trials by half). In monkeys, during checkerboard presentations, the split-half consistency was calculated by correlating the average activity of the 16 timepoints (0–400 ms; 25 ms bins) of checkerboard presentation of 25 random trial repetitions with the average activity of another non-overlapping 25 random repetitions, followed by Spearman-Brown correction. During moving bar presentations, the 40 timepoints (0–1s; 25 ms bins) during 25 random trial repetitions were first concatenated across the 4 directions (total of 160 timepoints), and then correlated to the concatenated averaged activity of another nonoverlapping 25 random trial repetitions, followed by Spearman-Brown correction. For all split-half consistency calculations, we randomly sampled trials 20 times.

Inter-areal regression

Let _Ar_i,t be the activity of neuron or site i in area A at timepoint t, where A can be L4 or L2/3 in the mouse data and V1 or V4 in the monkey data. Neuronal activity from one area (e.g., mouse V1 L4 or macaque V1) was used to predict activity in the other area (e.g., mouse V1 L2/3 or macaque V4) using ridge regression (Figure 1E). The activity of each neuron i in area A2 was predicted from n_A1 neurons in area A1 as follows:

During fitting, we minimized the residual sum of squares (RSS), defined as:

where w is the weight vector for predicting the activity of neuron i, n_T is the number of images/time points and α controls the regularization strength (α was tuned for each dataset with an independent sample and ranged from 10³ to 10⁵). Predictability for each neuron was evaluated using 10fold cross-validation across trials and quantified as squared Pearson’s r, referred to as explained variance fraction (EV fraction) throughout.

To remove temporal auto-correlation that would inflate the apparent prediction despite crossvalidation, we removed training timepoints near the test timepoints closer than the decay window of the activity auto-correlation (mouse: 5 s; macaque: 1.5 s). The auto-correlation decay window was determined using time-series forecasting Ridge Regression (using r_t to predict r_t+d , where d represents a delay). The delay was increased until the EV fraction approached chance.

Prediction directionality

We compared predictability across layers in different directions (in mice: L4→L2/3 vs. L2/3→L4) and also predictability across areas in different directions (in macaques: V1→V4 vs. V4→V1) (Figure 3). To ensure that results were not dependent on the number of neurons/sites, we randomly subsampled the number of neurons/sites of the area containing the larger number of neurons/sites (L2/3 for mouse; V1 for macaques) to match the number of predictors in both directions (10 permutations, neuron count details in Table 1 and Table 2). To account for potential changes in intrinsic predictability, we ensured that the neurons from both areas were matched in terms of the distribution of split-half correlation values so that the difference between individual area neurons/sites was less than 0.002. To assess the intrinsic predictability of neurons/sites in each region, the areas were used to predict themselves, where one neuron/site in the area was predicted by the remaining neurons in the same area. This “intra-areal” prediction was used to normalize EV fraction to compare directionality of prediction.

Stimulus types and spontaneous activity comparison

We compared predictability for different stimulus conditions Figure 4, Figure 5). To compare inter-areal prediction across stimulus types and between the presence or absence of stimuli , the number of predictors (neurons or sites) and timepoints was sub-sampled to be the same across all datasets. In the macaque, the time spent recording the lights-off condition was much greater than during stimulus or gray screen presentations. To account for the difference in time duration and therefore training size, we subsampled time periods to be the same across all stimulus, gray-screen, and lights-off, lights-off eyes open, and lights-off eyes closed conditions.

Neuron properties

We compared different neuronal properties with predictability measurements (Figure 6). The SNR and split-half correlation has been defined above. The absolute max pairwise correlation value of each neuron/site i in one area with all neurons in the other area was calculated as

where _A2r_i represents the activity of neuron/site i in area A2, which are correlated with the activity of every neuron j in area A1 (denoted as _A1r_j ).

The one-vs-rest correlation was calculated as follows. In the mouse data, we correlated the activity for the 32 stimuli during 1 trial repetition with the averaged activity of the remaining trial repetitions. In monkey, during checkerboard presentations, the one-vs-rest correlation was calculated by correlating the activity of the 16 timepoints (0–400 ms; 25 ms bins) during 1 trial repetition with the averaged activity of the remaining trial repetitions. For moving bar presentations, the 40 timepoints (0–1s; 25 ms bins) during 1 trial repetition were first concatenated across the 4 directions (total of 160 timepoints), and then correlated to the concatenated, trial-averaged activity of the remaining trial repetitions. For all one-vs-rest correlation calculations, we held out each trial in turn and averaged across the samples.

Receptive field overlap comparisons

In macaque, receptive field (RF) ellipses were calculated using center and edge locations in the dataset (Figure 6E, F). To calculate the percentage of RF overlap between the neuronal sites to be predicted and the predictor, the intersection area between ellipses was retrieved using the Shapely python package, and divided by the area of the predicted site. Sites that had predictors that overlap both more than 80% and less than 10% were selected to compare inter-areal predictions. To control for predictor size, 14 random predictor sites from all the sites in each overlap type were subsampled (10 random samples without replacement).

Trial repeat shuffling and time offset predictions

For the shuffled-trial experiments, we shuffled the predictor activity across repeat trials showing the same stimulus (Figure 7). (Thus, the stimulus order remained the same.) For the mouse time-offset analysis, the activity of predictor neurons was time-shifted in the positive or negative direction, with 1 bin corresponding to 1 stimulus presentation (800–900ms). For the monkey dataset, the predictor activity (400 or 1000 ms per presentation, 16–50 bins of 25 ms each) was offset across time bins. We used sub-windows of 200 ms to avoid window-length differences that would otherwise be introduced if we shifted the entire trial response.

Data and code availability

All the computational models and data analysis code developed in this work is publicly available at this link: https://github.com/4sdch/inter-area-neural-prediction.git. All the data are publicly available for mouse: https://figshare.com/articles/dataset/Recordings_of_ ten_thousand_neurons_in_visual_cortex_in_response_to_2_800_natural_images/6845348?file= 12462734 (Stringer et al., 2019a) and for macaques: https://gin.g-node.org/NIN/V1_V4_1024_ electrode_resting_state_data (Chen et al., 2022).

Supplemental Material

Figure Supplement 1.

Figure Supplement 2.

Figure Supplement 3.

Figure Supplement 4.

Figure Supplement 5.

Figure Supplement 6.

Acknowledgements

The authors would like to thank Elisa Pavarino, Leonardo Polina, Sara Djambazovksa, Carlos Ponce, Jan Drugowitsch, and Wei-Chung Allen Lee for providing comments on the manuscript.