Introduction

To predict the activity of neurons in the visual cortex, multiple studies have focused on correlating external stimuli with trial-averaged responses (Hubel and Wiesel, 1962; Pasupathy et al., 2020). Between the stimulus and specific cortical neurons, there is a complex signal processing cascade in-volving multiple processing stages. Therefore, computational models of visual processing typically gloss over most of the relevant biological machinery in an attempt to fit average firing rates from images (Serre et al., 2007a; Yamins et al., 2014). A mechanistic understanding of the factors that govern firing in the visual cortex requires models that can capture the trial-by-trial transformations across those processing stages. Moreover, neurons throughout the cortex fire “spontaneously” in the absence of any visual input. Thus, by definition, any model predicting neuronal activity that is exclusively dependent on visual stimulation does not account for such fluctuations. Previous studies in mice have revealed significant non-visual influences in neuronal activity in cortex, even in V1, partly accounted for by movement (Stringer et al., 2019b; Avitan and Stringer, 2022; Polack et al., 2013; Niell and Stryker, 2010; Dadarlat and Stryker, 2017). These observations contrast with a recent macaque study which did not find the same motor-related spontaneous activity in either V1, V2, or V3 (Talluri et al., 2023). Nevertheless, variables that are not related to movement, such as attention, expectation, and arousal, also modulate stimulus- and non-stimulus driven neuronal activity in monkeys (Reynolds and Chelazzi, 2004; Gazzaley et al., 2007; Okazaki et al., 2008; Gilbert and Li, 2013), potentially adding to the response variability across stimulus repeats and to neuronal activity in the absence of visual stimuli.

Neuronal interactions between visual areas occur in the presence and absence of visual stimuli (Chen et al., 2022; Stringer et al., 2019b; Wosniack et al., 2021; Ringach, 2009; Avitan and Stringer, 2022). Therefore, such interactions can and should be studied both as a function of sensory inputs and contextual cues but also in the absence of external stimulation or task demands (Chacron et al., 2003; Hsu et al., 2004; Ringach, 2009). A paradigmatic example of inter-areal interactions is the series of synaptically-connected laminar (e.g. layer 4 → layer 2/3) and cortical areas (e.g. V1→V2→V4→IT) within the ventral visual stream (Lee et al., 2016; Felleman and Van Essen, 1991; Markov et al., 2014; Douglas and Martin, 2004; Wang and Burkhalter, 2007; The MICrONS Consortium et al., 2021). Due to feedforward, feedback, and horizontal connections in the ventral visual stream, the inter-areal interactions could represent reliable shared visual and non-visual information. Several studies examined interactions between visual areas in mice and macaques, focusing on the entire population level (Semedo et al., 2019, 2022; Tang et al., 2023; Morales-Gregorio et al., 2024), trial-averaged responses removing transient fluctuations (Semedo et al., 2019), neuronal activity in response to only one image presentation (Semedo et al., 2019, 2022), or in the absence of any stimulus (Morales-Gregorio et al., 2024). Here we investigated interactions between areas in single trials at the level of cortical layers or brain areas across different stimulus types or in the absence of visual stimulation, across different species, and across different recording techniques and temporal resolutions. Throughout, we use the term “interactions” to denote shared variability at the single-trial level, including stimulus-induced correlations and trialbytrial stimulus-independent fluctuations which can arise from internal state, contextual inputs, or other top-down factors. We focused on multiple simultaneously recorded areas of the ventral visual stream to assess the stimulus- and non-stimulus-driven variability shared between cortical subnetworks. We found that it is possible to reciprocally predict neuronal activity, both during visual stimulation but also during spontaneous activity, and that this predictability depends on the intrinsic properties of each neuron, the degree of receptive field overlap, and the relative timing of activity across areas.

Results

Layer 4 activity predicts layer 2/3 activity and V1 activity predicts V4 activity in single trials

We studied neuronal activity from two open datasets: mouse neurons in V1 layer 4 and layers 2/3 (L4 and L2/3; calcium imaging; Figure 1A) (Stringer et al., 2019a), and macaque monkey multiunit sites in areas V1 and V4 (extracellular electrophysiology; Figure 1B) (Chen et al., 2022). In addition, we recorded new data from an additional monkey during spontaneous activity and using the same visual stimuli as in Chen et al. (2022) (Methods). The mouse neuronal recordings we used for this experiment were based on approx. 5,500 neurons per mouse (n=4, Table 1) responding to visual stimuli (drifting gratings or static natural black and white images; total of 7 recording days), in addition to “spontaneous” activity during approximately 30 minutes of gray/black screen presentation on 6 of the 7 recording days. The monkey recordings were based on a range of 32-803 electrodes per animal (3 monkeys, L, A, and D; Table 2) responding to visual stimuli (full-size static checkerboard image; n=3 recordings for monkey L, n=1 for monkey A, n=2 for monkey D), small and thin bar slow-moving in a small clockwise square direction (n=1 for monkeys L and A); or large and thick bar fast-moving in a big clockwise square direction (n=1 for monkeys L and A) in addition to spontaneous activity during gray screen presentation in all recording days (n=5 for monkey L, n=3 for monkey A, n=2 for monkey D). The recordings consisted of multi-unit activity (MUAe) and local field potentials (LFP) in monkeys L and A retrieved from the open dataset by Chen et al. (Chen et al., 2022). Monkey D recordings were acquired using the same checkerboard stimulus presentation conditions as stated in Chen et al. (2022). There was also a lights-off condition, where the head-fixed monkeys were free to open or close eyes for approximately 10-25 minutes (monkeys L/D; 2-3 recording days). We omitted channels with signal-to-noise ratio of less than 2, or channels that were considered “spurious” by the authors of the open dataset (Chen et al., 2022).

Figure 1.

Table 1.

Table 2.

We evaluated two types of interactions between areas: inter-laminar (Figure 1C; mouse V1) and inter-cortical (Figure 1D; monkey). We used linear ridge regression to predict neuronal activity in one area from activity in the other area in single trials (Figure 1E, see Methods) (Semedo et al., 2019). For each neuron/site in the target area we fit a linear readout from the other area’s simultaneous population activity on the same trial, with an L₂ (“ridge”) penalty. Performance was evaluated using cross-validation over trials and quantified as squared Pearson’s r (hereafter, “explained variance” or EV, Methods). Figure 2A shows sample neuronal activity from three example mouse V1 L2/3 cells during image presentation (black traces). Overlaid, the figure also shows the predicted neuronal activity (red). The predicted neuronal activity is shown as a function of the actual activity in response to every image presentation for the same example cells in Figure 2C. The top cell illustrates a case where the predicted activity closely matches the actual activity (EV = 0.67), the middle cell shows a typical case (EV = 0.39), and the bottom cell illustrates a case where the predictions deviated from the actual neuronal activity (EV = 0.07). We focused on neurons deemed “visually reliable” (∼17% of total L2/3 neurons; Table 1, Methods, see results for all neurons in Figure Supplement 1C). The ridge regression model predicted single-trial L2/3 activity from L4 activity across both types of visual stimuli with an average EV of 0.28 ± 0.16 (mean ± stdev. across neurons, Figure 2E) whereas the shuffle control mean EV was 0.004 ± 0.002.

Figure 2.

In monkeys, trial-to-trial variations in V4 activity were predicted from V1 activity across the three types of visual stimuli. Example recording sites for monkey L are shown in Figure 2B, D (see also Figure Supplement 1 D–G for examples from two additional monkeys, A and D). The ridge regression model predicted the single-trial responses in V4 activity from V1 activity with an average EV of 0.30 ± 0.15 (Figure 2F for monkey L, see Figure Supplement 1H for monkey A and Figure Supplement 1I for monkey D), whereas the shuffle control mean EV was 0.005 ± 0.005. Most sites in V4 (72.3%) were deemed visually reliable; EV results for all monkey sites are shown in Figure Supplement 1J). EV performance varied across the three monkeys (EV_L = 0.34, EV_A = 0.21, EV_D = 0.11). This varia-tion may be due to predictor population size (Table 2). We subsampled sites in monkey L to match the numbers of monkey A and monkey D (20 permutations; Figure Supplement 1L; subsampling was performed for all analyses hereafter unless noted). Subsampled monkey L matching the num-ber of sites in monkey A (monkey L_A) showed an average EV of 0.35 ± 0.14, statistically higher compared to the EV for monkey A (p < 0.001, permutation test). Similarly, subsampled monkey L matching the number of sites in monkey D (monkey L_D) showed an average EV of 0.19 ± 0.11, statistically higher compared to the EV for monkey D (p < 0.001, permutation test).

We asked whether the conclusions were dependent on the model and parameter choices. As an alternative approach to ridge regression, we also fit a Poisson general linear model (GLM) enforcing non-negativity to monkey MUAe without baseline subtraction, using identical folds, bin widths, and temporal gaps as in the ridge regression analysis. The results from the Poisson GLM were consistent with the results obtained using ridge regression (Figure Supplement 1M). Additionally, to evaluate the impact of the bin size on the results, we re-estimated EV after binning MUAe into 10–200 ms. EV increased with bin size and remained well above shuffle controls across conditions (Figure Supplement 4A–C, p < 0.001, paired permutation test), indicating that inter-areal predictability is robust across a broad range of physiologically relevant timescales.

In sum, it was possible to provide estimates of neuronal activity in single trials in both species, across different layers within primary visual cortex in mice and across different visual cortical areas in monkeys.

Inter-cortical predictions are asymmetrical

In the previous section, we demonstrated the possibility of predicting L2/3 activity from L4 activity and V4 from V1. We asked whether we could also predict neuronal responses in the opposite direction. We could predict L4 activity from L2/3 activity in mice and V1 activity from V4 activity in monkeys (Figure Supplement 1A for mice, Figure Supplement 1K for monkeys). To directly compare predictability between directions in mice and monkeys, we matched the number of predictors (i.e., number of neurons/sites used to predict activity) and the degree of self-consistency (split-half r values) by randomly subsampling in each layer or cortical region prior to computing the predictability metrics (Figure 3A, C, Methods).

Figure 3.

In mice, there was no statistically significant difference between L4→L2/3 and L2/3→L4 directions (p > 0.05, hierarchical permutation test, Figure 3B). In monkeys, after matching predictor count and split-half correlation values, the EV fraction in the V1→V4 direction was higher than in the V4→V1 direction (p < 0.001 for monkey L, Figure 3D; p < 0.05 for monkey A, Figure Supplement 3C). This directional asymmetry was stable across temporal bin sizes in monkey L (Figure Supplement 4D). In monkey A, the asymmetry was small and was not robust across different bin sizes (Figure Supplement 4E), consistent with the noisier data and weaker effect size in that ani-mal. There were not enough sites with similar reliability distributions in monkey D to perform the matched subsampling directionality tests.

Although we accounted for predictor size and reliability distributions, differences in prediction performance between directions could still reflect additional inherent differences in target-population properties (Figure Supplement 2F–J). To control for these differences, we modeled EV as a function of target–population properties and used the residuals for inference, EV_resid = EV−^ÊV.

Covariates included self-consistency, SNR, one–vs–rest r², and variance metrics (Methods). After adjustment, the difference between inter-cortical directions remained statistically significant overall (p < 0.001 for monkey L, p < 0.001 for monkey A, Figure Supplement 3F–G). These results suggest that the differences between directions cannot be attributed to target-related neuronal proper-ties. We also analyzed the residuals in mice prediction directions, and after controlling for neuron properties, we still found no statistically significant difference in predictability directions (Figure Supplement 3E, p = 0.22 for stimulus activity, p = 0.48 for gray screen activity).

Predictability of neuronal activity is dependent on the visual stimulus

We evaluated whether the predictability of neuronal activity varied with the type of visual stimulus presented to the animal. In mice, we compared the inter-laminar prediction of neuronal activity of visually reliable neurons in response to drifting gratings versus natural images (Figure 4A). We could predict mouse L4 and L2/3 activity under both stimulus conditions (p < 0.001, paired permutation test of prediction vs. shuffled frames prediction). Predictability was higher for drifting gratings than natural images in the L4→L2/3 direction (Figure 4B; p < 0.001, hierarchical permutation test).

Figure 4.

In monkeys L and A, we compared inter-cortical predictability of visually reliable sites in responses to a slow-moving small thin bar, fast-moving large thick bar, and a full-size checkerboard image (Figure 4C). As expected, V1 and V4 could predict each other’s neural activity across all stimulus types (p < 0.001, paired permutation test of prediction vs. shuffled frames prediction). The predictability was highest in both directions for neuronal activity in response to a full-field checker-board image (monkey L Figure 4D; monkey A Figure Supplement 5F). In the V1→V4 direction, the EV fraction was higher when predicting a slow-moving small thin bar compared to a fast-moving large thick bar (monkey L Figure 4D, left; monkey A Figure Supplement 5F, left).

In the V4→V1 direction, there was no consistency in prediction differences between monkey L and monkey A. In monkey L (V4→V1), the EV for fast-moving large thick bars was larger than for the slow-moving thin bars (p < 0.001, permutation test; Figure 4D, right), whereas this difference was flipped in monkey A (p < 0.001; Figure Supplement 5F, right). This observation could be due to differences in the degree of reliability in the neuronal responses between monkeys (Figure Supplement 5H, L for monkeys L, A, respectively). We modeled EV as a function of SNR, self-consistency (split-half r), variance across time within stimulus, and variance across trials within timepoint, and used EV_resid = EV − ^ÊV for inference (Methods). In monkey L (V4→V1), the difference in EV residuals between thin and thick bars was not statistically significant (p = 0.44; Figure Supplement 5K, right), whereas this difference remained statistically in monkey A (p < 0.001; Figure Supplement 5O, right). Lastly, in the V1→V4 direction, EV residuals remained statistically higher when predicting the responses to the thin bar compared to the thick bar in both monkeys L and A (p < 0.001 for both monkeys; Figure Supplement 5K,O, left).

Neuronal activity could be predicted even during spontaneous activity

Given the dependence on the visual stimulus, we next asked whether it would be possible to predict neuronal responses in the absence of any visual stimulus, during “spontaneous activity”. We com-pared the predictability of stimulus-evoked activity in mice (drifting gratings and natural images) versus the predictability of activity recorded during gray screen presentations. This comparison was conducted in both visually (SNR >2 & split-half r >0.8) and non-visually (SNR <2 & split-half r <0.8) reliable neurons (n=3 mice; mouse MP027 did not undergo 30 min. of gray screen presentation). In visually reliable neurons, neuronal activity could still be predicted during the gray screen condition (p < 0.001, hierarchical paired permutation test compared with shuffled frames). There was a significant reduction in EV during gray screen compared to visual stimulus presentation (Figure 5A left, p < 0.001, hierarchical paired permutation test). In non-visually reliable neurons, predictability was higher during the gray screen condition compared to stimulus presentation (Figure 5A right, p < 0.001, hierarchical paired permutation test).

Figure 5.

While there was no correlation between neuronal predictability in the responses to visual stim-ulus presentations and in the response to gray screen presentations in visually reliable neurons (Figure 5B), there was a strong correlation for non-visually reliable neurons (Figure 5C). The difference in predictability in the absence of a stimulus (gray screen presentations) could in principle change according to the directionality in inter-laminar interactions. However, there was no statistically significant difference in the EV fraction between laminar directions (L4→L2/3 vs. L2/3→L4), in the same “visually reliable” group and even when subsampling “non-visual” neurons (Figure Supplement 3A).

In monkeys, we only focused on visually reliable sites since the majority of the neuronal population was visually reliable (Figure Supplement 1J). Additionally, an SNR of less than 2 (one of the requirements to define non-visual neurons in the mouse data) most likely reflects artifactual is-sues with the electrode recording the multiunit site (Chen et al., 2022). We compared inter-areal prediction of stimulus presentation activity (all monkeys for checkerboard presentation; monkeys L and A for moving bars), gray screen presentation (all monkeys), and during lights-off (monkeys L and D). The predictability of neuronal activity in response to gray screen presentation remained statistically above chance (p < 0.001 paired permutation test of prediction vs. shuffled frames pre-diction for all monkeys). The predictability of neuronal activity in the absence of visual stimulus was lower compared to checkerboard presentations (Figure 5D, p < 0.001 for monkey L; Figure Supplement 6E, p < 0.001 for monkey A; Figure Supplement 6F left, p < 0.01 for monkey D). The same conclusions held when comparing moving bar vs. gray screen presentations (Figure Supplement 6D for monkey L; Figure Supplement 6E for monkey A). Intriguingly, the EV fraction in the lights-off condition was statistically higher than during the stimulus presentations in both directions in monkey L, whereas the EV fraction in the lights-off condition in monkey D was lower (Figure Supplement 6F). Eye closure and sleep can induce global oscillations (Hohaia et al., 2022) and therefore may lead to correlated neuronal activity, causing an increase in predictability. Mon-key L spent a significant portion with its eyes closed (Chen et al., 2022), unlike monkey D, who spent almost the entire time with its eyes open (eyes closed duration < 3s in both recording days). To test the effects of eye closure, we further separated the lights-off neuronal activity of into periods where the eyes of monkey L were open or closed. The EV fraction was statistically significantly higher than EV fraction during stimulus presentation activity only during the eyes-closed period (Figure 5D).

The correlation in visual predictability between stimulus presentation and spontaneous activity was high across all types of spontaneous conditions (Figure 5E and Figure Supplement 6C for monkey L; Figure Supplement 6I, J for monkeys A, D, respectively). When assessing the inter-cortical prediction directionality during spontaneous conditions, in monkey L, we found the same asymmetrical relationship as in Figure 3, where V1→V4 EV fraction was significantly higher than V4→V1 prediction in both gray screen (p < 0.001, permutation test) and lights-off (p < 0.001, permutation test) conditions (Figure Supplement 3B). In monkey A gray screen presentation activity, we did not find any statistically significant difference between direction predictions (Figure Supplement 3C, right). However, when accounting for neuronal property differences between the areas, we found the same direction EV asymmetry across both monkeys (p < 0.001; Figure Supplement 3F for monkey L; Figure Supplement 3G for monkey A).

Behavioral metrics account for only a small portion of predictability

Because inter-areal predictability persists in the absence of visual drive, we asked whether behavioral signals could partly account for the inter-laminar and inter-cortical interactions. In mice, augmenting the neural predictability model with face–motion singular value decomposition (SVD) components and running speed did not significantly increase EV relative to the neural-only model in the L2/3 to L4 direction (p = 0.77, paired permutation test), and had a mild statistical increase in the L4 to L2/3 direction (p < 0.05, paired permutation test; Figure Supplement 10A). During spontaneous activity, the same type of augmentation did not increase EV relative to the neural-only model in either direction (p = 0.095 for L4→L2/3 direction, p = 0.93 for L2/3→L4 direction, paired permutation tests; Figure Supplement 10C). Behavior-only models captured a modest yet statistically significant fraction of variance (stimulus: EV=0.02 ± 0.03, spontaneous: EV=0.01 ± 0.01; p < 0.001 paired permutation test with shuffled frames). Behavior-only EV correlated with neu-ral EV within areas (stimulus: ∼ 0.47–0.50; spontaneous: ∼ 0.68–0.69; Figure Supplement 10B,D), indicating that animals/neurons with stronger neural predictability also tended to show stronger behavior-related structure, but in all cases behavior was not driving the main effect.

In monkey neural recordings during resting state condition (eyes open only), pupil diameter provided a small but significant predictive power (EV=0.01 ± 0.01; p < 0.001 paired permutation test of prediction vs. shuffled frames prediction). Adding pupil size to the neural model yielded no significant EV gain in either direction (p = 1 in the V1→V4 direction, p = 0.70 in the V4→V1 direction; Figure Supplement 10E).

Together, these results indicate that behavioral metrics contributed a measurable shared com-ponent to cross-area predictability, but they are not the principal driver: most explained variance remains tied to the neural population activity.

Receptive field overlap and neuronal response properties impact predictability

We investigated which neuronal properties are related to the ability to predict responses by comparing EV and key indicators of neuronal response reliability and receptive field properties, in both visually- and non-visually reliable neurons, during either visual presentation or spontaneous conditions. First, we considered the following properties: (i) max r² value (i.e., maximum squared correlation between each neuron in the predictor population and the predicted neuron), (ii) 1-vs-rest r² self-consistency (squared correlation between each neuron’s response to one trial repetition of all stimuli/timepoint with the average response of the rest of the trial repetitions; Methods), (iii) SNR of the predicted neuron, (iv) variance across stimuli, (V) variance across repeats of the same stimulus, (vii) variance across timepoints (monkey data only, since we have multiple timepoints per trial), and (viii) split-half r (Methods). We plotted EV against each of these variables (mouse: Figure 6B; monkey L: Figure 6E, monkey A: Figure Supplement 8B, monkey D: Figure Supplement 9B) and report the correlation coefficient between EV and each variable in the y-axis in mice (Figure 6A) and monkeys (monkey L: Figure 6D, monkey A: Figure Supplement 8A, monkey D: Figure Supplment 9A).

Figure 6.

In mice, during both stimulus presentation and gray screen presentation, the most correlated property with a neuron’s inter-areal predictability was the max r² (Figure 6A). For the other 5 properties, there was a strong distinction between stimulus presentation (dark bars) and gray screen presentation (light bars): all 5 properties were positively correlated with the neural activity predictability EV fraction during stimulus presentation, but they were slightly anticorrelated with their predictability EV fraction during gray screen presentation. Because the split-half correlation calculation averages out the non-stimulus-dependent variability in both halves of the trials, it showed a weaker correlation with EV, which depends on trial-by-trial modulation. The one-vs-rest r² metric, which also examines trial-by-trial modulation and does not average split-half trials, yielded a stronger correlation with EV during stimulus presentation.

When examining the relationship between 1-vs-rest self-consistency and inter-laminar prediction EV in mice, we observed a bimodal distribution of neurons: one group of neurons showed high EV despite having low self-consistency, and another group showed EV correlated with self-consistency (Figure 6B third column; Figure Supplement 7A; p < 0.001, Hartigan’s dip test after thresholding neurons with EV > 0.4). The responses of neurons with low self-consistency also showed high EV during gray screen presentation. This bimodality was present in two of the three mice (p < 0.001 for mouse MP031, MP032, p = 0.69 for mouse MP033, Hartigan’s dip test; self-consistency and EV fraction relationships across all mice can be seen in Figure Supplement 7B). To better understand the responses of neurons with low self-consistency, we projected out the “non-visual ongoing neuronal activity” from the neuronal responses (Stringer et al., 2019a) (Methods). This non-visual ongoing activity is deemed to be influenced by spontaneous behavior (Stringer et al., 2019b). Projecting out this non-visual activity eliminated this bimodality (p = 0.71 and p = 0.98 for L4→L2/3 and L2/3→L4, respectively; Hartigan’s dip test, Figure 6C,D). This observation could be because the responses of neurons with low self-consistency can no longer be predicted, or be-cause the responses of those neurons became more reliable and therefore were highly predicted. To distinguish between these two possibilities, we compared both the 1-vs-rest self-consistency and the prediction EV before and after removing the non-visual activity. Removing the non-visual ongoing activity increased the self-consistency value across the three mice (Figure Supplement 7C; p < 0.001, paired permutation test). Interestingly, the inter-laminar EV fraction decreased in MP031 and MP032 mice, yet increased in MP033 (Figure Supplement 7Fp < 0.001, paired permutation test). When examining individual pairwise relationships in a fraction of highly predictable neurons (lines in Figure Supplement 7F), we found that some of the highly predictable neurons remained predictable after removing the non-visual activity, whereas other highly predictable neurons dropped their EV fraction dramatically. Together with the small yet reliable shared behavior contributions (Figure Supplement 10), these results indicate that a non-trivial portion of inter-areal predictability in mice reflects shared internal-state fluctuations, whereas a complementary portion persists after removing those dimensions, consistent with stimulus-driven interactions.

In monkeys, one of the neuronal properties that showed high correlation with inter-areal pre-diction EV across all conditions was also the maximum correlation value (first column in Figure 6D, E, Figure Supplement 8A, and Figure Supplement 9). Other neuron properties like SNR, split-half correlation, and one-vs-rest correlation were also highly correlated with inter-cortical predictability EV(Figure 6D, Figure Supplement 8A, and Figure Supplement 9A). The split-half correlation was highly correlated with EV fraction, although the relationship was highly non-linear (middle column in Figure 6D, Figure Supplement 8B, and Figure Supplement 9B). Using the one-vs-rest squared correlation removed some of this non-linearity and further increased the correlation with the EV fraction in monkeys L and A (third column in Figure 6D, Figure Supplement 8B). In addition, the relationship between one-vs-rest correlation squared and EV fraction did not show evidence of bimodality (p = 0.99, Hartigan’s dip test focusing on sites with EV > 0.4).

We conjectured that neurons that have overlapping receptive fields (RFs) should share more information, and therefore their responses would better predict each other than neurons with non-overlapping RFs. In addition, even when all neurons are exposed to the same stimulus (full field symmetrical checkerboard image, gray screen, darkness, etc), neurons with overlapping RFs may be more synaptically connected, resulting in better inter-cortical predictions. To test this hypothesis, we compared inter-cortical predictions in different ensembles of neurons with RFs that differed in their degree of overlap. This hypothesis was only tested in monkeys L and A because we did not have access to RF estimates in the mouse data, and we did not have enough sites with RF overlap in monkey D. For each V4 site whose responses we predicted, we separated the predictors into two groups with matched size: one group where all the V1 predictor sites had <10% RF overlap (sample of one V4 site in monkey L, Figure 6F, top), and one group where all the V1 predictor sites had >80% RF overlap (sample of one monkey L V4 site, Figure 6F, bottom). A similar procedure was followed when predicting the activity of V1 neurons from V4 predictor neurons (Figure 6H). Inter-areal prediction was higher in the >80% RF overlap condition compared to the <10% RF overlap ensembles in both directions and across all stimulus conditions (Figure 6G, I for monkey L; Figure Supplement 8C, D for monkey A). To compare these numbers with those obtained with larger numbers of sites, we calculated the EV performance while using all predictors. In most cases, predictions using the >80% RF overlap group were lower than when using all predictors, showing that predictor neurons with low RF overlap still contributed to the EV (Figure 6G,I, for monkey L; Figure Supplement 8C,D for monkey A).

Inter-areal predictability is both stimulus and non-stimulus driven

The results in Figure 5 and Figure 6 pointed to components of the predictable responses that are stimulus-driven and other components that are not stimulus-driven. We considered two alternative hypotheses: (1) If shared information were strictly stimulus-driven, prediction across repeats of the same stimulus should remain accurate; (2) If shared information were entirely stimulus-independent, prediction across repeats should approach chance (EV ≈ 0). To test these hypotheses, we compared the inter-areal prediction EV fractions using unshuffled versus shuffled trials. Shuffling was performed across repeat trials of the same images (mice: Figure 7A, monkeys: Figure 7D). In mice, one stimulus presentation was either a drifting grating or a natural static image. In monkeys, one stimulus presentation was either the one checkerboard image, a large, thick, fast-moving bar, or a small, thin, slow-moving bar. In both species and in both directions, inter-areal prediction EV fraction decreased after shuffling stimulus repeats compared to before shuffling (Figure 7B for mouse; Figure 7E for monkey L; Figure Supplement 11A for monkey A; Figure Supplement 11C for monkey D; p < 0.001 for both species and directions, paired permutation test).

Figure 7.

Interestingly, the EV fraction during shuffled trial-repeats still remained significantly above chance (p < 0.001, paired permutation test of prediction vs. shuffled frames prediction). In mice, neurons showed two types of distribution in terms of their response predictability in shuffled and unshuffled trials. For a subset of neurons, the EV fraction was still high in the shuffled condition, albeit their EV was lower than in the unshuffled case (Figure 7C; points below but near the diagonal line). For another subset of neurons, the EV fraction during shuffled trials was much lower or even near zero. The responses of the latter group had the highest predictability during gray screen activity.

In monkeys, neurons farther away from the diagonal line showed higher EV fraction during gray screen activity (Figure 7F for monkey L; Figure Supplement 11F for monkey A; Figure Supplement 11G for monkey D). In addition, we examined whether shuffling repeat presentations of gray screen images (simulating spontaneous activity) would result in any prediction at all. There was a more profound decrease in inter-cortical performance in all monkeys (Figure Supplement 12A–C) with no neurons that remained as predictable during shuffled trials compared to unshuffled trials (Figure Supplement 12E–G).

Accounting for latency differences improves inter-areal activity predictions in monkeys visual area sub-populations

Given the latency differences in neuronal responses between V1 and V4 Schmolesky et al., 1998, we asked whether accounting for this latency could result in better inter-cortical prediction. To test this hypothesis, we offset the neuronal activity using different lags for each area (Figure 7G, H) and recalculated the ridge regression predictions. For each offset level, we calculated the percentage of neurons where the EV fraction peaked at that offset. For the checkerboard image, in monkeys L and A V1→V4 predictions, the biggest percentage of neurons had a peak performance when there was no time offset between areas, with a substantial proportion of neurons with a peak performance for 10–30 ms offsets in the negative direction (i.e., V1 activity preceding V4 activity; Figure 7I, left for monkey L, Figure Supplement 11E, left for monkey A). This distribution of peak EV values was only present during early visual responses (first 275 ms of stimulus onset). In monkeys L and A V4→V1 direction, there was a large proportion of neurons with peak EV when considering 10–20 ms offsets in the positive direction (i.e., V4 after V1, Figure 7I, right for monkey L, Figure Supplement 11E, right for monkey A). These differences were apparent in the early part of the visual response, before 250 ms. When offsetting the neuronal responses to gray screen presentations, across all times and areas, the highest percentage of neurons with peak EV was when there was no time offset (Figure Supplement 12D).

Neural signals can also be predicted at the ensemble level using local field potential signals

We repeated the analyses in monkeys L and A using the local field potential (LFP) signals. We evaluated whether the predictability of LFP signals across areas depended on the signal frequency by considering band-limited LFP amplitudes (Low: 2–12 Hz; Beta: 12–30 Hz; Gamma: 30–45 Hz; High-gamma: 55–95 Hz; Hilbert envelopes, z-scored). In both monkeys, the Gamma band showed a feed-forward signature in the early evoked period: the V1→V4 predictability peaked at negative offsets (∼10–30 ms; V1 leads), and the V4→V1 predictability peaked at positive offsets of similar magnitude Figure Supplement 13A,D for monkey L; Figure Supplement 13B,E for monkey A), consistent with previous findings (Semedo et al., 2022; van Kerkoerle et al., 2014; Bastos et al., 2015). The Low and Beta frequency bands exhibited a broader structure with less consistent peaks in predictability offset timing.

Discussion

Neuronal activity in one brain region or layer within the visual cortex can be used to predict neuronal activity in another nearby and anatomically connected region or layer in single trials (Figure 2). In monkeys, predictability was asymmetric: V1 activity better accounted for V4 activity than vice versa (Figure 3, Figure 7). This inter-areal prediction persisted across different stimuli (Figure 4) but also in the absence of a visual stimulus, during gray-screen and lights-off periods (Figure 5). The degree of predictability increased with signal-to-noise ratio, response variance, and the degree of overlap between receptive fields (Figure 6).

In line with other studies in mice (Stringer et al., 2019b; Niell and Stryker, 2008; Andermann et al., 2011), we observed an approximately bimodal distribution of neuronal responses, with a large subset of neurons that do not show reliable responses to visual stimuli both in L4 and L2/3 (Figure 6B, third column; Figure Supplement 7A). However, even for the subset of neurons considered to be “non-visual”, at least within the set of stimuli and conditions examined here, their activity remains predictable. This bimodal distribution disappears when projecting out potential non-sensory ongoing activity (Figure 6C; Figure Supplement 7D) (Stringer et al., 2019b, 2021). At the population level, neuronal encoding subspaces in mouse visual cortex have been shown to have little overlap between visual sensory and non-sensory (behavioral) information, with only one shared dimension (Stringer et al., 2019b). The visually unreliable, yet highly predictable, sub-set of neurons described here could be the neuronal group driving this orthogonality. As expected, the activity of “visual” neurons can be better predicted during visual presentation and not during gray screen presentation. In stark contrast, the activity of non-visual neurons can be predicted even better during gray screen presentation than during visual stimulation. There was no such bimodal distribution in the data from monkeys. One possibility is that there may be no (or very few) non-visual neurons in monkey V1 or V4. Indeed, the overwhelming majority of recording sites in V1 and V4 responded strongly to visual stimulation. Yet, the comparisons between the results in mice and monkeys reported here need to be interpreted with caution because the two datasets differ in terms of recording techniques (electrophysiology versus two-photon imaging), consequently also in their temporal resolution (one millisecond versus hundreds of milliseconds), and the type of interaction studied (across areas versus across layers), in addition to any differences between species.

In monkeys, sites where the receptive fields (RF) of V1 and V4 overlap can better predict each other compared to other sites showing little RF overlap. This observation could reflect RF-dependent intrinsic connectivity between areas, but also RF-dependent shared inputs from other areas, such as the thalamus. In the latter case, those putative shared inputs cannot be strictly dependent on visual inputs, given that the effect of RF overllap persists even during gray screen conditions.

Many computational models that aim to explain neuronal activity in the visual cortex are based on feedforward signal propagation, with increased receptive field sizes, selectivity, and feature invariance along the visual hierarchy (Serre et al., 2007b; Kreiman, 2021; Connor et al., 2007). Consistent with this idea, we described an asymmetry in the degree of predictability, with V1 neurons explaining V4 responses better than the other way around. This observation persisted after control-ling for neuronal count and split-half correlation values and was also apparent during the lights-off condition. In contrast, there was no asymmetry when comparing inter-laminar prediction directions in mice. The lack of asymmetry in inter-laminar prediction directions in mice could be due to the slow dynamics in calcium imaging, the lack of a clear inter-areal hierarchy, or differences between species.

The asymmetry in directionality was also observed when implementing temporal delays to inter-areal prediction, consistent with processing delays across areas (Semedo et al., 2022; Gokcen et al., 2022; Schmolesky et al., 1998). A substantial proportion of neurons increased their inter-areal predictability when offsetting the times between areas, specifically in the direction that aligns their neuronal activities. In contrast to the temporal delays associated with processing visual stimulation, during gray screen presentation, the activity of most neurons was best predicted in the absence of any time offsets, suggesting that the internally generated neuronal activity during spontaneous conditions may be largely driven in a non-feedforward manner. Further evidence supporting the distinction between visually-driven and non-visually-driven interactions comes from the observation that trial repeat shuffling reduced, but did not eliminate, predictability in both mice and monkeys. In mice, when comparing shuffled versus unshuffled activity, we encountered again a bimodal distribution, where a group of neurons was closer to the diagonal line (their responses were predicted as well during the shuffled compared to the non-shuffled condition), and another group of neurons that were closer to the x-axis (their responses could not be predicted during the shuffled condition). The responses of the latter group were best predicted during gray screen activity, suggesting that they mostly shared non-visual information. The predictive power in mouse V1 from layer 4 to layer 2/3 during spontaneous conditions has been recently shown in Papadopouli et al. (2024), consistent with our findings. The overall area population decrease in predictability after shuffling may be due to the influence of non-visual activity, such as movement (Stringer et al., 2019b), especially since these non-visual stimulus effects have been shown to occur on a timescale of approximately one second, consistent with the results in our study. In macaques, context-dependent effects are likely not due to movement, since the monkeys maintained fixation during the stimulus task, and visually-evoked activity was not driven by movement (Talluri et al., 2023).

The neural recordings in mice and monkeys are different in terms of: (i) recording modalities (calcium versus electrophysiology), (ii) temporal resolution (hundreds of milliseconds versus milliseconds), (iii) neuronal types, (iv) SNR, (v) cortical targets (layers versus areas), (vi) sample sizes, (vii) stimuli, and (viii) task conditions (Stringer, 2018; Chen et al., 2022). The goal of this study was not to make any direct quantitative comparison across species, but rather to investigate the properties of inter-areal interactions within each species.

Several limitations in this study are worth considering in future work. First, neither calcium imaging nor MUAe recordings capture the timing of single spikes in individual neurons. One of many ways in which the timing of individual spikes could be important is that synchronous or close-to-synchronous inputs can have a larger impact on post-synaptic responses (Salinas and Sejnowski, 2001). Second, this study focused on linear predictability, but other non-linear models could better capture neuronal activity. Third, and critically, the experimental data evaluated here provide only a fraction of the inputs to a given neuron. For example, the predictor signals studied here are likely to exclude most (in monkeys) if not all (in mice) inhibitory inputs (Jiang et al., 2015; Shen et al., 2020; Ibrahim et al., 2020; Schuman et al., 2021). Any cortical neuron receives extensive local inputs and non-local inputs from many other areas. Fourth, we only had limited access to non-visual sources that could contribute to predicting neuronal responses; other sources of input could include finer behavioral measurements, attention, familiarity, and arousal state. Finally, biophysically realistic models of the transformation between inputs and outputs of a given neuron should include their dendritic locations and specific synaptic potentials (Park et al., 2019; Petousakis et al., 2023). Therefore, the quantitative results on the prediction of neuronal responses reported here constitute a lower bound.

We evaluated inter-areal interactions in different types of neuronal recordings, timescales, and species. These interactions were assessed in single trials, separating visually-driven and non-visual contributions, and accounting for the directionality and dynamics of neuronal responses. These efforts constitute an initial step towards, and a lower bound for, systematically building computational models that can account for the transformations from sensory inputs to their encoding across multiple processing steps in the cortex. Whereas many efforts with artificial neural networks merely map inputs to outputs (Kreiman, 2021), a detailed and systematic understanding of individual processing steps can lead to better and biologically more relevant computational models of sensory processing.

Methods

Datasets

We used the mouse dataset from (Stringer et al., 2019a) containing calcium-imaging activity measurements from 43,630 neurons in layer 4 (L4) and 12,060 neurons in layers 2/3 (L2/3) in V1 of 4 mice during 32 different randomly interleaved presentations of either drifting gratings or gray-scale natural images (each one repeated more than 90 times), along with spontaneous activity during 30 minutes of exposure to a gray/black screen (Figure 1A, data acquisition details in Stringer et al. (2019a,b)). Calcium imaging activity was recorded during stimulus presentations at a scan rate of 2.5 Hz or 3 Hz (each frame was acquired every 400 ms or 333 ms). The computed stimulus responses per stimulus presentation were averaged based on two frames immediately post stimulus onset. Cortical layers were determined using the 10-12 planar z-positions retrieved during the multi-plane calcium activity acquisition. For stimulus-response and spontaneous recordings, neuronal activity of each neuron was z-scored using its 30-minute gray screen spontaneous activity (mean gray-screen activity subtracted and divided by gray-screen activity standard deviation).

We used the dataset from Chen et al. (2022) for monkeys A and L and conducted new neural recordings from one additional monkey (monkey D). The Chen et al. dataset consists of envelope multiunit activity (MUAe) and LFP from 1,024 recording sites in one monkey (monkey L) in response to either multiple-day recordings of more than 60 repetitions of a full-size checkerboard image, moving small-thin or large-thick bars in 4 directions, gray screen presentations, or more than 30 minutes of baseline activity where the monkey was in a room with the lights off (Figure 1B). We also used data from a second monkey (monkey A) from the same published dataset. However, due to the degraded state of the electrodes at the time of the recording, we were only able to obtain enough data from V4 sites for responses to the full-size checkerboard image, moving bars, and gray screen presentations. Neuronal activity was summed over 25 ms non-overlapping bins. Activity duration was 300 ms, 400 ms, and 1 s for gray screen, checkerboard, and moving bar presentations, respectively. For the recordings during visual stimulation, the neuronal activity was normalized by subtracting the mean activity during the gray screen presentations separately for each site.

We collected new data from an additional monkey in response to the full-size checkerboard image presentations, gray screen presentations, and lights-off activity (∼15 minutes). Monkey D (12 years old, 14.4 kg) was chronically implanted with floating multielectrode arrays (Microprobes for Life Sciences, MD) targeting areas including V1 and V4. Arrays consisted of 16 channels per area, yielding a total of 32 electrodes (V1–16, V4–16). All procedures received ethical approval by Harvard Medical School Institutional Animal Care and Use Committees and conformed to NIH guidelines provided in the Guide for the Care and Use of Laboratory Animals. All relevant ethical regulations for animal and nonhuman primate testing and research were followed.

The same conditions of stimulus presentation (image/square size, presentation time, etc) were presented as in Chen et al. (2022). Stimulus presentation and data acquisition (including electro-physiology and eye tracking) were coordinated using MonkeyLogic2 software (Hwang et al., 2019) and OmniPlex Neural Recording Data Acquisition Systems (Plexon Inc.), integrated via custom MAT-LAB scripts. Online spike sorting was performed with the PlexControl client using waveform-based classification. Visual stimuli were displayed on ViewPixx EEG monitors (ViewPixx Technologies; 1920×1080, 120 Hz), and eye position was monitored via ISCAN (ISCAN Inc.). Raw analog signals were used to calculate MUAe using same processing logic as Chen et al. (2022).

Visual Reliability

A neuron or site was defined to be visually reliable if its signal-to-noise ratio (SNR) was 2 or higher and its split-half correlation value was 0.8 or higher. Due to the high number of repetitions of visual stimuli, the split-half correlation was skewed toward high values, which is why we used a higher split-half correlation threshold than commonly used in other studies. In mice, the SNR for each neuron was calculated as:

where <> denotes mean, std denotes the standard deviation, r_stim is the average activity in response to stimuli, and r_spont indicates the average activity over the 30-minute gray screen presentation.

In monkeys, we followed the definition in Chen et al. (2022) and calculated the SNR for each site as:

using the peak activity during the checkerboard presentation for the signal, and the average gray screen neuronal activity as background (denoted as r_spont).

In mice, the split-half consistency was calculated by correlating the average activity for the 32 stimuli in half of the trials, randomly chosen, with the average activity in the other half of the trials, followed by Spearman-Brown correction (used to correct for the division of trials by half). In monkeys, during checkerboard presentations, the split-half consistency was calculated by correlating the average activity of the 16 timepoints (0–400 ms; 25 ms bins) of checkerboard presentation of 25 random trial repetitions with the average activity of another non-overlapping 25 random repetitions, followed by Spearman-Brown correction. During moving bar presentations, the 40 time-points (0–1s; 25 ms bins) during 25 random trial repetitions were first concatenated across the 4 directions (total of 160 timepoints), and then correlated to the concatenated averaged activity of another nonoverlapping 25 random trial repetitions, followed by Spearman-Brown correction. For all split-half consistency calculations, we randomly sampled trials 20 times.

Inter-areal predictions

Let _Ar_i,t be the activity of neuron or site i in area A at timepoint t, where A can be L4 or L2/3 in the mouse data and V1 or V4 in the monkey data. Neuronal activity from one area (e.g., mouse V1 L4 or monkey V1) was used to predict activity in the other area (e.g., mouse V1 L2/3 or monkey V4) using ridge regression (Figure 1E). The activity of each neuron i in area A2 was predicted from n_A1 neurons in area A1 as follows:

During fitting, we minimized the residual sum of squares (RSS), defined as:

where w is the weight vector for predicting the activity of neuron i, n_T is the number of images/time points and α controls the regularization strength (α was tuned for each dataset with an independent sample and ranged from 10³ to 10⁵). Predictability for each neuron was evaluated using 10-fold cross-validation across trials and quantified as squared Pearson’s r between held-out _A2r_i,t and _A2r^_i,t, referred to as explained variance fraction (EV fraction) throughout.

To remove temporal auto-correlation that would inflate the apparent prediction despite cross-validation, we removed training timepoints near the test timepoints closer than the decay window of the activity auto-correlation (mouse: 5 s; monkey: 1.5 s). The auto-correlation decay window was determined using time-series forecasting Ridge Regression (using r_t to predict r_t+d, where d represents a delay). The delay was increased until the EV fraction approached chance.

To model a Poisson distribution of MUAe responses, we also fit a Poisson generalized linear model (GLM) using the same predictors as above.

We implemented this model with scikit-learn’s PoissonRegressor (log link; intercept b_i unpenalized) and tuned α per dataset using an independent sample. Model evaluation followed the same 10-fold cross-validation and temporal exclusion procedure as ridge regression (1.5 s exclusion window for monkey). Predictability was quantified as the squared Pearson correlation between observed and predicted MUAe on held-out folds (EV fraction), allowing direct comparison between ridge regression and Poisson GLM (Figure Supplement 1M).

Prediction directionality

We compared predictability across layers in different directions (in mice: L4→L2/3 vs. L2/3→L4) and also predictability across areas in different directions (in mon-keys: V1→V4 vs. V4→V1) (Figure 3). To ensure that results were not dependent on the number of neurons/sites, we randomly subsampled the number of neurons/sites of the area containing the larger number of neurons/sites (L2/3 for mouse; V1 for monkeys) to match the number of predictors in both directions (10 iterations per recording type, neuron count details in Table 1 and Table 2. Results in Figure 3 represent the median across the 10 subsamples). To account for potential changes in intrinsic predictability, we ensured that the neurons from both areas were matched in terms of the distribution of split-half correlation values so that the difference between individual area neurons/sites was less than 0.002. To assess the intrinsic predictability of neurons/sites in each region, the areas were used to predict themselves, where one neuron/site in the area was predicted by the remaining neurons in the same area. This “intra-areal” prediction was used to normalize EV fraction to compare directionality of prediction.

Stimulus types and spontaneous activity comparison

We compared predictability for different stimulus conditions (Figure 4, Figure 5). To compare inter-areal prediction across stimulus types and between the presence or absence of stimuli, the number of predictors (neurons or sites) and timepoints was sub-sampled to be the same across all datasets. In monkeys, the time spent record-ing the lights-off condition was much greater than during stimulus or gray screen presentations. To account for the difference in time duration and therefore training size, we subsampled time periods to be the same across all stimulus, gray-screen, and lights-off, lights-off eyes open, and lights-off eyes closed conditions.

Neuron properties

We compared different neuronal properties with predictability measurements (Figure 6). The SNR and split-half correlation has been defined above. The absolute max pairwise correlation value of each neuron/site i in one area with all neurons in the other area was calculated as

where _A2r_i represents the activity of neuron/site i in area A2, which are correlated with the activity of every neuron j in area A1 (denoted as _A1r_j).

One–vs–rest self-consistency (1-vs-rest r²). For each neuron/site i with T repeat trials, let r^(t) ∈ ℝ^M be its activity vector from repetition t across all stimuli/timepoints (length M, defined below). The leave-one-out mean response over repetitions excluding t is

Throughout, the superscript (rest) means “all repetitions excluding repeat t”.

The one–vs–rest correlation for the held-out repetition t is

We then average these correlations across all held-out repetitions:

In mice, r^(t) stacks the responses to the S = 32 stimuli for repetition t (each element is the mean of the two frames immediately post-onset). In monkeys, during checkerboard presentations, r^(t) stacks the T = 16 timepoints (0–400 ms; 25 ms bins). In monkey moving-bar presentations, r^(t) concatenates the T = 160 timepoints (4 directions × 40 bins; 0–1 s, 25 ms bins).

Variance metrics. All variances were computed on gray–baseline–subtracted signals. In mice (32 gratings, K repeats), for neuron i, let r^(k,s) denote the scalar response to stimulus s ∈ {1, …, S} on repeat k ∈ {1, …, K} (repeat-averaged within the analysis window). We defined:

In monkeys, for the checkerboard stimuli, for site i let r^(k,t) denote the scalar response to time-point t ∈ {1, …, T } on repeat k ∈ {1, …, K}

In monkeys, for the moving bar stimuli, the same two metrics were computed per stimulus s and then averaged across the four stimuli. In addition, we defined an across-stimuli variance of the time-averaged, trial-mean response:

These variance covariates (together with SNR, split-half self-consistency, and one-vs-rest r², where applicable) were included as regressors in the within–dataset-type linear models used to obtain residual explained variance, EV_resid = EV − ^ÊV.

EV residualization

For each direction (V1→V4, V4→V1) and stimulus, we fit an ordinary least squares (OLS) model:

where EV^{^}_i is the predicted explained variance of each neuron/site i. Predictors were z-scored within each direction or stimulus. SNR_i is its signal-to-noise ratio (defined in the Visual Reliability section), split-half r_i is its split-half consistency (also defined above), Var_time,i and Var_trials,i are the temporal and across-trial variance metrics (defined in the Variance metrics section), β₀, …, β₄ are re-gression coefficients, and ε_i is the residual error term. Predictors were z-scored within each direc-tion or stimulus. Residual EV was defined as EV_resid,i = EV_i − EV^{^} _i and used for all post-adjustment comparisons (Figure Supplement 3E–H; Figure Supplement 5K,O).

Dip test of unimodality

We used Hartigan’s dip test to evaluate whether the distribution of 1-vs-rest squared correlation values was unimodal or not (Figure Supplement 7A,D). The test was per-formed after restricting the analysis to highly predictable neurons/sites (EV > 0.4). The dip statistic measures the maximum deviation between the empirical cumulative distribution function and the closest unimodal distribution (Hartigan and Hartigan, 1985). Reported p-values correspond to the probability of obtaining a dip statistic at least as large under the null hypothesis of unimodality. Dip tests were computed separately for each cortical layer and mouse (Python package diptest).

Receptive field overlap comparisons

In monkeys L and A, receptive field (RF) ellipses were calculated using center and edge locations in the dataset (Figure 6E, F). To calculate the percentage of RF overlap between the neuronal sites to be predicted and the predictor, the intersection area between ellipses was retrieved using the Shapely Python package, and divided by the area of the predicted site. Sites that had predictors that overlap both more than 80% and less than 10% were selected to compare inter-areal predictions. To control for predictor size, 14 random predictor sites from all the sites in each overlap type were subsampled (10 random samples without replacement).

Trial repeat shuffling and time offset predictions

For the shuffled-trial experiments, we shuffled the predictor activity across repeat trials showing the same stimulus (Figure 7). Thus, the stimulus order remained the same. For the mouse time-offset analysis, the activity of predictor neurons was time-shifted in the positive or negative direction, with 1 bin corresponding to 1 stim-ulus presentation (800–900ms). For the monkey dataset, the predictor activity (400 or 1000 ms per presentation, 16–50 bins of 10 ms each) was offset across time bins. We used sub-windows of 200 ms to avoid window-length differences that would otherwise be introduced if we shifted the entire trial response.

Supplemental Material

Figure Supplement 1.

Figure Supplement 2.

Figure Supplement 3.

Figure Supplement 4.

Figure Supplement 5.

Figure Supplement 6.

Figure Supplement 7.

Figure Supplement 8.

Figure Supplement 9.

Figure Supplement 10.

Figure Supplement 11.

Figure Supplement 12.

Figure Supplement 13.

Data availability

All the computational models and code for data analyses are publicly available at https://github.com/4sdch/Trial-by-trial-inter-areal-interactions-in-visual-cortex. All the data are publicly available (mice: https://figshare.com/articles/dataset/Recordings_of_ten_thousand_neurons_in_visual_cortex_in_response_to_2_800_natural_images/6845348?file=12462734, monkeys L and A: https://gin.g-node.org/NIN/V1_V4_1024_electrode_resting_state_data and monkey D: https://osf.io/9gz6f.

Acknowledgements

The authors would like to thank Elisa Pavarino, Leonardo Polina, Sara Djambazovksa, Jan Drugowitsch, Wei-Chung Allen Lee, Morgan Talbot, for providing comments on the manuscript.

Additional information

Funding

This work was supported by NIH grant R01EY026025 (GK) and NSF grant CCF-1231216 (GK). This work was partially supported by the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 101007926, and RO1 award: NINDS R01 NS113890 to SS

Funding

NIH (R01EY026025)

• Gabriel Kreiman

NSF (CCF-1231216)

• Gabriel Kreiman

NINDS (R01NS113890)

• Stelios Smirnakis