Predicting trial-to-trial and timepoint-to-timepoint neuronal activity between areas.

A. Top: Experimental set-up to record two-photon Calcium imaging activity data from layers 2/3 (L2/3) and layer 4 (L4) in rodent V1 upon presentation of gratings, natural stimuli or gray screen images (represented as imgn) (Stringer et al., 2019a). Deconvolved calcium imaging traces were z-scored using baseline activity during 30 minutes of gray screen presentation before/after image presentation (Table 1). Bottom: Sample z-scored neuronal activity from 3 different neurons in response to 100 presentations of drifting gratings (left) or gray screen presentations (right). Each activity value corresponds to one image presentation, and calculated as the average of two calcium imaging video frames (666 ms or 800 ms, see details in Methods). B. Top: Experimental set-up for the neuronal activity data from macaque V1 and V4 (Chen et al., 2022). Electrophysiological activity was simultaneously recorded across 1,024 channels from 16 Utah arrays (Table 2). Bottom: Envelope multiunit spiking activity (MUAe) from 3 different sites in response to multiple presentations of a repeated 400 ms full-field checkerboard image (left, baseline mean-subtracted), 200 ms gray screen (middle), or during a lights-off condition (30 minutes total; right). Each value corresponds to aggregated MUAe activity in a 25 ms bin. C. Overview of inter-laminar relationships examined in mouse V1. “lower-level” layer 4 (L4) neuronal activity is used to predict “higher level” layer 2/3 (L2/3) activity (blue arrow) and vice versa (red arrow). D. Overview of inter-cortical relationships examined in macaque, where lower-level V1 is used to predict higher-level V4 (blue arrow) and vice versa (red arrow). E. Illustration of linear ridge regression method used for inter-areal prediction. Neuronal activity in response to presentation number i (labeled “ri”) at time t from one area (e.g., mouse V1 L2/3 or macaque V1) was used to predict activity in the other area (e.g., mouse V1 L4 or macaque V4) (Semedo et al., 2019). Predictability was evaluated using 10-fold cross-validation across presentation trials in mouse, and across 25 ms timepoints in macaque (Methods).

Mouse neuron counts used for inter-layer prediction and analyses.

A total of 7 recordings were used to perform prediction experiments. Each row corresponds to a recording day, containing the dataset recording type (Mouse Dataset), total number of neurons, and visually responsive neurons (see Methods). Fourth column: In the directionality prediction experiments, the area containing more neurons (L2/3) was further subsampled to match the number of L4 neurons. The dataset recording type names contain either “ori32” or “natimg32”, in addition to the mouse name (MP0-). “natimg32” represents dataset of the 32 natural image presentation. “ori32” represents dataset of the 32 drifting gratings.

Monkey site counts used for inter-cortical prediction and analyses.

Dates 090817,100817,250717 correspond to neuronal activity in response to checkerboard presentations, gray screen presentations, and lights off condition. Date 260617 corresponds to small thin moving bars presentation. Date 280617 corresponds to large thick moving bars presentation. Fourth column: In the directionality prediction experiments, the area containing more sites (V1) was further subsampled to match the number of V4 sites.

Lower level activity can predict higher level activity in both rodent and primate brains.

A. Example neuronal activity (z-scored, black) in response to stimulus presentations (drifting gratings) in mouse V1 L2/3 along with regression-model predictions (red) for a typical cell (2, middle), cell in the top 10% percentile of predictability (1, top), and bottom 10% percentile (3, bottom). B. Same as A for macaque MUAe activity in response to a full-field checkerboard image in three V4 neuronal sites. C. Predicted neuronal activity versus actual neuronal activity in response to stimuli for the mouse L2/3 cells 1, 2, and 3 shown in A. Each point represents 800 ms corresponding to a stimulus presentation. r values (top left) indicate the correlation coefficient. D. Same as C for macaque V4 neuronal sites 1, 2, and 3. Each point represents one 25ms timepoint during the 400 ms presentation. E. Distribution of EV fraction in L4→L2/3 regressions of neural activity in response to visual stimuli in cells that were deemed visually responsive in 4 mice and 7 recording days (n = 7, 265 neurons, Methods). Performance using 10-fold cross-validation across trials was quantified as squared Pearson’s r, referred to as explained variance (EV) fraction. The three vertical lines show the 3 examples in part A, C. The blue solid shaded rectangle (here and throughout) represents the interquartile range (IQR) shuffle control performance, where the activity timepoints of one area were randomly shuffled. F. Distribution of EV fraction in V1→V4 regressions of neural activity in response to vistual stimuli in sites deemed visually responsive (One macaque, 5 recording days, 68–82 V4 sites recorded per day; n = 376 total site recordings).

Asymmetry in inter-cortical predictability in macaque but not inter-laminar predictability in mouse.

A. Split-half reliability (Methods) for the n = 298 neurons (per area) in mouse MP033 drifting gratings presentation recording of V1 L2/3 (green) and L4 (coral) used to perform directionality comparisons. Neurons were randomly sub-sampled to match the numbers and self-reliability in the two distributions. Here and throughout, asterisks indicate statistically significant differences using a hierarchical independent permutation test (10,000 permutations): * p < 0.05, ** p < 0.01, *** p < 0.001; “n.s.” indicates p > 0.05. B. Violin plots describing the distribution of EV fraction for L4→L2/3 (green) and L2/3→L4 (coral) predictions across all 7 stimulus recordings (n = 1, 113 neurons per area). Violin plots (here and throughout) represent the distribution of neuron/site values, with width representing density and inner boxplot representing the interquartile range. Whiskers of innerbox represent range of the data. C. Split-half reliability for the n = 74 sites (per area) in macaque checkerboard recording (date=090817) of V4 (green) and V1 (coral) used to perform directionality comparisons. D. Violin plots describing the distribution of EV fraction for V1→V4 (green) and V4→V1 (coral) across all 5 stimulus recordings (n = 786 sites recordings per area).

Stimulus type influences neuronal predictability.

A. Illustration of the two types of stimuli (drifting gratings and static natural images) presented to the mouse during calcium imaging. B. Across-layer predictability in mouse V1 for each stimulus type (dark: drifting gratings, light: natural images) and prediction direction. C. Illustration of the three types of stimuli presented to the monkeys (Chen et al., 2022). The slow-moving small thin bar moved near the fixation point for 1 s in each of the four directions, while the fast moving large thick bar moved towards the edges of the screen for 1 s in each of the four directions. The full-field checkerboard image was presented repeatedly (400 ms each presentation). D. Across-area predictability for each stimulus type (dark: slow bars, medium: fast bars, light: checkerboard) and direction.

Spontaneous activity can also be predicted.

A. EV fraction of neuronal activity in response stimulus presentation (dark violins) or gray screen presentation (light violins) for neurons deemed visually (left) or non-visually (right) responsive (Methods). B. Correlation between EV in responses to gray screen (y-axis) versus stimulus presentation (x-axis) in mouse V1 visually responsive neurons (L4→L2/3:left, green; L2/3→L4: right, coral). Diagonal line represents the line of equality (y=x). r is the Pearson’s r coefficient. C. Same as B but for non-visually responsive neurons. D. EV during stimulus presentations (checkerboard image, green), gray screen presentations (light green) or during lights off (dark green). The lights-off condition is further separated into periods when the eyes were open or closed. All lights-off conditions were sub-sampled (10 permutations) to contain similar training lengths as the stimulus and gray screen presentation recordings. E. Correlation between EV in responses to gray screen (y-axis) versus stimulus presentation (x-axis) in macaque visually responsive neurons (V1→V4:left, green; V4→V1: right, coral). Diagonal line represents the line of equality (y=x). r is the Pearson’s r coefficient. All recorded sites were pulled from the 3 recording days of checkerboard presentations.

neuronal predictability depends on SNR, stimulus response variance, and receptive field overlap.

A. Correlation between different neuronal properties with the predictability of L2/3 (green) and L4 (coral) neuronal responses during the presence (dark color) or absence (light color) of visual stimulus. Neuronal properties measured in mouse V1 include the correlation value of the most correlated pair to each cell (max correlation value, squared), a modified metric of self-consistency (one-vs-rest correlation, squared), SNR, variance in the neuronal activity in response to different stimuli, and the traditional metric of self-consistency (split-half correlation r) (Methods). B. Relationship between three neuronal properties and their predictability in a randomly chosen sub-sample of neurons (n = 4, 000) for mouse L2/3 (green) and L4 (coral) neuronal responses from both drifting gratings and natural images conditions (combined). Hue represents the degree of predictability for the same neurons during the 30 minutes of gray screen presentation (see color map on bottom right).C. 1-vs-rest square correlation relationship with predictability after projecting out dimensions of “non-visual” activity (using gray screen activity (Stringer et al., 2019a). D. Correlation between different neuronal properties with the predictability of monkey V4 (green) and V1 (coral) neuronal site recordings during the full-field checkerboard presentation (dark color), gray screen presentation (light color), and lights-off condition (darkest color; solid, hatch lines, and hatch dots). Neuronal properties measured in the macaque visual cortex include the max correlation squared value, one-vs-rest squared correlation, SNR, and split-half correlation r. E. Same as B for the macaque V1 and V4 neuronal sites. F. Top: Receptive fields of one sample V4 neuronal site (green circle, array 2 electrode 187) and 14 randomly selected V1 neuronal sites as predictors (black circles), constrained on sites that share less than 10% receptive field overlap with the V4 site. Bottom: Receptive fields of the same neuronal site 187 and 14 randomly selected V1 neuronal sites used as predictors, constrained on sites that share at least 80% receptive field overlap with the V4 site. G. Differences in predictability of V4 neural activity (n = 110 site recordings) in terms of 14 V1 predictor sites with less than 10% RF overlap (light green), 14 predictor sites with at least 80% RF overlap (green), and all predictors (dark green). Predictions were computed for recordings in response to the stimulus presentation (sliding bars and full-field checkerboard images), gray screen presentation, and lights off. H. Bottom and top left: Same as F but for macaque sample V1 site 810. I. Same as D, but for V1 (n = 970 site recordings).

Predicting neuronal activity across time reveals shared stimulusand non-stimulus driven information in both mouse and macaque visual cortex, along with latency and non-latency specific correlations in macaque V1/V4.

A. Shuffled-trial-repeat experiment set-up for comparing unshuffled vs. shuffled prediction in mouse V1 L2/3 and L4. Neuronal activity in response to stimulus repeats was shuffled within their respective image. B. EV fraction for unshuffled (dark) and shuffled (light) trials in the L4 → L2/3 (green) and L2/3 → L4 (coral) directions. C. Relationship between shuffled (y-axis) and unshuffled (x-axis) trial repeat EVs in the mouse L4 → L2/3 (left, green) and L2/3 → L4 (right, coral) directions. Hue represents EV fraction during gray screen activity. D. Shuffled-trial-repeat experiment set-up comparing unshuffled vs. shuffled prediction in macaque data. Neuronal activity (including all timepoints) in response to stimulus repeats were shuffled within their respective image. Since checkerboard presentation was only one stimulus, visualization of experiment only applies to the “Stimulus A” portion. E. Same as B for macaque V1 → V4 (green) and V4 → V1 (coral). F. Same as C for macaque V1 → V4 (green) and V4 → V1 (coral). G. Illustration of time offsets applied to macaque neuronal activity for inter-areal predictions. Instead of neuronal activity prediction between areas being done on simultaneous activity (middle coral and bottom green box), the V4 neuronal activity (green) at time tm was predicted using V1 neuronal activity (coral) at time tm±offset, were offset represents 1–8 timebins (25 ms per timebin) before (if negative; left coral box) or after (if positive; right coral box) time tm. Time offset experiment was done in both prediction directions (V1 → V4 and V4 → V1). H. Experimental set-up example for predicting neuronal activity in V4 using V1 neuronal activity from 25 ms prior to V4 activity. Neural activity is in response to a repeated checkerboard image. A 200 ms section of the cortical area was used to represent the image presentation response, and was offset -1 timebin (25 ms) to predict a 200 ms target cortical area. During the prediction experiments, the 200 ms window was slid across the entire duration of the stimulus I. Time offset prediction results across both V1→V4 (left,green) and V4→V1 (right, coral) prediction directions. Each square corresponds to the fraction of neuronal sites whose neural activity were best predicted during that offset period and time window.

EV fraction in mouse L4 neurons and macaque V1 neuronal sites and comparison between visual and non-visual neurons/sites.

A. Distribution of EV fraction in L2/3→L4 regressions in cells deemed visually responsive in 4 mice and 7 recording days. B. Same as A, but for macaque V4→V1 neuronal site regressions. C Distribution of visually (purple) and non-visually (gray) responsive neurons in mouse L2/3 and L4. In mouse, we used a conservative criterion to select neurons to be visually responsive, based on an average signal to noise ratio of over 2 and a split-half correlation value of at least 0.8 (more details in Methods). D. Same as C but for V1 and V4 sites in macaque. E. Differences in EV fractions using filtering methods to determine visually responsive neurons in mouse L2/3 and L4 across the 4 mice. Both a SNR of over 2 along with a split-half correlation value of over 0.8 was used to determine a neuron to be visually responsive. F. Same as E, but for the one macaque V1 and V4 sites.

Neuronal property differences between areas in mouse and monkey.

Differences in self-consistency (A), SNR (B), and max correlation value (C) between entire visually responsive neuronal populations in mouse L2/3 and L4 (independent permutation test, here and throughout figure). D–F Same as A–C but for macaque V4 and V1. G. Differences in inter-laminar predictability directions in mouse when using all predictors in their respective L2/3 and L4 layers. H. Differences in inter-laminar predictability directions in mouse during gray screen presentation neuronal activity. I. Same as G, but for macaque V1 and V4. J. Differences in inter-cortical predictability between macaque V1 and V4 neuronal activity in response to gray screen presentations and during lights off conditions.

Neuronal activity properties for different stimulus types.

A. Sample stimuli for mouse drifting grating and static natural images. B-D. Split-half correlation (B), SNR (C), and maximum correlation values (D) for each mouse layer and stimulus type (see color scheme in part A. E. Sample stimuli for monkeys: full-size checkerboard, slow and fast moving bars. F-H. Same as B-D for macaque V1 and V4 (see color map for each stimulus condition in E.

Comparing stimulus presentation vs. gray screen activity predictions in mouse and macaque.

A. Differences in inter-laminar predictability between stimulus presentation and gray screen presentation neuronal activity in L2/3 across the three different mice (MP027 did not undergo gray screen presentation recordings). Left: visually responsive L2/3 neurons. Right: non-visually responsive L2/3 neurons. B. Same as A, but for mouse L4. C. Correlation coefficient values between checkerboard presentation and gray screen and lights off conditions in macaque inter-cortical predictability. D. Differences in inter-cortical predictability between moving bar presentation and gray screen activity in macaque V1 and V4 (paired permutation test).

Bimodal distributions of visual and nonvisually active inter-laminar predictability across mice.

A. Relationship between 1-vs-rest squared correlation and EV fraction in L2/3 (top, green) and L4 (bottom, coral) neurons across all mice. B. Same relationship between 1-vs-rest squared correlation and EV performanc, but after projecting out “non-visual ongoing” activity (Stringer et al., 2019a). C. Differences in 1-vs-rest squared correlation values between including and not including non-visual ongoing activity dimensions across three mice (paired permutation test). D. Differences in inter-laminar predictability between including and not including non-visual ongoing activity dimensions in L2/3 (top, green) and L4(bottom, coral) across three mice. Sample subset of neurons with initial prediction values of over 0.4 visualized with lineplots.

Predictability across time using gray screen neuronal activity.

A. Sample averaged raw MUAe activity in macaque V1 and V4 across the three different conditions. B. Differences in EV fraction between unshuffled and shuffled trialrepeat activity during gray screen presentations (paired permutation test). Visualization of relationship between unshuffled and shuffled EV fraction in V4 (C) and V1(D) during gray screen presentations. E. Percentage of neurons with max performance across predictor time offsets in V1→V4 (E) and V4→V1 (F) directions during gray screen presentations.