Introduction

In addition to the position and orientation of simple visual cues, like Gabor patches and drifting gratings8, primary visual cortical responses are also direction selective9, and show predictive coding10, suggesting that the temporal sequence of visual cues influences neural firing. Accordingly, these and higher visual cortical neurons also encode a sequence of visual images, i.e., a movie11–13. The hippocampus is farthest downstream from the retina in the visual circuit. The rodent hippocampal place cells encode spatial or temporal sequences2,14–21 and episode-like responses22–24. However, these responses typically require active locomotion25, and they are thought to be non-sensory responses26. Primate and human hippocampal responses are selective to specific sets of visual cues, e.g., the object-place association27, their short-term1 and long-term28 memories, cognitive boundaries between episodic movies29, and event integration for narrative association30. However, despite strong evidence for hippocampal role in episodic memory, the hippocampal encoding of a continuous sequence of images, i.e., a visual episode, is unknown.

Results

We used a publicly available dataset (Allen Brain Observatory – Neuropixels Visual Coding, © 2019 Allen Institute). Mice were monocularly shown a 30s clip of a continuous segment from the movie Touch of Evil (Welles, 1958)31 (Extended Data Fig. 1). Mice were head-fixed but were free to run on a circular disk. A total of 17048 broad spiking, active, putatively excitatory neurons were analyzed, recorded using 4-6 Neuropixel probes in 24 sessions from 24 mice (See Methods).

The majority of neurons in the visual areas (Lateral geniculate nucleus LGN, primary visual cortex V1, higher visual areas: antero-medial and medio-lateral AM-PM) were modulated by the movie, consistent with previous reports (Extended Data Fig. 2)11–13,32. Surprisingly, neurons from all parts of the hippocampus (dentate gyrus DG, CA3, CA1, subiculum SUB) were also clearly modulated (Fig. 1), with reliable, elevated spiking across many trials in small movie segments. To quantify selectivity in an identical, firing rate- and threshold-independent fashion across brain regions, we computed the z-scored sparsity33,34 of neural selectivity (See Methods). Cells with z-scored sparsity >2 were considered significantly (p<0.03) modulated. The areas V1 (97.3%) and AM-PM (97.1%) had the largest percentage of movie tuned cells, much higher than reported earlier11 (∼40%), perhaps because we analyzed extracellular spikes, while the previous study used calcium imaging. The majority of neurons in LGN (89.2%) too showed significant modulation by the movie. Thus, the vast majority of thalamo-cortical neurons were significantly modulated by the movie.

Movie frame selectivity in hippocampal neurons

(a) Raster plots of two different dentate gyrus (DG) neurons as a function of the movie frame (top). The corresponding mean firing rate response over 60 trials is also shown (bottom). These two cells had significantly increased firing activity in specific parts of the movie. 33.1% of dentate neurons were significantly modulated by the movie (right, green bar), far greater than chance (gray bar). Total active, broad spiking neurons for each brain region indicated at top (Ntuned /Ncells=506/1531). (b) Same as (a), for CA3 (168/969, 17.3%), (c) CA1 (2326/6914, 33.6%) and (d) subiculum (379/849, 44.6%) neurons.

Movie selectivity was prevalent in the hippocampal regions too, despite head fixation, dissociation between self-movements and visual cues as well as the absence of rewards, task, or memory demands (Fig. 1a-d). Subiculum, the output region of the hippocampus, farthest removed from the retina, had the largest fraction (44.6% Fig. 1d) of movie-tuned neurons, followed by the upstream CA1 (33.6%, Fig. 1c) and dentate gyrus (33.1%, Fig. 1a). However, CA3 movie selectivity was nearly half as much (17.3%, Fig. 1b). This is unlike place cells, where CA3 and CA1 selectivity are comparable35,36 and subiculum selectivity is weaker37.

To confirm these findings, we did several controls. Running alters neural activity in visual areas38,39 and hippocampus40,41. Hence, we used the data from only the stationary epochs (see Methods) and only from sessions with at least 300 seconds of stationary data (17 sessions, 24906 cells). Movie tuning was unchanged in this data (Extended Data Fig. 3). This is unlike place cells where spatial selectivity is greatly reduced during immobility5,6. Neurons recorded simultaneously from the same brain region also showed different selectivity patterns (Extended Data Fig. 4). Thus, nonspecific effects such as running cannot explain the brain wide movie selectivity.

Hippocampal neurons have one or two place fields in typical mazes which take a few seconds to traverse42. In larger arenas that take tens of seconds to traverse, the number of peaks per cell and the peak duration increases43–46. Peak detection for movie tuning is nontrivial because neurons have nonzero background firing rate, and the elevated rates cover a wide range (Fig. 1). We developed a novel algorithm to address this (see Methods). On average, V1 neurons had the largest number of movie-fields (Fig. 2a, mean±s.e.m.=10.4±0.1), followed by LGN (8.6±0.3) and AM-PM (6.3±0.07). Hippocampal areas had significantly fewer movie-fields per cell: dentate gyrus (2.1±0.1), CA3 (2.8±0.3), CA1(2.0±0.02) and subiculum (2.1±0.05). Thus, the number of movie-fields per cell was smaller than the number of place-fields per cell in comparably long spatial tracks43–48, but a handful of hippocampal cells had more than 5 movie-fields (Extended Data Fig. 5).

Multi-peaked, mega-scale movie-fields across all brain areas

(a) Distribution of the number of movie-fields per tuned cell (See Methods) in different brain regions (shown by different colors, top line inset, arranged in their hierarchical order). Hippocampal regions (blue-green shades) were significantly different from each other (KS-test p<0.04), except DG-CA3. All visual regions were significantly different from each other (KS-test p<7.0×10−11). All visual-hippocampal region pair-wise comparisons were also significantly different (KS-test p<1.8×10−44). CA1 had the lowest number of movie-fields per cell (2.0±0.02, mean±s.e.m.) while V1 had the highest (10.4±0.1). (b) Distribution of the durations of movie-fields identified in (a), across all tuned neurons from a given brain region. These were significantly different for all brain region pairs (KS-test p<7.3×10−3). The longest movie-fields were in subiculum (3169.9±169.8 ms), and the shortest in V1 (156.6±9.2ms). (c) Snippets of movie-fields from an example cell from V1, with 2 of the fields zoomed in, showing 60x difference in duration. Black bar at top indicates 50ms, and gray bar indicates 1s. Each frame corresponds to 33.3ms. Average response (solid trace, y-axis on the right) is superimposed on the trial wise spiking response (dots, y-axis on the left). Color of dots corresponds to frame numbers as in Fig. 1. (d) Same as (c), for a CA1 neuron with 54x difference in duration. (e) The ratio of longest to shortest field duration within a single cell, i.e., mega-scale index, was largest in V1 (56.7±2.2) and least in subiculum (8.0±9.7). All visual-visual and visual-hippocampal brain region pairs were significantly different on this metric (KS-test p<0.02). Among the hippocampal-hippocampal pairs, only CA3-SUB were significantly different (p=0.03). (f) For each cell, the total duration of all movie-fields, i.e., cumulative duration of significantly elevated activity, was comparable across brain regions. The largest cumulative duration (10.2±0.46s, CA3) was only 1.66x of the smallest (6.2±0.09 sec, V1). Visual-hippocampal and visual-visual brain region pairs’ cumulative duration distributions were significantly different (KS-test p<0.001), but not hippocampal pairs (p>0.07). Distribution of the firing within fields, normalized by that in the shuffle response. All fields from all tuned neurons in a brain region were used. Firing in movie-fields was significantly different across all brain region pairs (KS-test, p<1.0×10−7), except DG-CA3. Movie-field firing was largest in V1 (2.9±0.03) and smallest in subiculum (1.14±0.03). (h) Snippets of movie-fields from representative tuned cells, from LGN showing a long movie-field (233 frames, or 7.8s, panel 1), and from AM-PM and from hippocampus showing short fields (2 frames or 66.6ms wide or less).

Typical receptive field size increases as one moves away from the retina in the visual hierarchy31. A similar effect was seen for movie-field durations. On average, hippocampal movie-fields were longer than visual regions (Fig. 2b). But there were many exceptions –movie-fields of LGN (median±s.e.m., here and subsequently, unless stated otherwise, 308.5±33.9 ms) were twice as long as in V1 (156.6±9.2ms). Movie-fields of subiculum (3169.9±169.8 ms) were significantly longer than CA1 (2786.1±77.5 ms) and nearly three-fold longer than the upstream CA3 (979.1±241.1 ms). However, the dentate movie-fields (2113.2±172.4 ms) were two-fold longer than the downstream CA3. This is similar to the patterns reported for CA3, CA1 and dentate gyrus place cells48. But others have claimed that CA3 place fields are slightly bigger than CA149, whereas movie-fields showed the opposite pattern.

The movie-field durations spanned a 500-1000-fold range in every brain region investigated (Fig. 2e). This mega-scale scale is unprecedentedly large, nearly 2 orders of magnitude greater than previous reports in place cells43,45. Even individual neurons showed 100-fold mega-scale responses (Fig. 2c & d) compared to less than 10-fold scale within single place cells43,45. The mega-scale tuning within a neuron was largest in V1 and smallest in subiculum (Fig. 2e). This is partly because the short duration movie-fields in hippocampal regions were typically neither as narrow and nor as prominent as in the visual areas (Extended Data Fig. 6).

Despite these differences in mega-scale tuning across different brain areas, the total duration of elevated activity, i.e., the sum of movie-field durations within a single cell, was remarkably conserved across neurons within and across brain regions (Fig. 2f). Unlike movie-field durations, which differed by more than ten-fold between hippocampal and visual regions, cumulative durations were quite comparable, ranging from 6.2s (V1) to 10.2s (CA3) (Fig. 2f, LGN=8.8±0.21sec, V1=6.2±0.09, AM-PM=7.8±0.09, DG=9.4±0.26, CA3=10.2±0.46, CA1=9.1±0.12, SUB=9.5±0.27). Thus, hippocampal movie-fields are longer and less multi-peaked than visual areas, such that the total duration of elevated activity was similar across all areas, spanning about a fourth of the movie, comparable to the fraction of large environments in which place cells are active45,47,48. To quantify the net activity in the movie-fields, we computed the total firing in the movie-fields (i.e., the area under the curve for the duration of the movie-fields), normalized by the expected discharge from the shuffled response. Net movie-field discharge was also comparable across brain areas, but maximal in V1 and least in subiculum (Fig. 2g).

Many movie-fields showed elevated activity spanning up to several seconds, suggesting rate-code like encoding (Fig. 2h). However, some cells showed movie-fields with elevated spiking restricted to less than 50ms, similar to responses to briefly flashed stimuli in anesthetized cats12,13,50. This is suggestive of a temporal code, characterized by low spike timing jitter51. Such short-duration movie-fields were not only common in the thalamus, but also AM-PM, three synapses away from the retina. A small fraction of cells in the hippocampal areas, more than five synapses away from the retina, too showed such temporally coded fields (Fig. 2h).

To determine the stability and temporal-continuity of movie tuning across the neural ensembles we computed the population vector overlap between even and odd trials52 (see Methods). Population response stability was significantly greater for tuned than for untuned neurons (Extended Data Fig. 7). The population vector overlap around the diagonal was broader in hippocampal regions than visual cortical and LGN, indicating longer temporal-continuity, reflective of their longer movie-fields. Further, the population vector overlap away from the diagonal was longer around frames 400-800 in all brain areas due to the longer movie-fields in that movie segment (see below).

Are all movie frames represented equally by all brain areas? The duration and density of movie-fields varied as a function of the movie frame and brain region (Extended Data Fig. 8). We hypothesized that this variation could correspond to the change in visual content from one frame to the next. Hence for comparison, we quantified the similarity between adjacent movie frames as the correlation coefficient between corresponding pixels and termed it as frame-to-frame (F2F) correlation. The majority of brain regions had substantially reduced density of movie-fields between the movie frames 400 to 800, but the movie-fields were longer in this region. This effect was greater in the visual areas than hippocampal. Using significantly tuned neurons, we computed the average neural activity in each brain region at each point in the movie (see Methods). Although movie-fields (Fig. 3a), or just the tallest movie-field per cell (Fig. 3b), covered the entire movie, the peak normalized, ensemble activity level of all brain regions showed significant overrepresentation, i.e., deviation from the norm, in certain parts of the movie (Fig. 3c, see Methods). This was most pronounced in V1 and higher visual areas AM-PM. The number of movie frames with elevated ensemble activity was higher in visual areas than hippocampal regions (Fig. 3d), and also this modulation (see Methods) was smaller in hippocampus and LGN, compared to visual cortical regions (Fig. 3e).

Population averaged movie-tuning varies across brain areas.

(a) Stack plot of all the movie-fields detected from all tuned neurons of a brain region. Color indicates relative firing rate, normalized by the maximum firing rate in that movie-field. The movie-fields are sorted according to the frame with the maximal response. Note accumulation of fields in certain parts of the movie, especially in subiculum and AM-PM. (b) Similar to (a), but using only a single, tallest movie-field peak from each neuron showing a similar pattern, with pronounced overrepresentation of some portions of the movie in most brain areas. Each neuron’s response is normalized by its maximum firing rate. (c) Multiple single unit activity (MSUA) in a given brain region, obtained as the average response across all tuned cells, by using maxima-normalized response for each cell from (b). Gray lines indicate mean±4*std response from the shuffle data corresponding to p=0.025 after Bonferroni correction for multiple comparisons (see Methods). AM-PM had the largest MSUA modulation (sparsity=0.01) and CA1 had the smallest (sparsity=1.8×10−4). The MSUA modulation across several brain region pairs –AM&PM-DG, V1-CA3, DG-CA3, CA3-CA1 and CA1-SUB were not significantly correlated (Pearson correlation coefficient p>0.05). Some brain region pairs, DG-LGN, DG-V1, AM&PM-CA3, LGN-CA1, V1-CA1, DG-SUB and CA3-SUB, were significantly negatively correlated (r<-0.18, p<4.0×10−7). All other brain region pairs were significantly positively correlated (r>0.07, p<0.03). (d) Number of frames for which the observed MSUA deviates from the z=±4 range from (c), termed significant deviation. V1 and AM-PM had the largest positive deviant frames (289), and CA3 had the least (zero). (e) Firing in deviant frames above (or below) chance level, as a percentage of the average response. Above chance level deviation was greater or equal to that below, for all brain regions, with the largest positive deviation in AM-PM (9.3%), largest negative deviation in V1 (6.0%), and least in CA3 (zero each). (f) Total firing rate response of visual regions across tuned neurons. All regions had significant negative correlation (r<-0.39, p<3.4×10−34) between the ensemble response and the frame-to-frame (F2F) correlation (gray line, y-axis on the left) across movie frames. (g) Similar to (f), for hippocampal regions. CA3 response were not significantly correlated with the frame-to-frame correlation, dentate gyrus (r=0.26, p=4.0×10−15) and CA1 (r=0.21, p=1.5×10−10) responses were positively correlated, and subiculum response was negatively correlated (r=-0.44, p=2.2×10−43). Note the substantially higher mean firing rates of LGN in (f) and subiculum neurons in (g) (colored lines closer to the top) compared to other brain areas.

Using the significantly tuned neurons, we also computed the average neural activity in each brain region corresponding to each frame in the movie, without peak rate normalization (see Methods). The degree of continuity between the movie frames, quantified as above (F2F correlation), was inversely correlated with the ensemble rate modulation in all areas except DG, CA3 and CA1 (Fig. 3f and g). As expected for a continuous movie, this F2F correlation was close to unity for most frames, but highest in the latter part of the movie where the images changed more slowly. The population wide elevated firing rates, as well as the smallest movie-fields, occurred during the earlier parts (Extended Data Fig. 8). Thus, the movie-code was stronger in the segments with greatest change across movie frames. These results show differential ensemble-wide representation of the movie across brain regions.

If these responses were purely visual, a movie made of scrambled sequence of images would generate equally strong or even stronger selectivity due to the even larger change across movie frame, despite the absence of similarity between adjacent frames. To explore this possibility, we investigated neural selectivity when the same movie frames were presented in a fixed but scrambled sequence (scrambled movie). The within frame and the total visual content was identical between the continuous and scrambled movies, and the same sequence of images was repeated many times in both experiments (see Methods). But there was no correlation between adjacent frames, i.e., visual continuity, in the latter (Fig.4a).

Larger reduction of selectivity in hippocampal than visual regions due to scrambled presentation.

(a) Similarity between the visual content of one frame with the subsequent one, quantified as the Pearson correlation coefficient between pixel-pixel across adjacent frames for the continuous movie (pink) and the scrambled sequence (lavender), termed F2F correlation. Similar to Fig. 3g. For the scrambled movie, the frame number here corresponds to the chronological frame sequence, as presented. (b) Fraction of broad spiking neurons significantly modulated by either the continuous movie or the scrambled sequence using z-scored sparsity measures (similar to Fig. 1, see Methods). For all brain regions, continuous movie generated greater selectivity than scrambled sequence (KS-test p<7.4×10−4). (c) Percentage change in the magnitude of tuning between the continuous and scrambled movies for cells significantly modulated by either continuous or scrambled movie, termed visual continuity index. Largest drop in selectivity due to scrambled movie occurred in CA1 (90.3±2.0%), and least in V1 (−1.5±0.6%). Continuous to scrambled tuning change was significantly different between all brain region pairs (KS-test p<0.03) and significantly greater for hippocampal areas than visual (8.2-fold, p<10−100). (d) Raster plots (top) and mean rate responses (color, bottom) showing increased spiking responses to only one or two scrambled movie frames, lasting about 50ms. Tuned responses to scrambled movie were found in all brain regions, but these were the least frequent in DG and CA1. (e) One representative cell each from V1 (left) and CA1 (right), where the frame rearrangement of scrambled responses resulted in a response with high correlation to the continuous movie response for V1, but not CA1. Pearson correlation coefficient values of continuous movie and rearranged scrambled responses are indicated on top. (f) Average decoding error for observed data (see Methods), over 60 trials for continuous movie (maroon), was significantly lower than shuffled data (gray) (KS-test p<8.2×10− 22). Solid line – mean error across 60 trials, shaded box – s.e.m. (g) Similar to (f), decoding of scrambled trials was significantly worse than that for the continuous movie (KS-test p<3.6×10−8), except V1 (p=0.13), where the errors were not significantly different (2.6 vs. 2.7 frames). Scrambled responses, in their “as is”, chronological order were used herein. LGN decoding error for scrambled presentation was 6.5x greater than that for continuous movie, whereas the difference in errors was least for V1 (1.04x). Scrambled movie decoding error for all visual areas and for CA1 and subiculum was significantly smaller than chance level (KS-test p<2.6×10−3), but not DG and CA3 (p>0.13). Only the middle 20 trials of the continuous movie were used for comparison with the scrambled movie since the scrambled movie was only presented 20 times. Middle trials of the continuous movie were chosen as the appropriate subset since they were chronologically closest to the scrambled movie presentation.

For all brain regions investigated, the continuous movie generated significantly greater modulation of neural activity than the scrambled sequence (Fig. 4b). Middle 20 trials of the continuous movie were chosen as the appropriate subset for comparison since they were chronologically closest to the scrambled movie presentation. This preference for sequential over scrambled movie was the greatest in hippocampal regions where the percentage of significantly tuned neurons (4.4%, near chance level of 2.3%) reduced more than 4-fold compared to the continuous movie (17.8%, after accounting for the lesser number of trials, see Methods). This was unlike visual areas where the scrambled (80.4%) and the continuous movie (92.4%) generated similar prevalence levels of selectivity (Fig. 4b). The few hippocampal cells which had significant selectivity to the scrambled sequence, did not have long-duration responses, but only very short, ∼50ms long responses (Fig. 4d), reminiscent of, but even sharper than human hippocampal responses to flashed images28. To estimate the effect of continuous movie compared to the scrambled sequence on individual cells, we computed the normalized difference between the continuous and scrambled movie selectivity for cells which were selective in either condition (Fig. 4c, see Methods). This visual continuity index was more than eight-fold higher in hippocampal areas (87.8%) compared to the visual areas (10.6%).

The pattern of increasing visual continuity index as we moved up the visual hierarchy, largely paralleled the anatomic organization53, with the greatest sensitivity to visual continuity in the hippocampal output regions, CA1 and subiculum, but there were notable exceptions. The primary visual cortical neurons showed the least reduction in selectivity due to the loss of temporally contiguous content, whereas LGN neurons, the primary source of input to the visual cortex and closer to the periphery, showed far greater sensitivity (Fig. 4c).

Many visual cortical neurons were significantly modulated by the scrambled sequence, but their number of movie-fields per cell was greater and their duration was shorter than during the continuous movie (Extended Data Fig. 9 and 10). This could occur due to the loss of frame-to- frame correlation in the scrambled sequence. The average activity of the neural population in V1 and AM-PM showed significant deviation even with the scrambled movie, comparable to the continuous movie, but this multi-unit ensemble response was uncorrelated with the frame-to- frame correlation in the scrambled sequence (Extended Data Fig. 11). A substantial fraction of visual cortical and LGN responses to the scrambled sequence could be rearranged to resemble continuous movie responses (Extended Data Fig. 12, see Methods). The latency needed to shift the responses was least in LGN and largest in AM-PM, as expected from the feed-forward anatomy and the model of visual information processing (Extended Data Fig. 12). This rearrangement did not recreate the continuous movie responses above chance levels in the hippocampal regions (example cells in Fig. 4e, also see Extended Data Fig. 12 for statistics and details).

Population vector decoding of the ensemble of a few hundred place cells is sufficient to decode the rat’s position using place cells54, and the position of a passively moving object55. Using similar methods, we decoded the movie frame number (see Methods). Continuous movie decoding was better than chance in all brain regions analyzed (Fig. 4f). Scrambled movie decoding was significantly weaker yet above chance level (as expected from shuffles, see Methods) in visual areas, but not in CA3 and dentate gyrus. But CA1 and subiculum neuronal ensembles could be used to decode scrambled movie frame number slightly above chance levels (Fig. 4g). Similarly, the population overlap between even and odd trials for scrambled sequence was strong for visual areas, and weaker in hippocampal regions, but significantly greater than untuned neurons in hippocampal regions (Extended Data Fig. 13).

Discussion

To understand how neurons encode a continuously unfolding visual episode, we investigated the neural responses in the rodent brain to an isoluminant human movie. As expected, neural activity in all thalamo-cortical visual areas was significantly modulated, with elevated activity in response to specific visual content. Most (96.6%, 6554/6785) of thalamo-cortical neurons showed significant movie tuning, nearly double that reported for the classic stimuli such as Gabor patches in the same dataset31. Remarkably, a third of hippocampal neurons (32.9%, 3379/10263) were also movie-tuned, comparable to the fraction of neurons with significant spatial selectivity in mice56 and bats57. While the hippocampus is implicated in episodic memory, rodent hippocampal responses are largely studied in the context of spatial maps or place cells, and more recently in other tasks which requires active locomotion or active engagement7,58.

However, unlike place cells5,6, movie-tuning remained intact during immobility in all brain areas studied. This dissociation of the effect of mobility on spatial and movie selectivity agrees with the recent reports of dissociated mechanisms of episodic encoding and spatial navigation in human amnesia59.

Across all brain regions, neurons showed a mega-scale encoding by movie-fields varying in duration by up to 1000-fold, similar to, but far greater than recent reports of 10-fold multi-scale responses in the hippocampus43–48,60. Importantly, mega-scale responses were found within single neurons as well, which can enable encoding of both finer and broader details of an episode in the same neuron. This could provide greatly enhanced episodic memory and recall.

The response latency was highest in AM-PM, then V1 and least in LGN, thus following the visual hierarchy. Among the visual areas, continuous movie tuning was greater in V1, more than the upstream LGN. This is surprising but can explain the recent findings that silencing V1 reduces movie tuning in the thalamus61. Similarly, the preference for the continuous movie over scrambled sequence was greater in LGN and AM-PM than V1, which does not follow the expected hierarchy of visual processing. This effect was even greater in the hippocampal areas than any visual regions. Temporal integration window62–64 as well as intrinsic timescale of firing31 increases along the anatomical hierarchy in the cortex, with the hippocampus being farthest removed from the retina53. This hierarchical organization could explain the longer movie-fields and several fold greater preference for the continuous movie over scrambled sequence in the hippocampus. But, unlike reports of image-association memory in the inferior temporal cortex for unrelated images65, only a handful hippocampal neurons showed selective responses to the scrambled sequence. These results, along with the longer duration of hippocampal movie-fields could mediate visual-chunking or binding of a sequence of events.

Could the brain-wide mega-scale tuning be an artifact of poor unit isolation, e.g., due to an erroneous mixing of two neurons, one with very short and another with very long movie-fields? This is unlikely since the LGN and visual cortical neural selectivity to classic stimuli (Gabor patches, drifting gratings etc.) in the same dataset was similar to that reported in most studies31 whereas poor unit isolation should reduce these selective responses. However, to directly test this possibility, we calculated the correlation between the unit isolation index (or fraction of refractory violations) and the mega-scale index of the cell, while factoring out the contribution of mean firing rate (Extended Data Fig. 14). This correlation was not significant (p>0.05) for any brain areas.

Do the movie fields arise from the same mechanism as place fields? Studies have shown that when rodents are passively moved along a linear track that they had explored6, or when the images of the environment around a linear track was played back to them5, some hippocampal neurons generated spatially selective activity. Since the movie clip involved change of spatial view, one could hypothesize that the movie fields are just place fields generated by passive viewing. This is unlikely for several reasons. Mega-scale movie fields were found in all brain regions investigated, including the lateral geniculate nucleus and the primary visual cortex. There is no evidence of place cells in these brain regions. Further, in prior passive viewing experiments, the rodents were shown the same narrow linear track, like a tunnel, that they had previously explored actively to get food rewards at specific places. In contrast, in current experiments, these mice had never actively explored the space shown in the movie, nor obtained any rewards. Active exploration of a maze, combined with spatially localized rewards engages multisensory mechanisms resulting in increased place cell activation17,23,66 which are entirely missing in these experiments during passive viewing of a movie, presented monocularly, without any other multisensory stimuli and without any rewards. Compared to the exploration of a real-world maze, exploration of a visually identical virtual world causes 60% reduction in CA1 place cell activation67. In contrast, there was no evidence of significant CA1 neural shutdown during movie viewing, compared to the blank screen presentation (Extended Data Fig. 14).

A recent study showed that CA1 neurons encode the distance, angle, and movement direction of motion of a vertical bar of light55, consistent with the position of hippocampus in the visual circuitry53. Do those findings predict the movie tuning herein? There are indeed some similarities between the two experimental protocols –purely passive optical motion without any self-motion or rewards. However, there are significant differences too; similar to place cells in the real and virtual worlds34, all the cells tuned to the moving bar of light had single receptive fields with elevated responses lasting a few seconds; there were neither punctate responses nor even 10-fold variation in neural field durations, let alone the 1000-fold change reported here. Finally, those results were reported only in area CA1, while the results presented here cover nearly all the major stations of the visual hierarchy.

Notably, hippocampal neurons did not encode Gabor patches or drifting gratings in the same dataset, indicating the importance of temporally continuous sequences of images for hippocampal activation31. This is consistent with the hypothesis that the hippocampus is involved in coding spatial sequences16,24,68. However, unlike place cells that degrade in immobile rats, hippocampal movie tuning was unchanged in the immobile mouse. Further, the scrambled sequence too was presented in the same sequence many times, yet movie tuning dropped to chance level in the hippocampal areas. Unlike visual areas, scrambled sequence response of hippocampal neurons could not be rearranged to obtain the continuous movie response. This shows the importance of continuous, episodic content instead of mere sequential recurrence of unrelated content for rodent hippocampal activation.

These results complement recent findings of spatial modulation of visual cortical neurons and coordinated activity of visual and hippocampal neurons during navigation69,70. Our findings open up the possibility of studying thalamic, cortical, and hippocampal brain regions in a simple, passive, and purely visual experimental paradigm and extend comparable convolutional neural networks11 to have the hippocampus at the apex53. Further, our results here bridge the long-standing gap between the hippocampal rodent and human studies29,71–73, where natural movies can be decoded from fMRI signals in immobile humans74. This brain-wide mega-scale encoding of a human movie episode and enhanced preference for visual continuity in the hippocampus compared to visual areas supports the hypothesis that the rodent hippocampus could play a role in non-spatial episodic memories, consistent with classic findings in humans1 and agrees with a more generalized, representational framework75,76 of episodic memory where it encodes temporal patterns. Similar responses are likely across different species, including primates. Thus movie-coding can provide a unified platform to investigate the neural mechanisms of episodic coding, learning and memory.

Methods

Experiments

We used the Allen Brain Observatory – Neuropixels Visual Coding dataset (© 2019 Allen Institute, https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels). This website and related publication31 contain detailed experimental protocol, neural recording techniques, spike sorting etc. Briefly, and of relevance here, prior to implantation with Neuropixel probes, mice passively viewed the entire range of images including drifting gratings, Gabor patches and the movies of interest here. Videos of the body and eye movements were obtained at 30Hz and synced to the neural data and stimulus presentation using a photodiode. Movies were presented monocularly on an LCD monitor with a refresh rate of 60Hz, positioned 15cm away from the mouse’s right eye and spanned 120°x95°. 30 trials of the continuous movie presentation were followed by 10 trials of the scrambled movie. Next was a presentation of drifting gratings, followed by a quiet period of 30 minutes where the screen was blank. Then the second block of drifting gratings, scrambled movie and continuous movie was presented.

Neural spiking data was sampled at 30 kHz with a 500Hz high pass filter. Spike sorting was automated using Kilosort277. Output of Kilosort2 was post-processed to remove noise units, characterized by unphysiological waveforms. Neuropixel probes were registered to a common co-ordinate framework78. Each recorded unit was assigned to a recording channel corresponding to the maximum spike amplitude and then to the corresponding brain region. Broad spiking units identified as those with average spike waveform duration (peak to trough) between 0.45 to 1.5ms were analyzed herein.

Movie tuning quantification

The movie consisted of 900 frames: 30s total, 30Hz refresh rate, 33.3ms per frame. At the first level of analysis, spike data were split into 900 bins, each 33.3ms wide (the bin size was later varied systematically to detect mega-scale tuning, see below). The resulting tuning curves were smoothed with a Gaussian window of σ=66.6 ms or 2 frames. The degree of modulation and its significance was estimated by the sparsity s as below, and as previously described methods55.

where rn is the firing rate in the nth frame or bin and N=900 is the total number of bins. Statistical significance of sparsity was computed using a bootstrapping procedure, which does not assume a normal distribution. Briefly, for each cell, the spike train as a function of the frame number from each trial were circularly shifted by different amounts and the sparsity of the randomized data computed. This procedure was repeated 100 times with different amounts of random shifts. The mean value and standard deviation of the sparsity of randomized data were used to compute the z-scored sparsity of observed data using the function zscore in MATLAB. The observed sparsity was considered statistically significant if the z-scored sparsity of the observed spike train was greater 2, which corresponds to p<0.023 in a one tailed t-test. Similar method was used to quantify significance of the scrambled movie tuning, as well as for the subset of data with only stationary epochs, or its equivalent subsample (see below). Middle 20 trials of the continuous movie were used in comparisons with the scrambled movie in Fig. 4, to ensure a fair comparison by using same number of trials, with similar time delays across measurements.

Stationary epoch identification

To eliminate the confounding effects of changes in behavioral state associated with running, we repeated our analysis in stationary epochs, defined as epochs when the running speed remained less than 2cm/sec for this period, as well as at least 5 seconds before and after this period. Analysis was further restricted to sessions with at least 5 total minutes of these epochs during the 60 trials of continuous movie presentation. To account for using lesser data of the stationary epochs, we compared the tuning using a random subsample of data, regardless of running or stopping and compared the two results for difference in selectivity.

Mega-scale movie-field detection in tuned neurons

For neurons with significant movie-sparsity, i.e., movie-tuned, the movie response was first recalculated at a higher resolution of 3.33ms (10 times the frame rate of 33.3ms). The findpeaks function in MATLAB was used to obtain peaks with prominence larger than 110% (1.1x) the range of firing variation obtained by chance, as determined from a sample shuffled response. This calculation was repeated at different smoothing values (logarithmically spaced in 10 Gaussian smoothing schemes with σ ranging from 6.7ms to 3430ms), to ensure that long as well as short movie-fields were reliably detected and treated equally. For frames where overlapping peaks were found at different smoothing levels, we employed a comparative algorithm to only select the peak(s) with higher prominence score. This score was obtained as the ratio of the peak’s prominence to the range of fluctuations in the correspondingly smoothed shuffle. This procedure was conducted iteratively, in increasing order of smoothing. If a broad peak overlapped with multiple narrow ones, the sum of scores of the narrow ones was compared with the broad one. To ensure that peaks at the beginning as well as the end of the movie frames were reliably detected, we circularly wrapped the movie response, for the observed as well as shuffle data.

Identifying frames with significant deviations in multiple single-unit activity (MSUA)

First, the average response across tuned neurons for each brain region was computed for each movie frame, after normalizing the response of each cell by the peak firing response. This average response was used as the observed “Multiple single unit activity (MSUA)” in Fig. 3. To compute chance level, individual neuron responses were circularly shifted with respect to the movie frames to break the frame to firing rate association but maintain overall firing rate modulation. 100 such shuffles were used, and for each shuffle, the shuffled MSUA response was computed by averaging across neurons. Across these 100 shuffles, mean and standard deviation was obtained for all frames, and used to compute the z-score of the observed MSUA. To obtain significance at p=0.025 level, Bonferroni correction was applied, and the appropriate z-score (4.04) level was chosen. The number of frames in the observed MSUA above (and below) this level were further quantified in Fig. 3. The firing deviation for these frames was computed as the ratio between the mean observed MSUA and the mean shuffled MSUA, reported as a percentage, for frames corresponding to z-score greater than +4 or less than -4. To obtain a total firing rate report, where each spike gets equal vote, we computed the total firing response by computing the total rate across all tuned neurons (and averaging by the number of neurons) in Fig. 3 and across all neurons in Extended Data Fig. 8.

Population Vector Overlap

To evaluate the properties of a population of cells, movie presentations were divided into alternate trials, yielding even and odd blocks52. Population vector overlap was computed between the movie responses calculated separately for these 2 blocks of trials. Population vector overlap between frames x of the even trials & frame y of the odd trials was defined as the Pearson correlation coefficient between the vectors (R1,x, R 2,x, … R N,x ) & (R 1,y, R 2,y, … R N,y ), where R n,x is the mean firing rate response of the nth neuron to the xth movie frame. N is the total number of neurons used, for each brain region. This calculation was done for x and y ranging from 1 to 900, corresponding to the 900 movie frames. The same method was used for tuned and untuned neurons in continuous movie responses in Extended Data Fig. 7, and for scrambled sequence responses in Extended Data Fig. 13.

Decoding analysis

Methods similar to those previously described were used54,55. For tuned cells, the 60 trials of continuous movie were each decoded using all other trials. Mean firing rate responses in the 59 trials for 900 frames were used to compute a “look-up” matrix. Each neuron’s response was normalized between 0 and 1. At each frame in the “observed” trial, the correlation coefficient was computed between the population vector response in this trial and the look-up matrix. The frame corresponding to the maximal correlation was denoted as the decoded frame. Decoding error was computed as the average of the absolute difference between actual and decoded frames, across the 900 frames of the movie. For comparison, shuffle data was generated by randomly shuffling the cell-cell pairing of the look-up matrix and “observed response”. A similar procedure was used for the 20 trials of the scrambled sequence, and the corresponding middle 20 trials of the continuous movie were used.

Rearranged scrambled movie analysis

To differentiate the effects of visual content versus visual continuity between consecutive frames, we compared the responses of the same neuron to the continuous movie and the scrambled sequence. In the scrambled movie, the same visual frames as the continuous movie were used, but they were shuffled in a pseudo random fashion. The same scrambled sequence was repeated for 20 trials. The neural response was computed at each frame of the scrambled sequence, keeping the frames in the chronological order of presentation. Then the scrambled sequence of frames was rearranged to recreate the continuous movie and the corresponding neural responses computed. To address the latency between movie frame presentation and its evoked neural response, which can differ across brain regions and neurons, this calculation was repeated for rearranged scrambled sequences with variable delays between τ= -500 to +500 ms (i.e., -150 to +150 frames of 3.33ms resolution, in steps of 5 frames or 16.6ms). The correlation coefficient was computed between the continuous movie response and this variable delayed response at each delay as rmeasured (τ) = corrcoef (Rcontinuous, Rscramble-rearranged(τ)). Rcontinuous is the continuous movie response, obtained at 3.33ms resolution and similarly, Rscramble-rearranged corresponds to the scrambled response after rearrangement, at the latency τ. The latency τ yielding the largest correlation between the continuous and rearranged scrambled movie was designated as the putative response latency for that neuron. This was used in Extended Data Fig.

12. The value of rmeasured(τmax) was bootstrapped using 100 randomly generated frame reassignments, and this was used to z-score rmeasured(τmax), with z-score > 2 as criterion for significance. The resultant z-score is reported in Extended Data Fig. 12.

Acknowledgements

We thank the Allen Brain Institute for provision of the dataset, Dr. Josh Siegle for help with the dataset, Dr. Krishna Choudhary for proof-reading of the text and Dr. Massimo Scanziani for input and feedback. This work was supported by grants to M.R.M. by the National Institutes of Health NIH 1U01MH115746.

Author contributions

C.S.P. performed the analyses with input from M.R.M. C.S.P. and

M.R.M. wrote the manuscript.

Competing interests

Authors declare that they have no competing interests.

Data availability

All data is publicly available at the Allen Brain Observatory – Neuropixels Visual Coding dataset (© 2019 Allen Institute, https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels).

Code availability

All analyses were performed using custom-written code in MATLAB version R2020a. Codes written for analysis and visualization are available from the corresponding authors at reasonable request.

Extended Data Figures

The movie.

The 30 second long, isoluminant movie with frame numbers denoting key episodes in this continuous segment.

Movie selectivity across brain areas.

(a) Similar to Fig. 1, representative single cells from LGN showing selective movie responses. Fraction selective are shown by the bar chart on the right. (b) Same as (a), for V1. (c) Same as (a), for higher visual areas AM-PM. (d) Cumulative distribution of movie selectivity across all broad spiking cells, including tuned cells (z>2 vertical black line, see Methods). Largest prevalence of selectivity in broad spiking neurons was seen in primary visual cortex (V1, 97.3%, 2606 out of 2679) and least in CA3 hippocampus (17.3%, 168 out of 969). (e) All brain regions analyzed showed far greater selectivity than the chance level (dashed gray line).

Movie tuning is unaffected by locomotive state of the mouse

(a) Similar to Fig. 1, a representative cell from each brain region showing significant modulation movie tuning using only the data when the mouse was immobile, while excluding the data when the mouse was running (stationary data, see Methods). All cells except for DG are from Fig. 1 or Extended Data Fig. 2. (b) Fraction of selective neurons was significantly above chance in all brain regions, ranging from 94.7% in V1 up to 7.1% in CA3 in the stationary data. (c) To explicitly test the effect of running on movie selectivity, we compared the results in (b) with a random subsample of data that included running and stationary, to control for the loss of data (see Methods). Prevalence of movie selectivity was not significantly different (KS-test p>0.18) in these 2 subsamples, except in CA1 (p=0.007, 13.1% in stationary data, 15.0% in the equivalent subsample). Only sessions with at least 300 seconds of stationary data were used in this analysis to ensure sufficient statistical power.

Simultaneously recorded hippocampal cells have different movie tuning.

Four simultaneously recorded and significantly movie-tuned cells from each from (a) Dentate gyrus, (b) CA3, (c) CA1 and (d) Subiculum. Each cell shows different movie selectivity. Average responses are overlaid (on raster plots), and their color corresponds to the different brain regions, described in Fig. 1 legend.

Few hippocampal neurons had greater than 5 movie-fields.

Handful of movie-tuned neurons from dentate gyrus (row 1 and 2), CA3 (row 3 and 4), CA1 and subiculum (bottom-right), had multiple movie-fields. Average responses are overlaid (on raster plots), similar to Fig. 1 and Extended Data Fig. 4.

Movie-field duration ratios are shorter than expected by chance in visual, but not hippocampal areas.

(a) Distribution of median duration of movie-fields, computed across all fields of a neuron. All visual-hippocampal region pairs were significantly different (KS-test p<7.1×10−31). DG-CA3 and DG-CA1 were not significantly different, but other visual-visual and hippocampal-hippocampal region pairs were significantly different. (KS-test p<0.04). CA3 had the largest median field duration (6.3±0.48s), and V1 had the smallest (0.27±0.03sec). Surprisingly, LGN movie-field durations (0.57±0.13s) were significantly longer than V1 (p=2.5×10−21), and comparable to those in the higher order brain areas (0.71±0.05s). (b) Median firing in movie-fields, normalized by that in the shuffle response, obtained as the median from all fields of a neuron. This metric is significantly different across all brain region pairs (KS-test p<3.4×10−5), except DG-CA3, CA3-CA1 and DG-CA1 pairs. The largest median firing was seen for V1 (2.5±0.05), and the smallest in subiculum (1.13±0.03). (c) Cumulative firing in movie-fields, normalized by that in the shuffle response, obtained by adding the firing within all fields of a neuron was significantly different across all brain region pairs (KS-test p<3.0×10−7), except DG-CA3, CA3-CA1 and DG-CA1. V1 response was largest (1.93±0.04), and subiculum was the smallest (1.11±0.02). (d) For each brain region, the movie-field duration ratio was recalculated by randomly reassigning the cell ids to all the movie peaks from that brain region. Using this new assignment of movie peaks to a cell, we obtain the chance level of mega-scale index (largest/smallest peak duration) within a cell. The observed mega-scale index was lesser than chance in all the visual areas (KS-test p<3.2×10−3, median was 77.5%, 56.2% and 41.7% of chance for LGN, V1 and AM-PM respectively). This was not the case in hippocampal regions (p>0.23). (e) Histogram of movie-fields, binned for their durations and their prominence, on a log-log scale. The most prominent fields tended to be wider in most brain areas, and this effect was stronger in hippocampal regions, than visual. Note that the histogram color is also logarithmically scaled.

Population vector overlap is broader in hippocampus and more stable for tuned than untuned cells.

(a) Population vector overlap between even and odd trials for the population of tuned neurons show highest overlap along the diagonal, i.e. the same movie frame, for all brain regions. Each neuron’s response was normalized by its mean rate and the average response in even as well as odd trials was smoothed by a Gaussian window of 2 frames (66.6ms, see Methods). Dashed black lines indicate the -300 and +300 frames away from the diagonal. Notice large correlations (close to unity, horizontal color bar) indicating stable responses. The correlations decay quickly to smaller values for the visual areas but slowly for hippocampal areas, due to their broader movie-fields. (b) Same as (a), but for untuned neurons, resulting in a salt and pepper overlap pattern and low values of correlation, indicating lesser stability than the tuned neurons. Since the majority of cells in the visual areas were tuned, the untuned population was smaller, leading to more variable population vector overlap. (c) The average overlap as a function of the number of movie frames away from the diagonal. It had a large value in visual regions for the 0th diagonal (colored lines) indicating stable responses, whereas the untuned neuron population (gray lines) were unstable, with values near zero, or chance level. The highest population vector overlap in hippocampal regions was smaller than visual areas but persisted for more frames, due to their broader movie-fields (Full width at half maximum of the peak – 17.3 frames for LGN, 22.7-V1, 39.0-AM&PM, 49.8-DG, 57.4-CA3, 64.7-CA1 and 59.2-subiculum).

Distribution of movie-fields reflects frame to frame correlation structure of the movie.

(a) Histogram of movie-fields across all tuned neurons in a brain region, as a function of the movie frame, showing non-uniform distribution. Framen to framen+1 correlation coefficient (F2F correlation), indicating the similarity of 2 consecutive frames, is shown for reference in gray. All distributions were deemed significantly different than a uniform distribution based on a Chi-square goodness-of-fit test (p<3.8×10−6). All distributions were significantly negatively correlated with F2F correlation (r<-0.18, p<10−7). (b) Same as (a), but for the median duration of movie-fields. F2F correlation shown in gray, with large correlation between consecutive frames between frames 400-800 reflected in larger movie-field durations in visual areas. All distributions were significantly different than a uniform distribution based on a Chi-square goodness-of-fit test (p<10−100) and all distributions were significantly positively correlated with F2F correlation (r>0.24, p<2×10−13). Note-y-axes for the histogram are log-scaled and show larger median durations for hippocampal regions than visual. (c) Total firing rate across all broad spiking neurons in different brain regions, showing similar non-uniformity as Fig. 3. All brain regions had significantly negative correlation with the F2F correlation (r<-0.08, p<0.03), except DG, which was significantly positively correlated (r=0.21, p=2.4×10−10). Largest number of above chance deviations were seen for AM-PM (340 frames), and least for CA3 (57 frames). Below chance level deviations were least common in LGN (25 frames), and most common in AM-PM (441 frames). Similar to Fig. 3c-e.

Scrambled movie elicits narrower but more movie-fields per cell than the continuous movie in the visual regions.

(a) Total number of fields per cell for the scrambled sequence were hierarchically arranged, with largest number of fields in LGN (mean±s.e.m., 31.8±2.0), followed by V1 (24.0±0.38) and last AM-PM (11.1±2.1). All three brain regions were significantly different from each other (KS-test p<2.0×10−5). (b) Median scrambled movie-field duration was shortest in LGN (43.9±131.2ms), intermediate in V1 (46.2±24.8) and widest in AM-PM (77.6±40.1ms), and differences were significant (p<7.0×10−4). This was much smaller than for the continuous movie (Fig. 2). (c) Durations of fields for scrambled sequence across all fields of all neurons from a brain region. These were narrowest in LGN (31.3±6.5ms), followed by V1 (38.6±0.2) and last AM-PM (64.3±6.7). All differences were significant (KS-test p<7.2×10−136). (d) Despite these differences, the cumulative duration of movie-fields was comparable across the three brain regions (1.69±0.05sec for V1, 2.03±0.07 for AM-PM and 2.4±0.2 for LGN), but significantly different (p<1.7×10−5). Note the linear scale on the x-axis in this panel compared to the log-scale in other panels. (e) Ratio of field durations, i.e., mega-scale index, was smallest in V1 (15.5±1.6), intermediate in LGN (16.3±5.4) and largest in AM-PM (23.4±2.1), and not significantly different between V1 and LGN (p=0.28). V1-AM&PM and LGN-AM&PM were significantly different (p<5.7×10−5). (f) Cumulative firing activity, summed across all movie-fields of a given neuron was largest in V1 (3.8±0.1), intermediate in LGN (2.3±0.1) and smallest in AM-PM (2.0±0.07), and significantly different between all brain region pairs (p<0.02).

Scrambled sequence evoked movie-fields were narrower than the continuous movie-fields in all visual regions, cell by cell comparison.

Data for only those visual area neurons that were significantly modulated by both the continuous and scrambled movie were used. (a) The number of movie-fields per cell for the continuous movie was significantly smaller than that for scrambled sequence in all brain areas (LGN – continuous mean±s.e.m.=10.7±0.42, scrambled=31.8±2.0, KS-test p=2.0×10−23, V1-10.8±0.11 vs. 24.0±0.38, KS-test p=3.7×10−210, AM&PM-6.9±0.07, vs. 11.1±0.21, KS-test p=1.3×10−57). Data are additionally scattered by a small random number for ease of visualization. (b) Median duration of movie-fields for a cell was significantly larger for continuous movie, compared to scrambled sequence in all visual regions. (LGN continuous=0.46±0.08sec, scrambled=0.04±0.13, KS-test p=7.1×10−65, V1-0.25±0.03 vs. 0.04±0.02, KS-test p<10−150, AM&PM-0.65±0.04, vs. 0.08±0.04, KS-test p<10−150). (c) Cumulative duration of all movie-fields for a cell was significantly larger for continuous movie, compared to scrambled sequence in all visual regions. (continuous=8.9±0.23sec, scrambled=2.4±0.19, KS-test p=3.2×10−69, V1-6.1±0.09 vs. 1.69±0.05, KS-test p=3.3x10−296, AM&PM-7.8±0.1, vs. 2.0±0.07, KS-test p=9.0×10−318). (d) Histogram of number of fields per cell, for continuous and scrambled movies. (e) Logarithmically spaced histogram of median field durations was significantly different between continuous and scrambled sequence. (f) Similar to (e), histogram of cumulative duration of movie-fields for each cell. (g) The ratio of number of fields per cell between continuous and scrambled movies was biased to smaller than unity values for all brain regions, with the largest bias for LGN (0.46±0.08), intermediate for V1 (0.5±0.04), and least for AM-PM (0.77±0.05). (h) The median field duration ratio was biased to values greater than unity, with the largest bias for LGN (7.4±1.4), least for V1 (4.5±0.68), and intermediate for AM-PM (5.5±0.82). (i) The cumulative field duration ratio was also biased to values greater than unity, with similar biases for LGN (3.37±0.36), V1 (3.1±0.3), and AM-PM (3.3±0.67).

Multiple-single unit activity (MSUA) across all movie-tuned neurons in a brain region shows greater modulation than chance levels for scrambled sequence in all visual areas.

(a) Stack plot of tuned responses to the scrambled movie presentation from each brain region, arranged in their increasing order of the frame corresponding to the peak firing response. Each response is normalized by the peak response. (b) Colored trace-average response, across all tuned responses from (b). gray trace - chance level, z=±4, corresponding to the p=0.025 level after Bonferroni correction. (c) Number of frames for which the observed response exceeds (or falls below) z=±4 cutoff from (b), called significantly deviant frames. V1 had the largest number of positive (279 frames) and negative (297) deviant frames, AM-PM had intermediate (225 & 235), and LGN had the least (31 & 29). (d) Firing rate deviation above chance levels, corresponding to the significant frames, as identified in (c), normalized by the mean rate of the MSUA. Largest deviation was observed in V1 (above-3.1 and below-2.7%), and least in LGN (1.1% and 0.45%) Compare with Fig. 3. (e) Frame to frame correlation, from Fig. 4a for comparison. This was not significantly correlated with the MSUA responses in (b), for any of the brain regions (Pearson correlation coefficient LGN p=0.06, V1 p=0.07, AM-PM p=0.26).

Latency of responses to scrambled-sequence correspond to the anatomical hierarchy of visual areas.

(a) Average response for one representative cell from each visual region, that had high similarity between continuous movie and rearranged scrambled sequence responses (see Methods). Gray response in background corresponds to the chronological scrambled sequence. (b) Cumulative histogram of z-scored correlation between continuous and scrambled-rearranged tuning responses (see Methods). Dotted black line indicates significance threshold of z>2. (c) The latency at which continuous and scrambled-rearranged responses were maximally correlated showed high values (heuristically above 0.25) in a short range of positive latencies for LGN, V1 and AM-PM neurons. This analysis was restricted to neurons tuned in continuous as well as scrambled movies. Similar analysis for hippocampal regions resulted in almost no correlations above 0.25. (d) Cumulative histogram of latencies when the continuous and scrambled-rearranged responses were maximally correlated was smallest for LGN (59.5±4.6ms), and largest for higher visual areas, AM-PM (91.6±1.6ms). Hippocampal regions were excluded, owing to lack of data with correlation about 0.25.

Population vector overlap was narrow at the diagonal with scrambled movie

(a) Population vector overlap between even and odd trials for tuned neurons showing higher overlap along the diagonal for all brain regions. Black lines indicate the -300 and +300 diagonal, whereas the main diagonal is the 0th diagonal. (b) Same as (a) but for untuned neurons, resulting in a salt and pepper overlap without higher correlation around the diagonal. (c) The average overlap along diagonals had a large value in visual regions for the 0th diagonal, which was not true for the untuned neuron population. Average correlation in hippocampal regions was broader and lesser in magnitude compared to visual regions. Similar to Extended Data Fig. 7. Full width at half maximum of the peak – 4.4 frames for LGN, 4.8-V1, 5.2-AM&PM, 7.6-DG, 5.7-CA3, 10.8-CA1 and 15.1-subiculum.

Movie presentation did not alter hippocampal firing rates and the mega-scale coding was unrelated to cluster quality.

(a) More than 50% of hippocampal place cells shut down during maze exploration. In contrast, there was no consistent pattern of neural activation or shutdown during the movie presentation in all brain areas. To make a more conservative estimate, this comparison was restricted to units whose firing rates did not differ by more than 20% across the two movie blocks. Further only the data when the animals were immobile was used to avoid confounding effects of running. (b) The mega-scale index was only weakly correlated with the mean firing rate of a neuron in V1 (Pearson’s correlation coefficient r=0.08, p=7.3×10−5), CA1 (r=-0.14, p=3.5×10−8) and subiculum (r=-0.14, p=0.02), and was uncorrelated for other brain regions (p>0.05) (c) The refractory violations index was uncorrelated with the mega-scale index (lower index means better cluster quality31,79) for all brain regions (p>0.05). To remove potential confounding effect of mean firing rates, we computed the partial correlation coefficient, by factoring out the mean firing rate). (d) Similar to (c), the isolation index (greater isolation index means better cluster quality31,80) was uncorrelated with the mega-scale index for all brain regions (partial correlation coefficient, by factoring out the mean firing rate, p>0.12). Factoring out mean firing rate was deemed necessary since the isolation index was typically positively correlated, and the refractory violations index was typically negatively correlated with mean rate. The mega-scale index comparisons were restricted to movie active, tuned neurons with at least two movie peaks. Note-log spaced axes for (a)-(d)