Overview of dataset, features, and decoding approach.

a) Example of the recorded neuronal activity. Patients watched the complete commercial film 500 Days of Summer while neuronal activity was recorded via depth electrodes. Top row: example movie frames. Due to copyright, the original movie frames have been replaced with images generated using stable diffusion 25. Bottom row: spike trains from ten amygdala neurons of a single patient, where each row shows data from an individual neuron (corresponding ID number given as a label). b) Spike density plot showing the waveforms of each neuron in a (corresponding neuron ID given in top right). Neurons shown include both single- and multi-neurons. c) Distribution of labels across the entire movie (runtime: 83 minutes). Occurrences of character-related features are in magenta, visual transition events in blue, and location events in yellow. d) Distribution of the 2286 neurons across the recorded regions (A: amygdala, H: hippocampus, EC: entorhinal cortex, PHC: parahippocampal cortex, PIC: piriform cortex) for all 29 patients.

Responsive single-neurons in the parahippocampal cortex.

a) Example peri-stimulus activity for representative parahippocampal (PHC) neurons, for labels with a significant PHC response. Upper plots: spike density plot showing the waveforms a given responsive neuron (label name given as title). Middle plots: spike time rasters showing the neuron’s activity surrounding the onset of the corresponding label throughout the movie. Note: onsets for Scene Cuts and Camera Cuts were randomly subsampled to match the number of Summer appearances. Lower plots: average firing rate across 100 ms bins, for 1000 ms before and 1000 ms after the onset or event. Solid lines show the mean across all neurons, within group, and the transparent area shows the 95% confidence interval. b) Region-wise single-neuron activity surrounding the onset of labeled entity. Number of cells exhibiting a significant response over the total number of PHC cells are given as the title, followed by the corresponding percentage. Upper plots (heatmaps): averages of peri-stimulus spike rates per neuron (spikes per 100 ms bin, z-scored across the pseudotrial) for 1000 ms before and 1000 ms after label onset. Each row of the heatmap represents the average binned activity for one neuron. Neurons are sorted in descending order by the p-value of the response—the dotted grey line shows the threshold for responsive neurons (p → 0.001). Lower plots (line plots): average z-scored firing rate across bins. Neurons are separated into responsive (orange line) and non-responsive (black line). Solid lines show the mean for each group of neurons (responsive vs. non-responsive) and the transparent area depicts the 95% confidence interval. Significant differences between the responsive and non-responsive firing rates are shown as solid black lines (*, p → 0.05, cluster permutation test).

Categories of labels can be decoded from the neuronal population activity.

a) Overview of the neuronal decoding pipeline. Spiking data (individual neurons shown as columns) was sectioned into 1600 ms sequences, 800 ms before and after a frame onset (purple highlight; bins shown as horizontal lines, not to scale), and given as input to a two-layer Long Short-Term Memory (LSTM) network. The output of the fully connected layer (FC) predicts the presence of a given label in a frame. b) Assessment of individual-neuron decoding performance by classifying data samples into positive or negative predictions for the label Summer based on the firing activity of a neuron. c) Decoding performances on labels of the movie (reported performance using Cohen’s Kappa, mean performance across five different data splits, error bars indicate standard error of the mean). Labels fall into one of three categories—characters (pink), visual transitions (blue), or location (yellow)—with a separate model for each label. All performances were significantly better than chance level with an alpha level of 0.001. Decoding performances for the logistic regression model in lighter colors. d) Impact of temporal information in spike trains for recurrent neural networks. Trained models were evaluated using temporally altered test data (sequence order shuned, repeated 100 times). Colored bars depict performance without shuning, while grey bars represent shuned scenarios (reported performance using Cohen’s Kappa, results show the mean performance across five different data splits, variance given as standard error of the mean). e) Decoding performance of the main movie character (Summer) for different temporal gap sizes between samples of training, validation, and test sets. Colored temporal gap of 32 s indicates the chosen gap size for all reported performances in this study.

Patient-specific decoding performance.

Decoding performances for the main character (Summer), visual transitions (Scene Cuts), and location (Indoor/Outdoor) are reported using Cohen’s Kappa and compared to the performance obtained from the total population (pooled across all patients, dashed line). a) Decoding performance based on the total population of 2286 units, with neurons pooled across all patients. b-d) Patient-specific decoding performances for Summer, Scene Cuts and Indoor/Outdoor.

Semantic information is distributed differently across MTL regions based on category.

Decoding performances for semantic features, by MTL region. All performances were significantly better than chance level with an alpha level of 0.001 (reported performance using Cohen’s Kappa, mean performance across five different data splits, error bars indicate standard error of the mean. a) Decoding performances for visual transitions and location features. b) Character visibility could be decoded from the entire population of neurons, and with variable performance when training only on individual MTL regions. c) Decoding performances for face-specific character appearances and Presence features, by region.

Responsive neurons drive performance for visual transitions, but not characters.

To assess the contribution of the responsive neurons on decoding (identified in Stimulus-aligned responsive neurons found primarily in parahippocampal cortex), we compared the decoding performance for subpopulations which did or did not contain these neurons. a) Illustration of neuronal sets used in the decoding comparisons (triangles represent neurons). An example of the complete population is shown in the left-most section, which depicts Non-responsive (only) and Responsive (only) cells, with a simulated example of a respective peri-stimulus time histogram (onset raster, grey and magenta). A Non-responsive (size-matched) group (middle section) was randomly subsampled from the total population to have a size-matched comparison to the total set of responsive neurons. The Responsive (size-matched) set (right-most section) consisted of all responsive neurons padded with randomly selected non-responsive neurons to match the total Non-responsive (only) population. b,c) Decoding performances for discussed subpopulations for the character label Summer and Camera Cuts. Number of responsive neurons for the respective subpopulation reported in parentheses.

500 neurons are sufficient to reach peak decoding performance.

a) Decoding performances for the character Summer for subpopulations of top-performing neurons, testing various sizes ranging from 1% to 100% of the full population (absolute numbers of neurons are reported in parenthesis). Mean performance across different splits is reported, and standard error of the mean is visualized by the error bars. b) Number of overlapping neurons across rankings for different sizes of subpopulations of top-performing neurons (pink). As a baseline, we compare the number of overlapping neurons to the number expected by chance (grey), and we observed a notably higher intersection of top-ranked neurons across the splits. Additionally, the overlap between the intersection of top-ranked neurons and the previously defined responsive neurons is shown (black). c, d) Overlapping neurons (in total 78) in subpopulations of 500 top-performing neurons for each ranking were distributed across patients and MTL regions.