A hierarchical coordinate system for sequence memory in human entorhinal cortex

Reviewer #1 (Public review):

Summary:

Shpektor et al. propose a link between how humans learn abstract and hierarchical structures to support memory (for example, remembering the event of the first landing on the moon) and the medial temporal lobe (MTL) and grid cells in particular. Given that there is solid work on how grid cells in different modules jointly encode position in rodents, providing evidence for the existence of a similar code in humans in the non-spatial domain and in relation to memory formation, would constitute a valuable finding.

The authors first examine a small human intracranial dataset to demonstrate that sequence position is decodable in MTL population codes. They then examine behavioral data from two larger groups of participants who passively viewed content presented in a hierarchical sequence and show that errors in recall of positions within that sequence qualitatively match hierarchical predictions. The task design enabled distinct signatures of memory representations at different levels of hierarchy. While there were no multivariate patterns in MTL or any brain region that matched these patterns reliably, a follow-up analysis in MTL revealed a gradient along the anterior-posterior axis, such that lower levels of the hierarchy tended to have representational peaks in more anterior regions of the MTL, which was consistent across the two fMRI datasets.

Major strengths of the study include the novelty of the experimental paradigm and data.

In particular, single cell recording in MTL from a small number of human participants during sequence learning and testing a larger group of human participants on a sequence amenable to hierarchical structure learning, and collecting fMRI data during retrieval.

Furthermore, the paper tackles an important question and does so from both directions, using inspirations from both biology and computational science to navigate it.

The primary weaknesses of the paper are a lack of compelling support for the overarching claim about hierarchical representation and a lack of clarity and consistency about exactly what those hierarchical representations should and do look like. My concerns regarding these weaknesses are described below, and I believe that most, if not all, of them could be addressed through additional analysis and paper revisions.

In the first part of the paper, the authors provide single-cell recordings in MTL, and they report the existence of cells that are sensitive to position (more so than to picture). However, they don't elaborate on this result with a model for an abstract sequence code. This is an issue because one possible explanation for the sequential position decoding is that neurons just fire at the presentation of the first image and decay at different rates, or ramp up toward action or feedback. One might be able to decode the position in sequence from these cells' activity, but can hardly call this an abstract code of position in a sequence. However, the authors don't provide further investigation into what the single-cell result might suggest and move on to a completely different fMRI experiment in the second part of the paper. Being able to decode sequence position does not, in my view, necessarily imply an abstract positional code - and I felt that further analysis of the single unit data would be required to identify what representations gave rise to that decoding ability.

The most compelling evidence that participants were encoding temporal order hierarchically came from behavioral data in the second part of the paper. However, these results were not presented clearly enough to evaluate their reliability and specificity. Figure 2i shows histograms of errors across participants with arrows pointing to bars that apparently correspond to errors of different levels of hierarchy. There are three colored bars, corresponding to errors of one unit at the first, second, or third levels of hierarchy. The first level is not diagnostic of hierarchy, but the other two colored bars appear higher than the colors nearby them. However, my understanding is that these bars correspond to situations with the same tone - which seems like an obvious reason that two positions might be confused, which in my view would weaken the argument for hierarchical encoding. Furthermore, there is no display of variability in the plot or indication of individual differences, so it is hard to tell whether the histogram is dominated by a few participants who made a lot of errors or is reflective of a general tendency across participants.

The fMRI analyses, while creative, raise questions regarding interpretability. The authors report no representations of hierarchical position at any level, either in MTL or across the whole brain, which would typically be taken as a lack of evidence for the representations existing. Follow-up analyses revealed that what shadows of representations do exist seem to line up along the anterior-posterior gradient. But what does that mean if we can't be sure that the representations are really there? Typically, we tally up evidence supporting an overarching claim by testing multiple predictions that are all consistent with the same story - but in this case, it seems that not all such test results are consistent.

In many cases, it was difficult to judge the strength of evidence due to somewhat minimal reporting on the exact hypotheses tested and test statistics.

On a high level, I found the overarching story linking the two datasets together to be somewhat tenuous. While I understand that science rarely rolls out as a coherent story, presenting the authors' valuable experiments in this fashion makes it harder for the reader to digest the information and reach a conclusion. The relevance of the first section of the paper to the second is not immediately apparent. Each section provides somewhat incomplete evidence for a set of claims on its own - but my view was that combining the two studies led to more questions than answers - since the paradigms and measurements are so different.

In conclusion, the authors propose an interesting account of how memories are formed in the human brain, by building an abstract and hierarchical code. The paper identifies a few separate findings that are suggestive of hierarchical abstract memory encoding in the MTL - yet I believe that more work would need to be done to irrefutably support that claim.

Reviewer #2 (Public review):

Overall, I think these are exciting results that make a very nice contribution to the literature. I thought the picture-tagging of sequence locations in the fMRI study was clever, and the across-sequence RSA results were especially compelling. But there are several aspects of the presentation of the results that reduced my confidence and enthusiasm.

(1) This is an unusual paper in that there is one human intracranial study and two fMRI studies. The paradigm for the intracranial study is very different than the fMRI paradigm. The key differences are that the fMRI paradigm is hierarchical, while the intracranial is flat, with no sequence learning component, and the fMRI is auditory, while the intracranial is auditory. The justification for the switch from intracranial to fMRI was that intracranial does not allow anterior-posterior axis analysis, but there are so many differences between the studies that this feels like an awkward transition and justification. Also, anterior-posterior analysis in the MTL may not be feasible in EC with intracranial data, but it can be feasible in the hippocampus, and indeed this could be very worthwhile and relevant to pursue (see point 2).

While the two independent fMRI datasets is a strength, the replications would have been much more compelling had the analysis for the second dataset been preregistered.

(2) The intracranial results are pitched as a novel "abstract coordinate representation" but there is a substantial prior literature on MTL "ordinal position codes", which I believe is the same thing in this paradigm. Most of this literature is in the hippocampus, which is, of course, very relevant given the hippocampal findings here, but there is also evidence for this kind of information in EC, e.g., https://elifesciences.org/articles/45333.

(3) Given the intracranial results in the hippocampus as well as the prior relevant literature on position coding, it was not clear why the hippocampus was not an ROI in the fMRI studies.

(4) It wasn't until reading the Methods section carefully that I understood that the results do not hold for the right EC, only the left. This deserves more acknowledgment.

(5) The use of one-sided t-tests with an alpha of .05 reduced my confidence in the robustness of the results.

Reviewer #3 (Public review):

Summary:

Shpektor et al. investigate how hierarchical sequence structure is represented in the entorhinal cortex (EC) and medial temporal lobe (MTL) using a combination of single-unit recordings and fMRI. In the single-unit recordings, they find abstract representations of ordinal position within short sequences in both the EC and the hippocampus. Next, they use two fMRI datasets to examine representations of hierarchical sequence structure in EC. They find that these representations (1) are organized along a posterior-to-anterior hierarchy, with finer sequence structure represented in posterior EC and coarser structure in anterior EC, and (2) generalize across sensory features, suggesting an abstract representation of sequence position. The authors take these findings as evidence of a non-spatial hierarchical coordinate system in the human EC, analogous to grid cells in rodents.

Strengths:

The methodological approach presented in this study is commendable, combining single-unit recordings in the MTL with two fMRI datasets. The finding of hierarchical and abstract sequence representations in the EC is compelling and is replicated across these datasets and modalities. The manuscript addresses important questions about how the MTL abstracts across experiences that share hierarchical structure, a topic of considerable current interest. As such, the work is likely to be of broad interest to researchers studying these processes in both rodents and humans.

Weaknesses:

In my view, the main weaknesses concern the interpretation of the results, as well as several areas where additional analyses and methodological clarification would strengthen the manuscript. My point-by-point comments are as follows:

(1) I found the evidence for hierarchical and abstract sequence-position representations interesting. However, I am less convinced by the stronger claim that these findings demonstrate a coordinate system analogous to grid-cell coding. The current results appear to provide stronger support for abstract sequence-position coding than for grid-like coding per se. In particular, it is not clear to me that hierarchical sequence representations necessarily imply a grid-like representational format or a coordinate system. Many neural systems exhibit gradients of representational scale along the anterior-posterior axis, both within and across brain regions, without being considered grid-like. I would encourage the authors to clarify why it should be interpreted specifically in terms of a coordinate system rather than more general hierarchical sequence representations. The manuscript would benefit either from a more explicit justification of this link to grid-cell coding or from a more cautious framing of the conclusions.

(2) Relatedly, the emphasis on grid-cell-like coding naturally centers the story on entorhinal cortex (EC). Yet, the single-neuron results indicate that the hippocampus contained a comparable number of position-selective cells. In addition, a large body of literature has implicated the hippocampus in hierarchical representations of memories, sequences, and relational structure. For completeness, I encourage the authors to repeat the key fMRI analyses within the hippocampus, rather than focusing exclusively on EC.

(3) I have some concerns regarding the amount of information available to distinguish representations at different levels of the sequence hierarchy. As I understand the design, each 113-tone sequence was associated with only eight images, meaning there were approximately 14 tones between successive image events. It would be helpful to provide additional detail regarding how image coordinates were assigned and selected, how many observations contributed to each hierarchical level, and how much statistical power was available to distinguish representations at different scales.

(4) I was also uncertain about the potential influence of visual similarity in Dataset 1. My understanding is that the images were not entirely unique but instead consisted of rotated versions of the same images. If so, this visual similarity could potentially complicate the interpretation of representational structure. It would therefore be useful to clarify whether repeated images occurred within the same or different locations in the hierarchy and to provide analyses demonstrating that the reported effects cannot be explained by visual similarity. This seems particularly important given that the corresponding effects in Dataset 2 were weaker.

(5) The rationale for using a custom orderness metric could be explained more clearly. It would be helpful to understand why a custom metric was preferred over rank-order measures such as Kendall's tau or Spearman's rho. I would be interested in seeing whether the orderness results replicate using one of these more conventional metrics.

(6) I had difficulty reconciling the finding that sequence representation effects are stronger across rather than within sequences. Intuitively, I would have expected representations within a sequence to reflect both shared hierarchical position and sensory experience, thus yielding stronger within-sequence effects than across sequences. The opposite pattern seems somewhat counterintuitive. I would appreciate additional discussion of this pattern and what it implies about the nature of the underlying representation. It would also be informative to know whether similar effects are observed elsewhere in the brain, and why EC might preferentially express a purely abstract representation more strongly than representations that additionally share sensory features.

(7) The authors' theory is that hierarchical representations of sequences in EC are used as a scaffold for memory, yet the current paper does not link their behavioural results to their neural ones. I think making such a link would greatly strengthen the results presented here. For example, is displacement error or sequence memory related to ordered representations of the sequence structure?

(8) I thought the manuscript would benefit from a broader discussion of prior work on (1) sequence representations and (2) hierarchical representations in the hippocampus and related regions. As it stands, the manuscript does a good job of situating its findings within the literature on grid cells in the EC but gives comparatively little attention to the literature on sequence representations in the hippocampus. Placing the current findings within this broader body of work would help clarify which aspects of the results are specific to a grid-like interpretation and which may instead reflect more general principles of hierarchical representation in the MTL or across the brain.