Episodic boundaries affect neural features of representational drift in humans

Reviewer #1 (Public Review):

Summary:
This study applied pattern similarity analyses to intracranial EEG recordings to determine how neural drift is related to memory performance in a free recall task. The authors compared neural similarity within and across lists, in order to contrast signals related to contextual drift vs. the onset of event boundaries. They find that within-list neural differentiation in the lateral temporal cortex correlates with the probability of word recall; in contrast, across-list pattern similarity in the medial parietal lobe correlates with recall for items near event boundaries (early-list serial positions). This primacy effect persists for the first three items of a list. Medial parietal similarity is also enhanced across lists for end-of-list items, however, this effect then predicts forgetting. The authors do not find that within- or across-list pattern similarity in the hippocampus is related to recall probability.

Strengths:
The authors use a large dataset of human intracranial electrophysiological recordings, which gives them high statistical power to compare neural activity and memory across three important memory encoding regions. In so doing, the authors also address a timely and important question about the neural mechanisms that underlie the formation of memories for events.

The use of both within and across event pattern similarity analyses, combined with linear mixed effects modeling, is a marriage of techniques that is novel and translatable in principle to other types of data.

Weaknesses:
In several instances the paper does not address apparent inconsistencies between the prior literature and the findings. For example, the first main finding is that recalled items have more differentiated lateral temporal cortex representations within lists than not recalled items. This seems to be the opposite of the prediction from temporal context models that are used to motivate the paper-context models would predict that greater contextual similarity within a list should lead to greater memory through enhanced temporal clustering in recall. This is what El-Kalliny et al (2019) found, using a highly similar design (free recall, intracranial recordings from the lateral temporal lobe). The authors never address this contradiction in any depth in order to reconcile it with the previous literature and with the motivating theoretical model.

The way that the authors conduct the analysis of medial parietal neural similarity at boundaries leads to results that cannot be conclusively interpreted. The authors report enhanced similarity across lists for the first item in each list, which they interpret as reflecting a qualitatively distinct boundary signal. However, this finding can readily be explained by contextual drift if one assumes that whatever happens at the start of each list is similar or identical across lists (for example, a get ready prompt or reminder of instructions). The authors do not include analyses to rule this out, which undermines one of the main findings.

Although several previous studies have linked hippocampal fMRI and electrophysiological activity at event boundaries with memory performance, the authors do not find similar relationships between hippocampal activity, event boundaries, and memory. There are potential explanations for why this might be the case, including the distinction between item vs. associative memory, which has been a prominent feature of previous work examining this question. However, the authors do not address these potential explanations (or others) to explain their findings' divergence from prior work -this makes it difficult to interpret and to draw conclusions from the data about the hippocampus' mechanistic role in forming event memories.

There is a similar absence of interpretation with respect to the previous literature for the data showing enhanced boundary-related similarity in the medial parietal cortex. The authors' interpretation seems to be that they have identified a boundary-specific signal that reflects a large and abrupt change in context, however, another plausible interpretation is that enhanced similarity in the medial parietal cortex is related to a representation of a schema for the task structure that has been acquired across repeated instances.

The authors do not directly compare their model to other models that could explain how variability in neural activity predicts memory. One example is the neural fatigue hypothesis, which the authors mention, however there are no analyses or data to suggest that their data is better fit by a boundary/contextual drift mechanism as opposed to neural fatigue.

Reviewer #2 (Public Review):

Summary: The goal of this study is to clarify how the brain simultaneously represents item-specific temporal information and item-independent boundary information. The authors report spectral EEG data from intracranial patients performing a delayed free recall task. They perform cosine similarity analyses on principal components derived from gamma band power across stimulus duration. The authors find that similarity between items in serial position 1 (SP1) and all other within-list items decreases as a function of serial position, consistent with temporal context models. The authors find that across-list item similarity to SP1 is greatest for SP1 items relative to items from other serial positions, an effect that is greater in medial parietal lobe compared to lateral temporal cortex and hippocampus. The authors conclude that their findings suggest that perceptual boundary information is represented in medial parietal lobe. Despite a robust dataset, the methodological limitations of the study design prevent strong interpretations from being made from these data. The same-serial position across-list similarity may be driven by attentional mechanisms that are distinct from boundary information.

Strengths:
1. The motivation of the study is strong as how both temporal contextual drift and event boundaries contribute to memory mechanisms is an important open question.

2. The dataset of spectral EEG data from 99 intracranial patients provides the opportunity for precise spatiotemporal investigation of neural memory mechanisms.

Weaknesses:
1. Because this is not a traditional event boundary study, the data are not ideally positioned to demonstrate boundary specific effects. In a typical study investigating event boundary effects, a series of stimuli are presented and within that series occurs an event boundary -- for instance, a change in background color. The power of this design is that all aspects between stimuli are strictly controlled -- in particular, the timing -- meaning that the only difference between boundary-bridging items is the boundary itself. The current study was not designed in this manner, thus it is not possible to fully control for effects of time or that multiple boundaries occur between study lists (study to distractor, distractor to recall, recall to study). Each list in a free recall study can be considered its own "mini" experiment such that the same mechanisms should theoretically be recruited across any/all lists. There are multiple possible processes engaged at the start of a free recall study list which may not be specific to event boundaries per se. For example, and as cited by the authors, neural fatigue/attentional decline (and concurrent gamma power decline) may account for serial position effects. Thus, SP1 on all lists will be similar by virtue of the fact that attention/gamma decrease across serial position, which may or may not be a boundary-specific effect. In an extreme example, the analyses currently reported could be performed on an independent dataset with the same design (e.g. 12 word delayed free recall) and such analyses could potentially reveal high similarity between SP1-list1 in the current study and SP1-list1 in the second dataset, effects which could not be specifically attributed to boundaries.

2. Comparisons of recalled "pairs" does not account for the lag between those items during study or recall, which based on retrieved context theory and prior findings (e.g. Manning et al., 2011), should modulate similarity between item representations. Although the GLM will capture a linear trend, it will not reveal serial position specific effects. It appears that the betas reported for the SP12 analyses are driven by the fact that similarity with SP12 generally increases across serial position, rather a specific effect of "high similarity to SP12 in adjacent lists" (Page 5, excluding perhaps the comparison with list x+1). It is also unclear how the SP12 similarity analyses support the statement that "end-list items are represented more distinctly, or less similarly, to all succeeding items" (Page 5). It is not clear how the authors account for the fact that the same participants do not contribute equally to all ROIs or if the effects are consistent if only participants who have electrodes in all ROIs are included.

3. The authors use the term "perceptual" boundary which is confusing. First, "perceptual boundary" seems to be a specific subset of the broader term "event boundary," and it is unclear why/how the current study is investigating "perceptual" boundaries specifically. Second and relatedly, the current study does not have a sole "perceptual" boundary (as discussed in point 1 above), it is really a combination of perceptual and conceptual since the task is changing (from recalling the words in the previous list to studying the words in the current list OR studying the words in the current list to solving math problems in the current list) in addition to changes in stimulus presentation.

4. Although the results show that item-item similarity in the gamma band decreases across serial position, it is unclear how the present findings further describe "how gamma activity facilitates contextual associations" (Page 5). As mentioned in point 1 above, such effects could be driven by attentional declines across serial position -- and a concurrent decline in gamma power -- which may be unrelated to, and actually potentially impair, the formation of contextual associations, given evidence from the literature that increased gamma power facilitates binding processes.

5. Some of the logic and interpretations are inconsistent with the literature. For example, the authors state that "The temporal context model (TCM) suggests that gradual drift in item similarity provides context information to support recovery of individual items" however, this does not seem like an accurate characterization of TCM. According to TCM, context is a recency-weighted average of previous experience. Context "drifts" insofar as information is added to/removed from context. Context drift thus influences item similarity -- it is not that item similarity itself drifts, but that any change in item-item similarity is due to context drift. The current findings do not appear at odds with the conceptualization of drift and context in current version of the context maintenance and retrieval model. Furthermore, the context representation is posited to include information beyond basic item representations. Two items, regardless of their temporal distance, can be associated with similar contexts if related information is included in both context representations, as predicted and shown for multiple forms of relatedness including semantic relatedness (Manning & Kahana, 2012) and task relatedness (Polyn et al., 2012).

Reviewer #3 (Public Review):

Summary:
In this study, the authors analyzed data from 99 individuals with implanted electrodes who were performing a word-list recall task. Because the task involves successively encoding and then recalling 25 lists in a row, they were able to measure the similarity in neural responses for items within the same list as well as items across different lists, allowing them to test hypotheses about the impact of between-list boundaries on neural responses. They find that, in addition to slow drift in responses across items, there is boundary-related structure in the medial parietal lobe such that early items in each list show similarity (for recalled items) and late items in each list show similarity (for not recalled items).

Strengths:
The dataset used in this paper is substantially larger than most iEEG datasets, allowing for the detection of nuanced differences between item positions and for analyses of individual differences in boundary-related responses. There are excellent visualizations of the similarity structure between items for each region, and this work connects to a growing literature on the role of event boundaries in structuring neural responses.

Weaknesses:

1. My primary confusion in the current version of this paper is that the analyses don't seem to directly compare the two proposed models illustrated in Fig 1B, i.e. the temporal context model (with smooth drifts between items, including across lists) versus the boundary model (with similarities across all lists for items near boundaries). After examining smooth drift in the within-list analysis (Fig 2), the across-list analyses (Figs 3-5) use a model with two predictors (boundary proximity and list distance), neither of which is a smoothly-drifting context. Therefore there does not appear to be a quantitative analysis supporting the conclusion that in lateral temporal cortex "drift exhibits a relationship with elapsed time regardless of the presences of intervening boundaries" (lines 272-3).

2. The feature representation used for the neural response to each item is a gamma power time-frequency matrix. This makes it unclear what characteristics of the neural response are driving the observed similarity effects. It appears that a simple overall scaling of the response after boundaries (stronger responses to initial items during the beginning portion of the 1.6s time window) would lead to the increased cosine similarity between initial items, but wouldn't necessarily reflect meaningful differences in the neural representation or context of these items.

3. The specific form of the boundary proximity models is not well justified. For initial items, a model of e^(1-d) is used (with d being serial position), but it is not stated how the falloff scale of this model was selected (as opposed to e.g. e^((1-d)/2)). For final items, a different model of d/#items is used, which seems to have a somewhat different interpretation (about drift between boundaries, rather than an effect specific to items near a final boundary). The schematic in Fig 1B appears to show a hypothesis which is not tested, with symmetric effects at initial and final boundaries.

4. The main text description of Fig 2 only describes drift effects in lateral temporal cortex, but Fig 2 - supplement 1 shows that there is also drift and a significant subsequent memory effect in the other two ROIs as well. There is not a significant memory x drift slope interaction in these regions; are the authors arguing that the lack of this interaction (different drift rates for remembered versus forgotten items) is critical for interpreting the roles of lateral temporal cortex versus medial parietal and hippocampal regions?

5. The parameter fits for the "list distance" regressor are not shown or analyzed, though they do appear to be important for the observed similarity structure (e.g. Fig 3E). I would interpret this regressor as also being "boundary-related" in the sense that it assumes discrete changes in similarity at boundaries.

6. It is unclear to me whether the authors believe that the observed similarity after boundaries is due to an active process in which "the medial parietal lobe uses drift-resets" (line 16) to reinstate a boundary-related context, or that this similarity is simply because "the context for the first item may be the boundary itself" (lines 246-7), and therefore this effect would emerge naturally from a temporal context model that incorporates the full task structure as the "items."