Introduction

Internal models of the world are constructed using memories of past experiences. Knowledge retained from events in the past enables predictions about ongoing and future events, allowing for adaptive behavior. As events unfold, alignment between what is experienced and what is predicted provides validation for the current internal model and signals that continuing to retrieve information from memory may be useful for guiding behavior. On the other hand, divergence of ongoing experience from mnemonic predictions signals that the environment has potentially changed and that past experiences may not be informative or relevant for current and future behavior. Such mnemonic prediction errors (MPEs) may elicit increased attention and arousal and signal that more cognitive control –– the collection of cognitive and neural mechanisms that effortfully influence information processing to be aligned with one’s goals¹ – is needed. Control may support increased attention to and prioritization of bottom-up sensory information from the external environment and the overcoming of memory-guided, but now irrelevant, representations and behavioral responses². The extent to which control, attention, and arousal are increased following MPEs may depend on the strength of the initial prediction³, with stronger predictions resulting in larger MPE responses when violated. Moreover, changes in control, attention, and arousal may not only impact immediate cognition, but are also posited to support future behavior by facilitating the encoding of novel (unexpected) information into memory^4,5. Storage of novel information allows for knowledge updating and therefore more accurate predictions in the future^6,7. While there has been considerable progress in understanding the neurobiological mechanisms underlying MPE detection, the broader impacts of MPEs on systems distributed throughout the brain, namely on cognitive control, attention, and arousal, are not fully understood. Characterizing how these effects interact, vary with mnemonic prediction strength, and impact learning will further elucidate how multiple cognitive and neural systems cooperate to facilitate adaptive behavior when faced with violations of memory-guided predictions about how the world operates.

Insights into the effects of MPEs on control, attention, and arousal have come from studies of event segmentation. According to Event Segmentation Theory^8,9, the mismatch between one’s internal model and ongoing events can elicit a prediction error, leading to event segmentation (i.e., the psychological perception of an event boundary along with accompanying neural computations), which discretizes current experience into “episodes” in memory^10,11. Extant data indicate that increases in pupil size are observed at event boundaries^12,13 and in response to prediction errors of varying content, signs, and modalities^14,15. Such changes in pupil size reflect increases in attention and arousal^16–18; moreover, pupil size is linked with noradrenergic activity in the locus coeruleus (LC)^18,19, which is associated with signaling strong prediction errors^20,21. The LC may be selectively sensitive to prediction errors that are strong²⁰, as LC is theorized to release noradrenaline in response to unexpected uncertainty²¹ and a strong prediction reflects high certainty in prediction accuracy. Strong prediction errors have therefore been characterized as “global model failure signals”²⁰ which may initiate a network reset²² and increase attention, arousal, and control along with learning rates. Pupil size may therefore selectively increase in response to strong, but not weak, MPEs.

Complementing evidence of MPE-related increases in pupil size^12,13, time-resolved electrophysiological measures of visuospatial attention –– specifically, electroencephalography (EEG) posterior alpha power suppression^23,24 –– may also show that attention increases following MPEs, evidenced by greater alpha suppression²⁵. The underlying sources of these measures of attention may be partially shared, given that pupil size and posterior alpha are coupled in certain contexts^26,27. Moreover, LC lesions in monkeys reduce the amplitude of the P3 event-related potential (ERP), another electrophysiological measure of attention/alertness²⁸. However, it remains unclear whether LC activity additionally modulates posterior alpha suppression. Posterior alpha and pupil size also could be differentially influenced by other mechanisms of MPE detection, such as those subserved by the hippocampus and/or prefrontal cortex (PFC)²⁹.

The hippocampus is thought to compute MPEs^30–32, with the CA1 subfield being anatomically well-suited for mismatch detection given that it is a convergence zone for mnemonic predictions from hippocampal subregion CA3 and sensory input from entorhinal cortex^33–36. Functional neuroimaging studies provide support for CA1 as a mismatch “detector” in humans^3,37,38. Moreover, following MPE signals from the hippocampus, LC, or other regions, learning rates in the hippocampus are posited to increase³³. Such increases may be underpinned, at least in part, by a biasing of hippocampal functional connectivity towards an encoding-favored configuration³⁹, which may enhance the integration of bottom-up sensory information during hippocampal encoding computations⁴⁰. Separately, increases in hippocampal theta power^41,42 may increase the likelihood of encoding novel information into memory⁴³. Finally, dynamic causal modeling of magnetoencephalography (MEG) data suggests that mismatch detection in the hippocampus, as assayed by theta power, may be driven by theta-related signals from ventromedial prefrontal cortex (vmPFC)⁴². Hippocampal theta, which may be challenging to detect with scalp EEG⁴⁴, could be a driver of cognitive control and attention signals that are often assayed using scalp EEG.

PFC theta, observed using MEG^42,45 and scalp EEG^46–48, is known canonically as frontal theta⁴⁹ or frontal midline theta⁵⁰ and is considered a marker of cognitive control^49,51. Frontal theta may initiate downstream effects on information processing⁴⁹ by engaging attention to support goal-directed behavior. Frontal theta increases following reward prediction errors^46,48,49 and fosters behavioral adaptation⁴⁶. Whether MPEs are similarly followed by an increase in frontal theta, and how frontal theta responses interact with posterior alpha suppression and pupil size warrants closer examination.

Higher learning rates following MPEs may be mediated, in part, by increases in attention and arousal, which facilitate memory encoding^52,53. Increases in pupil-linked attention and arousal following MPEs may reflect direct involvement of LC activity in fostering learning, given that optogenetic activation of LC neurons in mice enhances memory encoding⁵⁴ and higher LC integrity among older adults is associated with better episodic memory performance⁵⁵. LC effects on learning may be mediated by the hippocampus, which receives profuse projections from LC neurons⁵⁴. Increases in attention and arousal may also be triggered by MPE-driven increases in cognitive control, as instantiated via frontal theta⁴⁹. Higher frontal theta prior to⁵⁶ and during^51,57 encoding is associated with better subsequent memory. In addition, MPE-driven increases in frontal theta may also enhance learning by driving hippocampal theta⁴², which also is associated with better subsequent memory⁴³.

There are thus multiple possible pathways by which MPEs impact cognition. Here we focus on two mechanisms: (1) increases in cognitive control (assayed via frontal theta) and (2) increases in attention and arousal (assayed via posterior alpha and pupil size). To elicit MPEs, we used an associative memory task wherein participants were tested on their memory for verb-picture pairings they had studied four times (strong memories) or once (weak memories). Associative memories (pictures) were cued by the presentation of a studied verb. On a subset of trials, verb presentation was followed by the presentation of a mismatched picture. Report of a mismatch would indicate a violation of a mnemonic prediction and the elicitation of an MPE. Using time-resolved scalp EEG (Experiment 1) and pupillometry (Experiments 1 and 2) measured as participants completed the associative memory task, we document the effects of strong vs. weak MPEs on cognitive control, attention, and arousal. Then, we characterize how control signals interact with attention and arousal signals, and how EEG and pupil assays of attention/arousal relate to each other. Throughout, we examine how these electrophysiological and pupillary measures differ as a function of memory strength to investigate how trial-level assays of prediction strength relate to MPE responses. Finally, we examine whether MPE-related changes in each of these physiological assays relate to the encoding of prediction violating stimuli, as evidenced by subsequent memory for mismatch probes.

Collectively, our findings show that strong MPEs, relative to weak MPEs, evoke greater increases in cognitive control (frontal theta), attention (posterior alpha suppression and pupil), and arousal (pupil). The magnitude of immediate MPE responses not only varies as a function of strong vs. weak predictions, but also with trial-level assays of the amount of attention allocated to the mnemonic prediction (posterior alpha suppression). Moreover, increases in frontal theta, but not posterior alpha or pupil size, are linked to enhanced learning of strong prediction-violating (i.e., mismatch) stimuli.

Methods

We conducted and, for most analyses, pooled data across two experiments that used a common associative memory task paradigm with pupillometry. Experiments 1 and 2 differed in: participant recruitment pool; the duration of the delay between the Associative Memory Task and a final subsequent Recognition Test on the mismatch probes; day of administration of the gradual continuous performance task (gradCPT); inclusion of a Localizer Task (only administered in Experiment 1); and concurrent recording of EEG (only collected in Experiment 1). Images of animate and inanimate objects were taken from the BOSS database⁵⁸, an Animacy x Size stimulus set⁵⁹, and the internet. Objects were isolated from their white or complex backgrounds and were then overlaid on a gray background for stimulus presentation. Scene stimuli in the Localizer Task in Experiment 1 were taken from the “Massive Memory” Scene Categories stimulus set⁶⁰. Luminance of image stimuli in all tasks, with the exception of the gradCPT, were controlled using the SHINE Toolbox⁶¹.

Experiment 1: Associative Memory Task with Same-Day Mismatch Probe Recognition Test (scalp EEG and pupillometry)

Fifty-one healthy young adults (aged 18-35 yrs) enrolled in Experiment 1. One participant withdrew early and all data from this participant were excluded. The final sample included 50 participants (25 female; mean age = 23.1 yrs, s.d. = 3.3). Pupil data were not properly tracked from three participants and were excluded from the corresponding pupil analyses. Participants were recruited from Stanford University and the surrounding community, were right-handed, and had normal or corrected-to-normal vision. The study was approved by the Stanford University Institutional Review Board and all participants provided written informed consent. In a single experimental session, participants completed 2.5hrs of behavioral tasks, with the subsequent recognition memory task occurring 6-min following the associative memory task. Throughout, EEG data were recorded using a 128-channel HydroCell Sensor Net (Electrical Geodesics) and pupillometry data were collected using an Eyelink 1000 eye-tracker; EEG and pupillometry data were collected at a sampling rate of 1000 Hz. Following EEG net application, electrode positions were recorded using a LiDAR sensor on an iPad using the Polycam application (https://poly.cam/). Before each task (ranging from 3–16min in duration), EEG impedances were checked and calibration and validation of the eye-tracker were performed.

Experiment 2: Associative Memory Task with Delayed Mismatch Probe Recognition Test (pupillometry)

Fifty-three healthy young adults (aged 18-23 yrs) enrolled in Experiment 2. Participants were recruited from the Stanford Psychology Credit Pool and had normal or corrected-to-normal vision. Two participants withdrew on Day 1 and all data from these participants were excluded. One participant withdrew before Day 2 of the experiment; Day 1 data from this participant were retained. The final sample included 51 participants (23 female; mean age = 19.8 yrs, s.d. = 1.4). Other data loss/inclusion: one participant’s pupil was not properly tracked throughout the experiment and these pupil data were excluded from corresponding analyses; due to experimenter error, the subsequent memory test was administered incorrectly for four participants on Day 2 and these data were excluded from the corresponding analyses; one participant’s subsequent recognition data were excluded because the participant responded “Old” on all trials. The study was approved by the Stanford University Institutional Review Board and all participants provided written informed consent. In Session 1, participants completed 2hrs of behavioral tasks. Session 2 took place 48hrs after Session 1, during which participants completed the subsequent recognition task and the gradCPT. In both sessions of the experiment, pupillometry data were collected during behavioral tasks using an Eyelink 1000 eye-tracker at a sampling rate of 1000Hz. Calibration and validation were performed before each task.

Experimental Design

1.1 Associative Memory Task

Participants performed an associative memory task, which consisted of an associative encoding phase and an associative retrieval test. The associative memory task was administered in four study-test cycles, with encoding and retrieval phases separated by a 3min distractor task wherein participants reported the parity of sequentially integers ranging from 1 to 100. Each integer was presented immediately following a response. The purpose of the distractor task was to prevent participants from maintaining the studied pairs in working memory prior to the associative retrieval test.

In the associative encoding phase, participants were instructed to learn verb-picture pairs by forming a mental image involving the verb and the image in each pair (Fig. 1a). On each trial, an action verb was presented at the center of the screen for 1.5s. The subsequent inter-stimulus interval (ISI) ranged from 0.5-1.5s after which a picture of an animate or inanimate object was then displayed for 2.5s. During the presentation of the object, participants generated a mental image and reported (using a button press) whether their mental image of the verb-picture pairing was “very vivid” or “not very vivid.” The inter-trial interval (ITI) ranged from 1.25–1.5s. Participants were shown 52 pairs in each encoding phase; 26 were presented four times over the course of an encoding phase cycle (“Strong” pairings) and 26 were presented once (“Weak” pairings). Repeated exposures to strong pairings were distributed across the encoding phase cycle, such that each strong pairing was viewed once during each quartile of trials. As such, each cycle of encoding included 130 trials (26 pairs presented four times and 26 pairs presented once). A black fixation dot was presented at the center of the screen during all trials and participants were instructed to fixate throughout. Verb-picture pairings were randomly assigned per participant, as was the assignment of pairs to the Strong and Weak conditions.

Figure 1:

After the distractor phase, an associative retrieval test examined memory for the verb-picture pairings (Fig. 1b). On each trial, a verb cue was presented for 1s. Participants were instructed to try to bring to mind the associated picture as soon as the cue appeared on the screen. Following a 1s delay, during which only the central fixation dot was presented on the screen, a picture was then presented for 0.75s. Participants were instructed to respond as quickly as possible, using a button press to indicate whether or not they thought they had studied the picture with that verb. Memory for all 52 pairs in the encoding phase of a given cycle was tested. The brief (0.75s) presentation duration of the test probe and instruction to respond rapidly were designed to encourage participants to retrieve the associate upon presentation of the verb cue, setting up a situation in which memory of the associate and perception of the probe either match or mismatch³⁸. To focus on trials on which retrieval likely preceded probe onset, retrieval task trials were retained for analysis if responses were made within 1s after probe onset³⁸. In each study-test cycle, 36 trials were “Match” trials, wherein the picture after the verb matched that studied in the encoding phase, and 16 were “Mismatch” trials, wherein the picture did not match what was studied. On Mismatch trials, the probe picture was from a different category as the correct associate (i.e., if the picture associated with a verb cue was animate, the mismatch probe was inanimate, and vice versa). To control for stimulus novelty during retrieval, mismatch stimuli were initially introduced to participants at the outset of the experiment in a pre-exposure 1-back task (see below). ITIs ranged from 1.25–1.5s. A black fixation dot was presented at the center of the screen at all times and participants were instructed to fixate throughout. The test order of pairings was randomized.

Each study-test cycle included unique verbs and pictures. The assignment of each picture to each verb for Match and Mismatch pairings was randomized across participants.

1.2 Pre-Exposure Task

At the outset of the experiment, prior to the associative memory task, participants performed a 1-back task in which they were shown a sequence of images (each presented for 2s) separated by variable-length ITIs (1.0-1.2s). For each stimulus, participants were instructed to report, using a button press, whether the current stimulus was the same as or different from the previous stimulus. To maximize the duration with which participants attended to each stimulus, they were instructed to provide their response when each stimulus had left the screen. There were 96 unique stimuli in total (48 animate and 48 inanimate objects); 64 of the stimuli were later presented as mismatch probes during the retrieval phase of the associative memory task (see above), and all stimuli were later presented in the subsequent recognition test; the order of the stimuli was randomized. Of the 32 items not presented in the retrieval phase of the associative memory task, nine were repeated in the 1-back task. A black fixation dot was presented at the center of the screen at all times during all trials and participants were instructed to fixate on the central dot throughout the task. The purpose of this pre-exposure task was to familiarize participants to the stimuli that would later serve as mismatch probes during the associative memory task. With mismatch probes that were familiar rather than novel, item novelty would not be a primary source of memory evidence guiding decisions in the retrieval phase of the associative memory task.

1.3 GradCPT

To obtain a subject-level assay of attention, we collected 5-min of gradCPT⁶² data in each experiment with modified stimulus presentation timings. In Experiment 1, participants performed the gradCPT after four cycles of the associative memory task to introduce a delay before the surprise recognition test (see below) and prevent maintenance of material studied in the associative memory task in working memory. In Experiment 2, where the surprise recognition test took place 48hrs after the associative memory test, the gradCPT was performed after the recognition test in Session 2. In the gradCPT, a centrally presented picture gradually transitioned between different mountain and city images (90% of images were of cities and 10% were of mountains). Each image was presented for 0.05s and transitioned to the next image over 0.8s. Participants were instructed to press the spacebar when they saw a city scene and to withhold responding for mountain scenes. Performance in the gradCPT was not central to the core hypotheses and is not reported in the current manuscript.

1.4 Recognition Test

To assess the effects of MPEs on learning, a recognition test was administered. In this surprise recognition memory test, participants were tested on their memory for the 64 mismatch probes shown during the associative retrieval test. Non-mismatch foils on the recognition test consisted of the 32 stimuli from the pre-exposure 1-back that were not repeated in that task nor used as mismatch probes during the associative retrieval test; as such, they were old but not associated with MPEs. An additional 64 novel foils were included in the recognition test. On each recognition trial, participants reported whether they had seen the image at any point during the experiment, by pressing one button for “old” and another for “new”. Each image was displayed for a minimum of 1.2s and a maximum of 2s, such that if participants failed to respond within 1.2s, the image remained on the screen for an additional 0.8s. ITIs varied from 0.5–1.5s and the order of the stimuli was randomized. A black fixation dot was presented at the center of the screen at all times and participants were instructed to fixate. In Experiment 1, the recognition test was administered after the gradCPT. In Experiment 2, the recognition test was the first task administered in Session 2. The recognition test examined subsequent memory for mismatch probes either ∼5-10min (Expt 1) or ∼48hrs (Expt 2) after the associative memory task.

1.5 Localizer Task

At the end of Experiment 1, participants performed a second 1-back task designed to enable training of an EEG classifier that discriminates between evoked electrophysiological activity patterns associated with animate and inanimate objects. The localizer task was included in Experiment 1 to address other research questions and results from this task are not reported in the current manuscript. For completeness, the task is described here. On each trial, an image of an animate object, an inanimate object, or a scene⁶⁰ was displayed on the screen for 2s. Images were separated by a 1.0-1.2s ITI. The instructions were the same as for the pre-exposure 1-back task, such that participants were instructed to report whether a given image was the same as or different from the previous image, and to respond after the image had left the screen. The stimuli were novel (i.e., had not appeared in any other experimental phase), with 270 unique stimuli presented in total (90 per category); the order of the stimuli was randomized. Ten percent of the stimuli repeated once (as 1-backs) over the course of the task.

EEG and Pupillometry Analyses

2.1 EEG Preprocessing

EEG data from the associative retrieval test in Experiment 1 are a focus of the present manuscript, given the open questions/hypotheses regarding mnemonic prediction errors. Preprocessing of these EEG data was performed using the MNE package⁶³. EEG data were first downsampled from 1000 Hz to 250 Hz and then filtered with a high-pass filter of 1 Hz and a low-pass filter of 30 Hz. Bad channels were then identified via visual inspection of the power spectrum and were subsequently interpolated. An ICA was fitted to the data; ICs resembling blinks and muscle artifacts were identified automatically using functions in MNE and by visual inspection. Data were then re-referenced to the average.

2.2 EEG Spectral Analyses

For analyses of spectral power in the associative retrieval task, trial-level power was computed on epochs of data from −1s to 5s with respect to trial onset. Epoched data were decimated by a factor of two to reduce computation time, yielding a sampling frequency of 125Hz. Epochs wherein the voltage of any channel of interest exceeded 100 μV were excluded from analyses. Frontal theta (4-7Hz) power was computed using electrodes 5, 6, 7, 11, 12, 13, 106, 112 and posterior alpha (8-12Hz) power was computed using electrodes 62, 67, 71, 72, 75, 76, 77 (Supplementary Fig. 1). These electrodes were selected based on prior research on attention lapsing and goal coding⁶⁴. Instantaneous power was computed at each time point using a Morlet wavelet with 7 cycles and epochs were subsequently trimmed to 0s–4s after trial onset for subsequent analyses.

2.3 Pupillometry Preprocessing

Pupillometry size data from both experiments were preprocessed⁶⁵, with data first being decimated from 1000 Hz to 100 Hz. A pupil dilation speed threshold was determined by first computing the median absolute deviation (MAD) in pupil size in each block of associative retrieval task data. Visual inspection was used to assign 10 as the constant value for multiplicatively scaling the MAD. A rejection threshold was then determined as the sum of this value and the median dilation speed. Samples of data for which the change in pupil diameter from the previous sample exceeded this threshold were removed. Blinks were padded by 50ms on each side and linear interpolation was used to fill in missing data due to blinks or dilation artifacts. The data were then smoothed using a third order Butterworth filter and additional outlier samples were identified using the dilation speed threshold procedure described above, but with a constant factor of 0.2. This procedure of interpolation, smoothing, and outlier detection was then performed a second time. Finally, trials were excluded if more than 10% of data points were missing in the original time series. Data were subsequently de-trended by taking the residuals of a linear regression model fit to the time series for each run of each task. For all analyses, the z-scored residuals were analyzed.

2.4 Statistical Models for Time Series Analysis

Trial-level regression analyses were conducted on the frontal theta, posterior alpha, and pupil time series from the associative retrieval test to identify temporal clusters that were sensitive to the factors of Mismatch (i.e., mismatch vs. match probes) and/or Strength (i.e., strong vs. weak associative pairs). For each participant and each time point, a linear regression model testing main effects of Mismatch and Strength, and a Mismatch × Strength interaction was run using R⁶⁶ to compute beta weights for each regressor. For these models, only correct trials (i.e., hits to match probes and correct rejections to mismatch probes) were included to identify temporal clusters associated with memory retrieval strength and/or MPEs. For frontal theta and posterior alpha analyses, two participants in Experiment 1 were excluded because they were missing trials in at least one of the four conditions and model fitting could not be performed; for pupil analyses, two participants from Experiment 2 were excluded for missing trials in at least one of the four conditions. For analyses of subsequent recognition memory for mismatch probes, 10 participants from Experiment 1 and seven from Experiment 2 were excluded for missing trials in at least one of the four conditions; of these, seven participants from Experiment 1 were excluded from the frontal theta and posterior alpha time series analyses of subsequent recognition memory.

For primary pupil analyses, data were pooled across Experiments 1 and 2 (N=94); see Supplement for results by experiment. The beta weights computed for each time point were averaged across participants to generate time series for the effects of Strength, Mismatch, and Mismatch × Strength interactions for frontal theta, posterior alpha, and pupil size. Significance testing for each regressor was performed by running a one-sample t-test at each time point. To determine whether temporal clusters of significant beta weights were significant, a permutation test was run in which the sign of each participant’s beta weights was randomly assigned; this procedure was repeated 1000 times and t-tests for each time point were computed on each iteration to generate a null distribution. Significance testing for each temporal cluster was performed by computing the probability of a cluster that was at least as long in the null distribution.

2.5 Pupillary Temporal PCA

In addition to canonical pupil size analyses, the pupil data were deconstructed using a temporal principal components analysis (PCA). Temporal PCA was used to obtain trial-level assays of attention and arousal with (1) greater sensitivity to cognitive processes engaged during the task and (2) that could be related to electrophysiological measures on a trial-by-trial basis. PCA was performed on the pupil time series aggregated across all correct trials and all participants (N=96); the size of the input matrix was the number of trials × the number of time points (10346 × 401 time points). We retained the first six components, using at least 95% variance explained as the cutoff; cumulative variance explained by all six components was 96.16%. To obtain interpretable scores for each trial and each PC, the loadings were then orthogonalized using a varimax rotation on the rotation matrix produced by the PCA. For each component, we fit linear mixed effects models using lme4⁶⁷ to examine the effects of Mismatch and Strength (score ∼ Mismatch * Strength + (Mismatch * Strength | participant)). We also examined how scores related to reaction time (RT) on hit and correct rejection trials (RT ∼ score ∼ Mismatch * Strength + (Mismatch * Strength | participant)). A maximal random effects approach was adopted as recommended by⁶⁸; for models that failed to converge, we successively simplified the random effects structure.

Results

1. Effects of mnemonic prediction errors

Integrating behavior, electrophysiology, and pupillometry, we examined the core hypothesis that mnemonic prediction errors (MPEs) trigger an increase in cognitive control, attention, and arousal. We tested this hypothesis through condition comparisons of prediction errors following strong vs. weak associative encoding, and through trial-level assays of varying associative retrieval/prediction strength (via readout from EEG spectral power during retrieval; canonical and principal component pupil assays). We then explored whether such increases in control, attention, and/or arousal predict subsequent memory for the unexpected/mismatch stimuli.

Prediction strength impacts match/mismatch choice behavior

In a combined sample of 101 participants, associative memory d’ (Z_{associative hit}–Z_{false alarm}) was higher for strong (3.11±0.93) compared to weak pairings (1.67±0.76) (t₁₀₀=20.988, p<0.001; Fig. 1c; see Supplementary Fig. 2 for data separated by experiment). Consistent with participants retrieving the associate prior to probe onset, probe-locked response times (RT) were relatively rapid, with across-participant mean RT occurring prior to offset of the probe. RTs were faster for strong compared to weak trials (main effect of Strength: β=0.211, CI=[0.202, 0.219], p<0.001), correct than incorrect trials (Accuracy: β=0.200, CI=[0.171, 0.229], p<0.001), and match than mismatch trials (Mismatch: β=0.165, CI=[0.154, 0.175], p<0.001). The RT difference between (a) correct and incorrect trials was larger for strong than weak pairings (Strength × Accuracy: β=−0.167, CI=[−0.194, −0.140], p<0.001); (b) match and mismatch trials was larger for strong compared to weak pairings (Strength × Mismatch: β=㌒0.126, CI=[−0.141, −0.111], p<0.001); and (c) correct and incorrect trials was larger for match than mismatch trials (Accuracy × Mismatch: β=−0.152, CI=[−0.189, −0.115], p<0.001). The Strength × Accuracy × Mismatch interaction also was significant (β=0.140, CI=[0.096, 0.184], p<0.001). Collectively, these outcomes indicate that: participants followed task instructions, with retrieval of the cued associate tending to begin prior to probe onset, thus establishing conditions for a MPE on mismatch trials; memory was superior following repeated learning (higher d’ and faster RTs in the strong condition); responses were slower when the probe was a mismatch; and strong MPEs had a greater impact on RTs relative to weak MPEs.

Strong MPEs signal a need for cognitive control

Frontal theta is posited to mark the engagement of cognitive control and support goal-directed behavior⁴⁹. We examined how MPEs impact frontal theta power, leveraging trial-level regression models to examine the effects of match vs. mismatch probes and condition-level differences between strong vs. weak predictions. We focus here on temporal clusters where there were effects of Strength, Mismatch, and/or an interaction that emerged after probe onset. Because some participants had few to no error trials, models only included correct trials (for models including incorrect trials, see Supplementary Fig. 3 and Supplementary Tables 1-3).

Post-probe onset, there was a main effect of Mismatch (0.032 to 0.704s; mean β=0.35, 95% CI=[0.26, 0.44]; p<0.001; Fig. 1d), a main effect of Strength (0.504 to 0.872s; β=0.14, CI=[0.07, 0.21]; p<0.001) and a Mismatch × Strength interaction (0.048 to 0.728s; β=−0.35, CI=[−0.46, −0.24]; p<0.001; all p-values for temporal clusters are permutation-corrected, unless indicated otherwise). The interaction reflects a greater difference in frontal theta power between mismatch and match trials on strong compared to weak trials (Mismatch effect size, strong: Cohen’s d=1.28; weak: d=0.85). These data suggest that strong MPEs elicit a greater increase in cognitive control relative to weak MPEs. Qualitatively, the main effect of Strength emerged later than the main effect of Mismatch, suggesting that the increase in frontal theta evoked by the probe was more sustained for weak compared to strong trials (or, as a corollary, that the greater control elicited by strong MPEs enabled more rapid resolution of conflict and ultimate choice selection). Consistent with this possibility, changes in frontal theta elicited by strong MPEs primarily preceded responses (Supplementary Fig. 4a).

After probe offset (i.e., during the ITI), there was a main effect of Mismatch (0.394 to 0.698s; β=−0.07, CI=[−0.12, −0.02]; p<0.001), a main effect of Strength (0.466 to 0.626s; β=−0.06, CI=[−0.12, 0.00]; p<0.001), and a Mismatch × Strength interaction (0.194 to 0.738s; β=0.11, CI=[0.05, 0.17]; p<0.001); the signs of all three effects were inverted relative to the post-probe onset effects. Indeed, the interaction indicates that the difference in the magnitude of the post-probe offset dip in frontal theta between match and mismatch trials was larger for strong compared to weak trials.

Strong MPEs increase attentional engagement

Attentional engagement can be indexed by multiple physiological measures, including a decrease in posterior alpha power²³. After probe onset, there was a main effect of Mismatch (0.512 to 0.816s; β=−0.13, CI=[−0.18, −0.08]; p<0.001; Fig. 1e), a main effect of Strength (0.504 to 1.480s; β=−0.18, CI=[−0.22, −0.14]; p<0.001), and a Mismatch × Strength interaction (0.536 to 1.376s; β=0.15, CI=[0.10, 0.20]; p<0.001). The interaction indicates that, relative to weak MPEs, strong MPEs evoked greater posterior alpha suppression, putatively marking greater attentional engagement. Moreover, weak matches evoked greater alpha suppression relative to strong matches. Response-locked analyses showed that responses coincided with a drop in posterior alpha and strength-modulated MPEs emerged primarily after responses (Supplementary Fig. 4b).

Attentional engagement and arousal also can be indexed by analyses of pupil size. Across 94 participants with suitable pupil data and sufficient trial counts per condition, we found that relative to probe onset, there was a main effect of Mismatch (−0.14 to 2.00s; β=0.16, CI=[0.09, 0.23]; p<0.001; Fig. 1f), a main effect of Strength (1.29 to 2.00s; β=0.22, CI=[0.15, 0.29]; p<0.001), and a Mismatch × Strength interaction (0.52 to 2.00s; β=−0.32, CI=[−0.40, −0.24]; p<0.001) (see Supplementary Fig. 5 for data separated by experiment). Complementing the posterior alpha findings, the interaction indicates that the increase in pupil diameter was larger for strong compared to weak MPEs. This strength-modulated MPE effect specifically emerged after responses (Supplementary Fig. 4c).

Distinct pupil features reflect mnemonic prediction errors and attentional orienting

We leveraged temporal PCA over pupil data (N=96) to gain further insights into the sequence of cognitive operations that were engaged over the course of the associative retrieval trials and in response to MPEs. As detailed in corresponding Results sections below, the temporal profile of each of six components (Fig. 2a) reveal relationships with trial structure, condition, and retrieval behavior, and suggest that the analysis isolated: (a) the effects of pupil-linked arousal at the time of cue presentation on subsequent associative retrieval success (PC2); (b) responses to the strength of the cue (PC2 and PC5); (c) engagement of cognitive effort in support of controlled retrieval (PC5); (d) memory retrieval strength (PC1); (e) responses to probe onset and detection of mismatch stimuli (PC6); (f) responses to MPEs (PC3 and PC4); and (g) post-error orienting responses (PC4). In the following sections, we concentrate on and further delineate specific pupil components that most closely relate to our core hypothesis (for discussion of PC2 and PC6, see Supplementary Fig. 6-7 and Supplementary Tables 4-8).

Fig. 2:

By relating PC scores to behavior and experimental conditions (memory strength and match/mismatch), we identified two PCs (PC3 and PC4) that were sensitive to MPEs (Fig. 2b and 2c). PC3, explaining 19.29% of variance, peaked 0.99s after probe onset (Fig. 2b; 95% CI = [0.97, 1.01]; loading = 0.88, 95% CI=[0.88, 0.90]) and had a full width at half maximum (FWHM) of 1.18s (CI=[1.16, 1.21]). PC4, explaining 14.68% of variance, peaked towards the end of the trial (Fig. 2c; 1.82s after probe onset, CI=[1.81, 1.83]; loading = 0.93, CI=[0.92, 0.93]). To interpret these components and test their sensitivity to MPEs, we examined how the scores for each PC related to prediction strength and match/mismatch conditions; we further assessed how RTs related to PC scores and experimental conditions.

For PC3, there was a Mismatch × Strength interaction (Table 1), indicating a strength-sensitive MPE pupil response. Pairwise comparisons showed higher PC3 scores for strong mismatches (correct rejections) than strong matches (hits) (β=−0.079, CI=[−0.133, −0.024], p=0.005); for weak pairings, PC3 scores were higher for matches than mismatches (β=0.093, CI=[0.034, 0.152], p=0.002). Using data from all trials, including misses and false alarms, tPCA extracted highly similar components (r(399)>0.99, p<0.001 for all six PCs); using these component scores, we performed a model comparison to examine whether Accuracy explained a significant amount of variance in PC3 scores. Indeed, an augmented model including effects of and interactions with Accuracy (in addition to Mismatch and Strength) fit PC3 scores better than a model that only included effects of and an interaction between Mismatch and Strength (χ²(8)=63.90, p<0.001; Table 2). The enhanced fit of the augmented model further supports the interpretation that PC3 reflects a strength-sensitive MPE pupil response.

Table 1:

Table 2:

Altogether, the RT findings suggest that stronger MPEs (i.e., faster mismatch RTs) evoked larger pupillary responses. This was manifest in there being a Mismatch × Strength × PC3 score interaction (Table 3), with post-hoc analyses showing that higher PC3 scores for strong mismatches were coupled with faster responses (β=−0.046, CI=[−0.060, −0.031], p<0.001), whereas RTs and PC3 scores were unrelated for strong matches (β=0.009, CI=[−0.003, 0.021], p=0.143); the difference between these slopes was significant (β=0.055, CI=[0.038, 0.072], p<0.001). Furthermore, RTs and PC3 scores were more strongly related for strong compared to weak mismatches (β=−0.032, CI=[−0.053, −0.012], p<0.001). Together, these results further support the interpretation that PC3 is a strength-sensitive MPE component that is sensitive to differences in prediction strength at both the condition-level (strong vs. weak) and trial-level (e.g., trial-level strong mismatch RTs).

Table 3:

A different pattern was observed for weak trials, with no relationship between PC3 scores and RTs for mismatches (β=−0.014, CI=[−0.028, 0.001], p=0.076), a negative relationship for matches (β=−0.002, CI=[−0.035, −0.010], p=0.001), and no significant difference between these slopes (β=−0.009, CI=[−0.027, 0.010], p=0.599). These outcomes are consistent with the behavioral findings that MPEs were experienced primarily on strong but not weak trials (Fig. 1). Altogether, these findings indicate that when stronger mnemonic predictions (as indexed by shorter RTs) were violated, larger pupillary responses were evoked.

Similar to PC3, PC4 responses displayed strength-sensitive MPE modulation (Fig 2c; Table 4). A Mismatch × Strength interaction revealed a larger difference between mismatch and match scores in the strong than the weak condition; strong mismatch scores were higher than strong match scores (β=−0.275, CI=[−0.327, −0.224], p<0.001) and weak mismatch scores were lower than weak match scores (β=0.202, CI=[0.145, 0.258], p<0.001). PC4 scores from the dataset including error trials were better fit by a model that additionally included effects of and interactions with Accuracy (χ²(8)=213.83, p<0.001); moreover, component scores were highest for strong error trials (Table 5; Accuracy effect; Accuracy × Strength).

Table 4:

Table 5:

The RT model showed that weaker mnemonic predictions (or greater difficulty in correctly retrieving memories), as indexed by longer RTs, were associated with higher PC4 scores (Table 6; effect of PC4 score). The relationship between PC4 scores and RTs was stronger for weak compared to strong trials (PC4 score × Strength) and for match compared to mismatch trials (PC4 score × Mismatch). There was no three-way interaction (PC4 score × Mismatch × Strength). As such, more effortful memory retrieval, manifesting as slower RTs and/or erroneous responses, elicited larger delayed pupillary responses.

Table 6:

Together, these results suggest that PC4, which peaked after responses, reflects post-decision cognitive processing and perhaps a change in attentional engagement. PC4 appears to primarily reflect an orienting response to errors – not only strong erroneous predictions, but also erroneous memory decisions. This component is sensitive to the amount of effort (assayed by RTs) exerted to generate those predictions, and the strength of those errors (strong vs. weak). Elevated PC4 scores on weak hits, where no error was made, may instead reflect retrieval practice and thus internally oriented attention. On these trials, when cue-elicited retrieval may have been weaker, the probe may have provided additional support for pattern completion of the learning episode for that association, and this engagement in memory retrieval may have elicited pupil dilation (c.f., Strength effect in Fig. 1f). Overall, PC4 may therefore reflect an attentional orienting response that, depending on the relative success of memory retrieval and probe identity, may direct attention internally or externally.

PC3 and PC4 scores were differentially associated with RTs, with PC3 showing an inverse association during strong MPEs and PC4 showing positive associations in all conditions. This differential relationship suggests for stronger MPEs (i.e., strong MPEs that were responded to quickly), higher PC3 scores may reflect more rapid resolution of conflict and lower PC4 scores may therefore reflect less attentional orienting. In other cases, when PC4 scores are high, conflict resolution may have taken longer and subsequent orienting responses may not resolve before the onset of the subsequent trial. As such, PC3 and PC4 may have differential effects on cognition and behavioral performance on subsequent trials.

Carryover effects of pupil-linked MPE responses on cognition: enhanced readiness-to-remember

Next, we examined how the effects of MPEs on pupil responses (PC3 and PC4) carried forward in time, focusing on strong MPEs, as these elicited larger effects on RTs (Fig. 1c), cognitive control (Fig. 1d), attention (Fig. 1e and Fig. 1f), and arousal (Fig. 1f). Immediate pupil responses following memory decisions (PC3) impacted retrieval success on the subsequent trial. Higher PC3 scores on strong MPE trials were associated with a greater likelihood of remembering on the next trial (main effect of PC3 scores: odds ratio (O.R.)=1.492, CI=[1.057, 2.106], p=0.023). This relationship did not differ as a function of Strength or Mismatch (PC3 score × Strength: O.R.=0.755, CI=[0.512, 1.114], p=0.157; PC3 score × Mismatch: O.R.=0.570, CI=[0.305, 1.062], p=0.077) and there was no three-way interaction (O.R.=1.316, CI=[0.628, 2.757], p=0.467). In contrast, PC4 scores did not predict memory performance on the next trial (no main effect of PC4 scores: O.R.=0.952, CI=[0.657, 1.381], p=0.797; PC4 × Strength: O.R.=0.861, CI=[0.568, 1.305], p=0.481; no PC4 × Mismatch interaction: O.R.=1.458, CI=[0.760, 2.798], p=0.257; no PC4 × Strength × Mismatch interaction: O.R.=0.877, CI=[0.397, 1.937], p=0.746).

These outcomes indicate that enhanced attention and arousal immediately evoked by strong MPEs impact readiness-to-remember^52,69 on the subsequent trial. These effects could be mediated in part by an enhancement of preparatory attention following strong MPEs (see Supplementary Fig. 6 for effects on PC2 scores). Altogether, these results indicate that the increases in attention and arousal following memory decisions and strong MPEs adaptively enhance future memory behavior.

Summary

Following probe onset, detection of a mismatch may elicit engagement of cognitive control (i.e., an increase in frontal theta) to facilitate comparison of mnemonic and sensory evidence. Sustained attention to mnemonic evidence and evidence accumulation during the comparison process may drive an anticipatory pupil response up until the decision; the amplitude of this response may ultimately be impacted by the magnitude of the MPE (PC3) and in turn impact attention and readiness-to-remember on the subsequent trial. Upon completion of the trial, after a response is made, additional cognitive processes may lead to subsequent changes in pupil size (PC4) that reflect additional changes in attention/arousal.

We next investigated how control processes interacted with attention/arousal by examining relationships between MPE-related frontal theta, posterior alpha, and pupil responses.

Frontal theta predicts pupillary responses, but not posterior alpha

Altogether, our behavioral, electrophysiological, and pupillary findings provide converging evidence that stronger MPEs were experienced in the strong condition relative to the weak condition. Accordingly, we examined the relationship between MPE-related impacts on frontal theta, posterior alpha, and pupil, focusing on strong trials and the effects of matches and mismatches.

Changes in attention and arousal following MPEs may be elicited by error detection alone or by a separate signal that more control is needed. To investigate this question, we examined whether putative cognitive control signals, assayed by increased frontal theta power, predicted changes in posterior alpha and pupil size (Fig. 2d). First, we observed no relationship between MPE-evoked frontal theta (second Mismatch × Strength cluster) and posterior alpha (third Mismatch × Strength cluster) (Fig. 2e; main effect of frontal theta: β=0.010, CI=[−0.012, 0.032], p=0.369; frontal theta × Mismatch interaction: β=−0.023, CI=[−0.056, 0.011], p=0.182]). Second, while higher frontal theta on strong mismatches was associated with higher decision-related pupil components — both immediate (Fig. 2f; PC3: main effect of frontal theta: β=0.050, CI=[0.008, 0.092], p=0.020) and lagged (Fig. 2g; PC4: main effect of frontal theta: β=0.054, CI=[0.005, 0.104], p=0.033) — the strength of this coupling did not differ between matches and mismatches for PC3 (frontal theta × Mismatch interaction for PC3: β=0.001, CI=[−0.056, 0.059], p=0.966) or PC4 (frontal theta × Mismatch interaction for PC4: β=0.024, CI=[−0.055, 0.103], p=0.544]). Finally, frontal theta predicted mean pupil diameter (Mismatch × Strength cluster) (Supplementary Fig. 8a; main effect of frontal theta: β=0.046, CI=[0.007, 0.084], p=0.022), with no difference in the strength of this relationship between mismatch and match trials (frontal theta × Mismatch interaction (β=0.038, CI=[−0.021, 0.097], p=0.212).

Together, these results indicate that the need for cognitive control, as assayed by frontal theta, was associated with immediate and lagged pupil-linked increases in attention and arousal (i.e., PC3, PC4, and mean pupil diameter). Although frontal theta was higher for strong mismatches than strong matches (Fig. 2d), the strength of the relationship between frontal theta and pupillary responses did not differ as a function of MPE detection. Increases in attention as assayed by posterior alpha were independent of frontal theta control signals. Thus, these outcomes suggest that some changes in attention following MPEs are elicited by error detection alone (i.e., posterior alpha), whereas other changes in attention/arousal (i.e., pupil) are associated with control signals.

Posterior alpha couples with lagged pupillary orienting responses, but not immediate decision-related pupil responses

We next examined whether MPE-driven changes in posterior alpha were associated with pupil-linked attention and arousal responses. There was no relationship between posterior alpha power and PC3 scores (Fig. 2h; main effect of posterior alpha: β=−0.005, CI=[−0.077, 0.067], p=0.894; PC3 score × Mismatch: β=0.017, CI=[−0.108, 0.141], p=0.793). By contrast, PC4 scores and mean pupil diameter were negatively coupled with posterior alpha power (Fig. 2i; main effect of posterior alpha on PC4 scores: β=−0.155, CI=[−0.229, −0.080], p<0.001; Supplementary Fig. 8b; main effect of posterior alpha on mean pupil: β=−0.107, CI=[−0.182, −0.032], p=0.007), indicating that greater posterior alpha suppression (a marker of attentional engagement) predicted larger pupillary orienting responses (a marker of increased arousal and/or attention). There was no PC4 score × Mismatch interaction (β=−0.053, CI=[−0.192, 0.086], p=0.452) nor was there a mean pupil × Mismatch interaction (β=0.028, CI=[−0.112, 0.168], p=0.687). As such, greater attentional engagement following memory decisions, as assayed by decreases in posterior alpha, was followed by stronger pupillary orienting responses.

Together, these findings suggest that the magnitude of alpha suppression following memory decisions is associated with lagged orienting pupillary responses, but not immediate decision-related pupillary responses. In other words, the initial MPE-evoked changes in posterior alpha and pupil size (PC3) are independent of each other.

Summary

In sum, strong MPEs evoked changes in frontal theta (Fig. 1d), posterior alpha (Fig. 1e), and pupil diameter (Fig. 1f). While the changes in frontal theta and posterior alpha elicited by strong MPEs were independent of each other, changes in frontal theta and posterior alpha were both associated with the magnitude of late pupil responses (PC4). Initial decision-related pupil responses (PC3) were associated with frontal theta but not posterior alpha. These relationships did not differ between Match and Mismatch, suggesting that coupled changes in attention/arousal and control during/following memory decisions are not specific to MPEs. Differences in the patterns of relationships for pupil components and mean pupil diameter highlight the increased sensitivity offered by temporal PCA in identifying pupil responses to varying cognitive processes engaged at different timepoints. Critically, these outcomes highlight the distinct temporal profiles of the relationship between control and attention/arousal: pupillary assays of increased attention/arousal evoked by memory decisions are yoked to cognitive control signals (i.e., PC3, PC4, mean pupil), whereas increases in attention as assayed by posterior alpha only relate to later pupil responses (i.e., PC4 but not PC3 or mean pupil).

2. Controlled retrieval and memory retrieval strength

MPEs reflect the mismatch between a cue-evoked retrieved memory (i.e., a mnemonic prediction) and the future state of the world (i.e. the probe). We next examined how cue-evoked electrophysiological and pupillary markers of processes that precede MPEs –– namely cognitive control and memory retrieval strength –– relate to MPE responses. First, we examined how cue-evoked differences between strong and weak trials (which presumably reflect, at least in part, differences in the strength of retrieved memories) manifest in frontal theta, posterior alpha, and pupillary signals prior to the onset of the match/mismatch retrieval probe. Next, we identified trial-level indices of pupil-linked cognitive effort and memory retrieval signals. Finally, we examined whether trial-by-trial variability in cognitive control and retrieval strength interact with MPE responses.

Memory retrieval signals in frontal theta, posterior alpha, and pupil

To identify temporal clusters reflecting engagement of cognitive control during memory retrieval and memory retrieval strength, we examined cue-evoked and delay period main effects of Strength. We observed three key findings. First, frontal theta varied by memory strength (main effect of Strength from 0.768 to 1.744s after cue onset; β=0.07, CI=[0.03, 0.12]; p<0.001; Fig. 1d), with frontal theta power being higher on weak compared to strong trials. This difference may reflect increased engagement of cognitive control during attempts to retrieve weak associations. Second, for posterior alpha, there were main effects of Strength from 0.128 to 0.232s (β=0.07, CI=[0.01, 0.12]), 0.952 to 2.112s (β=0.11, CI=[0.07, 0.15]); Fig. 1e) after cue onset. For both clusters, posterior alpha power was lower on strong than weak trials, with this cue-evoked alpha suppression possibly reflecting attention to strongly familiar retrieval cues and/or to retrieved contents. Third, pupil diameter paralleled the findings for posterior alpha, showing a main effect of Strength from 0.91 to 2.94s after cue onset (β=−0.12, CI=[−0.18, −0.05]; p<0.001; Fig. 1f), with pupil diameter being larger on strong compared to weak trials. As with posterior alpha, this difference could reflect attention to strongly familiar retrieval cues and/or to retrieved contents.

Distinct pupil components reflect cognitive effort and memory retrieval strength

Leveraging temporal PCA, we disentangled cue-evoked pupillary responses and identified two distinct components sensitive to cognitive effort and memory retrieval success.

PC1 scores were higher for stronger memories both at the condition level and the trial level, leading us to interpret PC1 as indicative of the strength of memory retrieval. In particular, PC1 scores were higher for strong compared to weak trials (Table 7; effect of Strength) and higher PC1 scores predicted faster match RTs (Table 8; effect of PC1 score). The relationship between PC1 scores and RTs was attenuated for mismatch trials (Mismatch × PC1 score interaction), which may be a consequence of the response delay induced by MPEs on mismatch trials (Fig. 1c). PC1 explained the most variance (28.81%; Fig. 3a), onset during the cue period, peaked predominantly at the time of probe onset (2.00s, CI=[1.94, 2.05]; loading = 0.90, CI=[0.90, 0.91]), and had a FWHM of 1.75s (CI=[1.70, 1.78]). Altogether, these findings suggest that PC1 tracks graded retrieval success or strength of memory retrieval.

Fig. 3:

Table 7:

Table 8:

For PC5, the relationship with the memory strength conditions and RTs indicated that this component was a condition-level marker of cue strength and a trial-level marker of cognitive effort during associative retrieval. PC5 scores were higher for strong than weak trials (Table 9; main effect of Strength), but in contrast to PC1, higher PC5 scores were associated with longer RTs (Table 10; main effect of PC5 scores). The relationship between PC5 scores and RTs was stronger for weak compared to strong pairings (PC5 score × Strength). Thus, PC5 showed the opposite pattern as PC1, with PC5 scores scaling inversely with trial-level memory strength, suggesting that PC5 reflects a mixture of cue strength (i.e., higher scores for strong cues) and the amount of cognitive effort expended during associative memory retrieval (i.e., inverse relationship with RT). The temporal profile of PC5 is consistent with this interpretation, as PC5 onset during the cue period, peaked during the delay period (1.15s after cue onset, CI=[1.12, 1.18]; loading = 0.56, CI=[0.52, 0.61]; Fig. 3b) and had a FWHM of 0.98s (CI=[0.92, 1.04]). This component explained 5.66% of variance.

Table 9:

Table 10:

Posterior alpha (attention) during memory retrieval is coupled with frontal theta (cognitive control) and pupil (attention/arousal)

Having used condition-level memory strength differences to identify temporal clusters associated with associative memory retrieval in frontal theta, posterior alpha, and pupil size (Fig. 3c-e), we then examined whether these electrophysiological and pupillary assays of cognitive control, attention, and arousal during memory retrieval aligned with each other trial-to-trial. In analyses of posterior alpha, we focus on the second Strength cluster (Fig. 1e), as this cluster spanned most of the delay period, when retrieval-related differences were most likely to be present (n.b., mean power for the longest cluster may also have the highest SNR). Similar results were found for the first Strength cluster (Supplementary Fig. 9-10). In addition, since temporal PCA isolated one pupil component related to associative retrieval strength (PC1) and another with cognitive effort during memory retrieval (PC5), pupil analyses focused on these component scores.

Stronger memories may require less cognitive control for bringing memory contents to mind and more content with which to allocate attention. As such, frontal theta and posterior alpha may be positively associated with each other on a trial-by-trial basis. Consistent with this idea, there was a main effect of frontal theta on posterior alpha (β=0.132, CI=[0.104, 0.161], p<0.001), indicating that higher frontal theta predicted less posterior alpha suppression. Moreover, a frontal theta × Strength interaction (β=0.062, CI=[0.018, 0.106], p=0.006; Fig. 3c) revealed that this relationship was stronger for weak compared to strong pairings. There was no frontal theta × Mismatch interaction (β=−0.031, CI=[−0.085, 0.023], p=0.264) and no frontal theta × Strength × Mismatch interaction (β=−0.048, CI=[−0.122, 0.027], p=0.212). Altogether, these results are consistent with frontal theta marking engagement of cognitive control during cued retrieval of weaker memories and that the amount of control engaged during retrieval scales trial-by-trial with less posterior alpha suppression; less suppression may reflect weaker memory reinstatement and fewer retrieval products to allocate attention to.

Posterior alpha and pupil size may both assay attention to memory contents and thus may be inversely related. There were negative relationships between posterior alpha power (second Strength cluster) and both PC1 scores and mean pupil size (main effect of PC1: β=−0.068, CI=[−0.092, −0.043], p<0.001; Fig. 3e; main effect of pupil: β=−0.064, CI=[−0.091, −0.037], p<0.001; Supplementary Fig. 11a). These relationships did not interact with other factors (Strength – PC1 × Strength interaction: β=0.013, CI=[−0.023, 0.049], p=0.472 or Mismatch – PC1 × Mismatch interaction: β=−0.012, CI=[−0.054, 0.030], p=0.578; three-way interaction: β=0.033, CI=[−0.030, 0.096], p=0.298; Strength – pupil × Strength interaction: β=0.000, CI=[−0.041, 0.041], p=0.994; or Mismatch – pupil × Mismatch interaction: β=−0.028, CI=[−0.076, 0.020]. p=0.252); three-way interaction: β=0.039, CI=[−0.033, 0.112], p=0.284) and there was no relationship between posterior alpha and PC5 (Supplementary Fig. 10). See Supplementary Figs. 10-11 for similar results for mean pupil diameter and the second posterior alpha Strength cluster. There were no relationships between frontal theta and PC1 or PC5 scores (Supplementary Fig. 12). Together, these findings complement the above relationship between posterior alpha and frontal theta, indicating that signals of memory retrieval in pupillary responses are associated trial-by-trial with more posterior alpha suppression during memory retrieval; this suppression may reflect attention to and maintenance of retrieved contents.

3. MPE responses interact with cognitive processes during mnemonic prediction generation

To understand whether cognitive processes during retrieval (i.e., cognitive control and attention) interact with changes in control, attention, and arousal following MPEs, we investigated whether cue-evoked frontal theta (a marker of cognitive control) predicted subsequent probe-evoked posterior alpha or pupillary MPE responses, and whether posterior alpha or pupil diameter during memory retrieval predicted frontal theta MPE responses.

While there were no relationships between frontal theta during memory retrieval and MPE-related posterior alpha or pupil responses (Supplementary Fig. 13), nor were there effects of posterior alpha or pupil diameter on frontal theta MPE responses (Supplementary Fig. 14a-b), attention during memory retrieval predicted MPE-related pupil effects. In particular, there was a positive relationship between posterior alpha suppression during memory retrieval (second Strength cluster) and PC3 scores (main effect of posterior alpha: β=−0.097, CI=[−0.182, −0.013], p=0.025; Fig. 3c). This relationship interacted with Mismatch (Mismatch × posterior alpha interaction: β=−0.111, CI=[−0.217, −0.005], p=0.039), such that the relationship between posterior alpha and PC3 scores was stronger when mnemonic predictions were violated than when they were confirmed. A positive relationship was also observed for posterior alpha and mean pupil size, though no interaction was observed (Supplementary Fig. 14c). By contrast, there was no relationship between posterior alpha and PC4 scores (Supplementary Fig. 14d). These outcomes indicate that greater attentional allocation during memory retrieval, which occurs when memories are stronger and more retrieval products can be reinstated and attended to (Fig. 1e; Fig. 3c), predicts the magnitude of immediate pupil responses to MPEs.

4. Learning from MPEs

Frontal theta during strong MPEs enhances learning

Finally, to understand whether strong MPE-driven increases in cognitive control, attention, and arousal support learning, we examined whether electrophysiological and pupillary responses to MPEs predict subsequent memory for mismatch probes in Experiment 1. There was a main effect of subsequent memory outcome on mean MPE-related frontal theta (the first Mismatch × Strength cluster) (β=−0.152, CI=[−0.284, −0.021], p=0.024; Fig. 4) and a significant frontal theta × Strength interaction (β=0.218, CI=[0.035, 0.401], p=0.020), indicating that frontal theta was higher for subsequent hits compared to subsequent misses and that this difference varied as a function of weak compared to strong mismatch probes. MPE-associated changes in posterior alpha and pupil were not associated with subsequent memory (see Supplementary Fig. 15; for subsequent memory effects in pupil in Experiment 2, see Supplementary Fig. 15a; see Supplementary Fig. 16 for recognition memory performance separated by memory condition and experiment). Trial-level regression models (Supplementary Fig. 17) further revealed main effects of Subsequent Memory on frontal theta from 1.088 to 1.624s (β=−0.15, CI=[−0.25, −0.05]; p<0.001), 1.944 to 2.544s (β=−0.24, CI=[−0.42, −0.06]; p<0.001), and 2.960 to 3.152s (β=−0.09, CI=[−0.15, −0.02]; p<0.001) after cue onset, such that frontal theta was higher for subsequent hits than subsequent misses. There was a significant Subsequent Memory × Strength interaction from 0.256 to 0.552s after probe onset (β=0.54, CI=[0.11, 0.98]; p<0.001) whereby the difference in power between hits and misses was larger for strong compared to weak probes. Collectively, these findings suggest that cue-evoked and delay period frontal theta, strength-modulated increases in frontal theta following MPEs, and the magnitude of the post-peak frontal theta responses enhanced learning of probes that violated mnemonic predictions.

Fig. 4:

Discussion

Features of present experiences often overlap with those of the past. Shared features allow for aspects of current experience to cue memories and enable knowledge about how similar events unfolded in the past to inform predictions about ongoing experience. When ongoing experience unexpectedly diverges from mnemonic predictions, multiple neurocognitive processes may be elicited, fostering adaptation to novel circumstances and facilitating more accurate predictions in the future. Drawing on behavioral, electrophysiological, and pupillary assays, the current study revealed that strong MPEs elicit increases in cognitive control, attention, and arousal, with potential impacts on later cognition and new learning.

Immediate, strength-sensitive neurocognitive impacts of MPEs

Across three physiological measures – frontal theta, posterior alpha, and pupil size – MPE-related responses were larger when predictions were strong compared to weak. These findings are consistent with the idea that responses to prediction errors are modulated by how unexpected the error is^20,21 and relatedly, how much uncertainty there is when making a decision⁷⁰. In the current study, weak predictions may give rise to less certainty about an upcoming event (i.e., the probe) such that weak MPEs may be considered an outcome of an expected degree of uncertainty following a weak memory cue. On the other hand, more strongly cued predictions – evidenced in the current study by better memory and faster RTs for strong vs. weak pairs – may elicit lower levels of uncertainty; a MPE in this case would be more unexpected and warrant increasing attention to facilitate model updating. Theories of expected and unexpected uncertainty implicate different neuromodulators in tracking different types of uncertainty, with LC norepinephrine release posited to track unexpected uncertainty²¹. The present pupil size findings, when interpreted as an indirect assay of noradrenergic LC activity^18,19, are consistent with this theory. Mnemonic prediction strength critically modulates demands on control, attention, and arousal.

Memory modes following mnemonic predictions: Open questions and future directions

Notably, for pupil size, the direction of the effect following weak MPEs was the inverse of that observed following strong MPEs, such that there was greater pupil dilation for weak matches than weak mismatches. We propose that this difference may reflect attention being oriented internally to retrieved contents after a weak mnemonic prediction is confirmed, and that increased pupil dilation following weak matches may reflect the probe facilitating pattern completion of the learning episode for the weak association. These findings suggest that the engagement of retrieval^71,72 or encoding modes³⁹ following confirmation or violation of mnemonic predictions may be dependent on prediction strength. However, whether increased pupil dilation following the confirmation of weak mnemonic predictions indeed reflects retrieval requires investigation of the mental contents and targets of increased attention following weak matches.

Pupil responses to MPEs: Temporal dynamics and neural interactions

Pupil responses during associative retrieval trials indexed multiple cognitive processes. As observed by others^12,13, temporal PCA revealed multiple pupil responses sensitive to task structure. Moreover, we observed strong coupling between PC scores and trial-level behavioral responses, which enabled interpretation of individual components as reflections of distinct cognitive processes. With respect to MPEs, temporal PCA yielded finer-grained insights into the temporal dynamics of pupillary responses. In particular, this analysis isolated two temporally distinct pupil responses to MPEs: a more immediate strength-modulated prediction error response (PC3) and a later attentional orienting response (PC4). By obtaining trial-level PC scores, as opposed to condition- and subject-level scores as in prior work^12,13, we were able to relate the multifaceted pupil responses to electrophysiological signals on a trial-by-trial basis. These trial-level analyses offered greater sensitivity into relationships that the pupil has with frontal theta markers of cognitive control and posterior alpha markers of attention. These analyses highlighted that PC3 scores were predicted by engagement of cognitive control (frontal theta) during memory decisions regardless of prediction error, and that for strong MPEs, PC3 scores scaled with the amount of attention allocated to strong mnemonic predictions during memory retrieval (posterior alpha suppression). PC4, like PC3, was associated with cognitive control (frontal theta) during memory decisions. In addition, PC4 scores varied with the degree of posterior alpha suppression evoked following memory decisions. Altogether, these findings highlight that initial MPE-related changes in attention and/or arousal assayed separately by pupil and posterior alpha are distinct, but that over time, these measures converge and may subsequently reflect a unitary focus of attention.

Frontal theta and posterior alpha dynamics

Frontal theta and posterior alpha were correlated during memory retrieval, yet MPE-driven changes in frontal theta and posterior alpha were not robustly coupled. This outcome suggests that MPE-evoked changes in cognitive control and posterior alpha-metrics of attention are generated independently; this additionally implies that different mismatch detection systems may underlie MPE responses in frontal theta and posterior alpha. Moreover, frontal theta cognitive control signals appeared to have an earlier influence than posterior alpha on pupil-linked attention and/or arousal (i.e., there was a differential association with PC3 and PC4). Future work should examine the directionality of these effects to understand whether frontal theta/posterior alpha drives pupil responses after MPEs or vice versa. Furthermore, direct assays of LC activity, which likely precede cognition-related pupil responses, would provide additional insights into underlying mechanisms driving these responses.

Evidence accumulation during memory-based decisions

Beyond being impacted by MPEs, pupil size also indexed cognitive processes during the generation of mnemonic predictions. In particular, PC1 reflected trial-level mnemonic prediction strength, providing additional support for pupil systems tracking evidence accumulation processes⁷³; for the current study, in evidence accumulation from memory. Given that PC1 scores were coupled with posterior alpha during memory retrieval, future work should tease apart whether pupil size and posterior alpha index distinct processes during memory reactivation and memory-based decisions beyond what has previously been observed in perceptual decision-making for parameters that modulate integration of evidence or neuronal noise⁷³. Such investigations could also elucidate the commonalities and differences between pupil-linked and posterior alpha-linked attention signals.

Beyond mismatch signals in PFC: Potential interactions with the hippocampus, VTA/SN, and LC

Drawing on frontal theta, the current study characterized the temporal profile of cognitive control following strong MPEs. Yet additional and more direct corroboration of prior MEG findings that vmPFC theta drives hippocampal theta⁴² is needed to resolve questions about their relative timing. Are MPEs detected first by PFC or by hippocampus (e.g., within subfield CA1)? Does prediction strength modulate the magnitude of frontal and hippocampal theta responses similarly? Alternatively, does the strength of MPEs modulate hippocampal theta linearly and frontal theta nonlinearly, such that frontal theta primarily responds to strong MPEs?

Hippocampal-PFC dynamics are a focus of current theories of mismatch detection. From one perspective, PFC is theorized to play a key role in conveying goal information to support selective encoding of novel information that is goal-relevant³³. Another model posits that predictions and plans for upcoming decisions are generated by the dorsolateral PFC (dlPFC) and relayed to the hippocampus, which later sends mismatch signals to medial PFC⁷⁴. The latter flow of information to medial PFC, depending on whether the hippocampus is conveying a match or a mismatch in expectations, can further bolster the planned action or enable action switching⁷⁴. The dynamics of this information flow imply that control signals mediated by medial PFC are evoked following hippocampal theta mismatch signals, rather than with vmPFC theta driving hippocampal theta⁴². Whether and when PFC subregions drive hippocampal activity, and vice versa, remain unclear. Some studies of hippocampal-PFC dynamics during retrieval have shown that signals from PFC initiate memory retrieval processes in the hippocampus/temporal lobe^75,76, whereas others demonstrate that hippocampal theta may precede PFC theta during free recall⁷⁷. Other studies of cross-regional communication suggest that information flows from hippocampus to PFC via theta oscillations in both humans⁷⁷ and non-human animals^78,79. With respect to error-driven dynamics, cross-regional communication following MPEs may reflect engagement in memory encoding as opposed to memory retrieval, and these two processes may exhibit differential hippocampal-PFC dynamics. Altogether, these observations highlight critical open questions about (a) the relationship between MPE-driven PFC and hippocampal activity, (b) the importance of theta oscillations⁴⁴ in mediating cross-regional communication, (c) whether hippocampal-PFC dynamics differ during memory encoding vs. retrieval, and (d) whether the temporal dynamics of hippocampal-PFC information flow vary as a function of PFC subregion. Answers to these questions will advance understanding of cross-regional communication upon MPE detection³³.

Given limitations in the spatial resolution of scalp EEG, key questions about other components of models of mismatch detection were not examined in the current study. Future work should further examine the role of the nucleus accumbens, which is theorized to integrate goal information from PFC and MPE signals from the hippocampus and the ventral tegmental area/substantia nigra (VTA/SN)⁸⁰. The VTA/SN are posited to release dopamine in response to MPEs to tag hippocampal memory traces for consolidation³³. Whether dopamine and norepinephrine differentially contribute to learning from MPEs remains unclear. Prior work examining the integrity of LC and SN among older adults implicates the former in episodic memory performance and the latter in working memory⁵⁵. The relative roles of noradrenaline release from LC and dopamine release from VTA/SN during MPEs warrant further investigation. Notably, the hippocampus receives more profuse projections from LC than VTA neurons in mice⁵⁴. Moreover, LC neurons influence PFC neuronal firing^81,82 and stimulation of the LC releases both noradrenaline and dopamine in PFC^83,84; PFC neurons also project to LC^85,86. The bidirectional connectivity between LC and PFC raises questions about the temporal dynamics of their interactions following MPEs and how LC modulates PFC-hippocampus interactions.

Future oriented: How MPEs facilitate future behavior

In addition to impacting current cognition and behavior, adaptive responses to MPEs should facilitate future behavior. In the short-term, we observed that larger pupil responses to strong MPEs enhanced preparatory attention and readiness-to-remember⁶⁹ on the subsequent trial, cued ∼1–2s later. These findings further link pupil responses to post-error behavioral adaptation⁸⁷. On a longer timescale, MPEs could facilitate future behavior by increasing the probability that unexpected information is encoded into memory and thus update future predictions. The likelihood of learning from strong MPEs was associated with frontal theta magnitude during MPEs, but not posterior alpha or pupil size. Subsequent memory effects in frontal theta for strong mismatch probes were not selectively observed during MPEs, but also prior to and following MPEs. These findings are consistent with prior work showing that higher peri-stimulus frontal theta boosts memory formation^51,56,57. Moreover, these findings critically highlight the circumstances in which frontal theta may play a key role in learning: when violations of strong mnemonic predictions elicit increases in cognitive control. The degree of cognitive control engaged during prediction generation additionally impacts the likelihood of future remembering. Whether MPE-driven changes in frontal theta are a mechanism by which PFC conveys the goal relevance of surprising information³³ remains unclear and could be investigated in future studies by experimentally manipulating the goal relevance of the mismatch probe.

Mnemonic predictions guide everyday behavior. The strength of these predictions varies depending on frequency of prior occurrences³, emotional salience^88,89, and attention during learning^53,90,91, amongst other factors. Here, we report that violations of strong mnemonic predictions elicited increases in cognitive control, attention, and arousal. The level of pupil-linked attention/arousal in response to strong MPEs was influenced by the amount of attention allocated to the mnemonic prediction and impacted preparatory attention and readiness-to-remember later in time. Furthermore, the magnitude of cognitive control responses to strong MPEs modulated the likelihood of remembering that information in the future. Altogether, these findings advance understanding of the learning and retrieval rules of the mind and brain and add to a growing literature on how dynamic interactions between knowledge about the past and the present give rise to rapid shifts in cognition that impact learning and future remembering.

Data availability

Data and code for this manuscript are accessible at https://github.com/alicexue/mpe-attention-cog-ctrl.

Acknowledgements

This work was supported by the National Institute on Aging R01AG065255 to A.D.W., National Science Foundation GRF DGE-2146755 to A.M.X., and a Stanford Interdisciplinary Graduate Fellowship to A.M.X. The authors would like to thank the members of the Stanford Memory Lab for useful discussions.

Additional files

Supplemental Figures and Tables

Additional information

Funding

HHS | NIH | National Institute on Aging (NIA) (R01AG065255)

• Anthony D Wagner

NSF | National Science Foundation Graduate Research Fellowship Program (GRFP) (DGE-2146755)

• Alice M Xue