Schema-based predictive eye movements support sequential memory encoding
- Department of Psychology, Columbia University, United States
- Center for Neural Science and Department of Psychology, New York University, United States
Figures
Figure 1
A schematic of the experimental method.
(a) Task structure across six (non-consecutive) days. (b) Memory task. In each of the 30 trials, participants saw an initial board for 5 s, and then a move was added to the board every 5 s. After viewing a sequence of 4four to eight moves and completing a distractor task, participants were shown the initial board and asked to reconstruct the sequence by placing the pieces on the board. (c) Gameplay task. Participants played 40 games against an adaptive-difficulty AI agent.
Figure 2
Participants’ performance in memory and gameplay.
(a) On average, Participants become better at remembering sequences over the course of the training (red line, N = 35). In each session, memory accuracy is much higher than the performance that would be achieved if people were simply guessing according to the gameplay model (green line, simulated, N = 100). (b) On average, participants become better at playing the game across sessions. Error bars represent the standard error of the mean. (c) There is a positive correlation between people’s playing strength and their recall accuracy (each dot corresponds to one session of one participant).
Figure 3
The effect of schema consistency on memory and its development across training sessions.
(a) An example of the model’s evaluation of a board where the next player is black. Based on features that would be generated by each possible next move (e.g. creating three-in-a-row), the model generates a probability distribution over potential next moves. We apply this model to the stimuli in the memory task to estimate the probability of each move. (b) The effect of schema consistency of a move on the recall accuracy for the move. The x-axis is the log probability of a move, which reflects schema consistency. The y-axis is the probability that a participant will remember the move. In session 1, there is no relationship between the probability of moves and subsequent memory. In sessions 2–6, people are more likely to remember schema-consistent moves. The histogram at the top and bottom of the figure is the frequency of moves with certain log probabilities that are remembered and forgotten, respectively (c) The relationship between move probability and recall accuracy over the 6 sessions. (*** denotes p<0.001). (d) The relationship between move probability and reaction time at recall for correctly remembered moves. (*** denotes p<0001). (e) The proportion of times the first mistake in a sequence is a more probable move than the actual move that was observed. Error bars represent the standard error of the mean, and the sample size is 35 participants.
Figure 4
Using eye movements to reveal encoding strategies.
Left: A participant’s fixation heatmaps over a 5 s encoding period. right: the 6 regressors that we consider as potential encoding strategies. 1: Participants could look at the last (most recent) move, which is what they need to remember. 2: Participants could look at occupied tiles that might be relevant to the most recent move, to try to see what features the move forms. 3: Participants could be anticipating the upcoming move, meaning they will look at the empty squares on the board that are likely to be the next move. 4: Participants could also look at current pieces that are relevant for predicting the next move, i.e., pieces related to empty squares that are likely to be the next moves. 5: Participants could be looking at moves that previously appeared, in order to rehearse the observed move sequence. 6: Participants might have an overall tendency to look at occupied or unoccupied tiles.
Figure 5
The relationship between prediction, playing strengths, and recall accuracy.
(a) The extent to which eye movements on empty squares align with the gameplay model (prediction coefficient; see Figure 4) increases over training sessions. (b) Correlation between Elo (playing strength) and prediction coefficient in the 35 participants. (c) Correlation between prediction coefficient and recall accuracy. (d) Mediation analysis: the effect of playing strength on recall accuracy is mediated by the amount of prediction participants made during encoding. The values on the arrows represent regression coefficients for standardized measures of Elo, prediction coefficient, and recall accuracy (* denotes p<0.05, ** denotes p<0.01). (e) Move-level analysis on the effect of schema consistency on memory, predicting whether a move will be remembered based on the probability of the move (brighter colors are more schema-consistent) and prediction coefficient in the previous move. Error bars represent the standard error of the mean.
Figure 6
Model-free measure of prediction and their relationship with memory.
(a) Example fixation heatmaps (in red) of high and low prediction confidence and surprise. Confidence measures the extent to which fixations were focused on a small number of unoccupied squares, while surprise measures how well these fixations aligned with the actual next move (indicated with an empty circle). (b) Recall accuracy was best when prediction confidence was high and surprise was low. Error bars represent the standard error of the mean.
Author response image 1
Author response image 2
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Schema-based predictive eye movements support sequential memory encoding
eLife 12:e82599.
https://doi.org/10.7554/eLife.82599
