To make choices in structured environments, human and non-human animals must forecast the long-term consequences of candidate actions, which often are delayed and play out serially over multiple steps. Predicting these outcomes often requires integrating information from disparate learning episodes, which occurred at different times in the past. In other words, successful planning requires both (a) accessing memories of multiple previous events and (b) integrating these memories to make useful predictions about an action’s consequences: a type of integrative memory replay (Lengyel and Dayan, 2008). Indeed, both human and non-human animals can solve reward learning tasks designed to require this sort of integration, such as latent learning tasks (Tolman, 1948) in which information about the outcome contingencies of actions (e.g., the spatial layout of a maze) is presented in a separate stage of training from the associated rewards (e.g. receipt of food).

Little is known, however, about how and when such integrative planning occurs. A longstanding assumption, dating back to Tolman’s proposal in 1948 (Tolman, 1948), is that planning unfolds at decision time, for example during mental simulation of the candidate future action trajectories. More recent proposals suggest that such planning need not occur only when the animal is presented with a choice, but can also occur earlier, during encoding of choice-relevant information (Wimmer and Shohamy, 2012; Shohamy and Daw, 2015). In particular, when new information is first learned (e.g. food discovered in some location), its consequences for distal actions (e.g. a high value for routes leading there) can be inferred and immediately encoded. Indeed, human fMRI work provides support for both mechanisms: Studies have found that people’s ability to solve integrative evaluation problems depends on memories of distal task states accessed either at the time of choice (Doll et al., 2015) or encoding (Wimmer and Shohamy, 2012).

What the encoding- and choice-time mechanisms share is that both occur during active task performance, which artificial intelligence refers to as learning online (Sutton, 1991). However, during both encoding/learning and decision-making periods, people often have limited time for exploring the set of interrelationships among decisions and outcomes, which may be prohibitive in complex environments. A third, more general possibility, which we examine here, is that integration of memories to support action evaluation can also occur offline, such as during off-task rest periods or even sleep. An advantage of offline planning is that it offloads computational costs to periods that are not already occupied by task behavior. This way, before the organism faces a problem that requires a decision (but after it has initially encoded the constituent events), it can update action valuations or choice policies offline, thereby accomplishing at least part of the evaluation required for later choice (Gershman et al., 2014).

Here we investigate integrative memory processes in humans that help piece together information offline, thus supporting planning and improving decision-making behavior. In the formalism of reinforcement learning in artificial intelligence, the Dyna family of algorithms implements offline planning, which is accomplished by learning from replay of experiences that were initially encoded during online behavior (Van Seijen and Sutton, 2015; Sutton, 1991). Inspired by this work, and together with behavioral evidence of offline revaluation (Gershman et al., 2014; Momennejad et al., 2017), computational models suggest that Dyna-like offline replay mechanisms might be implemented in the brain (Momennejad et al., 2017; Johnson and Redish, 2005; Ludvig et al., 2017; Russek et al., 2017). Indeed, this hypothesis fits closely with a substantial rodent literature revealing that hippocampal replay of previously experienced spatial trajectories is observed during both awake behavior and sleep (Takahashi, 2015; Ólafsdóttir et al., 2015).

It has also been shown that hippocampal replay (Pfeiffer and Foster, 2013) captures the topological structure of the task environment (Wu and Foster, 2014), and hippocampal reverse replay of trajectories occurs offline when the animal experiences changes in reward (prediction errors) at the goal state (Ambrose et al., 2016a). In parallel, human fMRI research work reveals how memories are reactivated offline, such as during periods of rest that follow rewarding episodes (Gruber et al., 2016; Murty et al., 2016). This offline reactivation could potentially aid in memory consolidation and integration, evidenced by enhanced subsequent memory performance (Takashima et al., 2009; Staresina et al., 2013; Tambini and Davachi, 2013). Consistent with the hypothesis that memory integration can be accomplished via offline replay, there is behavioral evidence that manipulations affecting rest periods influence humans’ ability to solve integrative decision tasks (Gershman et al., 2014). However, so far, no animal or human study has provided neural evidence that integration during offline replay supports planning.

Our main question in this study is: does replay predict a change in choice behavior? In this study, we use fMRI and multivariate pattern analysis (MVPA) to test the hypothesis, outlined above, that offline replay drives integrative re-evaluation and update of past policies. We designed a paradigm where such integration would be reflected in the behavioral choices participants make. It has been shown that replay of a particular episode can lead to the reinstatement of neural activity in sensory cortical regions (Ji and Wilson, 2007; Polyn et al., 2005). Accordingly, we use MVPA here to track cortical reinstatement of a particular state during off-task rest periods, as a measure of offline replay of that state. We then examine the relationship between neural replay evidence (from MVPA) and the change in subsequent choice behavior, to test whether replay facilitates the brain’s ability to integrate information and update planning policies accordingly.

Having found evidence for this relationship between replay and replanning, our secondary question in this study was to explore the antecedents of this process: What determines which memories are replayed? The idea that integrative replay occurs offline (rather than being triggered by particular task events) raises the question of prioritization: Out of all the episodes stored, which memories should the brain replay to ensure integrative and goal-directed learning? What is it about an experience that privileges it for later replay? Here we draw insight again from artificial intelligence, where the standard approach for this problem is an algorithm known as prioritized sweeping (Moore and Atkeson, 1993; Peng and Williams, 1993). The key insight behind this algorithm is that when surprising information is encountered at some state, it may have consequences for the value of states or actions that could have led there, and therefore those predecessors should be prioritized for replay. One representative and standard signal of such surprise or uncertainty, which is leveraged as a heuristic to trigger replay in prioritized sweeping, is the experience of prediction errors—that is the deviation between the agent’s predictions and experiences of the world. To this end, both negative and positive prediction errors (i.e. the unsigned error) are meaningful as they signify the need for improved value estimates. To test this hypothesis, we use model-based fMRI analysis to evaluate whether, across blocks and subjects, larger neural correlates of prediction errors are associated with greater replay during subsequent rest periods, and ultimately greater change in behavior.

In short, our major proposal is that offline neural replay drives the integration of distal memories to update previously learned representations and support future planning. Our secondary question is which earlier events predict replay and ultimately replanning. We hypothesize that episodes marked with large prediction errors (PEs), whether positive or negative, signal uncertainty about the optimal reward trajectory and tag related memories for future replay. This replay, in turn, enables the integration of separate memories in the service of updating optimal reward policy (action policies that maximize reward). We combine model-based fMRI, MVPA, and a sequential decision-making task to test our hypotheses.

To test the hypothesis that offline replay supports integration of memories to serve future decisions, we operationalized planning using a reward revaluation paradigm (Gershman et al., 2014; Momennejad et al., 2017). In reward revaluation, participants first learn the multi-step sequential decisions that lead them to reward from a starting point; later, they experience a local change to later stages of the decision trajectory. Upon return to the beginning of the trajectory, we can measure the extent to which a participant’s choices adapt to the new local changes they experienced.

Here we used a 2-stage sequential decision task with a within-subject design that allowed us to manipulate whether or not revaluation was required for reward-maximizing choice behavior (Gershman et al., 2014; Momennejad et al., 2017). In both the revaluation and control conditions, participants (n = 24) repeatedly visited a starting stimulus (Stage I) and chose between two actions that would lead them to one of two subsequent stimuli depending on their choice (Stage II), as displayed in Figure 1. Once a participant reached either of those Stage II stimuli (or states), they selected one of two actions to obtain reward. We will refer to the Stage I stimulus as state one and the two Stage II stimuli as state 2 and state 3. As a cover story, participants were told they were playing a game as a thief, exploring different buildings to steal the most money. Participants were given four separate blocks of this task, each with a different reward structure; half these blocks were randomly assigned to the revaluation condition, and half the blocks were assigned to the control condition, as described below. To ensure that participants did not transfer what they learned from one block to another, each block had a different colored background that indicated they were in a different city, and different stimuli for the Stage I and Stage II actions (Figure 1—figure supplement 1).

Figure 1 with 1 supplement see all

Download asset Open asset

The task in both revaluation and control blocks consisted of three phases: a Learning phase (where the participant explored the entire decision tree), a Relearning phase (in which the participant visited Stage II but never Stage I), and a Test phase (where participants revisited Stage I). During the Learning phase, participants learned the structure of states and rewards.

During the Relearning phase, the rewards associated with Stage II stimuli changed for the revaluation blocks (but not for the control blocks). The new reward amounts in revaluation blocks were chosen so as to require a change in the decision policy leading to optimal reward in Stage I (e.g., during the Learning phase, the optimal decision policy might be to go left at Stage I, but after taking into account the Stage II reward changes experienced during Relearning, the new optimal decision policy might be to go right at Stage I).

The Test phase assayed whether participants reversed their initial preference according to the Stage II changes they experienced in the Relearning phase (in control blocks we did not expect any change in preference to occur). Note that a change in Stage I policy preference would require integrating information about the action-stimulus contingencies from the Learning phase with the re-learned reward amounts associated with Stage II states from the Relearning phase. The Test phase choice was repeated four times; we computed the average proportion correct across these four choices. Importantly (to prevent any influence of new learning during the test phase), participants were not given any feedback about the consequent Stage II stimulus and reward after these choices.

Participants were incentivized to search for and choose higher-value states. They were explicitly instructed prior to the learning phase to find the highest monetary reward and navigate there in order to increase their bonus money. The instructions below appeared prior to the learning phase of every city/MDP during the experiment in order to reiterate this incentive (the names of the city and location were adjusted for each session):

"In this game you will be robbing a theater in Berlin.

By learning how to get around the theater, and learning where the most money is, you can steal the most money possible.

And the more money you steal in the game, the more REAL BONUS MONEY you will be paid on top of the normal experiment payment. You could earn an extra $2–3!’

Note that the critical difference between the control and revaluation conditions was in the Relearning phase: in the control condition there was no change in the rewards associated with Stage II choices (such that integrative consideration of this new information should not result in systematic change in Stage I preferences), whereas in the revaluation condition Stage II rewards changed. Hence, during the Test phase we could test whether decisions made in Stage I were affected by information learned about Stage II in the Relearning phase (see Figure 1 and Materials and methods). The inclusion of the control condition helps to rule out effects related to any nonspecific change in preferences (as opposed to integrative replanning), such as forgetting or inattention. As noted above, participants completed four separate blocks with this structure but different stimuli and rewards (two revaluation and two control). They did not know to which condition each block belonged (or about the logic of the manipulation), and the order of revaluation vs. control blocks was randomized.

In addition to the revaluation vs. control manipulation, we also manipulated the level of noise (variance) associated with rewards experienced following an action in a given state. This led to a 2 × 2 factorial design with revaluation and noise manipulations. For both revaluation and control blocks, half the blocks had variable (noisy) rewards selected from a distribution with a fixed mean (noisy rewards condition) leading to a persistent level of prediction errors even in the control condition (e.g. choosing left at state two would lead to different values that average to $5), and half of the blocks had no noise in rewards (noiseless rewards condition) leading to no prediction errors after learning unless there was a revaluation (e.g. choosing left at state two would always lead to $5).

This noise manipulation—that is the noiseless-rewards versus noisy-rewards conditions—ensured that the difference between revaluation and control condition blocks was not limited to the absence or presence of PEs or surprise in general. In particular, the inclusion of a noisy control block (in which there is elevated surprise but no actionable new information during the Relearning phase) further helps to control for nonspecific changes in preference (e.g., mediated by attention or motivation) that might be associated with surprise per se, rather than surprise enabling integrative planning. Note that under the Dyna and prioritized sweeping algorithms that underlie our hypotheses, replay by itself is not sufficient to produce change in preference. Our hypothesis, instead, is that surprise leads to replay later on (to investigate whether the surprising information has consequences for choice), but whether this replay leads to replanning/revaluation depends on whether the replayed experiences in fact support a change in preference. Accordingly, the inclusion of reward noise in the noisy-rewards condition (even in control blocks) allowed us to rule out the possibility that change in choice preferences is brought about by ‘any’ experience of uncertainty, whether or not experience actually supported replanning. However, given that our primary interest is in replanning per se, in what follows we mainly report revaluation vs. control effects, collapsing noiseless and noisy reward conditions, while reporting statistical differences and further fMRI analyses of the noise manipulation in the Supplementary material.

Behavioral results

Comparison of replanning magnitudes revealed that participants showed significant revaluation behavior (i.e., reversed their Stage I choice from Learning to Test, as indicated by positive replanning magnitudes) in revaluation blocks (t(23) = 4.89, p < 0.0001), but not during control blocks (t(23) = −0.10, p = 0.92), and the difference in magnitudes between the two conditions was significant (t(23) = 4.333, p = 0.0002) (Figure 2B). The comparison with the control condition verifies that this apparent revaluation behavior is not simply attributable to general forgetting or increased memory noise from Learning to the Test phase; rather, it is due to revaluation of past policies in the face of novel information in the revaluation condition—but not the control condition. A 2-way analysis of variance confirmed the main effect of the reward vs. control factor but revealed no significant main effect of the reward stability vs. noise manipulation (see Figure 2—figure supplement 1).

Figure 2 with 2 supplements see all

Download asset Open asset

The brain’s response to unsigned prediction error tags memories for replay

A key question for any account of learning from replay is how the brain selects which memories to replay. We thus explored what events favor replay, and ultimately successful revaluation. A general, appealing possibility is that events with some particular property when first experienced, like surprising ones, are privileged for later offline replay. (Mechanistically, this might occur either because they are ‘tagged’ for replay when first experienced, or because they are singled out later at retrieval.) We tested this idea in terms of one representative notion of surprise – neural sensitivity to unsigned PEs (i.e., the absolute value of PEs), as in the prioritized sweeping algorithm (Moore and Atkeson, 1993). Offline replay in turn would support updating past policies and hence predict future revaluation behavior (Figure 3, note the relationships indicated in the diagram, top left). We have shown in previous sections that offline replay during rest is correlated with revaluation during a later Test phase. To examine evidence for the hypothesized relationship between the brain’s response to PE and offline replay, as well as subsequent replanning behavior (Figure 3, top left), we ran parametric modulation analyses as follows.

Figure 3 with 1 supplement see all

Download asset Open asset

For each participant, we conducted a parametric regression analysis within the Revaluation condition to identify regions with sensitivity to stimulus-by-stimulus unsigned PEs. Prediction errors were estimated from a Temporal Difference (TD) learning model (see Methods). We then tested, for each subject and each Revaluation block, the relationship between the magnitude of the brain’s PE response (in that block) and two different covariates: subsequent replay and subsequent replanning score (Figure 3, top right, purple and blue maps respectively). This allowed us to determine whether (and in which brain areas) the magnitude of activity in response to unsigned PE was related to off-line replay (Figure 3, regions indicated in purple, see Table 1 for coordinates) and subsequent replanning (Figure 3, regions indicated in Blue, see Table 2 for coordinates), as hypothesized (Figure 3, top left). In order to identify the combined effect and given the correlation between the two covariates, we used a conjunction analysis over the two resulting correlation brain maps (Figure 3, regions indicated in green). This conjunction analysis yielded regions where the brain’s response to unsigned PE on each block correlated with both (a) subsequent mean replay evidence (for Stage I stimuli) during rest, and (b) subsequent replanning magnitude of the choice behavior during that block’s Test phase (Figure 3).

Table 1

Region	Z score	X	Y	Z	K voxels	P
Right Cuneus Left Cuneus Right Lingual Left Lingual Calcarine	3.31 3.29 3.21 3.15 3.14	-2 4 −14 16 6	−68 −66 −68 −46 −70	4 22 22 -8 4	1898	.000488

Table 2

Region	Z score	X	Y	Z	K voxels	P
Right precentral Orbitofrontal cortex (OFC) Supplementary motor area Right superior frontal cortex ACC Right OFC	3.67 3.62 3.59 3.49 3.49 3.42	26 0 -8 18 −14 6	−20 58 4 32 50 58	68 −12 48 38 0 −18	5240	1.36e-08
Right superior temporal Right supra-marginal Inferior parietal Right putamen Right superior temporal pole	3.67 3.6 3.58 3.39 3.35	56 34 36 26 46	−30 −36 −42 8 18	14 44 44 2 −16	5165	1.67e-08
Left superior temporal pole Left putamen	3.4 3.35	−46 −20	−10 18	0 8	2161	.00021

Table 3

Region (cluster)	Z score	X	Y	Z	K voxels	P
Left hippocampus	4.74	−20	−38	10	297	.014
Left insula Left superior temporal	4.57 4.57	−36 −42	-8 −24	6 6	1959	.000
Right superior temporal	4.11	44	−12	2	2249	.000
Right anterior cingulate, ventromedial PFC	4.65	2	38	6	238	.048
Left mid cingulate Left Supplementary motor area	3.31 3.33	−10 -6	−10 −12	46 62	258	.030

This conjunction analysis yielded multiple regions where neural sensitivity to unsigned PEs significantly predicted both future offline replay and revaluation magnitudes: the anterior cingulate cortex (ACC), the mid cingulate, posterior medial cortex (PMC) including dorsal and ventral precuneus, and the basal ganglia, especially the putamen (Figure 3, bottom). These results are consistent with the hypothesized relationship between sensitivity to prediction errors during learning, offline replay during rest, and revaluation behavior at test, and suggest an involvement of ACC, PMC, and basal ganglia in signaling the relevant PEs (see Figure 3—figure supplement 1 for breakdown of this analysis for the noisy-rewards and noiseless-rewards blocks).

Brain activity during revaluation vs. control rest

Next, we report a number of observations further unpacking the key results discussed above. Our hypothesis is that learning and updating take place during offline replay periods—indeed, we revealed above that replay in category-sensitive brain regions during rest periods of revaluation blocks (but not control blocks) predicted subsequent planning behavior. This hypothesis also implies that brain regions related to learning and memory should show more activity during the rest periods of revaluation trials, where engaging in offline replay could result in better choices, compared to control trials in which the most learning has already occurred during the Learning phase. To test this, we used a GLM to compare overall differences in univariate activity during the rest periods of revaluation vs. control trials. Contrasting rest periods from revaluation trials versus control trials (Rest_revaluation> Rest_control) revealed higher activity in the hippocampus/medial temporal lobe and the anterior cingulate cortex (p < 0.005, cluster corrected at p < 0.05, Figure 4, see Table 3 for coordinates, and Figure 4—figure supplement 1 for breakdown of this analysis for the noisy-rewards and noiseless-rewards blocks).

Figure 4 with 1 supplement see all

Download asset Open asset

Learning during rest

So far, we have shown that evidence for offline replay, across subjects and blocks, correlated with subsequent replanning behavior, and that regions broadly involved in memory and evaluation were more active during the rest period of revaluation trials. Finally, we broke down these effects over the course of the three rest periods in the Relearning phase to examine how they evolved over the progression of relearning and seek evidence of learning dynamics.

A property of the learning model is that PEs during the Relearning period should decrease from the period preceding Rest 1 (when participants are first encountering the new rewards in the revaluation condition) to the period preceding Rest 3 (after participants have had more time to learn these new rewards); this was the case for the PE values estimated by the learning model (Figure 5, top left). Given the decreasing pattern in unsigned PE during re-learning, we predicted that this within-block variation might be echoed in patterns of replay across rest periods – reduced PE should lead to less replay. To focus the analysis on differences among rest periods within each block, we mean-centered MVPA evidence for the Stage I stimulus during the three rest periods for each block of each subject. This allowed us to compare patterns of within-block differences between replay in the three rest periods of revaluation and control trials. In control blocks, we found no differences in MVPA evidence for S1 replay during the three rest periods (Figure 5). During revaluation blocks, however, evidence for offline replay significantly decreased from Rest one to Rest 3 (t(23) = 2.9, p = 0.007), similar to the decreasing pattern of unsigned prediction errors (Figure 5). Furthermore, offline replay was higher during Rest 1 of revaluation blocks compared to the Rest 1 of control blocks (t(23) = 2.17, p = 0.039). Thus, consistent with our hypothesis, prediction errors and replay track each other over the course of the block, as well as across blocks (as previously shown in Figure 3).

Figure 5 with 1 supplement see all

Download asset Open asset

Analyzing the rest periods of revaluation trials in the noisy-rewards and noiseless-rewards conditions separately (Figure 5—figure supplement 1), we found a similar trend: In the noisy-rewards condition, replay was marginally higher during Rest one than both Rest 2 (t(23) = 2.07, p = 0.049) and Rest 3 (t(23) = 2.02, p = 0.05) – in the noiseless-rewards condition, replay levels were low overall and did not vary across rest periods.

Finally, we considered how this replay related to subsequent replanning behavior in the Test phase. We compared the replay-replanning correlation across the different rest periods (Figure 5; unpacking over time the correlation from Figure 2B). We found that, for the average of all revaluation blocks, this correlation was numerically stronger (and only statistically significant) during later rest periods (however, there was no significant difference between the correlation coefficient for Rest one versus the correlation coefficient for Rest 3, Fisher’s z = −1.01, p = 0.31). We then analyzed this effect separately for revaluation blocks in the noisy rewards and revaluation blocks in the noiseless-rewards condition (Figure 5—figure supplement 1), and found again that the replay-behavior correlation was mainly driven by the noise condition, where the most significant replay-behavior correlation is observed during Rest 2.

We present fMRI evidence for the hypotheses that uncertainty-driven replay can help integrate past episodes offline, and in turn, update representations and policies that support planning (Van Seijen and Sutton, 2015). We address two questions. Our main question was: does off-task replay support planning? Inspired by the Dyna architecture in reinforcement learning, our main contribution is to demonstrate a clear and selective association between offline replay and change in choice preferences, that is replanning. We found that MVPA evidence for offline replay during rest correlates with subsequent choice behavior in reward revaluation. We suggest that during offline replay the brain integrates and pieces together simulated trajectories of information acquired during multiple learning episodes and updates planning policies accordingly.

Our secondary question was: what predicts such offline replay? Inspired by prioritized sweeping in reinforcement learning, our second hypothesis was that certain classes of events – such as surprising ones – privilege trajectories for future replay that helps the brain to learn whether they have consequences for choice preferences. We examined this question using a representative index of surprise, the brain’s response to unsigned prediction error. Whether this subsequent replay leads to revaluation or not will depend on whether the replayed experiences themselves support a change in behavior. Therefore, not all prediction errors will lead to revaluation, even if they increase replay. We found that the brain’s sensitivity to unsigned prediction errors in the ACC, basal ganglia, and posterior medial cortex (the precuneus) across subjects and blocks predicted both the extent of subsequent offline replay as well as subsequent replanning behavior. We suggest that the brain’s sensitivity to surprising information and increased uncertainty, here signaled by unsigned prediction errors, may tag certain episodes and trajectories of experience for selective offline replay later on (Moore and Atkeson, 1993; Peng and Williams, 1993).

Finally, we observed a general increase in hippocampal, posterior medial, anterior cingulate (extending to ventromedial prefrontal cortex upon relaxing the threshold), and superior temporal cortex activity during the rest periods of revaluation trials compared to control. The hippocampus has long been identified as the locus of episodic memory (Ranganath and Ritchey, 2012), posterior medial cortex is shown to be involved in recall and integration of memories over long time-scales (Chen et al., 2017; Shenhav et al., 2013), the anterior cingulate cortex is known to enable comparisons and signal conflicts the resolution of which may be valuable enough to recruit controlled processes (van Kesteren et al., 2013), and the ventromedial prefrontal cortex has been established as a region involved in memory retrieval of relevant information (Preston and Eichenbaum, 2013; Gläscher et al., 2009), value representation (Fellows and Farah, 2007), and decision making (Anderson et al., 2016) respectively. Hippocampal-prefrontal interactions have been shown to mediate control of memory in memory inhibition (Kalisch et al., 2006) and updating memory in extinction (Behrens et al., 2007). Thus, this finding supports an increased role for memory and evaluation mechanisms during the rest periods of revaluation blocks.

In short, inspired by work on the Dyna architecture in reinforcement learning (Sutton, 1991) and prioritized sweeping (Moore and Atkeson, 1993; Peng and Williams, 1993) in machine learning, as well as behavioral paradigms of reward revaluation (Gershman et al., 2014; Momennejad et al., 2017), these results are the first to show a functional role for offline replay in planning under uncertainty. The key novel aspect of our results is our finding that offline replay can lead to replanning by integrating multiple, temporally distal learning episodes (here, computing the implication of Stage II experiences on choice policy at Stage I). The predominant assumption, going back to Tolman, is that humans and animals solve tasks of this sort by online computation, e.g. forward planning when a choice is required during the Test phase (Tolman, 1948), or earlier during acquisition. Accordingly, previous human studies have shown benefits of online reinstatement of past episodes on decision-making in a sensory preconditioning paradigm (Wimmer and Shohamy, 2012), and that goal-directed behavior in sequential decision-making is accompanied by the online reinstatement of prospective states (Doll et al., 2015). Other studies have explored properties of offline reinstatement, showing that it favors past episodes associated with higher reward (Gruber et al., 2016) and supports upcoming learning of related material (Cichy et al., 2012). However, to our knowledge, no human or rodent study has tested the role of offline replay in piecing together distal episodes to support planning. Tolman famously designed revaluation tasks that require the animal to integrate information experienced at various points in time (Tolman, 1948). Thus, we used a simplified Tolman-style revaluation task to test the role of offline replay in integration of distant memories and address the crucial question of which memories the brain would preferentially replay to improve planning. While the rodent literature has shown in various paradigms that offline replay is associated with better task performance (Johnson and Redish, 2005; Ólafsdóttir et al., 2015; Ambrose et al., 2016a), none verifies that these improvements are specifically due to the offline integration of memory. Our present findings further these previous efforts and elucidate the role of offline replay in goal-directed memory integration that supports planning.

It is important to consider alternative hypotheses and interpretations of our results. An important general point is that our results are correlational, and we of course can’t directly draw causal conclusions from the associations we report. Our main result is the association between offline neural reinstatement and subsequent preference change, supporting our hypothesis of a role for the former in replanning. In general, alternative hypotheses in the literature for how replanning occurs tend to focus on online computation. However, evidence for the importance of online computation is not mutually exclusive with the mechanism discussed here, and it cannot itself explain the reported association between planning and offline replay. In order to rule out alternative interpretations, we included a number of controls in the design of our study. As a general matter, the factorial design of our experiment, which includes no-revaluation controls and noise manipulations, and our use of a relative replanning score to control for initial learning, all help to verify that the association we report between replay and behavior is specific to replanning. For instance, if the brain-behavior relationship was confounded by many sorts of nonspecific factors such as variation in overall attention or motivation, or by forgetting rather than strategic replanning, similar brain-behavior relationships should be present in the control blocks as well. Accordingly, we can rule out the alternative interpretation that our results are due to behavioral change driven by ‘any’ surprising event regardless of whether it warrants a change in behavior. To control for this possibility, we included a noisy control condition where participants experienced prediction errors in the experienced rewards but the global structure of the task, and therefore the most rewarding choice, did not change. We did not observe a relationship between replay and revaluation behavior in this control condition. A second alternative interpretation is that any replay, regardless of the category, is associated with revaluation of choice behavior. Such a relationship between nonspecific replay and behavior could reflect a more generic index of task engagement, e.g., higher attention or motivation in participants who are better at the task. This interpretation would predict a relationship between Stage two category replay and replanning as well. However, we found no significant correlation between Stage two replay and replanning (Figure 2—figure supplement 2). Altogether, the specificity of the findings is consistent with the hypothesis that learning from simulated experience during replay determines whether revaluation occurs.

Our second key result concerns a further pair of associations, between earlier signatures of PE and subsequent replay and behavioral planning. This is consistent with a general family of explanations, in which events with certain characteristics (e.g., surprising one) are privileged for later replay. We operationalize this in terms of the neural response to unsigned prediction error during encoding, as suggested by prioritized sweeping (the standard embodiment of this general idea in computer science), but this is meant to be representative of a broader class of heuristics. Our task design does not definitively allow us to distinguish this particular heuristic measure of surprise from other, correlated ones that could support similar effects. For instance, in more elaborate models, overall variability in outcomes can be decomposed into contributions of volatility and stochasticity, and PE can be scaled relative to these (Ambrose et al., 2016b). Similarly, there is evidence that neural replay is sensitive to the sign of prediction errors (Gläscher et al., 2010). However, in the present task, signed and unsigned PEs are highly correlated (in particular, most PEs are positive, because subjects are incentivized to make rewarding choices and not return to disappointing states). Thus, the present design does not have sufficient power to dissociate the role of signed vs. unsigned prediction errors in replay.

Furthermore, although prioritized sweeping works by marking high-PE states when they are experienced – and this echoes many notions of ‘tagging’ in the cognitive neuroscience of memory – given the correlational design we can’t rule out other mechanisms by which the decision which events to replay happens later. For instance, in principle, even without tagging, the brain could retrieve memories offline in an initially unbiased way to search for high-PE ones to focus on. Our data do not directly dissociate the two hypotheses, and both are consistent with our core contribution: that offline replay of surprising events supports planning. Having said this, we noted that, in realistic scenarios (as well as task designs with larger decision trees) such unguided search would be computationally expensive and potentially intractable. We experience many events and form many memories on a daily basis. In order to enhance learning and planning, it would be useful for the brain to have a mechanism to focus simulated experience on those that may lead to policy updates. Thus, while acknowledging other possibilities, we suggest that some form of prioritized replay is more likely.

Given that our interest is primarily in the relationships between prediction errors, replay, and revaluation behavior, we focused our analysis on the difference between revaluation and control conditions, collapsing over blocks with noisy vs. noiseless rewards. We did, however, also consider all results separately across those conditions. Perhaps unsurprisingly (given lower power), we did not find significant differences between the conditions in any of our measures, but we note a few findings. First, we only observed a significant correlation between replay and replanning behavior in revaluation blocks in the noisy reward condition (Figure 2—figure supplement 1). These results may be due to a ceiling effect in the noiseless reward conditions, where updating past policies was sufficiently easy for online processes to adequately do the job. Second, we only found significant conjunction results for the PE analyses in the combined condition, but neither in the noisy-rewards condition nor the noiseless-rewards conditions alone. This is most likely due to the fact that the data in the noisy and noiseless conditions were half the number of runs included in the combined analysis reported in the main text (Figure 3). Future experiments should focus on the differential effect of the noisiness vs. stability of reward structures, and the volatility of event and reward structures more broadly, on brain regions recruited in offline processes that support planning. One possibility is that the brain is more likely to benefit from offline replay for planning when the reward structure in the world is noisier, in which case changes in mean reward may be neglected as variance, making online integration more difficult. While the present study has focused on reward prediction errors, future studies are required to study the role of other forms of prediction errors, e.g. state prediction errors (Dunsmoor and Murphy, 2015).

We also examined the dynamics of our results over the progression of the relearning phase. The key difference between the three rest periods was in the amount of unsigned prediction error (PE, or uncertainty) experienced prior to each (Figure 5). Here, unsigned prediction errors are how the brain notices something is not right, i.e. some prediction does not match the observation, and needs to determine whether the surprising experience may have further consequences for the brain’s stored representations and policies (its model of the world). Before Rest one during revaluation trials, participants had just noticed that rewards have changed from what they expected, leading to the experience of high unsigned PEs while making Stage II decisions. Importantly, they did not visit the Stage I state during the Relearning phase, but only Stage II stimuli, therefore they did not experience any PEs after choosing a policy from Stage I. Before Rest 3 of revaluation trials, participants experienced smaller prediction errors because they had already had the chance to learn the new reward structures for Stage II stimuli; correspondingly, overall levels of replay were lower after Rest three than after Rest 1. However, crucially, while levels of replay were lower after Rest 3, there was still a robust correlation between Rest three replay and behavior (the size of the replay-behavior correlation did not significantly differ across rest periods).

Taken together, these findings suggest a potential two-stage functional role for replay in the present study. First, replay of trajectories marked with unsigned PE could allow identifying distal parts of the state space that may need to be updated. When the brain encounters prediction errors, initial replay can enable search through a graph of memory states to identify other states affected by the experienced PE: this is the ‘sample to tag with priority’ function of replay. The present experiment consists of one such past state, but it is possible that in larger decision trees this search for relevant past states leads to ‘reverse replay’ (Wu and Foster, 2014; Ambrose et al., 2016a). As such key past states that lead to the present state in which high PE was experienced may be tagged with priority over other memories to be replayed later on. Second, once the new structure of the environment is adequately learned, and high PEs are no longer experienced; the tagged relevant past states and their corresponding trajectories can be replayed, simulated if never directly experienced, and updated. A similar tagging idea had been proposed in the domain of fear learning, where memory traces could be retroactively altered in accordance with new behavioral relevance of fear associations (Kurth-Nelson et al., 2016), whereas here we propose a more general case where unsigned PEs tag memories with priority for subsequent integrative replay. As such, the second function of replay may be ‘learning from replay’: to actually enable the brain to piece together and simulate trajectories in order to update past representations and planning policies offline (Pfeiffer and Foster, 2013).

It is worth commenting on how the notion of replay used here relates to the notion of sequential replay in the rodent literature – in this literature, the term ‘replay’ often refers to the replay of sequences at sub-second speed. The temporal resolution of fMRI poses a challenge to providing direct neural evidence of sequential replay. That said, we designed the study such that replanning the optimal Stage I choice required integration of past information by piecing together different memories sequentially: The transition from Stage I to Stage II, on the one hand, and the new Stage II rewards, on the other, were originally experienced at separate distal points in time during online task performance. Unlike animal studies, but similar to previous human fMRI studies of offline replay (Staresina et al., 2013; Cichy et al., 2012; Gruber et al., 2016), here we leverage a controlled design together with evidence of Stage I reinstatement to investigate the role of offline replay in planning. Future work using methods with higher temporal resolution, such as MEG and direct recordings from the hippocampus in patients, is required to shed light on the details of forward and reverse sequential replay in updating representations and memories that serve planning.

In future work, we plan to conduct more detailed explorations of the functional role of replay in planning, using paradigms with larger decision trees and higher temporal resolutions that make it possible to track replay trajectories in more detail; e.g., a recent human study used MEG to identify a role for reverse replay of trajectories (Dayan, 1993). We also plan to explore how offline replay might contribute to generalization and temporal abstraction, e.g. via multi-step predictive representations of upcoming states (or the successor representation, (Russek et al., 2017; Momennejad et al., 2017). Caching of multi-step predictive representations during replay can in turn enable generalization (Botvinick and Weinstein, 2014), faster planning especially under time pressure, and sub-goal discovery (Beckmann et al., 2003) across longer scales. After consolidation, the generalized multi-step representations could lead to abstract representations of states and policies such as schemas. Using generalized representations, both discovery of relevant past states (as in the first function of replay above) and piecing together episodes to simulate trajectories to goal (as the second function of replay above) could also take place in a multi-step fashion consistent with the successor representation (Russek et al., 2017), enabling more efficiency simulation of past and future trajectories.

To summarize, we have provided neural and behavioral evidence for the hypothesis that offline replay supports planning. We further suggest that the brain’s sensitivity to uncertainty, operationalized as unsigned prediction errors, mediates the amount of offline replay and, through this, revaluation behavior. Our findings further our understanding of how the brain leverages offline memory mechanisms in planning and goal-directed behavior.

Neural and behavioral data have been made available online at OpenNeuro (https://openneuro.org/datasets/ds001612/versions/1.0.0).

1. 1. Ida Momennejad
   2. A Ross Otto
   3. Nathaniel D Daw
   4. Kenneth A Norman

   (2018) OpenNeuro

   Neural and behavioral data from Offline Replay Supports Planning in Human Reinforcement Learning.

   https://doi.org/10.18112/openneuro.ds001612.v1.0.1

1. 1. Ambrose RE
   2. Pfeiffer BE
   3. Foster DJ

   (2016a) Reverse Replay of Hippocampal Place Cells Is Uniquely Modulated by Changing Reward

   Neuron 91:1124–1136.

   https://doi.org/10.1016/j.neuron.2016.07.047

   • PubMed
   • Google Scholar
2. 1. Ambrose RE
   2. Pfeiffer BE
   3. Foster DJ

   (2016b) Reverse Replay of Hippocampal Place Cells Is Uniquely Modulated by Changing Reward

   Neuron 91:1124–1136.

   https://doi.org/10.1016/j.neuron.2016.07.047

   • PubMed
   • Google Scholar
3. 1. Anderson MC
   2. Bunce JG
   3. Barbas H

   (2016) Prefrontal-hippocampal pathways underlying inhibitory control over memory

   Neurobiology of Learning and Memory 134:145–161.

   https://doi.org/10.1016/j.nlm.2015.11.008

   • PubMed
   • Google Scholar
4. 1. Beckmann CF
   2. Jenkinson M
   3. Smith SM

   (2003) General multilevel linear modeling for group analysis in FMRI

   NeuroImage 20:1052–1063.

   https://doi.org/10.1016/S1053-8119(03)00435-X

   • PubMed
   • Google Scholar
5. 1. Behrens TE
   2. Woolrich MW
   3. Walton ME
   4. Rushworth MF

   (2007) Learning the value of information in an uncertain world

   Nature Neuroscience 10:1214–1221.

   https://doi.org/10.1038/nn1954

   • PubMed
   • Google Scholar
6. 1. Botvinick M
   2. Weinstein A

   (2014) Model-based hierarchical reinforcement learning and human action control

   Philosophical Transactions of the Royal Society B: Biological Sciences 369:20130480.

   https://doi.org/10.1098/rstb.2013.0480

   • Google Scholar
7. 1. Chen J
   2. Leong YC
   3. Honey CJ
   4. Yong CH
   5. Norman KA
   6. Hasson U

   (2017) Shared memories reveal shared structure in neural activity across individuals

   Nature Neuroscience 20:115–125.

   https://doi.org/10.1038/nn.4450

   • PubMed
   • Google Scholar
8. 1. Cichy RM
   2. Heinzle J
   3. Haynes JD

   (2012) Imagery and perception share cortical representations of content and location

   Cerebral Cortex 22:372–380.

   https://doi.org/10.1093/cercor/bhr106

   • PubMed
   • Google Scholar
9. 1. Dayan P

   (1993) Improving generalization for temporal difference learning: the successor representation

   Neural Computation 5:613–624.

   https://doi.org/10.1162/neco.1993.5.4.613

   • Google Scholar
10. 1. Doll BB
    2. Duncan KD
    3. Simon DA
    4. Shohamy D
    5. Daw ND

    (2015) Model-based choices involve prospective neural activity

    Nature Neuroscience 18:767–772.

    https://doi.org/10.1038/nn.3981

    • PubMed
    • Google Scholar
11. 1. Dunsmoor JE
    2. Murphy GL

    (2015) Categories, concepts, and conditioning: how humans generalize fear

    Trends in Cognitive Sciences 19:73–77.

    https://doi.org/10.1016/j.tics.2014.12.003

    • Google Scholar
12. 1. Eichenbaum H

    (2004) Hippocampus: cognitive processes and neural representations that underlie declarative memory

    Neuron 44:109–120.

    https://doi.org/10.1016/j.neuron.2004.08.028

    • PubMed
    • Google Scholar
13. 1. Fellows LK
    2. Farah MJ

    (2007) The role of ventromedial prefrontal cortex in decision making: judgment under uncertainty or judgment per se?

    Cerebral Cortex 17:2669–2674.

    https://doi.org/10.1093/cercor/bhl176

    • PubMed
    • Google Scholar
14. 1. Gershman SJ
    2. Markman AB
    3. Otto AR

    (2014) Retrospective revaluation in sequential decision making: A tale of two systems

    Journal of Experimental Psychology: General 143:182–194.

    https://doi.org/10.1037/a0030844

    • Google Scholar
15. 1. Gläscher J
    2. Hampton AN
    3. O'Doherty JP

    (2009) Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making

    Cerebral Cortex 19:483–495.

    https://doi.org/10.1093/cercor/bhn098

    • PubMed
    • Google Scholar
16. 1. Gläscher J
    2. Daw N
    3. Dayan P
    4. O'Doherty JP

    (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning

    Neuron 66:585–595.

    https://doi.org/10.1016/j.neuron.2010.04.016

    • PubMed
    • Google Scholar
17. 1. Gruber MJ
    2. Ritchey M
    3. Wang SF
    4. Doss MK
    5. Ranganath C

    (2016) Post-learning Hippocampal Dynamics Promote Preferential Retention of Rewarding Events

    Neuron 89:1110–1120.

    https://doi.org/10.1016/j.neuron.2016.01.017

    • PubMed
    • Google Scholar
18. 1. Ji D
    2. Wilson MA

    (2007) Coordinated memory replay in the visual cortex and hippocampus during sleep

    Nature Neuroscience 10:100–107.

    https://doi.org/10.1038/nn1825

    • PubMed
    • Google Scholar
19. 1. Johnson A
    2. Redish AD

    (2005) Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model

    Neural Networks 18:1163–1171.

    https://doi.org/10.1016/j.neunet.2005.08.009

    • PubMed
    • Google Scholar
20. 1. Kalisch R
    2. Korenfeld E
    3. Stephan KE
    4. Weiskopf N
    5. Seymour B
    6. Dolan RJ

    (2006) Context-dependent human extinction memory is mediated by a ventromedial prefrontal and hippocampal network

    Journal of Neuroscience 26:9503–9511.

    https://doi.org/10.1523/JNEUROSCI.2021-06.2006

    • PubMed
    • Google Scholar
21. 1. Kuhl BA
    2. Rissman J
    3. Chun MM
    4. Wagner AD

    (2011) Fidelity of neural reactivation reveals competition between memories

    PNAS 108:5903–5908.

    https://doi.org/10.1073/pnas.1016939108

    • PubMed
    • Google Scholar
22. 1. Kurth-Nelson Z
    2. Economides M
    3. Dolan RJ
    4. Dayan P

    (2016) Fast Sequences of Non-spatial State Representations in Humans

    Neuron 91:194–204.

    https://doi.org/10.1016/j.neuron.2016.05.028

    • PubMed
    • Google Scholar
23. 1. Lengyel M
    2. Dayan P

    (2008)

    Hippocampal Contributions to Control: The Third Way

    Advances in Neural Information Processing Systems  20:889–896.

    • Google Scholar
24. Preprint

    1. Ludvig EA
    2. Mirian MS
    3. Kehoe EJ
    4. Sutton RS

    (2017) Associative learning from replayed experience

    bioRxiv.

    https://doi.org/10.1101/100800

    • Google Scholar
25. 1. Momennejad I
    2. Russek EM
    3. Cheong JH
    4. Botvinick MM
    5. Daw ND
    6. Gershman SJ

    (2017) The successor representation in human reinforcement learning

    Nature Human Behaviour 1:680–692.

    https://doi.org/10.1038/s41562-017-0180-8

    • Google Scholar
26. 1. Moore AW
    2. Atkeson CG

    (1993) Prioritized sweeping: Reinforcement learning with less data and less time

    Machine Learning 13:103–130.

    https://doi.org/10.1007/BF00993104

    • Google Scholar
27. 1. Murty VP
    2. Tompary A
    3. Adcock RA
    4. Davachi L

    (2016) Selectivity in post-encoding connectivity with high-level visual cortex is associated with reward-motivated memory

    Journal of Neuroscience 15:537–545.

    https://doi.org/10.1523/JNEUROSCI.4032-15.2016

    • Google Scholar
28. 1. Ólafsdóttir HF
    2. Barry C
    3. Saleem AB
    4. Hassabis D
    5. Spiers HJ

    (2015) Hippocampal place cells construct reward related sequences through unexplored space

    eLife 4:e06063.

    https://doi.org/10.7554/eLife.06063

    • PubMed
    • Google Scholar
29. Conference

    1. Peng J
    2. Williams RJ

    (1993) Efficient learning and planning within the Dyna framework

    IEEE International Conference on Neural Networks. pp. 168–174.

    https://doi.org/10.1109/ICNN.1993.298551

    • Google Scholar
30. 1. Pfeiffer BE
    2. Foster DJ

    (2013) Hippocampal place-cell sequences depict future paths to remembered goals

    Nature 497:74–79.

    https://doi.org/10.1038/nature12112

    • PubMed
    • Google Scholar
31. 1. Polyn SM
    2. Natu VS
    3. Cohen JD
    4. Norman KA

    (2005) Category-specific cortical activity precedes retrieval during memory search

    Science 310:1963–1966.

    https://doi.org/10.1126/science.1117645

    • PubMed
    • Google Scholar
32. 1. Preston AR
    2. Eichenbaum H

    (2013) Interplay of hippocampus and prefrontal cortex in memory

    Current Biology 23:R764–R773.

    https://doi.org/10.1016/j.cub.2013.05.041

    • PubMed
    • Google Scholar
33. 1. Ranganath C
    2. Ritchey M

    (2012) Two cortical systems for memory-guided behaviour

    Nature Reviews Neuroscience 13:713–726.

    https://doi.org/10.1038/nrn3338

    • PubMed
    • Google Scholar
34. 1. Russek EM
    2. Momennejad I
    3. Botvinick MM
    4. Gershman SJ
    5. Daw ND

    (2017) Predictive representations can link model-based reinforcement learning to model-free mechanisms

    PLOS Computational Biology 13:e1005768.

    https://doi.org/10.1371/journal.pcbi.1005768

    • PubMed
    • Google Scholar
35. 1. Schlichting ML
    2. Preston AR

    (2014) Memory reactivation during rest supports upcoming learning of related content

    PNAS 111:15845–15850.

    https://doi.org/10.1073/pnas.1404396111

    • PubMed
    • Google Scholar
36. 1. Shenhav A
    2. Botvinick MM
    3. Cohen JD

    (2013) The expected value of control: an integrative theory of anterior cingulate cortex function

    Neuron 79:217–240.

    https://doi.org/10.1016/j.neuron.2013.07.007

    • PubMed
    • Google Scholar
37. 1. Shohamy D
    2. Daw ND

    (2015) Integrating memories to guide decisions

    Current Opinion in Behavioral Sciences 5:85–90.

    https://doi.org/10.1016/j.cobeha.2015.08.010

    • Google Scholar
38. 1. Staresina BP
    2. Alink A
    3. Kriegeskorte N
    4. Henson RN

    (2013) Awake reactivation predicts memory in humans

    PNAS 110:21159–21164.

    https://doi.org/10.1073/pnas.1311989110

    • PubMed
    • Google Scholar
39. 1. Sutton RS

    (1991) Dyna, an integrated architecture for learning, planning, and reacting

    ACM SIGART Bulletin 2:160–163.

    https://doi.org/10.1145/122344.122377

    • Google Scholar
40. 1. Takahashi S

    (2015) Episodic-like memory trace in awake replay of hippocampal place cell activity sequences

    eLife 4:e08105.

    https://doi.org/10.7554/eLife.08105

    • PubMed
    • Google Scholar
41. 1. Takashima A
    2. Nieuwenhuis IL
    3. Jensen O
    4. Talamini LM
    5. Rijpkema M
    6. Fernández G

    (2009) Shift from hippocampal to neocortical centered retrieval network with consolidation

    Journal of Neuroscience 29:10087–10093.

    https://doi.org/10.1523/JNEUROSCI.0799-09.2009

    • PubMed
    • Google Scholar
42. 1. Tambini A
    2. Davachi L

    (2013) Persistence of hippocampal multivoxel patterns into postencoding rest is related to memory

    PNAS 110:19591–19596.

    https://doi.org/10.1073/pnas.1308499110

    • PubMed
    • Google Scholar
43. 1. Tolman EC

    (1948) Cognitive maps in rats and men

    Psychological Review 55:189–208.

    https://doi.org/10.1037/h0061626

    • PubMed
    • Google Scholar
44. 1. van Kesteren MT
    2. Beul SF
    3. Takashima A
    4. Henson RN
    5. Ruiter DJ
    6. Fernández G

    (2013) Differential roles for medial prefrontal and medial temporal cortices in schema-dependent encoding: from congruent to incongruent

    Neuropsychologia 51:2352–2359.

    https://doi.org/10.1016/j.neuropsychologia.2013.05.027

    • PubMed
    • Google Scholar
45. Conference

    1. Van Seijen H
    2. Sutton RS

    (2015)

    A deeper look at planning as learning from replay

    Proceedings of the 32nd International Conference on International Conference on Machine Learning. pp. 2314–2322.

    • Google Scholar
46. 1. Wilson RC
    2. Niv Y

    (2015) Is Model Fitting Necessary for Model-Based fMRI?

    PLOS Computational Biology 11:e1004237.

    https://doi.org/10.1371/journal.pcbi.1004237

    • PubMed
    • Google Scholar
47. 1. Wimmer GE
    2. Shohamy D

    (2012) Preference by association: how memory mechanisms in the hippocampus bias decisions

    Science 338:270–273.

    https://doi.org/10.1126/science.1223252

    • PubMed
    • Google Scholar
48. 1. Woolrich MW
    2. Behrens TE
    3. Beckmann CF
    4. Jenkinson M
    5. Smith SM

    (2004) Multilevel linear modelling for FMRI group analysis using Bayesian inference

    NeuroImage 21:1732–1747.

    https://doi.org/10.1016/j.neuroimage.2003.12.023

    • PubMed
    • Google Scholar
49. 1. Woolrich M

    (2008) Robust group analysis using outlier inference

    NeuroImage 41:286–301.

    https://doi.org/10.1016/j.neuroimage.2008.02.042

    • PubMed
    • Google Scholar
50. 1. Wu X
    2. Foster DJ

    (2014) Hippocampal replay captures the unique topological structure of a novel environment

    Journal of Neuroscience 34:6459–6469.

    https://doi.org/10.1523/JNEUROSCI.3414-13.2014

    • PubMed
    • Google Scholar

Author details

1. Ida Momennejad

   Princeton Neuroscience Institute, Princeton University, New Jersey, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Supervision, Validation, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   idam@princeton.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-0830-3973

2. A Ross Otto

   Department of Psychology, McGill University, Montreal, Canada

   Contribution

   Conceptualization, Data curation, Formal analysis, Methodology, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-9997-1901

3. Nathaniel D Daw

   Princeton Neuroscience Institute, Princeton University, New Jersey, United States

   Contribution

   Conceptualization, Resources, Software, Supervision, Funding acquisition, Methodology, Project administration, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5029-1430

4. Kenneth A Norman

   Princeton Neuroscience Institute, Princeton University, New Jersey, United States

   Contribution

   Conceptualization, Resources, Software, Supervision, Funding acquisition, Methodology, Project administration, Writing—review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5887-9682

• 5,157

  views

• 773

  downloads

• 97

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.