In an uncertain world adaptive behavior requires us to carefully balance the exploration of new opportunities with the exploitation of known resources. Finding the optimal balance between exploration and exploitation is a hard computational problem and there is considerable interest in understanding how humans and animals strike this balance in practice (Badre et al., 2012; Cavanagh et al., 2011; Cohen et al., 2007; Daw et al., 2006; Frank et al., 2009; Hills et al., 2015; Mehlhorn et al., 2015; Wilson et al., 2014). Recent work has suggested that humans use two distinct strategies to solve the explore-exploit dilemma: directed exploration, based on information seeking, and random exploration, based on decision noise (Wilson et al., 2014). Even though both of these strategies serve the same purpose, that is, balancing exploration and exploitation, it is likely they rely on different cognitive mechanisms. Directed exploration is driven by information and is thought to be computationally complex (Gittins and Jones, 1979; Auer et al., 2002; Gittins, 1974). On the other hand, random exploration can be implemented in a simpler fashion by using neural or environmental noise to randomize choice (Thompson, 1933).

A key question is whether these dissociable behavioral strategies rely on dissociable neural systems. Of particular interest is the frontopolar cortex (FPC) – an area that has been associated with a number of functions, such as tracking pending and/or alternate options (Koechlin and Hyafil, 2007; Boorman et al., 2009), strategies (Domenech and Koechlin, 2015) and goals (Pollmann, 2016) and that has been implicated in exploration itself (Badre et al., 2012; Cavanagh et al., 2011; Daw et al., 2006). Importantly, however, the exact role that FPC plays in exploration is unknown as how exploration is defined varies from paper to paper. In one line of work, exploration is defined as information seeking. Understood this way, exploration correlates with RFPC activity measured via fMRI (Badre et al., 2012) and a frontal theta component in EEG (Cavanagh et al., 2011), suggesting a role for RFPC in directed exploration. However, in another line of work, exploration is operationalized differently, as choosing the low value option, not the most informative. Such a measure of exploration is more consistent with random exploration where decision noise drives the sampling of low value options by chance. Defined in this way, exploratory choice correlates with lateral FPC activation (Daw et al., 2006) and stimulation and inhibition of RFPC with direct current (tDCS) can increase and decrease the frequency with which such exploratory choices occur (Raja Beharelle et al., 2015).

Taken together, these two sets of findings suggest that RFPC plays a crucial role in both directed and random exploration. However, we believe that such a conclusion is premature because of a subtle confound that arises between reward and information in most explore-exploit tasks. This confound arises because participants only gain information from the options they choose, yet are incentivized to choose more rewarding options. Thus, over many trials, participants gain more information about more rewarding options and the two ways of defining exploration, that is, choosing high information or low reward options, become confounded (Wilson et al., 2014). This makes it impossible to tell whether the link between RFPC and exploration is specific to either directed or random exploration, or whether it is general to both.

To distinguish these interpretations and investigate the causal role of FPC in directed and random exploration, we used continuous theta-burst TMS (Huang et al., 2005) to selectively inhibit right frontopolar cortex (RFPC) in participants performing the ‘Horizon Task’, an explore-exploit task specifically designed to separate directed and random exploration (Wilson et al., 2014). Using this task we find evidence that inhibition of RFPC selectively inhibits directed exploration while leaving random exploration intact.

We used our previously published ‘Horizon Task’ (Figure 1) to measure the effects of TMS stimulation of RFPC on directed and random exploration. In this task, participants play a set of games in which they make choices between two slot machines (one-armed bandits) that pay out rewards from different Gaussian distributions. To maximize their rewards in each game, participants need to exploit the slot machine with the highest mean, but they cannot identify this best option without exploring both options first.

Figure 1

Download asset Open asset

The Horizon Task has two key manipulations that allow us to measure directed and random exploration. The first manipulation is the horizon itself, i.e. the number of decisions remaining in each game. The idea behind this manipulation is that when the horizon is long (6 trials), participants should explore more frequently, because any information they acquire from exploring can be used to make better choices later on. In contrast, when the horizon is short (1 trial), participants should exploit the option they believe to be best. Thus, this task allows us to quantify directed and random exploration as changes in information seeking and behavioral variability that occur with horizon.

The second manipulation is the amount of information participants have about each option before making their first choice. This information manipulation is achieved by using four forced-choice trials, in which participants are told which option to pick, at the start of each game. We use these forced-choice trials to setup one of two information conditions: an unequal, or (Aston-Jones and Cohen, 2005; Badre et al., 2012), condition, in which participants see 1 play from one option and 3 plays from the other option, and an unequal, or (Auer et al., 2002; Auer et al., 2002), condition, in which participants see two outcomes from both options. By varying the amount of information participants have about each option independent of the mean payout of that option, this information manipulation allows us to remove the reward-information confound, at least on the first free-choice trial (Figure 2). After the first free-choice trial, however, participants tend to choose more rewarding options more frequently and reward and information are rapidly confounded. For this reason the bulk of our analyses are focussed on the first free-choice trial where the confound has been removed.

Figure 2

Download asset Open asset

RFPC stimulation selectively inhibits directed exploration on the first free-choice

In this section we analyze behavior on the first free-choice trial in each game. This way we are able to remove any effect of the reward-information confound and fairly compare behavior between horizon conditions. We analyze the data with both a model-free approach, using simple statistics of the data to quantify directed and random exploration, as well as a model-based approach, using a cognitive model of the behavior to draw more precise conclusions. Both analyses point to the same conclusion that RFPC stimulation selectively inhibits directed, but not random, exploration.

Model-free analysis

The two information conditions in the Horizon Task allow us to quantify directed and random exploration in a model-free way. In particular, directed exploration, which involves information seeking, can be quantified as the probability of choosing the high information option, ${\displaystyle p(\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o})}$ in the [1 3] condition, while random exploration, which involves decision noise, can be quantified as the probability of making a mistake, or choosing the low mean reward option, ${\displaystyle p(\mathrm{l}\mathrm{o}\mathrm{w}\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n})}$, in the [2 2] condition.

Using these measures of exploration, we found that inhibiting the RFPC had a significant effect on directed exploration but not random exploration (Figure 3A,B). In particular, for directed exploration, a repeated measures ANOVA with horizon, TMS condition and order as factors revealed a significant interaction between stimulation condition and horizon ($F(1,24)=4.96$, $p=0.036$). Conversely, a similar analysis for random exploration revealed no effects of stimulation condition (main effect of stimulation condition, $F(1,24)=0.88$, $p=0.36$; interaction of stimulation condition with horizon, $F(1,24)=1.24$, $p=0.28$). Post hoc analyses revealed that the change in directed exploration was driven by changes in information seeking in horizon 6 (one-sided t-test, $t(24)=2.62$, $p=0.008$) and not in horizon 1 (two-sided t-test, ${\displaystyle t(24)=-0.30}$).

Figure 3

Download asset Open asset

Model-based analysis

While the model-free analyses are intuitive, the model-free statistics, ${\displaystyle p(\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o})}$ and ${\displaystyle p(\mathrm{l}\mathrm{o}\mathrm{w}\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n})}$, are not pure reflections of information seeking and behavioral variability and could be influenced by other factors such as spatial bias and learning. To account for these possibilities we performed a model-based analysis using a model that extends our earlier work (Wilson et al., 2014; Somerville et al., 2017; Krueger et al., 2017) see Materials and methods for a complete description. In this model, the level of directed and random exploration is captured by two parameters: an information bonus for directed exploration, and decision noise for random exploration. In addition the model includes terms for the spatial bias and to describe learning.

Overview of model

Before presenting the results of the model-based analysis we begin with a brief overview of the most salient points of the model. A full description of the model can be found in the Methods and code to implement the model can be found in the Supplementary Material.

Conceptually, the model breaks the explore-exploit choice down into two components: a learning component, in which participants estimate the mean payoff of each option from the rewards they see, and a decision component, in which participants use this estimated payoff to guide their choice. The learning component assumes that participants compute an estimate of the average payoff for each slot machine, $R_t^i$, using a simple delta rule update equation (based on a Kalman filter (Kalman, 1960), see Materials and methods):

(1) $R_{t+1}^i=R_t^i+{\alpha }_t^i(r_t-R_t^i)$

where $r_t$ is the reward on trial $t$ and ${\alpha }_t^i$ is the time-varying learning rate that determines the extent to which the prediction error, $(r_t-R_t^i)$, updates the estimate of the mean of bandit $i$. The learning process is described by three free parameters: the initial value of the estimated payoff, $R_0$, and two learning rates, the initial learning rate, ${\alpha }_1$, and the asymptotic learning rate, ${\alpha }_{inf}$, which together describe the evolution of the actual learning rate, ${\alpha }_t$, over time. For simplicity, we assume that these parameters are independent of horizon and uncertainty condition (Table 1).

Table 1

Parameter	Horizon dependent?	Uncertainty dependent?	TMS dependent?
prior mean, $R_0$	no	no	yes
initial learning rate, ${\alpha }_1$	no	no	yes
asymptotic learning rate, ${\alpha }_{\mathrm{\infty }}$	no	no	yes
information bonus, $A$	yes	n/a	yes
spatial bias, $B$	yes	yes	yes
decision noise, $\sigma$	yes	yes	yes

The decision component of the model assumes that participants choose between the two options (left and right) probabilistically according to.

(2) ${\displaystyle p(\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{o}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t})=\frac{1}{1+\exp \left(\frac{\mathrm{\Delta }R+A\mathrm{\Delta }I+B}{\sigma }\right)}}$

where $\mathrm{\Delta }R$ ( $=R_t^{left}-R_t^{right}$ ) is the difference in expected reward between left and right options and $\mathrm{\Delta }I$ is the difference in information between left and right options (which we define as +1 when left is more informative, −1 when right is more informative, and 0 when both options convey equal information in the [2 2] condition). The decision process is described by three free parameters: the information bonus $A$, the spatial bias $B$, and the decision noise $\sigma$. We estimate separate values of the decision parameters for each horizon and (since the information bonus is only used in the [1 3] condition) separate values of only the bias and decision noise for each uncertainty condition.

Overall, subject’s behavior in each session (vertex vs RFPC stimulation) is described by 13 free parameters (Table 1): three describing learning ($R_0$, ${\alpha }_1$ and ${\alpha }_{\mathrm{\infty }}$) and 10 describing the decision process ($A$ in the two horizon conditions, $B$ and $\sigma$ in the four horizon-x-uncertainty conditions). These 13 parameters were fit to each subject in each stimulation condition using a hierarchical Bayesian approach (Lee and Wagenmakers, 2014) (see Materials and methods).

Model fitting results

Posterior distributions over the group-level means are shown in the left column of Figure 4, while posteriors over the TMS-related change in parameters are shown in the right column. Both columns suggest a selective effect of RFPC stimulation on the information bonus in horizon 6.

Figure 4

Download asset Open asset

Focussing on the left column first, overall the parameter values seem reasonable. The prior mean is close to the generative mean of 50 used in the actual experiment, and the decision parameters are comparable to those found in our previous work (Wilson et al., 2014). The learning rate parameters, ${\alpha }_1$ and ${\alpha }_{\mathrm{\infty }}$, were not included in our previous models and are worth discussing in more detail. As expected for Bayesian learning (Kalman, 1960; Nassar et al., 2010), the initial learning rate is higher than the asymptotic learning rate (95% of samples in the vertex condition, 94% in the RFPC condition). However, the actual values of the learning rates are quite far from their ‘optimal’ settings of ${\alpha }_1=1$ and ${\alpha }_{\mathrm{\infty }}=0$ that would correspond to perfectly computing the mean reward. This suggests a greater than optimal reliance on the prior (${\alpha }_1<1$) and a pronounced recency bias (${\alpha }_{\mathrm{\infty }}>0$) such that the most recent rewards are weighted more heavily in the computation of expected reward, $R_t^i$. Both of these findings are likely due to the fact that the version of the task we employed did not keep the outcomes of the forced trials on screen and instead relied on people’s memories to compute the expected value.

Turning to the right hand column of Figure 4, we can see that the model-based analysis yields similar result to the model-free analysis. In particular we see a reduction (of about 4.8 points) in the information bonus in horizon 6 (with 99% of samples showing a reduced information bonus in the RFPC stimulation condition) and no effect on decision noise in either horizon in either the [2 2] or [1 3] uncertainty conditions (with between 40% and 63% of samples below zero).

In addition to the effect on the information bonus in horizon 6, there is also a hint of an effect on the information bonus in horizon 1 (85% samples less than zero) and on the prior mean $R_0$ (88% samples above zero). While these results may suggest that RFPC stimulation affects more than just information bonus in horizon 6, they more likely reflect an inherent tradeoff between prior mean and information bonus that is peculiar to this task. In particular, because the prior mean has a stronger effect on the more uncertain option, an increase in $R_0$ increases the value of the more informative option in much the same way as an information bonus. Thus, when applied to this task, the model has a built in tradeoff between prior mean and information bonus that can muddy the interpretation of both. Note that this tradeoff is not a general feature of the model and could be removed with a different task design that employed more forced choice trials and hence more time for the effects of the prior to be removed.

Figure 5 exposes the tradeoff between $R_0$ and $A$ in more detail. Panels A and B plot samples from the posterior over the TMS-related change in information bonus, ${\displaystyle A(\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{e}\mathrm{x})-A(\mathrm{R}\mathrm{F}\mathrm{P}\mathrm{C})}$, against the TMS-related change in prior mean, ${\displaystyle R_0(\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{e}\mathrm{x})-R_0(\mathrm{R}\mathrm{F}\mathrm{P}\mathrm{C})}$. For both horizon conditions we see a strong negative correlation such that increasing $R_0$ decreases $A$. This negative correlation is especially problematic for the interpretation of the horizon 1 change in information bonus where a sizable fraction of the posterior centers on no change in either variable. In contrast the negative correlation between $A$ and $R_0$ does not affect our interpretation of the horizon 6 result where the TMS-related change in $A$ is negative regardless of of the change in $R_0$.

Figure 5

Download asset Open asset

Finally we asked whether the horizon-dependent change in information seeking, i.e. $\mathrm{\Delta }A=A(h=6)-A(h=1)$, was different in each TMS condition. As shown in Figure 5C, the TMS-related change in $\mathrm{\Delta }A$ is about −3.1 points (94% samples below 0) and is uncorrelated with the TMS-related change in $R_0$. Taken together, this suggests that we can be fairly confident in our claim that RFPC stimulation has a selective effect on directed exploration.

The effect of RFPC stimulation on later trials

Our analyses so far have focussed on just the first free choice and have ignored the remaining five choices in the horizon six games. The reason for this is the reward-information confound, illustrated in Figure 2, which makes interpretation of the later trials more difficult. Despite this difficulty, we note that in Figure 2 the size of the confound is almost identical in the two stimulation conditions and so we proceed, with caution, to present a model-free analysis of the later trials below.

In Figure 6 we plot the model-free measures, ${\displaystyle p(\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o})}$ and ${\displaystyle p(\mathrm{l}\mathrm{o}\mathrm{w}\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n})}$, as a function of trial number. Both measures show a decrease over the course of the horizon six games although, because of the confound, it is difficult to say whether these changes reflect a reduction in directed exploration, random exploration, or both. Interestingly, the differences in ${\displaystyle p(\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o})}$ between vertex and RFPC conditions on the first free-choice trial appear to persist into the second, a result that becomes more apparent when we plot the TMS-related change, that is, ${\displaystyle p(\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o},\mathrm{R}\mathrm{F}\mathrm{P}\mathrm{C})-p(\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o},\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\mathrm{e}\mathrm{x})}$ (Figure 6C,D). More formally a repeated measures ANOVA with trial number, TMS condition as factors reveals a significant main effect of trial number ($F(5,120)=126$, $p<{10}^{-45}$), no main effect of TMS condition ($F(1,120)=1.17$, $p=0.29$) and a near significant interaction between trial number and TMS condition ($F(5,120)=2.26,p=0.053$). A post hoc, one-sided t-test on the second trial reveals a marginally significant reduction in ${\displaystyle p(\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{i}\mathrm{n}\mathrm{f}\mathrm{o})}$ on the second trial (${\displaystyle t(24)=1.61}$). In contrast, a similar analysis for random exploration shows no evidence for any effect of TMS condition on ${\displaystyle p(\mathrm{l}\mathrm{o}\mathrm{w}\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n})}$ (main effect of TMS, $F(1,120)=0.16$, $p=0.69$; TMS x trial number, $F(5,120)=0.69$, $p=0.63$) although the main effect of trial number persists ($F(5,120)=13.7$, $p<{10}^{-9}$). Thus, the analysis of later trials provides additional, albeit modest, support for the idea that RFPC stimulation selectively disrupts directed but not random exploration at long horizons.

Figure 6

Download asset Open asset

In this work we used continuous theta-burst transcranial magnetic stimulation (cTBS) to investigate whether right frontopolar cortex (RFPC) is causally involved in directed and random exploration. Using a task that is able to behaviorally dissociate these two types of exploration, we found that inhibition of RFPC caused a selective reduction in directed, but not random exploration. To the best of our knowledge, this finding represents the first causal evidence that directed and random exploration rely on dissociable neural systems and is consistent with our recent findings showing that directed and random exploration have different developmental profiles (Somerville et al., 2017). This suggests that, contrary to the assumption underlying many contemporary studies (Daw et al., 2006; Badre et al., 2012), exploration is not a unitary process, but a dual process in which the distinct strategies of information seeking and choice randomization are implemented via distinct neural systems.

Such a dual-process view of exploration is consistent with the classical idea that there are multiple types of exploration (Berlyne, 1966). In particular Berlyne’s constructs of ‘specific exploration’, involving a drive for information and ‘diversive exploration’, involving a drive for variety, bear a striking resemblance to our definitions of directed and random exploration. Despite the importance of Berlyne’s work, more modern views of exploration tend not to make the distinction between different types of exploration, considering instead a single exploratory state or exploratory drive that controls information seeking across a wide range of tasks (Berlyne, 1966; Aston-Jones and Cohen, 2005; Hills et al., 2015; Kidd and Hayden, 2015). At face value, such unitary accounts seem at odds with a dual-process view of exploration. However, these two viewpoints can be reconciled if we allow for the possibility that, while directed and random exploration are implemented by different systems, their levels are set by a common exploratory drive.

Intriguingly, individual differences in behavior on the Horizon Task provide some support for the idea that directed and random exploration are driven by a common source. In particular, in a large behavioral data set of 277 people performing the Horizon Task, we find a positive correlation between the levels of directed and random exploration such that people with high levels of directed exploration also tend to have high levels random exploration ($r(275)=0.29$, $p<{10}^{-5}$), Figure 7. This is consistent with the idea that the levels of directed and random exploration are set by the strength of an exploratory drive that varies as an individual difference between people.

Figure 7

Download asset Open asset

While the present study does allow us to conclude that directed and random exploration rely on different neural systems, the limited spatial specificity of TMS limits our ability to say exactly what those systems are. In particular, because the spatial extent of TMS is quite large, stimulation aimed at frontal pole may directly affect activity in nearby areas such as ventromedial prefrontal cortex (vmPFC) and orbitofrontal cortex (OFC), both areas that have been implicated in exploratory decision making and that may be contributing to our effect (Daw et al., 2006). In addition to such direct effects of TMS on nearby regions, indirect changes in areas that are connected to the frontal pole could also be driving our effect. For example, cTBS of left frontal pole has been associated with changes in blood perfusion in areas such as amygdala, fusiform gyrus and posterior parietal cortex (Volman et al., 2011) and by changes in the fMRI BOLD signal in OFC, insula and striatum (Hanlon et al., 2017). In addition (Volman et al., 2011) showed that unilateral cTBS of left frontal pole is associated with changes in blood perfusion to the right frontal pole. Indeed, such a bilateral effect of cTBS may explain why our intervention was effective at all given that a number of neuroimaging studies have shown bilateral activation of the frontal pole associated with exploration (Daw et al., 2006; Badre et al., 2012). Future work combining cTBS with neuroimaging will be necessary to shed light on these issues.

With the above caveats that our results may not be entirely due to disruption of frontal pole, the interpretation that RFPC plays a role in directed, but not random, exploration is consistent with a number of previous findings. For example, frontal pole has been associated with tracking the value of the best unchosen option (Boorman et al., 2009), inferring the reliability of alternate strategies (Boorman et al., 2009; Domenech and Koechlin, 2015), arbitrating between old and new strategies (Donoso et al., 2014; Mansouri et al., 2015), and reallocating cognitive resources among potential goals in underspecified situations (Pollmann, 2016). Taken together, these findings suggest a role for frontal pole in model-based decisions (Daw et al., 2006) that involve long-term planning and the consideration of alternative actions. From this perspective, it is perhaps not surprising that directed exploration relies on RFPC, since computing an information bonus relies heavily on an internal model of the world. It is also perhaps not surprising that random exploration is independent of RFPC, as this simpler strategy could be implemented without reference to an internal model. Indeed, the ability to explore effectively in a model-free manner, may be an important function of random exploration as it allows us to explore even when our model of the world is wrong.

More generally, it is unlikely that frontal pole is the only area involved in directed exploration, and more work will be needed to map out the areas involved in directed and random exploration and expose their causal relationship to explore-exploit behavior.

1. 1. Zajkowski
   2. W
   3. Kossut
   4. M
   5. Wilson
   6. RC

   (2017) A causal role for right frontopolar cortex in directed, but not random, exploration

   Publicly accessible via the Harvard Dataverse website (https://dx.doi.org/10.7910/DVN/CZT6EE).

   https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/CZT6EE

1. 1. Aston-Jones G
   2. Cohen JD

   (2005) Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance

   The Journal of Comparative Neurology 493:99–110.

   https://doi.org/10.1002/cne.20723

   • PubMed
   • Google Scholar
2. 1. Auer P
   2. Cesa-Bianchi N
   3. Fischer P

   (2002) Finite-time analysis of the multiarmed bandit problem

   Machine Learning 47:235–256.

   https://doi.org/10.1023/A:1013689704352

   • Google Scholar
3. 1. Badre D
   2. Doll BB
   3. Long NM
   4. Frank MJ

   (2012) Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration

   Neuron 73:595–607.

   https://doi.org/10.1016/j.neuron.2011.12.025

   • PubMed
   • Google Scholar
4. 1. Berlyne DE

   (1966) Curiosity and exploration

   Science 153:25–33.

   https://doi.org/10.1126/science.153.3731.25

   • PubMed
   • Google Scholar
5. 1. Boorman ED
   2. Behrens TE
   3. Woolrich MW
   4. Rushworth MF

   (2009) How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action

   Neuron 62:733–743.

   https://doi.org/10.1016/j.neuron.2009.05.014

   • PubMed
   • Google Scholar
6. 1. Cavanagh JF
   2. Bismark AJ
   3. Frank MJ
   4. Allen JJ

   (2011) Larger Error Signals in Major Depression are Associated with Better Avoidance Learning

   Frontiers in Psychology 2:331.

   https://doi.org/10.3389/fpsyg.2011.00331

   • PubMed
   • Google Scholar
7. 1. Cohen JD
   2. McClure SM
   3. Yu AJ

   (2007) Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration

   Philosophical Transactions of the Royal Society B: Biological Sciences 362:933–942.

   https://doi.org/10.1098/rstb.2007.2098

   • PubMed
   • Google Scholar
8. 1. Costa A
   2. Oliveri M
   3. Barban F
   4. Bonnì S
   5. Koch G
   6. Caltagirone C
   7. Carlesimo GA

   (2013) The right frontopolar cortex is involved in visual-spatial prospective memory

   PLoS ONE 8:e56039.

   https://doi.org/10.1371/journal.pone.0056039

   • PubMed
   • Google Scholar
9. 1. Costa A
   2. Oliveri M
   3. Barban F
   4. Torriero S
   5. Salerno S
   6. Lo Gerfo E
   7. Koch G
   8. Caltagirone C
   9. Carlesimo GA

   (2011) Keeping memory for intentions: a cTBS investigation of the frontopolar cortex

   Cerebral Cortex 21:2696–2703.

   https://doi.org/10.1093/cercor/bhr052

   • PubMed
   • Google Scholar
10. 1. Daw ND
    2. O'Doherty JP
    3. Dayan P
    4. Seymour B
    5. Dolan RJ

    (2006) Cortical substrates for exploratory decisions in humans

    Nature 441:876–879.

    https://doi.org/10.1038/nature04766

    • PubMed
    • Google Scholar
11. 1. Domenech P
    2. Koechlin E

    (2015) Executive control and decision-making in the prefrontal cortex

    Current Opinion in Behavioral Sciences 1:101–106.

    https://doi.org/10.1016/j.cobeha.2014.10.007

    • Google Scholar
12. 1. Donoso M
    2. Collins AG
    3. Koechlin E

    (2014) Human cognition. Foundations of human reasoning in the prefrontal cortex

    Science 344:1481–1486.

    https://doi.org/10.1126/science.1252254

    • PubMed
    • Google Scholar
13. 1. Frank MJ
    2. Doll BB
    3. Oas-Terpstra J
    4. Moreno F

    (2009) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation

    Nature Neuroscience 12:1062–1068.

    https://doi.org/10.1038/nn.2342

    • PubMed
    • Google Scholar
14. 1. Gittins JC
    2. Jones DM

    (1979) A dynamic allocation index for the discounted multiarmed bandit problem

    Biometrika 66:561.

    https://doi.org/10.1093/biomet/66.3.561

    • Google Scholar
15. 1. Gittins JC

    (1974) Resource allocation in speculative chemical research

    Journal of Applied Probability 11:255.

    https://doi.org/10.1017/S0021900200036718

    • Google Scholar
16. 1. Gratton C
    2. Lee TG
    3. Nomura EM
    4. D'Esposito M

    (2013) The effect of theta-burst TMS on cognitive control networks measured with resting state fMRI

    Frontiers in Systems Neuroscience 7:124.

    https://doi.org/10.3389/fnsys.2013.00124

    • PubMed
    • Google Scholar
17. 1. Hanlon CA
    2. Dowdle LT
    3. Correia B
    4. Mithoefer O
    5. Kearney-Ramos T
    6. Lench D
    7. Griffin M
    8. Anton RF
    9. George MS

    (2017) Left frontal pole theta burst stimulation decreases orbitofrontal and insula activity in cocaine users and alcohol users

    Drug and Alcohol Dependence 178:310–317.

    https://doi.org/10.1016/j.drugalcdep.2017.03.039

    • PubMed
    • Google Scholar
18. 1. Hills TT
    2. Todd PM
    3. Lazer D
    4. Redish AD
    5. Couzin ID
    6. Cognitive Search Research Group

    (2015) Exploration versus exploitation in space, mind, and society

    Trends in Cognitive Sciences 19:46–54.

    https://doi.org/10.1016/j.tics.2014.10.004

    • PubMed
    • Google Scholar
19. 1. Huang YZ
    2. Edwards MJ
    3. Rounis E
    4. Bhatia KP
    5. Rothwell JC

    (2005) Theta burst stimulation of the human motor cortex

    Neuron 45:201–206.

    https://doi.org/10.1016/j.neuron.2004.12.033

    • PubMed
    • Google Scholar
20. 1. Kalman RE

    (1960) A New Approach to Linear Filtering and Prediction Problems

    Journal of Basic Engineering 82:35–45.

    https://doi.org/10.1115/1.3662552

    • Google Scholar
21. 1. Kidd C
    2. Hayden BY

    (2015) The psychology and neuroscience of curiosity

    Neuron 88:449–460.

    https://doi.org/10.1016/j.neuron.2015.09.010

    • PubMed
    • Google Scholar
22. 1. Koechlin E
    2. Hyafil A

    (2007) Anterior prefrontal function and the limits of human decision-making

    Science 318:594–598.

    https://doi.org/10.1126/science.1142995

    • PubMed
    • Google Scholar
23. 1. Krueger PM
    2. Wilson RC
    3. Cohen JD

    (2017)

    Strategies for exploration in the domain of losses

    Judgment and Decision Making 12:104–117.

    • Google Scholar
24. Book

    1. Lee MD
    2. Wagenmakers E

    (2014)

    Bayesian Cognitive Modeling: A Practical Course

    Cambridge University Press.

    • Google Scholar
25. 1. Mansouri FA
    2. Buckley MJ
    3. Mahboubi M
    4. Tanaka K

    (2015) Behavioral consequences of selective damage to frontal pole and posterior cingulate cortices

    PNAS 112:E3940–E3949.

    https://doi.org/10.1073/pnas.1422629112

    • PubMed
    • Google Scholar
26. 1. Mehlhorn K
    2. Newell BR
    3. Todd PM
    4. Lee MD
    5. Morgan K
    6. Braithwaite VA
    7. Hausmann D
    8. Fiedler K
    9. Gonzalez C

    (2015) Unpacking the exploration–exploitation tradeoff: A synthesis of human and animal literatures

    Decision 2:191–215.

    https://doi.org/10.1037/dec0000033

    • Google Scholar
27. 1. Nassar MR
    2. Wilson RC
    3. Heasly B
    4. Gold JI

    (2010) An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment

    Journal of Neuroscience 30:12366–12378.

    https://doi.org/10.1523/JNEUROSCI.0822-10.2010

    • PubMed
    • Google Scholar
28. 1. Peirce JW

    (2007) PsychoPy--Psychophysics software in Python

    Journal of Neuroscience Methods 162:8–13.

    https://doi.org/10.1016/j.jneumeth.2006.11.017

    • PubMed
    • Google Scholar
29. 1. Plummer M

    (2003)

    JAGS: a program for analysis of bayesian graphical models using gibbs sampling

    Proceedings of the 3rd International Workshop on Distributed Statistical Computing 124:125.

    • Google Scholar
30. 1. Pollmann S

    (2016) Frontopolar resource allocation in human and nonhuman primates

    Trends in Cognitive Sciences 20:84–86.

    https://doi.org/10.1016/j.tics.2015.11.006

    • PubMed
    • Google Scholar
31. 1. Raja Beharelle A
    2. Polanía R
    3. Hare TA
    4. Ruff CC

    (2015) Transcranial stimulation over frontopolar cortex elucidates the choice attributes and neural mechanisms used to resolve exploration-exploitation trade-offs

    Journal of Neuroscience 35:14544–14556.

    https://doi.org/10.1523/JNEUROSCI.2322-15.2015

    • PubMed
    • Google Scholar
32. 1. Rescorla RA
    2. Wagner AR

    (1972)

    A theory of pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement

    Classical Conditioning II: Current Research and Theory 2:64–99.

    • Google Scholar
33. 1. Rossi S
    2. Hallett M
    3. Rossini PM
    4. Pascual-Leone A
    5. Safety of TMS Consensus Group

    (2009) Safety, ethical considerations, and application guidelines for the use of transcranial magnetic stimulation in clinical practice and research

    Clinical Neurophysiology 120:2008–2039.

    https://doi.org/10.1016/j.clinph.2009.08.016

    • PubMed
    • Google Scholar
34. 1. Schultz W
    2. Dayan P
    3. Montague PR

    (1997) A neural substrate of prediction and reward

    Science 275:1593–1599.

    https://doi.org/10.1126/science.275.5306.1593

    • PubMed
    • Google Scholar
35. 1. Somerville LH
    2. Sasse SF
    3. Garrad MC
    4. Drysdale AT
    5. Abi Akar N
    6. Insel C
    7. Wilson RC

    (2017) Charting the expansion of strategic exploratory behavior during adolescence

    Journal of Experimental Psychology: General 146:155–164.

    https://doi.org/10.1037/xge0000250

    • PubMed
    • Google Scholar
36. 1. Thompson WR

    (1933) On the likelihood that one unknown probability exceeds another in view of the evidence of two samples

    Biometrika 25:285–294.

    https://doi.org/10.1093/biomet/25.3-4.285

    • Google Scholar
37. 1. Volman I
    2. Roelofs K
    3. Koch S
    4. Verhagen L
    5. Toni I

    (2011) Anterior prefrontal cortex inhibition impairs control over social emotional actions

    Current Biology 21:1766–1770.

    https://doi.org/10.1016/j.cub.2011.08.050

    • PubMed
    • Google Scholar
38. 1. Wilson RC
    2. Geana A
    3. White JM
    4. Ludvig EA
    5. Cohen JD

    (2014) Humans use directed and random exploration to solve the explore-exploit dilemma

    Journal of Experimental Psychology: General 143:2074–2081.

    https://doi.org/10.1037/a0038199

    • PubMed
    • Google Scholar
39. 1. Wischnewski M
    2. Schutter DJ

    (2015) Efficacy and time course of theta burst stimulation in healthy humans

    Brain Stimulation 8:685–692.

    https://doi.org/10.1016/j.brs.2015.03.004

    • PubMed
    • Google Scholar

Author details

1. Wojciech K Zajkowski

   University of Social Sciences and Humanities, Warsaw, Poland

   Contribution

   Conceptualization, Formal analysis, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   Competing interests

   No competing interests declared

2. Malgorzata Kossut

   1. Department of Psychology, University of Social Sciences and Humanities, Warsaw, Poland
   2. Nencki Institute of Experimental Biology, Warsaw, Poland

   Contribution

   Resources, Supervision, Project administration

   Competing interests

   No competing interests declared

3. Robert C Wilson

   1. Department of Psychology, University of Arizona, Tucson, United States
   2. Cognitive Science Program, University of Arizona, Tucson, United States

   Contribution

   Conceptualization, Software, Formal analysis, Supervision, Visualization, Writing—original draft, Writing—review and editing

   For correspondence

   bob@arizona.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-2963-2971

• 3,580

  views

• 468

  downloads

• 114

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.