In recent decades, cognitive neuroscience has made breakthroughs in computational modeling, demonstrating that reinforcement learning (RL) models can explain foundational aspects of human thought and behavior. RL models can explain not only simple cognitive processes such as stimulus-outcome and stimulus-response learning (Schultz et al., 1997; O’Doherty et al., 2004; Gläscher et al., 2009), but also highly complex processes, including goal-directed, temporally extended behavior (Ribas-Fernandes et al., 2011; Daw et al., 2011), meta-learning (Wang et al., 2018), and abstract problem solving requiring hierarchical thinking (Eckstein and Collins, 2020; Botvinick, 2012; Collins and Koechlin, 2012; Werchan et al., 2016). Underlining their centrality in the study of human cognition, RL models have been applied across the lifespan (van den Bos et al., 2018; Bolenz et al., 2017; Nussenbaum and Hartley, 2019), and in both healthy participants and those experiencing psychiatric illness (Huys et al., 2016; Adams et al., 2016; Hauser et al., 2019; Ahn and Busemeyer, 2016; Deserno et al., 2013). RL models are of particular interest because they also promise a close link to brain function: A specialized network of brain regions, including the basal ganglia and prefrontal cortex, implement computations that mirror specific components of RL algorithms, including action values and reward prediction errors (Frank and Claus, 2006; Niv, 2009; Lee et al., 2012; O’Doherty et al., 2015; Glimcher, 2011; Garrison et al., 2013; Dayan and Niv, 2008). In sum, RL, explaining behavior ranging from simple conditioning to complex problem solving, appropriate for diverse human (and nonhuman) populations, based on a compelling theoretical foundation (Sutton and Barto, 2017), and with strong ties to brain function, has seen a surge in published studies since its introduction (Palminteri et al., 2017), and emerged as a powerful and potentially unifying modeling framework for cognitive and neural processing.

Computational modeling enables researchers to condense rich behavioral datasets into simple, falsifiable models (e.g. RL) and fitted model parameters (e.g. learning rate, decision temperature) (van den Bos et al., 2018; Palminteri et al., 2017; Daw, 2011; Wilson and Collins, 2019; Guest and Martin, 2021; Blohm et al., 2020). These models and parameters are often interpreted as a reflection of (or ‘window into’) cognitive and/or neural processes, with the ability to dissect these processes into specific, unique components, and to measure participants’ inherent characteristics along these components. For example, RL models have been praised for their ability to separate the decision making process into value updating and choice selection stages, allowing for the separate investigation of each dimension. Hereby, RL models infer person-specific parameters for each dimension (e.g. learning rate and decision noise), seemingly providing a direct measure of individuals’ inherent characteristics. Crucially, many current research practices are firmly based on these (often implicit) assumptions, which give rise to the expectation that parameters have a task- and model-independent interpretation and will seamlessly generalize between studies. However, there is growing—though indirect—evidence that these assumptions might not (or not always) be valid. The following section lays out existing evidence in favor and in opposition of model generalizability and interpretability. Building on our previous opinion piece, which—based on a review of published studies—argued that there is less evidence for model generalizability and interpretability than expected based on current research practices (Eckstein et al., 2021), this study seeks to directly address the matter empirically.

Many current research practices are implicitly based on the interpretability and generalizability of computational model parameters (despite the fact that many researchers explicitly distance themselves from them). For our purposes, we define a model variable (e.g. fitted parameter) as generalizable if it is consistent across uses, such that a person would be characterized with the same values independent of the specific model or task used to estimate the variable. Generalizability is a consequence of the assumption that parameters are intrinsic to participants rather than task dependent (e.g. a high learning rate is a personal characteristic that might reflect an individual’s unique brain structure). One example of our implicit assumptions about generalizability is the fact that we often directly compare model parameters between studies—for example, comparing our findings related to learning rate parameters to a previous study’s findings related to learning rate parameters. Note that such a comparison is only valid if parameters capture the same underlying constructs across studies, tasks, and model variations, that is, if parameters generalize. The literature has implicitly equated parameters in this way in review articles (Huys et al., 2016; Adams et al., 2016; Hauser et al., 2019; Frank and Claus, 2006; Niv, 2009; Lee et al., 2012; O’Doherty et al., 2015; Glimcher, 2011; Dayan and Niv, 2008), meta-analyses (Garrison et al., 2013; Yaple and Yu, 2019; Liu et al., 2016), and also most empirical papers, by relating parameter-specific findings across studies. We also implicitly evoke parameter generalizability when we study task-independent empirical parameter priors (Gershman, 2016), or task-independent parameter relationships (e.g. interplay between different kinds of learning rates [Harada, 2020]), because we presuppose that parameter settings are inherent to participants, rather than task specific.

We define a model variable as interpretable if it isolates specific and unique cognitive elements, and/or is implemented in separable and unique neural substrates. Interpretability follows from the assumption that the decomposition of behavior into model parameters ‘carves cognition at its joints’, and provides fundamental, meaningful, and factual components (e.g. separating value updating from decision making). We implicitly invoke interpretability when we tie model variables to neural substrates in a task-general way (e.g. reward prediction errors to dopamine function [Schultz and Dickinson, 2000]), or when we use parameters as markers of psychiatric conditions in a model-independent way (e.g. working-memory deficits in schizophrenia [Collins et al., 2014]). Interpretability is also required when we relate abstract parameters to aspects of real-world decision making (Heinz et al., 2017), and generally, when we assume that model variables are particularly ‘theoretically meaningful’ (Huys et al., 2016).

However, in the midst of the growing application of computational modeling of behavior, the focus has also shifted toward inconsistencies and apparent contradictions in the emerging literature, which are becoming apparent in cognitive (Nassar and Frank, 2016), developmental (Nussenbaum and Hartley, 2019; Javadi et al., 2014; Blakemore and Robbins, 2012; DePasque and Galván, 2017), clinical (Adams et al., 2016; Hauser et al., 2019; Ahn and Busemeyer, 2016; Deserno et al., 2013), and neuroscience studies (Garrison et al., 2013; Yaple and Yu, 2019; Liu et al., 2016; Mohebi et al., 2019), and have recently become the focus of targeted investigations (Robinson and Chase, 2017; Weidinger et al., 2019; Brown et al., 2020; Pratt et al., 2021). For example, some developmental studies have shown that learning rates increased with age (Master et al., 2020; Davidow et al., 2016), whereas others have shown that they decrease (Decker et al., 2015). Yet others have reported U-shaped trajectories with either peaks (Rosenbaum et al., 2020) or troughs (Eckstein et al., 2022) during adolescence, or stability within this age range (Palminteri et al., 2016) (for a comprehensive review, see Nussenbaum and Hartley, 2019; for specific examples, see Nassar and Frank, 2016). This is just one striking example of inconsistencies in the cognitive modeling literature, and many more exist (Eckstein et al., 2022). These inconsistencies could signify that computational modeling is fundamentally flawed or inappropriate to answer our research questions. Alternatively, inconsistencies could signify that the method is valid, but our current implementations are inappropriate (Palminteri et al., 2017; Uttal, 1990; Webb, 2001; Navarro, 2019; Yarkoni, 2020; Wilson and Collins, 2019). However, we hypothesize that inconsistencies can also arise for a third reason: Even if both method and implementation are appropriate, inconsistencies like the ones above are expected—and not a sign of failure—if implicit assumptions of generalizability and interpretability are not always valid. For example, model parameters might be more context-dependent and less person-specific than we often appreciate (Nussenbaum and Hartley, 2019; Nassar and Frank, 2016; Yaple and Yu, 2019; Behrens et al., 2007; McGuire et al., 2014).

To illustrate this point, the current project began as an investigation into the development of learning in adolescence, with the aim of combining the insights of three different learning tasks to gain a more complete understanding of the underlying mechanisms. However, even though each task individually showed strong and interesting developmental patterns in terms of model parameters (Master et al., 2020; Eckstein et al., 2022; Xia et al., 2021), these patterns were very different—and even contradictory—across tasks. This implied that specific model parameters (e.g. learning rate) did not necessarily isolate specific cognitive processes (e.g. value updating) and consistently measure individuals on these processes, but that they captured different processes depending on the learning context of the task (lack of generalizability). In addition, the processes identified by one parameter were not necessarily distinct from the cognitive processes (e.g. decision making) identified by other parameters (e.g. decision temperature), but could overlap between parameters (lack of interpretability). In a nutshell, the ‘same’ parameters seemed to measure something different in each task.

The goal of the current project was to assess these patterns formally: We determined the degree to which parameters generalized between three different RL tasks, investigated whether parameters were interpretable as unique and specific processes, and provide initial evidence for context factors that potentially modulate generalizability and interpretability of model parameters, including feedback stochasticity, task volatility, and memory demands. To this aim, we compared the same individuals’ RL parameters, fit to different learning tasks in a single study, in a developmental dataset (291 participants, ages 8–30 years). Using a developmental dataset had several advantages: It provided large between-participant variance and hence better coverage of the parameter space, and allowed us to specifically target outstanding discrepancies in the developmental psychology literature (Nussenbaum and Hartley, 2019). The three learning tasks we used varied on several common dimensions, including feedback stochasticity, task volatility, and memory demands (Figure 1E), and have been used previously to study RL processes (Davidow et al., 2016; Collins and Frank, 2012; Javadi et al., 2014; Master et al., 2020; Eckstein et al., 2022; Xia et al., 2021). However, like many tasks in the literature, the tasks likely also engaged other cognitive processes besides RL, such as working memory and reasoning. The within-participant design of our study allowed us to test directly whether the same participants showed the same parameters across tasks (generalizability), and the combination of multiple tasks shed light on which cognitive processes the same parameters captured in each task (interpretability). We extensively compared and validated all RL models (Palminteri et al., 2017; Wilson and Collins, 2019; Lee, 2011) and have reported each task’s unique developmental results separately (Master et al., 2020; Eckstein et al., 2022; Xia et al., 2021).

Figure 1

Download asset Open asset

Our results show a striking lack of generalizability and interpretability for some tasks and parameters, but convincing generalizability for others. This reveals an urgent need for future research to address the role of context factors in computational modeling, and reveals the necessity of taking context factors into account when interpreting and generalizing results. It also suggests that some prior discrepancies are likely explained by differences in context.

This section gives a brief overview of the experimental tasks (Figure 1B–D) and computational models (Figure 1F; also see sections 'Task Design', 'Computational Models', and 'Appendix 2'; for details, refer to original publications [Master et al., 2020; Eckstein et al., 2022; Xia et al., 2021]). We then show our main findings on parameter generalizability (section 'Part I: parameter generalizability') and interpretability (section 'Part II: parameter interpretability'). All three tasks are learning tasks and have been previously well-captured by RL models, yet with differences in parameterization (Javadi et al., 2014; Davidow et al., 2016; Collins and Frank, 2012). In our study as well, the best-fitting RL models differed between tasks, containing some parameters that were the same across tasks, and some that were task-specific (Figure 1F). Thus, our setup provides a realistic reflection of the diversity of computational models in the literature.

Task A required participants to learn the correct associations between each of four stimuli (butterflies) and two responses (flowers) based on probabilistic feedback (Figure 1B). The best-fitting model contained three free parameters: learning rate from positive outcomes ${\alpha }_{+}$, inverse decision temperature $\beta$, and forgetting $F$. It also contained one fixed parameter: learning rate from negative outcomes ${\alpha }_{-}=0$ (Xia et al., 2021). Task B required participants to adapt to unexpected switches in the action-outcome contingencies of a simple bandit task (only one of two boxes contained a gold coin at any time) based on semi-probabilistic feedback (Figure 1C). The best-fitting RL model contained four free parameters: ${\alpha }_{+}$, ${\alpha }_{-}$, $\beta$, and choice persistence $p$ (Eckstein et al., 2022). Task C required learning of stimulus-response associations like task A, but over several task blocks with varying numbers of stimuli, and using deterministic feedback (Figure 1D). The best model for this task combined RL and working memory mechanisms, containing RL parameters ${\alpha }_{+}$ and ${\alpha }_{-}$; working memory parameters capacity $K$, forgetting $F$, and noise $ϵ$; and mixture parameter $\rho$, which determined the relative weights of RL and working memory (Master et al., 2020; Collins and Frank, 2012). The Markov decision process (MDP) framework provides a common language to describe learning tasks like ours, by breaking them down into states, actions, and reward functions. Appendix 2 summarizes the tasks in this way and highlights major differences.

We employed rigorous model fitting, comparison, and validation to obtain the best-fitting models presented here (see Appendix 4 and Palminteri et al., 2017; Daw, 2011; Wilson and Collins, 2019; Lee, 2011): For each task, we compared a large number of competing models, based on different parameterizations and cognitive mechanisms, and selected the best one based on quantitative model comparison scores as well as the models’ abilities to reproduce participants’ behavior in simulation (Appendix 4—figure 1). We also used hierarchical Bayesian methods for model fitting and comparison where possible, to obtain the most accurate parameter estimates (Lee, 2011; Brown et al., 2020). Individual publications provide further details on the set of models compared and validate the claim that the models presented here are the best-fitting ones for each task (Master et al., 2020; Eckstein et al., 2022; Xia et al., 2021), an important premise for the claim that individual parameters are well estimated. This qualitative validation step for each dataset ensures that potential parameter discrepancies between tasks are not due to a lack of modeling quality, and can indeed provide accurate information about parameter generalizability and interpretability. (Though we acknowledge that no model is ever right.)

Part I: parameter generalizability

Crucially, the parameter inconsistencies observed in previous literature could be caused by non-specific differences between studies (e.g. participant samples, testing procedures, modeling approaches, research labs). Our within-participant design allows us to rule these out by testing whether the same participants show different parameter values when assessed using different tasks; this finding would be strong evidence for the hypothesized lack of parameter generalizability. To assess this, we first determined whether participants showed similar parameter values across tasks, and then whether tasks showed similar parameter age trajectories.

Differences in absolute parameter values

We first used repeated-measures analyses of variance (ANOVAs) to test for task effects on absolute parameter values (Figure 2A). When ANOVAs showed significant task effects, we followed up with repeated-measures t-tests to compare each pair of tasks, using the Bonferroni correction.

Figure 2

Download asset Open asset

Learning rates ${\alpha }_{+}$ and ${\alpha }_{-}$ were so dissimilar across tasks that they occupied largely separate ranges: They were very low in task C (${\alpha }_{+}$ mean: 0.07, sd: 0.18; ${\alpha }_{-}$ mean: 0.03, sd: 0.13), intermediate in task A (${\alpha }_{+}$ mean: 0.22, sd: 0.09; ${\alpha }_{-}$ was fixed at 0), but fairly high in task B (${\alpha }_{+}$ mean: 0.77, sd: 0.11; ${\alpha }_{-}$ mean: 0.62, sd: 0.14; for statistical comparison, see Table 1). Decision noise was high in task B ($\frac{1}{\beta }$ mean: 0.33, sd: 0.15), but low in tasks A ($\frac{1}{\beta }$ mean: 0.095, sd: 0.0087) and C ($ϵ$ mean: 0.025, sd: 0.032; statistics in Table 1 ignore $ϵ$ because its absolute values were not comparable to $\frac{1}{\beta }$ due to the different parameterization; see section 'Computational models'). Forgetting was significantly higher in task C (mean: 0.19, sd: 0.17) than A (mean: 0.056, sd: 0.028). Task B was best fit without forgetting.

Table 1

Parameter	Model	Tasks	F / t	df	$p$	sig.
$\frac{1}{\beta }$	ANOVA	A, B	830	1	${\displaystyle p<0.001}$	***
	t-test	A vs B	25	246	${\displaystyle p<0.001}$	***
${\alpha }_{+}$	ANOVA	A, B, C	2,018	2	${\displaystyle p<0.001}$	***
	t-test	A vs B	66	246	${\displaystyle p<0.001}$	***
	t-test	A vs C	12	246	${\displaystyle p<0.001}$	***
	t-test	B vs C	51	246	${\displaystyle p<0.001}$	***
${\alpha }_{-}$	ANOVA	B, C	2,357	1	${\displaystyle p<0.001}$	***
	t-test	B vs C	49	246	${\displaystyle p<0.001}$	***
Forgetting	ANOVA	A, C	161	1	${\displaystyle p<0.001}$	***
	t-test	A vs C	49	246	${\displaystyle p<0.001}$	***

For all parameters, absolute parameter values hence differed substantially between tasks. This shows that the three tasks produced significantly different estimates of learning rates, decision noise/exploration, and forgetting for the same participants (Figure 2B). Interestingly, these parameter differences echoed specific task demands: Learning rates and noise/exploration were highest in task B, where frequent switches required quick updating and high levels of exploration. Similarly, forgetting was highest in task C, which posed the largest memory demands. Using regression models that controlled for age (instead of ANOVA) led to similar results (Table Appendix 8—table 2).

Relative parameter differences

However, comparing parameters in terms of their absolute values has shortcomings because it minimizes the role of relative variance between participants, which reflects participants’ mutual relationships to each other, and might be an important component of parameters. To test whether parameters generalized in relative, rather than absolute terms, we first correlated corresponding parameters between each pair of tasks, using Spearman correlation (Appendix 8—figure 1). Indeed, both ${\alpha }_{+}$ (Appendix 8—figure 1A) and noise/exploration parameters (Appendix 8—figure 1B) were significantly positively correlated between tasks A and B as well as between tasks A and C. Significant correlations were lacking between tasks B and C. This suggests that both ${\alpha }_{+}$ and noise/exploration generalized in terms of the relationships they captured between participants; however, this generalization was only evident between tasks A and B or A and C, potentially due to the fact that task A was more similar to tasks B and C than these were to each other (Figure 1E; also see section 'Main axes of variation'). Fig. Appendix 8—figure 3 shows the correlations between all pairs of features in the dataset (model parameters and behavioral measures). Note that noise parameters generalized between tasks A and C despite differences in parameterization ($ϵ$ vs. $1/\beta$), showing robustness in the characterization of choice stochasticity (Appendix 8—figure 1B).

Parameter age trajectories

This correlation analysis, however, is limited in its failure to account for age, an evident source of variance in our dataset. This means that apparent parameter generalization could be driven by a common dependence on age, rather than underlying age-independent similarities. To address this, we next focused on parameter age trajectories, aiming to remove differences between tasks that are potentially arbitrary (e.g. absolute mean and variance), while conserving patterns that are potentially more meaningful (e.g. shape of variance, i.e. participants’ values relative to each other). Age trajectories were calculated by z-scoring each parameter within each task (Figure 2C). To test for differences, mixed-effects regression was used to predict parameters of all tasks from two age predictors (age and squared age) and task (A, B, or C). A better fit of this model compared to the corresponding one without task indicates that task characteristics affected age trajectories. In this case, we followed up with post-hoc models comparing individual pairs of tasks.

For ${\alpha }_{-}$, the task-based regression model showed a significantly better fit, revealing an effect of task on ${\alpha }_{-}$’s age trajectory (Table 2). Indeed, ${\alpha }_{-}$ showed fundamentally different trajectories in task B compared to C (in task A, ${\alpha }_{-}$ was fixed): In task B, ${\alpha }_{-}$ decreased linearly, modulated by a U-shaped curvature (linear effect of age: $\beta =-0.11$, ${\displaystyle p<0.001}$; quadratic: $\beta =0.003$, ${\displaystyle p<0.001}$), but in task C, it increased linearly, modulated by an inverse-U curvature (linear: $\beta =0.32$, ${\displaystyle p<0.001}$; quadratic: $\beta =-0.07$, ${\displaystyle p<0.001}$; Figure 2C). The fact that these patterns are opposites of each other was reflected in the significant interaction terms of the overall regression model (Table 3). Indeed, we previously reported a U-shaped trajectory of ${\alpha }_{-}$ in task B, showing a minimum around age 13–15 (Eckstein et al., 2022), but a consistent increase up to early adulthood in task C (Xia et al., 2021). This shows striking differences when estimating ${\alpha }_{-}$ using task B compared to C. These differences might reflect differences in task demands: Negative feedback was diagnostic in task C, requiring large learning rates from negative feedback ${\alpha }_{-}$ for optimal performance, whereas negative feedback was not diagnostic in task B, requiring small ${\alpha }_{-}$ for optimal performance.

Table 2

Parameter	AIC without task	AIC with task	F(df)	p	sig.
$\frac{1}{\beta }/ϵ$	2,044	2,054	NA	NA	–
${\alpha }_{+}$	2,044	2,042	$F(4,245)=2.34$	$p=0.056$	–
${\alpha }_{-}$	1,395	1,373	$F(2,245)=6.99$	$p=0.0011$	**
Forgetting	1,406	1,411	NA	NA	–

Table 3

Parameter	Tasks	Predictor	${\displaystyle \beta }$	$p$(Bonf.)	sig.
$\frac{1}{\beta }/ϵ$	A, B, C	Intercept	1.86	${\displaystyle <0.001}$	***
		Age (linear)	–0.17	0.003	**
		Age (quadratic)	0.004	${\displaystyle <0.001}$	***
${\alpha }_{+}$	A, B, C	Intercept	–2.10	${\displaystyle <0.001}$	***
		Age (linear)	0.20	${\displaystyle <0.001}$	***
		Age (quadratic)	–0.004	${\displaystyle <0.001}$	***
${\alpha }_{-}$	B, C	Task (main effect)	4.15	${\displaystyle <0.001}$	***
		Task * linear age (interaction)	0.43	${\displaystyle <0.001}$	***
		Task * quadratic age (interaction)	–0.010	${\displaystyle <0.001}$	***
Forgetting	A, C	Intercept	0.37	0.44
		Age (linear)	–0.034	0.53
		Age (quadratic)	0.001	0.63

For ${\alpha }_{+}$, adding task as a predictor did not improve model fit, suggesting that ${\alpha }_{+}$ showed similar age trajectories across tasks (Table 2). Indeed, ${\alpha }_{+}$ showed a linear increase that tapered off with age in all tasks (linear increase: task A: $\beta =0.33$, ${\displaystyle p<0.001}$; task B: $\beta =0.052$, ${\displaystyle p<0.001}$; task C: $\beta =0.28$, ${\displaystyle p<0.001}$; quadratic modulation: task A: $\beta =-0.007$, ${\displaystyle p<0.001}$; task B: $\beta =-0.001$, ${\displaystyle p<0.001}$; task C: $\beta =-0.006$, ${\displaystyle p<0.001}$). For noise/exploration and forgetting parameters, adding task as a predictor also did not improve model fit (Table 2), suggesting similar age trajectories across tasks. For decision noise/exploration, the grand model revealed a linear decrease and tapering off with age (Figure 2C; Table 3), in accordance with previous findings (Nussenbaum and Hartley, 2019). For forgetting, the grand model did not reveal any age effects (Figure 2C; Table 3), suggesting inconsistent or lacking developmental changes.

In summary, ${\alpha }_{-}$ showed different age trajectories depending on the task. This suggests a lack of generalizability: The estimated developmental trajectories of learning rates for negative outcomes might not generalize between experimental paradigms. However, the age trajectories of noise/exploration parameters, ${\alpha }_{+}$, and forgetting did not differ between tasks. This lack of statistically-significant task differences might indicate parameter generalizability—but it could also reflect high levels of parameter estimation noise. Subsequent sections will disentangle these two possibilities.

Predicting age trajectories

The previous analysis, focusing on parameter differences, revealed some lack of generalization (e.g. ${\alpha }_{-}$). The next analysis takes the inverse approach, assessing similarities in an effort to provide evidence for generalization: We used linear regression to predict participants’ parameters in one task from the corresponding parameter on another task, controlling for age and squared age.

For both ${\alpha }_{+}$ and noise/exploration parameters, task A predicted tasks B and C, and tasks B and C predicted task A, but tasks B and C did not predict each other (Table 4; Figure 2D), reminiscent of the correlation results (section 'Relative parameter differences'). For ${\alpha }_{-}$, tasks B and C showed a marginally significant negative relationship (Table 4), suggesting that predicting ${\alpha }_{-}$ between tasks can lead to systematically biased predictions, confirming the striking differences observed before (section 'Parameter age trajectories'). For forgetting, tasks A and C were not predictive of each other (Table 4), suggesting that the lack of significant differences we observed previously (Table 3) did not necessarily imply successful generalization, but might have been caused by other factors, for example, elevated noise.

Table 4

Parameter	Tasks	$\beta$	p	sig.
$\frac{1}{\beta }$, ${\displaystyle ϵ}$	A & B	0.28		***
	A & C	0.19	0.0022	**
	B & C	0.039	0.54
${\alpha }_{+}$	A & B	0.13	0.035	*
	A & C	0.23		***
	B & C	–0.073	0.25
${\alpha }_{-}$	B & C	–0.12	0.058	$
Forgetting	A & C	0.097	0.13

Statistical comparison to generalizability ceiling

Our analyses so far suggest that some parameters did not generalize between tasks: We observed differences in age trajectories (section 'Parameter age trajectories') and a lack of mutual prediction (section 'Predicting age trajectories'). However, the lack of correspondence could also arise due to other factors, including behavioral noise, noise in parameter fitting, and parameter trade-offs within tasks. To rule these out, we next established the ceiling of generalizability attainable using our method.

We established the ceiling in the following way: We first created a dataset with perfect generalizability, simulating behavior from agents that use the same parameters across all tasks (Appendix 5—figure 1). We then fitted this dataset in the same way as the human dataset (e.g. using the same models), and performed the same analyses on the fitted parameters, including an assessment of age trajectories (Appendix 5—table 1) and prediction between tasks (Appendix 5—table 2, Appendix 5—table 3, and Appendix 5—table 4). These results provide the practical ceiling of generalizability, given the limitations of our data and modeling approach. We then compared the human results to this ceiling to ensure that the apparent lack of generalization was a valid conclusion, rather than stemming from methodological constraints: If the empirical human dataset is significantly below ceiling, we can conclude a lack of generalization, but if it is not significantly different from the expected ceiling, our approach might lack validity.

The results of this analysis support our conclusions. Specifically, whereas humans had shown divergent trajectories for parameter ${\alpha }_{-}$ (Figure 2B; Table 1), the simulated agents (that used the same parameters for all tasks) did not show task differences for ${\alpha }_{-}$ or any other parameter (Appendix 5—figure 1B, Appendix 5—table 1), even when controlling for age (Appendix 5—table 2, Appendix 5—table 3). Furthermore, the same parameters were predictive between tasks in all cases (Appendix 5—table 4). These results show that our method reliably detected parameter generalization in a dataset that exhibited generalization.

Lastly, we established whether the degree of generalization in humans was significantly different from agents. To this aim, we calculated the Spearman correlations between each pair of tasks for each parameter, for both humans (section 'Relative parameter differences'; Appendix 8—figure 1) and agents, and then compared humans and agents using bootstrapped confidence intervals (Appendix 5). Human parameter correlations were significantly below the ceiling for most parameters (exceptions: ${\alpha }_{+}$ in A vs B; $ϵ$ / $\frac{1}{\beta }$ in A vs C; Appendix 5—figure 1C). This suggests that the human sample showed less-than-perfect generalization for most task combinations and most parameters: Generalization was lower than in agents for parameters forgetting, ${\alpha }_{-}$, ${\alpha }_{+}$ (in two of three task combinations), and $ϵ$ / $\frac{1}{\beta }$ (in two of three task combinations).

Summary part I: Generalizability

So far, no parameter has shown generalization between tasks in terms of absolute values (Figure 2A and B), but noise/exploration and ${\alpha }_{+}$ showed similar age trajectories (Figure 2C), at least in tasks that were sufficiently similar (Figure 2D). To summarize, (1) all parameters differed significantly between tasks in terms of absolute values (Figure 2A and B). Intriguingly, absolute parameter values varied more between tasks than between participants within tasks, suggesting that task demands played a larger role in determining parameter values than participants’ individual characteristics. This was the case for all four model parameters (Noise/Exploration, ${\alpha }_{+}$, ${\alpha }_{-}$, and Forgetting). (2) However, there was evidence that in some cases, parameter age trajectories generalized between tasks: Task identity did not affect the age trajectories of noise/exploration, forgetting, or learning rate ${\alpha }_{+}$ (Figure 2C), suggesting possible generalization. However, only noise/exploration and ${\alpha }_{+}$ age trajectories were the same between tasks, hence revealing deeper similarities, and this was only possible when tasks were sufficiently similar (Table 4), highlighting the limits of generalization. No generalization was possible for ${\alpha }_{-}$, whose age trajectory differed both qualitatively and quantitatively between tasks, showing striking inverse patterns. Like for absolute parameter values, differences in parameter age trajectories were likely caused by differences in task demands. (3) Parameter ${\alpha }_{+}$ reached ceiling generalizability between tasks A and B, and parameter $ϵ$ / $\frac{1}{\beta }$ between tasks A and C. Generalizability of all other task combinations and parameters was significantly lower than expected from a perfectly-generalizing population. (4) For the parameters whose age trajectories showed signs of generalization, our results replicated patterns in the literature, with noise/exploration decreasing and ${\alpha }_{+}$ increasing from childhood to early adulthood (Nussenbaum and Hartley, 2019).

Part II: Parameter interpretability

To address the second assumption identified above, Part II focuses on parameter interpretability, testing whether parameters captured specific, unique, and meaningful cognitive processes. To this end, we first investigated the relations between different parameters to assess whether individual parameters were uniquely interpretable (i.e. specific and distinct from each other). We then determined how parameters were related to observed behavior, seeking evidence for external interpretability.

Main axes of variation

To build a foundation for parameter interpretation, we first aimed to understand which cognitive processes and aspects of participant behavior were captured by each parameter. We opted for a data-driven approach, interpreting parameters based on the major axes of variance that emerged in our large dataset, identified without a priori hypotheses. Concretely, we used PCA to identify the principal components (PCs) of our joint dataset of behavioral features and model parameters (Abdi and Williams, 2010). We first gained a thorough understanding of these PCs, and then employed them to better understand what was captured by model parameters. Detailed information on our approach is provided in sections 'Principal component analysis (PCA)' (PCA methods), 'Appendix 6' (behavioral features), and Appendix 8—figure 4 (additional PCA results).

We first analyzed PC1, the axis of largest variation and main source of individual differences in our dataset (25.1% of explained variance; Appendix 8—figure 4A). We found that behaviors that indicated good task participation (e.g. higher percentage of correct choices) loaded positively on PC1, whereas behaviors that indicated poor participation loaded negatively (e.g. more missed trials, longer response times; Figure 3A). This was the case for performance measures in the narrow sense of maximizing choice accuracy (e.g. percentage correct choices, trials to criterion, proportion of win-stay choices), but also in the wider sense of reflecting task engagement (e.g. number of missed trials, response times, response time variability). PC1 therefore captured a range of ‘good’, task-engaged behaviors, and is likely similar to the construct of ‘decision acuity’ (Moutoussis et al., 2021): Decision acuity was recently identified as the first component of a factor analysis (variant of PCA) conducted on 32 decision-making measures on 830 young people, and separated good and bad performance indices. Decision acuity reflected generic decision-making ability, predicted mental health factors, and was reflected in resting-state functional connectivity, but distinct from IQ (Moutoussis et al., 2021). Like decision acuity Moutoussis et al., 2021, our PC1 increased significantly with age, consistent with increasing performance (Appendix 3—figure 1B; age effects of subsequent PCs in Appendix 8—figure 4; Appendix 8—table 1).

Figure 3

Download asset Open asset

How can this understanding of PC1 (decision acuity) help us interpret model parameters? In all three tasks, noise/exploration and forgetting parameters loaded negatively on PC1 (Figure 3A), showing that elevated decision stochasticity and the decay of learned information were associated with poorer performance in all tasks. ${\alpha }_{+}$ showed positive loadings throughout, suggesting that faster integration of positive feedback was associated with better performance in all tasks. Taken together, noise/exploration, forgetting, and ${\alpha }_{+}$ showed consistency across tasks in terms of their interpretation with respect to decision acuity. Contrary to this, ${\alpha }_{-}$ loaded positively in task C, but negatively in task B, suggesting that performance increased when participants integrated negative feedback faster in task C, but performance decreased when they did the same in task B. As mentioned before, contradictory patterns of ${\alpha }_{-}$ were likely related to task demands: The fact that negative feedback was diagnostic in task C likely favored fast integration of negative feedback, while the fact that negative feedback was not diagnostic in task B likely favored slower integration (Figure 1E). This interpretation is supported by behavioral findings: ‘lose-stay’ behavior (repeating choices that produce negative feedback) showed the same contrasting pattern as ${\alpha }_{-}$ on PC1, loading positively in task B, which shows that lose-stay behavior benefited performance, but negatively on task C, which shows that it hurt performance (Figure 3A). This supports the claim that lower ${\alpha }_{-}$ was beneficial in task B, while higher ${\alpha }_{-}$ was beneficial in task C, in accordance with participant behavior and developmental differences.

We next analyzed PC2 and PC3. For easier visualization, we flipped the loadings of all features with negative loadings on PC1 to remove the effects of task engagement (PC1) when interpreting subsequent PCs (for details, see section 'Principal component analysis (PCA)'). This revealed that PC2 and PC3 encoded task contrasts: PC2 contrasted task B to task C (loadings were positive / negative / near-zero for corresponding features of tasks B / C / A; Figure 3B). PC3 contrasted task A to both B and C (loadings were positive / negative for corresponding features on task A / tasks B and C; Figure 3C). (As opposed to most features of our dataset, missed trials and response times did not show these task contrasts, suggesting that these features did not differentiate between tasks). The ordering of PC2 before PC3 shows that participants’ behavior differed more between task B compared to C (PC2: 8.9% explained variance) than between B and C compared to A (PC3: 6.2%; Appendix 8—figure 4), as expected based on task similarity (Figure 1E). PC2 and PC3 therefore show that, after task engagement, the main variation in our dataset arose from behavioral differences between tasks.

How can this understanding of PC2-3 promote our understanding of model parameters? The task contrasts encoded by the main behavioral measures were also evident in several parameters, including noise/exploration parameters, ${\alpha }_{+}$, and ${\alpha }_{-}$: These parameters showed positive loadings for task B in PC2 (A in PC3), and negative loadings for task C (B and C; PC2: Figure 3B, PC3: 3 C). This indicates that noise/exploration parameters, ${\alpha }_{+}$, and ${\alpha }_{-}$ captured different behavioral patterns depending on the task: The variance present in these parameters allowed for the discrimination of all tasks from each other, with PC2 discriminating task B from C, and PC3 discriminating tasks B and C from A. In other words, these parameters were clearly distinguishable between tasks, showing that they did not capture the same processes. Had they captured the same processes across tasks, they would not be differentiable between tasks, similar to, for example, response times. What is more, each parameter captured sufficient task-specific variance to indicate in which task it was measured. In sum, these findings contradict the assumption that parameters are specific or interpretable in a task-independent way.

Taken together, the PCA revealed that the emerging major axes of variation in our large dataset, together capturing 40.2% of explained variance, were task engagement (PC1) and task differences (PC2-PC3). These dimensions can be employed to better understand model parameters: Task engagement / decision acuity (PC1) played a crucial role for all four parameters (Figure 3A), and this role was consistent across tasks for noise/exploration, forgetting, and ${\alpha }_{+}$. This consistency supports the claim that parameters captured specific, task-independent processes in terms of PC1. For ${\alpha }_{-}$, however, PC1 played inverse roles across tasks, showing a lack of task-independent specificity that was likely due to differences in task demands. Furthermore, PC2 and PC3 revealed that noise/exploration, ${\alpha }_{+}$, and ${\alpha }_{-}$ specifically encoded task contrasts, suggesting that the parameters captured different cognitive processes across tasks, lacking a task-independent core of meaning.

Parameters and cognitive processes

Whereas the previous analysis revealed that parameter roles were not entirely consistent across tasks, it did not distinguish between parameter specificity (whether the same parameter captures the same cognitive processes across tasks) and distinctiveness (whether different parameters capture different cognitive processes).

To assess this, we probed how much parameter variance was explained by both corresponding and non-corresponding parameters across tasks: We predicted one parameter from all others to get a sense for which relationships were least and most explanatory, while accounting for all relationships of all parameters, using regression. We assumed that parameters reflected one or more cognitive processes, such that shared variance implies overlapping cognitive processes. If parameters are specific (i.e. reflect similar cognitive processes across tasks), then corresponding parameters should be predictive of each other (e.g. when predicting task B’s $\frac{1}{\beta }$ from task A’s parameters, task A’s $\frac{1}{\beta }$ should show a significant regression coefficient). If parameters are also distinct, then non-corresponding parameters should furthermore not be predictive (e.g. no other parameters beside task A’s $\frac{1}{\beta }$ should predict task B’s $\frac{1}{\beta }$). We used repeated, k-fold cross-validated Ridge regression to avoid overfitting, obtaining unbiased out-of-sample estimates of the means and variances of explained variance $R^2$ and regression coefficients $w$ (for methods, see section 'Ridge regression').

Assessing general patterns that arose in this analysis, we found that all significant coefficients connected tasks A and B or tasks A and C but never tasks B and C, mirroring previous results (Figure 2D; section 'Relative parameter differences') with regard to task similarity (Figure 1E). This suggests that no parameter had a specific core that extended across all three tasks—the largest shared variance encompassed two tasks.

We first address parameter specificity. Focusing on noise/exploration parameters, coefficients were significant when predicting noise/exploration in task A from noise/exploration in tasks B or C, but the inverse was not true, such that coefficients were not significant when predicting tasks B or C from task A (Figure 4A; Table 5). The first result implies parameter specificity, showing that noise/exploration parameters captured variance (cognitive processes) in task A that they also captured in tasks B and C. The second result, however, implies a lack of specificity, showing that noise/exploration parameters captured additional cognitive processes in tasks B and C that they did not capture in task A. A further lack of specificity was evident in that even the variance that both B and C captured in A was not the same: Prediction accuracy increased when combining tasks B and C to predict task A, showing that noise/exploration parameters in tasks B and C captured partly non-overlapping aspects of noise/exploration (Figure 4B, left-most set of bars, compare purple to orange and blue).

Figure 4

Download asset Open asset

Table 5

Predicted parameter (Task)	Predicting parameter (Task)	Coefficient	$p$	sig.
Noise/exploration (A)	Exploration $\frac{1}{\beta }$ (B)	0.14	0.031	*
	${\alpha }_{-}$ (B)	0.14	0.032	*
	Persistence (B)	–0.19	0.0029	**
	Noise ${\displaystyle ϵ}$ (C)	0.12	0.038	*
	${\alpha }_{-}$ (C)	–0.18	0.045	*
		–0.19	0.023	*
Noise/exploration (B)	Noise/exploration (A)	0.09	0.27	–
Noise/exploration (C)	Noise/exploration (A)	0.04	0.63	–
${\alpha }_{+}$ (A)	${\alpha }_{-}$ (C)	0.22	0.011	*
	${\displaystyle \rho }$ (C)	0.16	0.050	*
	$K$ (C)	0.15	0.020	*
	Exploration ${\displaystyle \frac{1}{\beta }}$ (B)	0.19	0.0026	**
	${\alpha }_{-}$ (B)	–0.21	${\displaystyle <0.001}$	***
	${\alpha }_{+}$ (B)	0.0042	0.94	–
	Persistence (B)	0.23	${\displaystyle <0.001}$	***
${\alpha }_{+}$(B)	$\frac{1}{\beta }$ (A)	–0.077	0.37	–
	${\alpha }_{+}$ (A)	0.058	0.48	–
	${\alpha }_{+}$ (C)	–0.00018	0.99	–
	${\alpha }_{-}$ (C)	–0.000055	1.00	–
	Forgetting (A)	0.015	0.82	–
${\alpha }_{+}$ (C)	${\alpha }_{+}$ (A)	0.20	0.013	*
${\alpha }_{-}$ (B)	${\alpha }_{+}$ (A)	–0.25	0.0018	**
${\alpha }_{-}$ (C)(C)	${\alpha }_{+}$ (A)	0.24	0.0022	**

Table 6

Predicted parameter (Task)	Predicting parameter (Task)	coefficient	$p$	sig.
Noise/exploration (A)	Win-stay (A)	–0.30	<0.001	***
	Lose-stay (A)	–0.23	<0.001	***
	Accuracy (A)	–0.19	0.0076	**
	Response times (A)	0.092	0.029	*
	Delay (A)	0.25	<0.001	***
Noise/exploration (B)	Win-stay (B)	–0.58	<0.001	***
	Lose-stay (B)	0.091	0.0034	**
	Accuracy (B)	–0.36	<0.001	***
	Win-stay (A)	–0.12	0.032	*
	Response times (A)	0.059	0.051	–
${\alpha }_{+}$ (A)	Win-stay (A)	0.74	<0.001	***
${\alpha }_{+}$ (B)	Win-stay (B)	0.27	<0.001	***
${\alpha }_{+}$ (C)	Accuracy (C)	0.24	0.033	*
${\alpha }_{-}$ (B)	Win-stay (B)	0.29	<0.001	***
	Lose-stay (B)	–0.71	<0.001	***
	Accuracy (B)	–0.28	<0.001	***
${\alpha }_{-}$ (C)	Win-stay (C)	0.16	0.009	**
	Lose-stay (C)	–0.41	<0.001	***

Focusing next on learning rates, specificity was evident in that learning rate ${\alpha }_{+}$ in task A showed a significant regression coefficient when predicting learning rates ${\alpha }_{+}$ and ${\alpha }_{-}$ in task C, and learning rate ${\alpha }_{-}$ in task C showed a significant coefficient when predicting learning rate ${\alpha }_{+}$ in task A (Figure 4A; Table 5). This suggests a shared core of cognitive processes between learning rates ${\alpha }_{+}$ and ${\alpha }_{-}$ in tasks A and C. However, a lack of specificity was evident in task B: When predicting ${\alpha }_{+}$ in task B, no parameter of any task showed a significant coefficient (including ${\alpha }_{+}$ in other tasks; Table 5), and it was impossible to predict variance in task B’s ${\alpha }_{+}$ even when combining all parameters of the other tasks (Figure 4B, ‘Task B’ panel). This reveals that ${\alpha }_{+}$ captured fundamentally different cognitive processes in task B compared to the other tasks. The case was similar for parameter ${\alpha }_{-}$, which strikingly was inversely related between tasks A and B (Table 5), and impossible to predict in task B from all other parameters (Figure 4B). This reveals a fundamental lack of specificity, implying that learning rates in task B did not capture the same core of cognitive processes compared to other tasks.

We next turned to distinctiveness, that is, whether different parameters capture different cognitive processes. Noise/exploration in task A was predicted by Persistence and ${\alpha }_{-}$ in task B, and by ${\alpha }_{-}$ and working memory weight $\rho$ in task C (Figure 4A; Table 5). This shows that processes that were captured by noise/exploration parameters in task A were captured by different parameters in other tasks, such that noise/exploration parameters did not capture distinct cognitive processes.

In the case of learning rates, ${\alpha }_{+}$ in task A was predicted nonspecifically by all parameters of task B (with the notable exception of ${\alpha }_{+}$ itself; Figure 4A; Table 5), suggesting that the cognitive processes that ${\alpha }_{+}$ captured in task A were captured by an interplay of several parameters in task B. Furthermore, task A’s ${\alpha }_{+}$ was predicted by task C’s working memory parameters $\rho$ and $K$ (Figure 4A; Table 5), suggesting that ${\alpha }_{+}$ captured a conglomerate of RL and working memory processes in task A that was isolated by different parameters in task C (Collins and Frank, 2012). In support of this interpretation, no variance in task C’s working memory parameters could be explained by any other parameters (Figure 4B), suggesting that they captured unique working memory processes that were not captured by other parameters. Task C’s RL parameters, on the other hand, could be explained by parameters in other tasks (Figure 4B), suggesting they captured overlapping RL processes. In tasks B and C, ${\alpha }_{+}$ and ${\alpha }_{-}$ were partly predicted by other learning rate parameters (specific and distinct), partly not predicted at all (lack of specificity), and partly predicted by several parameters (lack of distinctiveness; Figure 4A).

In sum, in the case of noise/exploration, there was evidence for both specificity and a lack thereof (mutual prediction between some, but not all noise/exploration parameters). Noise/exploration parameters were also not perfectly distinct, being predicted by a small set of other parameters from different tasks. In the case of learning rates, some specificity was evident in the shared variance between tasks A and C, but that specificity was missing in task B. Distinctiveness was particularly low for learning rates, with variance shared widely between multiple different parameters. When conducting the same analyses in simulated agents using the same parameters across tasks (section 'Statistical comparison to generalizability ceiling'), we obtained much higher specificity and distinctiveness.

Parameters and behavior

The previous sections suggested that parameters captured different cognitive processes across tasks (i.e. different internal characteristics of learning and choice). We lastly examined whether parameters also captured different behavioral features across tasks (e.g. tendency to stay after positive feedback), and whether behavioral features generalized better. To investigate this question, we assessed the relationships between model parameters and behavioral features across tasks, using regularized Ridge regression as before, and predicting each model parameter from each task’s behavioral features (15 predictors, see 'Appendix 1' and 'Appendix '6; for regression methods, see section 'Ridge regression').

We found that noise/exploration parameters were predicted by the same behavioral features in tasks A and B, such that task A’s accuracy, win-stay, and lose-stay behavior predicted task A’s $\frac{1}{\beta }$; and task B’s accuracy, win-stay, and lose-stay behavior predicted task B’s $\frac{1}{\beta }$ (Figure 4C; Table 6). This shows consistency in terms of which (task-specific) behaviors were related to (task-specific) parameter $\frac{1}{\beta }$. Similarly for learning rates, ${\alpha }_{+}$ was predicted by the same behavior (win-stay) in tasks A and B, and ${\alpha }_{-}$ was predicted by the same behaviors (lose-stay, win-stay) in tasks B and C (Figure 4C; Table 6). This consistency in ${\alpha }_{-}$ is especially noteworthy given the pronounced lack of consistency in the previous analyses.

In sum, noise/exploration parameters, ${\alpha }_{+}$, and ${\alpha }_{-}$ successfully generalized between tasks in terms of which behaviors they reflected (Figure 4C), despite the fact that many of the same parameters did not generalize in terms of how they characterized participants (sections 'Differences in absolute parameter values', 'Relative parameter differences', and 'Parameter age trajectories'), and which cognitive processes they captured (sections 'Main axes of variation and parameters and cognitive processes'). Notably, the behavioral and parameter differences we observed between tasks often seemed tuned to specific task characteristics (Figure 1E), both in the case of parameters (most notably ${\alpha }_{-}$; Figures 2C and 3A) and behavior (most notably lose-stay behavior; Appendix 3—figure 1B), suggesting that both behavioral responses and model parameters were shaped by task characteristics. This suggests a succinct explanation for why parameters did not generalize between tasks: Because different tasks elicited different behaviors (Appendix 3—figure 1B), and because each behavior was captured by the same parameter across tasks (Figure 4C), parameters necessarily differed between tasks.

Both generalizability (Nassar and Frank, 2016) and interpretability (i.e. the inherent ‘meaningfulness’ of parameters Huys et al., 2016) have been stated as advantages of computational modeling, and many current research practices (e.g. comparing parameter-specific findings between studies) endorse them (Eckstein et al., 2021). However, RL model generalizability and interpretability has so far eluded investigation, and growing inconsistencies in the literature potentially cast doubt on these assumptions. It is hence unclear whether, to what degree, and under which circumstances we should assume generalizability and interpretability. Our developmental, within-participant study revealed that these assumptions warrant both increased scepticism and continued investigation: Generalizability and interpretability were suprisingly low for most parameters and tasks, but reassuringly high for a few others:

Exploration/noise parameters showed considerable generalizability in the form of correlated variance and age trajectories. Furthermore, the decline in exploration/noise we observed between ages 8–17 was consistent with previous studies (Nussenbaum and Hartley, 2019; Somerville et al., 2017; Gopnik, 2020), revealing consistency across tasks, models, and research groups that supports the generalizability of exploration/noise parameters. Still, for 2/3 pairs of tasks, the degree of generalization was significantly below the level of generalization expected by agents with perfect generalization.

Interpretability of exploration/noise parameters was mixed: Despite evidence for specificity in some cases (overlap in parameter variance between tasks), it was missing in others (lack of overlap), and crucially, parameters lacked distinctiveness (substantial overlap in variance with other parameters). Thus, while exploration/noise parameters were generalizable across tasks, they were not neurocognitively “interpretable” (as defined above).

Learning rate from negative feedback showed a substantial lack of generalizability: parameters were less consistent within participants than within tasks, and age trajectories differed both quantitatively and qualitatively. Learning rates from positive feedback, however, showed some convincing patterns of generalization. These results are consistent with the previous literature, which shows mixed results for learning rate parameters (Nussenbaum and Hartley, 2019). In terms of interpretability, learning rates from positive and negative feedback combined were somewhat specific (overlap in variance between some tasks). However, a lack of specificity (lack of shared core variance) and distinctiveness (fundamental entangling with several other parameters, most notably working memory parameters) overshadowed this result.

Taken together, our study confirms the patterns of generalizable exploration/noise parameters and task-specific learning rate parameters that are emerging from the literature (Nussenbaum and Hartley, 2019). Furthermore, we show that this is not a result of between-participant comparisons, but that the same participants will show different parameters when tested using different tasks. The inconsistency of learning rate parameters leads to the important conclusion that we cannot measure an individual’s ‘intrinsic learning rate’, and that we should not draw general conclusions about ‘the development of learning rates’ with the implication that they apply to all contexts.

These findings help clarify the source of parameter inconsistencies in previous literature (besides replication problems and technical issues, such as model misspecification [Nussenbaum and Hartley, 2019], lack of model comparison and validation [Palminteri et al., 2017; Wilson and Collins, 2019], lack of model critique [Nassar and Frank, 2016], inappropriate fitting methods [Daw, 2011; Lee, 2011], and lack of parameter reliability [Brown et al., 2020]): Our results show that discrepancies are expected even with a consistent methodological pipeline and up-to-date modeling techniques, because they are an expected consequence of variations in context (e.g. features of the experimental task [section Parameters and behavior] and the computational model). The results also suggest that the mapping between cognitive processes and exhibited behavior is many-to-many, such that different cognitive mechanisms (e.g. reasoning, value learning, episodic memory) can give rise to the same behaviors (e.g. lose-stay behavior) and parameters (e.g. ${\alpha }_{-}$), while the same cognitive mechanism (e.g. value learning) can give rise to different behaviors (e.g. win-stay, lose-shift) and influence several parameters (e.g. ${\alpha }_{+}$, ${\alpha }_{-}$), depending on context factors. Under this view, analysis of model parameters alone does not permit unequivocal conclusions about cognitive processes if the context varies (Figure 5B), and the interpretation of model parameter results requires careful contextualization.

Figure 5

Download asset Open asset

Future research needs to investigate context factors to characterize these issues in more detail. For example, which task characteristics determine which parameters will generalize and which will not, and to what extent? Does context impact whether parameters capture overlapping versus distinct variance? Here, the systematic investigation of task space (i.e., testing the same participants on a large battery of learning tasks created as the full factorial of all task features) could elucidate the relationships between parameter generalizability and task-based context factors (e.g., stochasticity, volatility, reward probability). To determine the distance between tasks, the MDP framework might be especially useful because it decomposes tasks along theoretically meaningful features. Future research will also need to determine the relative contributions of different sources of inconsistency, differentiating those caused by technical issues from those caused by context differences.

In sum, our results suggest that relating model parameters to cognitive constructs and real-world behavior might require us to carefully account for task variables and environmental variability in general. This ties into a broader open question of how neurocognitive processes are shared between tasks (Eisenberg et al., 2019; Moutoussis et al., 2021), and reflects a larger pattern of thought in psychology that we cannot objectively assess an individual’s cognitive processing while ignoring context. We have shown that in lab studies, different task contexts recruit different system settings within an individual; similarly, our real-life surroundings, the way they change during development, and our past environments (Lin et al., 2020; Lin et al., 2022) may also modulate which cognitive processes we recruit.

The data collected for this project have been made available online on the OSF data servers.

1. 1. Eckstein M

   (2021) Open Science Framework

   ID jm2c8. The Unique Advantage of Adolescents in Probabilistic Reversal.

   https://osf.io/jm2c8/
2. 1. Xia L
   2. Collins A

   (2021) Open Science Framework

   ID wq4te. Modeling Changes in Probabilistic Reinforcement Learning during Adolescence.

   https://osf.io/wq4te/
3. 1. Eckstein M
   2. Master SL
   3. Zou AR
   4. Collins A

   (2021) Open Science Framework

   ID h4qr6. Data for "The interpretation of computational model parameters depends on the context".

   https://osf.io/h4qr6/

1. 1. Abdi H
   2. Williams LJ

   (2010) Principal component analysis

   Wiley Interdisciplinary Reviews: Computational Statistics 2:433–459.

   https://doi.org/10.1002/wics.101

   • Google Scholar
2. 1. Adams RA
   2. Huys QJM
   3. Roiser JP

   (2016) Computational psychiatry: towards a mathematically informed understanding of mental illness

   Journal of Neurology, Neurosurgery, and Psychiatry 87:53–63.

   https://doi.org/10.1136/jnnp-2015-310737

   • PubMed
   • Google Scholar
3. 1. Ahn WY
   2. Busemeyer JR

   (2016) Challenges and promises for translating computational tools into clinical practice

   Current Opinion in Behavioral Sciences 11:1–7.

   https://doi.org/10.1016/j.cobeha.2016.02.001

   • PubMed
   • Google Scholar
4. 1. Behrens TEJ
   2. Woolrich MW
   3. Walton ME
   4. Rushworth MFS

   (2007) Learning the value of information in an uncertain world

   Nature Neuroscience 10:1214–1221.

   https://doi.org/10.1038/nn1954

   • PubMed
   • Google Scholar
5. 1. Berridge KC

   (2007) The debate over dopamine ’ S role in reward: the case for incentive salience

   Psychopharmacology 191:391–431.

   https://doi.org/10.1007/s00213-006-0578-x

   • PubMed
   • Google Scholar
6. 1. Blakemore SJ
   2. Robbins TW

   (2012) Decision-Making in the adolescent brain

   Nature Neuroscience 15:1184–1191.

   https://doi.org/10.1038/nn.3177

   • PubMed
   • Google Scholar
7. 1. Blohm G
   2. Kording KP
   3. Schrater PR

   (2020) A how-to-model guide for neuroscience

   ENeuro 7:ENEURO.0352-19.2019.

   https://doi.org/10.1523/ENEURO.0352-19.2019

   • PubMed
   • Google Scholar
8. 1. Bolenz F
   2. Reiter AMF
   3. Eppinger B

   (2017) Developmental changes in learning: computational mechanisms and social influences

   Frontiers in Psychology 8:2048.

   https://doi.org/10.3389/fpsyg.2017.02048

   • PubMed
   • Google Scholar
9. 1. Bornstein AM
   2. Norman KA

   (2017) Reinstated episodic context guides sampling-based decisions for reward

   Nature Neuroscience 20:997–1003.

   https://doi.org/10.1038/nn.4573

   • PubMed
   • Google Scholar
10. 1. Botvinick MM

    (2012) Hierarchical reinforcement learning and decision making

    Current Opinion in Neurobiology 22:956–962.

    https://doi.org/10.1016/j.conb.2012.05.008

    • PubMed
    • Google Scholar
11. 1. Bouret S
    2. Sara SJ

    (2005) Network reset: a simplified overarching theory of locus coeruleus noradrenaline function

    Trends in Neurosciences 28:574–582.

    https://doi.org/10.1016/j.tins.2005.09.002

    • PubMed
    • Google Scholar
12. 1. Brown VM
    2. Chen J
    3. Gillan CM
    4. Price RB

    (2020) Improving the reliability of computational analyses: model-based planning and its relationship with compulsivity

    Biological Psychiatry. Cognitive Neuroscience and Neuroimaging 5:601–609.

    https://doi.org/10.1016/j.bpsc.2019.12.019

    • PubMed
    • Google Scholar
13. 1. Cazé RD
    2. van der Meer MAA

    (2013) Adaptive properties of differential learning rates for positive and negative outcomes

    Biol Cybern 107:711–719.

    https://doi.org/10.1007/s00422-013-0571-5

    • PubMed
    • Google Scholar
14. 1. Christakou A
    2. Gershman SJ
    3. Niv Y
    4. Simmons A
    5. Brammer M
    6. Rubia K

    (2013) Neural and psychological maturation of decision-making in adolescence and young adulthood

    Journal of Cognitive Neuroscience 25:1807–1823.

    https://doi.org/10.1162/jocn_a_00447

    • PubMed
    • Google Scholar
15. 1. Collins AGE
    2. Frank MJ

    (2012) How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis

    The European Journal of Neuroscience 35:1024–1035.

    https://doi.org/10.1111/j.1460-9568.2011.07980.x

    • PubMed
    • Google Scholar
16. 1. Collins AGE
    2. Koechlin E

    (2012) Reasoning, learning, and creativity: frontal lobe function and human decision-making

    PLOS Biology 10:e1001293.

    https://doi.org/10.1371/journal.pbio.1001293

    • PubMed
    • Google Scholar
17. 1. Collins AGE
    2. Brown JK
    3. Gold JM
    4. Waltz JA
    5. Frank MJ

    (2014) Working memory contributions to reinforcement learning impairments in schizophrenia

    The Journal of Neuroscience 34:13747–13756.

    https://doi.org/10.1523/JNEUROSCI.0989-14.2014

    • PubMed
    • Google Scholar
18. 1. Collins AGE
    2. Albrecht MA
    3. Waltz JA
    4. Gold JM
    5. Frank MJ

    (2017a) Interactions among working memory, reinforcement learning, and effort in value-based choice: a new paradigm and selective deficits in schizophrenia

    Biological Psychiatry 82:431–439.

    https://doi.org/10.1016/j.biopsych.2017.05.017

    • PubMed
    • Google Scholar
19. 1. Collins AGE
    2. Ciullo B
    3. Frank MJ
    4. Badre D

    (2017b) Working memory load strengthens reward prediction errors

    The Journal of Neuroscience 37:4332–4342.

    https://doi.org/10.1523/JNEUROSCI.2700-16.2017

    • PubMed
    • Google Scholar
20. 1. Collins AGE

    (2018) The tortoise and the hare: interactions between reinforcement learning and working memory

    Journal of Cognitive Neuroscience 30:1422–1432.

    https://doi.org/10.1162/jocn_a_01238

    • PubMed
    • Google Scholar
21. 1. Collins AGE
    2. Frank MJ

    (2018) Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory

    PNAS 115:2502–2507.

    https://doi.org/10.1073/pnas.1720963115

    • PubMed
    • Google Scholar
22. 1. Collins AGE

    (2019) Reinforcement learning: bringing together computation and cognition

    Current Opinion in Behavioral Sciences 29:63–68.

    https://doi.org/10.1016/j.cobeha.2019.04.011

    • Google Scholar
23. 1. Cools R
    2. Clark L
    3. Owen AM
    4. Robbins TW

    (2002)

    Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging

    The Journal of Neuroscience 22:4563–4567.

    • PubMed
    • Google Scholar
24. 1. Cools R
    2. Frank MJ
    3. Gibbs SE
    4. Miyakawa A
    5. Jagust W
    6. D’Esposito M

    (2009) Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to dopaminergic drug administration

    The Journal of Neuroscience 29:1538–1543.

    https://doi.org/10.1523/JNEUROSCI.4467-08.2009

    • PubMed
    • Google Scholar
25. 1. Dabney W
    2. Kurth-Nelson Z
    3. Uchida N
    4. Starkweather CK
    5. Hassabis D
    6. Munos R
    7. Botvinick M

    (2020) A distributional code for value in dopamine-based reinforcement learning

    Nature 577:671–675.

    https://doi.org/10.1038/s41586-019-1924-6

    • PubMed
    • Google Scholar
26. 1. Davidow JY
    2. Foerde K
    3. Galván A
    4. Shohamy D

    (2016) An upside to reward sensitivity: the hippocampus supports enhanced reinforcement learning in adolescence

    Neuron 92:93–99.

    https://doi.org/10.1016/j.neuron.2016.08.031

    • PubMed
    • Google Scholar
27. Book

    1. Daw ND

    (2011) Trial-by-trial data analysis using computational models

    In: Delgado MR, Phelps EA, Robbins TW, editors. Decision Making, Affect, and Learning: Attention and Performance XXIII. New York: Oxford University Press. pp. 3–38.

    https://doi.org/10.1093/acprof:oso/9780199600434.001.0001

    • Google Scholar
28. 1. Daw ND
    2. Gershman SJ
    3. Seymour B
    4. Dayan P
    5. Dolan RJ

    (2011) Model-Based influences on humans ’ choices and striatal prediction errors

    Neuron 69:1204–1215.

    https://doi.org/10.1016/j.neuron.2011.02.027

    • PubMed
    • Google Scholar
29. 1. Dayan P
    2. Niv Y

    (2008) Reinforcement learning: the good, the bad and the ugly

    Current Opinion in Neurobiology 18:185–196.

    https://doi.org/10.1016/j.conb.2008.08.003

    • PubMed
    • Google Scholar
30. 1. Decker JH
    2. Lourenco FS
    3. Doll BB
    4. Hartley CA

    (2015) Experiential reward learning outweighs instruction prior to adulthood

    Cognitive, Affective & Behavioral Neuroscience 15:310–320.

    https://doi.org/10.3758/s13415-014-0332-5

    • PubMed
    • Google Scholar
31. 1. DePasque S
    2. Galván A

    (2017) Frontostriatal development and probabilistic reinforcement learning during adolescence

    Neurobiology of Learning and Memory 143:1–7.

    https://doi.org/10.1016/j.nlm.2017.04.009

    • PubMed
    • Google Scholar
32. 1. Deserno L
    2. Boehme R
    3. Heinz A
    4. Schlagenhauf F

    (2013) Reinforcement learning and dopamine in schizophrenia: dimensions of symptoms or specific features of a disease group?

    Frontiers in Psychiatry 4:172.

    https://doi.org/10.3389/fpsyt.2013.00172

    • PubMed
    • Google Scholar
33. 1. Dickstein DP
    2. Finger EC
    3. Brotman MA
    4. Rich BA
    5. Pine DS
    6. Blair JR
    7. Leibenluft E

    (2010) Impaired probabilistic reversal learning in youths with mood and anxiety disorders

    Psychological Medicine 40:1089–1100.

    https://doi.org/10.1017/S0033291709991462

    • PubMed
    • Google Scholar
34. 1. Donoso M
    2. Collins AGE
    3. Koechlin E

    (2014) Human cognition. Foundations of human Reasoning in the prefrontal cortex

    Science 344:1481–1486.

    https://doi.org/10.1126/science.1252254

    • PubMed
    • Google Scholar
35. 1. Eckstein MK
    2. Collins AGE

    (2020) Computational evidence for hierarchically structured reinforcement learning in humans

    PNAS 117:29381–29389.

    https://doi.org/10.1073/pnas.1912330117

    • PubMed
    • Google Scholar
36. 1. Eckstein MK
    2. Wilbrecht L
    3. Collins AGE

    (2021) What do reinforcement learning models measure? interpreting model parameters in cognition and neuroscience

    Current Opinion in Behavioral Sciences 41:128–137.

    https://doi.org/10.1016/j.cobeha.2021.06.004

    • PubMed
    • Google Scholar
37. Software

    1. Eckstein MK

    (2022) About the SLCN project, version swh:1:rev:4fb5955c1142fcbd8ec80d7fccdf6b35dbfd1616

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:2208f3398c0661225348312d6b0915cfd2f59cda;origin=https://github.com/MariaEckstein/SLCN;visit=swh:1:snp:2e353f04cfbb36a541776a2cbcc0ea45eac969a5;anchor=swh:1:rev:4fb5955c1142fcbd8ec80d7fccdf6b35dbfd1616
38. 1. Eckstein MK
    2. Master SL
    3. Dahl RE
    4. Wilbrecht L
    5. Collins AGE

    (2022) Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal

    Developmental Cognitive Neuroscience 55:101106.

    https://doi.org/10.1016/j.dcn.2022.101106

    • PubMed
    • Google Scholar
39. 1. Eisenberg IW
    2. Bissett PG
    3. Zeynep Enkavi A
    4. Li J
    5. MacKinnon DP
    6. Marsch LA
    7. Poldrack RA

    (2019) Uncovering the structure of self-regulation through data-driven ontology discovery

    Nature Communications 10:1–13.

    https://doi.org/10.1038/s41467-019-10301-1

    • PubMed
    • Google Scholar
40. 1. Frank MJ
    2. Seeberger LC
    3. O’reilly RC

    (2004) By carrot or by stick: cognitive reinforcement learning in parkinsonism

    Science 306:1940–1943.

    https://doi.org/10.1126/science.1102941

    • PubMed
    • Google Scholar
41. 1. Frank MJ
    2. Claus ED

    (2006) Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal

    Psychological Review 113:300–326.

    https://doi.org/10.1037/0033-295X.113.2.300

    • PubMed
    • Google Scholar
42. 1. Garrison J
    2. Erdeniz B
    3. Done J

    (2013) Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies

    Neuroscience and Biobehavioral Reviews 37:1297–1310.

    https://doi.org/10.1016/j.neubiorev.2013.03.023

    • PubMed
    • Google Scholar
43. Book

    1. Gelman A
    2. Carlin JB
    3. Stern HS
    4. Dunson DB
    5. Vehtari A
    6. Rubin DB

    (2013)

    Bayesian Data Analysis

    Chapman and Hall/CRC, Boca Raton.

    • Google Scholar
44. 1. Gershman SJ

    (2016) Empirical priors for reinforcement learning models

    Journal of Mathematical Psychology 71:1–6.

    https://doi.org/10.1016/j.jmp.2016.01.006

    • Google Scholar
45. 1. Gershman SJ

    (2017) Dopamine, inference, and uncertainty

    Neural Computation 29:3311–3326.

    https://doi.org/10.1162/neco_a_01023

    • PubMed
    • Google Scholar
46. 1. Gershman SJ
    2. Uchida N

    (2019) Believing in dopamine

    Nature Reviews. Neuroscience 20:703–714.

    https://doi.org/10.1038/s41583-019-0220-7

    • PubMed
    • Google Scholar
47. 1. Gläscher J
    2. Hampton AN
    3. O’Doherty JP

    (2009) Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making

    Cerebral Cortex 19:483–495.

    https://doi.org/10.1093/cercor/bhn098

    • PubMed
    • Google Scholar
48. 1. Glimcher PW

    (2011) Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis

    PNAS 108 Suppl 3:15647–15654.

    https://doi.org/10.1073/pnas.1014269108

    • PubMed
    • Google Scholar
49. 1. Gopnik A

    (2020) Childhood as a solution to explore-exploit tensions

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 375:20190502.

    https://doi.org/10.1098/rstb.2019.0502

    • PubMed
    • Google Scholar
50. 1. Guest O
    2. Martin AE

    (2021) How computational modeling can force theory building in psychological science

    Perspectives on Psychological Science 16:789–802.

    https://doi.org/10.1177/1745691620970585

    • PubMed
    • Google Scholar
51. 1. Harada T

    (2020) Learning from success or failure?-positivity biases revisited

    Frontiers in Psychology 11:1627.

    https://doi.org/10.3389/fpsyg.2020.01627

    • PubMed
    • Google Scholar
52. Preprint

    1. Hare TA

    (2020) Humans Are Primarily Model-Based Learners in the Two-Stage Task

    bioRxiv.

    https://doi.org/10.1101/682922

    • Google Scholar
53. 1. Hauser TU
    2. Will GJ
    3. Dubois M
    4. Dolan RJ

    (2019) Annual research review: developmental computational psychiatry

    Journal of Child Psychology and Psychiatry, and Allied Disciplines 60:412–426.

    https://doi.org/10.1111/jcpp.12964

    • PubMed
    • Google Scholar
54. 1. Heinz A
    2. Deserno L
    3. Zimmermann US
    4. Smolka MN
    5. Beck A
    6. Schlagenhauf F

    (2017) Targeted intervention: computational approaches to elucidate and predict relapse in alcoholism

    NeuroImage 151:33–44.

    https://doi.org/10.1016/j.neuroimage.2016.07.055

    • PubMed
    • Google Scholar
55. 1. Huys QJM
    2. Maia TV
    3. Frank MJ

    (2016) Computational psychiatry as a bridge from neuroscience to clinical applications

    Nature Neuroscience 19:404–413.

    https://doi.org/10.1038/nn.4238

    • PubMed
    • Google Scholar
56. 1. Izquierdo A
    2. Brigman JL
    3. Radke AK
    4. Rudebeck PH
    5. Holmes A

    (2017) The neural basis of reversal learning: an updated perspective

    Neuroscience 345:12–26.

    https://doi.org/10.1016/j.neuroscience.2016.03.021

    • PubMed
    • Google Scholar
57. 1. Javadi AH
    2. Schmidt DHK
    3. Smolka MN

    (2014) Adolescents adapt more slowly than adults to varying reward contingencies

    Journal of Cognitive Neuroscience 26:2670–2681.

    https://doi.org/10.1162/jocn_a_00677

    • PubMed
    • Google Scholar
58. 1. Jepma M
    2. Murphy PR
    3. Nassar MR
    4. Rangel-Gomez M
    5. Meeter M
    6. Nieuwenhuis S

    (2016) Catecholaminergic regulation of learning rate in a dynamic environment

    PLOS Computational Biology 12:e1005171.

    https://doi.org/10.1371/journal.pcbi.1005171

    • PubMed
    • Google Scholar
59. 1. Johnson C
    2. Wilbrecht L

    (2011) Juvenile mice show greater flexibility in multiple choice reversal learning than adults

    Developmental Cognitive Neuroscience 1:540–551.

    https://doi.org/10.1016/j.dcn.2011.05.008

    • PubMed
    • Google Scholar
60. 1. Katahira K

    (2016) How hierarchical models improve point estimates of model parameters at the individual level

    Journal of Mathematical Psychology 73:37–58.

    https://doi.org/10.1016/j.jmp.2016.03.007

    • Google Scholar
61. 1. Katahira K

    (2018) The statistical structures of reinforcement learning with asymmetric value updates

    Journal of Mathematical Psychology 87:31–45.

    https://doi.org/10.1016/j.jmp.2018.09.002

    • Google Scholar
62. 1. Konovalov A
    2. Krajbich I

    (2018) Neurocomputational dynamics of sequence learning

    Neuron 98:1282–1293.

    https://doi.org/10.1016/j.neuron.2018.05.013

    • PubMed
    • Google Scholar
63. 1. Kool W
    2. Cushman FA
    3. Gershman SJ

    (2016) When does model-based control pay off?

    PLOS Computational Biology 12:e1005090.

    https://doi.org/10.1371/journal.pcbi.1005090

    • PubMed
    • Google Scholar
64. 1. Lake BM
    2. Ullman TD
    3. Tenenbaum JB
    4. Gershman SJ

    (2017) Building machines that learn and think like people

    The Behavioral and Brain Sciences 40:e253.

    https://doi.org/10.1017/S0140525X16001837

    • PubMed
    • Google Scholar
65. 1. Lee MD

    (2011) How cognitive modeling can benefit from hierarchical Bayesian models

    Journal of Mathematical Psychology 55:1–7.

    https://doi.org/10.1016/j.jmp.2010.08.013

    • Google Scholar
66. 1. Lee D
    2. Seo H
    3. Jung MW

    (2012) Neural basis of reinforcement learning and decision making

    Annual Review of Neuroscience 35:287–308.

    https://doi.org/10.1146/annurev-neuro-062111-150512

    • PubMed
    • Google Scholar
67. 1. Lefebvre G
    2. Lebreton M
    3. Meyniel F
    4. Bourgeois-Gironde S
    5. Palminteri S

    (2017) Behavioural and neural characterization of optimistic reinforcement learning

    Nature Human Behaviour 1:0067.

    https://doi.org/10.1038/s41562-017-0067

    • Google Scholar
68. 1. Lin WC
    2. Delevich K
    3. Wilbrecht L

    (2020) A role for adaptive developmental plasticity in learning and decision making

    Current Opinion in Behavioral Sciences 36:48–54.

    https://doi.org/10.1016/j.cobeha.2020.07.010

    • PubMed
    • Google Scholar
69. 1. Lin WC
    2. Liu C
    3. Kosillo P
    4. Tai LH
    5. Galarce E
    6. Bateup HS
    7. Lammel S
    8. Wilbrecht L

    (2022) Transient food insecurity during the juvenile-adolescent period affects adult weight, cognitive flexibility, and dopamine neurobiology

    Current Biology 32:3690–3703.

    https://doi.org/10.1016/j.cub.2022.06.089

    • PubMed
    • Google Scholar
70. 1. Liu X
    2. Hairston J
    3. Schrier M
    4. Fan J

    (2016) Common and distinct networks underlying reward valence and processing stages: a meta-analysis of functional neuroimaging studies

    Neuroscience & Biobehavioral Reviews 35:1219–1236.

    https://doi.org/10.1016/j.neubiorev.2010.12.012

    • Google Scholar
71. 1. Lourenco F
    2. Casey BJ

    (2013) Adjusting behavior to changing environmental demands with development

    Neuroscience and Biobehavioral Reviews 37:2233–2242.

    https://doi.org/10.1016/j.neubiorev.2013.03.003

    • PubMed
    • Google Scholar
72. 1. Master SL
    2. Eckstein MK
    3. Gotlieb N
    4. Dahl R
    5. Wilbrecht L
    6. Collins AGE

    (2020) Distentangling the systems contributing to changes in learning during adolescence

    Developmental Cognitive Neuroscience 41:100732.

    https://doi.org/10.1016/j.dcn.2019.100732

    • PubMed
    • Google Scholar
73. 1. McDougle SD
    2. Collins AGE

    (2021) Modeling the influence of working memory, reinforcement, and action uncertainty on reaction time and choice during instrumental learning

    Psychonomic Bulletin & Review 28:20–39.

    https://doi.org/10.3758/s13423-020-01774-z

    • PubMed
    • Google Scholar
74. 1. McGuire JT
    2. Nassar MR
    3. Gold JI
    4. Kable JW

    (2014) Functionally dissociable influences on learning rate in a dynamic environment

    Neuron 84:870–881.

    https://doi.org/10.1016/j.neuron.2014.10.013

    • PubMed
    • Google Scholar
75. 1. Mohebi A
    2. Pettibone JR
    3. Hamid AA
    4. Wong JMT
    5. Vinson LT
    6. Patriarchi T
    7. Tian L
    8. Kennedy RT
    9. Berke JD

    (2019) Dissociable dopamine dynamics for learning and motivation

    Nature 570:65–70.

    https://doi.org/10.1038/s41586-019-1235-y

    • PubMed
    • Google Scholar
76. 1. Moutoussis M
    2. Garzón B
    3. Neufeld S
    4. Bach DR
    5. Rigoli F
    6. Goodyer I
    7. Bullmore E
    8. Guitart-Masip M
    9. Dolan RJ
    10. NSPN Consortium

    (2021) Decision-Making ability, psychopathology, and brain connectivity

    Neuron 109:2025–2040.

    https://doi.org/10.1016/j.neuron.2021.04.019

    • PubMed
    • Google Scholar
77. 1. Nassar MR
    2. Bruckner R
    3. Gold JI
    4. Li SC
    5. Heekeren HR
    6. Eppinger B

    (2016) Age differences in learning emerge from an insufficient representation of uncertainty in older adults

    Nature Communications 7:11609.

    https://doi.org/10.1038/ncomms11609

    • PubMed
    • Google Scholar
78. 1. Nassar MR
    2. Frank MJ

    (2016) Taming the beast: extracting generalizable knowledge from computational models of cognition

    Current Opinion in Behavioral Sciences 11:49–54.

    https://doi.org/10.1016/j.cobeha.2016.04.003

    • PubMed
    • Google Scholar
79. 1. Navarro DJ

    (2019) Between the devil and the deep blue sea: tensions between scientific judgement and statistical model selection

    Computational Brain & Behavior 2:28–34.

    https://doi.org/10.1007/s42113-018-0019-z

    • Google Scholar
80. 1. Niv Y

    (2009) Reinforcement learning in the brain

    Journal of Mathematical Psychology 53:139–154.

    https://doi.org/10.1016/j.jmp.2008.12.005

    • Google Scholar
81. 1. Nussenbaum K
    2. Hartley CA

    (2019) Reinforcement learning across development: what insights can we draw from a decade of research?

    Developmental Cognitive Neuroscience 40:100733.

    https://doi.org/10.1016/j.dcn.2019.100733

    • PubMed
    • Google Scholar
82. 1. O’Doherty J
    2. Dayan P
    3. Schultz J
    4. Deichmann R
    5. Friston K
    6. Dolan RJ

    (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning

    Science 304:452–454.

    https://doi.org/10.1126/science.1094285

    • PubMed
    • Google Scholar
83. 1. O’Doherty JP
    2. Lee SW
    3. McNamee D

    (2015) The structure of reinforcement-learning mechanisms in the human brain

    Current Opinion in Behavioral Sciences 1:94–100.

    https://doi.org/10.1016/j.cobeha.2014.10.004

    • Google Scholar
84. 1. Palminteri S
    2. Kilford EJ
    3. Coricelli G
    4. Blakemore SJ
    5. O’Reilly JX

    (2016) The computational development of reinforcement learning during adolescence

    PLOS Computational Biology 12:e1004953.

    https://doi.org/10.1371/journal.pcbi.1004953

    • Google Scholar
85. 1. Palminteri S
    2. Wyart V
    3. Koechlin E

    (2017) The importance of falsification in computational cognitive modeling

    Trends in Cognitive Sciences 21:425–433.

    https://doi.org/10.1016/j.tics.2017.03.011

    • PubMed
    • Google Scholar
86. 1. Pedregosa F
    2. Varoquaux G
    3. Gramfort A
    4. Michel V
    5. Thirion B
    6. Grisel O
    7. Blondel M
    8. Prettenhofer P
    9. Weiss R
    10. Dubourg V
    11. Vanderplas J
    12. Passos A
    13. Cournapeau D
    14. Brucher M
    15. Perrot M

    (2011)

    Scikit-learn: machine learning in python

    Journal of Machine Learning Research 12:2825–2830.

    • Google Scholar
87. 1. Petersen AC
    2. Crockett L
    3. Richards M
    4. Boxer A

    (1988) A self-report measure of pubertal status: reliability, validity, and initial norms

    Journal of Youth and Adolescence 17:117–133.

    https://doi.org/10.1007/BF01537962

    • PubMed
    • Google Scholar
88. 1. Peterson DA
    2. Elliott C
    3. Song DD
    4. Makeig S
    5. Sejnowski TJ
    6. Poizner H

    (2009) Probabilistic reversal learning is impaired in Parkinson ’ S disease

    Neuroscience 163:1092–1101.

    https://doi.org/10.1016/j.neuroscience.2009.07.033

    • PubMed
    • Google Scholar
89. 1. Pratt DN
    2. Barch DM
    3. Carter CS
    4. Gold JM
    5. Ragland JD
    6. Silverstein SM
    7. MacDonald AW

    (2021) Reliability and replicability of implicit and explicit reinforcement learning paradigms in people with psychotic disorders

    Schizophrenia Bulletin 47:731–739.

    https://doi.org/10.1093/schbul/sbaa165

    • Google Scholar
90. 1. Radulescu A
    2. Niv Y
    3. Ballard I

    (2019) Holistic reinforcement learning: the role of structure and attention

    Trends in Cognitive Sciences 23:278–292.

    https://doi.org/10.1016/j.tics.2019.01.010

    • PubMed
    • Google Scholar
91. 1. Ribas-Fernandes JJF
    2. Solway A
    3. Diuk C
    4. McGuire JT
    5. Barto AG
    6. Niv Y
    7. Botvinick MM

    (2011) A neural signature of hierarchical reinforcement learning

    Neuron 71:370–379.

    https://doi.org/10.1016/j.neuron.2011.05.042

    • PubMed
    • Google Scholar
92. 1. Robinson OJ
    2. Chase HW

    (2017) Learning and choice in mood disorders: searching for the computational parameters of anhedonia

    Computational Psychiatry 1:208–233.

    https://doi.org/10.1162/CPSY_a_00009

    • PubMed
    • Google Scholar
93. Preprint

    1. Rosenbaum G
    2. Grassie H
    3. Hartley CA

    (2020) Valence Biases in Reinforcement Learning Shift across Adolescence and Modulate Subsequent Memory

    PsyArXiv.

    https://psyarxiv.com/n3vsr/

    • Google Scholar
94. 1. Schultz W
    2. Dayan P
    3. Montague PR

    (1997) A neural substrate of prediction and reward

    Science 275:1593–1599.

    https://doi.org/10.1126/science.275.5306.1593

    • PubMed
    • Google Scholar
95. 1. Schultz W
    2. Dickinson A

    (2000) Neuronal coding of prediction errors

    Annual Review of Neuroscience 23:473–500.

    https://doi.org/10.1146/annurev.neuro.23.1.473

    • PubMed
    • Google Scholar
96. 1. Sendhilnathan N
    2. Semework M
    3. Goldberg ME
    4. Ipata AE

    (2020) Neural correlates of reinforcement learning in mid-lateral cerebellum

    Neuron 106:188–198.

    https://doi.org/10.1016/j.neuron.2019.12.032

    • PubMed
    • Google Scholar
97. 1. Shahar N
    2. Hauser TU
    3. Moutoussis M
    4. Moran R
    5. Keramati M
    6. Consortium N
    7. Dolan RJ

    (2019) Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling

    PLOS Computational Biology 15:e1006803.

    https://doi.org/10.1371/journal.pcbi.1006803

    • PubMed
    • Google Scholar
98. 1. Somerville LH
    2. Sasse SF
    3. Garrad MC
    4. Drysdale AT
    5. Abi Akar N
    6. Insel C
    7. Wilson RC

    (2017) Charting the expansion of strategic exploratory behavior during adolescence

    Journal of Experimental Psychology. General 146:155–164.

    https://doi.org/10.1037/xge0000250

    • PubMed
    • Google Scholar
99. 1. Starkweather CK
    2. Gershman SJ
    3. Uchida N

    (2018) The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty

    Neuron 98:616–629.

    https://doi.org/10.1016/j.neuron.2018.03.036

    • PubMed
    • Google Scholar
100. 1. Sugawara M
     2. Katahira K

     (2021) Dissociation between asymmetric value updating and perseverance in human reinforcement learning

     Scientific Reports 11:3574.

     https://doi.org/10.1038/s41598-020-80593-7

     • PubMed
     • Google Scholar
101. Book

     1. Sutton RS
     2. Barto AG

     (2017)

     Reinforcement Learning: An Introduction (2nd Edition)

     Cambridge, MA; London, England: MIT Press.

     • Google Scholar
102. 1. Swainson R
     2. Rogers RD
     3. Sahakian BJ
     4. Summers BA
     5. Polkey CE
     6. Robbins TW

     (2000) Probabilistic learning and reversal deficits in patients with Parkinson ’ S disease or frontal or temporal lobe lesions: possible adverse effects of dopaminergic medication

     Neuropsychologia 38:596–612.

     https://doi.org/10.1016/s0028-3932(99)00103-7

     • PubMed
     • Google Scholar
103. 1. Tai LH
     2. Lee AM
     3. Benavidez N
     4. Bonci A
     5. Wilbrecht L

     (2012) Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value

     Nature Neuroscience 15:1281–1289.

     https://doi.org/10.1038/nn.3188

     • PubMed
     • Google Scholar
104. 1. Uttal WR

     (1990) On some two-way barriers between models and mechanisms

     Perception & Psychophysics 48:188–203.

     https://doi.org/10.3758/bf03207086

     • PubMed
     • Google Scholar
105. 1. van den Bos W
     2. Cohen MX
     3. Kahnt T
     4. Crone EA

     (2012) Striatum-medial prefrontal cortex connectivity predicts developmental changes in reinforcement learning

     Cerebral Cortex 22:1247–1255.

     https://doi.org/10.1093/cercor/bhr198

     • PubMed
     • Google Scholar
106. 1. van den Bos W
     2. Bruckner R
     3. Nassar MR
     4. Mata R
     5. Eppinger B

     (2018) Computational neuroscience across the lifespan: promises and pitfalls

     Developmental Cognitive Neuroscience 33:42–53.

     https://doi.org/10.1016/j.dcn.2017.09.008

     • PubMed
     • Google Scholar
107. 1. van der Schaaf ME
     2. Warmerdam E
     3. Crone EA
     4. Cools R

     (2011) Distinct linear and non-linear trajectories of reward and punishment reversal learning during development: relevance for dopamine ’ S role in adolescent decision making

     Developmental Cognitive Neuroscience 1:578–590.

     https://doi.org/10.1016/j.dcn.2011.06.007

     • PubMed
     • Google Scholar
108. 1. Vikbladh OM
     2. Meager MR
     3. King J
     4. Blackmon K
     5. Devinsky O
     6. Shohamy D
     7. Burgess N
     8. Daw ND

     (2019) Hippocampal contributions to model-based planning and spatial memory

     Neuron 102:683–693.

     https://doi.org/10.1016/j.neuron.2019.02.014

     • PubMed
     • Google Scholar
109. 1. Waltmann M
     2. Schlagenhauf F
     3. Deserno L

     (2022) Sufficient reliability of the behavioral and computational readouts of a probabilistic reversal learning task

     Behavior Research Methods pp. 1–22.

     https://doi.org/10.3758/s13428-021-01739-7

     • PubMed
     • Google Scholar
110. 1. Waltz JA
     2. Gold JM

     (2007) Probabilistic reversal learning impairments in schizophrenia: further evidence of orbitofrontal dysfunction

     Schizophrenia Research 93:296–303.

     https://doi.org/10.1016/j.schres.2007.03.010

     • PubMed
     • Google Scholar
111. 1. Wang JX
     2. Kurth-Nelson Z
     3. Kumaran D
     4. Tirumala D
     5. Soyer H
     6. Leibo JZ
     7. Hassabis D
     8. Botvinick M

     (2018) Prefrontal cortex as a meta-reinforcement learning system

     Nature Neuroscience 21:860–868.

     https://doi.org/10.1038/s41593-018-0147-8

     • PubMed
     • Google Scholar
112. 1. Watanabe S

     (2013)

     A widely applicable Bayesian information criterion

     Journal of Machine Learning Research 14:867–897.

     • Google Scholar
113. 1. Webb B

     (2001) Can robots make good models of biological behaviour?

     The Behavioral and Brain Sciences 24:1033–1050.

     https://doi.org/10.1017/s0140525x01000127

     • PubMed
     • Google Scholar
114. Conference

     1. Weidinger L
     2. Gradassi A
     3. Molleman L
     4. van den Bos W

     (2019) Test-retest reliability of canonical reinforcement learning models

     2019 Conference on Cognitive Computational Neuroscience.

     https://doi.org/10.32470/CCN.2019.1053-0

     • Google Scholar
115. 1. Werchan DM
     2. Collins AGE
     3. Frank MJ
     4. Amso D

     (2016) Role of prefrontal cortex in learning and generalizing hierarchical rules in 8-month-old infants

     The Journal of Neuroscience 36:10314–10322.

     https://doi.org/10.1523/JNEUROSCI.1351-16.2016

     • PubMed
     • Google Scholar
116. 1. Wilson RC
     2. Collins AG

     (2019) Ten simple rules for the computational modeling of behavioral data

     eLife 8:e49547.

     https://doi.org/10.7554/eLife.49547

     • PubMed
     • Google Scholar
117. 1. Xia L
     2. Master SL
     3. Eckstein MK
     4. Baribault B
     5. Dahl RE
     6. Wilbrecht L
     7. Collins AGE

     (2021) Modeling changes in probabilistic reinforcement learning during adolescence

     PLOS Computational Biology 17:e1008524.

     https://doi.org/10.1371/journal.pcbi.1008524

     • PubMed
     • Google Scholar
118. 1. Yaple ZA
     2. Yu R

     (2019) Fractionating adaptive learning: a meta-analysis of the reversal learning paradigm

     Neuroscience and Biobehavioral Reviews 102:85–94.

     https://doi.org/10.1016/j.neubiorev.2019.04.006

     • PubMed
     • Google Scholar
119. 1. Yarkoni T

     (2020) The generalizability crisis

     The Behavioral and Brain Sciences 45:e1.

     https://doi.org/10.1017/S0140525X20001685

     • PubMed
     • Google Scholar
120. 1. Yu AJ
     2. Dayan P

     (2005) Uncertainty, neuromodulation, and attention

     Neuron 46:681–692.

     https://doi.org/10.1016/j.neuron.2005.04.026

     • PubMed
     • Google Scholar