People and animals undergo a number of cognitive changes across healthy aging, and while some of these changes reflect improvements emerging from an extended lifespan of continual learning, others are characterized by declines in function with advanced age (Cattell, 1943). In particular, recent work has highlighted age-related deficits in decision-making under uncertainty, particularly in situations where decision outcomes are learned through experience (Eppinger et al., 2013). While such deficits could have critical importance for both aging individuals and society, the exact mechanisms underlying these deficits remain unclear. One possibility is that such impairments stem from an inability to retain the task-relevant information (i.e. working memory, Salthouse and Babcock, 1991), whereas another possibility is that such impairments stem from failures to learn state action values through reinforcement (i.e. reinforcement learning, Chowdhury et al., 2013). These different cognitive processes are supported by different neural systems. Understanding how they contribute to impairments in decision-making across lifespan could also help shed light on biological age-related mechanisms of impaired decision-making.

Previous work has shown that working memory declines with aging (Salthouse and Babcock, 1991). Such declines have been shown in both cross-sectional and longitudinal studies and across a range of working memory tasks (Salthouse and Babcock, 1991). Working memory is thought to be implemented through persistent activation of neural firing in similarly tuned neurons within recurrent excitatory networks in prefrontal and parietal cortices (Goldman-Rakic, 1995; Andersen and Buneo, 2002). Within such networks, successful maintenance of activity over a delay period depends on glutamatergic signaling, and in particular activation of NMDA receptors (Durstewitz et al., 2000; van Vugt et al., 2020). In aged monkeys, such persistent activations in prefrontal cortex are diminished and can be restored through local pharmacological manipulations that affect intrinsic neural excitability (Wang et al., 2011). While the mechanisms by which persistent activity deteriorates with age are not fully established, observed reductions in synapses and dendritic spines on aged prefrontal pyramidal neurons (Dumitriu et al., 2010) suggest that they result at least in part from an overall reduction in glutamatergic signaling. Thus, if age-related behavioral deficits stem from diminished working memory functions, they may result from reductions in glutamate signaling in prefrontal and parietal regions that support working memory.

Reinforcement learning (RL) systems are also thought to decline across healthy aging. Evidence for such declines comes from behavioral impairments in tasks where participants learn to map stimuli onto rewarded actions (Eppinger et al., 2013; Frank and Kong, 2008; Hämmerer et al., 2011). Learning in such tasks is thought to occur through reinforcement of associations between stimulus and action in the striatum driven by dopamine reward prediction error signals (Frank et al., 2004; Collins and Frank, 2014). Such reward prediction error signals are blunted in healthy aging (Samanez-Larkin et al., 2014) supporting the accounts of age-related decline in RL (Chowdhury et al., 2013; Eppinger et al., 2013). Chowdhury et al., 2013 found that boosting dopamine signaling with dopamine precursor levodopa (L-DOPA) led to restoration of RPEs and recovery of learning performance in older adults. The results from these studies appear to indicate that neural mechanisms underlying reinforcement learning are impaired in older adults; however, a number of other studies have suggested that RL systems are intact in older adults (Radulescu et al., 2016; Grogan et al., 2019). Some of the discrepancy in these results may stem from inconsistencies in the tasks used to measure RL across different studies (Eckstein et al., 2022) as many tasks that could be solved through incremental learning in the striatum according to reward prediction errors (RL) could also be solved using other cognitive systems, working memory systems being one of particular importance (Yoo and Collins, 2022).

Typical reinforcement learning tasks require identifying the best action to choose when confronted with a given stimulus based on previous feedback. While RL models of such tasks assume that action values are learned incrementally, as if through adjustment of synaptic weights in the striatum, it is also possible for participants to achieve success on such tasks through short-term storage of recent trial information in working memory. Recent work has used task designs that can dissociate these potential contributors to goal directed behavior, along with models that can quantify them, to show that behavioral deficits previously thought to reflect reinforcement learning were actually attributable to working memory systems (Collins et al., 2014). The reinforcement learning–working memory task (Collins and Frank, 2012; Collins, 2018; Collins et al., 2014; Collins and Frank, 2018) is a simple stimulus–response association learning task with a format that is commonly observed in RL studies (Frank and Kong, 2008). However, this task includes a WM manipulation – varying the number of stimulus–response actions (set size) participants need to learn. This manipulation targets WM as a capacity-limited, short-term process since increasing set-size increases both load and average duration between stimulus repetitions. Thus, varying the set size can shift contribution of RL/WM to performance and help tease apart which of the two participants are relying on when performing the task. This task has yielded important mechanistic insights into general as well as clinical populations (Collins et al., 2014; Collins and Frank, 2012; Master et al., 2020).

In this study, we combined cognitive, computational, and neural approaches to address the question of (1) how the age-related deficits in making decisions from experience emerge from RL and WM computational mechanisms and (2) how the differences in these mechanisms relate to age-related changes in relevant neural systems. To this end, we administered an RL-WM task to young and older adults and modeled their behavior using computational model that can distinguish between deficits in RL and WM systems. We used magnetic resonance spectroscopy (MRS) to measure levels of glutamate and GABA in regions thought to support working memory (prefrontal/parietal cortices) and reinforcement learning (striatum).

We found that older participants performed worse in the RL-WM task than younger adults. Group comparison of model parameters revealed that older participants had a more rapid decay of task relevant information in working memory. Older adults also had reduced glutamate levels, particularly in prefrontal and parietal cortices where glutamate is thought to support the persistent activation that underlies working memory. Furthermore, reductions in prefrontal glutamate were correlated with working memory decay across individuals – such that those with the lowest levels of prefrontal glutamate had the most rapidly decaying working memories. Taken together, our results suggest that age-related working memory differences likely contribute to decision-making deficits in older adults, and that such deficits may result from failures of recurrent excitatory networks to sustain persistent representations due to reduced levels of glutamate.

42 older (age mean [SD] = 68 [8.5]) and 36 younger (age mean (SD) = 21 [4.4]) participants were enrolled in a two-session study. Age-normed total The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) (Randolph et al., 1998) scores were similar in both groups (mean [std] younger: 106.4 [12.0], older: 109.0 [10.8]), suggesting that our cohorts reflected comparable samples of the population with respect to overall cognitive ability. Furthermore, all participants scored in cognitively healthy range (younger adults range: 83–140; older adults range: 81–141) since cognitive impairment is defined as score 70 or below. The first session required performance of an RL-WM task designed to dissociate the contributions of working memory and reinforcement learning to decision-making under uncertainty. The second study session included an MRI session in which MRS was used to measure glutamate and GABA in key regions thought to support working memory (middle frontal gyrus [MFG] of prefrontal cortex and intraparietal sulcus [IPS]) and reinforcement learning (striatum).

In the RL-WM task, older and young adults were required to learn correct stimulus–response (key press) associations, with the goal of earning as many points as possible. After the stimulus appeared, the participants had 1 s to make their response, following which they received deterministic feedback. Each stimulus appeared nine times in a block, pseudo-randomly interleaved with other stimuli, allowing the participants to learn correct associations from feedback. The task has a common RL experiment structure (Frank et al., 2004), with an important difference: the number of associations varied (either three or six) between different, fully independent blocks. This enabled us to manipulate the degree to which working memory (WM), a capacity-limited system, might contribute to behavior by storing recent stimulus–action–outcome information. In particular, previous computational and empirical work using this task suggests that working memory will contribute less to decisions in the high set size condition (six associations), but that slower learning in high set size paradoxically leads to better retention relative to learning in a surprise ‘test phase’ in which learned associations are tested in the absence of feedback (Collins, 2018).

Behavioral results; model-independent

Learning

Both age groups learned to make accurate responses in the learning phase and did so more quickly in blocks with fewer interleaved stimuli, but younger adults learned more successfully than their older counterparts (Figure 1B and C). The learning curves reveal patterns that are consistent with RL and WM involvement in young adults (early spike in accuracy for lower set size that is absent for larger set size and that is considered a marker of fast/one-shot learning characteristic of WM but not RL; Collins and Frank, 2012). This pattern is less clear in older adults. The average accuracy of individuals in the young and old groups was 0.74 (SD = 0.01) and 0.60 (SD = 0.01), respectively (ANOVA for group difference: $F(1,76)=45.42,p=2.71e-09$). Younger adults had higher accuracy compared to older adults for both set sizes 3 ($t(76)=4.71$,$p=1.05e-05$) and 6 ($t(76)=7.03$,$p=7.61e-10$). Furthermore, we confirmed that the difference between set size 3 and set size 6 performance was greater in older adults ($t(76)=4.12,p=9.55e-05$). Training performance deficits in both set sizes were greatest in the oldest individuals, as revealed by significant correlations between age and training accuracy within the older group (Figure 1C; $r(n_S=3accuracy,age)=-0.42$, ${\displaystyle p=0.03}$; ${\displaystyle r(n_S=6accuracy,age)=-0.41}$, ${\displaystyle p=0.04}$).

Figure 1

Download asset Open asset

To test what factors (i.e. set size, reward history, delay) impact accuracy on a trial-by-trial level, we ran a mixed-effects general linear model (GLM). The GLM analysis of accuracy data indicated that both groups showed signatures of RL and WM in their behavior. We found a positive effect of reward history (${\displaystyle \beta =1.74}$, $t=43.51$, ${\displaystyle p<0.0003}$) that was apparent in both young and old groups (young: $\beta =1.66$, $t=27.60$, ${\displaystyle p<0.0003}$ ; older: $\beta =1.74$, $t=32.5$ ,${\displaystyle p<0.0003}$). We also found negative effects of set size and delay on accuracy (set size: $\beta =-0.26$, $t=-9.91$, ${\displaystyle p<0.0003}$ ; delay: ${\displaystyle \beta =-0.30}$, $t=-13.3$, ${\displaystyle p<0.0003}$) that were apparent in both young (set size: $\beta =-0.23$, $t=-7.55$, ${\displaystyle p<0.0003}$; delay: $\beta =-0.21$, $t=-5.11$, ${\displaystyle p<0.0003}$) and older adults (set size: $\beta =-0.34$, $t=-9.73$, ${\displaystyle p<0.0003}$; delay: $\beta =-0.29$, $t=-8.92$, ${\displaystyle p<0.0003}$).

Coefficients from the GLM, particularly those capturing behavioral hallmarks of working memory (i.e. set size, delay), could be used to infer participants age. A linear regression to predict the individual participants’ age based on their respective coefficients from the behavioral logistic regression model yielded negative coefficients for the accuracy fixed effect ($\beta =-17.2$, $t(71)=-5.64$, $p=3.1455e-07$) and its sensitivity to set size ($\beta =-7.44$, $t(71)=-3.01$, $p=0.0035$), suggesting that older adults had lower overall performance but were also more affected by the set size manipulation. Reward history coefficients took positive values ($\beta =12.73$, $t(71)=4.24$, $p=6.5262e-05$) in the same model, suggesting that older adults were also more sensitive to previous rewards than their younger counterparts.

Testing

Both age groups experienced set size-dependent declines in performance during the test phase of the task, with older adults achieving lower levels of performance overall (Figure 1B, inset figures). Consistent with the notion that greater use of WM during learning in small set sizes weakens RL-dependent learning as expressed in the test phase (Collins, 2018), participants had greater declines in performance for set size 3 than for set size 6 between training and testing (t(${\mathrm{\Delta }}_{3^L,3^T}$, ${\mathrm{\Delta }}_{6^L,6^T}$) young: $t(35)=4.64$, ${\displaystyle \mathrm{p}<0.001}$ ; older: $t(41)=8.1$, ${\displaystyle \mathrm{p}<0.001}$). The set size-based asymmetry in accuracy decline from training to testing was numerically greater in older adults, but the effect was not significant ($t(76)=1.62$, ${\displaystyle \mathrm{p}=0.1}$).

A GLM fit to test phase accuracy confirmed that reward history was still strongly linked to performance in the test phase, but that set size was less important. An analogous mixed effect model to that used on the training data revealed positive reward history effects in both young and older adults (young: ${\displaystyle \beta =1.03}$, $t=16.2$, ${\displaystyle p<0.001}$; older: ${\displaystyle \beta =0.92}$, ${\displaystyle t=0.92}$, ${\displaystyle \mathrm{p}<0.001}$). In contrast, in the test phase there was limited evidence for an effect of set size on performance in young adults ($\beta =-0.15$, $t=-1.85$, ${\displaystyle p=0.06}$) and only a very modest effect in the older adults ($\beta =-0.20$, $t=-3.07$, ${\displaystyle \mathrm{p}=0.002}$).

In summary of our basic behavioral results, general patterns in the data are consistent with contributions of both working memory and reinforcement learning to behavior, replicate previous findings showing interactions between the two revealed in the test phase, and confirm age-related variability in performance. To better understand how the WM and RL mechanisms vary with age, we used computational modeling to further dissect the computational factors giving rise to differences in participant behavior.

Modeling results

To gain more insight into how specific learning mechanisms vary with age, we fit a computational model to the RL-WM task behavioral data. Specifically, we applied a hybrid RL-WM model that quantifies how WM and RL contribute to learning together (Figure 2; for a detailed description of how the model captures WM and RL processes see ‘Methods’ section). Parameters attributed to WM consisted of (1) WM decay ($\varphi$) – capturing the rate at which stimulus–response associations decayed during learning, and (2) set size-dependent WM reliance parameters (${\omega }^3$ and ${\omega }^6$) – capturing relative reliance on WM (as opposed to RL) to make choices. RL parameters consisted of the learning rate (${\alpha }^{+}$) and test phase softmax inverse temperature (${\beta }^T$), governing the rate at which participants updated the stimulus–response association values based on observed feedback, and the extent to which participants’ test phase choices were deterministic vs. exploratory, respectively. Learning phase softmax beta (${\beta }^L$) captured the same dynamic during the learning phase (but unlike all other RL-WM model parameters was estimated at the group-level only; see ‘Methods’ for details). We also incorporated a negative learning rate (${\alpha }^{-}$) to differentiate learning from negative versus positive feedback, which was present in both RL and WM modules of the model. Finally, the model included a noise parameter in response selection ($ϵ$) to capture random lapses in task performance.

Figure 2

Download asset Open asset

We implemented the RL-WM model in a hierarchical Bayesian framework, as Bayesian model fitting offers many statistical benefits (e.g. Kruschke and Liddell, 2018) and incorporating hierarchical structure allowed us to account for individual- and group-level effects simultaneously (among other practical benefits; e.g. Lee, 2011). As this represents a novel extension of the RL-WM model, we offer full model specification in the ‘Methods’ section; further details on the exact approach to Bayesian model fitting are also available in ‘Methods.

Model comparison

We first compared the performance among the three versions of our Bayesian RL-WM model (with different hierarchical structures; see ‘Methods’ for details) in order to select a model to use in all model-based analyses of our data. Our model comparison indicated that the ‘two-group’ version of the model, which incorporated separate hierarchies over the participants within each age group, offered the best description of our data (see Appendix 1—figure 2). This two-group, hierarchical version of the model outperformed both the non-hierarchical model ($\mathrm{\Delta }\mathrm{WAIC}=1.62\times {10}^3$) and the model that included a single hierarchy over all participants, regardless of their age group ($\mathrm{\Delta }\mathrm{WAIC}=1.86\times {10}^2$). This model’s posterior predictive learning curves and posterior predictive asymptotic means (Figure 2B) successfully captured the general patterns in performance of both age groups on the RL-WM task.

Model parameter recovery

We used model parameter recovery to ensure that parameters in our best fitting model are separable, and that our model is sensitive to individual variability in model parameters. We ran a recovery study where we simulated data from estimated parameters according to the RL-WM model using the same task design as in our human experiment and then fit the synthetic data with our model. We confirmed that both individual-level parameters and group-level hyperparameters were recovered successfully (see Appendix 1—figure 9).

Parameter analysis

We next examined which parameters of the model differed substantially across age groups. The strongest difference between the age groups was with respect to decay rate $\varphi$ in the WM module. The group mean decay rate was much higher in the older age group ($\mathrm{\Delta }{\mu }^{\varphi }=-0.151$, 95% equal-tailed credible interval [CrI] = [-0.227, -0.070]) indicating that the stimulus–action–outcome associations stored in WM degraded faster for older adults. The mean decay rate for the younger group was 0.174, indicating that an association would decay to 83% of its original strength after a single trial, whereas in older adults the decay of 0.324 would lead to working memory degradation about twice as fast.

Next, we focused on the relative WM reliance parameter $\omega$. We were interested in exploring whether the WM reliance might follow the similar pattern observed in group-related difference in the WM decay. For instance, if older participants’ WM module is markedly more forgetful, they would accordingly rely less on the association weights stored in WM to guide their action selection. In our analysis, we observed that the relative reliance on WM over RL — as captured by the set size-dependent policy mixture parameters ${\omega }^{n_S}$ — depended heavily on the set size in both groups. While learning stimulus–action associations in set size 3, participants in both groups relied more on WM (${\displaystyle {\omega }^3>0.5}$ for 36 of 36 younger and 40 of 42 older participants). In set size 6, participants in both groups tended to rely more on RL (${\displaystyle {\omega }^6<0.5}$ for 32 of 36 younger and 35 of 42 older participants). Importantly, ${\omega }^3$ was greater than ${\omega }^6$ for 36 of 36 younger and 40 of 42 older participants. We observed the same effect for the group mean parameters, with similar magnitudes for both groups (${\mu }^{{\omega }^3}-{\mu }^{{\omega }^6}$, young: [0.336, 0549]; older: [0.214, 0.420]). Older adults had a somewhat reduced tendency to rely on WM ($\mathrm{\Delta }{\mu }^{\omega 3}=0.090[-0.003,0.179]$) compared to younger adults, even when doing so is appropriate (i.e. for the smaller set size), but this difference was not robust. Thus, older and young adults showed similar patterns in WM reliance, contingent on the set size.

The model fits revealed very little indication of group differences in parameters specific to the RL system. We found no group differences in the test phase beta ${\beta }^T$ ([-0.465, 7.673]) or the learning rates $\alpha$ ([-0.007, 0.042]). Our model did have a learning rate asymmetry parameter that affected both RL and WM systems, and allowed the model to capture a consistent trend toward higher learning from positive relative to negative outcomes in both groups. This was evident in group means (${\mu }^{{\alpha }^{+}}-{\mu }^{{\alpha }^{-}}$, young: [0.004, 0.049], older: [0.009, 0.0041]) as well as for individual participants (${\displaystyle {\alpha }^{+}>{\alpha }^{-}}$ for 34 of 36 young and 42 of 42 older participants). This asymmetry differed across groups, such that negative learning rates were higher in the young adult group compared to the older adults (Figure 2C) ($\mathrm{\Delta }{\mu }^{{\alpha }^{-}}$ = $0.016,[0.002,0.032]$). This suggests that older adults neglected negative feedback even more relative to young adults. This result is inconsistent with the previous work, suggesting that older adults exhibit bias in learning from negative outcomes (Frank and Kong, 2008). However, this finding has not been consistent across studies, and in our paradigm a bias toward learning from positive outcomes could be viewed as an adaptive strategy when resources are limited, in that positive outcomes perfectly prescribe the correct future action, whereas negative ones only rule a single response out.

Importantly, the noise/random lapse parameter did not differ between the two age groups ($\mathrm{\Delta }{\mu }^{ϵ}=[-0.011,0.047]$), which suggests it is unlikely that the observed differences in accuracy are due to older adults simply having noisier data.

To demonstrate the impact of their higher decay rate on older adults’ learning, we again simulated older adults’ behavior from the fitted model, except we used the young adults’ group-level decay rate in place of every older adult’s individual-level decay rate. To do this, we used the same procedure as was used to generate the posterior predictive plots (as in Figure 2b; see ‘Methods’), except with the aforementioned parameter swaps. This demonstration (see Appendix 1—figure 5) suggests that the older adult’s higher decay rate could potentially account for most of the difference in learning behavior between the age groups.

Our behavioral results revealed that older adults performed more poorly in set size 6, in which performance is more driven by RL than in set size 3, due to capacity limitations of WM. However, our modeling revealed that WM parameters, specifically WM decay $\varphi$, show greatest group differences, rather than RL parameters. Simulation results (see Appendix 1—figure 5) provided some insight into this finding by showing that decreasing the working memory decay in older adults improves training performance most notably in the set size 6 condition. The primary explanation for this is that higher set size assumes increased delay between successive presentations of the same stimuli. Longer delays, in turn, make differences in WM decay more pronounced. Furthermore, a WM impairment can have an indirect effect in RL, in that frequent failure to select correct action through WM leads to reduced ability to train RL on encoding correct responses (especially earlier in training, when the incremental RL has not ‘caught up’ yet), and thus worse performance overall. Thus, our behavioral and modeling results are not incongruous and simply highlight the importance of modeling as it allows us to identify the mechanisms that drive behavioral patterns with greater specificity (Radulescu et al., 2019; Collins and Frank, 2012).

Linking MRS measures, model parameters, and behavioral performance

Thus far, our results suggest that (1) there is a significant difference in performance between two age groups, and (2) that this difference is best explained by working memory deficit in older group (most significantly WM decay, and marginally reliance on WM in smaller set size condition). Next, we aimed to test how these age-related differences relate to metabolic changes affecting brain neurochemistry. We used MRS to measure neurotransmitters (glutamate and GABA) that support working memory (Wong and Wang, 2006) in regions important for both reinforcement learning (striatum [STR]) and working memory (middle frontal gyrus of prefrontal cortex [MFG], intraparietal sulcus [IPS]). To control for larger structural changes that may covary with differences in glutamate and GABA levels, we included gray volume, white matter volume and cortical thickness (in caudal middle frontal [CMF], superior frontal [SF], and rostral middle frontal [RMF]) cortex in our analyses and to control for nonspecific metabolic changes we included measures of N-acetylaspartate (NAA).

In order to understand how differences in performance and their computational mechanisms relate to the neural measures. we took a data-driven multistep analysis approach. First, we constructed a GLM to explain overall learning performance according to structural measures collected as part of our high-resolution anatomical MRI scans (white matter, gray matter, cortical thickness), as well as neurochemical measures from MRS (glutamate, GABA, NAA). Given the correlations in our neurochemical measures across brain regions, we attempted to maximize power to detect relationships in our initial linear model by aggregating neurochemical measures across brain regions. Specifically, we averaged the MRS measures across regions (i.e. average glutamate was the average of glutamate measures from medial frontal gyrus, intraparietal sulcus, and striatum). Anatomical specificity of significant predictors was then examined in follow-up analyses. We set up a linear model predicting average learning performance using the following z-scored predictors: average GABA, glutamate, NAA, gray matter, white matter, and cortical thickness in caudal middle frontal (CMF), superior frontal (SF), and rostral middle frontal (RMF) areas (cortical thickness measure was not averaged across the three areas because these measures were selected a priori from a Freesurfer parcellation and are unrelated to the voxels; see ‘Methods’ for details). The best model identified through a regularized and cross-validated fitting procedure (see ‘Methods’) included the following predictors: GABA, glutamate, NAA, CMF cortical thickness, and SF cortical thickness (${\displaystyle R^{2}adjusted=0.41}$, $F=7.8$, ${\displaystyle p<0.001}$, variance inflation factors: [1.16, 2.06, 1.76, 2.71, 2.37]). However, out of all the predictors in this model, the only coefficient within this model for which a value of zero could be reliably rejected was our aggregate measure of glutamate concentration (${\displaystyle \beta =0.04}$, $t=2.5$, ${\displaystyle \mathrm{p}=0.01}$; Figure 3B).

Figure 3

Download asset Open asset

Next, we used our best-fitting model to generate brain-based predictions of performance so that we could evaluate their computational specificity. Note that these predictions depended on all biological factors in the model with best out-of-sample accuracy, which outperformed simpler models, including one that only had glutamate as a predictor. These predictions can be thought of as projections of behavioral learning performance onto the axis of brain measures most closely related to it. We then fit a linear model that regressed the brain-predicted performance onto an explanatory matrix that contained all parameter estimates from our behavioral model in order to determine which computational elements are most tightly linked to the neural fingerprint of performance (or performance failures). The model had the following structure: brain-predicted performance ∼1 + all RL-WM parameters. This model explained a significant amount variance in our brain-based performance measure (${\displaystyle R_{\mathrm{a}\mathrm{d}\mathrm{j}\mathrm{u}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{d}}^2=0.48}$, $F=7.46$, ${\displaystyle \mathrm{p}<0.001}$) and revealed that WM decay $\varphi$ (${\displaystyle \beta =–0.02}$, $t=-2.24$, ${\displaystyle \mathrm{p}=0.03}$) and set size 3 WM weight ${\omega }_3$ (${\displaystyle \beta =0.02}$, $t=2.35$, ${\displaystyle \mathrm{p}=0.03}$) contributed substantially to these predictions, suggesting that the brain-based predictions largely reflected the integrity of a working memory system (Figure 3C).

Having established that (1) glutamate contributes to accuracy and that (2) WM parameters $\varphi$ and ${\omega }_3$ capture the performance predictions based on neural measures (including glutamate), we next tested the anatomical specificity of the glutamate–working memory relationship (see ‘Methods’). To do so, we constructed two additional regression models, in which we regressed $\varphi$ (${\displaystyle R^{2}adjusted=0.32}$, $F=9.22$, ${\displaystyle \mathrm{p}<0.001}$) and ${\omega }_3$ (${\displaystyle R^{2}adjusted=0.23}$, $F=6.42$, ${\displaystyle \mathrm{p}<0.001}$) onto three separate glutamate measures extracted from MFG, IPS, and striatum (model 1: $\varphi \sim$ 1 + MFG glutamate + IPS glutamate + STR glutamate; model 2: ${\omega }_3$ 1 + MFG glutamate + IPS glutamate + STR glutamate). From these two models, the only significant coefficient was the effect of MFG glutamate on WM decay $\varphi$ (${\displaystyle \varphi }$: MFG glutamate ${\displaystyle \beta =-0.04}$, $t=-2.95$, ${\displaystyle \mathrm{p}=0.004}$; STR glutamate ${\displaystyle \beta =-0.008}$, $t=-0.66$, ${\displaystyle \mathrm{p}=0.50}$; IPS glutamate $\beta =-0.023$, $t=-1.67$, ${\displaystyle \mathrm{p}=0.10;{\omega }_3}$ : MFG glutamate ${\displaystyle \beta =0.02}$, $t=1.48$, ${\displaystyle \mathrm{p}=0.14}$; STR glutamate ${\displaystyle \beta =0.02}$, $t=1.77$, ${\displaystyle \mathrm{p}=0.08}$; IPS glutamate $\beta =0.02$, $t=1.70$, ${\displaystyle \mathrm{p}=0.09}$) (Figure 3D). Specifically, lower glutamate levels were associated with higher WM decay. This relationship was specific to glutamate and was not observed when we substituted measures of glutamine (${\displaystyle r=-0.05}$, ${\displaystyle \mathrm{p}=0.70}$), a molecule with a nearby spectral peak. The relationship between MFG glutamate and WM memory decay persisted even after regressing out variance related to MFG gray matter (GM) and MFG creatine (Cr) (${\displaystyle r=-0.39}$, ${\displaystyle \mathrm{p}=0.0035}$), suggesting that our results are specifically related to glutamate rather than picking up on nonspecific anatomical differences such as gray matter or tissue density. For completeness, we also tested the correlation between MFG glutamate and other model parameters. We found that MFG glutamate also correlated with $\alpha$, ${\omega }_3$ and ${\alpha }^{-}$ (see Appendix 1—figure 7). However, when these parameters were included in a model that also contained the WM decay parameter (MFG glutamate ∼ 1 + $\varphi$ + $\alpha$ + ${\omega }_3$ + ${\alpha }^{-}$), only ${\displaystyle \varphi }$ was a significant predictor (${\displaystyle \varphi :\beta =0.24}$, ${\displaystyle t=-3.14}$, ${\displaystyle \mathrm{p}=0.002}$, ${\displaystyle \alpha :\beta =-0.03}$, ${\displaystyle t=-0.42}$, ${\displaystyle \mathrm{p}=0.67}$, ${\displaystyle {\omega }_3:\beta =0.12}$, ${\displaystyle t=1.84}$, ${\displaystyle \mathrm{p}=0.07}$, ${\displaystyle {\alpha }^{-}\beta =0.04}$, ${\displaystyle t=0.56}$, ${\displaystyle \mathrm{p}=0.57}$). Taken together, these results suggest that higher levels of WM decay were associated with lower prefrontal glutamate levels.

Next, we examined how age factors into this relationship. We found that age negatively correlated with glutamate (${\displaystyle r_{\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{n}}=-0.59}$, $p=2.93e-06$) and positively correlated with decay (${\displaystyle r_{\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{n}}=0.63}$, $p=3.15e-07$). The visualization of this depicted in Figure 3D suggests that older adults have lower glutamate levels and higher decay, whereas inverse is true for younger adults. Since age was negatively correlated with glutamate and positively correlated with decay, there is a possibility that age alone could fully explain the correlation between glutamate and WM decay. To address this question, we (1) ran a model predicting WM decay using MFG glutamate and age, as well as age-by-MFG glutamate interaction, and (2) correlation between MFG glutamate and WM decay in the two age groups separately. The interaction was not significant (${\displaystyle \beta =-0.01}$, ${\displaystyle \mathrm{t}=0.69,\mathrm{p}=0.48}$), and the main effects of glutamate and age were not significant when entered together in the model (age: ${\displaystyle \beta =0.14}$, ${\displaystyle \mathrm{t}=1.24,\mathrm{p}=0.21}$; glutamate: ${\displaystyle \beta =-0.02}$, ${\displaystyle t=-1.03,p=0.30}$), likely due to the fact that glutamate and age are correlated. We found that glutamate did not correlate with WM decay in young adults (${\displaystyle r_{\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{n}}=0.11,\mathrm{p}=0.54}$), but there was a trending correlation between glutamate and WM decay in older adults (${\displaystyle r_{\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{n}}=-0.36,\mathrm{p}=0.08}$). Finally, we used a formal mediation model to examine whether the relationship between age and working memory decay was weakened after accounting for glutamate. While the sign of the mediating effect in the model was consistent with glutamate reducing the explanatory effects of age, the effect was not large enough to reject the null hypothesis that glutamate does not mediate the effects of age on working memory (${\displaystyle ACME=0.009,p=0.28}$, proportion mediated = 0.13). Taken together, these results reiterate that age is closely associated with working memory decay and reduced glutamate levels, but provide very little conclusive evidence regarding the exact nature of these relationships.

Our results suggest that age-related differences in the ability to make decisions from experience are driven by increased forgetting in working memory that is linked to underlying levels of glutamate in prefrontal cortex. Our results provide a computationally specific account of learning impairments that have previously been observed over the course of healthy aging, and also provide insight into its biological underpinnings, in particular suggesting that reduced glutamate impairs the ability of prefrontal recurrent networks to maintain task relevant information over extended periods of time.

While it has long been established that working memory declines over healthy aging (Salthouse and Babcock, 1991), our work shows that these deficits are also at the root of age-related impairments in decisions from experience (Eppinger et al., 2013). Importantly, this result held even when simultaneously accounting for RL contributions to behavior, which have also been theorized to be impacted by cognitive aging (Frank and Kong, 2008; Li et al., 2015). As shown by examples of previous work (Collins et al., 2014; Leong et al., 2017; Radulescu et al., 2019), it is important to consider the interactions between neurocognitive systems rather than studying them in isolation as this could lead to an incorrect attribution of observed behavioral patterns. Our behavioral modeling results are complementary to those of another recent study that leveraged varying delay between training and testing to examine contributions of working memory, and concluded that older adults engaged WM less than young adults during the short delays (van de Vijver and Ligneul, 2020). Our results indicated that older adults had more rapid decay of task relevant working memories, but did not substantially reduce their deployment for selecting actions. Thus, across studies, there is converging evidence that working memory deficits in older adults can negatively impact decisions from experience, perhaps providing insight into why decisions from description are relatively spared by the cognitive aging process as such decisions do not typically require working memory (Li et al., 2015). However, our results imply that older adults did not compensate for their unreliable memory system by using working memory less, thereby increasing the impact of working memory deficits on behavior. Though our framework provided an ideal setup to disentangle contributions of learning and memory to decisions, one limitation of our task is that it does not provide any behavioral information about why working memory systems fail. For example, higher working memory decay in our model could reflect failures to appropriately prioritize storage of task-relevant information, including susceptibility to interference effects such as have previously been proposed to play a major role in cognitive aging (Amer et al., 2022), but could also be explained by a gradually increasing failure to retrieve previously stored information. Thus, although our computational approach was able to show that much of the age-related decision-making impairment previously attributed to learning is actually attributable to failures of working memory, we rely on future work to better understand the exact computational nature of those failures.

Our conclusion that age-related performance deficits were attributable to WM rather than RL is in conflict with previous literature (Frank and Kong, 2008; Hämmerer et al., 2011) suggesting age-related impairment of RL systems. Previous work on aging and RL, however, has yielded mixed conclusions, with some results suggesting that RL is not impaired in older adults (Grogan et al., 2019; Radulescu et al., 2016). Placing our study in the context of the existing literature, we believe that inconsistent results might have occurred due to a failure to separate and account for additional processes (WM, attention, etc.) that might contribute to performance deficits in older populations. This in particular reflects our motivation for conducting this study as quantifying contribution of cognitive processes to performance should not be studied in isolation (Radulescu et al., 2016; Radulescu et al., 2019; Collins and Frank, 2012; Collins, 2018).

On the biological side, our results suggest that age-related differences in working memory, but no other age-related behavioral changes, were associated with reductions in levels of glutamate in prefrontal cortex. Previous work has noted structural changes in the aging brain, including reduced brain volume and specific reductions in gray matter in certain brain regions, including prefrontal cortex (West, 1996; Buckner et al., 2004). While we did note some structural differences between younger and older participants, these structural changes were not related to decision-making, and instead the best neural predictions of task performance relied primarily on neurochemical levels, in particular, glutmate. Glutamate is the primary excitatory neurotransmitter in the brain, and in prefrontal cortex, is thought to support local recurrent signaling that supports the active maintenance of information in cortical neural networks. An enticing interpretation of these results is that our noninvasive MRS measures, which provide regional quantification of glutamate, may provide a readout of changes in the local synaptic architecture (e.g. decreased density of excitatory synapses) that play a causal role in the cognitive aging process.

Nonetheless, our study is limited in the support that it can provide for this idea. We found a relationship between glutamate and the working memory decay parameter that explained age-related differences in task performance. However, the bulk of this relationship was driven by the group differences in levels of glutamate and working memory decay, with only a trend-level correlation between the factors existing in the older adult group. Furthermore, formal mediation modeling was unable to confirm that glutamate mediates the observed relationships between age and working memory decay. There are several potential explanations for the remaining causal ambiguity, one being that our study was not particularly well powered to see within group relationships between biological variables and behavior (24 older participants with both brain and behavioral measurements), making the observed correlation in older adults (r = −0.39) difficult to interpret one way or the other. Mediation analysis typically requires very large samples to ensure reasonable power levels (Fritz and Mackinnon, 2007) and has an additional obstacle in our case that we cannot measure our hypothesized mediating factor directly and instead rely on a noisy proxy measurement (MRS glutamate), potentially muddying the interpretation of mediation analyses (Westfall and Yarkoni, 2016). An alternative explanation would be that age really is the causal factor driving both metabolic changes in glutamate and, separately, cognitive declines in working memory that affect performance on learning tasks. While we are unable to conclusively arbitrate between these possibilities in this study, we hope that our results inspire future work with larger samples of older adults that can effectively do so.

Our study is also limited by the MRS methodology that we employed. A major limitation of MRS is that we are unable to verify that the age-related differences in glutamate that we measure are localized to synapses, or even to neurons. The chemical specificity of our findings lends some support to the idea that glutamate is playing a functional role rather than simply serving as one of many markers for the metabolic changes that occur over healthy aging, yet additional work leveraging animal models would be necessary to verify this idea and fully elucidate a causal mechanism. Beyond specificity, our MRS methodology limited the breadth of neurochemical measurements we could make. We did not measure levels of dopamine (DA), which is a critical biological marker of learning by reinforcement (Schultz et al., 1997), and could further inform our understanding of the age-related changes in neurochemistry, and its effect on cognition (Bäckman et al., 2006; Li et al., 2001). Furthermore, dopamine also might modulate WM in addition to RL (O’Reilly and Frank, 2006). Thus, the role of dopamine in RL-WM interaction in the aging population would be an interesting topic to explore in future research.

A third limitation of this study is the cross-sectional design. Specifically, we compared a group of young and older adults, without directly observing age-related changes within subjects (i.e. a longitudinal design). As such, the observed age group differences in performance could have been, in part, attributed to cohort effects (e.g. young adults being more comfortable with technology and computer games compared to old adults). Despite this theoretical limitation, several aspects of our data suggest that the differences we observed are related to age rather than other factors that differed across our young and older cohorts. First, we collected RBANS scores, which revealed that our two groups reflected similar samples of their age groups with respect to overall cognitive function. Second, task performance decreased with age within the older adult group rather than simply across the two groups.

To analyze our results, we developed a Bayesian hierarchical RL-WM model that captured qualitative trends in both training and testing performance as well as group differences. However, our model validation revealed that the RL-WM model predictions deviated from older participants’ data in set size 6 learning (Figure 2B). Further inspection revealed that this occurred because in some set size 6 blocks some older participants learned a correct response for only a subset of stimuli, while entirely neglecting to learn correct responses for the remaining stimuli in the set. One interpretation is that older adults might have used this as a strategy to minimize the difficulty of the high set size condition (ignoring some stimuli to effectively make it a smaller set size), leading participants to deviate from our model, which learned from all stimuli. This interpretation was supported by model predictions for older adults who did not exhibit such selective learning, which displayed reduced model misfit (see Appendix 1—figure 4). Thus, our hierarchical modeling not only captured core trends in behavior, but it also highlighted a previously undocumented discrepancy between RL-WM model predictions and learning behavior in specific individuals due to alternative strategies (e.g. neglecting stimuli). While we were unable to further characterize or quantify these strategies in our dataset, we hope that this observation inspires future work to design tasks better positioned to do so.

We have focused on testing healthy older adults in order to isolate the aging as dimension along which we tested the differences in RL and WM. In the future, it would also be valuable to examine these changes in clinical populations (e.g. individuals with mild cognitive impairment or individuals with Alzheimer’s disease/dementia). Testing how neural mechanisms in these neurodegenerative disorders relate to RL/WM would further expand our understanding of the underpinnings of dementia-related symptoms. Furthermore, recognizing the signs of specific impairments at different stage of dementia development could inform cognitive-training programs that have been used to mitigate/delay cognitive symptoms in these neurodegenerative disorders (Clare et al., 2003). One interesting related question is that of the dynamics of age-related differences, in particular as to whether the measured age-related neural/cognitive changes are gradual or characterized by a sudden onset at a critical time point in lifespan? Based on the absence of age-related correlation with performance in young adults and negative correlation with age within the old group, it seems that the relationship between age and cognitive functions might not be linear and instead begin to decline more rapidly at older ages. Future work employing longitudinal studies could address this question and potentially link nonlinear declines in neural and behavioral measures to protracted onset of age-related disease.

In summary, we show that age-related differences in the ability to make decisions from experience are reflected in the age group variability in working memory decay. These behavioral deficits are accompanied by changes in neurochemistry, most notably reductions in prefrontal glutamate levels, that map specifically onto computational markers for working memory impairment and tend to increase with age. These findings suggest that a major component of age-related differences in decision-making could result from old adults experiencing reductions in prefrontal excitatory signaling that impair working memory systems necessary to translate recent information into appropriate action.

All data and code has been made available on OSF.

1. 1. Milena R
   2. Nassar M

   (2023) Open Science Framework

   Age-related differences in prefrontal glutamate are associated with increased working memory decay that gives appearance of learning deficits.

   https://doi.org/10.17605/OSF.IO/2U7PM

1. 1. Amer T
   2. Wynn JS
   3. Hasher L

   (2022) Cluttered memory representations shape cognition in old age

   Trends in Cognitive Sciences 26:255–267.

   https://doi.org/10.1016/j.tics.2021.12.002

   • Google Scholar
2. Conference

   1. Andersen RA
   2. Buneo CA

   (2002)

   The correlative triad among aging, dopamine, and cognition: Current status and future prospects

   Annual Review of Neuroscience. pp. 189–220.

   • Google Scholar
3. 1. Bäckman L
   2. Nyberg L
   3. Lindenberger U
   4. Li S-C
   5. Farde L

   (2006) The correlative triad among aging, dopamine, and cognition: current status and future prospects

   Neuroscience & Biobehavioral Reviews 30:791–807.

   https://doi.org/10.1016/j.neubiorev.2006.06.005

   • Google Scholar
4. Preprint

   1. Baribault B
   2. Collins A

   (2023) Troubleshooting Bayesian Cognitive Models

   PsyArXiv.

   https://doi.org/10.31234/osf.io/rtgew

   • Google Scholar
5. Preprint

   1. Betancourt M

   (2016) Diagnosing Suboptimal Cotangent Disintegrations in Hamiltonian Monte Carlo

   arXiv.

   https://arxiv.org/abs/1604.00695

   • Google Scholar
6. 1. Bornstein AM
   2. Norman KA

   (2017) Reinstated episodic context guides sampling-based decisions for reward

   Nature Neuroscience 20:997–1003.

   https://doi.org/10.1038/nn.4573

   • PubMed
   • Google Scholar
7. 1. Buckner RL
   2. Head D
   3. Parker J
   4. Fotenos AF
   5. Marcus D
   6. Morris JC
   7. Snyder AZ

   (2004) A unified approach for morphometric and functional data analysis in young, old, and demented adults using automated atlas-based head size normalization: reliability and validation against manual measurement of total intracranial volume

   NeuroImage 23:724–738.

   https://doi.org/10.1016/j.neuroimage.2004.06.018

   • Google Scholar
8. 1. Carpenter B
   2. Gelman A
   3. Hoffman MD
   4. Lee D
   5. Goodrich B
   6. Betancourt M
   7. Brubaker MA
   8. Guo J
   9. Li P
   10. Riddell A

   (2017) Stan: a probabilistic programming language

   Journal of Statistical Software 76:1.

   https://doi.org/10.18637/jss.v076.i01

   • PubMed
   • Google Scholar
9. 1. Cattell RB

   (1943) The measurement of adult intelligence

   Psychological Bulletin 40:153–193.

   https://doi.org/10.1037/h0059973

   • Google Scholar
10. 1. Chowdhury R
    2. Guitart-Masip M
    3. Lambert C
    4. Dayan P
    5. Huys Q
    6. Düzel E
    7. Dolan RJ

    (2013) Dopamine restores reward prediction errors in old age

    Nature Neuroscience 16:648–653.

    https://doi.org/10.1038/nn.3364

    • PubMed
    • Google Scholar
11. 1. Clare L
    2. Woods RT
    3. Moniz Cook ED
    4. Orrell M
    5. Spector A

    (2003) Cognitive rehabilitation and cognitive training for early-stage Alzheimer’s disease and vascular dementia

    The Cochrane Database of Systematic Reviews 1:CD003260.

    https://doi.org/10.1002/14651858.CD003260

    • PubMed
    • Google Scholar
12. 1. Collins AGE
    2. Frank MJ

    (2012) How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis

    The European Journal of Neuroscience 35:1024–1035.

    https://doi.org/10.1111/j.1460-9568.2011.07980.x

    • PubMed
    • Google Scholar
13. 1. Collins AGE
    2. Brown JK
    3. Gold JM
    4. Waltz JA
    5. Frank MJ

    (2014) Working memory contributions to reinforcement learning impairments in schizophrenia

    The Journal of Neuroscience 34:13747–13756.

    https://doi.org/10.1523/JNEUROSCI.0989-14.2014

    • Google Scholar
14. 1. Collins AGE
    2. Frank MJ

    (2014) Opponent actor learning (opal): Modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive

    Psychological Review 121:337–366.

    https://doi.org/10.1037/a0037015

    • PubMed
    • Google Scholar
15. 1. Collins AGE

    (2018) The tortoise and the hare: Interactions between reinforcement learning and working memory

    Journal of Cognitive Neuroscience 30:1422–1432.

    https://doi.org/10.1162/jocn_a_01238

    • PubMed
    • Google Scholar
16. 1. Collins AGE
    2. Frank MJ

    (2018) Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory

    PNAS 115:2502–2507.

    https://doi.org/10.1073/pnas.1720963115

    • Google Scholar
17. 1. Desikan RS
    2. Ségonne F
    3. Fischl B
    4. Quinn BT
    5. Dickerson BC
    6. Blacker D
    7. Buckner RL
    8. Dale AM
    9. Maguire RP
    10. Hyman BT
    11. Albert MS
    12. Killiany RJ

    (2006) An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest

    NeuroImage 31:968–980.

    https://doi.org/10.1016/j.neuroimage.2006.01.021

    • PubMed
    • Google Scholar
18. 1. Dumitriu D
    2. Hao J
    3. Hara Y
    4. Kaufmann J
    5. Janssen WGM
    6. Lou W
    7. Rapp PR
    8. Morrison JH

    (2010) Selective changes in thin spine density and morphology in monkey prefrontal cortex correlate with aging-related cognitive impairment

    The Journal of Neuroscience 30:7507–7515.

    https://doi.org/10.1523/JNEUROSCI.6410-09.2010

    • PubMed
    • Google Scholar
19. 1. Durstewitz D
    2. Seamans JK
    3. Sejnowski TJ

    (2000) Dopamine-Mediated stabilization of delay-period activity in a network model of prefrontal cortex

    Journal of Neurophysiology 83:1733–1750.

    https://doi.org/10.1152/jn.2000.83.3.1733

    • PubMed
    • Google Scholar
20. 1. Eckstein MK
    2. Master SL
    3. Xia L
    4. Dahl RE
    5. Wilbrecht L
    6. Collins AGE

    (2022) The interpretation of computational model parameters depends on the context

    eLife 11:e75474.

    https://doi.org/10.7554/eLife.75474

    • PubMed
    • Google Scholar
21. 1. Edden RAE
    2. Puts NAJ
    3. Harris AD
    4. Barker PB
    5. Evans CJ

    (2014) Gannet: a batch-processing tool for the quantitative analysis of gamma-aminobutyric acid–edited MR spectroscopy spectra

    Journal of Magnetic Resonance Imaging 40:1445–1452.

    https://doi.org/10.1002/jmri.24478

    • PubMed
    • Google Scholar
22. 1. Eppinger B
    2. Schuck NW
    3. Nystrom LE
    4. Cohen JD

    (2013) Reduced striatal responses to reward prediction errors in older compared with younger adults

    The Journal of Neuroscience 33:9905–9912.

    https://doi.org/10.1523/JNEUROSCI.2942-12.2013

    • PubMed
    • Google Scholar
23. 1. Fischl B
    2. Salat DH
    3. Busa E
    4. Albert M
    5. Dieterich M
    6. Haselgrove C
    7. van der Kouwe A
    8. Killiany R
    9. Kennedy D
    10. Klaveness S
    11. Montillo A
    12. Makris N
    13. Rosen B
    14. Dale AM

    (2002) Whole brain segmentation: Automated labeling of neuroanatomical structures in the human brain

    Neuron 33:341–355.

    https://doi.org/10.1016/s0896-6273(02)00569-x

    • PubMed
    • Google Scholar
24. 1. Frank MJ
    2. Seeberger LC
    3. O’reilly RC

    (2004) By carrot or by stick: Cognitive reinforcement learning in parkinsonism

    Science 306:1940–1943.

    https://doi.org/10.1126/science.1102941

    • PubMed
    • Google Scholar
25. 1. Frank MJ
    2. Moustafa AA
    3. Haughey HM
    4. Curran T
    5. Hutchison KE

    (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning

    PNAS 104:16311–16316.

    https://doi.org/10.1073/pnas.0706111104

    • Google Scholar
26. 1. Frank MJ
    2. Kong L

    (2008) Learning to avoid in older age

    Psychology and Aging 23:392–398.

    https://doi.org/10.1037/0882-7974.23.2.392

    • PubMed
    • Google Scholar
27. 1. Fritz MS
    2. Mackinnon DP

    (2007) Required sample size to detect the mediated effect

    Psychological Science 18:233–239.

    https://doi.org/10.1111/j.1467-9280.2007.01882.x

    • PubMed
    • Google Scholar
28. Book

    1. Gelman A
    2. Carlin JB
    3. Stern HS
    4. Dunson DB
    5. Vehtari A
    6. Rubin DB

    (2013) Bayesian Data Analysis

    Taylor Francis.

    https://doi.org/10.1201/b16018

    • Google Scholar
29. 1. Gershman SJ

    (2015) Do learning rates adapt to the distribution of rewards?

    Psychonomic Bulletin & Review 22:1320–1327.

    https://doi.org/10.3758/s13423-014-0790-3

    • Google Scholar
30. 1. Goldman-Rakic PS

    (1995) Cellular basis of working memory

    Neuron 14:477–485.

    https://doi.org/10.1016/0896-6273(95)90304-6

    • PubMed
    • Google Scholar
31. 1. Grogan JP
    2. Isotalus HK
    3. Howat A
    4. Irigoras Izagirre N
    5. Knight LE
    6. Coulthard EJ

    (2019) Levodopa does not affect expression of reinforcement learning in older adults

    Scientific Reports 9:6349.

    https://doi.org/10.1038/s41598-019-42904-5

    • PubMed
    • Google Scholar
32. 1. Hämmerer D
    2. Li S-C
    3. Müller V
    4. Lindenberger U

    (2011) Life span differences in electrophysiological Corre-Lates of monitoring gains and losses during probabilistic reinforcement learning

    Journal of Cognitive Neuroscience 23:579–592.

    https://doi.org/10.1162/jocn.2010.21475

    • PubMed
    • Google Scholar
33. 1. Hancu I

    (2009) Optimized glutamate detection at 3T

    Journal of Magnetic Resonance Imaging 30:1155–1162.

    https://doi.org/10.1002/jmri.21936

    • PubMed
    • Google Scholar
34. 1. Hu Y
    2. Chen X
    3. Gu H
    4. Yang Y

    (2013) Resting-State glutamate and GABA concentrations predict task-induced deactivation in the default mode network

    The Journal of Neuroscience 33:18566–18573.

    https://doi.org/10.1523/JNEUROSCI.1973-13.2013

    • PubMed
    • Google Scholar
35. 1. Kennedy L
    2. Simpson D
    3. Gelman A

    (2019) The experiment is just as important as the likelihood in understanding the prior: a cautionary note on robust cognitive modeling

    Computational Brain & Behavior 2:210–217.

    https://doi.org/10.1007/s42113-019-00051-0

    • Google Scholar
36. 1. Kruschke JK
    2. Liddell TM

    (2018) The Bayesian new statistics: hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective

    Psychonomic Bulletin & Review 25:178–206.

    https://doi.org/10.3758/s13423-016-1221-4

    • Google Scholar
37. 1. Lee MD

    (2011) How cognitive modeling can benefit from hierarchical Bayesian models

    Journal of Mathematical Psychology 55:1–7.

    https://doi.org/10.1016/j.jmp.2010.08.013

    • Google Scholar
38. 1. Leong YC
    2. Radulescu A
    3. Daniel R
    4. DeWoskin V
    5. Niv Y

    (2017) Dynamic interaction between reinforcement learning and attention in multidimensional environments

    Neuron 93:451–463.

    https://doi.org/10.1016/j.neuron.2016.12.040

    • PubMed
    • Google Scholar
39. 1. Li SC
    2. Lindenberger U
    3. Sikström S

    (2001) Aging cognition: from neuromodulation to representation

    Trends in Cognitive Sciences 5:479–486.

    https://doi.org/10.1016/s1364-6613(00)01769-1

    • PubMed
    • Google Scholar
40. 1. Li Y
    2. Gao J
    3. Enkavi AZ
    4. Zaval L
    5. Weber EU
    6. Johnson EJ

    (2015) Sound credit scores and financial decisions despite cognitive aging

    PNAS 112:65–69.

    https://doi.org/10.1073/pnas.1413570112

    • Google Scholar
41. 1. Master SL
    2. Eckstein MK
    3. Gotlieb N
    4. Dahl R
    5. Wilbrecht L
    6. Collins AGE

    (2020) Disentangling the systems contributing to changes in learning during adolescence

    Developmental Cognitive Neuroscience 41:100732.

    https://doi.org/10.1016/j.dcn.2019.100732

    • Google Scholar
42. 1. Nassar MR
    2. Frank MJ

    (2016) Taming the beast: extracting generalizable knowledge from computational models of cognition

    Current Opinion in Behavioral Sciences 11:49–54.

    https://doi.org/10.1016/j.cobeha.2016.04.003

    • PubMed
    • Google Scholar
43. 1. Niv Y
    2. Edlund JA
    3. Dayan P
    4. O’Doherty JP

    (2012) Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain

    The Journal of Neuroscience 32:551–562.

    https://doi.org/10.1523/JNEUROSCI.5498-10.2012

    • PubMed
    • Google Scholar
44. 1. O’Reilly RC
    2. Frank MJ

    (2006) Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia

    Neural Computation 18:283–328.

    https://doi.org/10.1162/089976606775093909

    • PubMed
    • Google Scholar
45. 1. Provencher SW

    (2001) Automatic quantitation of localized in vivo 1H spectra with lcmodel

    NMR in Biomedicine 14:260–264.

    https://doi.org/10.1002/nbm.698

    • PubMed
    • Google Scholar
46. 1. Radulescu A
    2. Daniel R
    3. Niv Y

    (2016) The effects of aging on the interaction between reinforcement learning and attention

    Psychology and Aging 31:747–757.

    https://doi.org/10.1037/pag0000112

    • PubMed
    • Google Scholar
47. 1. Radulescu A
    2. Niv Y
    3. Ballard I

    (2019) Holistic reinforcement learning: the role of structure and attention

    Trends in Cognitive Sciences 23:278–292.

    https://doi.org/10.1016/j.tics.2019.01.010

    • Google Scholar
48. 1. Randolph C
    2. Tierney MC
    3. Mohr E
    4. Chase TN

    (1998) The repeatable battery for the assessment of neuropsychological status (rbans): preliminary clinical validity

    Journal of Clinical and Experimental Neuropsychology 20:310–319.

    https://doi.org/10.1076/jcen.20.3.310.823

    • PubMed
    • Google Scholar
49. 1. Salthouse TA
    2. Babcock RL

    (1991) Decomposing adult age differences in working memory

    Developmental Psychology 27:763–776.

    https://doi.org/10.1037/0012-1649.27.5.763

    • Google Scholar
50. 1. Samanez-Larkin GR
    2. Worthy DA
    3. Mata R
    4. McClure SM
    5. Knutson B

    (2014) Adult age differences in frontostriatal representation of prediction error but not reward outcome

    Cognitive, Affective, & Behavioral Neuroscience 14:672–682.

    https://doi.org/10.3758/s13415-014-0297-4

    • Google Scholar
51. 1. Schultz W
    2. Dayan P
    3. Montague PR

    (1997) A neural substrate of prediction and reward

    Science 275:1593–1599.

    https://doi.org/10.1126/science.275.5306.1593

    • PubMed
    • Google Scholar
52. 1. Shibata K
    2. Sasaki Y
    3. Bang JW
    4. Walsh EG
    5. Machizawa MG
    6. Tamaki M
    7. Chang LH
    8. Watanabe T

    (2017) Overlearning hyperstabilizes a skill by rapidly making neurochemical processing inhibitory-dominant

    Nature Neuroscience 20:470–475.

    https://doi.org/10.1038/nn.4490

    • Google Scholar
53. Book

    1. Sutton RS
    2. Barto AG

    (2018)

    Reinforcement Learning: An Introduction

    MIT press.

    • Google Scholar
54. 1. Tkác I
    2. Starcuk Z
    3. Choi IY
    4. Gruetter R

    (1999) In vivo 1H NMR spectroscopy of rat brain at 1 MS echo time

    Magnetic Resonance in Medicine 41:649–656.

    https://doi.org/10.1002/(sici)1522-2594(199904)41:4<649::aid-mrm2>3.0.co;2-g

    • PubMed
    • Google Scholar
55. 1. van de Vijver I
    2. Ligneul R

    (2020) Relevance of working memory for reinforcement learning in older adults varies with timescale of learning

    Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition 27:654–676.

    https://doi.org/10.1080/13825585.2019.1664389

    • PubMed
    • Google Scholar
56. 1. van Vugt B
    2. van Kerkoerle T
    3. Vartak D
    4. Roelfsema PR

    (2020) The contribution of AMPA and NMDA receptors to persistent firing in the dorsolateral prefrontal cortex in working memory

    The Journal of Neuroscience 40:2458–2470.

    https://doi.org/10.1523/JNEUROSCI.2121-19.2020

    • PubMed
    • Google Scholar
57. 1. Vehtari A
    2. Gelman A
    3. Simpson D
    4. Carpenter B
    5. Bürkner P-C

    (2021) Rank-normalization, folding, and localization: an improved rˆ for assessing convergence of MCMC (with discussion)

    Bayesian Analysis 16:667–718.

    https://doi.org/10.1214/20-BA1221

    • Google Scholar
58. 1. Wang M
    2. Gamo NJ
    3. Yang Y
    4. Jin LE
    5. Wang X-J
    6. Laubach M
    7. Mazer JA
    8. Lee D
    9. Arnsten AFT

    (2011) Neuronal basis of age-related working memory decline

    Nature 476:210–213.

    https://doi.org/10.1038/nature10243

    • Google Scholar
59. 1. Watanabe S

    (2010)

    Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory

    Journal of Machine Learning Research 11:12.

    • Google Scholar
60. 1. West RL

    (1996) An application of prefrontal cortex function theory to cognitive aging

    Psychological Bulletin 120:272–292.

    https://doi.org/10.1037/0033-2909.120.2.272

    • PubMed
    • Google Scholar
61. 1. Westfall J
    2. Yarkoni T

    (2016) Statistically controlling for confounding constructs is harder than you think

    PLOS ONE 11:e0152719.

    https://doi.org/10.1371/journal.pone.0152719

    • PubMed
    • Google Scholar
62. 1. Wong KF
    2. Wang XJ

    (2006) A recurrent network mechanism of time integration in perceptual decisions

    The Journal of Neuroscience 26:1314–1328.

    https://doi.org/10.1523/JNEUROSCI.3733-05.2006

    • PubMed
    • Google Scholar
63. 1. Yoo AH
    2. Collins AGE

    (2022) How working memory and reinforcement learning are intertwined: a cognitive, neural, and computational perspective

    Journal of Cognitive Neuroscience 34:551–568.

    https://doi.org/10.1162/jocn_a_01808

    • PubMed
    • Google Scholar

Author details

1. Milena Rmus

   University of California, Berkeley, Berkeley, United States

   Contribution

   Data curation, Formal analysis, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   milena_rmus@berkeley.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-2044-048X

2. Mingjian He

   Massachusetts Institute of Technology, Boston, United States

   Contribution

   Data curation, Formal analysis, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-6688-8693

3. Beth Baribault

   University of California, Berkeley, Berkeley, United States

   Contribution

   Formal analysis, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   Competing interests

   No competing interests declared

4. Edward G Walsh

   Brown University, Providence, United States

   Contribution

   Investigation, Methodology, Project administration, Writing - review and editing

   Competing interests

   No competing interests declared

5. Elena K Festa

   Brown University, Providence, United States

   Contribution

   Supervision, Investigation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-3700-4270

6. Anne GE Collins

   University of California, Berkeley, Berkeley, United States

   Contribution

   Supervision, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3751-3662

7. Matthew R Nassar

   Brown University, Providence, United States

   Contribution

   Conceptualization, Supervision, Funding acquisition, Investigation, Methodology, Writing - original draft, Project administration, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5397-535X

• 1,338

  views

• 222

  downloads

• 3

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.