1. Neuroscience
Download icon

Associability-modulated loss learning is increased in posttraumatic stress disorder

  1. Vanessa M Brown
  2. Lusha Zhu
  3. John M Wang
  4. B Christopher Frueh
  5. Brooks King-Casas  Is a corresponding author
  6. Pearl H Chiu  Is a corresponding author
  1. Virginia Tech Carilion Research Institute, United States
  2. Virginia Tech, United States
  3. IDG/McGovern Institute for Brain Research, Beijing Key Laboratory of Behavior and Mental Health, Peking–Tsinghua Center for Life Sciences, Peking University, China
  4. University of Hawaii, United States
  5. Salem Veterans Affairs Medical Center, United States
  6. Virginia Tech-Wake Forest University, United States
Research Article
  • Cited 3
  • Views 1,244
  • Annotations
Cite this article as: eLife 2018;7:e30150 doi: 10.7554/eLife.30150

Abstract

Disproportionate reactions to unexpected stimuli in the environment are a cardinal symptom of posttraumatic stress disorder (PTSD). Here, we test whether these heightened responses are associated with disruptions in distinct components of reinforcement learning. Specifically, using functional neuroimaging, a loss-learning task, and a computational model-based approach, we assessed the mechanistic hypothesis that overreactions to stimuli in PTSD arise from anomalous gating of attention during learning (i.e., associability). Behavioral choices of combat-deployed veterans with and without PTSD were fit to a reinforcement learning model, generating trial-by-trial prediction errors (signaling unexpected outcomes) and associability values (signaling attention allocation to the unexpected outcomes). Neural substrates of associability value and behavioral parameter estimates of associability updating, but not prediction error, increased with PTSD during loss learning. Moreover, the interaction of PTSD severity with neural markers of associability value predicted behavioral choices. These results indicate that increased attention-based learning may underlie aspects of PTSD and suggest potential neuromechanistic treatment targets.

https://doi.org/10.7554/eLife.30150.001

eLife digest

Posttraumatic stress disorder, or PTSD for short, is a serious psychiatric disorder that sometimes occurs after someone has experienced a dangerous or threatening event. People with PTSD are prone to overreact to unexpected reminders of these events, and are often hypervigilant for danger. Why these symptoms occur is not yet clear, but it is thought that people with PTSD may have learning problems that lead them to overestimate the likelihood of danger.

Advanced tools from computer science and mathematics have helped scientists to study how the brain learns. These tools may now provide more insight into how diseases like PTSD disrupt learning. Scientists use computer models of learning to test how humans make choices and react to their outcomes. These models build on the idea that humans make choices based on what they predict an outcome will be, and then learn when they update their expectations based on the accuracy of their predictions.

Now, Brown et al. show that people with PTSD have an increased learning response to surprising events – these are defined in this study as outcomes that are inconsistent with participants’ predictions. In the experiments, 74 combat veterans who had experienced trauma in Iraq or Afghanistan underwent a type of brain scanning procedure, while they played a gambling-like game. Some participants had PTSD, others did not.

Both groups learned to make choices that minimized the loss of money. However, learning in veterans with PTSD was strongly influenced by how much attention they paid to surprising outcomes. Moreover, the brain areas that help to process attention to surprise were highly active in people with PTSD. Brown et al. added a third group of participants with depression to the study to verify that the learning changes were PTSD-specific. This depression-only group did not have differences in attention to surprise.

Many treatments for PTSD focus on exposing individuals to feared situations and trauma memories, so that individuals can learn that these situations are no longer dangerous. Computational modeling and neuroimaging may help scientists pinpoint the sources of learning deficits, such as increased attention to surprising outcomes. Identifying the different possible causes of learning problems may lead to new or more precise learning-based treatments for PTSD and other learning-related conditions. Understanding how learning-related brain changes develop may also help find ways to prevent and better diagnose PTSD and other psychiatric disorders.

https://doi.org/10.7554/eLife.30150.002

Introduction

Posttraumatic stress disorder (PTSD) is debilitating and characterized by excessive behavioral, psychological, and physiological responses to unexpected stimuli (Pitman et al., 2012). In particular, clinical and empirical observations have documented the negative impact of salient cues on neural and behavioral functioning in PTSD, including heightened orienting to unexpected events, impaired extinction of learned fear, and unstable attention biases toward perceived threatening stimuli (Aupperle et al., 2012; Bar-Haim et al., 2007; Blair et al., 2013; Morey et al., 2009; Naim et al., 2015). Together, these behavioral alterations in response to unexpected stimuli, uncontrollable reminders of trauma, and other negative, threatening, and trauma-related events point to PTSD as a disorder of disrupted learning from reminders of negative events; however, the specific components of anomalous learning in PTSD remain unknown. As an initial step toward addressing this issue, we adopt a computational psychiatry approach (Montague et al., 2012; Wang and Krystal, 2014; Maia and Frank, 2011), using quantitative specification of neural and behavioral learning processes to investigate the neurocomputational substrates of PTSD.

Computational model-based approaches to learning provide a mechanistic framework for understanding the detrimental impact of unexpected negative stimuli and reminders of negative events in PTSD. Error-guided models of reinforcement learning (RL) have robustly shown that unexpected outcomes (i.e., value ‘prediction errors’) drive learning by directly updating the value of the cues associated with those outcomes (Rescorla and Wagner, 1972; Sutton and Barto, 1998). A related family of hybrid reinforcement learning models combines prediction-error based learning with a dynamically changing attention modulation variable (i.e., a cue’s associability value) that scales with the magnitude of prediction errors previously associated with a particular cue. In these models, trial-by-trial associability values associated with particular cues gate the learning of subsequent outcomes associated with these cues (Li et al., 2011; Pearce and Hall, 1980; Le Pelley, 2004). Thus, these models contain separate parameters for error-based learning rate and associability updating that together govern how strongly current and past prediction errors, respectively, affect learning. While commonly used tasks that assess cue-salience (Todd et al., 2015) and attention to threat (Naim et al., 2015; Vythilingam et al., 2007) test important components of aversive processing in PTSD, they are more limited for explaining how changes in attention to negative stimuli may affect subsequent behavioral choices, as reinforcement learning algorithms allow in the context of value-based learning tasks. More generally, computational model-based approaches allow fitting of models of neural function to behavioral choices and imaging data and thus facilitate the separation of mechanistic processes (e.g., responses to associability value, associability updating, learning rate, prediction error responses) related to attention and learning from negative and positive events. In the hybrid RL framework, PTSD, and symptoms of hypervigilance in particular, may reflect disproportionate attentional processes that drive maladaptive, heightened responses to stimuli with a history of unexpected outcomes.

The neural substrates of reinforcement learning suggest further compelling links between associability-modulated learning and PTSD, as the brain networks involved in both overlap. In particular, work in humans and rodents has identified roles for the ventral striatum, anterior cingulate, and amygdala in encoding prediction error (Rangel et al., 2008; Pagnoni et al., 2002; Garrison et al., 2013) and for the amygdala and insula in encoding associability values (Li et al., 2011; Roesch et al., 2012). In PTSD, affective stimuli consistently elicit altered neural activation in a network that prominently also includes the amygdala, insula, and prefrontal regions (Hayes et al., 2012; Etkin and Wager, 2007). Of translational relevance, attention, learning, and the amygdala have all been behavioral and neural targets of promising new therapies for PTSD (Badura-Brack et al., 2015; Craske et al., 2014; Langevin et al., 2016); elucidating the component neurobehavioral mechanisms associated with learning in PTSD may refine these targets.

In brief, extant data indicate that neural and behavioral distinctions between solely prediction error-based and associability-modulated learning contribute uniquely to neural and behavioral correlates of learning and may clarify processes underlying the heightened sensitivity to unexpected stimuli in PTSD. To assess this possibility, we implemented a probabilistic learning task during functional magnetic resonance imaging (fMRI) in combat-deployed military veterans with and without posttraumatic stress disorder. We posited that if attention-modulated learning plays a role in PTSD, hybrid RL models incorporating associability should predict participants’ choices better than learning models without associability. Furthermore, PTSD severity, particularly symptoms related to hyperarousal (Lissek and van Meurs, 2015), should correlate preferentially both with enhanced associability updating and with increased activity in neural structures encoding associability values (i.e., amygdala and insula). Critically, these relationships would not be expected between PTSD and solely error-based learning rate or error-related activity in neural structures encoding value and prediction error (i.e., ventral striatum, ventromedial prefrontal cortex).

Results

Participant characteristics and model-agnostic behavioral performance

Combat-deployed military veterans (N = 74) completed a probabilistic learning task in the loss and gain domains while undergoing fMRI scanning (Figure 1; full task description in Materials and methods). All veterans had served at least one tour in Iraq or Afghanistan since 2001, had experienced Criterion A deployment-related trauma, and were recruited from a larger study by our group examining biomarkers of mood and anxiety disorders. To be considered in the present analyses, participants were further required to demonstrate behavioral engagement on the relevant portions of the probabilistic learning task (N = 68 veterans; see Materials and methods for exclusion breakdowns, full inclusion/exclusion criteria, and exclusion details for sub-analyses). Veterans were assessed for PTSD using the Clinician Administered PTSD Scale (CAPS [Blake et al., 1995]) and for other psychiatric disorders with the Structured Clinical Interview for DSM-IV (SCID [First et al., 1996]). Participants exhibited a range of PTSD symptoms, with 39 veterans meeting DSM-IV (American Psychiatric Association, 2000) criteria for PTSD (see Supplementary file 1, Table 1A for clinical and demographic information); the other 29 veterans had previous deployment-related trauma exposure as assessed by the CAPS interview but did not meet criteria for PTSD, resulting in 39 participants with PTSD and 29 participants without PTSD for primary analyses.

Reinforcement learning model.

Schematic description of reinforcement learning model. Participants choose between two stimuli, view the monetary outcome of the choice, and learn over time which is the ‘better’ option. A reinforcement learning model incorporating associability-modulated learning rate is illustrated here. Expected value is updated based on the static learning rate, the current trial’s prediction error, and the associability value from the previous outcome associated with the stimulus. Meanwhile, associability value is updated based on the static associability weight and the absolute value of the current trial’s prediction error. Green lines indicate the effect of prediction error through learning rate on expected value, while purple lines indicate the effect of prediction error through associability weight on associability value on the current trial, and then on learning rate on the subsequent trial. Thus, through trial-by-trial modulation of learning rates, associability values act as an attentional gate on learning. The task involved blocks consisting of all loss or all gain trials; loss trials are shown here as the model with associability fit well in this condition only.

https://doi.org/10.7554/eLife.30150.003

In both loss and gain domains, veterans showed robust learning, as evidenced by increasing likelihood of choosing the ‘better’ option from chance on the first trial to near 80% correct after ~15 trials (Figure 2a and Figure 2—figure supplement 1). An adaptive design titrated the task for participants to achieve sufficient learning (i.e., block length was adjusted once participants achieved learning; see Materials and methods for details); reflecting this, performance accuracy (% better choice) did not differ between participants with and without PTSD in the gain or loss domains (gain: t41 = −1.29, p>0.1; loss: t41 = 0.459, p>0.1).

Figure 2 with 1 supplement see all
Behavioral performance and relationship to parameter estimates.

(a) Loss performance. Performance was quantified as proportion of choices that were the ‘better’ option. Over time, participants show learning (running average over five trials, averaged over all blocks; mean ± SE). Behavior is separated by diagnostic group, with veteran control (No PTSD) participants’ behavior marked by a solid line and the behavior of veterans with PTSD marked by a dotted line. (b) Plot of trial-by-trial associability value and prediction error for loss learning blocks, averaged across sets of five trials and across participants. Values are derived from the associability reinforcement learning model described in Figure 1 using each participant’s individually estimated parameter values and behavioral choices. As expected, a gradual reduction in average associability value and reduction of prediction error towards zero across trials is observed. As the initial expected value of each stimulus was set at 0, prediction errors are initially negative. During learning, prediction errors become distributed around, and converge toward, 0, indicating learning. Similarly, the initial associability value is set at one and decreases as outcomes become better predicted. Note that the probabilistic outcomes of this task ensure that outcomes are not completely predicted; this feature maintains variation in prediction error and retains the influence of associability value. As the task progresses, new blocks requiring new learning are more likely to occur, leading to an overall increase in associability value and prediction errors further from 0. Gain learning blocks did not fit the associability model well; figures related to gain learning are shown in Figure 2—figure supplement 1.

https://doi.org/10.7554/eLife.30150.004

Behavioral model-fitting and relationship of model parameters with PTSD

As an initial step toward evaluating the role of associability in learning in PTSD, participants’ choices were fit to a prediction error-based reinforcement learning (RL) model with and without a dynamic associability value (κ)-modulated learning rate on the prediction error (δ), as in Li et al. (2011). In this model, the associability value κ of a chosen stimulus changes on a trial-by-trial basis based on a combination of the magnitude of previous prediction errors and a static associability weight η, a parameter which varies by participant and indicates the extent to which the magnitude of recent prediction errors updates trial-by-trial associability values (see Materials and methods for full model specifications and Figure 2b for a trial-by-trial plot of the time course and relationship between prediction error and associability values). We verified via simulation that associability weight (η) did not directly affect performance (Figure 3—figure supplement 1a), allowing us to dissociate the effects of associability updating from general performance deficits. Consistent with our hypothesis, including associability in the RL model significantly improved model fit for the majority of participants and did so during loss learning only (Figure 3a; protected exceedance probability of model with versus without associability: 0% in gain, 100% in loss). The role of associability in loss, but not gain, learning is consistent with prior data showing heightened orienting and attentional biases toward negative information in PTSD (Li et al., 2011; Boll et al., 2013); thus, our subsequent analyses focused on learning variables in the loss domain (see Appendix 2, Supplementary Results for supplemental data related to gain learning and model-agnostic support for the presence of associability-modulated updating during loss learning only).

Figure 3 with 1 supplement see all
Model fit and relationship to behavior.

(a) Protected exceedance probability and model frequency calculated by Bayesian Model Selection for the model with versus without associability for loss and gain trials, showing an improvement in model fit when adding associability during loss learning only. (b) Average regression beta values per subject between trial-by-trial associability value (κ) and reaction time (controlling for expected value and trial number) show a positive relationship between model-estimated associability value and choice latencies. Dots are individual subjects’ beta values. (c) Prediction of switching behavior is improved when previous trial’s prediction error (δ) is modulated by associability value (κ; negative change in AIC for logistic regression predicting switching; δ and δ * κ model, each compared against basic outcome-only model; see Supplementary Materials and methods). Error bars represent subject-level standard errors based on a leave-one-out standard error estimation. (d) Associability signaling independent of PTSD and covariates, displayed at p<0.05 FDR corrected. First level regression of trial-by-trial associability values on neural activity at time of outcome. Second level (shown) of constant term of regression. Additional model confirmation analyses (relationship between associability weight parameter and performance, relationship between actual and predicted choices, and model parameter recovery) are in Figure 3—figure supplement 1.

https://doi.org/10.7554/eLife.30150.006

The role of associability in loss learning was further corroborated by robust effects of associability values on behavior during the task. First, choices predicted by the associability RL model showed high correspondence with participants’ actual choices (correlation of bins of predicted vs. observed choices: r = 0.997, p<0.001; Figure 3—figure supplement 1b) and did not differ between participants with and without PTSD (t-test of subjects’ log likelihoods and PTSD diagnosis: t66 = −0.33, p>0.1). Reaction times were significantly and positively correlated with trial-by-trial associability values (κ), reflecting greater decision latency as the associability value of the chosen option increased (Figure 3b; average per-subject regression beta value of reaction time predicting associability, controlling for expected value of chosen option and trial number: .232; t-test assessing difference from 0: t68 = 8.28, p<0.001). Next, we regressed trial-by-trial estimates of prediction error (δ) and associability-modulated prediction error (κ*δ), respectively, computed from participants’ individually estimated parameter values, against their trial-by-trial switching behavior (switch or no switch, a measure of responsivity to outcomes). Associability-modulated prediction error significantly predicted switching choices above prediction error alone (χ21 = 323.0, p<0.001; Figure 3c). In addition, associability weight and other model parameters were recoverable through simulation (Figure 3—figure supplement 1c), and associability value showed a robust neural effect independent of PTSD diagnosis (Figure 3d; Supplementary file 1, Table 1B). These data together indicate that the associability RL model fit participants’ behavior and neural activity well and support the incorporation of associability in the RL model of loss learning.

If associability-modulated learning plays a role in PTSD, individuals with PTSD ought to have greater associability weights (η; see Materials and methods for model specifications). To test this possibility, participants’ individually estimated associability weights, reflecting the degree to which associability values are updated based on recent unsigned prediction errors during loss learning for each participant, and unmodulated learning rates (α) were compared between participants with and without a PTSD diagnosis (see Appendix 1: Supplementary Methods for individual parameter estimation details). Associability weights were increased in participants with PTSD (t62 = 4.01, p<0.001) while unmodulated learning rate did not differ between groups (t62 = 0.63, p>0.1; Figure 4a), supporting the hypothesis of a greater emphasis on modulation of loss learning by attention in PTSD.

Figure 4 with 1 supplement see all
Behavioral and neural substrates of associability are increased with PTSD.

(a) Loss associability weight increases with PTSD while unmodulated learning rate does not (individual estimates for learning parameters: circles indicate control participants [mean: solid line] and X’s indicate participants with PTSD [mean: dotted line]; insets display regression beta values for PTSD diagnosis variable from linear regression predicting learning parameters; error bars represent SEM). (b) Neural effect of PTSD diagnosis on trial-by-trial associability value activation (cluster-level FDR p<0.05 whole brain corrected with a cluster forming threshold of p<0.001; t value of PTSD diagnosis on parametric modulator of associability value at outcome event). Dashed circles indicate amygdala and insula regions. Prediction error ROIs from an independent cohort used to test for prediction error differences in PTSD are shown in Figure 4—figure supplement 1.

https://doi.org/10.7554/eLife.30150.008

Given the high co-occurrence of depression in PTSD (O'Donnell et al., 2004), we also tested the specificity of increased associability weight to PTSD versus depression. Specifically, we enrolled a separate cohort of gender-, estimated IQ-, and smoking status-matched participants with a current diagnosis of major depressive disorder (MDD; N = 20; see Materials and methods for MDD participant details), but not PTSD. These MDD-only participants performed the same learning task, and we also fit the behavior of these participants to the RL model with associability. Compared to MDD-only participants, the participants with PTSD had significantly higher associability weights (t57 = 7.25, p<0.001) but similar unmodulated learning rates (t57 = 1.44, p>0.1). Therefore, despite the high comorbidity between PTSD and depression (O'Donnell et al., 2004), increased associability-modulated learning appears specific to PTSD and not to mood-related psychopathology.

Neural substrates of associative learning in PTSD and relationship to behavioral choices

To investigate the instantiation of neural substrates of associability- and prediction error- based learning in PTSD, we first regressed trial-by-trial estimates of associability value (κ) and prediction error (δ), respectively, during loss learning against subjects’ neural responses to the outcome event (per [Li et al., 2011; Esber et al., 2012; Roesch et al., 2010a]; see Materials and methods for design matrix specifications). Corroborating the behavioral findings of greater associability value updating in participants with PTSD, neural encoding of associability values showed a significant relationship with PTSD at an FDR-corrected whole brain significance level in a network of regions including bilateral amygdala and insula, hypothesized areas of relevance for PTSD and associability-based learning (Pitman et al., 2012; Li et al., 2011) (Figure 4b; Supplementary file 1, Table 1C; see Figure 3d for associability-related signaling across participants after accounting for PTSD and covariates). To further test the localization of the increased associability-related activation in PTSD to our a priori hypothesized areas of amygdala and insula, we extracted beta values from these anatomical regions of interest (see Materials and methods for ROI definition). PTSD was significantly related to associability-related activation in these areas (amygdala: t37 = 2.45, p<0.05; insula: t37 = 3.87, p<0.001). Replacing the binary PTSD diagnosis with a dimensional measure of PTSD symptom severity (total CAPS score) across all veteran participants resulted in a similar pattern of effects (Figure 5a; Supplementary file 1, Table 1D). PTSD was not related to neural responses to prediction error at this whole brain level (Figure 5—figure supplement 1a). Follow up analyses covaried for presence of psychotropic medication, a positive screen for mild traumatic brain injury, or smoking status, and additionally tested effects within a subgroup of veterans with and without PTSD who were free from psychotropic medication and matched on estimated IQ; none of these covariates was significantly related to neural or behavioral results involving associability, and in the matched subgroup, PTSD remained related to behavioral and neural encoding of associability value (see Appendix 2, Supplemental Results for details).

Figure 5 with 2 supplements see all
PTSD symptom clusters of hyperarousal and avoidance/numbing have greatest neural and behavioral interaction with associability value.

(a) Relationship of total PTSD symptoms (CAPS score) and PTSD symptom clusters to neural encoding of loss associability value (FDR p<0.05; displayed at p<0.005 uncorrected to allow equivalent thresholding across images). (b) Switching behavior is explained by interaction of PTSD symptom clusters and neural associability value activation for total CAPS, avoidance/numbing and hyperarousal. Bars indicate improvement in model fit when adding an interaction term of symptom cluster severity to a mixed effects logistic regression model predicting switching behavior by an interaction of previous outcome with neural activity; likelihood ratio test of model fit improvement; χ2 >10.60 indicates significant improvement at p<0.005 uncorrected for multiple comparisons [αBonferroni =.006]). Relationship of total PTSD symptoms and symptom clusters with prediction error; associability value and prediction error analyses with vmPFC and insula ROIs; and jackknife error distributions are shown in Figure 5—figure supplement 1. Interaction of amygdala and insula ROIs with previous outcomes and PTSD is illustrated in Figure 5—figure supplement 2.

https://doi.org/10.7554/eLife.30150.010

The lack of whole-brain relationship between neural prediction error signals and PTSD could be due to our conservative multiple comparison correction. To assess this possibility, we further investigated the relationship between prediction error activation and PTSD within prediction error related regions of interest (ROIs) derived from a trauma-unexposed reference cohort, separate from the veteran cohort (described in Materials and methods). Neural responses in these ROIs (including ventral striatum and vmPFC) also did not show significant effects of PTSD for prediction error, even in neural regions strongly related to prediction error signaling (see Materials and methods for details; reference group prediction error activation is shown in Figure 4—figure supplement 1). Independent of PTSD, participants showed significant prediction error activation in striatum (left striatum: t42 = 3.56, p<0.001; right striatum: t42 = 3.02, p<.005), further supporting intact PE-related signal that is unaffected by PTSD.

To assess which PTSD symptom clusters are more associated with increased neural signaling of associability value, we examined the symptom clusters of re-experiencing, avoidance/numbing, and hyperarousal (American Psychiatric Association, 2000). Specifically, we tested the degree to which symptom severity in each cluster was related to neural correlates of trial-by-trial associability and prediction error, respectively. The hyperarousal and avoidance/numbing symptom clusters showed the most extensive neural responses corresponding with associability (Figure 5aSupplementary file 1,Tables 2E and 2F), with little relationship with re-experiencing symptoms (Figure 5a; Supplementary file 1, Table 2G).

Finally, if neural encoding of associability is relevant for real-world behavioral disruptions in PTSD, the combination of PTSD and neural responsivity to associability value ought to be related to participants’ likelihood of adjusting behavioral choices based on past experiences. As described above, PTSD and the hyperarousal and avoidance/numbing symptom clusters showed a strong relationship with neural activation to associability, suggesting that the interaction of PTSD and neural substrates of associability should predict switching behavior over and above neural activation to associability value alone. We therefore carried out mixed effects logistic regression models predicting switching behavior as a result of the interaction of the previous trial’s outcome and associability value-related neural activation, with and without PTSD symptoms as an additional interaction term (see Materials and methods for regression model specifications). The addition of PTSD diagnosis, hyperarousal symptoms, and avoidance/numbing symptoms each significantly improved model fit for bilateral amygdala after accounting for multiple comparisons (αBonferroni =.006; PTSD diagnosis:χ22 = 12.5 p<.005; hyperarousal: χ22 = 12.6, p<.005; avoidance/numbing: χ22 = 12.0, p<.005; Figure 5b), while re-experiencing symptoms did not (χ22 = 6.0, p>0.01; similar patterns were observed for bilateral insula; Figure 5—figure supplement 1b and c). A secondary analysis inspecting the components of this interaction revealed overall greater tendency to switch with greater associability value-related neural activation in participants with PTSD relative to controls, with additional effects of large relative to small outcomes leading to a greater tendency to switch in controls in the amygdala and insula ROIs and in PTSD in the amygdala ROI; the effect of outcomes in PTSD in the insula ROI was reversed (Figure 5—figure supplement 2a and b). Analogous regressions with prediction error related activation in striatum and vmPFC were implemented to test the specificity of effects to associability; these analyses indicated that no symptom clusters improved model fit for prediction error-related neural responses predicting behavioral choices (Figure 5—figure supplement 1b and c).

Discussion

Previous research has implicated altered learning and attentional processes in PTSD, but the neuromechanistic basis of these dysfunctions has been unclear. Here, we used a computational model-based approach and identified increased associability-based modulation of loss learning, neurally and behaviorally, in combat-deployed military veterans as a function of PTSD; prediction errors and unmodulated learning rate were unchanged by PTSD. The greater neural sensitivity to associability values was particularly strong with increasing hyperarousal and avoidance/numbing symptoms and interacted with symptom severity to predict participants’ behavioral choices. These results point to increased reliance on attention-based modulation of learning in PTSD that is guided by a network of brain regions including the amygdala and insula.

A number of earlier studies have connected greater attention to perceived threat, disrupted cognitive processing in the presence of negative, salient stimuli, and increased orienting to unexpected events in PTSD (Aupperle et al., 2012; Bar-Haim et al., 2007; Blair et al., 2013; Morey et al., 2009; Naim et al., 2015). However, the mechanistic underpinnings of this hypersensitivity to unexpected outcomes have been unclear. We posited that the hypersensitivity may derive from either (i) heightened solely error-related learning that increases learning directly from unexpected outcomes (i.e., prediction errors), or (ii) disproportionate attentional allocation to these prediction errors which magnifies the effects of these unexpected outcomes in future decision-making. Our approach differentiates between these possibilities, and the data support the latter hypothesis: neither static error-based learning rate nor neural prediction error responses were associated with PTSD, while increased associability updating based on loss prediction error signals increased with PTSD at both the behavioral and neural levels. These results suggest the disruptive effect of salient negative stimuli observed in PTSD is related to greater involvement of associability-related learning processes that are based on previous unexpected negative experiences with similar stimuli, rather than disrupted responses to immediate cues.

The current findings further link extant neural data showing amygdala and insula disruptions in PTSD (Hayes et al., 2012; Brown et al., 2014; Rauch et al., 2006) with the learning processes that these regions support. In particular, animal research has strongly implicated amygdala-mediated, attention-modulated processes in learning (Esber et al., 2012; Holland and Gallagher, 1999), and support for the amygdala’s role in these processes is emerging in human neuroimaging studies of associative learning (Li et al., 2011; Boll et al., 2013). In addition, intriguing evidence suggests that transient lesions of amygdala functioning through direct stimulation reduces PTSD symptoms (Langevin et al., 2016). The insula, which here similarly demonstrates increased associability-related activation with PTSD, also plays a key role in associability-based learning (Li et al., 2011) and related functions including altering outcome value and computing costs (Skvortsova et al., 2014; Palminteri et al., 2012). Previously found alterations in neural connectivity and responses to affective stimuli in amygdala/insula may further derive from increases in attention- or salience-modulated learning associated with these regions that emerges as hypersensitivity to potential threat (Etkin and Wager, 2007; Brown et al., 2014; Rauch et al., 2006; Seeley et al., 2007). We note that previous work on associability-based learning in non-psychiatric participants has shown a prominent role of associability in learning from other forms of aversive stimuli, such as threat of shock (Li et al., 2011; Boll et al., 2013), which also show disruptions in PTSD (Lissek and van Meurs, 2015). While the increased updating of associability values observed here in PTSD during monetary loss learning should apply generally to learning from other classes of negative stimuli, it will be important in future investigations to explicitly test the role of associability-based learning for PTSD in the context of threatening or trauma-related cues which have been shown to impair fear extinction in the disorder (Milad et al., 2009; Norrholm et al., 2011) as well as to connect our findings to models of other behavioral disruptions in mood/anxiety (e.g., [Mkrtchian et al., 2017]) and to explore under what conditions the greater associability weight in PTSD is adaptive versus maladaptive (as in [Vythilingam et al., 2007; Zinchenko et al., 2017]). Rich literatures support associability as a measure of cue-specific salience or attention allocation (Li et al., 2011; Pearce and Hall, 1980; Le Pelley, 2004; Roesch et al., 2012), but other stimulus properties and forms of attention may also affect learning in PTSD (Dayan et al., 2000; Yu and Dayan, 2005; de Berker et al., 2016). In addition, as previous work has found associability-related learning in the amygdala to be dopamine-dependent (Esber et al., 2012), the intact error-related but disrupted associability-related signaling in PTSD found in the present study raises questions about potential intermediate neural substrates involved in this dissociation (Nasser et al., 2017). Future work on the relationship between error- and associability-related learning signals is needed to further clarify the neural systems-level disruptions in this relationship in PTSD.

The associations among neural encoding of associability value, specific PTSD symptoms, and behavioral choices support the translational relevance of the present findings for identifying and refining new neuromechanistically-informed targets of treatment (Bowers and Ressler, 2015). Here, hyperarousal and the related numbing/avoidance symptoms (Flack et al., 2000) showed the strongest relationships with neural substrates of associability; interactions of these neural substrates with these PTSD symptoms also predicted participants’ behavioral choices during the loss learning task. The clinical manifestation of hyperarousal in PTSD includes excessive sustained vigilance (American Psychiatric Association, 2000) and resembles what heightened updating of associability values would predict; specifically, reminders of trauma can be conceptualized as stimuli associated with unexpected negative outcomes, with the greater associability updating seen here in PTSD causing stimuli associated with a recent history of these surprising outcomes to command greater attention and increased updating. Thus, addressing certain components of attention or cue-salience in learning-based interventions (e.g., prolonged exposure [Craske et al., 2014; Schnurr et al., 2007]) or incorporating a learning context into neural or cognitive therapies (Badura-Brack et al., 2015; Langevin et al., 2016; Khanna et al., 2015) may provide targeted benefit for reducing symptoms of PTSD. The increased updating of associability values in PTSD may reflect a higher prior belief that changes in prediction error will occur, a difference in processing changes in prediction errors, or a combination, which could also serve as mechanistic treatment targets. More generally, associability and other computational model-identified dysfunctions may serve as new, precise, functional targets for novel and extant treatment approaches. In particular, disruptions in associability-related attention or cue-salience neural circuits may be improved with computational model-based behavioral retraining techniques. Likewise, neural substrates of associability may be evoked during learning tasks and targeted with real-time neuromodulation.

The increased updating of associability values reported here also suggests mechanistic insight into the impaired fear extinction and retention of extinction memories that have previously been reported in PTSD (Milad et al., 2009; Norrholm et al., 2011). Specifically, greater updating of associability-modulated learning in PTSD may help explain several phenomena seen during extinction in PTSD. Greater associability signaling during cues associated with uncertain outcomes, such as during the early phases of fear extinction, would create a lack of habituation (Norrholm et al., 2011; Collins and Schiller, 2013). Meanwhile, as stimuli become more predictable, greater associability values lead to reduced learning, which may weaken retention of repeatedly presented information such as during later stages of extinction (Craske et al., 2014; Milad et al., 2009). Increased updating of associability values in PTSD may also increase the likelihood of inferring new states after surprising outcomes, rather than supporting inhibitory learning about current states, thus reducing the ability to connect conditioned and extinguished contexts and leading to impairments in extinction retention and generalization (Dunsmoor et al., 2015; Morey et al., 2015). We note that while the present work is consistent with earlier studies in non-psychiatric controls identifying associability-related neural activity at the outcome event (Li et al., 2011; Esber et al., 2012; Roesch et al., 2010a, 2010b), neural substrates of associability have also been observed with predictive cues (Roesch et al., 2012); this distinction deserves further study in future work investigating and addressing the nature of learning disruptions in PTSD.

Despite the high prevalence and great societal, personal, and fiscal cost of PTSD, and particularly combat-related PTSD (Richardson et al., 2010), the mechanistic underpinnings of the disorder remain unclear. Here, a neurocomputational psychiatry approach distinguishes between competing hypotheses about dysfunctional processes in PTSD (Montague et al., 2012; Stephan and Mathys, 2014) and shows that PTSD is related to increased neural and behavioral substrates of attention-based modulation of loss learning, while solely prediction error-based learning is unchanged. More generally, these findings suggest that by integrating human functional neuroscience, computational model-based analyses, and gold-standard clinical assessments, new neuromechanistic targets of intervention for PTSD and other mental disorders may be identified and tested.

Materials and methods

Participants

Participants were US military veterans (N = 74) with combat deployments to Iraq or Afghanistan since 2001. All veterans had served at least one tour in Iraq or Afghanistan since 2001 and were part of a larger study by our group examining biomarkers of mood and anxiety disorders. All participants provided written informed consent, and all procedures were approved by the Institutional Review Boards of Baylor College of Medicine, the Salem Veterans Affairs Medical Center, and Virginia Tech. Veterans served by the Houston, TX and Salem, VA Veterans Affairs Medical Centers were recruited through VA medical record searches, provider referrals, and community advertisements. Enrollment inclusion criteria for the larger biomarker study included: meeting criteria for current PTSD or non-PTSD, age 18 to 64, English speaking, normal or corrected to normal vision, and verbal IQ greater than 80. Enrollment exclusion criteria included: contraindications to MRI scanning (e.g., implanted ferrous metal, claustrophobia), loss of consciousness greater than 30 min, behaviors meeting criteria for substance abuse or dependence (excluding nicotine dependence) in the past 30 days, and current/past psychotic or bipolar disorders.

For the veteran cohort, psychiatric diagnoses were assessed using the Structured Clinical Interview for DSM-IV (SCID [First et al., 1996]) and the Clinician Administered PTSD Scale (CAPS [Blake et al., 1995]), administered by trained study staff. Using the ‘Rule of 3’ (frequency +intensity > 3 [Blanchard et al., 1995]) diagnostic criterion on the CAPS. Total PTSD severity was constructed from the summed severity score for all symptoms. Severity scores for the three DSM-IV symptom clusters, measuring re-experiencing (CAPS questions B1-B5), avoidance and numbing (questions C6-C12), and hyperarousal (questions D13-D17) symptoms were created to assess the severity of each symptom cluster. Participants also completed the Beck Depression Inventory-II (BDI [Steer et al., 1999]) measuring depression symptoms; the Combat Exposure Scale (CES [Lund et al., 1984]), measuring combat trauma exposure, and the Wechsler Test of Adult Reading (WTAR [Wechsler, 2001]), measuring approximate verbal IQ. In addition, participants completed a demographics questionnaire assessing age, gender, ethnicity, years of education, and medications taken. Two participants did not complete the CES; their scores were imputed based on other variables (CAPS, BDI, age, and gender) using the ‘mi’ package in R (Su et al., 2011).

To be included in the present analyses, veterans were further required to demonstrate behavioral engagement on the relevant portions of the probabilistic learning task (i.e., sufficient loss learning, criteria detailed in the Probabilistic learning task section) and have successfully completed the functional MRI portion of the study. In addition, all included veteran participants were required to have experienced trauma satisfying criterion A1 of PTSD according to the DSM-IV (American Psychiatric Association, 2000), and veteran participants without PTSD were required to be free from current DSM-IV diagnoses and not taking psychotropic medication. N = 68 veterans satisfied these additional criteria and were considered for all subsequent behavioral analyses (see fMRI data collection and preprocessing for imaging analysis inclusion criteria). Among these 68 veterans, thirty-nine (N = 39) met DSM-IV (American Psychiatric Association, 2000) criteria for PTSD (see Supplementary file 1, Table 2A for clinical and demographic information); the other N = 29 veterans had previous deployment-related trauma exposure as assessed by the CAPS interview but did not meet criteria for PTSD. Participants excluded did not differ from those retained for all analyses on demographic measures (age, gender, household income level, proportion nonwhite, proportion with greater than high school education), PTSD severity, or proportion with a PTSD diagnosis (all ps > 0.05).

Two additional cohorts of participants were recruited from the Houston, TX and southwest Virginia areas. The first, a non-psychiatric civilian reference cohort (N = 23) was not included in the full set of analyses but was used to construct independent regions of interest to test in the veteran cohort (see fMRI procedures for more information). The second cohort of participants (N = 20) consisted of participants with a current diagnosis of Major Depressive Disorder and no current or past PTSD, matched to the participants with PTSD on gender, smoking status, and estimated IQ. Behavior from this clinical comparison cohort was fit to the associability RL model and parameter estimates were compared to the participants with PTSD (see Relationship between model parameters and PTSD for more information). The focus of the present work was on learning in PTSD; as such, given the high co-occurrence of depression in PTSD, the MDD-only comparison group was used here to test the specificity of the main behavioral effects to PTSD.

Participants in the non-psychiatric reference cohort, used to construct independent neural regions of interest, had psychiatric diagnostic status assessed by the SCID (N = 17) or the Mini International Neuropsychiatric Interview (MINI [Lecrubier et al., 1997]; N = 6). Participants also completed the WTAR to estimate IQ and a demographics questionnaire. Participants included for ROI construction reported no trauma exposure that qualified for Criterion A1 for PTSD and were matched to the veteran cohort on age, gender, and estimated IQ. Participants in the MDD psychiatric reference cohort, whose behavioral learning parameters were compared to the participants with PTSD, had psychiatric diagnostic status assessed by the SCID and met criteria for current Major Depressive Disorder and did not meet current or lifetime criteria for PTSD. These participants also completed the BDI to assess current depression symptoms, the WTAR to estimate IQ, and a demographics questionnaire to assess age, gender, and smoking status.

fMRI data collection and preprocessing

Participants were scanned on a 3T Siemens Tim Trio MR scanner. Echoplanar images were collected in 34 4 mm slices at a 30° hyperangulation from the anterior-posterior commissure (AC-PC) line (TR = 2000 ms, TE = 30 ms, flip angle = 90°, matrix = 64×64, voxel size 3.4 × 3.4×4.0 mm3). A high-resolution (1 mm3) anatomical Magnetization Prepared Rapid Gradient Echo (MPRAGE) T1 image (TR = 1200 ms, TE = 2.66 ms, flip angle = 12°) was collected to aid in registration.

All imaging analyses were conducted using SPM8 for fMRI (Wellcome Trust Centre for Neuroimaging, http://www.fil.ion.ucl.ac.uk/spm/software/spm8). Preprocessing consisted of: slice timing correction, realignment to the first functional scan, coregistration to the participant’s structural image, normalization to the MNI template, and smoothing to ensure Gaussianity (6 mm FWHM). Participants with motion greater than 3 mm or 0.5 radians in any direction were excluded (n = 4). Functional images were visually inspected for signal drop out in ventral areas, including ventromedial prefrontal cortex and amygdala, and were excluded if significant signal loss was present (n = 3). An additional two participants were excluded due to other data quality issues.

Probabilistic learning task

Participants completed a probabilistic loss and gain learning task (two-arm bandit, adapted from [Pessiglione et al., 2006]) while undergoing fMRI scanning (Figure 1). The task was presented in pseudo-randomized alternating blocks consisting of all loss learning or gain learning trials, respectively. On each trial, participants were presented with two abstract stimuli. One stimulus had a higher (75%) probability of leading to a better monetary outcome and a lower probability (25%) of leading to a worse monetary outcome, while the probabilities for the other stimulus were reversed (i.e., smaller probability of better outcome and larger probability of worse outcome). Participants selected one stimulus using a MRI-compatible button box (Current Designs, Inc.). The participant’s choice was framed for a jittered viewing time of 2–4 s, after which the outcome (monetary amount gained or lost) was shown for 2 s. Participants were not shown the outcome associated with the unchosen stimulus. A fixation cross was shown between each trial for a jittered viewing time of 1–3 s. For gain trials, the better outcome ranged from +70 to +80 cents, while the worse outcome ranged from +20 to +30 cents. For loss trials, the outcomes similarly ranged from −20 to −30 and −70 to −80 cents. At the beginning of each block, high and low outcome values were randomly chosen with replacement from uniformly distributed outcome pairs {20,70}, {25,75}, or {30,80} and kept consistent within blocks (the outcome displays indicated ‘You Lose [amount]’ for loss blocks, and ‘You Gain [amount]’ for gain blocks). Blocks containing gain and loss trials alternated. Each block consisted of novel stimuli, which required participants to re-learn the contingencies between stimuli and outcomes within each block.

As our goal was to examine mechanistic processes associated with learning, including in individuals who may differ in learning processes, the task used an adaptive algorithm to ensure that participants successfully learned the contingencies and continued to learn throughout the task. In order to obtain a sufficient number of learning trials, block length was determined based on the proportion of correct choices (i.e., choosing the better option): blocks ended when performance reached at least 70% correct based on a running average of the last 10 choices, with the additional specification that the first block be at least 15 trials long. The task ended when the participant had at least 25 correct and 25 incorrect choices in each of the loss and gain conditions. The total number of trials per participant ranged from 50 to 70 for loss trials and 50–68 for gain trials. Participants completed an average of 4.14 gain blocks and 4.58 loss blocks; the number of blocks and trials completed did not differ between PTSD and veteran control participants for gain or loss (all ps > 0.2). The comparable number of trials and blocks completed in each group indicates that the amount of task titration was similar on average between the PTSD and veteran control participants.

Before entering the scanner, participants were presented with task instructions, completed a practice round, and were given the opportunity to ask questions. They were not provided information about the full statistical structure of the task but were provided an initial $10 endowment, and informed that payment would be based on actual performance. To ensure participants were attending to the task and had behavior suitable for model fitting (Sokol-Hessner et al., 2009), participants who switched options in either gain or loss blocks less than 10% of the time were excluded from analyses comparing gain and loss behavior (N = 19; 8 control veterans and 11 veterans with PTSD). For analyses of loss behavior only, participants excluded for low switching during gain blocks only were included in analyses (see Comparison of model fits for RL models for more information).

Reinforcement learning models

To identify the reinforcement learning (RL) model that best explained participants’ data, models with and without associability were tested; models were evaluated separately for gain and loss trials.

Following standard temporal difference RL (Sutton and Barto, 1998) the expected value (Q) of the stimulus on the next trial (t + 1) was updated with the product of the learning rate (α; range 0 to 1) and prediction error (δ):

(1) QA(t+1)=QA(t)+α δ(t)

Trial-by-trial prediction error δ(t) was computed as the difference between the actual outcome (R) and the expected value (Q):

(2) δt=R't-Q(t)

The outcome R was modulated by reward sensitivity (ρ) (Huys et al., 2013), comprising a multiplier on the value of the loss or gain value further from 0 (i.e., outcomes ranging ±0.70 to ±0.80):

(3) R'At=ρRAt

where A represents the chosen option. A decay parameter for the unchosen option B (γ; range 0+) (Boorman et al., 2009; Collins and Frank, 2016; Niv et al., 2015; Cavanagh, 2015) was also included:

(4) QBt+1=γQB(t)

The probability of each choice was modeled with a softmax function incorporating inverse temperature (β; range 0+):

(5) PA(t+1)=eβQA(t+1)eβQA(t+1)+eβQB(t+1)

Since inverse temperature and reward sensitivity may not be uniquely identifiable in the RL model, inverse temperature was first estimated in a model without reward sensitivity. The RL model with reward sensitivity was then estimated with inverse temperature fixed at the group mean rather than estimated as a free parameter (similar to [Pessiglione et al., 2006]).

For the associability RL model, learning rate was modulated on a trial-by-trial basis by an associability value for the chosen stimulus (κ) (Li et al., 2011). An associability weight parameter (η; range 0 to 1) controlled the extent to which the magnitude of previous prediction errors updated the trial-by-trial associability value of a particular stimulus. Associability values were initialized at one and updated for each stimulus separately. Associability values were constrained to stay above. 05 (Li et al., 2011Le Pelley, 2004) . Therefore, similar to the constant parameter of learning rate and the trial-by-trial estimate of prediction error in classic RL models, this model added a constant parameter of associability weight and a trial-by-trial estimate of associability value of the chosen stimulus.

(6) κAt+1=1-ηκAt+η|δt|
(7) QAt+1=QAt+α* κAt*δt

Regressors for imaging analyses used the prediction error, associability value, and probability at trial t, prior to updating associability value for the next trial. Because novel stimuli were used each block, the expected value and associability value were reset at the beginning of each block. The same parameters were used for all blocks of the same condition, but separate parameters were used per condition, resulting in one learning rate for loss and one learning rate for gain, one associability weight for gain and one for loss, and so on.

Comparison of model fits for RL models

Model comparison was computed using Bayesian Model Selection (Rigoux et al., 2014). Each participant’s probability for each model was transformed into corrected AIC (AICc) to penalize models with greater numbers of parameters. These values were then fed into a variational approximation of BMS, resulting in a predicted probability of each model per subject and an overall protected exceedance probability of each model over all subjects. This approach allows for between-subject heterogeneity in model probabilities while enabling a group-level model comparison.

Reinforcement learning model parameters were estimated separately for loss and gain trials, based on previous work showing differential influences of reward and punishment during learning in similar tasks (Palminteri et al., 2012; Pessiglione et al., 2006). To test the appropriateness of this approach, we first conducted a logistic regression predicting the probability of switching choices based on the previous three outcomes, including the condition (loss or gain) as an interaction term on the previous outcome. This interaction term was significant (z = 2.124, p<0.05), suggesting different learning patterns during loss and gain learning. As a second test of the appropriateness of separate estimates for loss and gain, we compared a model with parameters combined across loss and gain trials to a model separating all parameters by condition. The model with all parameters separated by condition fit better (AICc improvement in 91% of participants, protected exceedance probability of 100%), confirming different learning patterns in loss and gain conditions. Additionally, we tested the fit of a basic reinforcement learning model, with parameters of learning rate and inverse temperature, versus our model with parameters of learning rate, reward sensitivity, and decay (prior to adding the additional parameter of associability weight) and found that the more complex model fit better, with a protected exceedance probability of 100%.

Reinforcement learning models with and without associability were then compared within loss and gain conditions separately. 81% of participants (85% of veteran controls and 78% of PTSD participants) showed a greater probability of the model with associability in loss, while no participants showed a greater probability of the associability model during gain learning. This difference in frequencies between conditions was significantly different (χ21 = 59.0, p<0.001). The overall protected exceedance probability of the associability model reflected this difference, with a protected exceedance probability of 100% in loss and 0% in gain (Figure 3a). As a result of the improved model fit in the loss condition only, subsequent analyses focused on loss learning trials. To maximize power, participants with suitable loss behavior but a low rate of switching during gain learning (n = 16) or who were excluded from imaging analyses (n = 9) were included in model-based behavioral analyses, resulting in a sample size of 68 (39 PTSD and 29 veteran controls); analyses comparing loss and gain learning used the more restricted sample of 43 participants (23 PTSD and 20 veteran controls) with complete data across conditions and modalities (i.e., loss, gain, behavior, neuroimaging). Analyses of these participants with full data in all four categories are reported in Appendix 2, Supplementary Results, Model-based loss analyses in restricted sample and are consistent with results from the larger group.

Model testing and validation

To delineate the effect of associability weight on performance accuracy, we conducted a simulation of participants’ performance at different values of the associability weight. Other parameters were set at the group means and the associability weight was varied from. 05 to. 95 in steps of. 05. Data from fifty participants with two blocks of 25 trials each were simulated for each value of the associability weight. The mean and standard error of the proportion of correct options chosen were calculated for each value of associability weight. Performance did not systematically vary with the value of the associability weight and the correlation between associability weight and performance was not significant (r = 0.10, p>0.1; Figure 3—figure supplement 1a).

According to theories of associability-based learning (Le Pelley, 2004), stimuli that have been poorly predicted in the past should draw more attention during subsequent learning trials, resulting in increased associability values and longer decision times. Therefore, we tested whether reaction times scaled with the associability value of the chosen cue. Each participant’s reaction times were modeled in a regression equation with predictors of the chosen cue’s associability value, the chosen cue’s expected value, and the trial number; trials with reaction times shorter than 333 milliseconds or longer than five seconds were excluded (3.07% of trials). Figure 3b shows participants’ average beta values of the relationship between associability value and reaction time during loss learning. The average beta value was .232, and significantly greater than 0 (one sample t-test: t68 = 8.28, p<0.001), reflecting longer decision times for stimuli with larger associability values. The average beta value for associability value predicting reaction time when including the previous trial’s prediction error as an additional predictor was similarly significant (t68 = 7.21, p<0.001), indicating that the effect of associability value on reaction time was not due to concomitant prediction error. This effect was confirmed in a mixed effects regression using the ‘lme4’ package (Bates et al., 2015) in R including all participants’ reaction times and accounting for the nesting of responses within subjects. Model fit was compared between a model predicting reaction times from the chosen cue’s associability value, the chosen cue’s expected value, and trial number versus a model without associability value; the model with associability value was a significantly better fit (likelihood ratio test: χ21 = 8.67, p<.005). Trial number was included as a covariate in these analyses to account for general adaptation effects; however, trial number and associability were uncorrelated (average correlation across subjects: r = −0.12).

To investigate the associability RL model’s performance in predicting participants’ behavior, each participant’s trials were binned (using 5 bins total) based on the probability of choosing the correct choice as predicted by the model. This binning resulted in a proportion of correct choices made when the predicted probability of choosing the correct stimulus was 0–20%, 31–40%, and so on per participant. The mean and standard error of the proportion of correct choices per bin was calculated across subjects. The model showed good concordance with actual behavior; binned predicted performance was strongly correlated with mean proportion of correct choices per bin (r4 = 0.9986, p<0.001; Figure 3—figure supplement 1b).

As an additional test of the ability of the RL model’s associability value to predict behavior in the loss condition, mixed effects logistic regression models were used to assess the effects of prediction error alone and prediction error modulated by associability value on model free switching behavior using the ‘lme4’ package in R. The dependent variable was whether or not the participant switched choices from the previous trial to the current trial (0 = no switch, 1 = switch). The basic regression included terms for the previous trial’s outcome (1 = small loss, 0 = large loss) and for the nesting of trials within subjects (e.g., switch ~ outcome 1 trial previous + [1|subject]). The fit (AIC) for this model was compared against two more complex models: (1) adding an interaction of the previous trial’s outcome with the previous trial’s prediction error (e.g., outcome 1 trial previous * PE 1 trial previous) and (2) adding an interaction of the previous trial’s outcome, prediction error, and associability value (outcome 1 trial previous * PE 1 trial previous * associability value 1 trial previous). Prediction error and associability value estimates were taken from the associability RL model using individual parameter estimates. The difference in AIC between the basic regression model and models adding prediction error and prediction error * associability is plotted in Figure 3c. Standard error on the improvement in AIC was calculated using a jackknife approach (Efron and Gong, 1983).

Relationship between model parameters and PTSD

The relationships between PTSD diagnosis and individual behavioral parameter estimates for associability weight and learning rate were computed in R using a linear regression with additional regressors of BDI, CES, age, and gender. The variance inflation factor for PTSD in this analysis was within acceptable limits (VIF = 1.56). The beta values for PTSD diagnosis (control vs. PTSD; error bars of standard error) predicting associability weight and learning rate, respectively, are plotted in Figure 4a.

As an additional test of the specificity of these findings to PTSD versus general psychopathology, RL parameters from the behavior of an additional psychiatric reference cohort with diagnoses of MDD but not PTSD (see Participants, above) were compared to the participants with PTSD, covarying for age. A linear regression model was used to compare these groups accounting for age; a regression with additional covariates of gender, estimated IQ, smoking status, and depression severity showed similar results supporting the specificity of increased associability to PTSD (PTSD diagnosis effect: t53 = 7.29, p<0.001).

fMRI Analyses

For first level analyses, analyses of prediction error used prediction error (δ) as a parametric modulator at the time of outcome. Based on previous work showing electrophysiological and BOLD response to associability at time of outcome (Li et al., 2011; Esber et al., 2012; Roesch et al., 2010a), associability analyses used associability value (κ) as a parametric modulator on the outcome event. An additional analysis combined these parametric modulators in the same first-level analysis, with prediction error entered as the first parametric modulator and associability value as the second; this approach did not meaningfully differ from entering these parametric modulators in separate first-level analyses (see Supplementary file 1, Table 1H for second level results for this approach). All analyses used the probability of the chosen option (computed by the softmax function) as a parametric modulator at time of cue presentation and a second parametric modulator of outcome amount at the time of outcome. All outcome and cue events were modeled as stick functions. Prediction error, associability value, and probability were computed based on each subject’s individually estimated parameters and were z-transformed prior to entering at the first level (Lebreton and Palminteri, 2016). Additional regressors of no interest included button press, block number, and six motion regressors. Data were high pass filtered with a cutoff of 128 s. Group level analyses were set to a voxel-level cluster forming threshold of p<0.001 and then cluster-level corrected at p<0.05 topological false discovery rate (FDR) for the whole brain (Woo et al., 2014).

To assess the impact of PTSD (or symptom severity) on neural substrates of prediction error and associability, PTSD diagnosis or Clinician Administered PTSD Scale (CAPS) symptom cluster score was entered at the group level with covariates of depression, combat exposure, age, and gender. Follow-up analyses included the presence of psychotropic medication and mTBI as covariates (see Appendix 2, Supplementary Results, Subgroup Analyses for details). To test the robustness of our results to the thresholding method used, the effect of PTSD diagnosis on associability-related neural signal at the second level was checked using a nonparametric thresholding approach. Using SnPM13 (version 1.05; http://warwick.ac.us/snpm), 10,000 permutations were carried out with PTSD as the covariate of interest and depression, combat exposure, age, and gender as covariates of no interest. Results were thresholded at p<0.05 FWE, resulting in a corrected threshold of t > 5.36. Results with this permutation analysis were similar to those reported in the main text and are displayed in Supplementary file 1, Table 1I.

To test whether the lack of evidence for a relationship between neural prediction error and PTSD severity was due to conservative whole-brain correction, we followed up with a region of interest analysis in areas involved in prediction error. To select areas most likely to be involved in prediction error, functional regions of interest (ROIs) were created based on activations from the probabilistic learning task in a non-veteran reference cohort (see Participants, above). Group-level activations for prediction error and associability were thresholded at p<0.005 uncorrected with a cluster extent ≥20 voxels. Activation clusters were binarized as masks and used to create ROIs (21 for gain PE, 13 for loss PE). The first eigenvariate of the beta values in each ROI was extracted for the veteran group. Values were read into R and regressed against PTSD diagnosis along with covariates of BDI, CES, age, and gender. To verify intact prediction error signaling, ROIs in the striatum were used in one-way t-tests to determine if mean activation for the veteran cohort was significantly different from zero.

Neural interaction predicting behavioral switching

To evaluate the relevance of neural substrates of associability to behavioral disruptions in PTSD, we tested whether interactions among PTSD, neural activation in areas related to associability or prediction error, and previous outcomes predicted switching behavior. The anatomically defined regions of interest were created in the following manner: amygdala from centromedial and laterobasal subregions from the Jühlich atlas (Amunts et al., 2005); striatum from caudate, putamen, and globus pallidus from the AAL atlas (Tzourio-Mazoyer et al., 2002) and nucleus accumbens from the IBASPM atlas in WFU PickAtlas; anterior insula from portion of insula anterior to y = 10 from the AAL atlas; and ventromedial PFC from medial orbitofrontal cortex from AAL. Each subject’s beta values for associability value-related activation (in amygdala and insula) and prediction error-related activation (in striatum and vmPFC) were extracted from the respective ROIs. The regression included terms for the interaction of the ROI beta values with PTSD diagnosis or symptom cluster severity and the previous trial’s outcome (1 = small loss, 0 = large loss) as well as the outcomes of two and three trials previous and for the nesting of trials within subjects (e.g., switch ~ previous trial’s outcome * associability-related amygdala beta value * PTSD diagnosis + outcome 2 trials previous + outcome three trials previous + [1|subject], plus all lower order terms). The effects of the three previous outcomes were included in the regression as they were all found to affect switching behavior. Models were estimated using the ‘lme4’ package in R. Likelihood ratio tests compared this model to a model with the same ROI, but without the PTSD terms (main effect and interaction). Changes in model fit, rather than the significance of individual regressors, was chosen due the unreliability of p-values in mixed effects models such as the one used here (Snijders, 2011). Therefore, improvement in model fit with the addition of a term is analogous to this term being significant in the regression model. Alpha level was set at p<0.006 (Bonferroni corrected for 8 tests). Figure 5b shows the chi squared value from these likelihood ratio tests for amygdala and striatum, with chi squared values corresponding to p<0.005 noted for reference. Analyses with all ROIs are shown in Figure 5—figure supplement 1b. Although the dependent and independent variables in this analysis (switching behavior and associability-related neural activation) are related to behavioral associability parameters, this relationship reflects the underlying theorized relationship among behavior, the hybrid RL model, and neural activation rather than representing a circular analysis.

To assess subject-level variability in the likelihood ratio tests, jackknife analyses (Efron and Gong, 1983) were run on each hierarchical logistic regression. The distribution of the chi-squared values are shown in Figure 5—figure supplement 1c. Supporting the main analyses (Figure 5), the interaction of associability-related neural activation in amygdala with PTSD diagnosis, hyperarousal symptoms, and to a lesser extent avoidance/numbing symptoms, all showed the majority of jackknife chi squared estimates above the corrected significance levels. Meanwhile, the interaction of associability-related neural activation in amygdala with re-experiencing, as well as prediction error-related neural activation in striatum with PTSD diagnosis and all symptom clusters, all showed the majority of estimates below significance.

Appendix 1

Supplementary methods

Model-agnostic behavioral analyses

Model-agnostic analyses assessed proportion of correct choices (i.e., better option chosen) between veteran participants with and without PTSD. To investigate associability-based learning in PTSD in a model-agnostic way, we conducted logistic regressions predicting participants’ probability of switching based on the previous outcome as well as the unexpectedness of prior outcomes. To quantify the amount of surprise or unexpectedness in a manner similar to our model-based analyses, we defined the expected outcome in terms of the history of outcomes to that point in the block, such that the expected outcome of a stimulus equaled the proportion of time the chosen stimulus had led to the better outcome in the block or the other stimulus had led to the worse outcome. For example, if stimulus A had been picked three times, with two better outcomes, and stimulus B had been picked two times, with one worse outcome, the outcome history for stimulus A would be (2 + 1)/5 = 0.6. In addition, to account for the fact that each stimulus carried an associability value that was updated only when that stimulus was chosen, the amount of surprise was calculated anew each time a stimulus was chosen, and carried forward from the previous trial when it was not. The interaction of expectedness of the outcome two trials back with the value of the most recent outcome was prominent in loss trials (z = −3.08, p=0.002; gain trials: z = −1.74, p=0.082), supporting the role of associability-based learning in the loss domain. When examining each group separately during loss learning, the interaction of previously unexpected outcomes and the most recent outcome was more evident in the PTSD participants (z = −3.02, p=0.003; control participants: z = −1.37, p=0.17), supporting greater associability-based learning in participants with PTSD during loss learning.

Parameter estimation and recovery

Reinforcement learning models with and without associability were fit using hierarchical Bayesian estimation (Daw, 2011; Wiecki et al., 2015). Estimation was done using Hamiltonian Markov Chain Monte Carlo using Stan (Carpenter, 2017) in R. Group level parameters were specified as normally distributed, with an inverse logit transformation on parameters constrained to be between 0 and 1 (learning rate and associability weight) and a lower bound of 0 on parameters constrained to be positive (reward sensitivity and inverse temperature). Parameters were given a non-centered parameterization to aid in estimation by specifying mean, scale, and error distributions for each parameter (Betancourt and Girolami, 2015). Mean distributions, estimated at the group level, were specified as normally distributed with priors of mean = 0 and standard deviation = 10, for parameters that are not logit transformed, or with standard deviation = 2.5 for parameters that were logit transformed. Scale distributions, estimated at the group level, were given a half-Cauchy prior (Gelman, 2014) (bounded to be greater than 0) with values of 0 and 2.5. Error distributions, which were estimated for each subject, were given a normal prior with mean = 0 and standard deviation = 1. For learning rate and associability weight, these parameter values were then run through an inverse logit transformation bound between 0 and 1. Therefore, each subject’s parameter (for example, associability weight) consisted of a group estimated mean value plus the combined value of the group estimated scale value multiplied by the individually estimated error value. Models were estimated separately for each condition and within each veteran group. Four MCMC chains were run for each RL model, with 8000 samples per chain (4000 after discarding warm up samples). Chains from the RL models used in analyses were inspected for convergence and showed good mixing, with all values of the potential scale reduction factor (Gelman and Rubin, 1992) less than 1.1.

In the loss condition, the fit parameters for the veteran controls (inverse temperature fixed at 4.35) were: associability weight = 0.246 (SE. 033), learning rate = 0.708 (SE. 020), reward sensitivity = 1.06 (SE. 065), and decay of unchosen option = 0.857 (SE. 022) and for the PTSD subjects (inverse temperature fixed at 5.05): associability weight = 0.540 (SE. 039), learning rate = 0.672 (SE. 011), reward sensitivity = 1.06 (SE. 069), and decay of unchosen option = 0.883 (SE. 010). As reported throughout, the RL model with associability weight did not fit well in the gain condition (see Comparison of model fits for RL models section above); for completeness, we also report the parameters of the RL model without associability weight in gain: for the veteran controls (inverse temperature fixed at 6.35): learning rate = 0.10 (SE = 0.02), reward sensitivity = 3.01 (SE = 0.09), and decay of unchosen option = 0.82 (SE = 0.01); and for the PTSD subjects (inverse temperature fixed at 7.20): learning rate = 0.10 (SE = 0.02), reward sensitivity = 3.18 (SE = 0.09), and decay of unchosen option = 0.89 (SE = 0.01).

To test whether model parameters were independently recoverable and did not covary in a way that would prevent accurate estimation of each parameter separately, data were simulated based on different values of parameters along the range seen in participants’ data (learning rates of. 67,. 75, and. 83; associability weights of. 25,. 375, and. 5; reward sensitivity of 1.0, 1.25, and 1.5; and decay of unchosen option of. 9,. 95, and 1). These simulated data were then fit to the model and the resulting recovered parameters were compared to the simulated parameters. All parameters were recoverable, with simulated differences in parameter values similar to recovered differences (Figure 3—figure supplement 1c).

Appendix 2

Supplementary results

Gain learning

As reported in the main text, model fit comparison showed a model with an associability parameter to be a worse fit for the gain condition. The presence of associability-based learning during exclusively negative contexts is supported by previous studies on the topic (Li et al., 2011; Roesch et al., 2012; Boll et al., 2013).

Similar to loss prediction error, striatal ROIs showed significant activation with prediction error independent of PTSD symptoms (t42 = 2.87; p<0.01), confirming intact PE-related signal in the veteran cohort during gain learning.

Subgroup analyses (depression, combat exposure, medication, traumatic brain injury, smoking, IQ)

Depression and combat exposure

The covariates of depressive symptoms (as measured by BDI) and level of combat exposure (as measured by CES) were not related to variables of interest on a behavioral or neural level. Behavioral associability weight was not related to depressive symptoms (t62 = 0.31, p>0.1) or to level of combat exposure (t62 = 0.88, p>0.1). In addition, when depression diagnosis (from the SCID) was substituted for level of depressive symptoms in analyses, it also did not show a relationship with behavioral associability weight (t62 = −1.51, p>0.2). Level of depressive symptoms, depression diagnosis, and level of combat exposure all did not show a significant relationship with fMRI regressions against associability value.

Psychotropic medications and traumatic brain injury

Covariates accounting for the use of psychotropic medications or the presence of mTBI were not related to variables of interest on a behavioral or neural level. Behavioral associability weight was not related to the use of psychotropic medication (t61 = 0.43, p>0.2) or to the presence of mTBI (t61 = −1.14, p>0.2). For fMRI regressions of associability value against medication or mTBI status, no significant relationships were observed (p<0.001 cluster-forming threshold).

Smoking

Three out of the 29 veteran controls without PTSD and eight of the 39 veterans with PTSD reported being a current cigarette smoker. The chi-squared test for differences in frequencies of cigarette smokers in these two groups was not significant (χ21 = 0.692, p>0.2). We additionally investigated smoking status as a covariate in a series of follow-up analyses to determine if nicotine consumption was related to our effects of interest. Smoking status was not related to loss associability or learning rate (associability: t61 = 1.20, p>0.2; learning rate: t61 = −0.18, p>0.2) and including this variable did not meaningfully alter the relationship between PTSD diagnosis and these dependent variables (relationship with PTSD diagnosis: learning rate t61 = −0.004, p>0.2; associability t61 = 4.50, p<0.001). Smoking status also did not show any significant relationship with neural associability, and including this variable in the neural analyses did not meaningfully change the relationship between PTSD status and associability-related activation.

IQ

Since lower intelligence is a risk factor for PTSD, and has been widely documented to differ between those who develop PTSD after a trauma and those who do not (Breslau et al., 2006; Koenen et al., 2007), IQ was not included as a covariate in the primary analyses. When including estimated verbal IQ (Wechsler Test of Adult Reading score) as a covariate, the relationship between estimated verbal IQ and behavioral associability weight is nonsignificant (t61 = −0.94, p>0.2) as well as with behavioral learning rate (t61 = 1.93, p>0.05), while neural effects of associability do not show a relationship with IQ. Meanwhile, the effect of PTSD behaviorally and neurally remains significant when including estimated IQ as a regressor.

Subgroup matched on IQ and medication

As an additional test of the robustness of effects in PTSD to the potential confounds of estimated IQ and medication status, a subgroup of participants who were both (1) free from psychotropic medication and (2) matched between PTSD and veteran control groups on estimated IQ was analyzed. The effect of PTSD on behavioral associability weight (t29 = 2.99, p<0.01) and neural associability signal in ROIs of amygdala (t20 = 2.74, p<0.05) and insula (t20 = 4.42, p<0.001) remained significant in this subgroup.

Model-based loss analyses in restricted sample

As reported above, our initial analyses included any participants with suitable loss behavior (N = 68). We repeated these behavioral analyses in the subset of veteran participants with suitable behavior and imaging data for both loss and gain trials (N = 43; 23 patients with PTSD and 20 veteran controls). Mirroring the findings in the full set of participants, participants with PTSD in this smaller sample had a higher behavioral associability weight (t37 = 2.86, p<0.01) and no difference in learning rate (t37 = −0.45, p>0.2), an effect that was robust to additional covariates of medication, smoking status, and mild TBI.

References

  1. 1
    Diagnostic and Statistical Manual-Text Revision
    1. American Psychiatric Association
    (2000)
    American Psychiatric Association.
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
    Hamiltonian Monte Carlo for Hierarchical Models
    1. M Betancourt
    2. M Girolami
    (2015)
    In: S. K Upadhyay, U Singh, D. K Dey, A Loganathan, editors. Current Trends in Bayesian Methodology with Applications. Chapman and Hall/CRC. pp. 79–101.
    https://doi.org/10.1201/b18502-5
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
    Stan: a probabilistic programming language
    1. B Carpenter
    (2017)
    Journal of Statistical Software, 76, 10.18637/jss.v076.i01.
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
    Decision Making, Affect, and Learning: Attention and Performance XXIII
    1. ND Daw
    (2011)
    3–38, Decision Making, Affect, and Learning: Attention and Performance XXIII.
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
    User’s Guide for the Structured Interview for DSM-IV Axis I Disorders—Research Version (SCID-I)
    1. M First
    2. R Spitzer
    3. M Gibbon
    4. J Wiliams
    (1996)
    Biometrics Research.
  29. 29
  30. 30
  31. 31
  32. 32
    Bayesian Data Analysis
    1. A Gelman
    (2014)
    CRC Press.
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
  42. 42
  43. 43
  44. 44
  45. 45
  46. 46
  47. 47
  48. 48
  49. 49
  50. 50
  51. 51
  52. 52
  53. 53
  54. 54
  55. 55
  56. 56
  57. 57
  58. 58
  59. 59
  60. 60
  61. 61
  62. 62
  63. 63
    Classical Conditioning II: Current Research and Theory
    1. RA Rescorla
    2. AR Wagner
    (1972)
    64–99, Classical Conditioning II: Current Research and Theory.
  64. 64
  65. 65
  66. 66
  67. 67
  68. 68
  69. 69
  70. 70
  71. 71
  72. 72
  73. 73
  74. 74
  75. 75
  76. 76
  77. 77
    Reinforcement Learning: An Introduction
    1. R Sutton
    2. A Barto
    (1998)
    MIT Press.
  78. 78
  79. 79
  80. 80
  81. 81
  82. 82
    Wechsler Test of Adult Reading: WTAR
    1. D Wechsler
    (2001)
    Psychological Corporation.
  83. 83
  84. 84
  85. 85
  86. 86

Decision letter

  1. Michael J Frank
    Reviewing Editor; Brown University, United States

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your article "Associability-modulated loss learning is increased in posttraumatic stress disorder" for consideration by eLife. Your article has been reviewed by two peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Richard Ivry as the Senior Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

Summary:

This study addresses the question of how the hypervigilance in PTSD patients can be accounted for from a computational psychiatric perspective. The authors provided both behavioral modeling and neuroimaging data to support the hypothesis that allocation of attention (associability weight) is higher in the PTSD compared to the control subjects; neural markers associated with associability but not prediction error, such as amygdala and insula showed elevated activity in PTSD during loss learning. This is a well-written and timely research article with interesting results. The quantitative approach to a specific clinical population should be of interest to a wide audience of basic research and translational studies.

Essential revisions:

1) Both reviewers noted that the modeling part was potentially more complex than needed. For example, the paper specified reward sensitivity and learning rate. However, these two parameters might covariate in parameter estimation. They also separated the chosen and unchosen option learning rates yet did not run a nested model comparison to test whether these additional parameters are necessary. From an empirical perspective, the authors might want to restrict their model to a simpler one or provide enough evidence to support a more complicated one. Also, Equation 3 does not seem to make sense as currently written.

2) Please add some additional detail regarding the task to the main text. It is difficult to understand the model without understanding the task. Within the task description in the supplemental materials, several aspects of the task remain unclear:

- What was the probability of better/worse outcomes for the less rewarded stimulus?

- What was the probability distribution of payoffs within the specified ranges, i.e. within 20-30 or 70-80 (uniform, Gaussian, etc.)?

- Were outcomes shown for both chosen and unchosen stimuli? If not, how did subjects learn from the unchosen stimulus?

3) The authors might want to clarify the logic behind titrating the length of the task to achieve certain level of better choice selection rates. Different performance in the task itself is a behavioral marker between the control and PTSD groups. In fact, Figure 2A suggests that there is a difference in performance between the groups. Also, given the design of the task, gain and loss blocks alternated. Within the gain block, associability in general shows a declining trend. Thus, it is important to disentangle a non-specific adaptation signal from the associability signal that the paper focuses on.

4) There are significant problems with the conceptual interpretation of the results:

-Based on the equations, associability value is expected (unsigned) prediction error:

Equation 6 can be rewritten as 𝛋A(t + 1) = 𝛋A(t) +η(|𝛅 t | – 𝛋A(t))

Associability weight is the learning rate in learning this associability value (i.e. in learning expected unsigned prediction error). Presumably, this learning rate should be high if the prediction errors themselves are expected to be more likely to change over time.

If this interpretation is correct, many descriptions of associability weight in the text are misleading. For example, "behavioral parameter estimates of associability…increased with PTSD" (Abstract), "[Associability weight] indicates the extent to which the magnitude of previous prediction errors is used to update trial-by-trial associability values", "associability weights, reflecting the degree to which stimulus values are modulated or not by associability during loss learning for each participant". Associability weight is actually a learning rate and so reflects the extent to which recent prediction errors (as opposed to older prediction errors) are weighted in updating associability values. This is different from stating associability weight is a measure of associability or that it reflects the extent to which associability is used to make decisions. This distinction is crucially important to understanding the results conceptually.

5) It seems that associability does not play a stronger role in subjects with PTSD. It also does not appear to be the case that PTSD subjects are systematically overestimating associability values. Rather, subjects with PTSD more heavily weight recent prediction errors (as opposed to older prediction errors) when estimating associability values; this may be due to a higher prior belief in the likelihood that prediction errors will change over time.

6) Simulations revealed that associability weight did not affect performance. Is higher associability weight adaptive, maladaptive, or neither?

7) Regarding neural data: Was the associability value that was used as a regressor the updated value for the current trial (i.e. incorporating prediction error on the current trial)? If so, it would reflect the unsigned prediction error on the current trial, especially if the associability weight is high. Is it possible that the neural results simply reflect a stronger response to unsigned prediction error, as opposed to signed prediction error, in PTSD? Even if the updated value was not used, it seems that the associability value should be correlated with unsigned predication error on current trial.

8) Was there a brain-behavior relationship independent of PTSD. It is also not clear whether adding PTSD symptoms to the brain-behavior relationship model improved model fit because the brain-behavior interaction was stronger in PTSD or whether the brain-behavior interaction was weaker in PTSD.

9) In the Introduction, the correspondence between the model and clinical constructs related to PTSD could be further clarified conceptually. Are unexpected events, reminders of negative events, and threats all being equated? Is associability the same as attention to threat?

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Associability-modulated loss learning is increased in posttraumatic stress disorder" for further consideration at eLife. Your revised article has been favorably evaluated by a Senior Editor, Michael Frank as the Reviewing Editor, and two reviewers.

The manuscript has been improved but there are some remaining issues that need to be addressed. Given their specificity and brevity, I've decided to simply append the reviewers’ requests here:

Reviewer 1:

1) If the previous version was lacking certain details, this version gave me an impression that they might have overdone it. The authors might want to tighten the Materials and methods (for example, the MCMC part can be placed into supplementary materials).

2) In the response to the reviewers, the authors justified their modeling approach and pointed to previous literatures in terms of including a decay factor in their model. I'm curious whether removing this parameter would significantly changes the results reported in the paper since it seems irrelevant to the scientific question the authors were interested in.

3) In Figure 2B, the plotted associability value and prediction error seems to be inversely correlated (albeit these values were generated using a moving average method). But in their model, as the authors stated in their response to the reviewers, current associability was generated using PEs from previous trial. And if they indeed correlate with each other, then how did they look for the neural correlates of PE and associability values at the same time? Also, the y-axis in Figure 2B was labeled as "parameter value", which should be "variable value".

4) I'm confused with Figure 3D and 4B, are they both neural correlates of associability values? And the difference is 3D refers to the associability neural correlates independent of PTSD diagnosis but 4B does?

5) The model free logistic regression mentioned that "the outcome history for stimulus A would be (2+1)/5 = 0.6". I have a hard time understanding why it should be defined this way.

Reviewer 2:

6) My primary remaining concern is that I am still having difficulty understanding the three-way interaction between neural encoding of associability value, prior trial outcomes, and PTSD symptoms to predict switching behavior (subsection “Neural substrates of associative learning in PTSD and relationship to behavioral choices”, last paragraph). It is difficult for me to understand what it means that the interaction is "positive" in PTSD subjects and "negative" in non-PTSD subjects, and how to interpret this. Since a small loss was coded as 1 and a large loss as 0, does this mean that in PTSD, small losses were associated with a stronger relationship between neural activity and switching, while in non-PTSD, large losses were associated with a stronger relationship between neural activity and switching? If so, how do the authors interpret this? I do not find the figure showing chi-squared values (Figure 5B) illuminating on this point.

https://doi.org/10.7554/eLife.30150.020

Author response

Essential revisions:

1) Both reviewers noted that the modeling part was potentially more complex than needed. For example, the paper specified reward sensitivity and learning rate. However, these two parameters might covariate in parameter estimation. They also separated the chosen and unchosen option learning rates yet did not run a nested model comparison to test whether these additional parameters are necessary. From an empirical perspective, the authors might want to restrict their model to a simpler one or provide enough evidence to support a more complicated one. Also, Equation 3 does not seem to make sense as currently written.

We appreciate the feedback. The reviewers point out three broad areas for clarification regarding our modeling approach: i) potential covariation of parameter estimates; ii) justification of the final model used; and iii) equation 3. We discuss each of these in turn below.

i) Potential covariation of parameter estimates. We recognize the possibility that model parameters such as reward sensitivity and learning rate may not be uniquely identifiable under some conditions. We tested this potential issue with our data and estimation approach by examining parameter recovery and did not find problems with estimating parameters accurately. That is, the parameters were recovered based on data simulated from a range of parameter values, indicating that each parameter is identifiable (see Figure 3—figure supplement 1C for parameter recovery for learning rate, reward sensitivity, associability weight, decay parameters). We have expanded on the parameter recovery analysis in the text:

“To test whether model parameters were independently recoverable and did not covary in a way that would prevent accurate estimation of each parameter separately, data were simulated based on different values of parameters along the range seen in participants’ data.”

In our experience, the model estimation approach taken in our paper (Hamiltonian Monte Carlo as implemented in Stan) deals with potentially correlated parameters better than other estimation approaches, and avoids the problem of poor identifiability with these parameters. We have plotted an expanded version of the parameter recovery results (Author response image 1, expanded from Figure 3—figure supplement 1C) to show that we are able to separably estimate learning rate and reward sensitivity; this figure illustrates the recovered reward sensitivity parameters at different levels of learning rate (LR) and reward sensitivity (RS).

Author response image 1
Identifiability of reward sensitivity and learning rate.

Learning rate (LR) and reward sensitivity (RS) were simulated at three levels (LR:.666,.75, and.825; RS: 1, 1.25, and 1.5), chosen based on the range of actual participants’ values; additional parameters of associability weight and unchosen learning rate were also simulated at three levels, resulting in 81 different combinations of parameters. The average recovered parameter value for each combination of learning rate and reward sensitivity is plotted above, showing that reward sensitivity is recovered well at each level of learning rate.

https://doi.org/10.7554/eLife.30150.018

In contrast, reward sensitivity and inverse temperature are not identifiable when used in the same model, and so we tested which parameter to retain as a free parameter and which to fix, based on improvements in model fit, and found that allowing reward sensitivity to vary as a free parameter improved fit (see next point), and therefore used the model including reward sensitivity and not inverse temperature.

ii) Justification of final model. The reviewers also suggested the need for further evidence to support the final model. In arriving at the final model, we compared model fit for a basic RL model with parameters of learning rate and inverse temperature versus our model (prior to adding the associability weight parameter), which has parameters of learning rate, reward sensitivity, and unchosen learning rate (now termed decay; see response to comment 2 below). This more complex model fit better (using AICc as a fit metric to penalize the additional complexity), with a protected exceedance probability of ~100%. We have added this analysis to the text:

“…we tested the fit of a basic reinforcement learning model, with parameters of learning rate and inverse temperature, versus our model with parameters of learning rate, reward sensitivity, and decay (prior to adding the additional parameter of associability weight) and found that this more complex model fit better, with a protected exceedance probability of 100%.”

iii) Clarification of equation 3. We apologize for the confusion with Equation 3 – there was a typographical error, and it has been corrected to read:

RA(t) = ρ RA(t)

Similarly, Equation 2 has been edited:

𝛅(t) = 𝐑′(t) − 𝐐(t)

2) Please add some additional detail regarding the task to the main text. It is difficult to understand the model without understanding the task. Within the task description in the supplemental materials, several aspects of the task remain unclear:

We regret the confusion. These questions are addressed in turn below, and the text has been edited as indicated. We have merged the task description that was previously in the supplemental methods with the relevant section in the main text Materials and methods. Complete task details are now provided into the Probabilistic learning task section of the Materials and methods.

- What was the probability of better/worse outcomes for the less rewarded stimulus?

The probability of better/worse outcomes for the less rewarded stimulus was.25/.75. We have added these details to the text:

“On each trial, participants were presented with two abstract stimuli. One stimulus had a higher (75%) probability of leading to a better monetary outcome and a lower probability (25%) of leading to a worse monetary outcome, while the probabilities for the other stimulus were reversed (i.e., smaller probability of better outcome and larger probability of worse outcome).”

- What was the probability distribution of payoffs within the specified ranges, i.e. within 20-30 or 70-80 (uniform, Gaussian, etc.)?

The payoffs for the better and worse outcomes were calculated at the beginning of each block, fixed for the remainder each block, initialized at 20, 25, or 30 cents, and 50 cents apart. Better/worse outcome pairs were drawn (with replacement) from a uniform distribution from this set, so for each block payoffs were randomly chosen to be +/-20 and +/- 70, +/-25 and +/-75, or +/-30 and +/-80 cents. We have added these details to the text:

“At the beginning of each block, high and low outcome values were randomly chosen with replacement from uniformly distributed outcome pairs {20,70}, {25,75}, or {30,80} and kept consistent within blocks (the outcome displays indicated ‘You Lose [amount]’ for loss blocks, and ‘You Gain [amount]’ for gain blocks).”

- Were outcomes shown for both chosen and unchosen stimuli? If not, how did subjects learn from the unchosen stimulus?

Outcomes were shown only for the chosen stimulus; participants did not see the outcome of the unchosen stimulus. We acknowledge our original use of the term ‘unchosen learning rate’ was confusing in this context, as it implies updating based on counterfactual learning. Therefore, we have changed the nomenclature of this parameter to a ‘decay’ parameter, as the parameter functions to measure the extent to which participants forget the previously learned value of the unchosen stimuli (i.e., how quickly the value of unchosen options decay, as in: Boorman, Behrens, Woolrich, & Rushworth, 2009; Cavanagh, 2015; Collins & Frank, 2016; Niv et al., 2015; and others).

We have reworded the relevant text and changed the citations to similar models to read: “A decay parameter for the unchosen option B (γ; range 0+) (Boorman et al., 2009; Cavanagh, 2015; Collins and Frank, 2016; Niv et al., 2015) was also included.”

Additionally, we have edited other references to the “unchosen" learning rate parameter to now refer to the “decay” parameter.

We have also clarified this aspect of the task description in the text:

“The participant’s choice was framed for a jittered viewing time of 2-4 seconds, after which the outcome (monetary amount gained or lost) was shown for 2 seconds. Participants were not shown the outcome associated with the unchosen stimulus.”

3) The authors might want to clarify the logic behind titrating the length of the task to achieve certain level of better choice selection rates. Different performance in the task itself is a behavioral marker between the control and PTSD groups. In fact, Figure 2A suggests that there is a difference in performance between the groups. Also, given the design of the task, gain and loss blocks alternated. Within the gain block, associability in general shows a declining trend. Thus, it is important to disentangle a non-specific adaptation signal from the associability signal that the paper focuses on.

As the reviewer comment suggests, patient groups often differ in their speed or other behavioral markers of learning. When designing our learning task we wanted to maximize the likelihood that all participants, regardless of psychopathology, would achieve learning. Therefore, rather than have all participants complete blocks of equal numbers of trials, which may either a) extend beyond learning and lead to task disengagement or b) end prematurely while learning is still ongoing, we allowed block length to vary based on performance. An adaptive design thus allowed us to achieve the goal of examining differences in model-based processes associated with successful learning. We have added information about this to the text:

“As our goal was to examine mechanistic processes associated with learning, including in individuals who may differ in learning, we used an adaptive algorithm to ensure that participants successfully learned the contingencies and continued to learn throughout the task.”

We note that our participants with PTSD and no PTSD were in fact comparable in overall learning (measured by proportion of better options chosen, as noted in the last paragraph of the subsection “Participant characteristics and model- behavioral performance”), block length, and number of blocks completed (subsection “Probabilistic learning task”, second paragraph). We also note that in Figure 2A, the shaded area represents +/-1 SEM from the mean for each group and that these areas overlap throughout learning, supporting our quantitative findings of lack of overall performance differences in veterans with and without PTSD.

To address the possibility that associability values are confounded with adaptation or time spent on the task, we correlated trial number against ongoing associability value. These values were not correlated, with an average correlation across subjects of -.116. Additionally, we controlled for trial number when assessing the relationship between associability value and reaction time (subsection “Behavioral model-fitting and relationship of model parameters with PTSD”, second paragraph and Figure 3B); trial number itself was not related to reaction time while associability value was. These results provide converging evidence that associability and a general adaptation trend are well dissociated during loss learning in our task.

We have added information about this to the text:

“Trial number was included as a covariate in these analyses to account for general adaptation effects; however, trial number and associability were uncorrelated (average correlation across subjects: r = -.12).”

As the associability model did not fit gain learning well, we did not investigate associability during the gain blocks.

4) There are significant problems with the conceptual interpretation of the results:

-Based on the equations, associability value is expected (unsigned) prediction error:

Equation 6 can be rewritten as 𝛋A(t + 1) = 𝛋A(t) +η(|𝛅 t | – 𝛋A(t))

Associability weight is the learning rate in learning this associability value (i.e. in learning expected unsigned prediction error). Presumably, this learning rate should be high if the prediction errors themselves are expected to be more likely to change over time.

If this interpretation is correct, many descriptions of associability weight in the text are misleading. For example, "behavioral parameter estimates of associability…increased with PTSD" (Abstract), "[Associability weight] indicates the extent to which the magnitude of previous prediction errors is used to update trial-by-trial associability values", "associability weights, reflecting the degree to which stimulus values are modulated or not by associability during loss learning for each participant". Associability weight is actually a learning rate and so reflects the extent to which recent prediction errors (as opposed to older prediction errors) are weighted in updating associability values. This is different from stating associability weight is a measure of associability or that it reflects the extent to which associability is used to make decisions. This distinction is crucially important to understanding the results conceptually.

We acknowledge the reviewers’ comments on the interpretation of our model and fully agree that associability weight acts as a learning rate for learning associability values.

We believe that two areas in which we were unclear formed the basis of this reviewer comment. Specifically, we were i) inconsistent in specifying when we meant associability value of the stimulus vs associability weight and also ii) imprecise in how we previously described the function of the associability weight. To address these issues, we have first clarified throughout the text in each mention of ‘associability’ whether associability refers to the associability value of the stimulus or the associability updating weight. Second, we have also clarified that associability weight functions as a learning rate with which the associability value is updated (as the reviewers indicate, this reflects the degree to which recent prediction errors are weighted in updating associability values). Specifically, we have revised the passages noted by the reviewers, and other text throughout, to clarify that the associability weight is a learning rate and, as such, functions as an updating parameter:

Abstract: “Neural substrates of associability value and behavioral parameter estimates of associability updating, but not prediction error, increased with PTSD during loss learning. Moreover, the interaction of PTSD severity with neural markers of associability value predicted behavioral choices.”

Introduction: “Furthermore, PTSD severity, particularly symptoms related to hyperarousal (Lissek and van Meurs, 2014), should correlate preferentially both with enhanced associability updating and with increased activity in neural structures encoding associability (i.e., amygdala and insula).”

Results: “In this model, the associability value κ of a chosen stimulus changes on a trial-by-trial basis based on a combination of the magnitude of previous prediction errors and a static associability weight η, a parameter which varies by participant and indicates the extent to which the magnitude of recent prediction errors updates trial-by-trial associability values…”

Results: “To test this possibility, participants’ individually estimated associability weights, reflecting the degree to which associability values are updated based on recent unsigned prediction errors during loss learning for each participant, and unmodulated learning rates (α) were compared between participants with and without a PTSD diagnosis…”

Results: “Corroborating the behavioral findings of greater associability updating in participants with PTSD, associability value showed a significant relationship with PTSD at an FDR-corrected whole brain significance level…”

Discussion: “We posited that the hypersensitivity may derive from either i) heightened solely error-related learning that increases learning directly from unexpected outcomes, or ii) heightened updating of associability values in which disproportionate attention is dynamically allocated in a way that magnifies the effects of prediction errors in future decision-making.”

Discussion: “neither static error-based learning rate nor neural prediction error responses were associated with PTSD, while dynamic attentional modulation (i.e., associability updating) of loss learning signals increased with PTSD at both the behavioral and neural levels.”

Discussion: “While the increased updating of associability values observed here in PTSD during monetary loss learning…”

5) It seems that associability does not play a stronger role in subjects with PTSD. It also does not appear to be the case that PTSD subjects are systematically overestimating associability values. Rather, subjects with PTSD more heavily weight recent prediction errors (as opposed to older prediction errors) when estimating associability values; this may be due to a higher prior belief in the likelihood that prediction errors will change over time.

To clarify, participants with PTSD exhibited greater associability weights than those without PTSD. We fully agree with the interpretation that these increased weights indicate that participants with PTSD more heavily weight recent prediction errors when estimating associability values. We believe that some lack of precision in describing the function of associability weight in our original submission contributed to this reviewer comment. We clarify below:

Specifically, behavioral parameter estimates of associability weight (see also point 4 above) are increased in PTSD participants (shown in Figure 4A, main text). This increased associability weight indicates, per the reviewer comment, that PTSD participants more heavily weight recent prediction errors when estimating associability values. We have edited our text to clarify this (see changes listed in response to point 4 above).

This increased associability weight may indeed derive from a prior belief in a greater chance of fluctuating prediction errors, but also could arise from differences in processing prediction errors during the task or from other biases in updating associability values. Future work could manipulate and disentangle these and other potential sources of increased associability updating in PTSD. We have added these possibilities to the Discussion:

“The increased updating of associability values in PTSD may reflect a higher prior belief that changes in prediction error will occur, a difference in processing changes in prediction errors, or a combination, which could also serve as treatment targets.”

6) Simulations revealed that associability weight did not affect performance. Is higher associability weight adaptive, maladaptive, or neither?

We appreciate this question. Whether higher associability weight is adaptive or maladaptive likely depends on characteristics of the environment. That is, in situations involving unpredictable shifts in how often contingencies change, higher associability weight would be adaptive, but in more stable situations it would be maladaptive. Our task design likely falls in between these two extremes, and in environments similar to this task, associability weight is neither adaptive nor maladaptive. A mix of context-dependent adaptive and maladaptive effects has been found in other aspects of altered emotional processing and attention in PTSD (Vythilingam et al., 2007; Zinchenko et al., 2017).

We have added this information to the Discussion:

“While the increased updating of associability observed here in PTSD during monetary loss learning should apply generally to learning from other classes of negative stimuli, it will be important in future investigations to explicitly test the role of associability-based learning for PTSD in the context of threatening or trauma-related cues which have been shown to impair fear extinction in the disorder (Milad et al., 2009; Norrholm et al., 2011) as well as to connect our findings to models of other behavioral disruptions in mood/anxiety (e.g., (Mkrtchian, Aylward, Dayan, Roiser, and Robinson, 2017)) and to explore under what conditions the greater associability weight in PTSD is adaptive versus maladaptive (as in (Vythilingam et al., 2007; Zinchenko et al., 2017)).”

7) Regarding neural data: Was the associability value that was used as a regressor the updated value for the current trial (i.e. incorporating prediction error on the current trial)? If so, it would reflect the unsigned prediction error on the current trial, especially if the associability weight is high. Is it possible that the neural results simply reflect a stronger response to unsigned prediction error, as opposed to signed prediction error, in PTSD? Even if the updated value was not used, it seems that the associability value should be correlated with unsigned predication error on current trial.

The associability value used as a regressor in all neural analyses was the value prior to updating on the current trial and so did not include the current trial’s prediction error. We have clarified this in the Materials and methods section:

“Regressors for imaging analyses used the prediction error, associability value, and probability at trial t, prior to updating associability value for the next trial.”

To test the relationship between unsigned prediction error and PTSD, we ran an additional neural analysis with unsigned prediction error as a trial-by-trial regressor. This analysis showed robust relationships between unsigned prediction error and BOLD activation across all subjects, similar to previous studies (Author response image 2), but when regressed against PTSD diagnosis at the group level, showed no modulation by PTSD. This signal partially overlaps with, but is distinct from, the associability value-related brain activation shown in Figure 3D.

Author response image 2
Unsigned prediction error signal independent of PTSD.

Neural signaling of unsigned

https://doi.org/10.7554/eLife.30150.019

prediction error across PTSD and control participants, FDR corrected at p<.05 (p<.001 cluster defining threshold). As illustrated, participants showed a robust signal for unsigned prediction error, but this signal was not related to PTSD.</Author response image 2 title/legend>

We additionally tested the relationship between behavioral estimates of associability value and unsigned prediction error. Given the relationship in our model between unsigned prediction error and associability weight, we expected these measures to be related but not collinear. Consistent with this expectation, there was a moderate correlation between associability value and unsigned prediction error (average correlation across participants: r =.333). Using the Fisher’s r to z transformation on this correlation, we tested the relationship of this correlation with the associability weight parameter across participants and again found a moderate relationship (r =.25, p =.04); similarly, the relationship between unsigned prediction error and associability value was nearing a significant difference for participants with versus without PTSD (t66 = 1.92, p =.06). Therefore, associability value and unsigned prediction error are related, as would be predicted conceptually, and are more strongly related with higher associability weights. However, associability value and unsigned prediction error remain distinctly separate measurements, sharing around 10% of their variance in our sample on average. Thus, the increased associability weight parameter and increased neural reactivity to associability value in participants with PTSD cannot be solely attributed to unsigned prediction error.

8) Was there a brain-behavior relationship independent of PTSD. It is also not clear whether adding PTSD symptoms to the brain-behavior relationship model improved model fit because the brain-behavior interaction was stronger in PTSD or whether the brain-behavior interaction was weaker in PTSD.

As illustrated in main text Figure 3D, neural encoding of associability value is present independent of PTSD. Additional analyses analogous to main text Figure 5Bbut conducted within non-PTSD participants verified a robust brain-behavior relationship independent of PTSD (i.e., interaction of outcomes and neural encoding of associability values predicting switching behavior; insula: χ2 = 19.0, p <.001; amygdala χ2 = 13.6, p =.001).

As previously reported, across all participants neural encoding of associability value interacted with prior trial outcomes and PTSD symptoms to predict switching behavior (main text Figure 5B). The brain-behavior relationship (i.e., interaction of outcomes and neural encoding of associability values predicting switching behavior) was in the opposite direction for the PTSD and non-PTSD participants (positive in PTSD and negative in non-PTSD). These data thus indicate opposite brain-behavior interactions in participants with and without PTSD (rather than a weaker or stronger relationship in one group) whose direction is dependent on diagnosis.

We have added information about this to the Results section:

“The brain-behavior relationship (i.e., interaction of outcomes and neural encoding of associability values predicting switching behavior) was in the opposite direction for the PTSD and non-PTSD participants (positive in PTSD and negative in non-PTSD).”

9) In the Introduction, the correspondence between the model and clinical constructs related to PTSD could be further clarified conceptually. Are unexpected events, reminders of negative events, and threats all being equated? Is associability the same as attention to threat?

We appreciate this point. We did not intend to equate or differentiate among the listed clinical constructs and have attempted to clarify both this and the connections between the model and clinical constructs in the Introduction.

Clinically, the consequences of unexpected negative events and reminders of negative events (particularly trauma) often overlap in individuals with PTSD (Dunsmoor and Paz, 2015), and patients view unexpected, uncontrollable, or trauma-related experiences as threatening because of the connection to previous traumatic events (Craske, Treanor, Conway, Zbozinek, and Vervliet, 2014; Schnurr et al., 2007). We posit that altered associability-based learning in PTSD likely contributes to some of the consequences (particularly hyperarousal symptoms) of threatening events and may lend precision to understanding which of these constructs are most relevant, but is not the same as the clinical constructs themselves (including attention to threat). Rather associability-based learning and related model-based approaches provide mechanistic insight as to how these unexpected events, reminders of negative events etc. trigger symptoms and whether they and their sequelae are (or are not) related.

For example, neural and behavioral responses to prediction errors may track well with clinical responses to unexpected events, and responses to associability values of stimuli may track well with responses to reminders of unexpected negative events, and both may be threatening. Although such potential connections are intriguing, we are reluctant (given the sparseness of the clinical literature and the early stages of the model-based evidence) to make strong statements about precise correspondences between model components and such clinical constructs. The data we present here are nonetheless a promising step in this direction and indicate that heightened updating of associability values of stimuli (which cue or remind of previous events) specifically is related to hyperarousal symptoms in PTSD. Future studies will be required to need to empirically and thoroughly test the mapping of connections between model and clinical constructs related to PTSD. We have clarified the relevant portion of the Introduction:

“In particular, clinical and empirical observations have documented the negative impact of salient cues on neural and behavioral functioning in PTSD, including heightened orienting to unexpected events, impaired extinction of learned fear, and unstable attention biases toward perceived threatening stimuli (Aupperle, Melrose, Stein, and Paulus, 2012; Bar-Haim, Lamy, Pergamin, Bakermans-Kranenburg, and van IJzendoorn, 2007; Blair et al., 2013; Morey et al., 2009; Naim et al., 2015). Together, these behavioral alterations in response to unexpected stimuli, uncontrollable reminders of trauma, and other negative, threatening, and trauma-related events point to PTSD as a disorder of disrupted learning from reminders of negative events; however, the specific components of anomalous learning in PTSD remain unknown.”

And:

“Computational model-based approaches to learning provide a mechanistic framework for understanding the detrimental impact of unexpected negative stimuli and reminders of negative events in PTSD.”

We have also incorporated possible extensions of our model to clinical concepts in the Discussion:

“The clinical manifestation of hyperarousal in PTSD includes excessive sustained vigilance (American Psychiatric Association, 2000) and resembles what heightened updating of associability values would predict; specifically, reminders of trauma can be conceptualized as stimuli associated with unexpected negative outcomes, with the greater associability updating seen here in PTSD causing stimuli associated with a recent history of these surprising outcomes to command greater attention and increased updating.”

[Editors' note: further revisions were requested prior to acceptance, as described below.]

The manuscript has been improved but there are some remaining issues that need to be addressed. Given their specificity and brevity, I've decided to simply append the reviewers’ requests here:

Reviewer 1:

1) If the previous version was lacking certain details, this version gave me an impression that they might have overdone it. The authors might want to tighten the Materials and methods (for example, the MCMC part can be placed into supplementary materials).

We appreciate the concern for the need to balance brevity and explanation in the Materials and methods section. In preparing the resubmission, we reformatted portions of the manuscript to conform with eLife formatting guidelines. We acknowledge that combining the main text methods and supplementary methods as a part of this reformatting has made the methods somewhat unwieldly, and so have moved some of the methods, specifically those pertaining to MCMC parameter estimation and model free behavioral analyses, to Appendix 1: supplementary methods.

2) In the response to the reviewers, the authors justified their modeling approach and pointed to previous literatures in terms of including a decay factor in their model. I'm curious whether removing this parameter would significantly changes the results reported in the paper since it seems irrelevant to the scientific question the authors were interested in.

Thanks for this comment. We assert that the decay parameter is relevant to our question and explains significant aspects of behavior in our task, as evidenced by the improvement in model fit. Consistent with the role of this parameter in governing the retention of unchosen values, the decay parameter is highly correlated with participants’ model free proportion of switching between options (r =.664, p <.001), indicating that this parameter is related to important aspect of participants’ behavior that should not be neglected in our model. However, as we did not hypothesize that the retention of unchosen values would be responsible for different learning patterns in PTSD, we did not focus on this parameter when examining learning differences in PTSD.

We nonetheless tested the robustness of the results without the decay parameter (and preserving associability weight and learning rate), as suggested by this reviewer. To do so, we used the poorer fitting model without the decay parameter, and examined the relationship between the individual associability weight parameters estimated from this model with the associability weight parameters from the full model. The associability weight parameters from these two models were highly correlated, r =.75, p <.001, suggesting that the inclusion of the decay parameter did not have a large effect on estimates of the associability weight parameter. We then tested the relationships between PTSD and loss associability weight and learning rate, as described in the primary behavioral analyses, with parameters from the model without the decay parameter. The relationship between PTSD and associability weight remained significant (t62 = 3.36, p =.001), and the relationship between PTSD and learning rate remained nonsignificant (t62 =.035, p =.7). Therefore, the inclusion of the decay parameter does not substantively affect the results or conclusions reported in the paper.

3) In Figure 2B, the plotted associability value and prediction error seems to be inversely correlated (albeit these values were generated using a moving average method). But in their model, as the authors stated in their response to the reviewers, current associability was generated using PEs from previous trial. And if they indeed correlate with each other, then how did they look for the neural correlates of PE and associability values at the same time? Also, the y-axis in Figure 2B was labeled as "parameter value", which should be "variable value".

We fully agree with the importance of ensuring that associability value and prediction error measure separable influences on behavior. As this reviewer notes, Figure 2B uses a moving average method and so does not reflect the exact trial-wise relationship between prediction error and associability value. The actual relationship between signed prediction error and associability value is similar to the relationship between unsigned prediction error and associability value that was discussed in the previous response to reviewers- it is present but minor. Specifically, the average correlation between associability value and signed prediction error across participants is -.336, meaning that these two variables share about 11% of their variance. The independence of these measures is reflected in the fact that adding prediction error as the first parametric modulator, and associability value as the second parametric modulator, to first level imaging analyses results in a similar relationship at the group level between associability value and PTSD as when prediction error is not included.

We appreciate the feedback on the figure axis label – Figure 2B’s y-axis now reads ‘variable value’ as suggested.

4) I'm confused with Figure 3D and 4B, are they both neural correlates of associability values? And the difference is 3D refers to the associability neural correlates independent of PTSD diagnosis but 4B does?

The reviewer is correct. Both are correlates of associability value, and Figure 3D shows the intercept term in the regression of PTSD on neural associability values, and so shows the neural correlates of associability value independent of PTSD diagnosis. Meanwhile, Figure 4B shows the PTSD term in this regression and so shows the relationship between PTSD diagnosis and neural associability values.

We have clarified this distinction in the description of these results:

“…neural encoding of associability values showed a significant relationship with PTSD at an FDR-corrected whole brain significance level in a network of regions including bilateral amygdala and insula, hypothesized areas of relevance for PTSD and associability-based learning1,12 (Figure 4B; Supplementary file 1, Table 1C; see Figure 3D for associability-related signaling across participants after accounting for PTSD and covariates).”

and in the legend for Figure 3D:

“(D): Associability signaling independent of PTSD and covariates, displayed at p <.05 FDR corrected.”

5) The model free logistic regression mentioned that "the outcome history for stimulus A would be (2+1)/5 = 0.6". I have a hard time understanding why it should be defined this way.

As part of model free analog of associability modulated learning, we calculated expectedness as the probability of receiving the better outcome with a particular stimulus choice, and then used the difference between this expected outcome and whether or not the outcome was actually better. The outcome history for a particular stimulus in this analysis was therefore intended as the probability that choosing a stimulus would lead to the better outcome the next time it was chosen and was defined the sum of a) better outcomes after choosing that stimulus (2 in the quoted example) and b) worse outcomes after choosing the other stimulus (1 in the example), divided by c) the number of choices to that point in the block (5 in the example). This value approximated the predicted outcome on the next trial that this stimulus was chosen, and was subtracted from the actual outcome to represent the level of surprise or unexpectedness for use in the logistic regression.

Reviewer 2:

6) My primary remaining concern is that I am still having difficulty understanding the three-way interaction between neural encoding of associability value, prior trial outcomes, and PTSD symptoms to predict switching behavior (subsection “Neural substrates of associative learning in PTSD and relationship to behavioral choices”, last paragraph). It is difficult for me to understand what it means that the interaction is "positive" in PTSD subjects and "negative" in non-PTSD subjects, and how to interpret this. Since a small loss was coded as 1 and a large loss as 0, does this mean that in PTSD, small losses were associated with a stronger relationship between neural activity and switching, while in non-PTSD, large losses were associated with a stronger relationship between neural activity and switching? If so, how do the authors interpret this? I do not find the figure showing chi-squared values (Figure 5B) illuminating on this point.

We apologize for the confusion. As previously reported, across all participants, neural encoding of associability value interacted with prior trial outcomes and PTSD symptoms to predict switching behavior (main text Figure 5B). To parse this interaction, we examined the effects of the interaction of neural associability value with PTSD diagnosis outside of the interaction with previous outcomes in predicting switching behavior. The main effect in this model of the effect of neural associability value in predicting switching was negative for both ROIs (insula: z = -2.58; amygdala: z = -2.02), meaning that greater neural reactivity to associability value in these ROIs predicts less overall switching independent of previous outcomes. However, the interaction with PTSD for these ROIs was positive (z values of 2.39 for insula and 1.96 for amygdala), indicating that greater neural reactivity to associability value is related to a greater tendency to switch choices in people with PTSD, again independent of previous outcomes. Therefore, participants with PTSD show an overall higher tendency to switch with higher neural responsivity to associability values relative to controls.

The remaining portion of the interaction, the effect of the value of previous outcomes by diagnosis, depends on the region (amygdala versus insula), which was unclear in our previous response. For amygdala responsivity to associability value, participants with PTSD had a greater relationship between neural responsivity and switching behavior following large losses compared to small losses, as did the controls (see Figure 5—figure supplement 2A). For insula responsivity to associability value, participants with PTSD had a smaller relationship between neural responsivity and switching following large versus small losses (Figure 5—figure supplement 2B). Our previous response primarily examined the relationship between insula responsivity and switching, and so we apologize for the imprecision in our response. The difference in this part of the interaction between these two brain regions is interesting and could be followed up in future work.

We have added information describing the direction of the interaction to the manuscript:

“A secondary analysis inspecting the components of this interaction revealed overall greater tendency to switch with greater associability value-related neural activation in participants with PTSD relative to controls, with additional effects of large relative to small outcomes leading to a greater tendency to switch in controls in the amygdala and insula ROIs and in PTSD in the amygdala ROI; the effect of outcomes in PTSD in the insula ROI was reversed (Figure 5—figure supplement 2A and 2B).”

We have also added a plot of the effects of neural responsivity to associability value on switching for each level of outcomes (small vs. large loss) and group (PTSD vs. control) as Figure 5—figure supplement 2A and 2B.

We agree with this reviewer that understanding the interaction effects is important; however, the primary finding from this analysis was that neural responsivity to associability value, and not prediction error, interacted with hyperarousal and avoidance/numbing, but not reexperiencing, symptoms of PTSD to predict behavior. Due to the complexity of the three-way interaction, the multiple comparisons presented, and concerns about unreliability of specific parameter estimates in multilevel models, we believe the chi-squared values are a straightforward presentation of the primary aspect of the interaction results. For readers interested in a further explanation of the interaction, though, we do agree that additional explication is useful and so have added the text and figures referenced above as supplements to Figure 5.

https://doi.org/10.7554/eLife.30150.021

Article and author information

Author details

  1. Vanessa M Brown

    1. Virginia Tech Carilion Research Institute, Roanoke, United States
    2. Department of Psychology, Virginia Tech, Blacksburg, United States
    Contribution
    Software, Formal analysis, Validation, Visualization, Methodology, Writing—original draft, Writing—review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-0284-0890
  2. Lusha Zhu

    1. Virginia Tech Carilion Research Institute, Roanoke, United States
    2. School of Psychological and Cognitive Sciences, IDG/McGovern Institute for Brain Research, Beijing Key Laboratory of Behavior and Mental Health, Peking–Tsinghua Center for Life Sciences, Peking University, Beijing, China
    Contribution
    Software, Visualization, Methodology, Writing—review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8717-6356
  3. John M Wang

    1. Virginia Tech Carilion Research Institute, Roanoke, United States
    2. Department of Psychology, Virginia Tech, Blacksburg, United States
    Contribution
    Software, Visualization, Methodology, Writing—review and editing
    Competing interests
    No competing interests declared
  4. B Christopher Frueh

    University of Hawaii, Hilo, United States
    Contribution
    Conceptualization, Supervision, Project administration, Writing—review and editing
    Competing interests
    No competing interests declared
  5. Brooks King-Casas

    1. Virginia Tech Carilion Research Institute, Roanoke, United States
    2. Department of Psychology, Virginia Tech, Blacksburg, United States
    3. Salem Veterans Affairs Medical Center, Salem, United States
    4. School of Biomedical Engineering and Sciences, Virginia Tech-Wake Forest University, Blacksburg, United States
    Contribution
    Conceptualization, Resources, Formal analysis, Supervision, Funding acquisition, Visualization, Methodology, Project administration, Writing—review and editing
    Contributed equally with
    Pearl H Chiu
    For correspondence
    bkcasas@vtc.vt.edu
    Competing interests
    No competing interests declared
  6. Pearl H Chiu

    1. Virginia Tech Carilion Research Institute, Roanoke, United States
    2. Department of Psychology, Virginia Tech, Blacksburg, United States
    3. Salem Veterans Affairs Medical Center, Salem, United States
    Contribution
    Conceptualization, Resources, Formal analysis, Supervision, Funding acquisition, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing
    Contributed equally with
    Brooks King-Casas
    For correspondence
    chiup@vtc.vt.edu
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-8767-7406

Funding

National Natural Science Foundation of China (31671171)

  • Lusha Zhu

National Natural Science Foundation of China (31630034)

  • Lusha Zhu

Department of Veteran Affairs (D7030R)

  • Brooks King-Casas

National Institutes of Health (MH087692)

  • Pearl H Chiu

National Institutes of Health (MH106756)

  • Pearl H Chiu

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

This work was supported in part by the National Institutes of Health (MH087692, MH106756 to PC), the Department of Veteran Affairs (D7030R to BKC), and the National Natural Science Foundation of China (31671171, 31630034 to LZ). We acknowledge Read Montague and the assistance of Rizwan Ali, George Christopoulos, Dongil Chung, Alec Solway, Jessica Eiseman, Katherine Gardner, David Graham, Jacob Lee, Katherine McCurry, Robert McNamara, Cari Rosoff, Dharol Tankersley, and Wright Williams.

Ethics

Human subjects: All participants provided written informed consent. Study procedures were reviewed and approved by the Institutional Review Boards of Baylor College of Medicine, the Salem Veterans Affairs Medical Center, and Virginia Tech (protocols H-22764, H-21660; BKC-0001; 15-965, 10-1044).

Reviewing Editor

  1. Michael J Frank, Brown University, United States

Publication history

  1. Received: July 14, 2017
  2. Accepted: December 2, 2017
  3. Version of Record published: January 9, 2018 (version 1)

Copyright

© 2018, Brown et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 1,244
    Page views
  • 134
    Downloads
  • 3
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, Scopus, PubMed Central.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)

Further reading

    1. Neuroscience
    Jane Anne Horne et al.
    Tools and Resources Updated
    1. Neuroscience
    Éric Martineau et al.
    Research Article Updated