Nucleus accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error

  1. Jessica N Goedhoop
  2. Bastijn JG van den Boom
  3. Rhiannon Robke
  4. Felice Veen
  5. Lizz Fellinger
  6. Wouter van Elzelingen
  7. Tara Arbab
  8. Ingo Willuhn  Is a corresponding author
  1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Netherlands
  2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Netherlands

Abstract

There is active debate on the role of dopamine in processing aversive stimuli, where inferred roles range from no involvement at all, to signaling an aversive prediction error (APE). Here, we systematically investigate dopamine release in the nucleus accumbens core (NAC), which is closely linked to reward prediction errors, in rats exposed to white noise (WN, a versatile, underutilized, aversive stimulus) and its predictive cues. Both induced a negative dopamine ramp, followed by slow signal recovery upon stimulus cessation. In contrast to reward conditioning, this dopamine signal was unaffected by WN value, context valence, or probabilistic contingencies, and the WN dopamine response shifted only partially toward its predictive cue. However, unpredicted WN provoked slower post-stimulus signal recovery than predicted WN. Despite differing signal qualities, dopamine responses to simultaneous presentation of rewarding and aversive stimuli were additive. Together, our findings demonstrate that instead of an APE, NAC dopamine primarily tracks prediction and duration of aversive events.

Editor's evaluation

The article by Goedhoop et al. provides an important analysis of the role of terminal dopamine release in the nucleus accumbens in processing aversive events that will be of value to researchers interested in the neural mechanisms of reinforcement learning and computational modeling of dopamine function. Using a variety of conditions, the authors provide convincing data in support of the role of accumbal dopamine release in processing aversive events that situate the current report among growing interest and mounting investigations into the role of dopamine in aversion.

https://doi.org/10.7554/eLife.82711.sa0

Introduction

The midbrain dopamine system plays critical roles in motivation, learning, and movement; specifically for learning about rewards and creating motivational states that promote reward-seeking (Berridge and Robinson, 1998; Bromberg-Martin et al., 2010; Berke, 2018; Schultz, 2019). One of the most prominent functions of dopamine is the encoding of a so-called reward prediction error (RPE) signal (Schultz et al., 1997): when a reward is fully predicted by a cue, the increase in dopamine cell firing and terminal release of dopamine shifts ‘backward’ in time from the moment of reward delivery, to that of cue presentation (Schultz et al., 1997; Flagel et al., 2011). Furthermore, dopamine neurons pause their firing when a predicted reward is omitted and increase their firing in response to the delivery of an unpredicted reward. Thus, dopamine neurons encode the difference between predicted and obtained reward, which is corroborated by the fact that dopamine neuron activity scales with the relative value of reward and unexpected deviations from this value (Bromberg-Martin and Hikosaka, 2009).

Although the vast majority of studies focus on the relationship between dopamine and stimuli with a positive valence (rewards), the relevance of the dopamine system in processing stimuli with the opposite valence (aversive) has also generated great interest. In contrast to the primarily stimulatory response of rewards on dopamine activity, the reports on the effect of aversive events on the dopamine system are less consistent. For example, on the level of dopamine neuron cell bodies, aversive stimuli were demonstrated to result in inhibition of neuronal activity (Ungless et al., 2004; Mileykovskiy and Morales, 2011), excitation thereof (Anstrom et al., 2009; Valenti et al., 2011), or no effect at all (Mirenowicz and Schultz, 1996; Fiorillo, 2013). The widely accepted explanation for these varying results is that subpopulations of dopamine neurons exhibit different response profiles to aversive stimuli (Schultz and Romo, 1987; Guarraci and Kapp, 1999; Coizet et al., 2006; Bromberg-Martin and Hikosaka, 2009; Zweifel et al., 2011; Cohen et al., 2012; Lammel et al., 2011), whereby variance is presumably introduced by different types of aversive stimuli, by the fact that some studies were performed in awake and others in anesthetized animals, and by the location and projection targets of the recorded dopamine neurons (Brischoux et al., 2009; Matsumoto and Hikosaka, 2009; Lammel et al., 2011). However, activity at the level of dopamine neuron cell bodies does not necessarily always translate to their projection targets (Mohebi et al., 2019) as axonal terminal release of dopamine may be capable of operating independently from cell body activity (Threlfell et al., 2012). Therefore, in interrogating the entire spectrum of functions of the dopamine system, it is imperative to include measurements of extracellular dopamine concentrations in the projection target.

Midbrain dopamine neurons modulate their targets via population signals: dopamine release from a large number of extrasynaptic terminals, combined, constitutes a diffusion-based signal that is perpetuated by volume transmission (Rice and Cragg, 2008). The vast majority of projections from dopaminergic neurons target the striatum and its subregions. Inconsistent with the classic hypothesis positing that the dopamine system broadcasts a uniform signal across the striatum, it has been reported multiple times in recent years that dopamine signals display regional heterogeneity (Willuhn et al., 2012; Willuhn et al., 2014a, Lammel et al., 2011; de Jong et al., 2019; Menegas et al., 2017; Klanker et al., 2017; van Elzelingen et al., 2022a; van Elzelingen et al., 2022b). This heterogeneity is reflected in dopamine responses to aversive events throughout the striatum: whereas microdialysis studies report an increase in dopamine release in the nucleus accumbens in response to aversive events (Young et al., 1993; Young, 2004; Wilkinson et al., 1998; Bassareo et al., 2002; Pascucci et al., 2007; Ventura et al., 2007; Martinez et al., 2008 but see Mark et al., 1991; Liu et al., 2008), studies employing techniques with a higher, subsecond temporal resolution (e.g., fast-scan cyclic voltammetry [FSCV] or fluorescence fiber photometry) arrive at less consistent conclusions. For example, aversive stimuli produced an increase in dopaminergic activity in the nucleus accumbens shell (NAS) in some studies (Badrinarayan et al., 2012; de Jong et al., 2019), but a decrease in others (Roitman et al., 2008; Wheeler et al., 2011; McCutcheon et al., 2012; Twining et al., 2015). Similarly, contradictory findings are also reported in the neighboring nucleus accumbens core (NAC), where studies found both increased (Budygin et al., 2012; Mikhailova et al., 2019; Kutlu et al., 2021; Kutlu et al., 2022) and decreased dopamine activity (Badrinarayan et al., 2012; Oleson et al., 2012; de Jong et al., 2019; Stelly et al., 2019). In contrast, in the tail of the striatum, aversive events exclusively result in increased dopaminergic activity (Menegas et al., 2017; Menegas et al., 2018). Overall, it can be concluded that most studies observe a change in dopaminergic activity in response to aversive stimuli, that there are substantial differences between striatal regions in this response, and that it remains unclear what determines whether aversive events provoke an increase or a decrease in dopaminergic activity within striatal regions.

Delineating the role of dopamine in processing aversive events crucially requires understanding what the above-described changes in dopamine signaling encode specifically; or, in other words, whether these changes reflect aversive prediction errors (APEs, in which the dopamine response would reflect the discrepancy between expected and received aversive events) or merely individual aspects of aversive conditioning (such as the presence of aversive stimuli, and/or their prediction). A thorough analysis by Fiorillo, 2013 concluded that dopaminergic midbrain neurons do not encode aversive stimuli, but other studies have observed aspects of a dopamine APE, such as the predictive cue adopting the dopamine response of an aversive stimulus (Guarraci and Kapp, 1999; Oleson et al., 2012; Badrinarayan et al., 2012), or an APE-like response when the aversive stimulus was unpredicted or omitted (Matsumoto and Hikosaka, 2009; Matsumoto et al., 2016; Menegas et al., 2017; Salinas-Hernández et al., 2018; de Jong et al., 2019). However, it should be kept in mind that in case of omission or early termination of an expected aversive event, rewarding aspects of a milder-than-expected aversive event (RPE) may be mixed with an APE (Oleson et al., 2012; Salinas-Hernández et al., 2018; Stelly et al., 2019).

To consolidate these contradictory findings, we systematically evaluated whether dopamine truly signals an APE through a series of behavioral experiments in rats, in which we varied the value of the aversive stimulus, context valence, and probabilistic contingencies, and compared aversive and appetitive conditioning. Using FSCV, we measured the NAC dopamine response to these conditions since the NAC is a hotspot for RPE-like signals (Flagel et al., 2011; Papageorgiou et al., 2016) and is also tightly linked to motivational processes related to aversion avoidance (Badrinarayan et al., 2012; Oleson et al., 2012; Stelly et al., 2019). We employed loud white noise (WN) as the aversive stimulus. WN possesses several merits as it is well-tolerated by rats and is not painful, precisely controllable (intensity and duration can be effortlessly titrated), aversive without inducing freezing (most pertinent to this study as this might interfere with the dopamine signal), does not jeopardize the recording equipment, does not introduce artifacts to the recordings, and can be administered reliably (see ‘Discussion’). Based on the above-described findings, we hypothesized that NAC dopamine would exhibit an APE. However, we found that NAC dopamine concentration ramps down in response to both WN exposures and its predicting cue, ramps back upward upon stimulus cessation, that these ramps were qualitatively different from appetitive conditioning, and were inconsistent with a full APE signal.

Results

WN is aversive

We established the aversiveness of loud WN in a series of experiments. First, we used a real-time place aversion test in an open field, in which rats were exposed to 90 dB WN upon entry into one of the four quadrants of an open field (the WN quadrant was assigned randomly; Figure 1A). An example path traversed by a rat over the course of 30 min is depicted in Figure 1A (left panel), where the dark shaded area represents the quadrant paired with WN. On average, rats (n = 10) spent a significantly smaller percentage of time in the WN quadrant (9.71 ± 1.65) compared to chance level (t(9) = 9.585, p<0.0001; Figure 1A, right panel). Second, we exposed a new cohort of rats (n = 15) to the same open field without WN exposure to assess their preferred quadrant (Figure 1B, left panel). After this session, rats were placed in the open field in a second session, where now the entry into their preferred quadrant led to 90 dB WN exposure (Figure 1B, middle panel), which caused the rats to spend significantly less time in it (no WN: 48.5 ± 0.2474, 90 dB: 7.983 ± 0.3875, t(14) = 13.43, p<0.0001; Figure 1B, right panel). Next, a third cohort of rats (n = 12) was exposed to an open-field foraging task in which food pellets were placed between grid-floor bars in one of the quadrants. Entry into this grid-floor quadrant led to exposure of 0, 70, 80, 90, or 96 dB of WN, or a tone in separate sessions (Figure 1C, left panel). Without WN, rats spent 65.2% of the time in the maze foraging in the grid-floor quadrant. The greater the WN intensity, the shorter the amount of time rats spent foraging for food pellets (χ2(5) = 27.6, p<0.0001), whereas the tone did not reduce foraging (Figure 1C, middle panel), indicating that WN is more aversive than a pure tone (t(10) = 2.389, p=0.0381). The average number of entries into the grid-floor quadrant per session was 25.9 ± 2.5. The number of entries did not differ significantly across auditory stimuli. Rats were exposed to each auditory stimulus during foraging in two rounds of sessions, where they underwent exposure to all WN intensities and the tone in a first round of sessions, before exposure to each in a second round of sessions. The rats’ response to the different WN intensities and the tone was reliable across first and second sessions (Figure 1C, right panel), demonstrating the absence of sensitization or habituation of the behavioral response to WN across sessions and days. Interestingly, exposure to WN stimulated locomotor activity: rats in an operant box significantly increased locomotion speed in response to semi-random presentation of 6 s WN bouts (t(13) = 7.059, p<0.0001; Figure 1D, middle panel). This locomotor response was WN-intensity dependent as we found a main effect of intensity on baseline-subtracted locomotion speed (χ2(3) = 13.80, p=0.0032) in another cohort of rats (n = 13), and significant differences between 70 and 90 dB (p=0.005) and 70 and 96 dB (p=0.0143) (Figure 1D, right panel), which demonstrates that rats were able to discriminate between the different WN intensities. Rats responded with a comparable increase in locomotion speed to WN on days 1 and 6 of behavioral training, underlining the stability of the WN effect across sessions and days (t(16) = 0.9111, p=0.3757; Figure 1E). Next, we validated that rats can discern between different magnitudes of WN in an operant choice task. Here, rats could choose between pressing one of two extended levers, both of which prompted immediate delivery of a food pellet, but one of which additionally presented 5 s of 0, 70, 90, or 96 dB WN. Unsurprisingly, rats (n = 6) preferred the non-WN lever as indicated by a significant main effect of WN intensity using a Friedman test (χ2(3) = 11.57, p=0.0003). Importantly, when comparing the WN-paired lever presses, rats significantly preferred the 70 dB WN to the 96 dB WN (post-hoc Dunn’s tests, p=0.0027; Figure 1F).

White noise (WN) is an aversive stimulus that lowers dopamine concentration in the nucleus accumbens core (NAC).

(A) Left: example trajectory of a rat in the real-time place aversion test (30 min), in which entry into one quadrant (shaded), which was randomly determined prior to the session, led to 90 dB WN exposure. Right: aversiveness of WN in the place aversion test quantified as significantly decreased time spent in the WN quadrant (n = 10, 9.71 ± 1.65, t(9) = 9.585, p<0.0001). (B) Left: a second cohort of rats was placed in the open field to assess their preferred quadrant in a first session. In a second session, the entry into the preferred quadrant led to 90 dB WN exposure. Right: when WN was administered in this quadrant, rats spent significantly less time in it (n = 15, No WN: 48.5 ± 0.2474, 90 dB: 7.983 ± 0.3875, t(14) = 13.43, p<0.0001). (C) Left: a third cohort of rats (n = 12) was exposed to an open-field foraging task in which food pellets were placed between grid-floor bars in one of the quadrants. Entry into the grid-floor quadrant (shaded) led to exposure of 0, 70, 80, 90, or 96 dB of WN, or a 3 kHz tone in separate sessions. Middle: increasing WN intensity dose-dependently decreased the amount of time rats spent foraging for food pellets in the grid-floor quadrant (χ2(5) = 27.6, p<0.0001), whereas the tone did not reduce foraging (70 dB WN vs. 70 dB tone (#); t(10) = 2.389, p=0.0381). Right: the rats’ response to the different WN intensities and the tone was reliable across first and second sessions of exposure, where rats underwent exposure to all WN intensities and the tone in a first session, before exposure to each in a second session. (D) Left: all other experiments took place in operant boxes equipped with a food magazine, a multiple-tone generator (cue speaker), and a WN generator (WN speaker). Middle: semi-random presentations of 6 s WN bouts increased the locomotion speed of rats (cohort 4; n = 14) in an operant box during the WN epoch compared to pre-WN baseline (post-hoc Dunn’s test, t(13) = 7.059, p<0.0001). Right: in another cohort of rats (cohort 5; n = 13), we tested different WN intensities and found a main effect of intensity on locomotion speed (χ2(3) = 13.80, p=0.0032) and significant differences between 70 and 90 dB (p=0.005) and 70 and 96 dB (p=0.0143) (n=13). (E) Rats (n = 17) responded reliably with increased locomotion speed to WN across days. (F) Rats (cohort 6; n = 9) discern between different WN magnitudes in an operant choice task, where they had to choose between pressing a lever that resulted in a food-pellet delivery and a lever that resulted in a food-pellet delivery plus simultaneous 5 s of 0, 70, 90, or 96 dB of WN (pellet + WN; Friedman test, χ2(3) = 11.57, p=0.0003). (G) Left: example cresyl violet-stained brain slice depicting an electrolytic lesion in the NAC (outlined) at the tip of the fast-scan cyclic voltammetry (FSCV) electrode (vertical black line). Right: schematic overview of FSCV recording locations (blue dots) in the NAC (gray) of all animals. (H) Single-trial pseudocolor plot (top panel), dopamine trace (bottom panel), and cyclic voltammograms (inset in bottom panel) for representative, dopamine-specific current fluctuations recorded in NAC, 5 s before WN (dashed line), during 6 s of WN (gray bar), and 14 s after WN. Except for panel (H), data are mean ± SEM. *p<0.05, **p<0.01, ***p<0.001.

WN suppresses dopamine release in the NAC

All FSCV recordings were conducted in operant boxes equipped with a food magazine, a multiple-tone generator (cue speaker), and WN generators (WN speaker) (Med-Associates; Figure 1D, left panel). Electrolytic lesions at the FSCV electrode tip were used to histologically verify electrode placement (Figure 1G, left panel). Histological analysis confirmed for all animals included in this study that the sensing end of their FSCV electrodes was consistently placed in the NAC (Figure 1G, right panel). Unexpected exposure to 6 s of 90 dB WN strongly and reliably decreased extracellular concentrations of dopamine in the NAC (Figure 1H).

Different temporal NAC dopamine dynamics during aversive and appetitive Pavlovian conditioning

Dopamine release in the NAC is often consistent with a temporal difference RPE, where an increase in dopamine activity, initially time-locked to the delivery of a reward, shifts backward in time to its predicting cue. It is assumed that this phenomenon reflects the learned association between predictive cue and reward, where the reward becomes fully predicted by the cue and, therefore, no prediction error occurs at the time of reward delivery after sufficiently repeated cue-reward pairings (e.g., Schultz et al., 1997; Flagel et al., 2011). Our first experiment investigated whether a similar phenomenon also applies to aversive stimuli and their predictors, and in what time frame such a shift may occur. Rats (n = 16) were exposed to 30 pairings of cue (5 s) and 90 dB WN (6 s) that were separated by a variable intertrial interval (Figure 2A). In order to visualize the rapid changes in dopamine response presumably reflecting learning, the first five trials are depicted individually, and based on their stable visual appearance, trials 6–10 and 11–30 were binned together. Using one-way repeated-measures ANOVAs, in which we compared the average dopamine concentration during baseline, cue, and WN epochs, we found significant main effects in all trials (trial 1: F(1.403, 19.65) = 6.853, p=0.0102; trial 2: F(1.324, 19.86) = 8.205, p=0.0059; trial 3: F(1.265, 18.98) = 6.737, p=0.013; trial 4: F(1.100, 16.50) = 5.016, p=0.0363; trial 5: F(1.548, 23.23) = 21.90, p<0.0001; trials 6–10: F(1.025, 15.38) = 15.94, p=0.0011; trials 11–30: F(1.093, 16.39) = 38.42, p<0.0001). Post-hoc analyses using Wilcoxon signed-rank tests with a Holm–Bonferroni multiple-comparison correction revealed significantly lower dopamine concentrations during the WN epoch compared to pre-cue baseline during trial 1 (Z = –1.988, p=0.0235), trial 2 (Z = –2.430, p=0.0075), trial 3 (Z = –2.327, p=0.010), trial 5 (Z = –3.464, p<0.0005), trials 6–10 (Z = 3.103, p=0.001), and trials 11–30 (Z = –3.516, p<0.0001), indicating that the decrease in dopamine during WN is an unconditioned response, since it is observed already during the first trial. In contrast, we only observed significantly lower dopamine in the cue epoch compared to pre-cue baseline during trial 5 (Z = –1.965, p=0.0245), trials 6–10 (Z = –2.999, p=0.0015), and trials 11–30 (Z = –3.516, p<0.0001), indicating that this decrease develops over time and, therefore, is a conditioned response. In these first 30 trials, the decrease in extracellular dopamine during the WN epoch did not disappear or decrease. Thus, WN does not provoke a substantial temporal shift of NAC dopamine signaling from unconditioned to conditioned (predictive) cue within the first session of training.

Nucleus accumbens core (NAC) dopamine signaling and rat behavior during Pavlovian white noise (WN)-cue conditioning and varying WN intensities.

(A) Average extracellular concentrations of dopamine (DA; in nM) in the NAC (dark-blue line; SEM is shaded light blue) during the first 30 pairings of cue (5 s tone) and WN (6 s, 90 dB) (16 rats). To illustrate the immediate, unconditioned effects of WN, the first five trials are displayed individually. The bar graph insets depict dopamine release averaged for baseline, cue, and WN epochs. WN decreased dopamine significantly in all trials, except trial 4. The WN-paired cue began to decrease dopamine significantly starting at trial 5. (B) Comparison of dopamine release during aversive (left, n = 4) and appetitive (right, n = 10) Pavlovian conditioning. Top: subsecond changes in dopamine concentration (nM) on day 1 (blue) and day 6 (orange). Bottom: ratio of areas under the curve (AUC) between the CS and US (US/CS) on day 1 (early and late trials) and day 6 during aversive (left) and appetitive (right) conditioning. For aversive conditioning, dopamine differed between day 1 (early) and day 1 (late) (p=0.0138), and between day 1 (early) and day 6 (p=0.0318). For appetitive conditioning, a significant difference was found between early and late conditioning on day 1 (p=0.0441) and dopamine differed between day 1 (early) (p=0.0102) and day 1 (late) (p<0.0001) compared to day 6. (C) Conditioned behavioral response corresponding to (B). Left: during aversive conditioning, locomotion speed during cue presentation increased from day 1 to day 6 (Z = – 2.485, p=0.013), and also during WN (Z = –2.343, p=0.019) (n = 17). Right: during appetitive conditioning, time spent in proximity of the reward magazine increased between days 1 and 6 both during cue presentation (t(9) = 6.962, p<0.0001) and after pellet delivery (t(9) = 2.572, p=0.0301), (n = 10). (D) WN and its paired cue decreased dopamine concentration reliably across days. Cue-induced decrease was stable between days 5 and 9, and WN-induced decrease was stable between days 1, 5, and 9. (E) In an extinction session, for 20 consecutive trials, WN was withheld after cue presentation (n = 6), and dopamine differed significantly between trials 1–5 and 16–20 (p=0.0133). (F) In contrast, we detected no differences in dopamine release between exposure to varying WN intensities (70, 80, 90, or 96 dB; F(2.380, 11.90) = 0.1655, p=0.8813, n = 6). (G) We observed no significant differences in dopamine during cue (Z = –0.059, p=0.953) or WN (Z = –0.178, p=0.859) when two separate tones were used as predictors for 80 dB (blue) or 96 dB (orange) WN (n = 9). (H) Trial-by-trial correlation between locomotion speed and dopamine concentration during cue (blue) and WN (orange) for either 80 dB (top) or 96 dB WN (bottom) were not significant. Each dot represents one trial. Trials from all animals (n = 9) were pooled. Top left: no correlation during 80 dB WN (R2 = 0.0016, p=0.457) or its cue (R2 = 0.0004, p=0.7223). Bottom left: no correlation during 96 dB WN (R2 = 0.0011, p=5374) or its cue (R2 = 0.0003, p=0.7360). Right: R2 values calculated separately for each individual rat confirm that there is no significant correlation between locomotion speed and dopamine. Data are mean ± SEM, + SEM, or - SEM. *p<0.05, **p<0.01, ***p<0.001.

One possible explanation for an incomplete shift of dopamine signaling from WN to cue is that 30 pairings are insufficient to fully acquire the association. Therefore, we conditioned a subset of rats (n = 4) for five additional days. Another group of rats (n = 10) received food-pellet rewards paired with a predictive cue to compare the temporal dynamics of aversive (90 dB WN; Figure 2B, left) and appetitive (reward) conditioning (Figure 2B, right). Changes in dopamine are illustrated across time (Figure 2B, top), and in order to quantify the shift of the dopamine response from the US to the CS, we calculated the ratio between the areas under the curve (AUC) between them during the very first trials of conditioning (‘early day 1’), the rest of the trials of day 1 (‘late day 1’), and on the sixth day of conditioning (‘day 6’) (see Figure 2B, bottom). During aversive conditioning, we found, using a mixed-effects analysis, a significant main effect of the degree of conditioning (F(0.9586, 2.396) = 117.3, p=0.0043), and post-hoc testing using Tukey’s multiple-comparisons test reveals significant differences between the ratios of day 1 early trials and day 1 later trials (p=0.0138), as well as between day 1 early trials and day 6 (p=0.0318). However, no difference was observed between day 1 later trials and day 6 (p=0.9852). During appetitive conditioning, we also found a main effect of the degree of conditioning on the ratio between the US and the CS (F(1.034,8.788) = 13.88, p=0.0047), and, in contrast to aversive conditioning, the ratio on day 6 is significantly different from both day 1 early trials (p=0.0102) and day 1 later trials (p<0.0001). In addition, we found a significant difference between day 1 early trials and day 1 later trials (p=0.0441). The comparison of conditioned behavioral responses to 90 dB WN between day 1 and day 6 using Wilcoxon signed-rank tests and a Holm–Bonferroni correction for multiple comparisons reveals an increase in locomotion speed (Figure 2C), compared to baseline, which, on day 1, was restricted to the WN epoch alone (cue: Z = –0.876, p=0.381; WN: Z = –3.621, p<0.0001). On day 6, we observe an increase in locomotion speed during both the cue (Z = –3.053, p=0.002) and WN (Z = –3.621, p<0.0001) epoch compared to baseline, which are both also significantly higher compared to day 1 (cue: Z = – 2.485, p=0.013; WN: Z = –2.343, p=0.019). During appetitive conditioning, we see the same temporal evolution in the conditioned response, where rats approach the reward magazine more during the cue epoch on day 6 compared to day 1 (t(9) = 6.962, p<0.0001). However, during appetitive conditioning, but not during aversive conditioning, an almost complete shift of the dopamine response from the CS to the US occurred. Together, these results demonstrate distinct differences in the temporal dynamics of dopamine signaling during aversive and appetitive conditioning.

Reliability and extinction of cue-induced dopamine signaling

To test the reliability with which WN and its paired cue decrease dopamine concentration across days, we compared behavioral sessions on days 1, 5, and 9. The cue-induced decrease in dopamine concentration became more robust between days 1 and 5, which is explained by the cue acquisition during day 1 (Figure 2A). Cue-induced decrease was stable between days 5 and 9, and WN-induced decrease was stable between days 1, 5, and 9 (n = 6, cue: F(1.428, 7.141) = 9.402, p=0.0133; day 1 vs. day 5: p=0.0035; day 1 vs. day 9: p=0.0882; day 5 vs. day 9: 0.7916; WN: F (1.236, 6.178) = 0.03898, p=0.8936; Figure 2D). In addition to having monitored the quick acquisition of the cue’s dopamine-decreasing properties (Figure 2A), to further verify that these properties were a learned response, we tested how fast the association between cue and WN could be extinguished. Rats with well-established cue-WN associations (that were conditioned for more than 6 days) were exposed to 20 consecutive trials in which WN was omitted. Using a Friedman test, we found a significant effect of extinction (χ2(2) = 8.4, p=0.005). Post-hoc analysis using a Dunn’s multiple-comparison test revealed a significant difference between the decrease in extracellular dopamine concentration of trials 1–5 and trials 16–20 (p=0.0133, Figure 2E), with the latter no longer showing a decrease in dopamine. Thus, over the course of 15 extinction trials, the cue lost its conditioned dopamine response.

NAC dopamine does not reflect WN intensity and WN-induced behavior

For the previous experiments, we used WN with an intensity of 90 dB. We asked whether WN of different intensities would differentially influence extracellular dopamine in the NAC since we observed increased avoidance of higher intensities of WN (Figure 1C) and greater locomotion in response to higher intensities of WN (Figure 1D, right panel). First, we tested whether there is a dose–response relationship between different WN intensities and dopamine. We exposed rats (n = 6) to four different intensities of WN (70, 80, 90, and 96 dB), which were delivered in a semi-random order (Figure 2F). Although all WN intensities decreased dopamine release, we found no significant effect of intensity on extracellular dopamine (F(2.380, 11.90) = 0.1655, p=0.8813). In a different experiment, we trained rats (n = 9) on an aversive conditioning paradigm in which two cues (2 kHz or 8 kHz tones) predicted exposure to 6 s of either 80 dB or 96 dB WN, respectively (Figure 2G). Again, no significant differences were found in extracellular dopamine release between WN intensities (Z = –0.178, p=0.859), nor between the effects of their respective predicting cues (Z = –0.059, p=0.953). Both of these experiments indicate that extracellular dopamine in the NAC does not encode WN intensity, and, therefore, the relative aversiveness or aversive value of WN is not encoded by NAC dopamine.

Many studies have demonstrated the involvement of dopamine in movement (e.g., Syed et al., 2016; da Silva et al., 2018; Coddington and Dudman, 2019). Therefore, we tested whether a correlation between locomotion speed and extracellular dopamine concentration existed during cue and WN epochs. We performed a trial-by-trial analysis for both 80 dB (Figure 2H, top) and 96 dB (Figure 2H, bottom) WN exposures and found no correlation during the cue (80 dB: R2 = 0.0004, p=0.7223; 96 dB: R2 = 0.0003, p=0.7360) nor during WN (80 dB: R2 = 0.0016, p=0.4578; 96 dB: R2 = 0.0011, p=0.5374). The dots in the inset graphs represent the R2 values of the average locomotion speed and dopamine concentrations during the cue and WN period in the recording session for each individual animal.

NAC dopamine signals contain little prediction error

Although the WN-predictive, conditioned cue acquired the ability to reliably suppress NAC dopamine release, no substantial transfer of this effect from US to CS occurred (a prerequisite for a prediction error signal). To further evaluate whether NAC dopamine might function as an APE, we introduced several deviations from the expected outcomes. First, we exposed rats to two different cues (2 kHz or 8 kHz tones) that were associated with different probabilities followed by either 80 dB or 96 dB WN (see Figure 3A). Even though dopamine did not encode WN intensity, we hypothesized that dopamine may nonetheless convey an error component to be reflected as diminished dopamine decrease during the better-than-expected condition (occurrence of low-probability [25%] 80 dB WN), and augmented decrease in dopamine during the worse-than-expected condition (occurrence of low-probability [25%] 96 dB WN). As expected, we did not observe differences in dopamine release during the cue epoch due to the uncertainty of which intensity would follow (tone 1: t(8) = 0.5983, p=0.5662, Figure 3B; tone 2: t(8) = 1.432, p=0.1901, Figure 3C). However, during the WN epoch, we also did not find significant differences between the two intensities, neither for tone 1 (t(8) = 0.6698, p=0.5218), nor for tone 2 (t(8) = 0.4452, p=0.6680). Together, these results indicate that the prediction error of outcomes deviating from expected probability is not encoded by NAC dopamine concentrations.

Nucleus accumbens core (NAC) dopamine consistently tracks prediction and duration of white noise (WN) with little aversive prediction error function.

(A) Trial structure of the probabilistic Pavlovian WN task. (B) Dopamine concentration in the probabilistic task during the presentation of tone-1 cue, which was followed by 80 dB WN (blue) in 75% of trials and by 96 dB WN (orange) in the remaining 25% of trials. Bar graph inset: no significant differences in average dopamine concentration (n = 9) during cue (t(8) = 0.5983, p=0.5662) and WN (t(8) = 0.6698, p=0.5218). (C) Dopamine concentration in the probabilistic task during the presentation of tone-2 cue, which was followed by 80 dB WN (blue) in 25% of trials and by 96 dB WN (orange) in the remaining 75% of trials. Bar graph inset: no significant differences in average dopamine concentration (n = 9) during cue (t(8) = 1.432, p=0.1901) and WN (t(8) = 0.4452, p=0.6680). (D) Comparison of dopamine between predicted 6 s WN (orange) and worse-than-predicted 12 s WN (blue) (n = 11) demonstrates significantly lower average dopamine in the epoch between 11 and 25 s during worse-than-predicted 12 s WN (t(10) = 1.863, p=0.046). Bar graph (right): slopes of dopamine concentration trajectories (black dotted lines) show no significant difference between worse-than-predicted and predicted WN (Z = –1.432, p=0.0775). (E) Comparison of dopamine between predicted 6 s WN (orange) and better-than-predicted, omitted WN (blue) (n = 11) demonstrates significantly higher average dopamine in the epoch between 5 and 20 s during better-than-predicted, omitted WN (t(10) = 3.751, p=0.0019). Bar graph (right): slopes of dopamine concentration trajectories (black dotted lines) show no significant difference between better-than-predicted and predicted WN (Z = 1.334, p=0.091). (F) Comparison of dopamine between predicted 6 s WN (orange) and unpredicted 6 s WN (blue) (n = 11) demonstrates significantly lower average dopamine in the epoch between 11 and 25 s during unpredicted WN (t(10) = 2.453, p=0.0170). Bar graph (right): slopes of dopamine concentration trajectories (black dotted lines) show a significantly flatter slope during unpredicted WN compared to predicted WN (Z = –2.490, p=0.0065). (G) Dopamine release during prolonged 24 s WN exposure (blue) continues to incrementally decrease over time. Unexpected reward delivery (n = 6) during such 24 s WN (orange) induces an increase in dopamine in the epoch between 11 and 35 s (t(5) = 3.108, p=0.0266). (H) Comparison of dopamine after unexpected pellet delivery (blue) and after unexpected pellet delivery during WN exposure (orange) (n = 6) shows no significant difference in average dopamine in the epoch between 0 and 5 s (t(5) = 0.08753, p=0.9336). (I) Comparison of dopamine during WN exposure in a testing context without rewards (blue) and a testing context with intermittent rewards (orange) (n = 6) shows no significant difference in average dopamine during the cue (t(5) = 0.2841, p=0.7877) and WN (t(5) = 0.3151, p=0.7654). Data are mean + SEM or - SEM. *p<0.05, **p<0.01.

Since WN intensity had little influence on dopamine release, we investigated whether an APE was detectable when deviations from the expected outcome occurred in the temporal domain (i.e., duration of WN). We randomly exposed rats to a small number of probe trials with (a) a longer-than-predicted 90 dB WN (12 s instead of 6 s), in other words, a worse-than-predicted outcome (Figure 3D); (b) omitted WN, in other words, a better-than-predicted outcome (Figure 3E); or (c) unpredicted WN, in other words, another version of a worse-than-predicted outcome, but in this case lacking the prediction completely (Figure 3F). These three types of probe trials were implemented in a session in which the first 30 trials consisted of exclusive, deterministic pairings of cue (5 s) and 90 dB WN (6 s), after which these regular predicted WN trials were intermixed with the abovementioned probe trials. During the worse-than-predicted WN trials where the WN was extended by 6 s, we observed an extended suppression of dopamine, which decreased at the same rate as during the initial 6 s, and which ceased immediately upon termination of WN, resulting in a overall lower dopamine concentration in the 11–25 s epoch (Figure 3D; t(10) = 1.863, p=0.046, after Holm–Bonferroni correction for multiple comparisons). During better-than-predicted trials (omitted WN), we found overall higher concentrations of dopamine in the 5–20 s epoch (Figure 3E; t(10) = 3.751, p=0.0019, after Holm–Bonferroni correction). For the unpredicted WN, which constitutes a prediction error since the predictive cue is lacking, we aligned dopamine concentrations for predicted and unpredicted WN at its onset, in order to compare the impact of WN per se. We observed significantly lower dopamine concentrations exclusively in the epoch after the termination of the WN (Figure 3F; 11–25 s, t(10) = 2.453, p=0.0170, after Holm–Bonferroni correction), but not during the WN epoch itself.

Although we did not detect a dopamine error signal during exposure to WN (i.e., deviations from expected WN), we hypothesized that such unexpected events may alter dopamine after WN-offset. Thus, we compared the slope (or rate) of change of dopamine concentration during the recovery epoch (after WN cessation) since using slope allows for integration of the change in dopamine concentration over time, when the animals were presented with deviations from the predicted aversive event. Specifically, we found no significant slope difference between fully predicted WN trials and ‘worse-than-expected’ trials (t(10) = 0.1511, p=0.4415, after Holm–Bonferroni correction, Figure 3D, bar graph) or ‘better-than-expected’ trials (t(10) = 0.4809, p=0.3205, after Holm–Bonferroni correction, Figure 3E, bar graph). Thus, when our rats were exposed to unexpectedly extended WN or to the unexpected omission of WN, dopamine concentration reflected only the duration of WN exposure, but not a prediction error. In contrast, we found a significant difference in recovery slope between unexpected and expected WN (t(10) = 2.895, p=0.0080, after Holm–Bonferroni correction, Figure 3F, bar graph), which indicates that, in the case of an unexpected aversive stimulus, dopamine does not only track the duration of this aversive stimulus, but displays a differential response and, thus, may serve as a qualitative teaching signal.

Dopamine integrates information about appetitive and aversive stimuli

We then investigated whether dopamine could still encode rewards during ongoing WN exposure, that is, while extracellular dopamine concentrations are continuously decreasing. To test this, we delivered food pellets unexpectedly during a prolonged WN epoch. We observed a significant increase in dopamine release upon pellet delivery (t(5) = 3.108, p=0.0266, Figure 3G), which was comparable to the increase in dopamine release we observed upon pellet delivery in the absence of WN (t(5) = 0.08753, p=0.9336, Figure 3H). These results indicate that dopamine is still responsive to rewarding events during an aversive event and, thus, integrates information about appetitive and aversive events.

A previous study reported that dopamine is more prone to encode an APE in an experimental context with a low probability of intermittent reward delivery (Matsumoto et al., 2016). Thus, we compared dopamine release during cue and WN exposure embedded in two task contexts with different reward probabilities (i.e., different ‘reward contexts’). In the first task context, no rewards were delivered during the entire session, whereas in the second context a low chance of reward delivery existed (reward trial probability = 0.1). We did not observe a significant difference in dopamine concentration between these reward contexts during the cue epoch (t(5) = 0.2841, p=0.7877), nor during the WN epoch (t(5) = 0.3151, p=0.7654) (Figure 3I). Consistently, in another experiment, we did not observe a significant difference in dopamine concentration (during cue and WN epochs) when comparing a no-reward context with a high-reward context (reward trial probability = 0.5; cue: t(17) = 1.448, p=0.0829; WN: t(17) = 1.428, p=0.0857; data not shown).

Discussion

In this study, we set out to delineate the role of the dopamine system in processing aversive stimuli, by systematically investigating subsecond fluctuations in rat NAC dopamine concentration in response to an aversive auditory stimulus (WN), as well as its prediction by auditory tone cues. First, we demonstrated that WN-induced behavioral activation is WN-intensity-dependent and validated the aversiveness of WN in real-time place aversion, approach-avoidance foraging, and operant tasks, where we found that WN aversiveness scales with WN intensity. Trial-by-trial analysis of the first WN exposures revealed that WN as an unconditioned stimulus diminishes the concentration of extracellular dopamine in the NAC, and that a predicting cue rapidly takes on the role of conditioned stimulus (reversible by extinction), eliciting WN-like behavioral activation and dopamine depression. Dopamine during cue and WN was not correlated with locomotion speed. In contrast to appetitive conditioning, only a very limited temporal shift of the dopamine response from WN to the cue occurred. Dopamine responses to WN and its predictive cue were not affected by aversive value (WN intensity), context valence (introduction of intermittent rewards), or probabilistic contingencies. Instead, prediction and duration of the aversive WN were accompanied by a relatively slow and steady decrease in NAC dopamine concentration (a declining ramp that continued without plateauing), which was followed by an equally slow recovery of dopamine upon cessation of WN. The slope of this rebounding dopamine ramp was altered only by unpredicted presentation of WN (not by better-than-predicted or worse-than-predicted outcomes), revealing a function of dopamine that sometimes goes beyond simple real-time tracking the presence of conditioned and unconditioned aversive stimuli. Finally, we found the integration of rewarding and aversive stimuli is of parallel nature as WN-associated dopamine depression did not modify the rapid surge of dopamine triggered by unexpected reward delivery. Together, our findings indicate that negative dopamine signals in the NAC mostly track the prediction and duration of aversive events, with few aspects that are consistent with an APE.

WN is a versatile aversive stimulus that suppresses dopamine release and increases locomotion

We chose WN as an aversive stimulus to probe the limbic dopamine system’s role in aversive conditioning as it possesses several merits. First, at intensities that are not prone to cause loss of hearing (Escabi et al., 2019), WN is moderately aversive (Campbell and Bloom, 1965; Hughes and Bardo, 1981), and as such does not induce freezing, but instead provokes mild behavioral activation. This is particularly relevant with regard to studies relating dopamine function to behavioral read-outs since lack of movement is often associated with diminished activity of the dopamine system and, thus, may confound the interpretation of negative dopamine signals in the context of aversive events. Second, WN is reliably effective and tolerated across many trials and sessions, supporting the detection of neuronal signals by providing sufficient data for averaging across trials and enabling complex experiments with varying valence and contingencies. Third, WN is distinct, well-controllable, and easy to produce, where intensity and duration can be titrated effortlessly. Fourth, WN does not require attention to be detected (i.e., the animal will hear it anywhere in an experimental environment). Fifth, animals cannot interfere with WN delivery, as opposed to air puffs or electric foot shocks, which can be influenced by the animal’s actions and position (i.e., closing its eyelids or decreasing contact surface with the charged grid floor). Sixth, WN does not interfere with data recording in FSCV, electrophysiology, or fluorescence imaging. Together, the abovementioned merits make WN an experimentally valuable stimulus with great potential to uncover aversion-relevant brain mechanisms.

In this work, we report that WN diminishes extracellular dopamine concentration in the NAC upon first exposure, characteristic of an unconditioned stimulus or primary reinforcer. A predicting cue quickly adopted this property upon subsequent exposures, which was reversible by extinction. Such dopamine responses were stable across trials and sessions. Interestingly, the use of WN revealed a rare relationship between dopamine and behavior: increased locomotion speed was associated with a decrease in dopamine release. Behavioral activation is usually associated with increased dopamine signaling (Boureau and Dayan, 2011; Berridge and Robinson, 1998; da Silva et al., 2018; Coddington and Dudman, 2019), whereas a lack of movement or even freezing, depending on stimulus intensity, is often associated with decreased dopamine (e.g., Oleson et al., 2012; Badrinarayan et al., 2012). These frequently observed association patterns have prompted the hypothesis that the directionality of changes in dopamine concentration reflects the chosen behavioral strategy when confronted with aversive stimuli: an active or a passive reaction (Badrinarayan et al., 2012). Our results, however, prove that this hypothesis is not universally applicable. In this context, it would be interesting to assess dopamine responses to greater WN intensities than the ones used here. Such intensities could offer further insight into the relationship between behavior and dopamine dynamics because they are known to elicit freezing (Rescorla, 1973; Ledgerwood et al., 2005; Furlong et al., 2016), albeit with a higher risk of hearing loss (Escabi et al., 2019). Freezing elicited via electric shocks is accompanied by decreased NAC dopamine activity (Badrinarayan et al., 2012; Oleson et al., 2012; de Jong et al., 2019; Stelly et al., 2019). The behavioral activation we observe in response to WN might reflect an increased motivation to escape, which we cannot ascertain as our task was Pavlovian (thus, without an active avoidance component: the WN was inescapable). Notably, the observed decline in dopamine was not at all correlated with movement on a trial-by-trial basis; thus, it is conceivable that during mild WN exposure, NAC dopamine was uncoupled from its usual, more direct behavioral impact.

What is and what is not encoded by NAC dopamine?

Many studies have investigated the role of dopamine in aversion by testing the system’s reaction to the exposure to aversive stimuli (see above), but only a few scrutinize dopamine’s precise function therein, or whether dopamine encodes a ‘true’ APE. Their conclusions range from ‘dopamine is insensitive to aversiveness’ (Fiorillo, 2013), to the other extreme of ‘dopamine serves as an APE’ (Matsumoto et al., 2016). Fiorillo, 2013 ruled out the existence of a dopamine APE because (1) dopamine-neuron firing did not differ between presentation of aversive and neutral stimuli, (2) prediction of an aversive event did not affect firing, and (3) no integration of rewarding and aversive values was observed. In contrast, Matsumoto et al., 2016 found evidence for all three of these requirements and, therefore, concluded that dopamine neurons are capable of encoding a value prediction error (equally for both rewards and aversive stimuli). This discrepancy could partially be explained by the fact that Matsumoto et al., 2016 recorded from dopaminergic neurons in the VTA (of mice), whereas the majority of the neurons that Fiorillo, 2013 recorded were in the substantia nigra (of monkeys). Since we measured extracellular concentrations of NAC dopamine, which is released from terminals that originate from neurons in the VTA (Ikemoto, 2007), we expected to find an APE in our data.

Indeed, our results meet Fiorillo’s (2013) three requirements for an APE stated above: (1) during the first pairing of cue and WN, when the predictive auditory cue was still neutral, dopamine concentration during the WN epoch differed significantly from that during baseline and cue presentation, but the latter (cue and baseline dopamine) did not differ from each other. (2) After only four cue-WN pairings, cue presentation diminished dopamine concentration, thus prediction of the aversive event did alter dopamine activity. Although we did not find a significant difference in overall dopamine concentration between predicted and unpredicted WN, we did observe a difference in their post-WN recovery slopes. (3) Finally, although we did not detect an ‘interactive’ integration (modulated signal) of aversive and reward values during concurrent presentation of WN and a food pellet (as the absolute magnitude of released dopamine was equal to that of a pellet delivered outside of WN exposure), both the rewarding and aversive stimuli were encoded in parallel. The signals are thus integrated in the sense that both are processed at the same time, in an additive manner; as opposed to an exclusive organization, where the dopamine system may be ‘turned off’ or unresponsive towards rewards during the presence of an aversive stimulus. Taken together, up to this point, our results fit best with the conclusion of Matsumoto et al., 2016; although a noteworthy contrast is that in our data, context was irrelevant to the magnitude of dopamine response to the aversive event and reward: it made no difference for the acute dopamine response magnitude whether the aversive stimulus was delivered in rewarding contexts or not.

Next, however, we took inspiration from Hart et al., 2014, who used a mathematical approach developed by Caplin and Dean, 2007 to confirm the encoding of RPE signals by NAC dopamine. They used a deterministic and a probabilistic choice task in order to determine whether dopamine signals fulfilled three axioms that were considered necessary for a RPE signal: ‘consistent prize ordering,’ ‘consistent lottery ordering,’ and ‘no surprise equivalence.’ We employed the abovementioned deterministic and probabilistic Pavlovian conditioning tasks to identify an APE, instead of an RPE, by exposing rats to low- and high-dB WN, predicted by two different tones, with either a 100% probability (deterministic task) or with different probabilities (probabilistic task). We did not observe differences in dopamine concentration during the WN epoch in the deterministic task, which fulfills the third axiom (no surprise equivalence), since the prediction error is zero for both of these conditions. But the first two axioms were not fulfilled since we did not detect differences in dopamine during the WN epoch, when different WN intensities were presented with different probabilities. Rats avoided higher-dB WN more than lower-dB WN (Figure 1B) and exhibited WN dB-dependent locomotor activation (Figure 2F), indicating that WN aversiveness scales with WN intensity and that rats are able to discriminate between different WN intensities. Thus, we conclude that the NAC dopamine signals we observed did not fulfill the axiomatic criteria of an APE, when aversive stimulus intensity or value was varied.

Finally, we performed experiments to probe the dopamine signal in conditions where the aversive stimulus deviated from the expected duration; in other words, when trials were worse or better than predicted based on WN duration, but with a stable intensity of 90 dB. First, we extended WN duration or omitted WN in occasional trials. Extended WN elicited a continuation of the same declining dopamine concentration slope, which ceased promptly at WN cessation, after which dopamine slowly ramped backup toward baseline with a reversed, inclining slope. Thus, although the signal reflected the duration of extended WN, no discernible error component was evident. When WN was omitted, we did not observe an error signal either: instead, again, the signal slowly returned to baseline levels. Second, in another version of the ‘worse-than-expected’ condition, we occasionally delivered WN unexpectedly, without a preceding cue (after animals had learned the cue-WN association well), and observed a difference in the recovery slope after WN-offset compared to predicted WN. This flattened recovery slope indicates that NAC dopamine signals more than simply track the presence of aversive stimuli; in addition, it may relate to the failed anticipation of an aversive event (based on reliance on the predictive cue), and, thus, indicate altered cue-WN contingencies. In summary, we conclude that dopamine precisely tracks aversive stimulus duration, and the only evidence of an APE-like signal in our data was found after unpredicted WN, whereas several of our other experimental accounts are incompatible with an APE function of NAC dopamine. This places our results firmly in the middle ground between the no-APE (Fiorillo, 2013) and the full-APE conclusions (Matsumoto et al., 2016) described above.

Aversion versus reward

Consistent with most literature (Badrinarayan et al., 2012; Oleson et al., 2012; de Jong et al., 2019; Stelly et al., 2019), we found that NAC dopamine encodes rewarding and aversive events with opposite directionality. Furthermore, we report that a cue predicting an aversive stimulus can adopt the ability to prompt dopamine changes the way the aversive stimulus itself would. Taken together, this suggests that NAC dopamine encodes both reward and aversive prediction. However, decreases in dopamine concentration did not scale with WN intensity, unlike what is well-established for reward processing, where reward size or probability is encoded both for the reward itself and for predictive stimuli (Gan et al., 2010; Tobler et al., 2005; Watabe-Uchida et al., 2017). Thus, one could speculate that the diminishing effect of WN on NAC dopamine may be related to pre-attentive processes involved in saliency and novelty – dissociable from NAC value signals (Redgrave and Gurney, 2006; Kutlu et al., 2021; Kutlu et al., 2022). Furthermore, encoding of a prediction error, which is one of the best-characterized features of reward-related dopamine signaling, did not occur for aversive events. Thus, NAC dopamine does not encode aversive and appetitive stimuli (and their prediction) in the same way. Moreover, the basic nature of aversion-related dopamine signals in our data was different from that of rewards. For example, the temporal signal shift toward the earliest predictor of the respective reinforcing stimulus, as described for rewards, is incomplete for aversive conditioning. Another example is that reward-related changes in extracellular dopamine concentration are substantially larger and faster compared to aversive events. These discrepancies may be partially attributable to general differences between dopamine release into and removal from the extracellular space. More specifically, the dopamine system presumably has a bigger dynamic range for increasing activity; it can do so, for example, by increasing the number of cells firing and their firing frequency (and thereby the total number of dopamine-containing vesicles being released). In contrast, dopamine-signaling reduction cannot drop below a certain point since the cells’ maximum response is to cease firing altogether and extracellular dopamine can only be removed relatively slowly or must diffuse away. This disparity could translate into a structurally limiting factor on what can be encoded by a reduction in dopamine concentration and explain some of the abovementioned differences in function. However, the slow-ramping declining and recovery slopes we observed do not reflect the system limits since the very first exposure to WN resulted in a steeper decline and rewards given during WN resulted in steeper increases. Furthermore, disparate qualitative differences were also found in NAC-dopamine responses to the presentation of ultrasonic vocalizations that are associated with rewarding and aversive events (Willuhn et al., 2014b). Taken together, our results indicate that there are a few similarities between dopamine encoding of rewards and aversive stimuli, but overall we found more differences between them – hinting at aversive events being encoded by NAC dopamine more rudimentarily in a qualitative instead of quantitative fashion.

In summary, our findings demonstrate that WN is a valuable and versatile aversive stimulus that is well-suited to probe how the brain processes aversive stimuli. Overall, we conclude that dopamine tracks the anticipation and duration of an aversive event. This tracking materializes as a perpetually declining dopamine ramp that progresses without altering its slope until offset of the aversive stimulus (even WN lasting for 24 s did not reach a plateau of minimal dopamine concentration). Such aversion tracking may play an anticipatory role for certain defensive behaviors since the animals were behaviorally activated during the aversive event. Furthermore, we speculate that these slowly ramping aversion signals may contribute to a qualitative learning signal (other than a quantitative or scalar APE signal) since the unexpected aversive stimulus elicited a response beyond simply tracking the stimulus. Thus, we conclude that dopamine tracks both positive and negative valence in their temporal aspects and prediction, but that quantitatively speaking, the exact value and error is only encoded for rewards, in the upward direction of NAC dopamine concentration. This implies that aversive value and APEs are encoded in other brain regions.

Materials and methods

Animals

Adult male Long–Evans rats (300–400 g; Janvier Labs, France) were housed individually and kept on a reversed light–dark cycle (light on from 20:00 till 8:00) with controlled temperature and humidity. All animal procedures were in accordance with the Dutch and European laws and approved by the Animal Experimentation Committee of the Royal Netherlands Academy of Arts and Sciences under CCD license numbers AVD801002015126 and AVD80100202014245. In total, 37 rats underwent surgery, 21 of which exhibited a functional FSCV electrode with a histologically verified location in the NAC, and were therefore included in the study. An additional 64 rats, which did not undergo surgery, were used for behavioral tasks (Source data 1). All rats were food-restricted to 90% of their free-feeding bodyweight, and water was provided ad libitum.

Stereotaxic surgery

Request a detailed protocol

Rats were induced under isoflurane anesthesia and placed into the stereotaxic frame on an isothermal pad maintaining body temperature. The analgesic Metacam (0.2 mg meloxicam/100 g) was injected subcutaneously and the shaved scalp was disinfected using 70% ethanol. Upon incision of the scalp, it was treated with lidocaine (100 mg/ml). Holes were drilled in the cranium and the dura mater was cleared for targeting the NAC (1.2 mm AP, 1.5 mm ML, and –7.1 DV; Paxinos and Watson, 2007). Chronic carbon-fiber electrodes (Clark et al., 2010), made in-house, were positioned in the NAC, and an Ag/AgCl reference electrode was placed in a separate part of the forebrain. The electrodes were secured to screws in the skull using cranioplastic cement. Following surgery, rats received subcutaneous injection of 2 ml saline and were placed in a temperature-controlled cabinet to be monitored for an hour. Rats were given 1–2 weeks post-surgery to recover before food restriction, behavioral training, and recording.

Behavioral procedures

All behavioral experiments, except the place-aversion and foraging tasks, were conducted in modified operant boxes (32 × 30 × 29 cm, Med Associates Inc), equipped with a food magazine (connected to an automated food-pellet dispenser) flanked by two retractable levers (with cue lights), a house light, multiple tone generators, two WN generators, and metal grid floors (Med Associates Inc). Each operant box was surveilled by a video camera. The boxes were housed in metal Faraday cages that were insulated with sound-absorbing polyurethane foam.

Real-time place aversion

Request a detailed protocol

Rats (n = 10) were placed for 30 min in a light-shielded, square, Perspex open field (60 × 60 × 60 cm), made in-house (Netherlands Institute for Neuroscience [NIN] mechanical workshop). A camera mounted in the center above the open field recorded the position of the rat, which was tracked in real time by the open-source software Bonsai (Lopes et al., 2015). One quadrant of the open field was paired with exposure to 90 dB WN, which was produced by a WN generator from Med Associates Inc, mounted on top of one of the open-field walls. WN was automatically switched on as long as the head of the rat was present in the chosen quadrant, and switched off as soon as the rat exited the quadrant. The WN-quadrant position was fixed throughout the session.

In the first experiment, the WN-quadrant position was assigned randomly for each rat (Figure 1A). The percentage of time rats spent in the WN-paired quadrant was compared to chance level (25%). In the second experiment, another cohort of rats was placed into the open field without WN exposure to assess their preferred quadrant (Figure 1B). After this session, rats were placed in the open field in a second session, where now the entry into their preferred quadrant led to 90 dB WN exposure (Figure 1B). The percentage of time rats spent in the preferred quadrant was compared to the time spent in the WN-paired quadrant.

Approach-avoidance foraging task

Request a detailed protocol

A cohort of rats (n = 12) performed in an open-field foraging task (Figure 1C). For this purpose, a moveable grid floor that covered one quadrant of the open field was installed. Ten food pellets were placed between the grid-floor bars before the rats were placed in the middle of the open field for each 10 min session. After 1 day of habituation to the setup, rats underwent a series of sessions, where they were exposed to 0, 70, 80, 90, or 96 dB of WN, or a 70 dB 3 kHz tone upon entrance to the grid-floor quadrant, where they could forage for the pellets by reaching between the grid bars. Rats were exposed to each auditory stimulus twice during foraging in two rounds of sessions, where rats underwent exposure to all WN intensities and the tone in a first session, before exposure to each in a second session. The sequence of sessions proceeded from lowest to highest WN intensity, followed by the tone session. The 3 kHz tone was administered to evaluate whether WN was more aversive than other auditory stimuli. The location of the grid-floor quadrant was changed between rounds.

WN and reward choice task

Request a detailed protocol

Rats (n = 9) were trained to press one of the two levers in the operant box to receive food-pellet rewards (Dustless Precision Pellets, 45 mg, Bio-Serv). During the first training days, a single lever was inserted at variable intertrial intervals. Pressing this lever prompted delivery of a single food pellet and immediate retraction of the lever. Omissions (no lever press for 10 s) resulted in 10 s house-light illumination. After reaching a 90% success rate, the other lever was introduced in training sessions that consisted of 20 ‘forced’ trials, in which one of the two levers was presented, followed by 80 choice trials where both levers were presented. Any lever press resulted in delivery of a single food pellet and retraction of extended levers (marking the end of the trial). After five consecutive sessions with over 90% success rate, we paired reward delivery of one of the levers with simultaneous 5 s of 90 dB WN exposure. Rats were trained under these contingencies for 4 days, after which half of the animals (n = 5) were switched to 96 dB and the other half to 70 dB WN (n = 4). After four sessions, WN intensities were transposed between the two groups of animals for an additional four sessions, so that every animal received each WN intensity. In each of these WN sessions, animals could earn a maximum of 100 pellets (in 100 trials). Both levers were presented simultaneously at a variable intertrial-interval averaging 25 s (range: 15–35 s). Just as at the start of training, one lever press induced immediate retraction of both levers and prompted reward delivery (end of trial), whereas omissions (no press within 10 s upon lever insertion) ended the trial and resulted in 10 s house-light illumination. We compared the relative number of WN-paired lever presses across different WN intensities.

FSCV during aversive Pavlovian conditioning with 90 dB WN

Request a detailed protocol

On the first day of aversive Pavlovian conditioning, a new group of 16 rats was tethered to the FSCV recording equipment and placed in the operant box. In this and all paradigms described below, prior to behavioral session start, two unexpected deliveries of a single food pellet (spaced apart by 2 min) confirmed electrode viability to detect dopamine. The session started with the illumination of the house light. The first 30 trials consisted of the presentation of a 5 s cue (1.5 kHz, 75 dB tone) followed by 6 s of WN (90 dB). Trials were separated by a variable intertrial interval of 60 s (range: 30–90 s). For a subset of the rats (n = 11), these initial 30 trials were followed by 55 trials, in which 5 s cue/6 s WN pairings were randomly mixed with four trials with unpredicted WN (6 s of 90 dB WN without cue), four trials with WN omission (5 s cue without WN), and four trials with 5 s cue followed by 12 s of 90 dB WN (longer-than-expected condition).

A subset of the initial 16 rats (n = 4) was conditioned for an additional 5 days (days 2–6), of which the first 4 days consisted of sessions with 30 trials of pairings of 5 s cue/6 s with 90 dB WN, and on the fifth day (sixth day of conditioning in total) another FSCV recording session took place (as described for day 1). An additional group of animals (n = 13) without implanted FSCV electrodes were conditioned for 6 days in order to characterize behavioral responses to different WN intensities.

For the analysis of the first 30 conditioning trials, we compared the average dopamine concentration during the cue (5 s) and the WN (6 s) epoch to baseline (−5 to 0 s before cue onset). To analyze trials with different contingencies (unpredicted, omitted, or longer WN), we compared average dopamine during the relevant epochs (unpredicted WN: 11–25 s after cue onset; better than predicted [omitted WN]: 5–20 s after cue onset; worse than predicted [longer WN]: 11–25 s after cue onset) with average dopamine in the respective epochs in immediately preceding trials 5 s cue/6 s WN pairings (trials 25–30), during which dopamine decreases had stabilized and were unaffected by different contingencies. Slopes of dopamine traces were compared between trials with different contingencies and predicted WN trials since using the slope allows for integration of the change in dopamine concentration over time as opposed to averaging concentrations over an epoch (in which there is no integration over time). All traces were aligned before WN onset.

To compare dopamine concentration during cue and WN between days 1 and 6 and to quantify the shift of dopamine release from the US to the CS, we subdivided the results of day 1 into ‘day 1 (early)’ (trials 2–4; trial 1 was excluded to remove the saliency response to the first cue exposure) and ‘day 1 (late)’ (trials 5–30). We calculated the ratio between US and CS dopamine signals as a deviation from baseline (in the respective up or down direction). For aversive conditioning, this ratio was determined by (area above the curve of the WN epoch)/(area above the curve of the cue epoch). For appetitive conditioning, this ratio was determined by (area under the curve of the pellet epoch)/(area under the curve of the cue epoch).

Appetitive Pavlovian conditioning

Request a detailed protocol

Rats (n = 10) were placed in the operant box, and on days 1 and 6, they were tethered to the FSCV recording equipment. Illumination of the house light signaled the beginning of the session. Sessions consisted of 40 pairings of cue-light illumination (5 s) with a pellet delivery (delivered immediately after cue offset), which were separated by variable intertrial intervals averaging 60 s (range: 30–90 s).

WN dose response

Request a detailed protocol

Rats (n = 6) were tethered to the FSCV recording equipment and placed into the operant box. The two WN generators with custom-made volume control dials (NIN mechanical workshop) were used to switch between different WN intensities. The FSCV recording session consisted of six blocks in which two different WN intensities (70, 80, 90, or 96 dB) were presented for 6 s in random order, four times each, with a variable intertrial-interval averaging 30 s (range: 25–35 s). Between blocks, the volume dial was used to change WN intensities. During the different blocks, all WN intensities were presented in pairs of two and, therefore, each intensity was presented in 3 of the 6 blocks and played 12 times in total. We compared the average dopamine concentration during the WN exposures between the different intensities.

An additional group of rats (n = 13) without implanted FSCV electrodes were placed into an operant box and underwent WN exposure in order to characterize behavioral responses to the four different WN intensities (70, 80, 90, or 96 dB; randomly ordered in blocks of 15 trials) presented for a duration of 6 s per trial, followed by an average variable intertrial interval of 60 s (range: 30–90 s). Before the start of each block, three food pellets were delivered with a variable intertrial interval averaging 30 s (range: 20–40). We compared the average baseline-subtracted locomotion speed of the animals during the WN exposures between the different intensities.

Aversive Pavlovian conditioning with 80 dB and 96 dB WN

Request a detailed protocol

Rats (n = 9) underwent four aversive conditioning sessions in the operant box in which a 2 kHz and 8 kHz tone (5 s cue) predicted the exposure to 80 dB or 96 dB WN (6 s), respectively. Sessions consisted of 88 trials, of which 40 trials with 80 dB WN and 40 trials with 96 dB WN, predicted by their respective cues, were presented in random order. In the remaining eight trials, an unpredicted food pellet was delivered. These deliveries were distributed across the session so that in every block of 10 WN exposures, one pellet was delivered at a random trial number. Trials were separated by variable intertrial-intervals averaging 60 s (range: 30–90 s). On the fourth conditioning day, a recording session took place, for which the rats were connected to the FSCV recording equipment. We compared average dopamine concentrations during the cue (5 s) and during the WN (6 s) epochs.

During the subsequent four aversive conditioning sessions, we changed the probability of exposure to 80 dB and 96 dB WN following their associated cues. The total number of presentations of tone 1, tone 2, WN, and pellet deliveries remained the same. However, tone 1 was now followed by 80 dB WN (6 s) in 75% of the trials and by 96 dB WN (6 s) during the remaining 25% of the trials. Tone 2 was followed by 96 dB WN (6 s) in 75% of the trials and 80 dB WN (6 s) during 25% of the trials. A recording session took place on the fourth conditioning day. We compared the average dopamine concentrations during the cue (5 s) and WN (6 s) epochs.

Concurrent reward and WN, and cue extinction

Request a detailed protocol

Rats (n = 6) were connected to the recording set up and placed in the operant box. The conditioning session began with 10 pairings of the 5 s cue (1.5 kHz tone) and 6 s WN (90 dB). Next followed a block of 20 trials pairing the 5 s cue and 24 s of WN; during half of these trials (randomized), a pellet was delivered 6 s into the WN exposure. The recording session was concluded with a block of 20 extinction trials, in which only the 5 s cue was delivered. This recording session was the last to take place, the rats had experienced 9–11 conditioning sessions prior to this recording.

FSCV measurements and analysis

Request a detailed protocol

As described previously (Willuhn et al., 2014a), FSCV was used to detect subsecond changes in extracellular concentration of dopamine using chronically implanted carbon-fiber microsensors that were connected to a head-mounted voltammetric amplifier, interfaced with a PC-driven data-acquisition and analysis system (National Instruments) through an electrical commutator (Crist), which was mounted above the test chamber. Every 100 ms, voltammetric scans were repeated to achieve a sampling rate of 10 Hz. The electrical potential of the carbon-fiber electrode was linearly ramped from –0.4 V versus Ag/AgCl to +1.3 V (anodic sweep) and back (cathodic sweep) at 400 V/s (8.5 ms total scan time) during each voltammetric scan, and held at –0.4 V between scans. Dopamine is oxidized during the anodic sweep, if present at the surface of the electrode, forming dopamine-o-quinone (peak reaction detected around +0.7 V), which is reduced back to dopamine in the cathodic sweep (peak reaction detected around –0.3 V). The ensuing flux of electrons is measured as current and is directly proportional to the number of molecules that undergo electrolysis. The background-subtracted, time-resolved current obtained from each scan provides a chemical signature characteristic of the analyte, allowing resolution of dopamine from other substances (Phillips and Wightman, 2003). Chemometric analysis with a standard training set was used to isolate dopamine from the voltammetric signal (Clark et al., 2010). All data was smoothed with a moving 10-point median filter and baseline (set at 1 s before cue onset or, in case of an absent cue, 1 s before WN onset) subtraction was performed on a trial-by-trial basis prior to analysis of average concentration. Analyses were performed on dopamine concentration during cue (5 s) and WN (6 s) epochs and were compared to baseline dopamine concentrations or to the same epoch in a different experimental condition. Prior to each FSCV recording session, two unexpected deliveries of a single food pellet (spaced apart by 2 min) confirmed electrode viability to detect dopamine. Animals were excluded from analysis when (1) a lack of dopamine release in response to unexpected pellets before start of the behavioral session, and (2) FSCV recording amplitude background noise that was larger than 1nA in amplitude.

Analysis of operant box behavior

Request a detailed protocol

DeepLabCut software (Mathis et al., 2018) was used to track rat movement in the operant box using video data recorded during FSCV measurements. This tracking data was analyzed in MATLAB (The MathWorks, Inc, version 2019a) to determine distance to the reward magazine and speed of movement (cm/s). Analyses were performed using the average distance or locomotion speed during the cue (5 s) or WN (6 s) epochs. During the WN- and reward-choice task, the number of presses on each lever was registered via an automated procedure.

Histological verification of recording sites

Request a detailed protocol

After completion of the experiments, rats were deeply anesthetized using a lethal dose of pentobarbital. Recording sites were marked with an electrolytic lesion before transcardial perfusion with saline, followed by 4% paraformaldehyde (PFA). Brains were removed and post-fixed in PFA for 24 hr after which they were placed in 30% sucrose for cryoprotection. The brains were rapidly frozen using an isopentane bath, sliced on a cryostat (50 µm coronal sections, –20°C), and stained with cresyl violet.

Statistical analysis

Request a detailed protocol

FSCV and behavioral data were analyzed using one- or two-tailed paired or unpaired t-tests, repeated-measures ANOVAs, regression analysis, or their nonparametric equivalents when appropriate. Post-hoc analyses were conducted when necessary and p-values were adjusted when multiple comparisons were made. Statistical analyses were performed using Prism (GraphPad Software) and SPSS statistics version 25.0 (IBM); graphical representations were made using Prism. Statistical significance was set to p<0.05. Sample size was not explicitly determined by a power analysis when the study was being designed, but was, instead, based on the lab’s experience with this type of data.

Data availability

Data, statistics, and code at https://osf.io/8p37x/.

The following data sets were generated
    1. Goedhoop J
    2. Willuhn I
    (2021) Open Science Framework
    ID 8p37x. Nucleus-accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error.

References

    1. Bassareo V
    2. De Luca MA
    3. Di Chiara G
    (2002)
    Differential expression of motivational stimulus properties by dopamine in nucleus accumbens shell versus core and prefrontal cortex
    The Journal of Neuroscience 22:4709–4719.
    1. Campbell BA
    2. Bloom JM
    (1965) Relative aversiveness of noise and shock
    Journal of Comparative and Physiological Psychology 60:440–442.
    https://doi.org/10.1037/h0022572
    1. Hughes RA
    2. Bardo MT
    (1981)
    Shuttlebox avoidance by rats using white noise intensities from 90-120 db spl as the UCS
    The Journal of Auditory Research 21:109–118.
  1. Book
    1. Paxinos G
    2. Watson C
    (2007)
    The Rat Brain in Stereotaxic Coordinates
    Elsevier.
    1. Rescorla RA
    (1973) Effect of US habituation following conditioning
    Journal of Comparative and Physiological Psychology 82:137–143.
    https://doi.org/10.1037/h0033815

Decision letter

  1. Mihaela D Iordanova
    Reviewing Editor; Concordia University, Canada
  2. Kate M Wassum
    Senior Editor; University of California, Los Angeles, United States
  3. Erik Oleson
    Reviewer; University of Colorado Denver, United States

In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.

Decision letter after peer review:

[Editors’ note: the authors submitted for reconsideration following the decision after peer review. What follows is the decision letter after the first round of review.]

Thank you for submitting your work entitled "Nucleus-accumbens dopamine tracks aversive-stimulus duration and prediction but not value or prediction error" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by a Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Erik Oleson (Reviewer #3).

Comments to the Authors:

We are sorry to say that, after consultation with the reviewers, we have decided that your work will not be considered further for publication by eLife.

There was overwhelming enthusiasm for the manuscript and agreement regarding the strengths of the paper by the reviewers. These include the timely and important question of the role of dopamine in aversion, as well as the experimental approach especially the variety of behavioural tasks used. The strengths have been expressed by the reviewers in their individual comments (see below). However, a number of concerns were raised that require substantial reframing (punishment vs. aversive stimulus) of the manuscript as well as additional data collection. Although the reviewers' individual comments are appended to the decision letter, I wanted to give a brief overview of the discussion that led to the decision. One of the key aspects of the paper is the idea that dopamine concentration should be modulated in a white noise intensity-specific manner. However, the behavioural evidence that the white could be discriminated at different intensities came using different parameters compared to the neurochemical experiment. It is paramount to show that given these parameters the animals can indeed discriminate the white intensities as this is key to the argument of whether dopamine tracks aversive prediction error. Relatedly, behavioural evidence of conditioning is needed in the context of the recordings. The influence of baseline choice and amplitude vs. duration of the signal on the data need to be explored further. A more specific concern regarding the lack of correspondence between Figure 2B trace and bar graphs. It was unclear to the reviewers how the trace data correspond to the bar graphs below the trace. The statistical methods and degrees of freedom were also unclear. While this is a short overview of some of the concerns that were flagged, it is not exhaustive, further concerns are noted in the individual reviews. The reviewers and editors felt that addressing these concerns is beyond the scope of a standard eLife revision. However, given the enthusiasm for the manuscript if you are able to well address each of the reviewers concerns, we would be willing reconsider the manuscript for publication at eLife as a new submission, with a point by point rebuttal to each concern.

Reviewer #1 (Recommendations for the authors):

Understanding the role of dopamine beyond reward and specifically in aversion is of fundamental importance. The paper set out to examine whether dopamine concentration in the nucleus accumbens tracks aversive prediction error as it does in reward. To do this, the authors used fast-scan cyclic voltammetry alongside a number of key behavioural manipulations that have been shown to provide conditions for detecting reward prediction error. The behavioral designs used are established and appropriate to the questions posed. While the aversive cue, a loud white noise, employed was not standard, the paper provides behavioural evidence for its effectiveness as an aversive outcome. Although a closer examination of the learning that underlies the task design is necessary. While the analyses of the neural signal are appropriate, some alternatives could be explored in order to better understand the profile of the dopamine signal during the behavioural tasks employed, including evaluation of baseline fluctuations. Relating dopamine concentration to aversive events to that seen to rewarding events will provide important insight into the role of dopamine in general and valence-specific learning mechanisms.

A thorough examination of accumbal dopamine concentration levels to a aversive event and its predictor using some of the key conditions under which a (reward) prediction error is reported. As a result of this thorough investigation the paper is a real tour de force. Some comments/concerns are outlined below.

Interpretation of the data.

The temporal window of DA. Across the experiments DA concentrations are examined during the predictive cue and the WN US and those concentrations are not in line with an error signal. This may be explained in terms of floor effects in the signal, but some aspects of the study argue against that – the signal during the CS for example. The authors have examined post WN period which shows a differential slope to an expected and an unexpected WN, which is in line with a PE account. A more complete analyses of the slope would be worth seeing. A justification for the slope is also necessary. Also, why not take the concentration during a specified post-WN period? The authors could analyze different temporal windows. Importantly, the post-WN period does suggest PE-like differences – most striking in Figure 2A, but potentially present in 2D, E,F.

The baseline subtraction. The authors have used a reasonable baseline, pre-CS period, of determining the change in DA release. However, it would be important to know if and how the baseline changes across the training sessions. It is unclear whether pre-cue baseline subtraction was done on a trial by trial basis or if one averaged baseline was calculated and then used to determine DA concentration change. This could influence the data.

Framing. While I really appreciated the framing of the approach within what has been reported for RPE, I wonder if failing to get the same profile is really conclusive regarding the absence of a PE signal using an aversive event. For example, a complete shift is not reported in all datasets that show a DA RPE signal. This can also be due to variability across animals. Further, it is possible that an aversive event may be tracked more categorically by DA , making the intensity and probability examinations less relevant for testing PE in the aversive case. Further, reduction in DA concentration may also be less sensitive to detecting these subtle changes. Is there anything in the data that can deal away with these points?

Methodology. It seems that a lot (all?) of the rats were ran across all experimental conditions. This raises concern over carry over effects.

Validation of the WN as aversive: The behavioural tasks do not include another auditory cue as a control comparison in the open field nor in the operant. The operant has different intensities of the WN, which helps matters as there is a difference between the 70dB and the 96dB. But it is unclear what the role of the WN is in these conditions. Is it just to show that it is aversive or that it can condition behaviour? I think there is evidence for the former but not the latter.

The manuscript should refer to the WN as an aversive stimulus, not punisher.

Please change all instances that refer to WN intensity (e.g. 70dB, 90dB, etc) as a volume. 'volume' is a colloquial way of referring to intensity and is therefore not appropriate in a scientific setting.

Reviewer #2 (Recommendations for the authors):

The manuscript by Goedhoop focuses on understanding the dopaminergic signals that are driven by aversive stimuli. The project uses fast scan cyclic voltammetry to directly record dopamine fluctuations in awake and behaving animals in response to a variety of task variables to parse their contribution to behavior across conditions.

There are a lot of strengths of this manuscript.

First – the use of white noise is innovative and powerful. The field often focuses on aversive footshocks, which are interesting but unique stimuli. The use of white noise allows for an aversive stimulus that is not painful and is not an electrical stimulus which is a significant advantage over previous studies.

Second – one issue in the field in general is that people focus on dopamine as an RPE encoder where dopamine in every context has been linked to RPE-like signaling. However, a shortcoming of previous work with direct dopaminergic recording approaches like voltammetry is that they are electrical in nature and thus, cannot record the response to the aversive stimulus (footshock) themselves. This is clearly and issue as the stimulus response is a critical variable to understand in order to make conclusions about whether something encodes "RPE" or not.

However, even with these strengths there are some significant weaknesses. These occur In both the conceptual presentation and the experimental execution and if addressed the manuscript would be much stronger.

1. Regarding the conceptual issues, the largest is the terminology used throughout the manuscript. One of the major issues in this manuscript is the definition of all aversive stimuli as punishers. A punishment has a specific definition that is incorrectly used here. A punisher is not defined by the valence of the stimulus, but rather the behavioral effect of that stimulus on future behavior. A punisher reduces rates of behavior – appetitive stimuli can also function as punishers. This is a huge problem and the wording in the manuscript should be changed to reflect this. This is incredibly problematic as it suggests that the findings are different than what they actually are on a conceptual level as they relate to what dopamine is doing.

2. There are many statements that are inherently problematic because of this mischaracterization of the behavior. For example: "this heterogeneity is reflected in dopamine responses to punishment throughout the striatum" is stated in the introduction; however, many of these studies are not punishment. Also, many people have suggested that dopamine controls motivational responses. In that case a "punisher" and "negative reinforcer" would show different dopaminergic signatures even though the maintaining stimulus is aversive in both cases. This is actually an important and overlooked aspect of this work and defining everything as a punisher makes it difficult to decipher what the data are showing and how that relates to the actual behavior of the animal.

3. These results can alternatively be explained by the novelty induced alterations of behavior in rodents. The literature has shown that rodents withhold consummatory behavior and novelty induces hyperactivity in rodents (e.g., Bardo et al.,1990; Psychopharmacol; but also see earlier paper from 1950s Berlyne 1955; Bindra and Spinner 1958; Welker 1959). The dopamine system is highly involved in both of these effects.

Regarding experimental issues:

1. The canonical unconditioned aversive response in rodents is freezing or immobility (e.g., Antoniadis and McDonald, 2001, Exp Brain Res). Here in Figure 1 and also in Figure 2C they show the whitenoise itself results in increase in locomotor activity. How do we know this is an aversive response comparable to other traditional aversive stimuli such as footshocks or tail pinches (which are shown to result in increase in NAc core dopamine release see Budygin et al., 2012; Mikhailova et al., 2019).

2. In Figure2A, the dopamine response to the white noise seems to be decreased. However, this is due to the baseline used to compute the white noise dopamine responses, which seems to shift lower due to the dopamine response to the antecedent cue. That is why the initial white noise dopamine response seems to be positive in Trial1 where the baseline is still above 0 but looks negative when the cue response becomes negative starting from Trial3. If the dopamine response to the white noise outcome were computed with a baseline of its own (1-2 sec before the WN outcome) that would result in a positive peak even in trials 11-30.

3. The authors claim that the decrease in dopamine response to the white noise during the first trial of aversive conditioning (Figure 2A) is an unconditioned response. However, there is an immediate positive peak after the white noise presentation on that trial, which lasts about 1 sec. How does the behavior map on to this timeline? Do rats move for the first second but then freeze for the remainder of the white noise presentation? At the very least a strong justification should be made for what is being normalized and if and how you can separate specific task components.

5. There are numerous studies where cues are paired with white noise as an aversive stimulus. It is important to determine if the predictive cue elicited a conditioned response. Without that how do you know the animals made the association? This is important to make conclusions about what the neural signal in response to these cues actually mean.

6. In response to the data with the white noise and different timing. Is this predicting the timing? Or the value of the outcome? These are not dissociable in this experiment and when you discuss timing this would be important to dissociate. This is a critical thing to parse as duration is in the title.

Overall this is an interesting manuscript however in order for it to be suitable for publication the authors should rephrase their terminology to accurately state what the stimuli are and how they relate to behavior as well as make sure to show that white noise does function as an aversive stimulus.

Reviewer #3 (Recommendations for the authors):

In this study a talented group of neurochemists performed real-time measurements of dopamine concentration in behaving rats to investigate whether transient accumbal release events encode the value of aversive stimuli. Directly measuring dopamine release events rather than phasic bursts of putative dopamine neural activity is particularly important to determine how transient dopamine signals encode aversive events because recent evidence shows that terminal-terminal modulation influences behaviorally relevant patterns of release that do not necessarily coincide with changes in neural activity. The authors also incorporated an impressive systematic behavioral design and a unique aversive stimulus (i.e., white noise) to address this unresolved controversy. First, they determined that high decibel white noise produced a conditioned place aversion and punished food seeking. Then, by presenting comparable levels of white noise within a Pavlovian context, they found that dopamine release events were suppressed during the presentation an aversive stimulus and its conditioned predictor. They further report that the magnitude by which dopamine release events were suppressed did not correlate with the amplitude of white noise; thereby leading them to conclude that transient dopamine signals in the core region of the nucleus accumbens respond to aversive stimuli, but do not necessarily encode the value of punishment. However, there remain several unresolved issues and points of contention regarding the interpretation of the authors' results. Of note, they did not measure dopamine release during punished behavior, but rather in the presence of an aversive stimulus that increased the behavior being assessed. In addition, it is not clear whether the rats were able to discriminate between the tightly dispersed decibels of white noise presented during the Pavlovian task in which dopamine concentration was measured. While the current results are intriguing and a technical advance over preceding electrochemical studies, the overall picture of how transient dopamine signals throughout the mesocorticolimbic pathway encode aversive stimuli still requires further clarification that the current group of authors are capable of providing.

The submission includes an excellent set of well-considered experiments; I am both impressed and intrigued. However, I do have some constructive criticism, suggestions, and alternative interpretations to consider.

A timeline or illustration of the different subgroups and conditions under which FSCV recordings occurred would increase the readability of the manuscript.

If the authors do not believe they are measuring dopamine (DA) value signals associated with aversive stimuli, have they considered whether they are measuring a correlate of the acoustic startle response? Acoustic startle is commonly associated with an increase in ambulation (as reported in the current manuscript). This alternative interpretation would provide an important missing piece of data from previously hypothesized neural circuitry underlying acoustic startle (see figure 6 of Koch and Schnitzler, 1997). Furthermore, the transient accumbal DA signal has previously been associated with pre-attentive sensory perception of salience. Might your results align more with Redgrave's work demonstrating that there are indeed distinct DA sensory responses that are dissociable from accumbal value signals; possibly also involved in the acoustic startle response?

Koch M, Schnitzler HU. The acoustic startle response in rats-circuits mediating evocation, inhibition and potentiation. Behavioural brain research. 1997 Dec 1;89(1-2):35-49.

The authors should discuss the current results in the context their previous work (specifically DA correlates with 22Kh USVs) with the Wohr lab, which was surprisingly not referenced in the current manuscript.

Willuhn I, Tose A, Wanat MJ, Hart AS, Hollon NG, Phillips PE, Schwarting RK, Wöhr M. Phasic dopamine release in the nucleus accumbens in response to pro-social 50 kHz ultrasonic vocalizations in rats. Journal of Neuroscience. 2014 Aug 6;34(32):10616-23.

It is not clear whether rats could actually discriminate between the different tightly dispersed white noise volumes in the Pavlovian task during FSCV measurements-a centrally important experiment to support the authors' conclusions about dopamine and value. Based on my current interpretation of the methods, rats were able to discriminate between the white noise volumes in the operant choice task, but the conditions were substantially different from those in which the FSCV recordings occurred. Aside from being instrumental rather than Pavlovian, five consecutive training sessions occurred for each of three white noise volumes (70, 90, 96db) before discrimination testing was tested; and approximately 100 trials occurred in each session, with a single white noise volume being tested per session. Then, during volume-response FSCV recordings, each of four white noise volumes (70, 80, 90, 96db) were randomly played 12 times each in a single session. In a separate group of recordings, the authors performed FSCV recordings in the presence of 80 vs. 96db white noise volumes. Compared to the volume-response FSCV recordings, more training and trials occurred but, the Pavlovian trials were still randomly presented within a session. Thus, I caution against assuming that the animals could discriminate between the less dispersed and randomly presented white noise volumes presented in the single session Pavlovian experiments (particularly those depicted in 2E) based on the results from a methodologically distinct instrumental choice experiment. The authors should further address (either experimentally or logically) why the reader should accept that the rats could discriminate between the tightly spaced decibel volumes (particularly in the volume-response experiment).

Please provide additional clarification on figure 2B and your interpretations of it. First, the dopamine response to the CS predicting the aversive stimulus does seem to increase with experience (albeit quickly), which I contend contradicts your statement starting on line 293: 'Although the WN-predictive, conditioned cue acquired the ability to reliably suppress NAC dopamine release, no substantial transfer of this effect from US to CS occurred a prerequisite for a prediction error signal.' Aversive stimuli are known to rapidly induce conditioned responses. For example, a conditioned fear response is often established in a single fear conditioning session with just a few pairings of the CS and aversive stimuli. Thus, I contend (as the authors generally indicate in the results) that the US to CS transfer does indeed occur during aversive conditioning, it just occurred rapidly (on day 1; as would occur in standard fear conditioning). Similar to the presentation of extinction data in 2A, what do the CS and US data look like trial-by-trial on day 1 of aversive conditioning?

Please provide an explanation regarding the degrees of freedom associated with the statistics used for your comparisons in figure 2B (lines 242-244). It is unclear to me how the data points from the top figures transfer to the bottom figures and the degrees of freedom are adding to my confusion.

Also, what affect did smoothing the data with a 10-point median filter prior to analysis have on the results? While smoothing the data for visual presentation is common, I question whether performing statistical analysis on the data after smoothing it might affect the results?

Furthermore, the maximal amplitude of the CS associated DA response during appetitive conditioning appears to be comparable between day 1 and day 6 in the top right panel of 2B. Thus, it seems that the duration of the signal, rather than it's amplitude, is responsible for the significant effect shown in the bar graph of the bottom panel of 2B. From this observation, is the duration of the signal not accounting for the majority of appetitive value coding? Did smoothing the data contribute to the longer duration?

The authors found that a longer duration of white-noise exposure produced a longer suppression in dopamine release. One could argue that both the frequency and amplitude of a signal should influence neural coding. Thus, why would a prolonged reduction in frequency not be reflective of greater aversive value? Why do the authors exclusively consider the amplitude of their dopamine in the context of aversive value determinations?

The use of white noise as an aversive stimulus is championed by the authors because it does not produce the behavioral confound of freezing observed in standard fear conditioning approaches using electrical footshock. However, white noise and its conditioned predictor increased ambulatory behavior in the current study. Thus, how is the logic of avoiding behavioral DA responses not flawed? Could DA responses correlate to the initiation of action (which could be dissociable from speed) not confound DA value coding assessments across different volumes of white noise?

I also have some related critiques to consider regarding the benefits of white noise espoused in the discussion, starting on line 435. I already pointed out that the logic of using white noise because it doesn't induce freezing and thereby avoids movement-DA related confounds is flawed as white noise increased ambulation-which again could confound DA value coding of aversive stimuli if indeed DA transients are directly related to movement. But the data from the current study and others (PMID: 24345819, figure 2) might suggest that accumbal transients are not actually correlated in a positive way to general increases in activity. Regardless, I also take issue with point 2 and 3 (line 438) because foot shock can also maintain avoidance across many trials and sessions and be titrated with varying valence and contingencies; what are you contrasting white noise too? Point 4 does not remove the potential confound of Redgrave's work, as he has repeatedly demonstrated that the accumbal transient DA response can be induced by 'pre-attentive' subcortical sensory input. Point 6 is also not exactly accurate as properly isolating the electrical components of a shock generator eliminates the noise artifacts it can produce; thus, white noise might be easier but I don't think it is fair to imply that properly set-up electrical foot shock interferes with or jeopardizes FSCV recordings. It is fair that electrical artifacts are detected during single-unit recordings but they are easy to detect and remove during sorting.

The study would be strengthened by a core vs. shell vs. PFC comparison but at the very least, the literature regarding the role of dopamine in these distinct regions should be addressed in the discussion. I acknowledge that performing measurements of DA release in the PFC is a fraught endeavor but, at this level of journal I would expect that you at least address the Tye lab's data on aversive stimuli and the PFC (generally reviewed in: Vander Weele et al. 2019). Is it possible that DA signals in the PFC but not the NAc core encode aversive value? You also point out that Badrinarayan et al., 2012 reported that the same CS that reduced dopamine in the core increased DA in the shell-a somewhat paradoxical finding that was neither explored in the current study nor addressed in the discussion. While it is an important replication to show that NAc DA in the core is reduced by conditioned predictors of aversive stimuli, and the analysis done in figure 3 is an impressive advance over previous studies, I am left questioning the advance provided by the current data set. The primary positive effect in the core is a replication. Building a story on the general role transient dopamine signals play in encoding aversive stimuli using negative FSCV effects that were determined using tightly spaced decibels of white noise that the rat may not have been able to discriminate is shaky.

Vander Weele CM, Siciliano CA, Tye KM. Dopamine tunes prefrontal outputs to orchestrate aversive processing. Brain research. 2019 Jun 15;1713:16-31

I strongly suggest that the authors not use the term punishment in the context of their results and instead use a term such as aversive stimulus. While multiple definitions of punishment exist in the literature, the one that is almost universally taught in the psychological context of animal behavior today is that of Azrin and Holz (1966), which considers punishment as a reduction in behavior in response to a stimulus. According to this definition, the behavioral response is the key element in determining whether you are observing punishment or a reinforcement; with punishment describing a decrease in behavior and reinforcement describing an increase in behavior. It is additionally worth considering that an aversive stimulus (e.g., electrical shock) can function as either a punisher or a reinforcer (i.e., something that increases behavior) in the operant context (McKearny, 1966; Morse and McKearney 1977), during imprinting (Hess, 1959), or in human interaction (Mello 1978; Sack and Miller, 1975). In the current study white noise does seem to punish food-maintained responding in one context, but also increases ambulatory behavior in another context.

Azrin NH, Holz WC. Punishment. Operant behavior: Areas of research and application. 1966:380-447

McKearney JW. Maintenance of responding under a fixed-interval schedule of electric shock-presentation. Science. 1968 Jun 14;160(3833):1249-51.

Morse WH, McKearney JW, Kelleher RT. Control of behavior by noxious stimuli. In Handbook of psychopharmacology 1977 (pp. 151-180). Springer, Boston, MA.

Hess EH. Imprinting. Science. 1959 Jul 17;130(3368):133-41.

Mello NK. Control of drug self-administration: The role of aversive consequences. Phencyclidine (PCP) Abuse: An appraisal. National Institute on Drug Abuse Research Monograph. 1978 Aug 1(21):289-308.

Sack RL, Miller W. Masochism: A clinical and theoretical overview. Psychiatry. 1975 Aug 1;38(3):244-57.

Along this same line of thought, could it not be concluded that DA might scale with the value of punishment but not of aversive stimuli that do not actually reduce the occurrence of a behavior? Punishment-associated DA signals in the basal ganglia might be more correlated to stimuli that actually reduce behavior given this neural circuit's well-determined role in goal-directed learning. Thus, if you want to conclude that DA fails to encode punishment (rather than an aversive stimulus), I would want to actually see transient DA signals failing to correlate to stimuli that reduce behavior to different magnitudes.

[Editors’ note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting the paper entitled "Nucleus-accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error" for further consideration by eLife. Your revised article has been evaluated by a Senior Editor and a Reviewing Editor. We are sorry to say that we have decided that this submission will not be considered further for publication by eLife.

During the review process, the editors and reviewers evaluated and discussed your revised manuscript and your responses to the initial concerns. Unfortunately, all agreed that the data provided are insufficient to convince that the white noise is aversive. The data in Figure 1A and B that are used to make this point do not have the necessary controls. Further, db do not scale linearly. Therefore, the level of aversiveness of the different db of white noise also needs to be verified behaviourally. The valence of the white noise is a the backbone to the story of the paper and the absence of strong evidence that speaks to this issue within the paper was judged to be problematic, precluding it from further consideration for publication. We're sure this is not the decision you were hoping for, but appreciate the chance to reconsider your manuscript and hope that our evaluation will be useful for you as you move forward with this work.

[Editors’ note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Nucleus-accumbens dopamine tracks aversive-stimulus duration and prediction but not value or prediction error" for further consideration by eLife. Your revised article has been evaluated by Kate Wassum (Senior Editor) and a Reviewing Editor.

The manuscript has been improved but there are some remaining issues that need to be addressed, as outlined below:

Reviewer #1 (Recommendations for the authors):

The authors did an excellent job addressing my previous concerns. Of note, their additional data provide convincing evidence that animals could discriminate the different db of white noise and that all intensities function as aversive stimuli in naive rats. The additional locomotor control experiments also strengthen the manuscript. The text is also significantly improved, particularly regarding the concept of punishment. I found the experiments to be more intriguing with better framing. The new additional summary table helped me navigate the manuscript and, importantly, illustrates the impressive breadth of experiments conducted to address a contentious and important question in the field.

Reviewer #2 (Recommendations for the authors):

The authors have done a good job at responding to previous comments. I do think that the study is interesting and important for the field. I have a few additional comments that should be addressed.

Several manuscripts have come out recently specifically looking at dopamine release and aversive associative learning. These are surprisingly not cited or mentioned at all in the current manuscript and are highly relevant to the current work (Kutlu et al., 2022, Nature Neuroscience; Kutlu et al., 2021,. Current Biology). Both of these studies record dopamine release in the NAc core during aversive conditioning and relate dopamine signals to aversive stimulus responses and omissions based on previous predictions.

It would be interesting and important for the authors to discuss how aversive stimuli that induce different unconditioned responses – freezing, vs increased motor activity – could relate to dopamine signatures that they induce. Would the authors expect that dopamine responses to aversive stimuli that induce freezing be opposite to those that drive increases in activity?

https://doi.org/10.7554/eLife.82711.sa1

Author response

[Editors’ note: the authors resubmitted a revised version of the paper for consideration. What follows is the authors’ response to the first round of review.]

Reviewer #1 (Recommendations for the authors):

Understanding the role of dopamine beyond reward and specifically in aversion is of fundamental importance. The paper set out to examine whether dopamine concentration in the nucleus accumbens tracks aversive prediction error as it does in reward. To do this, the authors used fast-scan cyclic voltammetry alongside a number of key behavioural manipulations that have been shown to provide conditions for detecting reward prediction error. The behavioral designs used are established and appropriate to the questions posed. While the aversive cue, a loud white noise, employed was not standard, the paper provides behavioural evidence for its effectiveness as an aversive outcome. Although a closer examination of the learning that underlies the task design is necessary. While the analyses of the neural signal are appropriate, some alternatives could be explored in order to better understand the profile of the dopamine signal during the behavioural tasks employed, including evaluation of baseline fluctuations. Relating dopamine concentration to aversive events to that seen to rewarding events will provide important insight into the role of dopamine in general and valence-specific learning mechanisms.

A thorough examination of accumbal dopamine concentration levels to a aversive event and its predictor using some of the key conditions under which a (reward) prediction error is reported. As a result of this thorough investigation the paper is a real tour de force. Some comments/concerns are outlined below.

Interpretation of the data.

The temporal window of DA. Across the experiments DA concentrations are examined during the predictive cue and the WN US and those concentrations are not in line with an error signal. This may be explained in terms of floor effects in the signal, but some aspects of the study argue against that – the signal during the CS for example. The authors have examined post WN period which shows a differential slope to an expected and an unexpected WN, which is in line with a PE account. A more complete analyses of the slope would be worth seeing. A justification for the slope is also necessary. Also, why not take the concentration during a specified post-WN period? The authors could analyze different temporal windows. Importantly, the post-WN period does suggest PE-like differences – most striking in Figure 2A, but potentially present in 2D, E,F.

This (no APE signal) may be explained in terms of floor effects in the signal, but some aspects of the study argue against that – the signal during the CS for example.

The reviewer expresses a very reasonable concern, but also points out that we present data that speaks against the floor-effect hypothesis: (1) In Figure 3C, we show that the overall dopamine decrease during CS+US is greater than during CS alone. This would not be the case if a floor effect was present during the 6 s WN presentation. (2) Figures3D and 3G demonstrate that when WN duration is extended, dopamine concentration continues to decrease further than what is seen for the 6 s WN we applied in most of the experiments. Thus, at no point did we detect a floor effect. Moreover, even in the presence of a hypothetical floor effect, the WN-dopamine response would still not align with an aversive-prediction error (APE), because in Figure 3E, the dopamine recovery slope in case of a better than-expected scenario (CS alone) is the same as for the expected scenario (CS+WN) (e.g., no observable rebound that encodes an “error”).

A justification for the slope is also necessary.

Our justification for using the slope in Figures3D-F as a readout for dopamine dynamics, is that it is more sensitive and informative for probing dopamine function in the framework of an APE (based on the results reported in figures prior) than averaging dopamine concentration over a defined time window/epoch (which we otherwise employed), or alternatives employed in other studies, such as area under the curve or peak concentration. Using the slope allows for integration of the change in dopamine concentration over time, as opposed to averaging concentrations over an epoch (in which there is no integration over time). Averaging over an epoch would be insensitive to whether dopamine rebound recovers rapidly or slowly, and unable to determine when (during the averaged epoch) a dopamine change takes place. This type of information is crucial in Figures3D-F, as it allows us to differentiate between tracking an aversive stimulus and encoding of an APE: Averaging the changes in dopamine concentration would incorrectly lead to the interpretation that dopamine encodes prediction errors, even though the data can be explained by a much easier computation, i.e. the tracking of WN or its predictor (the data in Figure 3F is an indirect exception, as dopamine concentration recovers slower following unpredicted WN). Thus, in this case, averaging over an epoch would yield false positive results and would have failed to detect the tracking. This level of information depth is not required to draw conclusions in scenarios where slope does not differ, as for example in Figure 2A. In conclusion, assessing slopes in dopamine-concentration change provides another level of information.

Also, why not take the concentration during a specified post-WN period? The authors could analyze different temporal windows.

The reviewer is correct, we could use different temporal windows to achieve similar sensitivity as for the slope analysis. However, there is no advantage in doing so compared to the slope. In fact, it would be less objective and more complicated (e.g., determining the size of appropriate temporal windows), as well as being less powerful statistically. Thus, we believe that the slope analysis provides a superior readout.

“Importantly, the post-WN period does suggest PE-like differences – most striking in Figure 2A, but potentially present in 2D, E,F.

It is unclear to us how Figure 2A suggests evidence for PE-like differences. It simply shows that the concentration goes down during WN onset and (later in the session) also in response to the CS predicting WN. And after WN cessation the concentration goes up again (post-WN period). Although these data are not completely inconsistent with an APE per se, they are better explained by the “tracking explanation”, as the dopamine dips do not shift over to the CS substantially (as analyzed in Figure 2B left), thus speaking against an APE (even before considering the additional evidence against an APE provided in later figures). Furthermore, statistical analysis demonstrates that there are no significant slope differences in Figure 2A.

Similarly, Figures2D-F (now Figures2D,E,G in the new version of the manuscript) do not provide convincing evidence of an APE. Particularly, experiments describing different WN intensities in Figures2E and G (formerly F) speak against an APE, since a prediction error should by definition respond to different outcome values.

We think, that the reviewer may be referring to “aversive prediction” (a function restricted to reacting to cues that predict an aversive stimulus, without reacting to prediction errors) instead of APE, since there is certainly strong evidence for aversive prediction (i.e., the CS acquires the ability to decrease dopamine on its own) in the figures that the reviewer refers to.

The baseline subtraction. The authors have used a reasonable baseline, pre-CS period, of determining the change in DA release. However, it would be important to know if and how the baseline changes across the training sessions. It is unclear whether pre-cue baseline subtraction was done on a trial by trial basis or if one averaged baseline was calculated and then used to determine DA concentration change. This could influence the data.

The reviewer addresses an important point. Pre-cue baseline subtraction was performed on a trial-by-trial basis, which minimizes the effects of changes over time (including across training sessions). Therefore, baseline changes across training sessions did not significantly influence the reported outcomes. This is further underlined by the fact that pre-cue baseline was stable for the most part, and did not drift up or down significantly over time (see the horizontal pre-event dopamine traces in all figures). We have added information to the Methods section to clarify how we performed the baseline subtraction.

While we completely agree that it would be very interesting to record baseline changes across training sessions, this lies both beyond the technical possibilities of the experiments conducted, as well as beyond the scope of our study. APE- or tracking-like changes occur on a short time scale (seconds). The only baseline difference that could hypothetically influence the experimental outcomes would be the above-mentioned floor effect, which, as we pointed out above, did not occur.

Framing. While I really appreciated the framing of the approach within what has been reported for RPE, I wonder if failing to get the same profile is really conclusive regarding the absence of a PE signal using an aversive event. For example, a complete shift is not reported in all datasets that show a DA RPE signal. This can also be due to variability across animals. Further, it is possible that an aversive event may be tracked more categorically by DA , making the intensity and probability examinations less relevant for testing PE in the aversive case. Further, reduction in DA concentration may also be less sensitive to detecting these subtle changes. Is there anything in the data that can deal away with these points?

I wonder if failing to get the same profile is really conclusive regarding the absence of a PE signal using an aversive event.

The reviewer brings up a good methodological point. An axiomatic approach as repeatedly validated and performed by Rutledge, Glimcher and colleagues is arguably the most objective way to formally identify a prediction error (Caplin et al., 2010; Hart et al., 2014), because it tests several fundamental assumptions regarding prediction errors systematically and simultaneously. We applied this approach to aversive-stimulus prediction as reported in Figures2G and H and Figures3A-C. Neither the results of this approach nor any other of our results support the hypothesis that an APE is encoded by NAC dopamine. We therefore believe it justified to conclude that WN-induced changes are best described by the “tracking explanation” than an “APE explanation” (especially since “tracking” describes dopamine dynamics well).

Caplin A, Dean M, Glimcher PW, Rutledge RB. Measuring beliefs and rewards: A neuroeconomic approach. Q J Econ. 2010 Dec 31;125(3):923960.

Hart AS, Rutledge RB, Glimcher PW, Phillips PE. Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J Neurosci. 2014 Jan 15;34(3):698-704.

For example, a complete shift is not reported in all datasets that show a DA RPE signal. This can also be due to variability across animals.

Although the reviewer is correct that perhaps not all reported datasets exhibit a complete dopamine RPE shift, we are not aware of any dataset (recorded with a similar technique) demonstrating as little of a shift as reported in our manuscript for WN-cue conditioning (e.g., Day et al., 2007; Clark et al., 2010; Brown et al., 2011; Hollon et al., 2014; not even in so-called goal-tracking animals (our experimental setup did not allow for goal-tracking): Flagel et al., 2011). Moreover, as a control, we provide reward-conditioning related dopamine measurements that were taken under almost identical circumstances as our WN data, that demonstrate a much more substantial shift to the cue (Figure 2B, right). The data from these animals displays very little variance. Importantly, a complete shift is not the point we intended to make (since this can be dependent on training duration); rather the point is that RPEs shift substantially more than the reported WN data with comparable training amount. In addition, to extend on the topic of training duration: For some of the experiments, we trained animals for 9 days (Figure 3B and C), which is a very substantial amount of training.

Day JJ, Roitman MF, Wightman RM, Carelli RM. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat Neurosci. 2007 Aug;10(8):1020-8.

Clark JJ, Sandberg SG, Wanat MJ, Gan JO, Horne EA, Hart AS, Akers CA, Parker JG, Willuhn I, Martinez V, Evans SB, Stella N, Phillips PE. Chronic microsensors for longitudinal, subsecond dopamine detection in behaving animals. Nat Methods. 2010 Feb;7(2):126-9.

Brown HD, McCutcheon JE, Cone JJ, Ragozzino ME, Roitman MF. Primary food reward and reward-predictive stimuli evoke different patterns of phasic dopamine signaling throughout the striatum. Eur J Neurosci. 2011; 34(12):1997-2006.

Hollon, N. G., Arnold, M. M., Gan, J. O., Walton, M. E. and Phillips, P. E. M. Dopamine-associated cached values are not sufficient as the basis for action selection. Proc. Natl. Acad. Sci. U. S. A. 111, 18357–18362 (2014).

Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, Willuhn I, Akers CA, Clinton SM, Phillips PE, Akil H. A selective role for dopamine in stimulusreward learning. Nature. 2011 Jan 6;469(7328):53-7.

…aversive event may be tracked more categorically by DA, making the intensity and probability examinations less relevant for testing PE in the aversive case.

We believe that the reviewer is making the exact point we are trying to convey: Aversive events are tracked more categorically by dopamine. It “only” tracks the presence of an aversive stimulus or its predictive stimulus via a slow but constant decrease in NAC dopamine concentration, irrespective of the size of the prediction error experienced. This is one example for how our findings are inconsistent with a formal prediction error, which is generally conceptualized as a quantitative discrepancy between the outcome expected and the outcome experienced and, thus, scales with the relative value of the encoded stimulus and unexpected deviations from this value (BrombergMartin and Hikosaka, 2009; Nasser et al., 2018). We believe this is sufficient evidence to rule out encoding of that what is commonly defined as a prediction error, and that this is a noteworthy finding, as very few studies have addressed this issue systematically.

Bromberg-Martin, E.S., and Hikosaka, O. (2009). Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron, 63(1), 119-126.

Nasser HM, Lafferty DS, Lesser EN, Bacharach SZ, Calu DJ (2018). Disconnection of basolateral amygdala and insular cortex disrupts conditioned approach in Pavlovian lever autoshaping. Neurobiol Learn Mem. 147:35-45.

Reduction in DA concentration may also be less sensitive to detecting these subtle changes.

Unfortunately, the meaning of this sentence is unclear to us. Is the reviewer referring to reductions as opposed to increases in dopamine concentration? Or that the measurement of reductions is more difficult technically? With regards to the latter, the applied technique, fast-scan cyclic voltammetry (FSCV), is very capable of detecting both (phasic) increases and decreases in extracellular dopamine concentration, as evidenced by Figure 1G. With regards to the former, we believe that we may have addressed the concern already in our discussion: “The dopamine system presumably has a bigger dynamic range for increasing activity; it can do so, for example, by increasing the number of cells firing and their firing frequency (and thereby the total number of dopamine-containing vesicles being released). In contrast, dopamine-signaling reduction cannot drop below a certain point, since the cells’ maximum response is to cease firing altogether, and extracellular dopamine can only be removed relatively slowly or must diffuse away. This disparity could translate into a structurally-limiting factor on what can be encoded by a reduction in dopamine concentration and explain some of the above-mentioned differences in function. However, the slow-ramping declining and recovery slopes we observed do not reflect the system limits, since the very first exposure to WN resulted in a steeper decline and rewards given during WN resulted in steeper increases.”

Methodology. It seems that a lot (all?) of the rats were ran across all experimental conditions. This raises concern over carry over effects.

Although we understand concerns relating to carry-over effects in animal behavioral paradigms in general, we are unsure what kind of carry-over effects the reviewer is referring to in the context of our experiments that may compromise our data. Even if there was such an effect conceivable, it would not affect our results, as all comparisons are made within-session. Besides, the initial multi-day cue conditioning reveals dopamine signals maintain the same size between days 1 and 6, thus, no habituation takes place in this context. Furthermore, we now provide new, additional data demonstrating that animals differentiate between different WN intensities, even after many sessions of WN exposure (Figure 2F). Finally, Figures 3D-F demonstrate that dopamine traces induced by WN still look similar to day 1 of exposure, yet change readily when experimental conditions are altered.

On the contrary, we believe our approach of re-using animals between experiments in fact increases consistency between experiments (especially regarding concerns about interindividual differences). Besides this, there is the ethical advantage of reducing the number of animals used.

However, we may address the reviewer’s concern about carry-over effects quantitatively by calculating data stability across time:

Author response image 1
Cue and WN induce dopamine decrease across repeated experiments in the same animals.

Average dopamine concentration during cue and WN epochs in the animals depicted in Figures2A (first 30 trials on day 1 of conditioning), 2G (deterministic experiment), and 3B and C (probabilistic experiment). There is a significant main effect (F(1.428, 7.141) = 9.402, p = 0.0133) of recording session during the cue epoch, and post-hoc testing determined there is a significant difference between day 1 of conditioning and the deterministic experiment (p = 0.0035; Tukey’s multiple comparisons test). However, we found no effect of recording session on dopamine concentration during the WN epoch (F(1.236, 6.178) = 0.03898, p = 0.8936) indicating that the neurochemical response to WN remains stable across days.

Validation of the WN as aversive: The behavioural tasks do not include another auditory cue as a control comparison in the open field nor in the operant. The operant has different intensities of the WN, which helps matters as there is a difference between the 70dB and the 96dB. But it is unclear what the role of the WN is in these conditions. Is it just to show that it is aversive or that it can condition behaviour? I think there is evidence for the former but not the latter.

The behavioural tasks do not include another auditory cue as a control comparison in the open field nor in the operant [box]”

The reviewer makes a good suggestion: Indeed, adding a control stimulus would enable useful comparisons. However, we do not believe that the lack of these control comparisons undermines our conclusions, since the animals avoid WN at higher intensities more than at lower intensities (Figure 1C; operant box), and in new data we show that animals differentiate between different WN intensities as exemplified by increasing locomotion with higher WN intensities (Figure 2F; operant box). A cue predictive of WN only slowly acquires locomotion-inducing effects with multiple pairings (Figure 2C, left). Once this cue is no longer predictive of WN, it loses its dopamine-reducing effects (Figure 2D). Finally, the auditory cues used here are standard sounds produced by commercially available and widely used Med Associates operant boxes, which have been used in a plethora of studies without obvious aversive effects.

Above all, we did not investigate what kind of sound or which aspect of WN is aversive. We investigated how the dopamine system reacts to an aversive stimulus and its prediction.

Is it just to show that it is aversive or that it can condition behaviour?

It is to show both. Our interpretation is that WN is not just aversive, but its conditioning to a cue can also elicit conditioned behavior. Figure 2C shows a cue-induced increase in speed (green trace during cue presentation from 0-5 s), which is a conditioned response.

The manuscript should refer to the WN as an aversive stimulus, not punisher.

We agree with the reviewer and have changed the manuscript accordingly.

Please change all instances that refer to WN intensity (e.g. 70dB, 90dB, etc) as a volume. 'volume' is a colloquial way of referring to intensity and is therefore not appropriate in a scientific setting.

We changed the manuscript accordingly (“intensity” instead of “volume”).

Reviewer #2 (Recommendations for the authors):

The manuscript by Goedhoop focuses on understanding the dopaminergic signals that are driven by aversive stimuli. The project uses fast scan cyclic voltammetry to directly record dopamine fluctuations in awake and behaving animals in response to a variety of task variables to parse their contribution to behavior across conditions.

There are a lot of strengths of this manuscript.

First – the use of white noise is innovative and powerful. The field often focuses on aversive footshocks, which are interesting but unique stimuli. The use of white noise allows for an aversive stimulus that is not painful and is not an electrical stimulus which is a significant advantage over previous studies.

Second – one issue in the field in general is that people focus on dopamine as an RPE encoder where dopamine in every context has been linked to RPE-like signaling. However, a shortcoming of previous work with direct dopaminergic recording approaches like voltammetry is that they are electrical in nature and thus, cannot record the response to the aversive stimulus (footshock) themselves. This is clearly and issue as the stimulus response is a critical variable to understand in order to make conclusions about whether something encodes "RPE" or not.

However, even with these strengths there are some significant weaknesses. These occur In both the conceptual presentation and the experimental execution and if addressed the manuscript would be much stronger.

1. Regarding the conceptual issues, the largest is the terminology used throughout the manuscript. One of the major issues in this manuscript is the definition of all aversive stimuli as punishers. A punishment has a specific definition that is incorrectly used here. A punisher is not defined by the valence of the stimulus, but rather the behavioral effect of that stimulus on future behavior. A punisher reduces rates of behavior – appetitive stimuli can also function as punishers. This is a huge problem and the wording in the manuscript should be changed to reflect this. This is incredibly problematic as it suggests that the findings are different than what they actually are on a conceptual level as they relate to what dopamine is doing.

The reviewer is correct. Although we were aware that our use of the word “punishment” was not in the pure sense of its definition in Psychology, and although we stated in the Introduction of the previous version of the manuscript that: “we will frequently refer to aversive stimuli as punishments, irrespective of whether conditioning was of instrumental or Pavlovian nature.”, we agree it was a suboptimal use of the word “punishment”. We understand the reviewer’s concern about the deviation from a definition based on the reduction rate of behavior. We have changed the manuscript accordingly to reflect that we indeed intend to describe and focus on the valence of the stimulus. We replaced the word “punishment” with “aversive stimulus” and the term “punishment-prediction error” with “aversive prediction error”. However, as much as we agree that this was a suboptimal choice in terminology, we do not share the reviewer’s opinion that this was a “huge problem” and “incredibly problematic”: Regardless of terminology, the data convincingly reflect the dopamine response to aversive white noise, and our observations on its contrast to dopamine dynamics around rewards are not compromised.

2. There are many statements that are inherently problematic because of this mischaracterization of the behavior. For example: "this heterogeneity is reflected in dopamine responses to punishment throughout the striatum" is stated in the introduction; however, many of these studies are not punishment. Also, many people have suggested that dopamine controls motivational responses. In that case a "punisher" and "negative reinforcer" would show different dopaminergic signatures even though the maintaining stimulus is aversive in both cases. This is actually an important and overlooked aspect of this work and defining everything as a punisher makes it difficult to decipher what the data are showing and how that relates to the actual behavior of the animal.

We agree with the reviewer. As stated above under point 1, we replaced the word “punishment” with “aversive stimulus”. We believe that has appropriately removed what the reviewer refers to as “mischaracterization of the behavior” and guides the focus of our manuscript to the valence of the WN stimulus. By refining our definition, we have also addressed the reviewers second point: the data reflect dopamine response and animal behavior associated with aversive stimuli. We agree with the reviewer that terminology is important to communicate our findings appropriately.

3. These results can alternatively be explained by the novelty induced alterations of behavior in rodents. The literature has shown that rodents withhold consummatory behavior and novelty induces hyperactivity in rodents (e.g., Bardo et al.,1990; Psychopharmacol; but also see earlier paper from 1950s Berlyne 1955; Bindra and Spinner 1958; Welker 1959). The dopamine system is highly involved in both of these effects.

The reviewer posits an interesting alternative hypothesis to explain some of our data. However, although we agree that novelty may also induce increased locomotion and although we cannot exclude that novelty may have played a role in the first few presentations of the WN, we believe that novelty does not explain our findings:

1) Novelty is defined as “the quality of being new and unusual” (https://dictionary.apa.org/novelty). The reviewer cites several articles that support the conclusion that novelty vanishes on the scale of minutes (Welker, 1959; Berlyne, 1955; Bindra and Spinner 1958). Our rats, however, were exposed to WN hundreds of times, over the course of up to 10 sessions, with each a duration of 1.5 hours, and with (on average) approximately 66 exposures per session; thus, to these rats, WN cannot be described as novel, new, or unusual. We see this reflected in the dopamine response, which is stable across many sessions with many WN presentations.

(As a side note: Bardo et al. (1990) studied the effect of dopaminergic drugs on locomotion in a novel environment, but did not compare to a non-novel environment).

2) We report that locomotion and dopamine are not correlated (linearly; Figure 2H).

3) Novelty tends to induce locomotion towards a novel object or sensory stimulus, whereas the animals in our study avoid the WN stimulus (Figure 1A).

4) We are unsure why the reviewer brings up the withholding of consummatory behavior as an alternative explanation or confounding factor, since our experiments do not deal with withholding of consummatory behavior. Thus, the reported decrease in dopamine concentration cannot be explained by stopping consummatory behavior. 5) Dopamine release is not tied to consummatory behavior itself, as has been shown by a number of studies that demonstrate a temporal shift in dopamine release away from consummatory behavior, to the cue that predicts foodreward delivery (e.g., Day et al., 2007; Clark et al., 2010; Brown et al., 2011; Hollon et al., 2014).

Berlyne DE. The arousal and satiation of perceptual curiosity in the rat. J Comp Physiol Psychol. 1955 Aug;48(4):238-46.

D Bindra, N Spinner. Response to different degrees of novelty: the incidence of various activities. J Exp Anal Behav. 1958 Oct;1(4):341-50. – Welker WI. Escape, exploratory, and food-seeking responses of rats in a novel situation. J Comp Physiol Psychol. 1959 Feb;52(1):106-11. – MT Bardo 1, SL Bowling, RC Pierce. Changes in locomotion and dopamine neurotransmission following amphetamine, haloperidol, and exposure to novel environmental stimuli. Psychopharmacology (Berl). 1990;101(3):338-43.

Day JJ, Roitman MF, Wightman RM, Carelli RM. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat Neurosci. 2007 Aug;10(8):1020-8.

Clark JJ, Sandberg SG, Wanat MJ, Gan JO, Horne EA, Hart AS, Akers CA, Parker JG, Willuhn I, Martinez V, Evans SB, Stella N, Phillips PE. Chronic microsensors for longitudinal, subsecond dopamine detection in behaving animals. Nat Methods. 2010 Feb;7(2):126-9.

Brown HD, McCutcheon JE, Cone JJ, Ragozzino ME, Roitman MF. Primary food reward and reward-predictive stimuli evoke different patterns of phasic dopamine signaling throughout the striatum. Eur J Neurosci. 2011; 34(12):1997-2006.

Hollon, N. G., Arnold, M. M., Gan, J. O., Walton, M. E. and Phillips, P. E. M. Dopamine-associated cached values are not sufficient as the basis for action selection. Proc. Natl. Acad. Sci. U. S. A. 111, 18357–18362 (2014).

Regarding experimental issues:

1. The canonical unconditioned aversive response in rodents is freezing or immobility (e.g., Antoniadis and McDonald, 2001, Exp Brain Res). Here in Figure 1 and also in Figure 2C they show the whitenoise itself results in increase in locomotor activity. How do we know this is an aversive response comparable to other traditional aversive stimuli such as footshocks or tail pinches (which are shown to result in increase in NAc core dopamine release see Budygin et al., 2012; Mikhailova et al., 2019).

As stated by Antoniadis and McDonald (2001), somatomotor immobility (freezing) is a common fear response in many species, however, the same paper also states that another common fear response is withdrawal (avoidance or escape) from the danger, which includes increased locomotion. In fact, unconditioned aversive response in rodents depends on the situation/environment and aversiveness of the stimulus. As Schoonover et al. (2017) state: “Rodents exhibit a variety of defensive behaviors in response to innately aversive and conditioned cues, including flight, freezing, crouching, defensive threat, defensive attack, and burying of potentially threatening objects (Blanchard and Blanchard, 2008; Blanchard et al., 1986; Blanchard et al., 1998). The category of response elicited by a given cue depends on the context in which it is presented (Bolles and Collier, 1976; Bouton and Bolles, 1980; Pinel and Triet, 1978; Yilmaz and Meister, 2013), the nature of the conditioned stimulus (Karpicke et al., 1977; Pinel and Triet, 1978), and the ongoing behavioral state of the animal (Fentress, 1968a, b).… The flight from danger, in contrast to freezing, is a stereotyped behavioral motif whose onset consists of a rapid transition in behavioral state (Blanchard et al., 1998; De Franceschi et al., 2016; Domenici and Blake, 1991; Walther, 1969; Yilmaz and Meister, 2013).” Thus, besides freezing, animals may increase locomotion when avoiding aversive stimuli such as foot shocks, when given the opportunity to escape them; in our experiment, the animals attempt to avoid the stimulus (Figures1A and B), which is likely the reason for increased locomotion. Therefore, we believe that these different aversive stimuli elicit comparable responses.

With regards to the reviewer’s final point, we are aware of this literature and had in fact discussed it in our manuscript, to illustrate that previous studies report dopamine-system responses in both directions: “Contradictory findings are also reported in the neighboring nucleus accumbens core (NAC), where studies found both increased (Budygin et al., 2012; Mikhailova et al., 2019) and decreased dopamine activity (Badrinarayan et al., 2012; Oleson et al., 2012; DeJong et al., 2019; Stelly et al., 2019).”. It is noteworthy that the experiments by Budygin et al. (2012) and by Mikhailova et al. (2019) were conducted in anesthetized animals, whereas the other experiments cited above were conducted in awake animals, as were ours.

EA Antoniadis, RJ McDonald. Amygdala, hippocampus, and unconditioned fear. Exp Brain Res. 2001 May;138(2):200-9. doi: 10.1007/s002210000645.

CE Schoonover, AJP Fink, R Axel (2017). A naturalistic assay for measuring behavioral responses to aversive stimuli at millisecond timescale. https://doi.org/10.1101/161885.

2. In Figure2A, the dopamine response to the white noise seems to be decreased. However, this is due to the baseline used to compute the white noise dopamine responses, which seems to shift lower due to the dopamine response to the antecedent cue. That is why the initial white noise dopamine response seems to be positive in Trial1 where the baseline is still above 0 but looks negative when the cue response becomes negative starting from Trial3. If the dopamine response to the white noise outcome were computed with a baseline of its own (1-2 sec before the WN outcome) that would result in a positive peak even in trials 11-30.

In Figure 2A, the dopamine response to the white noise seems to be decreased.

Correct, this is what we report: dopamine decreases during WN presentation in Figure 2A, compared to baseline. Please see the traces and their respective bar graphs plotted over them, averaging the dopamine-response across 5s for baseline and cue, and 6s for WN responses.

However, this is due to the baseline used to compute the white noise dopamine responses, which seems to shift lower due to the dopamine response to the antecedent cue.

We believe this is a misunderstanding: the baseline used for the WN dopamine trace is the same as the baseline used for the cue trace (i.e., the 5s before cue onset (see Methods)). The average values of the dopamine traces for baseline, cue, and WN periods are depicted in the bar graphs inset on the plots (respectively white, light-grey, and dark-grey backgrounds).

That is why the initial white noise dopamine response seems to be positive in Trial1 where the baseline is still above 0 but looks negative when the cue response becomes negative starting from Trial3.

We understand that the reviewer observes by eye an increase in the initial dopamine response to WN in Trial1, however, the statistical analysis does not support this observation (there is no significant increase of dopamine during the first second of exposure to WN (p = 0.6606)) and, concomitantly, we did not report this as a result; the same is true for Trial3. Note: we would like to strongly emphasize that the error bars on the dopamine traces are substantial, which should be taken into consideration when translating graphical observations to statistical verification.

If the dopamine response to the white noise outcome were computed with a baseline of its own (1-2 sec before the WN outcome) that would result in a positive peak even in trials 11-30.

There was no significant positive peak in trials 11-30, no matter where the baseline for this trace is taken.

Somewhat unrelatedly: In case the reviewer assumes that the dopamine concentration stays decreased for the period of the variable inter-trial interval of 60s (range: 30-90 s): the five seconds following WN offset (displayed in Figure 2A) indicate that dopamine concentration trails back to baseline. This becomes more obvious in Figures 3D and E that depicts longer periods of time after WN offset.

3. The authors claim that the decrease in dopamine response to the white noise during the first trial of aversive conditioning (Figure 2A) is an unconditioned response. However, there is an immediate positive peak after the white noise presentation on that trial, which lasts about 1 sec. How does the behavior map on to this timeline? Do rats move for the first second but then freeze for the remainder of the white noise presentation? At the very least a strong justification should be made for what is being normalized and if and how you can separate specific task components.

As explained in our response to the reviewer’s point 2 (see above), this “immediate positive peak” is not statistically significant, and concomitantly, we did not report this as a result. Note: the error bars on the trial-one dopamine traces are substantial. Even in trial one, there is no significant increase of dopamine during the first second of exposure to WN (p = 0.6606) or its predictive cue (p = 0.0757). The initial bump in dopamine in Trial1 might be due to the novelty of the cue and the relative novelty of the first WN presentation of the session.

Hypothetically speaking, even if there were a statistically significant initial small peak in dopamine response (which there is not), we believe it would not seriously challenge our claim that the dopamine decrease to WN is unconditioned, since this hypothetical one-second dopamine bump is only present in Trial1. We do not think that it is reasonable to assume that a mild stressor like WN would lead to “conditioning” within one second.

Our data in Figure 1C (left) do not indicate any freezing. Furthermore, locomotion speed never decreases during the first trial of aversive Pavlovian conditioning (see Author response image 2 (n = 29)), instead we see an increase in speed during the WN epoch. Thus, WN induces an unconditioned response from the first exposure onwards. At no point in time do we see a significant drop in locomotion speed indicative of freezing behavior.

Author response image 2

5. There are numerous studies where cues are paired with white noise as an aversive stimulus. It is important to determine if the predictive cue elicited a conditioned response. Without that how do you know the animals made the association? This is important to make conclusions about what the neural signal in response to these cues actually mean.

Although we do not agree with the reviewer that there are numerous studies where cues are paired with WN as an aversive stimulus, we completely agree that it is important to determine if the predictive cue elicited a conditioned response. To this end, we had already provided data that strongly supports this claim in Figure 2C (left) of our original version of the manuscript. In the revised version of the manuscript, we now add even more animals to the analysis in this figure panel. The result and its interpretation remain unchanged: The predictive cue elicits a conditioned response, as indicated by increased locomotion during cue presentation compared to baseline locomotion on day 6 of conditioning (Z = -3.053, p = 0.002), before the WN is presented (Figure 2C, left). Thus, we demonstrate that the animals made the association between cue and WN.

6. In response to the data with the white noise and different timing. Is this predicting the timing? Or the value of the outcome? These are not dissociable in this experiment and when you discuss timing this would be important to dissociate. This is a critical thing to parse as duration is in the title.

It is not 100% clear to us what the reviewer is referring to when stating: “Is this predicting the timing? Or the value of the outcome?”. We interpret the reviewer’s comment as asking whether our results indicate that dopamine tracks timing/duration or the value of the outcome. Which is an important point, a point that we addressed in our experiments. We concluded in the original manuscript that dopamine is “tracking the timing/duration” (i.e., presence of the WN), because we show that value was not encoded (Figures2E and G and Figure 3B), but that instead WN induces a categorical, slow and steady decrease in dopamine lasting the duration of the presence of the aversive stimulus (Figures3D-F), followed by a slow recovery period. The slope of this decrease is slower than it could be (compare to fast decreasing slope on Trial1 of Figure 2A), as is its recovery (compare to increase in dopamine in Figure 2B, top right). This, to us, conclusively demonstrates that the signal is tracking aversive duration, and that it is not tracking aversive value, as it is not conceivable to us how value could be represented in such a nearly binary fashion

Overall this is an interesting manuscript however in order for it to be suitable for publication the authors should rephrase their terminology to accurately state what the stimuli are and how they relate to behavior as well as make sure to show that white noise does function as an aversive stimulus.

We agree with the reviewer on terminology and have made appropriate changes throughout the manuscript.

…make sure to show that white noise does function as an aversive stimulus.

We demonstrated in two different experiments that WN is an aversive stimulus (Figures1A and B). Also, others have demonstrated the aversiveness of WN previously (e.g., Campbell 1955; Harrison and Tacy, 1955; Campbell and Bloom, 1965; Hughes and Bardo, 1981).

Campbell, BA (1955). The fractional reduction in noxious stimulation required to produce "just noticeable" learning. Journal of Comparative and Physiological Psychology, 48(3), 141–148.

Harrison JM, Tracy WH. Use of auditory stimuli to maintain lever-pressing behavior. Science. 1955 Mar 11;121(3141):373-4. 3.

Campbell, BA and Bloom, JM. Relative aversiveness of noise and shock. J. Comp. Physiol. Psychol. 60, 440–442 (1965).

Hughes, RA and Bardo, MT. Shuttlebox avoidance by rats using white noise intensities from 90-120 db SPL as the UCS. J. Aud. Res. 21, 109–118 (1981).

Reviewer #3 (Recommendations for the authors):

In this study a talented group of neurochemists performed real-time measurements of dopamine concentration in behaving rats to investigate whether transient accumbal release events encode the value of aversive stimuli. Directly measuring dopamine release events rather than phasic bursts of putative dopamine neural activity is particularly important to determine how transient dopamine signals encode aversive events because recent evidence shows that terminal-terminal modulation influences behaviorally relevant patterns of release that do not necessarily coincide with changes in neural activity. The authors also incorporated an impressive systematic behavioral design and a unique aversive stimulus (i.e., white noise) to address this unresolved controversy. First, they determined that high decibel white noise produced a conditioned place aversion and punished food seeking. Then, by presenting comparable levels of white noise within a Pavlovian context, they found that dopamine release events were suppressed during the presentation an aversive stimulus and its conditioned predictor. They further report that the magnitude by which dopamine release events were suppressed did not correlate with the amplitude of white noise; thereby leading them to conclude that transient dopamine signals in the core region of the nucleus accumbens respond to aversive stimuli, but do not necessarily encode the value of punishment. However, there remain several unresolved issues and points of contention regarding the interpretation of the authors' results. Of note, they did not measure dopamine release during punished behavior, but rather in the presence of an aversive stimulus that increased the behavior being assessed. In addition, it is not clear whether the rats were able to discriminate between the tightly dispersed decibels of white noise presented during the Pavlovian task in which dopamine concentration was measured. While the current results are intriguing and a technical advance over preceding electrochemical studies, the overall picture of how transient dopamine signals throughout the mesocorticolimbic pathway encode aversive stimuli still requires further clarification that the current group of authors are capable of providing.

The submission includes an excellent set of well-considered experiments; I am both impressed and intrigued. However, I do have some constructive criticism, suggestions, and alternative interpretations to consider.

A timeline or illustration of the different subgroups and conditions under which FSCV recordings occurred would increase the readability of the manuscript.

We thank the reviewer for their suggestion, and have added Author response table 1 to the supplementary material of the manuscript:

Author response table 1
ExperimentNFSCV readoutBehavioral readout
Real time place aversion of randomly assigned quadrant (Figure 1A)10X
Real time place aversion of preferred quadrant (Figure 1B)15X
Approach-avoidance foraging task (Figure 1C)12X
Operant box WN exposures (Figure 1D)14X
WN and reward choice task (Figure 1F)6X
Aversive Pavlovian conditioning day 1
  • First 30 trials (Figure 2A, B, C and E)

  • Mix trials (Figure 3D, E and F)

16 11X XX
Appetitive Pavlovian conditioning (Figure 2B and C)10XX
Aversive Pavlovian conditioning days 2–6 (Figures 1E and 2B and C)4 13XX X
Dose response (Figures 1D and 2F)6 13XX X
Deterministic experiment (Figure 2E and G)9X
Probabilistic experiment (Figures 2E, 3A and B and C)9X
Concurrent reward & WN and extinction (Figures 2E and 3G and H)6X

If the authors do not believe they are measuring dopamine (DA) value signals associated with aversive stimuli, have they considered whether they are measuring a correlate of the acoustic startle response? Acoustic startle is commonly associated with an increase in ambulation (as reported in the current manuscript). This alternative interpretation would provide an important missing piece of data from previously hypothesized neural circuitry underlying acoustic startle (see figure 6 of Koch and Schnitzler, 1997). Furthermore, the transient accumbal DA signal has previously been associated with pre-attentive sensory perception of salience. Might your results align more with Redgrave's work demonstrating that there are indeed distinct DA sensory responses that are dissociable from accumbal value signals; possibly also involved in the acoustic startle response?

Koch M, Schnitzler HU. The acoustic startle response in rats-circuits mediating evocation, inhibition and potentiation. Behavioural brain research. 1997 Dec 1;89(1-2):35-49.

The reviewer suggests a reasonable hypothesis. The acoustic startle response (ASR) is one of the most commonly used behavioral responses to study habituation. In order to startle, ASR studies use sound intensities that are comparable (or higher) to the WN intensities we used. However, these studies commonly use a shorter sound duration of one second (we used six seconds) and a significantly lower number of trials per session (e.g., 10x), which are spaced apart with a shorter ITI than we used.

ASR habituation occurs both in the short-term and the long-term. More specifically, ASR short-term habituation occurs across trials within a session and long-term habituation across behavioral sessions. Thus, to study short-term habituation (a), amplitudes of the startle response across trials are compared, where a habituation effect occurs across several trials. To study long-term habituation (b), amplitudes of the startle response of the very first trial are compared across sessions, where a habituation effect occurs within five days.

We performed equivalent analyses on our dataset, comparing the average speed of our animals during WN exposures (a) across trials on day one and (b) in the first trials of sessions one and six:

a) Across trials on day one, we do indeed observe a decreased increase in locomotion speed during the WN epoch, which suggests that there might be short-term habituation (see Author response image 3). However, since this decrease in locomotion is not reflected in the dopamine responses (no differences in dopamine during the WN across these trials; see Figure 2A), we conclude that the dopamine responses that we observe are not encoding ASRs (under the assumption the reported behavioral response to WN is an ASR).

Author response image 3
Locomotion speed across the first 30 trials on day 1 of conditioning of the rats (n=16) for which we show the dopamine traces in the NAC in Figure 2A.

b) When we compare locomotion speed during the WN epoch of the first trials on days one and six, we do not find a significant difference (t(16)=0.9111, p=0.3757) and, therefore, conclude that long-term habituation does not take place in our paradigm (see Author response image 4).

Author response image 4
Comparing locomotion speed during the first trials of days 1 and 6 of aversive conditioning (total n = 17).

No significant difference was found between average locomotion speed during the WN epoch in trials 1 between days 1 and 6 (t(16)=0.9111, p=0.3757), thus, no long-term habituation of the behavioral response occurred..

Together, these additional data do not support the interpretation that our reported dopamine signals encode an ASR to WN, because (1) the hypothetical long-term ASR does not habituate. We think it is more likely that our 6-s WN presentation (with longer ITIs) provokes a behavioral response that is inconsistent with startle. And (2) the hypothetical short-term ASR habituates, but this decrease in locomotion is not reflected in the dopamine signals. Furthermore, dopamine responses dopamine signals are not correlated with the behavioral response whatsoever.

Our results regarding negative dopamine signals in response to an aversive stimulus may, however, align with the idea of transient NAC dopamine signal associated with pre-attentive sensory perception of salience and, thus, with Redgrave's work demonstrating distinct dopamine sensory responses that are dissociable from NAC value signals.

The authors should discuss the current results in the context their previous work (specifically DA correlates with 22Kh USVs) with the Wohr lab, which was surprisingly not referenced in the current manuscript.

Willuhn I, Tose A, Wanat MJ, Hart AS, Hollon NG, Phillips PE, Schwarting RK, Wöhr M. Phasic dopamine release in the nucleus accumbens in response to pro-social 50 kHz ultrasonic vocalizations in rats. Journal of Neuroscience. 2014 Aug 6;34(32):10616-23.

We did not reference our previous work on NAC dopamine responses to USVs in our manuscript because we did not think it relevant enough. Rats emit 22-kHz ultrasonic vocalizations (USVs) in aversive situations such as predator exposure, fear conditioning, or social defeat. Such calls are considered “alarm calls” that serve a communicative function, as their presentation induces call-specific freezing behavior in the receiver. WN on the other hand is frequency-unspecific noise (thereby essentially blocking the entire auditory modality, a sense that rodents rely on heavily), serves no communicative function, and, as a mildly aversive stimulus, does not induce freezing. Thus, WN does not induce the same behavioral response as 22-kHz (and probably does not elicit 22-kHz USVs). Taken together, it is not too surprising that qualitative differences are found in dopamine responses to the presentation of WN and 22-kHz USVs, as well as between 50-kHz (‘positive’ USVs) and 22-kHz USVs. On request of the reviewer, we have added a sentence on this matter to the Discussion section.

It is not clear whether rats could actually discriminate between the different tightly dispersed white noise volumes in the Pavlovian task during FSCV measurements-a centrally important experiment to support the authors' conclusions about dopamine and value. Based on my current interpretation of the methods, rats were able to discriminate between the white noise volumes in the operant choice task, but the conditions were substantially different from those in which the FSCV recordings occurred. Aside from being instrumental rather than Pavlovian, five consecutive training sessions occurred for each of three white noise volumes (70, 90, 96db) before discrimination testing was tested; and approximately 100 trials occurred in each session, with a single white noise volume being tested per session. Then, during volume-response FSCV recordings, each of four white noise volumes (70, 80, 90, 96db) were randomly played 12 times each in a single session. In a separate group of recordings, the authors performed FSCV recordings in the presence of 80 vs. 96db white noise volumes. Compared to the volume-response FSCV recordings, more training and trials occurred but, the Pavlovian trials were still randomly presented within a session. Thus, I caution against assuming that the animals could discriminate between the less dispersed and randomly presented white noise volumes presented in the single session Pavlovian experiments (particularly those depicted in 2E) based on the results from a methodologically distinct instrumental choice experiment. The authors should further address (either experimentally or logically) why the reader should accept that the rats could discriminate between the tightly spaced decibel volumes (particularly in the volume-response experiment).

The reviewer asks an important question: whether rats could discriminate between the different WN intensities. Naturally, we asked ourselves this question, too. First, we should correct a misunderstanding: these intensities are not “tightly dispersed” or “tightly spaced”, as decibels are on a logarithmic scale, and a 10dB difference equals a 10fold increase in intensity (perceived as a 2-fold increase in sound volume). This is easiest explained with real-life examples of the extremes of our range: 70dB is comparable to the sound of a running shower or a flushing toilet, whereas 96dB is the sound of a jackhammer or a power drill. Rats are well-hearing animals that rely on this sensory modality for many things. Based on appropriate literature, we chose our loudness range carefully, making sure that it was large enough for meaningful discrimination, and that the loudest condition is not harmful to the animals.

As pointed out by the reviewer, there was indeed more training leading up to the choice test, however, for the Pavlovian exposure, we also administered many trials: for the deterministic experiment (where 80 and 96 dB WN followed a cue with a 100% probability), FSCV measurements were carried out on the fourth day of conditioning (with 80 WN exposures per day) and for the probabilistic experiment (where 80 and 96 dB WN followed a cue with a probability of 75% or 25%), FSCV measurements were carried out after an additional four days of conditioning (with 80 WN exposures per day).

As the reviewer mentions, we show that the rats were able to discriminate between the WN intensities in the operant choice task. We disagree with the reviewer that the conditions in the operant choice task (where rats readily discriminated between white noise intensities), were substantially different from the Pavlovian presentation (where the reviewer questions the rats’ ability to discriminate between the WN intensities): Both were carried out in the exact same operant box, noise was played from the same speakers therein, and animals had the same amount of habituation. Thus, we believe that the WN intensities were perceived identically across these experiments, and it is therefore reasonable to assume that since the rats could discriminate between them in one, they could discriminate between them in the other.

Finally, we present new behavioral data in Figure 2F, which demonstrate that rats respond differently to different WN intensities, and thus discriminate between them.

Please provide additional clarification on figure 2B and your interpretations of it. First, the dopamine response to the CS predicting the aversive stimulus does seem to increase with experience (albeit quickly), which I contend contradicts your statement starting on line 293: 'Although the WN-predictive, conditioned cue acquired the ability to reliably suppress NAC dopamine release, no substantial transfer of this effect from US to CS occurred a prerequisite for a prediction error signal.' Aversive stimuli are known to rapidly induce conditioned responses. For example, a conditioned fear response is often established in a single fear conditioning session with just a few pairings of the CS and aversive stimuli. Thus, I contend (as the authors generally indicate in the results) that the US to CS transfer does indeed occur during aversive conditioning, it just occurred rapidly (on day 1; as would occur in standard fear conditioning). Similar to the presentation of extinction data in 2A, what do the CS and US data look like trial-by-trial on day 1 of aversive conditioning?

We agree with the reviewer, the data in Figure 2B could be described more clearly. We added information and clarification in response to the following remark by the reviewer (see next point on degrees of freedom).

…the dopamine response to the CS predicting the aversive stimulus does seem to increase with experience (albeit quickly), which I contend contradicts your statement starting on line 293: 'Although the WN-predictive, conditioned cue acquired the ability to reliably suppress NAC dopamine release, no substantial transfer of this effect from US to CS occurred a prerequisite for a prediction error signal.’

We are in agreement that some transfer does indeed occur during aversive conditioning. We do not state that “no transfer” occurs, we state that “no substantial transfer” occurs. This is an important distinction, as it implies that learning and associated dopamine-signal transfer occurs during aversive conditioning, however, with the distinction that only a relatively small percentage of the dopamine signal transfers from US to CS. (1) This percentage is small in comparison to appetitive conditioning; and (2) equally importantly, this percentage is not anywhere close to the amount of transfer that is a prerequisite for a prediction-error signal is (which requires full transfer with full predictability).

Similar to the presentation of extinction data in 2A, what do the CS and US data look like trial-by-trial on day 1 of aversive conditioning.

We are confused by this request. Figure 2A depicts “trial-by-trial data on day 1 of aversive conditioning” (some trials have been pooled because data does not change noticeably). Furthermore, Figure 2D depicts extinction data, not Figure 2A.

Please provide an explanation regarding the degrees of freedom associated with the statistics used for your comparisons in figure 2B (lines 242-244). It is unclear to me how the data points from the top figures transfer to the bottom figures and the degrees of freedom are adding to my confusion.

In order to make our point regarding the differences in the shift of the dopamine response from US to CS between appetitive and aversive Pavlovian conditioning more clear and easier to understand, we have now quantified this shift in a different manner than in the original manuscript (using the same data): In the new manuscript, we quantify this shift by calculating the ratio between US and CS dopamine concentration as a deviation from baseline (in the respective up or down direction). For aversive conditioning, this ratio was determined by dividing the area above the curve of the WN epoch by area above the curve of the cue epoch (WN AUC / cue AUC). For appetitive conditioning, this ratio was determined by dividing the area under the curve of the pellet epoch by the area under the curve of the cue epoch (pellet AUC / cue AUC).

We subdivided the results of day 1 into “day 1 (early)” (trials 2-4; trial 1 was excluded to minimize the saliency response) and “day 1 (late)” (trials 5-30), and also calculated the ratio for day 6. We used mixed effects analyses (because of one missing value for both aversive and appetitive Pavlovian conditioning) to separately test for the differences between the ratios during aversive and appetitive Pavlovian conditioning. During aversive conditioning we found a significant main effect of the amount of conditioning (F(0.9586, 2.396)=117.3, p = 0.0043), and post-hoc testing using Tukey’s multiple comparisons test reveals significant differences between the ratios of day 1 early trials and day 1 later trials (p = 0.0138), as well as between day 1 early trials and day 6 (p = 0.0318). However, no difference was observed between day 1 later trials and day 6 (p = 0.9852). During appetitive conditioning we also found a main effect of the amount of conditioning on the ratio between the US and the CS (F(1.034,8.788)=13.88, p=0.0047), and, in contrast to aversive conditioning, the ratio on day 6 is significantly different from both day 1 early trials (p=0.0102) and day 1 later trials (p<0.0001). In addition, we found a significant difference between day 1 early trials and day 1 later trials (p=0.0441). We have added this information to the manuscript.

Also, what affect did smoothing the data with a 10-point median filter prior to analysis have on the results? While smoothing the data for visual presentation is common, I question whether performing statistical analysis on the data after smoothing it might affect the results?

Smoothing data with 5- to 10-point filters is common practice for this type of data (including for statistics). It provides a better signal-to-noise ratio. To illustrate that this smoothing had no distorting effect on the data, we include a figure that compares raw data to the same data after being processed with the 10-point median filter (see Author response image 5). We can conclude from this graph that the smoothing of the data does not change the overall shape of the dopamine responses, but just eliminates the noise that accompanies this type of measurement technique.

Author response image 5
Statistical effects are not distorted by 10-point median filter.

Furthermore, the maximal amplitude of the CS associated DA response during appetitive conditioning appears to be comparable between day 1 and day 6 in the top right panel of 2B. Thus, it seems that the duration of the signal, rather than its amplitude, is responsible for the significant effect shown in the bar graph of the bottom panel of 2B. From this observation, is the duration of the signal not accounting for the majority of appetitive value coding? Did smoothing the data contribute to the longer duration?

From this observation, is the duration of the signal not accounting for the majority of appetitive value coding?

Although there is a change in the duration of the appetitive CS signal between days 1 and 6, there are a number of qualitative differences of this duration change compared to aversive conditioning (if that is what the reviewer is hinting at): (1) We know from a number of studies (including our own) that this average or AUC dopamine signal is highly responsive to changes in reward value (including changes in both amplitude and duration of the signal – as opposed to the WN response). (2) Increased release of dopamine (amplitude) automatically prolongs signal duration, since the signal is (mostly) actively inactivated (e.g., reuptake) – unlike decreases associated with WN (release could theoretically bring dopamine concentration back to baseline very quickly). (3) As can be seen from Figure 2B bottom right: The appetitive signal shift (US to CS) continues to grow dramatically within the session on day 1, as well as between days 1 and 6 (which includes a large amplitude change – unlike aversive conditioning). (4) The slope of change for dopamine concentration is different between appetitive and aversive conditioning. (5) See further points made in the discussion paragraph that compares appetitive and aversive conditioning.

Together, these points indicate that for appetitive conditioning, both duration and amplitude of the dopamine signal are modulated depending on the stimulus, whereas for aversive conditioning only the duration is modulated (depending on the stimulus duration).

Did smoothing the data contribute to the longer duration?

No, smoothing did not contribute to the duration of our dopamine signals. We used a 10-point median filter on data that was acquired with a rate of 10 Hz. Since we smoothed each data point by taking the median value of the 10 surrounding points (thus in total a time window of 1 second), the signal can only be prolonged for a maximum of 0.5 second. In the figure that accompanies our answer to the previous question, you can see an example graph in which we compared data which was processed with the 10 point median filter to the raw data. We can conclude from this graph that the smoothing of the data does not change the overall shape of the dopamine responses but just eliminates noise that is accompanied by this type of measuring technique.

The authors found that a longer duration of white-noise exposure produced a longer suppression in dopamine release. One could argue that both the frequency and amplitude of a signal should influence neural coding. Thus, why would a prolonged reduction in frequency not be reflective of greater aversive value? Why do the authors exclusively consider the amplitude of their dopamine in the context of aversive value determinations?

In our manuscript, we basically argue that in order for a brain signal to be a value signal, it should reflect all changes in value of a stimulus. Such changes in value include both the duration and the intensity of the “appraised” stimulus. In the case of WN, WN intensity is not encoded in NAC dopamine signals as Figure 2E and Figure 2G (formerly 2F) show clearly (all WN intensity produce the same amplitude in dopamine dip), even though the animals discriminated between intensities behaviorally. This is also true for Figures3BandC, where different probabilities alter the value of the stimuli, but do not alter the dopamine signal. So, we believe that this is already sufficient to disqualify NAC dopamine as encoding aversive value, since a value signal should reflect all general changes in the value of a stimulus (and it is not encoding an obvious, salient one such as intensity).

Even though we consider the above point to be sufficient for our argument, we will speculate with the reviewer on his hypothesis about duration: As we understand it, the reviewer is arguing that (despite the amplitude of the dopamine signal) value may be encoded in the dopamine signal in a quantitative manner by the duration of the dopamine dip. Our interpretation is that the dopamine dip is indeed encoding aversiveness, but in a qualitative manner (i.e., this is what we mean by tracking negative valence) via a constant decrease in dopamine during the presence of WN, because this constant decrease in the presence of WN is not modulated by WN intensity. In case duration of the dopamine dip encodes aversive value this way, it should drop for a differently long period of time with differently aversive WN, but we don’t see such duration encoding of value in Figure 2E, Figure 2G (formerly 2F), and Figures3B and C. Additionally, when unexpected WN durations are presented (Figures3D and E), the dip and recovery slopes are the same compared to the expected WN duration. Thus, the signal is not reacting to changes in WN duration in any other way than just decreasing in the presence of WN and increasing in the absense of WN (with the exception of Figure 3F).

The use of white noise as an aversive stimulus is championed by the authors because it does not produce the behavioral confound of freezing observed in standard fear conditioning approaches using electrical footshock. However, white noise and its conditioned predictor increased ambulatory behavior in the current study. Thus, how is the logic of avoiding behavioral DA responses not flawed? Could DA responses correlate to the initiation of action (which could be dissociable from speed) not confound DA value coding assessments across different volumes of white noise?

The reviewer raises an interesting hypothetical point, if there was any evidence for increased ambulatory behavior corresponding to decreased dopamine transients, as we see in our data. However, to our knowledge, all relevant literature that links dopamine to action finds these variables to correlate positively (i.e., there are no reported observations of decreasing dopamine-neuron activity). Knowing this, we believe the most obvious interpretation of our statement that using WN overcomes the common confound of freezing in foot shock approaches, is that we may be certain in our data that the dopamine decrease we observe is not due to lack of ambulatory behavior, but instead due to the aversive stimulus itself.

I also have some related critiques to consider regarding the benefits of white noise espoused in the discussion, starting on line 435. I already pointed out that the logic of using white noise because it doesn't induce freezing and thereby avoids movement-DA related confounds is flawed as white noise increased ambulation-which again could confound DA value coding of aversive stimuli if indeed DA transients are directly related to movement. But the data from the current study and others (PMID: 24345819, figure 2) might suggest that accumbal transients are not actually correlated in a positive way to general increases in activity. Regardless, I also take issue with point 2 and 3 (line 438) because foot shock can also maintain avoidance across many trials and sessions and be titrated with varying valence and contingencies; what are you contrasting white noise too? Point 4 does not remove the potential confound of Redgrave's work, as he has repeatedly demonstrated that the accumbal transient DA response can be induced by 'pre-attentive' subcortical sensory input. Point 6 is also not exactly accurate as properly isolating the electrical components of a shock generator eliminates the noise artifacts it can produce; thus, white noise might be easier but I don't think it is fair to imply that properly set-up electrical foot shock interferes with or jeopardizes FSCV recordings. It is fair that electrical artifacts are detected during single-unit recordings but they are easy to detect and remove during sorting.

We have addressed the reviewer’s first point with regards to locomotion as a confounding factor in our response to the previous point.

Regarding the reviewer taking issue with points 2 and 3: As stated by us on line 433 (of the previous manuscript draft), we are contrasting WN to more commonly-used aversive stimuli. This includes electric footshock, as well as other stimuli such as tail pinch, fox urine, social defeat, restraint/immobilization stress etc. Although we agree that electric shock is easier to titrate than most commonly-used aversive stimuli, we disagree that electric footshock at intensities that are commonly used is well-tolerated across many hundreds of trials. This is also the opinion of the Animal-research Ethics Committee we are working with. And as pointed out by reviewer 1, electric shock often involves pain.

And as stated by others (doi: 10.1016/j.bbr.2010.06.020): “.. electric shock has its downsides: It often induces secondary effects such as long term sleep disruption [41], altered social behavior [27], reduction in locomotion, rearing, and grooming behaviors, as well as an increase in immobility and defecation [45].” – Thus, it is beneficial to explore other aversive stimuli, as we have done here.

Furthermore, commonly-used metal grids for electric foot-shock delivery frequently fail to deliver shock reliably and/or consistently due to several factors (addressing point 3).

Regarding the reviewer’s comment on point 4: First of all, this list of advantages of WN is not restricted to the use of WN in dopamine research. Furthermore, it is very possible that an animal does not detect a stimulus light because it is turned away from the light source or has closed its eyes or has buried its snout in its fur while grooming. In all of these situations, WN would still be detected.

Regarding the reviewer’s comment on point 6: We have toned down this statement about electrical interference by removing the term “jeopardize”. However, at the very least, the data collected during foot-shock application is unusable (and in FSCV recordings often more than that), which is a period of significant value in a study such as ours. We are contented to hear the reviewer has no problems dealing with electrical interference from foot shocks in FSCV and single-unit data, as this is a prohibitive drawback in most labs that specialize in these types of recordings. This opinion is also shared by researchers that are using techniques other than FSCV: “Finally, the nature of footshock stimulation precludes its full inclusion in some modern experimental techniques, such as electrophysiology.” (doi: 10.1016/j.bbr.2010.06.020).

None of these points are intended to discredit foot shock as an aversive stimulus, which is obviously a very good model that has been and still is used widely, but merely to point out the benefits of WN.

The study would be strengthened by a core vs. shell vs. PFC comparison but at the very least, the literature regarding the role of dopamine in these distinct regions should be addressed in the discussion. I acknowledge that performing measurements of DA release in the PFC is a fraught endeavor but, at this level of journal I would expect that you at least address the Tye lab's data on aversive stimuli and the PFC (generally reviewed in: Vander Weele et al. 2019). Is it possible that DA signals in the PFC but not the NAc core encode aversive value? You also point out that Badrinarayan et al., 2012 reported that the same CS that reduced dopamine in the core increased DA in the shell-a somewhat paradoxical finding that was neither explored in the current study nor addressed in the discussion. While it is an important replication to show that NAc DA in the core is reduced by conditioned predictors of aversive stimuli, and the analysis done in figure 3 is an impressive advance over previous studies, I am left questioning the advance provided by the current data set. The primary positive effect in the core is a replication. Building a story on the general role transient dopamine signals play in encoding aversive stimuli using negative FSCV effects that were determined using tightly spaced decibels of white noise that the rat may not have been able to discriminate is shaky.

Vander Weele CM, Siciliano CA, Tye KM. Dopamine tunes prefrontal outputs to orchestrate aversive processing. Brain research. 2019 Jun 15;1713:16-31

Although we agree with the reviewer that comparing our findings between accumbal subregions (including the shell and different dorso-striatal regions) and other, cortical regions is very interesting, this was explicitly not our question; we motivate our choice of brain region (NAC) in the introduction. Based on previous findings, the NAC was the most likely brain region to express a dopamine APE. The mentioned work by Vander Weele et al. (2019) on the effects of dopamine in the PFC on signal-to-noise ratio is very impressive and interesting, but it does not address APEs at all, which is why we did not discuss it in our manuscript.

We appreciate the reviewer’s question: “Is it possible that DA signals in the PFC but not the NAc core encode aversive value?” Yes, we believe that is very much possible, and indeed a very good point that is highly interesting and deserves the attention of future studies (especially, because anatomical findings indicate that different midbrain dopamine neurons project to NAC and cortex). But again, discussing this would be outside of the scope of our manuscript, since we do not have any experimental data to base such a discussion on. We believe that the best we can do in our manuscript is to acknowledge that our data suggest that aversive value and APEs are encoded in other brain regions (as mentioned in our discussion).

We believe it is more than just a replication to show that NAc DA in the core is reduced by conditioned predictors of aversive stimuli, in a field of research where results on this topic have been all over the place. We provide recordings that were conducted in several different paradigms that consistently improve clarity on this topic.

We have addressed in great detail the reviewer’s misunderstanding about the intensities of WN in a previous question, but to reiterate: these intensities are in no way tightly spaced. And, we have addressed the reviewer’s concern that rats may not have been able to discriminate the different WN intensities.

I strongly suggest that the authors not use the term punishment in the context of their results and instead use a term such as aversive stimulus. While multiple definitions of punishment exist in the literature, the one that is almost universally taught in the psychological context of animal behavior today is that of Azrin and Holz (1966), which considers punishment as a reduction in behavior in response to a stimulus. According to this definition, the behavioral response is the key element in determining whether you are observing punishment or a reinforcement; with punishment describing a decrease in behavior and reinforcement describing an increase in behavior. It is additionally worth considering that an aversive stimulus (e.g., electrical shock) can function as either a punisher or a reinforcer (i.e., something that increases behavior) in the operant context (McKearny, 1966; Morse and McKearney 1977), during imprinting (Hess, 1959), or in human interaction (Mello 1978; Sack and Miller, 1975). In the current study white noise does seem to punish food-maintained responding in one context, but also increases ambulatory behavior in another context.

We agree with the reviewer, and have made appropriate changes throughout the manuscript.

Along this same line of thought, could it not be concluded that DA might scale with the value of punishment but not of aversive stimuli that do not actually reduce the occurrence of a behavior? Punishment-associated DA signals in the basal ganglia might be more correlated to stimuli that actually reduce behavior given this neural circuit's well-determined role in goal-directed learning. Thus, if you want to conclude that DA fails to encode punishment (rather than an aversive stimulus), I would want to actually see transient DA signals failing to correlate to stimuli that reduce behavior to different magnitudes.

The reviewer raises a very good point, however we have followed his and the other reviewer’s rightly-made suggestion and replaced the term “punishment” throughout the manuscript.

[Editors’ note: what follows is the authors’ response to the second round of review.]

During the review process, the editors and reviewers evaluated and discussed your revised manuscript and your responses to the initial concerns. Unfortunately, all agreed that the data provided are insufficient to convince that the white noise is aversive. The data in Figure 1A and B that are used to make this point do not have the necessary controls. Further, db do not scale linearly. Therefore, the level of aversiveness of the different db of white noise also needs to be verified behaviourally. The valence of the white noise is a the backbone to the story of the paper and the absence of strong evidence that speaks to this issue within the paper was judged to be problematic, precluding it from further consideration for publication. We’re sure this is not the decision you were hoping for, but appreciate the chance to reconsider your manuscript and hope that our evaluation will be useful for you as you move forward with this work.

To address the above-mentioned concerns, we have added multiple new experiments to the current submission. For example, to improve the real-time place-aversion paradigm, we conducted an experiment that introduces white noise (WN) to the chamber quadrant that was previously preferred by the animals. Furthermore, we invented an approach-avoidance paradigm, which demonstrates that the utilized WN intensities do, in fact, scale linearly in their aversiveness. Repeated WN exposures in this paradigm demonstrate that the rats’ response to the different WN intensities were reliable across sessions, validating the absence of sensitization or habituation of the behavioral response to WN across sessions and days. Similarly, we now also demonstrate that (just like the behavioral response) the dopamine response to WN does not change across days.

Additionally, we control for the type of stimulus by introducing a 70-dB tone to the animals, which proved to be less aversive than 70-dB WN. Moreover, we provide data that shows how intensity-dependent and reliable across days the locomotor response to WN is. Together, we strongly believe, the results from these additional experiments address all remaining concerns raised after the previous article submission, strengthening the conclusions we drew in our previous submission.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

The manuscript has been improved but there are some remaining issues that need to be addressed, as outlined below:

Reviewer #2 (Recommendations for the authors):

The authors have done a good job at responding to previous comments. I do think that the study is interesting and important for the field. I have a few additional comments that should be addressed.

Several manuscripts have come out recently specifically looking at dopamine release and aversive associative learning. These are surprisingly not cited or mentioned at all in the current manuscript and are highly relevant to the current work (Kutlu et al., 2022, Nature Neuroscience; Kutlu et al., 2021,. Current Biology). Both of these studies record dopamine release in the NAc core during aversive conditioning and relate dopamine signals to aversive stimulus responses and omissions based on previous predictions.

We agree with the reviewer that these references deserve being mentioned in the context of our work. Thus, as requested, we now cite these publications in the manuscript.

It would be interesting and important for the authors to discuss how aversive stimuli that induce different unconditioned responses – freezing, vs increased motor activity – could relate to dopamine signatures that they induce. Would the authors expect that dopamine responses to aversive stimuli that induce freezing be opposite to those that drive increases in activity?

The reviewer brings up a relevant point. As indicated by the editor, studies by others, including Robert Rescorla and Gavan McNally, utilized even louder white noise (around 120db) than we have in this work. We refrained from using WN at such intensities due to an elevated risk of loss of hearing (Escabi et al., 2019), when used at durations that we applied in our experiments (6+ seconds per WN presentation). Brief 100-ms presentations of 120-dB WN induced fear responses such as freezing. Future studies will have to investigate whether such an opposite behavioral response (compared to behavioral activation reported here), is accompanied by differential dopamine dynamics, perhaps an increase in dopamine release (potentially related to increases in activity). Our analysis indicates that the NAC dopamine response is somewhat independent of behavioral actions (see Figure 2H). Thus, we speculate that despite opposite behavioral patterns, 120-dB WN will also induce decreased dopamine. This speculation is supported by the work of others, who found decreased dopamine release during the administration of electric shocks that induce freezing (Badrinarayan et al., 2012; Oleson et al., 2012; De Jong et al., 2019; Stelly et al., 2019).

We have added this point to the Discussion section of our manuscript.

https://doi.org/10.7554/eLife.82711.sa2

Article and author information

Author details

  1. Jessica N Goedhoop

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Conceptualization, Data curation, Formal analysis, Validation, Investigation, Methodology, Writing – original draft, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-5077-5697
  2. Bastijn JG van den Boom

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Data curation, Formal analysis, Validation, Investigation, Visualization
    Contributed equally with
    Rhiannon Robke
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-0853-3763
  3. Rhiannon Robke

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Data curation, Formal analysis, Investigation, Visualization
    Contributed equally with
    Bastijn JG van den Boom
    Competing interests
    No competing interests declared
  4. Felice Veen

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Data curation, Formal analysis, Investigation
    Competing interests
    No competing interests declared
  5. Lizz Fellinger

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Data curation, Investigation
    Competing interests
    No competing interests declared
  6. Wouter van Elzelingen

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Investigation, Methodology
    Competing interests
    No competing interests declared
  7. Tara Arbab

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Conceptualization, Visualization, Writing – original draft, Writing – review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-7294-7223
  8. Ingo Willuhn

    1. Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands
    2. Department of Psychiatry, Amsterdam UMC, University of Amsterdam, Amsterdam, Netherlands
    Contribution
    Conceptualization, Formal analysis, Supervision, Funding acquisition, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing
    For correspondence
    i.willuhn@nin.knaw.nl
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-6540-6894

Funding

European Research Council (ERC-2014-STG 638013)

  • Ingo Willuhn

Nederlandse Organisatie voor Wetenschappelijk Onderzoek (VIDI 864.14.010, 2015/06367/ALW)

  • Ingo Willuhn

Nederlandse Organisatie voor Wetenschappelijk Onderzoek (BRAINSCAPES 024.004.012, Gravitation program)

  • Ingo Willuhn

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

We thank Ralph Hamelink and Nicole Yee for their technical support, Matthijs Feenstra for his input on the manuscript, Lucia Economico for illustrations, and Linda Dekker for histology.

Ethics

All animal procedures were in accordance with the Dutch and European laws and approved by the Animal Experimentation Committee of the Royal Netherlands Academy of Arts and Sciences. CCD license numbers AVD801002015126 and AVD80100202014245.

Senior Editor

  1. Kate M Wassum, University of California, Los Angeles, United States

Reviewing Editor

  1. Mihaela D Iordanova, Concordia University, Canada

Reviewer

  1. Erik Oleson, University of Colorado Denver, United States

Publication history

  1. Preprint posted: January 17, 2021 (view preprint)
  2. Received: August 15, 2022
  3. Accepted: October 20, 2022
  4. Version of Record published: November 11, 2022 (version 1)

Copyright

© 2022, Goedhoop et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 1,379
    Page views
  • 201
    Downloads
  • 0
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Jessica N Goedhoop
  2. Bastijn JG van den Boom
  3. Rhiannon Robke
  4. Felice Veen
  5. Lizz Fellinger
  6. Wouter van Elzelingen
  7. Tara Arbab
  8. Ingo Willuhn
(2022)
Nucleus accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error
eLife 11:e82711.
https://doi.org/10.7554/eLife.82711
  1. Further reading

Further reading

    1. Neuroscience
    Andrew P Davison, Shailesh Appukuttan
    Insight

    Artificial neural networks could pave the way for efficiently simulating large-scale models of neuronal networks in the nervous system.

    1. Neuroscience
    Jonathan Nicholas, Nathaniel D Daw, Daphna Shohamy
    Research Article

    A key question in decision making is how humans arbitrate between competing learning and memory systems to maximize reward. We address this question by probing the balance between the effects, on choice, of incremental trial-and-error learning versus episodic memories of individual events. Although a rich literature has studied incremental learning in isolation, the role of episodic memory in decision making has only recently drawn focus, and little research disentangles their separate contributions. We hypothesized that the brain arbitrates rationally between these two systems, relying on each in circumstances to which it is most suited, as indicated by uncertainty. We tested this hypothesis by directly contrasting contributions of episodic and incremental influence to decisions, while manipulating the relative uncertainty of incremental learning using a well-established manipulation of reward volatility. Across two large, independent samples of young adults, participants traded these influences off rationally, depending more on episodic information when incremental summaries were more uncertain. These results support the proposal that the brain optimizes the balance between different forms of learning and memory according to their relative uncertainties and elucidate the circumstances under which episodic memory informs decisions.