There are many types of learning; one type of learning means that rewards and punishments can shape future behavior. Dopamine is a molecule that allows neurons in the brain to communicate with one another, and it is released in response to unexpected rewards. Most neuroscientists believe that dopamine is important to learn from the reward; however, there are different opinions about whether dopamine is important to learn from punishments or not.

Previous studies that tried to examine how dopamine activities change in response to punishment have reported different results. One of the likely reasons for the controversy is that it was difficult to measure only the activity of dopamine-releasing neurons.

To overcome this issue, Matsumoto et al. used genetically engineered mice in which shining a blue light into their brain would activate their dopamine neurons but not any other neurons. Tiny electrodes were inserted into the brains of these mice, and a blue light was used to confirm that these electrodes were recording from the dopamine-producing neurons. Specifically if the electrode detected an electrical impulse when blue light was beamed into the brain, then the recorded neuron was confirmed to be a dopamine-producing neuron.

Measuring the activities of these dopamine neurons revealed that they were indeed activated by reward but inhibited by punishment. In other words, dopamine neurons indeed can signal punishments as negative and rewards as positive on a single axis. Further experiments showed that, if the mice predicted both a reward and a punishment, the dopamine neurons could integrate information from both to direct learning.

Matsumoto et al. also saw that when mice received rewards too often, their dopamine neurons did not signal punishment correctly. These results suggest that how we feel about punishment may depend on how often we experience rewards.

In addition to learning, dopamine has also been linked to many psychiatric symptoms such as addiction and depression. The next challenge will be to examine how the frequency of rewards changes an animal’s state and responses to punishment in more detail, and how this relates to normal and abnormal behaviors.

Dopamine is thought to be a key regulator of learning from appetitive as well as aversive events (Schultz et al., 1997; Wenzel et al., 2015). It has been proposed that dopamine neurons act as a teaching signal in the brain by signaling the discrepancy between the values of actual and predicted rewards, that is, reward prediction error (RPE) (Bayer and Glimcher, 2005; Cohen et al., 2012; Hart et al., 2014; Roesch et al., 2007; Schultz, 2010; Schultz et al., 1997). Although accumulating evidence supports this idea with respect to rewarding and reward-predicting events (Bayer and Glimcher, 2005; Cohen et al., 2012; Eshel et al., 2015; Hart et al., 2014; Roesch et al., 2007; Schultz, 2010; Schultz et al., 1997), how dopamine neurons integrate information about aversive events remains highly controversial.

Pioneering work by Wolfram Schultz and colleagues introduced the idea that dopamine neurons signal RPE. This work demonstrated that dopamine neurons in the midbrain of monkeys exhibit a highly specific set of responses to reward (Mirenowicz and Schultz, 1994). When the animal receives reward unexpectedly, dopamine neurons fire a burst of action potentials. If a sensory cue reliably predicts reward, however, dopamine neurons decrease their response to reward, and instead burst to the cue. Finally, if an expected reward is omitted, dopamine neurons pause their firing at the time they usually receive reward (Hollerman and Schultz, 1998; Schultz et al., 1997). Subsequently, the idea of RPE coding by dopamine neurons has been substantiated by further experiments in a variety of species including monkeys (Bayer and Glimcher, 2005; Hollerman and Schultz, 1998; Waelti et al., 2001), rats (Flagel et al., 2011; Pan et al., 2005; Roesch et al., 2007), mice (Cohen et al., 2012; Eshel et al., 2015) and humans (D'Ardenne et al., 2008). This signal is proposed to underlie associative learning (Rescorla and Wagner, 1972), and bears a striking resemblance to machine learning algorithms (Sutton and Barto, 1998).

Many of the previous studies that characterized dopamine responses used rewarded outcomes with varying degrees of predictability. Comparatively fewer studies have used aversive stimuli in the context of prediction errors. Among studies that have used aversive stimuli, these provide differing reports as to how dopamine neurons respond to aversive stimuli (Fiorillo, 2013; Schultz, 2015; Wenzel et al., 2015).

It is thought that the majority of dopamine neurons are inhibited by aversive stimuli (Mileykovskiy and Morales, 2011; Mirenowicz and Schultz, 1996; Tan et al., 2012; Ungless et al., 2004). However, a number of electrophysiological recording studies have reported that dopamine neurons are activated by aversive stimuli both in anesthetized (Brischoux et al., 2009; Coizet et al., 2006; Schultz and Romo, 1987) and awake animals (Guarraci and Kapp, 1999; Joshua et al., 2008; Matsumoto and Hikosaka, 2009), although the proportions and locations of aversion-activated neurons differed among these studies. The differences in the results between these studies could be due to the heterogeneity of dopamine neurons or to differences in experimental conditions (e.g. type of aversive stimuli; type of anesthesia). Furthermore, another study using fast-scan cyclic voltammetry found that dopamine neurons are excited during successful avoidance of aversive stimuli (Oleson et al., 2012), which could be 'rewarding'. Therefore, some of the excitatory responses to aversive stimuli may not be due to aversiveness alone.

Some of these discrepancies could correspond to differences in dopamine signaling depending on the projection target. Roitman et al. (2008) monitored dopamine dynamics in the nucleus accumbens using cyclic voltammetry while the animal received intra-oral administrations of a sucrose or quinine solution (Roitman et al., 2008). This study found that these stimuli caused opposite responses: dopamine release was increased by sucrose and decreased by quinine (McCutcheon et al., 2012), suggesting that at least the majority of dopamine neurons projecting to the nucleus accumbens are inhibited by aversive stimuli. Matsumoto and Hikosaka (2009) examined the diversity of dopamine neurons in context of prediction error. They showed that dopamine neurons that are activated by the prediction of aversive stimuli are located in the lateral part of the substantia nigra pars compacta (SNc), supporting the notion that dopamine subpopulations are spatially segregated (Matsumoto and Hikosaka, 2009). Consistent with this finding, Lerner et al. (2015) showed, using calcium imaging with fiber photometry, that SNc neurons projecting to the dorsolateral striatum are activated by aversive stimuli (electric shock) whereas those projecting to the dorsomedial striatum are inhibited (Lerner et al., 2015). Lammel et al. (2011) provided further evidence for spatial heterogeneity by showing that dopamine neurons projecting to the medial prefrontal cortex, located in the medial ventral tegmental area (VTA) exhibited a form of synaptic plasticity (AMPA/NMDA ratio) in response to aversive stimuli (formalin injection) whereas dopamine neurons projecting to the dorsolateral striatum did not (Lammel et al., 2011) although how these neurons change their firing patterns in response to aversive stimuli remains unknown.

In contrast to the above findings suggesting that dopamine neurons are heterogeneous with respect to signaling aversive events, Schultz, Fiorillo and colleagues have argued that dopamine neurons largely ignore aversiveness (Fiorillo, 2013; Schultz, 2015; Stauffer et al., 2016). One argument is that the excitation of dopamine neurons caused by aversive stimuli may be due to a 'generalization' or 'spill-over' effect of rewarding stimuli. Specifically, Mirenowicz and Schultz (1996) showed that when rewarding and aversive stimuli are predicted by similar cues (e.g. in a same sensory modality), aversion-predicting cues increase their tendency to activate dopamine neurons ('generalization') (Mirenowicz and Schultz, 1996). Kobayashi and Schultz (2014) showed that in a high-reward context, cues that predict a neutral outcome (e.g. a salient picture) increased their tendency to activate dopamine neurons compared to the neutral cues in a low reward context (Kobayashi and Schultz, 2014). Based on these and other observations (Fiorillo et al., 2013; Nomoto et al., 2010), they proposed that the early response reflects attributes such as stimulus generalization and intensity, and the later response reflects the subjective reward value and utility (Schultz, 2016; Stauffer et al., 2016).

One influential paper by Fiorillo (2013) concluded that dopamine neurons represent prediction errors with respect to reward but not aversiveness (Fiorillo, 2013). That is, dopamine neurons ignore aversive events. Recording from non-human primates, Fiorillo used three pieces of evidence to support this claim: First, dopamine neurons’ responses to aversive outcomes (air puff) were indistinguishable from their responses to neutral outcomes. Second, although most dopamine neurons reduced their reward responses when the reward was predicted, their response to aversive events was unaffected by prediction. Third, dopamine neurons did not integrate the value of aversive events when combined with rewarding events. From these results, the author proposed that the brain represents reward and aversiveness independently along two dimensions (Fiorillo, 2013). As a result, the author proposed that different molecules regulate different types of reinforcement learning: dopamine for reward and a different molecule for aversiveness. If proven true, these ideas are fundamental in understanding how the brain learns from reward and aversion. However, it remains to be clarified whether these observations can be generalized.

The conclusions in many of the studies cited above relied upon indirect methods such as spike waveforms and firing properties (Ungless and Grace, 2012) in order to identify dopamine neurons. These identification methods differed among studies and have recently been called into question (Lammel et al., 2008; Margolis et al., 2006; Ungless and Grace, 2012). The ambiguity of cell-type identification criteria across studies makes it difficult to consolidate data on dopamine signaling. For example, Ungless et al. showed that some neurons in the VTA that were excited by aversive events and identified as dopaminergic using standard electrophysiological criteria were revealed not to be dopaminergic when they were examined with juxtacellular labeling (Ungless et al., 2004). Furthermore, Schultz has argued that some previous recording studies may not have targeted areas rich in dopamine neurons (Schultz, 2016).

To circumvent this problem, we tagged dopamine neurons with a light-gated cation channel, channelrhodopsin-2 (ChR2) and unambiguously identified dopamine neurons based on their responses to light (Cohen et al., 2012). In the present study, we monitored the activity of identified dopamine neurons using a series of behavioral tasks designed to determine how dopamine neurons encode prediction of aversive events in addition to reward. Our results demonstrate that, in contrast to the proposal by Fiorillo (2013), dopamine neurons in VTA indeed are able to encode complete VPE, integrating information about both appetitive and aversive events in a common currency. Importantly, the ability of dopamine neurons to encode VPE depends on both reward contexts and the animal’s trial-by-trial behavioral state.

There have been divergent and inconsistent results with respect to how dopamine neurons encode aversive events. In the present study, we aimed to specifically examine how dopamine neurons integrate aversiveness and reward by carefully controlling various experimental parameters. The following three points are particularly notable. First, we used an optogenetic tagging method to unambiguously identify dopamine neurons while recording spiking activity in behaving mice. Second, we used a series of behavioral paradigms with probabilistic outcomes that are designed to test specific hypotheses regarding the integration of different outcomes and the effect of reward contexts. Third, we monitored aversion-related behaviors (eye blinking) to examine the trial-by-trial relationship to dopamine responses. By harnessing these controlled experimental conditions, our results indicate that dopamine neurons have different modes of signaling. In low reward contexts (the mixed prediction task and the low reward probability task), dopamine neurons were inhibited by aversive events more strongly than by neutral events, tracking the aversiveness of stimuli on a trial-by-trial basis. Furthermore, dopamine neurons signaled VPEs by faithfully combining information about both appetitive and aversive events into a common currency of value. Thus, dopamine can function as a precise teaching signal for updating integrated values for both reward and aversion. However, in high reward contexts (the high reward probability task and the high reward probability task 2), dopamine neurons exhibited early excitations, which undermined their ability to produce negative responses for aversive events.

1. 1. Atasoy D
   2. Aponte Y
   3. Su HH
   4. Sternson SM

   (2008) A FLEX switch targets Channelrhodopsin-2 to multiple cell types for imaging and long-range circuit mapping

   Journal of Neuroscience 28:7025–7030.

   https://doi.org/10.1523/JNEUROSCI.1954-08.2008

   • Google Scholar
2. 1. Bäckman CM
   2. Malik N
   3. Zhang Y
   4. Shan L
   5. Grinberg A
   6. Hoffer BJ
   7. Westphal H
   8. Tomac AC

   (2006) Characterization of a mouse strain expressing Cre recombinase from the 3' untranslated region of the dopamine transporter locus

   Genesis 44:383–390.

   https://doi.org/10.1002/dvg.20228

   • Google Scholar
3. 1. Bayer HM
   2. Glimcher PW

   (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal

   Neuron 47:129–141.

   https://doi.org/10.1016/j.neuron.2005.05.020

   • Google Scholar
4. 1. Brischoux F
   2. Chakraborty S
   3. Brierley DI
   4. Ungless MA

   (2009) Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli

   PNAS 106:4894–4899.

   https://doi.org/10.1073/pnas.0811507106

   • Google Scholar
5. 1. Bromberg-Martin ES
   2. Matsumoto M
   3. Hikosaka O

   (2010) Dopamine in motivational control: rewarding, aversive, and alerting

   Neuron 68:815–834.

   https://doi.org/10.1016/j.neuron.2010.11.022

   • Google Scholar
6. 1. Chang CY
   2. Esber GR
   3. Marrero-Garcia Y
   4. Yau HJ
   5. Bonci A
   6. Schoenbaum G

   (2016) Brief optogenetic inhibition of dopamine neurons mimics endogenous negative reward prediction errors

   Nature Neuroscience 19:111–116.

   https://doi.org/10.1038/nn.4191

   • Google Scholar
7. 1. Cohen JY
   2. Haesler S
   3. Vong L
   4. Lowell BB
   5. Uchida N

   (2012) Neuron-type-specific signals for reward and punishment in the ventral tegmental area

   Nature 482:85–88.

   https://doi.org/10.1038/nature10754

   • Google Scholar
8. 1. Coizet V
   2. Dommett EJ
   3. Redgrave P
   4. Overton PG

   (2006) Nociceptive responses of midbrain dopaminergic neurones are modulated by the superior colliculus in the rat

   Neuroscience 139:1479–1493.

   https://doi.org/10.1016/j.neuroscience.2006.01.030

   • Google Scholar
9. 1. Cooper BR
   2. Howard JL
   3. Grant LD
   4. Smith RD
   5. Breese GR

   (1974) Alteration of avoidance and ingestive behavior after destruction of central catecholamine pathways with 6-hydroxydopamine

   Pharmacology Biochemistry and Behavior 2:639–649.

   https://doi.org/10.1016/0091-3057(74)90033-1

   • Google Scholar
10. 1. D'Ardenne K
    2. McClure SM
    3. Nystrom LE
    4. Cohen JD

    (2008) BOLD responses reflecting dopaminergic signals in the human ventral tegmental area

    Science 319:1264–1267.

    https://doi.org/10.1126/science.1150605

    • Google Scholar
11. 1. Danjo T
    2. Yoshimi K
    3. Funabiki K
    4. Yawata S
    5. Nakanishi S

    (2014) Aversive behavior induced by optogenetic inactivation of ventral tegmental area dopamine neurons is mediated by dopamine D2 receptors in the nucleus accumbens

    PNAS 111:6455–6460.

    https://doi.org/10.1073/pnas.1404323111

    • Google Scholar
12. 1. Eshel N
    2. Bukwich M
    3. Rao V
    4. Hemmelder V
    5. Tian J
    6. Uchida N

    (2015) Arithmetic and local circuitry underlying dopamine prediction errors

    Nature 525:243–246.

    https://doi.org/10.1038/nature14855

    • Google Scholar
13. 1. Eshel N
    2. Tian J
    3. Bukwich M
    4. Uchida N

    (2016) Dopamine neurons share common response function for reward prediction error

    Nature Neuroscience 19:479–486.

    https://doi.org/10.1038/nn.4239

    • Google Scholar
14. 1. Fiorillo CD

    (2013) Two dimensions of value: dopamine neurons represent reward but not aversiveness

    Science 341:546–549.

    https://doi.org/10.1126/science.1238699

    • Google Scholar
15. 1. Fiorillo CD
    2. Song MR
    3. Yun SR

    (2013) Multiphasic temporal dynamics in responses of midbrain dopamine neurons to appetitive and aversive stimuli

    Journal of Neuroscience 33:4710–4725.

    https://doi.org/10.1523/JNEUROSCI.3883-12.2013

    • Google Scholar
16. 1. Flagel SB
    2. Clark JJ
    3. Robinson TE
    4. Mayo L
    5. Czuj A
    6. Willuhn I
    7. Akers CA
    8. Clinton SM
    9. Phillips PE
    10. Akil H

    (2011) A selective role for dopamine in stimulus-reward learning

    Nature 469:53–57.

    https://doi.org/10.1038/nature09588

    • Google Scholar
17. Book

    1. Franklin KB
    2. Paxinos G

    (2008)

    The Mouse Brain in Stereotaxic Coordinates

    San Diego: Elsevier Academic Press.

    • Google Scholar
18. 1. Guarraci FA
    2. Kapp BS

    (1999) An electrophysiological characterization of ventral tegmental area dopaminergic neurons during differential pavlovian fear conditioning in the awake rabbit

    Behavioural Brain Research 99:169–179.

    https://doi.org/10.1016/S0166-4328(98)00102-8

    • Google Scholar
19. 1. Hart AS
    2. Rutledge RB
    3. Glimcher PW
    4. Phillips PE

    (2014) Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term

    Journal of Neuroscience 34:698–704.

    https://doi.org/10.1523/JNEUROSCI.2489-13.2014

    • Google Scholar
20. 1. Hollerman JR
    2. Schultz W

    (1998) Dopamine neurons report an error in the temporal prediction of reward during learning

    Nature Neuroscience 1:304–309.

    https://doi.org/10.1038/1124

    • Google Scholar
21. 1. Ilango A
    2. Kesner AJ
    3. Keller KL
    4. Stuber GD
    5. Bonci A
    6. Ikemoto S

    (2014) Similar roles of substantia nigra and ventral tegmental dopamine neurons in reward and aversion

    Journal of Neuroscience 34:817–822.

    https://doi.org/10.1523/JNEUROSCI.1703-13.2014

    • Google Scholar
22. 1. Joshua M
    2. Adler A
    3. Mitelman R
    4. Vaadia E
    5. Bergman H

    (2008) Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials

    Journal of Neuroscience 28:11673–11684.

    https://doi.org/10.1523/JNEUROSCI.3839-08.2008

    • Google Scholar
23. 1. Kim HF
    2. Ghazizadeh A
    3. Hikosaka O

    (2015) Dopamine neurons encoding long-term memory of object value for habitual behavior

    Cell 163:1165–1175.

    https://doi.org/10.1016/j.cell.2015.10.063

    • Google Scholar
24. 1. Kobayashi S
    2. Schultz W

    (2014) Reward contexts extend dopamine signals to unrewarded stimuli

    Current Biology 24:56–62.

    https://doi.org/10.1016/j.cub.2013.10.061

    • Google Scholar
25. 1. Kvitsiani D
    2. Ranade S
    3. Hangya B
    4. Taniguchi H
    5. Huang JZ
    6. Kepecs A

    (2013) Distinct behavioural and network correlates of two interneuron types in prefrontal cortex

    Nature 498:363–366.

    https://doi.org/10.1038/nature12176

    • Google Scholar
26. 1. Lammel S
    2. Hetzel A
    3. Häckel O
    4. Jones I
    5. Liss B
    6. Roeper J

    (2008) Unique properties of mesoprefrontal neurons within a dual mesocorticolimbic dopamine system

    Neuron 57:760–773.

    https://doi.org/10.1016/j.neuron.2008.01.022

    • Google Scholar
27. 1. Lammel S
    2. Ion DI
    3. Roeper J
    4. Malenka RC

    (2011) Projection-specific modulation of dopamine neuron synapses by aversive and rewarding stimuli

    Neuron 70:855–862.

    https://doi.org/10.1016/j.neuron.2011.03.025

    • Google Scholar
28. 1. Lammel S
    2. Lim BK
    3. Malenka RC

    (2014) Reward and aversion in a heterogeneous midbrain dopamine system

    Neuropharmacology 76:351–359.

    https://doi.org/10.1016/j.neuropharm.2013.03.019

    • Google Scholar
29. 1. Lerner TN
    2. Shilyansky C
    3. Davidson TJ
    4. Evans KE
    5. Beier KT
    6. Zalocusky KA
    7. Crow AK
    8. Malenka RC
    9. Luo L
    10. Tomer R
    11. Deisseroth K

    (2015) Intact-brain analyses reveal distinct information carried by SNc dopamine subcircuits

    Cell 162:635–647.

    https://doi.org/10.1016/j.cell.2015.07.014

    • Google Scholar
30. 1. Margolis EB
    2. Lock H
    3. Hjelmstad GO
    4. Fields HL

    (2006) The ventral tegmental area revisited: is there an electrophysiological marker for dopaminergic neurons?

    The Journal of Physiology 577:907–924.

    https://doi.org/10.1113/jphysiol.2006.117069

    • Google Scholar
31. 1. Matsumoto M
    2. Hikosaka O

    (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals

    Nature 459:837–841.

    https://doi.org/10.1038/nature08028

    • Google Scholar
32. 1. Matthews GA
    2. Nieh EH
    3. Vander Weele CM
    4. Halbert SA
    5. Pradhan RV
    6. Yosafat AS
    7. Glober GF
    8. Izadmehr EM
    9. Thomas RE
    10. Lacy GD
    11. Wildes CP
    12. Ungless MA
    13. Tye KM

    (2016) Dorsal raphe dopamine neurons represent the experience of social isolation

    Cell 164:617–631.

    https://doi.org/10.1016/j.cell.2015.12.040

    • Google Scholar
33. 1. McCutcheon JE
    2. Ebner SR
    3. Loriaux AL
    4. Roitman MF

    (2012) Encoding of aversion by dopamine and the nucleus accumbens

    Frontiers in Neuroscience 6:137.

    https://doi.org/10.3389/fnins.2012.00137

    • Google Scholar
34. 1. Menegas W
    2. Bergan JF
    3. Ogawa SK
    4. Isogai Y
    5. Umadevi Venkataraju K
    6. Osten P
    7. Uchida N
    8. Watabe-Uchida M

    (2015) Dopamine neurons projecting to the posterior striatum form an anatomically distinct subclass

    eLife 4:e10032.

    https://doi.org/10.7554/eLife.10032

    • Google Scholar
35. 1. Mileykovskiy B
    2. Morales M

    (2011) Duration of inhibition of ventral tegmental area dopamine neurons encodes a level of conditioned fear

    Journal of Neuroscience 31:7471–7476.

    https://doi.org/10.1523/JNEUROSCI.5731-10.2011

    • Google Scholar
36. 1. Mirenowicz J
    2. Schultz W

    (1994)

    Importance of unpredictability for reward responses in primate dopamine neurons

    Journal of Neurophysiology 72:1024–1027.

    • Google Scholar
37. 1. Mirenowicz J
    2. Schultz W

    (1996) Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli

    Nature 379:449–451.

    https://doi.org/10.1038/379449a0

    • Google Scholar
38. 1. Nomoto K
    2. Schultz W
    3. Watanabe T
    4. Sakagami M

    (2010) Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli

    Journal of Neuroscience 30:10692–10702.

    https://doi.org/10.1523/JNEUROSCI.4828-09.2010

    • Google Scholar
39. 1. Oleson EB
    2. Gentry RN
    3. Chioma VC
    4. Cheer JF

    (2012) Subsecond dopamine release in the nucleus accumbens predicts conditioned punishment and its successful avoidance

    Journal of Neuroscience 32:14804–14808.

    https://doi.org/10.1523/JNEUROSCI.3087-12.2012

    • Google Scholar
40. 1. Pan WX
    2. Schmidt R
    3. Wickens JR
    4. Hyland BI

    (2005) Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network

    Journal of Neuroscience 25:6235–6242.

    https://doi.org/10.1523/JNEUROSCI.1478-05.2005

    • Google Scholar
41. 1. Popescu AT
    2. Zhou MR
    3. Poo MM

    (2016) Phasic dopamine release in the medial prefrontal cortex enhances stimulus discrimination

    PNAS 113:E3169–3176.

    https://doi.org/10.1073/pnas.1606098113

    • Google Scholar
42. Book

    1. Rescorla RA
    2. Wagner AR

    (1972)

    A theory of pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement

    In: Black A. H, Prokasy W. F, editors. Classical Conditioning II: Current Research and Theory. New York: Appleton-Century-Crofts. pp. 64–99.

    • Google Scholar
43. 1. Roeper J

    (2013) Dissecting the diversity of midbrain dopamine neurons

    Trends in Neurosciences 36:336–342.

    https://doi.org/10.1016/j.tins.2013.03.003

    • Google Scholar
44. 1. Roesch MR
    2. Calu DJ
    3. Schoenbaum G

    (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

    Nature Neuroscience 10:1615–1624.

    https://doi.org/10.1038/nn2013

    • Google Scholar
45. 1. Roitman MF
    2. Wheeler RA
    3. Wightman RM
    4. Carelli RM

    (2008) Real-time chemical responses in the nucleus accumbens differentiate rewarding and aversive stimuli

    Nature Neuroscience 11:1376–1377.

    https://doi.org/10.1038/nn.2219

    • Google Scholar
46. 1. Schmitzer-Torbert N
    2. Redish AD

    (2004) Neuronal activity in the rodent dorsal striatum in sequential navigation: separation of spatial and reward responses on the multiple T task

    Journal of Neurophysiology 91:2259–2272.

    https://doi.org/10.1152/jn.00687.2003

    • Google Scholar
47. 1. Schultz W

    (2010) Dopamine signals for reward value and risk: basic and recent data

    Behavioral and Brain Functions 6:24.

    https://doi.org/10.1186/1744-9081-6-24

    • Google Scholar
48. 1. Schultz W

    (2015) Neuronal reward and decision signals: From theories to data

    Physiological Reviews 95:853–951.

    https://doi.org/10.1152/physrev.00023.2014

    • Google Scholar
49. 1. Schultz W

    (2016) Dopamine reward prediction-error signalling: a two-component response

    Nature Reviews Neuroscience 17:183–195.

    https://doi.org/10.1038/nrn.2015.26

    • Google Scholar
50. 1. Schultz W
    2. Romo R

    (1987)

    Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey

    Journal of Neurophysiology 57:201–217.

    • Google Scholar
51. 1. Schultz W
    2. Dayan P
    3. Montague PR

    (1997) A neural substrate of prediction and reward

    Science 275:1593–1599.

    https://doi.org/10.1126/science.275.5306.1593

    • Google Scholar
52. 1. Stauffer WR
    2. Lak A
    3. Kobayashi S
    4. Schultz W

    (2016) Components and characteristics of the dopamine reward utility signal

    Journal of Comparative Neurology 524:1699–1711.

    https://doi.org/10.1002/cne.23880

    • Google Scholar
53. 1. Steinberg EE
    2. Keiflin R
    3. Boivin JR
    4. Witten IB
    5. Deisseroth K
    6. Janak PH

    (2013) A causal link between prediction errors, dopamine neurons and learning

    Nature Neuroscience 16:966–973.

    https://doi.org/10.1038/nn.3413

    • Google Scholar
54. Book

    1. Sutton RS
    2. Barto AG

    (1998)

    Reinforcement Learning: An Introduction

    Cambridge Univ Press.

    • Google Scholar
55. 1. Tan KR
    2. Yvon C
    3. Turiault M
    4. Mirzabekov JJ
    5. Doehner J
    6. Labouèbe G
    7. Deisseroth K
    8. Tye KM
    9. Lüscher C

    (2012) GABA neurons of the VTA drive conditioned place aversion

    Neuron 73:1173–1183.

    https://doi.org/10.1016/j.neuron.2012.02.015

    • Google Scholar
56. 1. Tian J
    2. Uchida N

    (2015) Habenula lesions reveal that multiple mechanisms underlie dopamine prediction errors

    Neuron 87:1304–1316.

    https://doi.org/10.1016/j.neuron.2015.08.028

    • Google Scholar
57. 1. Tian J
    2. Huang R
    3. Cohen JY
    4. Osakada F
    5. Kobak D
    6. Machens CK
    7. Callaway EM
    8. Uchida N
    9. Watabe-Uchida M

    (2016) Distributed and mixed information in monosynaptic inputs to dopamine neurons

    Neuron 91:1–16.

    https://doi.org/10.1016/j.neuron.2016.08.018

    • Google Scholar
58. 1. Tsai HC
    2. Zhang F
    3. Adamantidis A
    4. Stuber GD
    5. Bonci A
    6. de Lecea L
    7. Deisseroth K

    (2009) Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning

    Science 324:1080–1084.

    https://doi.org/10.1126/science.1168878

    • Google Scholar
59. 1. Uchida N
    2. Mainen ZF

    (2003) Speed and accuracy of olfactory discrimination in the rat

    Nature Neuroscience 6:1224–1229.

    https://doi.org/10.1038/nn1142

    • Google Scholar
60. 1. Ungless MA
    2. Magill PJ
    3. Bolam JP

    (2004) Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli

    Science 303:2040–2042.

    https://doi.org/10.1126/science.1093360

    • Google Scholar
61. 1. Ungless MA
    2. Grace AA

    (2012) Are you or aren't you? Challenges associated with physiologically identifying dopamine neurons

    Trends in Neurosciences 35:422–430.

    https://doi.org/10.1016/j.tins.2012.02.003

    • Google Scholar
62. 1. van Zessen R
    2. Phillips JL
    3. Budygin EA
    4. Stuber GD

    (2012) Activation of VTA GABA neurons disrupts reward consumption

    Neuron 73:1184–1194.

    https://doi.org/10.1016/j.neuron.2012.02.016

    • Google Scholar
63. 1. Waelti P
    2. Dickinson A
    3. Schultz W

    (2001) Dopamine responses comply with basic assumptions of formal learning theory

    Nature 412:43–48.

    https://doi.org/10.1038/35083500

    • Google Scholar
64. 1. Wenzel JM
    2. Rauscher NA
    3. Cheer JF
    4. Oleson EB

    (2015) A role for phasic dopamine release within the nucleus accumbens in encoding aversion: a review of the neurochemical literature

    ACS Chemical Neuroscience 6:16–26.

    https://doi.org/10.1021/cn500255p

    • Google Scholar
65. 1. Witten IB
    2. Steinberg EE
    3. Lee SY
    4. Davidson TJ
    5. Zalocusky KA
    6. Brodsky M
    7. Yizhar O
    8. Cho SL
    9. Gong S
    10. Ramakrishnan C
    11. Stuber GD
    12. Tye KM
    13. Janak PH
    14. Deisseroth K

    (2011) Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement

    Neuron 72:721–733.

    https://doi.org/10.1016/j.neuron.2011.10.028

    • Google Scholar

Thank you for submitting your article "Midbrain dopamine neurons signal aversion in a reward-context-dependent manner" for consideration by eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Timothy Behrens as the Senior Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

The reviewers appreciated the new data evaluating the relationship between aversive and appetitive coding in dopamine neurons of the VTA in this study, and also the effort in doing these analyses with carefully identified dopamine. However, the overall opinion was that the authors have to more carefully analyze their existing data, and to ensure the data clearly support the conclusions they make. Below is a summary of the 3 main types of concern that should be addressed:

1) A general comment brought up by all reviewers is that the authors should analyze the phasic neuronal responses with better temporal specificity, i.e., not just averaging over one larger time bin. The reviewers felt that it is not justified to ignore temporal dynamics in the aversive CS response, and that the conclusions (for example regarding stimulus generalization) will be changed/tempered if this is done.

2) The reviewers felt that the relationship between the behavior and the neural activity was not being fully fleshed out, and so new analyses are important to substantiate some of the claims. Namely relationship between CS and US in different predicted and unpredicted conditions, meaning of control tasks, and analyses of anticipatory licking, etc.

3) Another concern was the small n for many conclusions and the need for some additional statistical analyses. Although new experiments may be beyond the scope of this review, some conclusions may be biased by a low n (no significance in some cases vs. significance in other cases). Also, ANOVA and multiple comparison correction may be needed in some cases.

Also below are the detailed points of each reviewer to help in your revised submission.

Reviewer #1:

Matsumoto and colleagues describe the responses of identified dopamine (DA) neurons recorded from the lateral VTA of mice during multiple variations of an odor discrimination task. The methodology, including the DA neuron identification and recording, is similar to prior work from this group, and reflects a careful and sophisticated approach towards investigating the responses of individual identified neurons to outcomes and the cues that predict them. The authors' aim is to investigate the encoding by DA neurons of aversive stimuli, given the mixed findings in the literature. The approach taken here is to compare the neuronal responses to cues in a discrimination procedure in which reward-predictive and airpuff-predictive cues, as well as cues predicting both, or nothing, are presented pseudorandomly. The authors find that the encoding of aversive-predicting cues varies with the relative probability of other cues being reinforced by reward, such that DA response to airpuff-paired cues are decreases in spiking in low density reward sessions, and DA responses to airpuff-paired cues are increases in spiking in high density reward sessions (although decreases are also clearly visible in the neural response – see below). These are interesting findings. The results do add substantially to our understanding of DA neural responses by providing careful parametric comparisons in, very critically, identified DA neurons, using opto-tagging. Importantly, the notion that overall reward density may alter responses to cues – reward paired, or not – was shown previously by Schultz and colleagues, and this is acknowledged by the authors. The present data add to our understanding of the regulation of DA neuron responses to reward- and punishment-paired cues and will be of strong interest to scientists studying dopamine systems, as well as more generally, reward, addiction, and learning. The paper would benefit from more careful consideration of the specific attributes and limitations of these studies in forming their interpretations.

In addition, an explanation of how the differing signals in response to the negative cue in high and low reward contexts might be expected to impact current or future behavior differentially is missing, given that the behavioral data presented indicate that anticipatory eye blinking is similar in both contexts – however their conclusion on this point may be a function of their analysis approach. Specifics follow.

1) The authors conclude that "In low reward contexts, dopamine neurons were inhibited by aversive events more strongly than by neutral events, tracking aversiveness of stimuli on a trial-by-trial basis […] However, in high reward contexts, dopamine neurons exhibited early excitations, which undermined their ability to produce negative responses for aversive events." While their data as currently analyzed support this statement, it begs the question of what these different "modes" mean for learning and behavior since they presumably would communicate different information to downstream structures. The authors show that mice tend to show anticipatory blinking for airpuff in response to the cue whether it is a low or high reward context, but the relationship between the suppression of firing in response to the airpuff-predictive cue only holds for the low reward context, because consistent suppression in the DA neuron response is not seen in response to this cue in the high reward context. Yet this does not take the biphasic nature of the neural response to the aversive stimulus in the high reward context into account. In light of this (as well as the comment that follows), the pupillary/blink response appears to occur on a timescale parallel to that of the late component of the neural response, which in the case of the aversive cue, is an inhibition. Thus, it appears to me that there may indeed by a correlation with behavior and the neural signal. Can the authors investigate this issue more deeply?

2) The authors further state that: "[…]dopamine neurons signaled VPEs by faithfully combining information about both appetitive and aversive events into a common currency of value." Their evidence for this is that the average firing rate of DA neurons following a cue that predicts either an airpuff or reward is between that of a reward-predicting and an airpuff-predicting cue – however, this conclusion may depend overmuch on the averaging of the signal over the first second of cue presentation. Examination of the population histograms clearly show that this signal is dynamic, and that averaging the response over the full first second of cue presentation may obscure a multiphasic response to the aversive cues – there looks to be an excitation followed by an inhibition in the high reward context. Again, this is a response pattern described by Schultz and colleagues, who suggests that the early response reflects attributes such as stimulus generalization and intensity, and it is the later response that reflects the prediction error/value signal. This interpretation should receive deeper treatment in the current manuscript. Maybe the averaging across the full second is the most direct statistical approach, but the biphasic nature of the signal should be acknowledged.

3) It has been proposed that excitatory responses to cues that do not predict reward in multi-cue procedures (wherein some cues do predict reward) reflect stimulus generalization, among other factors. This seems to be one possible explanation for the finding that increases in response to the airpuff cue are more often seen in a high reward context than a low reward context. In addition, the authors data further support this notion since the cue that predicts no reward also shows an initial excitation in the high reward, but not low reward, context. Have the authors thought to use cues of differing modalities to reduce the sensory similarity? If not, the authors should consider that their data do not contradict a generalization and/or salience notion, although they appear to say they do.

4) The authors conclude that "dopamine neurons do not encode aversion in a conventional classical conditioning task". Given the clear evidence provided by the authors that reward context impacts the response of VTA dopamine neurons provided by the authors, the authors have not provided an appropriate test of this hypothesis. To conclude this, the authors would need to conduct their recordings in a procedure without interleaved rewarded trials.

5) I am curious what the potential differences in the responses to the conditioned stimuli and the unconditioned stimuli for a given neuron might be. Indeed, it would be useful if the authors reported the extent to which the encoding observed during the cue is similar to encoding during the outcome, i.e., if a neuron is inhibited by an airpuff-predicting cue, is it also inhibited by unexpected air puff. This is especially interesting given the points above, that the high reward context increase in firing at the outset of the odor cues seems parsimoniously explained as generalization, while the neural responses to the respective outcomes could still be increases and decreases for sucrose and airpuffs, respectively. In other words, do DA neuron responses to presentation of airpuff itself change in different reward contexts, or is this strictly a feature of the conditioned stimuli.

Reviewer #2:

In this manuscript, Matsumoto et al. investigated the response of midbrain dopaminergic neurons to an aversive stimulus in a variety of contexts. Although dopamine is thought to be important for learning from both reward and punishment, there has been disagreement about how dopaminergic neurons integrate aversive and rewarding stimuli. The authors show that dopaminergic activity is inhibited by an air puff and an air puff-predicting odor in low reward contexts. The air puff response is modulated by expectation and individual dopaminergic neurons are able to integrate the value of reward and punishment. Finally, they show that manipulating reward context affects dopaminergic responses to punishment, such that in high reward contexts there is actually an excitatory response to punishments. The latter condition appears to replicate various previous studies. The data provides a valuable contribution to the debate over how dopaminergic neurons respond to aversive stimuli. The use of optogenetic tagging allows for compelling identification of dopaminergic neurons and the authors tested several task structures in order to reconcile discrepancies in the field.

There are several ways in which the analysis and presentation should be improved:

1) In Figure 2 the authors conclude that in a high reward context, dopaminergic neurons respond to aversive stimuli with short-latency excitation "masked inhibitory responses" such that there is no longer a significant difference between responses to odor B and odor C. The importance of reward context is a major conclusion of the paper but its complete disruption of aversive signaling is confusing (and not entirely convincing) as presented. Why not analyze the different temporal phases of the response separately? It appears that if the responses were compared during the late inhibitory phase of the response (evident in Figure 2B), then there would indeed be a difference in the response to odor B and odor C in both the high reward and low reward contexts. In other words, the simplest explanation of the effect of "high reward context" is to create a short-latency excitatory response to the aversive stimulus. The authors argue this response is different from the short-latency 'generalization' response described by Schultz and colleagues, but not clear what that means. Specifically, in the Discussion it is noted that a high reward context "compromised the monotonic value coding of dopamine neurons," but this was not clearly explained or analyzed. Additionally, the argument that the excitatory response to punishment in a high reward context is not due to generalization is unclear. The authors write that "dopamine neurons did not simply add the same amount of excitation on top of the monotonic value coding." In fact, in Figure 2, it seems like there is an excitatory "generalization" response that is being added to the pause in firing to the CS-.

2) Although there is a clear increase in the amplitude of the response to air puff when it is unpredicted, there is large variability in the response to an omitted air puff. Although on average there is a very small increase in firing rate, a very large number of neurons decrease their activity in response to air puff omission (Figure 3F). The authors should discuss the heterogeneity of this response, which appears to complicate their conclusions.

3) It seems surprising that there is no correlation between the response to unpredicted air puff and unpredicted water delivery (Figure 5C). In Eshel et al. (2016) there is a correlation between predicted air puff and unpredicted water responses as well as between the air puff cue and the unpredicted water responses. So why wouldn't there be a correlation between unpredicted air puff and reward? Doesn't this mean that predicted and unpredicted air puff responses are themselves uncorrelated, which would contradict the earlier panels in Figure 5?

4) The protocol for the classical conditioning experiment conducted for Figure 2—figure supplement 1 is unclear. It appears that the only difference from the high reward task is the probability of punishment following odor C which is 80% instead of 90%, yet the excitatory component of the response to the punishment-predicting CS appears considerably larger than in Figure 2.

5) Not clear exactly what the eye blink response analysis achieves in the paper, nor why eye blinks were analyzed but not anticipatory licking.

6) There is no reported method for the correction for multiple comparisons in multiple instances when more than 2 groups are being analyzed in the same panel. Additionally, a two-way ANOVA should be used to analyze the effect of reward probability and anticipatory blinking in Figure 4F.

Reviewer #3:

In this manuscript Matsumoto and colleagues present a series of experiments that address the issue of how dopamine neurons respond to aversive events. They make a number of important and interesting observations. On the whole I thought the manuscript was well-written. I have a few concerns that I list below:

1) I feel that the Introduction would benefit from some elaboration in a few places.

Introduction, fourth paragraph: should make it clear that Oleson et al. (2012) measured dopamine release using FSCV.

Introduction, fifth paragraph: I am not clear how Roitman et al. (2008) resolved the issue and this was published several years earlier, so the wording is confusing in my view.

Introduction, fifth paragraph: should make it clear that Lerner et al. (2015) measured calcium activity, not action potential firing.

Introduction, last paragraph: this paragraph alludes to the issue of neurochemical identity in a way that I think should be more explicit since it is of particular relevance to this study. For example, Ungless et al. (2004) showed that some VTA neurons that are excited by aversive events are in fact not dopaminergic. It is noted, for example by Schultz (2016) in a recent review, that Matsumoto & Hikosaka (2009) recorded many neurons in areas where dopamine neurons were rare. This more strongly justifies the use of the opto-tagging approach that the authors have taken.

2) One general concern I had was the low numbers of mice in several of the experimental groups (e.g., Figure 1, n=3).

3) How do the authors know that the mice can tell the difference between the low and high reward situations?

4) In the subsection “Integration of aversiveness and reward in dopamine neurons”, first paragraph: add the caveat that activation of meso-cortical dopamine neurons and dorsal raphe dopamine neurons can induce conditioned-place aversion (Lammel et al. 2014 & Matthews et al. 2016).

4) In the subsection “Diversity of dopamine neurons”: note also that Brischoux et al. (2009) reported that ventromedial dopamine neurons exhibited aversive activations.

5) I found some of the red/purple colour schemes difficult to tell apart. I would suggest some different colour schemes (particularly for Figure 1).

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Midbrain dopamine neurons signal aversion in a reward-context-dependent manner" for further consideration at eLife. Your revised article has been favorably evaluated by Timothy Behrens (Senior editor), a Reviewing editor, and two reviewers.

The manuscript has been improved but there are some remaining issues regarding its readability that need to be addressed before acceptance, as outlined below by Reviewer 2. After consultation between the reviewers, both thought that increased readability and explanation of why each experiment was performed would greatly enhance the clarity and impact of the study.

Reviewer #2:

I felt that the reviewers were responsive to our comments and that this study provides a valuable contribution to the discussion regarding the integration of rewarding and aversive cues in dopamine neurons.

My only remaining comment is that I find the text in the Results and Discussion a bit difficult to follow. There are a number of experiments/analyses, and as it currently stands, the motivation for each experiment/analysis, as well as the relevant conclusions, is not always completely transparent. Therefore, the authors could consider editing the Results/Discussion for improved readability.

Reviewer #3:

This new version addresses the previous concerns that I raised.

The reviewers appreciated the new data evaluating the relationship between aversive and appetitive coding in dopamine neurons of the VTA in this study, and also the effort in doing these analyses with carefully identified dopamine. However, the overall opinion was that the authors have to more carefully analyze their existing data, and to ensure the data clearly support the conclusions they make. Below is a summary of the 3 main types of concern that should be addressed:

We appreciate all of the reviewers for their constructive criticism and suggestions. We have performed additional analyses and modified our manuscript according to reviewer’s comments. We hope that our manuscript has improved.

1) A general comment brought up by all reviewers is that the authors should analyze the phasic neuronal responses with better temporal specificity, i.e., not just averaging over one larger time bin. The reviewers felt that it is not justified to ignore temporal dynamics in the aversive CS response, and that the conclusions (for example regarding stimulus generalization) will be changed/tempered if this is done.

We appreciate reviewers’ criticism on the choice of time windows. We are aware that, in previous studies, it has become a common practice to pick a time window somewhat arbitrarily, especially for the analysis of dopamine activities. Although these studies have shown that dopamine neurons often show multi-phasic responses and different information appears to be conveyed in different time windows, no study has shown that postsynaptic neurons are equipped with a special mechanism to separately read out information from these windows. We therefore chose to use a relatively large time window (0–1,000 ms) that covers the entire response period as this would be a more conservative approach, and used this method throughout the manuscript. This rationale is now explained in Results (subsection “Reward-context dependent representation of aversion in dopamine neurons”, fifth paragraph).

The above analyses used a relatively large time window that contains the entire response period (0–1,000 ms). Because dopamine responses in high reward contexts exhibited biphasic responses (early excitation followed by later inhibition), we further analyzed the data by separating these time windows into smaller bins. Because there is no known mechanism by which downstream neurons can read out these windows separately, analysis using a large window can be considered more conservative. However, previous studies have indicated that different information may be conveyed in these time windows (Schultz, 2016; Stauffer et al., 2016).

However, we agree that it is important to know what information is conveyed by different time windows, as Schultz recently proposed (Schultz, 2016, Nat Rev Neurosci). We therefore performed new analyses using the time window in which dopamine neurons are predominantly inhibited, i.e. the later response phase (200–1,000 ms). Analyses using the small time window generally confirmed that there is a similar trend. For instance, fewer neurons showed a significant difference between nothing-predictive- and air puff-predictive CSs in a high reward context. These results are now shown in Figure 4—figure supplement 2, and discussed in Results as follows:

“We obtained similar results even if we compared only later time bins (200–1,000 ms), excluding the early excitation phase (Figure 4—figure supplement 2). […] Neurons that showed a large excitation to the air puff CS were not necessarily the same group of neurons which showed excitation to the air puff itself, consistent with a previous study (Matsumoto and Hikosaka, 2009) (Figure 4—figure supplement 3).”

We have also performed analyses on the relation between trial-to-trial variability of dopamine activities and anticipatory blinking using the later time window (200–1,000 ms). This analysis is now shown in Figure 5—figure supplement 3 and discussed in Results as follows:

“The correlation between trial-by-trial dopamine activity and anticipatory blinking was even clearer if we consider reward contexts (Figure 5E). […] The results do not change even when we only used a later window of dopamine CS responses, excluding the early excitation period (0–200 ms) (Figure 5—figure supplement 3).”

2) The reviewers felt that the relationship between the behavior and the neural activity was not being fully fleshed out, and so new analyses are important to substantiate some of the claims. Namely relationship between CS and US in different predicted and unpredicted conditions, meaning of control tasks, and analyses of anticipatory licking, etc.

We found that both reward context and behavioral outcome (blink) were important for dopamine neurons to show inhibition to mildly aversive cues but one was not sufficient. We rephrased this part both in Results and Discussion. We also added a new analysis in Figure 4—figure supplement 3 for the analysis of the relationship between CS and US, and Figure 5F and Figure 5—figure supplement 4 for the analysis of the relationship between anticipatory licking behavior and responses to reward CS. For more details of each analysis, please see our Response to each reviewer’s comment.

3) Another concern was the small n for many conclusions and the need for some additional statistical analyses. Although new experiments may be beyond the scope of this review, some conclusions may be biased by a low n (no significance in some cases vs. significance in other cases). Also, ANOVA and multiple comparison correction may be needed in some cases.

In this study, we used 14 mice for electrophysiological experiments, recorded from 176 neurons and identified 72 dopamine neurons in total. We agree that some of the task conditions contain relatively fewer number of neurons. However, for most of the analyses, we replicated the same results using two separate sets of experiments. To demonstrate the robustness of our basic findings, now we show data analyses using combined data in Figure 4—figure supplement 1G,J,K and Las well as side-by-side comparisons of comparable task conditions (Figure 4—figure supplement 1E,F,H,I).

We performed one-way ANOVA for comparing the means of CS responses (1: reward-, nothing-, versus air puff-predicting CSs and 2: reward-, reward + air puff-, versus air puff-predicting CSs). We described the results in Results (subsection “Dopamine neurons integrate values of both valences, appetitive and aversive”, last two paragraphs). Also, Bonferroni correction was applied for significance tests with multiple comparisons. We added this sentence in Materials and methods (subsection “Data analysis”, last paragraph).

Also below are the detailed points of each reviewer to help in your revised submission.

Reviewer #1:

1) The authors conclude that "In low reward contexts, dopamine neurons were inhibited by aversive events more strongly than by neutral events, tracking aversiveness of stimuli on a trial-by-trial basis […] However, in high reward contexts, dopamine neurons exhibited early excitations, which undermined their ability to produce negative responses for aversive events." While their data as currently analyzed support this statement, it begs the question of what these different "modes" mean for learning and behavior since they presumably would communicate different information to downstream structures. The authors show that mice tend to show anticipatory blinking for airpuff in response to the cue whether it is a low or high reward context, but the relationship between the suppression of firing in response to the airpuff-predictive cue only holds for the low reward context, because consistent suppression in the DA neuron response is not seen in response to this cue in the high reward context. Yet this does not take the biphasic nature of the neural response to the aversive stimulus in the high reward context into account. In light of this (as well as the comment that follows), the pupillary/blink response appears to occur on a timescale parallel to that of the late component of the neural response, which in the case of the aversive cue, is an inhibition. Thus, it appears to me that there may indeed by a correlation with behavior and the neural signal. Can the authors investigate this issue more deeply?

Please also refer to our response to the editor’s point #1 regarding specific time windows. The question of “what these different modes mean for learning and behavior” is important. One possibility would be that dopamine directly controls or modifies some behaviors. In our task, however, temporal dynamics of dopamine activities and blinking are different: dopamine responses are tightly locked to salient events such as CS or US, while blinking occurs much later than dopamine responses. This appears to suggest that dopamine activities are not tightly time-locked with blinking behaviors. The suggested idea is that certain time window of inhibitory activities might cause blinking. We did not see this correlation between dopamine activities and blinking when we used a big time window. Now we have analyzed our data using only the later phase of responses in high reward contexts, and show that the results are very similar to those obtained using a big time window. We have now added this analysis in Figure 5—figure supplement 3.

We interpret these results to support the idea that the dopamine signal (value coding) is an abstract value representation rather than immediate trigger of actions. Blinking is a good indicator for mice predicting air puff but may not be perfectly correlated with the negative value of the air puff. Dopamine neurons may signal negative value but not action itself. As a result, depending on contexts, prediction of air puff may be correlated with dopamine activity (in low reward context) or not (in high reward context). We rephrased these parts for clarification in Results (e.g. subsection “Trial-to-trial variability dopamine responses to aversive stimuli”, third paragraph) and Discussion.

“The correlation between trial-by-trial dopamine activity and anticipatory blinking was even clearer if we consider reward contexts (Figure 5E). [,,,] The results do not change even when we only used a later window of dopamine CS responses, excluding the early excitation period (0–200 ms) (Figure 5—figure supplement 3).”

2) The authors further state that: "[…]dopamine neurons signaled VPEs by faithfully combining information about both appetitive and aversive events into a common currency of value." Their evidence for this is that the average firing rate of DA neurons following a cue that predicts either an airpuff or reward is between that of a reward-predicting and an airpuff-predicting cue – however, this conclusion may depend overmuch on the averaging of the signal over the first second of cue presentation. Examination of the population histograms clearly show that this signal is dynamic, and that averaging the response over the full first second of cue presentation may obscure a multiphasic response to the aversive cues – there looks to be an excitation followed by an inhibition in the high reward context. Again, this is a response pattern described by Schultz and colleagues, who suggests that the early response reflects attributes such as stimulus generalization and intensity, and it is the later response that reflects the prediction error/value signal. This interpretation should receive deeper treatment in the current manuscript. Maybe the averaging across the full second is the most direct statistical approach, but the biphasic nature of the signal should be acknowledged.

Please also refer to the answer to the editor’s point #1. As the referee points out, it is important that previous studies have indicated that different information may be conveyed at different time windows. We now discuss the difference between the early and late phases of responses observed in the Schultz’s and Fiorillo’s studies in Introduction (sixth paragraph) and other places (e.g., in Results; subsection “Reward-context dependent representation of aversion in dopamine neurons”, second and fifth paragraphs).

As explained in our response to the editor’s point #1, we focused on a big time window because it remains unclear whether downstream neurons can read out information from specific, somewhat arbitrary response windows. In addition, our data indicates that defining a value- or prediction-error-specific time window is not very easy.

In our tasks, we did not observe a time window which solely signaled generalization and/or stimulus intensity regardless of the stimulus identity. For instance, Figure 1B shows that CS responses diverse in as quickly as 60 ms reflecting the predicted values of upcoming USs suggesting that our dopamine neurons can change their responses in a value dependent manner even in the early phase of the responses. The discrimination of odor identity in dopamine neurons in our task is fast, different from the previous tasks which used ambiguous cue on purpose (Nomoto et al., 2010, J Neurosci), and temporally overlaps with the early excitation during biphasic responses in high reward contexts.

Despite these difficulties in defining a time window, we have now performed analyses using only the late phase of the responses (200–1,000 ms). We did not observe value coding in this time window in response to air puff CSs in the high reward context, consistent with Fiorillo’s study (Fiorillo, 2013, Science). We added this result in Figure 4—figure supplement 2B. As for our mixed prediction tasks in Figure 1, if we split into two phases, the first phase includes many positive values of reward cues and the second phase includes many negative values of aversive cues, and so these patterns do not correspond to generalization and/or stimulus intensity and value, respectively. We think that this temporal shift of response peaks reflects the timing of responses encoding reward vs. aversion values with aversive information arriving slightly later, rather than the idea that the early period encodes generalization/intensity and the later period encodes values. We speculate that these shifted responses reflect the different sources of inputs signaling reward or aversion to dopamine neurons, which were not integrated yet in inputs. We have added these points in Discussion (page 17, lines 483-503).

“In examining the temporal dynamics of dopamine responses, we realized that on average, the peak of excitation for reward cues occurred earlier than the trough of inhibition for aversive cues. […] The results in the previous study (Eshel et al., 2016) could be consistent with the present results if the inhibition represents “no reward” but not the negative value of aversive stimulus, and this no-reward response is correlated with other reward-related responses of dopamine neurons.”

Because of the two limitations discussed above, we still prefer to report our main results, especially those pertaining to Figure 1, using a large time window.

3) It has been proposed that excitatory responses to cues that do not predict reward in multi-cue procedures (wherein some cues do predict reward) reflect stimulus generalization, among other factors. This seems to be one possible explanation for the finding that increases in response to the airpuff cue are more often seen in a high reward context than a low reward context. In addition, the authors data further support this notion since the cue that predicts no reward also shows an initial excitation in the high reward, but not low reward, context. Have the authors thought to use cues of differing modalities to reduce the sensory similarity? If not, the authors should consider that their data do not contradict a generalization and/or salience notion, although they appear to say they do.

Thank you very much for the comment and suggestion. We think that our data do not contradict with previous important findings about stimulus generalization etc. but rather expanded those findings. We tested how much these components disturb value coding of dopamine neurons. In addition to stimulus generalization, there are many other interpretations about the cause of early excitations. In each experiment, authors specifically designed to extract each factor from others. We think that the excitation that we observed in our tasks is caused by a combination of these factors, not only by stimulus generalization. The proposed experiment is an important experiment to understand which component is most affected by reward contexts but does not affect our statement that these components disturb value coding in dopamine neurons in high reward contexts. We revised our discussion on this point in the Discussion to be clearer (e.g. page 18, lines 507-524).

4) The authors conclude that "dopamine neurons do not encode aversion in a conventional classical conditioning task". Given the clear evidence provided by the authors that reward context impacts the response of VTA dopamine neurons provided by the authors, the authors have not provided an appropriate test of this hypothesis. To conclude this, the authors would need to conduct their recordings in a procedure without interleaved rewarded trials.

We apologize that this figure title was very misleading. We fixed this title. Thank you for letting us know.

5) I am curious what the potential differences in the responses to the conditioned stimuli and the unconditioned stimuli for a given neuron might be. Indeed, it would be useful if the authors reported the extent to which the encoding observed during the cue is similar to encoding during the outcome, i.e., if a neuron is inhibited by an airpuff-predicting cue, is it also inhibited by unexpected air puff. This is especially interesting given the points above, that the high reward context increase in firing at the outset of the odor cues seems parsimoniously explained as generalization, while the neural responses to the respective outcomes could still be increases and decreases for sucrose and airpuffs, respectively. In other words, do DA neuron responses to presentation of airpuff itself change in different reward contexts, or is this strictly a feature of the conditioned stimuli.

Thank you for the helpful comment. We observed similar phenomena in the responses to the unconditioned stimulus, i.e., air puff itself. In the low reward context, the response to air puff was mostly negative whereas the response was variable in the high reward context. Although we see a significant difference between responses to unpredicted air puff in high- and low-reward contexts using the neurons in the current data set (p = 0.001, n = 34 and n = 38, respectively, unpaired t-test), we would like to examine these phenomena more carefully because individual variability is big across neurons.

At the single neuron level, neurons which showed a large excitation to air puff CS were not necessarily the same group of neurons which showed excitation to air puff itself, consistent with a previous study (Matsumoto and Hikosaka, 2009, Nature). We now report this result in Figure 4—figure supplement 3 and in Results (page 12, lines 348-351).

“Neurons that showed a large excitation to the air puff CS were not necessarily the same group of neurons which showed excitation to the air puff itself, consistent with a previous study (Matsumoto and Hikosaka, 2009) (Figure 4—figure supplement 3).”

Reviewer #2:

In this manuscript, Matsumoto et al. investigated the response of midbrain dopaminergic neurons to an aversive stimulus in a variety of contexts. Although dopamine is thought to be important for learning from both reward and punishment, there has been disagreement about how dopaminergic neurons integrate aversive and rewarding stimuli. The authors show that dopaminergic activity is inhibited by an air puff and an air puff-predicting odor in low reward contexts. The air puff response is modulated by expectation and individual dopaminergic neurons are able to integrate the value of reward and punishment. Finally, they show that manipulating reward context affects dopaminergic responses to punishment, such that in high reward contexts there is actually an excitatory response to punishments. The latter condition appears to replicate various previous studies. The data provides a valuable contribution to the debate over how dopaminergic neurons respond to aversive stimuli. The use of optogenetic tagging allows for compelling identification of dopaminergic neurons and the authors tested several task structures in order to reconcile discrepancies in the field.

There are several ways in which the analysis and presentation should be improved:

1) In Figure 2 the authors conclude that in a high reward context, dopaminergic neurons respond to aversive stimuli with short-latency excitation "masked inhibitory responses" such that there is no longer a significant difference between responses to odor B and odor C. The importance of reward context is a major conclusion of the paper but its complete disruption of aversive signaling is confusing (and not entirely convincing) as presented. Why not analyze the different temporal phases of the response separately? It appears that if the responses were compared during the late inhibitory phase of the response (evident in Figure 2B), then there would indeed be a difference in the response to odor B and odor C in both the high reward and low reward contexts. In other words, the simplest explanation of the effect of "high reward context" is to create a short-latency excitatory response to the aversive stimulus. The authors argue this response is different from the short-latency 'generalization' response described by Schultz and colleagues, but not clear what that means. Specifically, in the Discussion it is noted that a high reward context "compromised the monotonic value coding of dopamine neurons," but this was not clearly explained or analyzed. Additionally, the argument that the excitatory response to punishment in a high reward context is not due to generalization is unclear. The authors write that "dopamine neurons did not simply add the same amount of excitation on top of the monotonic value coding." In fact, in Figure 2, it seems like there is an excitatory "generalization" response that is being added to the pause in firing to the CS-.

Thank you for the criticism and suggestions. We now added these analyses in Figure 4—figure supplement 2. Please also see our responses to the editor’s point #1 and reviewer 1’s point #2 and #3.

2) Although there is a clear increase in the amplitude of the response to air puff when it is unpredicted, there is large variability in the response to an omitted air puff. Although on average there is a very small increase in firing rate, a very large number of neurons decrease their activity in response to air puff omission (Figure 3F). The authors should discuss the heterogeneity of this response, which appears to complicate their conclusions.

We agree that air puff omission responses are very small and variable. Because we only mentioned the smallness, we now mentioned the variability in Results.

“To further examine whether dopamine neurons showed prediction error coding for aversive events, we compared the firing rate during the outcome period in air puff omission trials with that in trials that predict nothing. We found that the omission of a predicted air puff slightly but significantly increased firing rates, compared to no change in nothing trials (Figure 2E–H) although we observed variability in air puff omission responses.”

The air puff omission responses are correlated with air puff prediction and we did not observe clear clusters using these factors. We now added this analysis in Figure 3—figure supplement 1.

3) It seems surprising that there is no correlation between the response to unpredicted air puff and unpredicted water delivery (Figure 5C). In Eshel et al. (2016) there is a correlation between predicted air puff and unpredicted water responses as well as between the air puff cue and the unpredicted water responses. So why wouldn't there be a correlation between unpredicted air puff and reward? Doesn't this mean that predicted and unpredicted air puff responses are themselves uncorrelated, which would contradict the earlier panels in Figure 5?

As we found in the present study, the negative value of aversive stimulus is well represented in low reward contexts and the responses to aversiveness-related events are correlated with one another in these conditions. Importantly, the previous study (Eshel et al., 2016, Nat Neurosci) was conducted using a high reward context. A previous study found that the inhibition phase after excitation in biphasic responses is not related to aversiveness but to “no reward” (Fiorillo, 2013, Science). Consistent with this idea, we observe that in high reward contexts, cues predicting nothing often caused inhibition in dopamine neurons instead of no responses. In the previous study (Eshel et al., 2016, Nat Neurosci), the authors used cues which signaled either water or air puff (no cue signaling nothing). In this sense, the cue signaling air puff actually signals “no reward”. We interpret that the correlation which authors observed in responses between unpredicted water and air puff CS was mainly due to this “no reward” responses. Further, as the paper stated, the correlation was weak and not statistically significant.

The correlation between predicted air puff and unpredicted water in the previous study (Eshel et al., 2016, Nat Neurosci) was also complicated due to similar reasons. We examined author’s data more carefully and found that the correlation came from rebound after inhibition, but not inhibition itself; as Fiorillo reported (Fiorillo et al., 2013a, J Neurosci; Fiorillo et al., 2013b, J Neurosci), dopamine neurons actually often showed three phases (excitation, inhibition and excitation). We observed high correlation between this rebound and water responses but not between inhibition and water responses in high reward contexts.

Our present study is improved to more directly test the relation between rewarding and aversive value representations in dopamine neurons. We observed a correlation both within aversiveness-related representation and within reward related-representation but not between these. We now added this statement more clearly in Results (subsection “Homogeneous response function of dopamine neurons”, third paragraph) and Discussion (subsection “Integration of aversiveness and reward in dopamine neurons”, last paragraph).

4) The protocol for the classical conditioning experiment conducted for Figure 2—figure supplement 1 is unclear. It appears that the only difference from the high reward task is the probability of punishment following odor C which is 80% instead of 90%, yet the excitatory component of the response to the punishment-predicting CS appears considerably larger than in Figure 2.

Thank you for pointing this out. Yes, this is a very similar task except that the reward probability was slightly higher and the training procedure was slightly different. In this task (high reward probability task 2), we used 6.5% unpredicted reward delivery in comparison with 6% unpredicted water delivery in the high reward probability task. In addition, in the task 2, the odor predicting 90% water during experiments was same with the odor predicting 100% water during the initial training (a few days). It might be possible that animals might predict water with slightly higher probability than 90% during recording in the task 2 (i.e. higher reward context in the task 2). We now added a detailed task description in Materials and methods (subsection “Behavioral task”, second and last paragraphs). We did not see a significant difference in air puff CS responses between these two tasks (p = 0.19, unpaired t-test, n = 17 versus 17). We agree that this was confusing and we now added the pooled data for responses to air puff- and nothing-predicting CSs in Figure 4—figure supplement 1G.

5) Not clear exactly what the eye blink response analysis achieves in the paper, nor why eye blinks were analyzed but not anticipatory licking.

Please see our responses to the editor’s point #2 and reviewer 1’s point #1. The main message from the analysis on the neural activity-behavior correlations is to point out the importance of not only the contexts but also behavior in order to understand potential causes of the discrepancies between studies. For reward responses, there are not much physiological discrepancies, which may be due to strong rewarding effects of water, compared to relatively mild aversion with mild air puff. Although anticipatory licking is much more consistent, we observed similar tendency of dopamine responses in relation to anticipatory licking. We now added these results in Figure 5F and Figure 5—figure supplement 4 and in Results (last paragraph).

6) There is no reported method for the correction for multiple comparisons in multiple instances when more than 2 groups are being analyzed in the same panel. Additionally, a two-way ANOVA should be used to analyze the effect of reward probability and anticipatory blinking in Figure 4F.

Thank you for the suggestions. A two-way ANOVA cannot be used in Figure 5E (the original Figure 4F) because these are two different data sets. We agree that the figure was confusing and we modified the graphs (boxplots derived from two different tasks were displaced separately). We applied Bonferroni’s correction for significance tests with multiple comparisons and added this statement in Materials and methods.

Reviewer #3:

In this manuscript Matsumoto and colleagues present a series of experiments that address the issue of how dopamine neurons respond to aversive events. They make a number of important and interesting observations. On the whole I thought the manuscript was well-written. I have a few concerns that I list below:

1) I feel that the Introduction would benefit from some elaboration in a few places.

Thank you for the advice. We expanded some discussions in Introduction.

Introduction, fourth paragraph: should make it clear that Oleson et al. (2012) measured dopamine release using FSCV.

RESPONSE: We added this information (Introduction, fourth paragraph).

Introduction, fifth paragraph: I am not clear how Roitman et al. (2008) resolved the issue and this was published several years earlier, so the wording is confusing in my view.

We rephrased the sentence (Introduction, fifth paragraph).

Introduction, fifth paragraph: should make it clear that Lerner et al. (2015) measured calcium activity, not action potential firing.

We added this information (Introduction, fifth paragraph).

Introduction, last paragraph: this paragraph alludes to the issue of neurochemical identity in a way that I think should be more explicit since it is of particular relevance to this study. For example, Ungless et al. (2004) showed that some VTA neurons that are excited by aversive events are in fact not dopaminergic. It is noted, for example by Schultz (2016) in a recent review, that Matsumoto & Hikosaka (2009) recorded many neurons in areas where dopamine neurons were rare. This more strongly justifies the use of the opto-tagging approach that the authors have taken.

Thank you for letting us know! We now added these sentences in Introduction (eighth paragraph) and emphasized more about identification in Abstract and Discussion.

However, regarding the citation of Schultz’s statement on potential problems in recording locations in some previous studies, our opinion is that we are not in a position to strongly support either view (Schultz’s or the original researchers’). We, therefore, would like to not include this sentence (“Furthermore, Schultz has argued that some previous recording studies may not have targeted areas rich in dopamine neurons (Schultz, 2016)”) if the reviewers are fine with it.

2) One general concern I had was the low numbers of mice in several of the experimental groups (e.g., Figure 1, n=3).

Please see the answer to the editor’s point #3. The number of animals is comparable to other electrophysiology studies of dopamine neurons using monkeys (Fiorillo, 2013, Science; Fiorillo et al., 2013a, J Neurosci; Fiorillo et al., 2013b, J Neurosci; Kobayashi and Schultz, 2014, Curr Biol; Matsumoto and Hikosaka, 2009, Nature) and mice (Cohen et al., 2012, Nature; Eshel et al., 2015, Nature; Tian and Uchida, 2015, Neuron). We also observed similar tendencies by using data from two instead of three animals (after leaving out one out of three animals). We added this data in Figure 1—figure supplement 1.

3) How do the authors know that the mice can tell the difference between the low and high reward situations?

This is an important point. We observed that the level of anticipatory licking to reward cue was different between these two contexts (stronger with the reward CS predicting high probability of water). We have added this result in Figure 5—figure supplement 4B. Dopamine responses are also different between these two contexts. However, how reward contexts changed mouse conditions (consciously or unconsciously) is an important question to be resolved in future experiments.

4) In the subsection “Integration of aversiveness and reward in dopamine neurons”, first paragraph: add the caveat that activation of meso-cortical dopamine neurons and dorsal raphe dopamine neurons can induce conditioned-place aversion (Lammel et al. 2014 & Matthews et al. 2016).

Thank you for pointing out these references. We have added these references in Discussion(subsection “Integration of aversiveness and reward in dopamine neurons”, first paragraph).

4) In the subsection “Diversity of dopamine neurons”: note also that Brischoux et al. (2009) reported that ventromedial dopamine neurons exhibited aversive activations.

Thank you. We have added this information in Discussion (subsection “Diversity of dopamine neurons”, first paragraph).

5) I found some of the red/purple colour schemes difficult to tell apart. I would suggest some different colour schemes (particularly for Figure 1).

Thank you for pointing this out. We changed the blue and purple colors lighter in Figures.

[Editors' note: further revisions were requested prior to acceptance, as described below.]

The manuscript has been improved but there are some remaining issues regarding its readability that need to be addressed before acceptance, as outlined below by Reviewer 2. After consultation between the reviewers, both thought that increased readability and explanation of why each experiment was performed would greatly enhance the clarity and impact of the study.

Thank you for the advice. We added one paragraph specifically devoted to explanations of each experiment and comparisons between task conditions at the beginning of the Results, together with Figure 1 and Table 1.

Reviewer #2:

I felt that the reviewers were responsive to our comments and that this study provides a valuable contribution to the discussion regarding the integration of rewarding and aversive cues in dopamine neurons.

My only remaining comment is that I find the text in the Results and Discussion a bit difficult to follow. There are a number of experiments/analyses, and as it currently stands, the motivation for each experiment/analysis, as well as the relevant conclusions, is not always completely transparent. Therefore, the authors could consider editing the Results/Discussion for improved readability.

We added explanations of each experiment as mentioned above. We also rearranged Discussion and deleted some sentences for better readability.

Author details

1. Hideyuki Matsumoto

   1. Center for Brain Science, Harvard University, Cambridge, United States
   2. Department of Molecular and Cellular Biology, Harvard University, Cambridge, United States

   Contribution

   HM, Performed video analysis of eye blinking behaviors, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   No competing interests declared.

2. Ju Tian

   1. Center for Brain Science, Harvard University, Cambridge, United States
   2. Department of Molecular and Cellular Biology, Harvard University, Cambridge, United States

   Contribution

   JT, Performed video analysis of eye blinking behaviors

   Competing interests

   No competing interests declared.

3. Naoshige Uchida

   1. Center for Brain Science, Harvard University, Cambridge, United States
   2. Department of Molecular and Cellular Biology, Harvard University, Cambridge, United States

   Contribution

   NU, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   Competing interests

   NU: Reviewing editor, eLife

   "This ORCID iD identifies the author of this article:" 0000-0002-5755-9409

4. Mitsuko Watabe-Uchida

   1. Center for Brain Science, Harvard University, Cambridge, United States
   2. Department of Molecular and Cellular Biology, Harvard University, Cambridge, United States

   Contribution

   MW-U, Conception and design, Analysis and interpretation of data, Drafting or revising the article

   For correspondence

   mitsuko@mcb.harvard.edu

   Competing interests

   No competing interests declared.

   "This ORCID iD identifies the author of this article:" 0000-0001-7864-754X

• 5,157

  Page views

• 1,024

  Downloads

• 59

  Citations

Article citation count generated by polling the highest count across the following sources: Scopus, Crossref, PubMed Central.