Model-based reasoning requires using information about the structure of the world to infer information on-the-fly. This ability relies on a circuit involving the orbitofrontal cortex (OFC), as evidenced by the finding that humans and laboratory animals cannot infer value in several situations when OFC function is impaired (Gallagher et al., 1999; Izquierdo et al., 2004; McDannald et al., 2005; Takahashi et al., 2009; West et al., 2011; Gremel and Costa, 2013; Reber et al., 2017; Howard et al., 2020). Model-based reasoning can be isolated in sensory preconditioning (Brogden, 1939). During this task, subjects first learn an association between neutral or valueless stimuli. If one of these stimuli is thereafter paired with a biologically meaningful stimulus (e.g. food), presentation of the never-reward-paired preconditioned stimulus will produce a strong food-approach response. This demonstrates that subjects formed an associative model in which the preconditioned cue predicts the conditioned cue, which can subsequently be mobilized to produce a food-approach response when the preconditioned cue is presented. Because the preconditioned cue itself was never reinforced, this response to the preconditioned cue cannot be explained by model-free mechanisms. Preconditioned responding is reflective of the ability of subjects to infer value on-the-fly using knowledge about the associative structure of the information acquired during preconditioning.

Previous work from our lab has shown that the lateral OFC activity is necessary for inferring the value of never-reinforced sensory stimuli in the probe test at the end of sensory preconditioning (Jones et al., 2012). Interestingly, subsequent single-unit recording showed that neural activity in the OFC reflects not only the value predictions in the final probe test, but also tracks acquisition of the sensory-sensory associations in the initial phase of training (Sadacca et al., 2018). This result suggested a wider role for OFC in building associative models, independent of encoding of overt biological significance or value, the role often ascribed to this region (Kringelbach, 2005; Padoa-Schioppa and Assad, 2006; Padoa-Schioppa, 2011; Levy and Glimcher, 2012).

Of course, single-unit recording is correlative and cannot demonstrate that OFC activity is necessary for proper acquisition of the sensory-sensory associations during preconditioning. The correlates could simply reflect activity in primary sensory (Engineer et al., 2008; Garrido et al., 2009; Headley and Weinberger, 2015) or other cortical areas that are necessary in the first phase (Port et al., 1987; Robinson et al., 2011; Robinson et al., 2014; Fournier et al., 2020). If this is the case, then interference with processing in OFC during preconditioning should not affect responding to the preconditioned cue during the probe test. On the other hand, if the representations of the sensory-sensory associations formed in OFC in the initial phase contribute to building the model of the task used to infer value and drive responding in the probe test, then interference with processing in OFC during preconditioning should affect probe test responding, either directly impairing normal responding to the preconditioned cue, if responding in our task is entirely model-based as we have suggested (Jones et al., 2012; Sadacca et al., 2018), or by making any responding insensitive to devaluation of the predicted food reward, if responding in the probe is partly maintained by transfer of cached value to the cue during conditioning (Doll and Daw, 2016). Here we tested for such functional effects by training rats on a sensory preconditioning task nearly identical to those previously used by our lab (Jones et al., 2012; Sharpe et al., 2017; Sadacca et al., 2018), optogenetically inactivating the lateral OFC during cue presentation in the preconditioning phase of training and adding an additional assessment of responding after devaluation at the end of the experiment. We found that interfering with processing in OFC during the preconditioning phase completely abolished responding to the preconditioned cue in the probe test after conditioning, responding that in controls was subsequently disrupted by devaluation of the expected food reward.

While the OFC is generally thought to contribute to behavior where there are changes in, or computations of, value (Gallagher et al., 1999; Kringelbach, 2005; Padoa-Schioppa and Assad, 2006; Schoenbaum et al., 2011), our lab recently showed that neural activity in lateral OFC can also track the development of valueless sensory-sensory associative information in a sensory preconditioning task (Sadacca et al., 2018). While performance in the final phase of this task is ultimately sensitive to OFC manipulations in both rats (Jones et al., 2012) and humans (Wang et al., 2020b), the development of neural correlates of neutral sensory associations in the initial phase of training suggested that OFC might not be restricted to encoding information once clear biological significance is established and instead could be more widely involved in representing structure among features of the environment, with value being just one of these features. Here we tested this question by optogenetically-inactivating the OFC during the initial phase of preconditioning, when correlates were previously observed, and we found that this manipulation also disrupted performance in the final phase of the task. These findings are consistent with the argument that the associative correlates we observed during preconditioning in our prior study are not simply downstream reflections of sensory or other cortical processing and instead must contribute to the construction of the model used to guide responding in the final probe test. These data are also in accord with the findings that OFC represents not only value, but also features of the environment such as trial type (Lopatina et al., 2017), sequence (Zhou et al., 2019), direction, and goal location (Feierstein et al., 2006; Mainen and Kepecs, 2009).

These results come with several important considerations that are worth discussing. The first is whether OFC inhibition caused the rats to simply generalize between the preconditioned cues. In considering this possibility, we would first note that generalization or failure to treat these cues differently would be to some extent the final common outcome of disrupting the more detailed encoding of their identities and different associative relationships to reward that we argue is affected. For that reason, it is difficult to formally rule this out. However we believe that generalization as a primary cause is unlikely given the low overall levels of responding to these cues during the probe, relative to responses to the reward-paired cue. Additionally, responding to the two preconditioned cues in the experimental group was no different than responding to the preconditioned cue predicting reward in the control group. If generalization alone accounted for the results, we would expect higher levels of responding to both cues during the probe in the OFC-inhibited rats. There is also no evidence from other studies that the OFC is necessary for discriminating the sensory properties of cues; rats without OFC or in whom the OFC is inactivated perform well in a number of settings that require them to distinguish different cues to guide responding (Schoenbaum et al., 2002; Boulougouris et al., 2007; Takahashi et al., 2009; Jones et al., 2012).

A second consideration to discuss is the timing of our manipulation and its specificity. We inhibited OFC only during cue presentation, precisely when we previously observed single unit correlates of cue-cue encoding; our design did not include control conditions in which similar inhibition was done before or after the cues or during the intertrial intervals. Thus, while we can conclude that OFC must be online at the time the neural correlates were observed, we cannot conclude that it does not need to be online at any other time during the first phase. Although OFC can be inactivated briefly between trials without affecting OFC-dependent learning and behavior (Takahashi et al., 2013), we would not be surprised if processing outside the strictly defined cue period – say immediately before or after – were also necessary for normal learning. Such a result would not invalidate what we have shown here, but it would show that learning is not fully completed during the external sensory events. While defining this critical period would be very interesting, it would require a succession of control groups that we did not deem practical (just after the cues, 20 s after the cues, 40 s after the cues, and so on).

A third consideration is whether the effect of optogenetic inhibition could reflect something other than the inhibition of the neural correlates demonstrated in our earlier study. While we know from our control group, that simply delivering light during the cue period is not sufficient, perhaps inhibiting OFC neurons during this period distracts the rats, because it is rewarding or punishing or for some other reason. If this were the case, then we would have expected to see an impact of this on learning and responding to these cues in the later periods of the task. We did not see any evidence of this. Further we know from a number of prior studies that brief periods of OFC inhibition, similar to that employed here, often have minimal or no effects on ongoing behavior (Takahashi et al., 2013; Gardner et al., 2017; Gardner et al., 2018). These results suggest this is not, by itself, rewarding, painful or otherwise distracting. Finally, it has been shown that lateral OFC inactivation is not sufficient to produce state-dependent learning, which provides very strong evidence that it does not create a tangible experience in and of itself (Panayi and Killcross, 2014).

A final consideration is that all rats received food cup approach shaping prior to preconditioning. It is possible that this changes the salience of the conditioning chamber context and causes increased attention to the cues during preconditioning, which may facilitate learning. If this is the case, then one interpretation of the current findings would be that OFC is necessary for normal acquisition of the associations during preconditioning because it contributes to this boost in attention, perhaps secondary to a role in processing the value of the context, rather than contributing to the encoding itself. This idea would be consistent with results showing correlates of risk (O'Neill and Schultz, 2010; Ogawa et al., 2013), uncertainty (van Duuren et al., 2009) and salience (Ogawa et al., 2013) in OFC neurons, as well as the finding that OFC value coding is attentionally gated (McGinty et al., 2016) and active in auditory oddball task (Nguyen and Lin, 2014). However, while it is possible that inactivating OFC affected learning of the sensory-sensory associations by interfering with a primary role for OFC in supporting the context or even cue salience, it seems more parsimonious to us to propose that the role OFC plays in supporting salience is secondary to the associative learning role that OFC clearly supports. This idea is consistent with attentional models, both old and new, that link cue salience to associative strength – changes in cue associative strength drive changes in salience (Mackintosh, 1975; Pearce and Hall, 1980; Haselgrove et al., 2010; Esber and Haselgrove, 2011).

Of course, we do not intend to suggest that OFC is the only area that is necessary for this sort of value-neutral sensory learning. This is surely not the case. Indeed even when preconditioning is the readout, we know that interfering with several cortical areas including hippocampus (Port et al., 1987), retrosplenial cortex (Robinson et al., 2011; Robinson et al., 2014; Fournier et al., 2020), and perirhinal cortex (Holmes et al., 2013; Wong et al., 2019) seems to impair the acquisition of sensory-sensory associations. While most cortical and subcortical areas encode variables relevant to predicting biologically-relevant outcomes (Milad and Quirk, 2002; Burgos-Robles et al., 2009; Kennerley et al., 2011; Cai and Padoa-Schioppa, 2012; Burgos-Robles et al., 2013; Giustino et al., 2016; Giustino et al., 2019; Halladay et al., 2020), few studies look at whether the same areas track incidental sensory-sensory associations. Perhaps correlates like those observed in the OFC are ubiquitous; since the organism does not yet understand whether this information is useful it is encoded and is available for some time to enter into relationships reflecting the function of the area. This account would predict that other cortical areas (e.g. prelimbic, infralimbic, cingulate, retrosplenial) will likely also exhibit correlates of incidental sensory-sensory pairings, like OFC, yet unique single unit responses would not be apparent until the probe test, when subjects must infer specific information about the preconditioned cues based on new information acquired during conditioning. Indeed, fMRI correlates of cue-cue learning were observed in precentral gyrus, middle occipital gyrus, insula, and hippocampus in a human sensory preconditioning task (Wang et al., 2020a), and single unit correlates also exist in nucleus accumbens (Cerri et al., 2014). Along similar lines, OFC might contribute something to how the sensory-sensory associations are laid down that is necessary for their subsequent recall for use in the probe test, when it involves appetitive responding, while not being involved more generally in all forms of sensory associations. This could be assessed with fear conditioning.

Lastly we would note that while preconditioned responding has been shown as sensitive to extinction of the conditioned cue in fear conditioning (Rizley and Rescorla, 1972) as well as to aversive conditioning when the conditioned cue is a tastant (Blundell et al., 2003), sensitivity of the preconditioned response to devaluation of the reward predicted by the conditioned cue is novel. This result is important because, while it is conciliant with a mechanism whereby preconditioned responding reflects inference about food in the probe test, it is less consistent with other mechanisms whereby the preconditioned cue has been proposed to acquire value directly through so-called mediated learning in the conditioning phase. Such directly acquired value should leave residual responding that is resistant to devaluation. We did not see any evidence of this, consistent with prior work showing that preconditioned cues trained as we have done here do not acquire value (Sharpe et al., 2017). The demonstration of the devaluation sensitivity of this response nicely supports our contention that its OFC-dependence, both here and in prior work, reflects the role of OFC in using the underlying associative model to infer the likely delivery of food.

All data generated are contained within the source data files for Figure 2 and 3.

1. 1. Blundell P
   2. Hall G
   3. Killcross S

   (2003) Preserved sensitivity to outcome value after lesions of the basolateral amygdala

   The Journal of Neuroscience 23:7702–7709.

   https://doi.org/10.1523/JNEUROSCI.23-20-07702.2003

   • PubMed
   • Google Scholar
2. 1. Boulougouris V
   2. Dalley JW
   3. Robbins TW

   (2007) Effects of orbitofrontal, infralimbic and prelimbic cortical lesions on serial spatial reversal learning in the rat

   Behavioural Brain Research 179:219–228.

   https://doi.org/10.1016/j.bbr.2007.02.005

   • PubMed
   • Google Scholar
3. 1. Brogden WJ

   (1939) Sensory pre-conditioning

   Journal of Experimental Psychology 25:323–332.

   https://doi.org/10.1037/h0058944

   • Google Scholar
4. 1. Burgos-Robles A
   2. Vidal-Gonzalez I
   3. Quirk GJ

   (2009) Sustained conditioned responses in prelimbic prefrontal neurons are correlated with fear expression and extinction failure

   Journal of Neuroscience 29:8474–8482.

   https://doi.org/10.1523/JNEUROSCI.0378-09.2009

   • PubMed
   • Google Scholar
5. 1. Burgos-Robles A
   2. Bravo-Rivera H
   3. Quirk GJ

   (2013) Prelimbic and infralimbic neurons signal distinct aspects of appetitive instrumental behavior

   PLOS ONE 8:e57575.

   https://doi.org/10.1371/journal.pone.0057575

   • PubMed
   • Google Scholar
6. 1. Cai X
   2. Padoa-Schioppa C

   (2012) Neuronal encoding of subjective value in dorsal and ventral anterior cingulate cortex

   Journal of Neuroscience 32:3791–3808.

   https://doi.org/10.1523/JNEUROSCI.3864-11.2012

   • PubMed
   • Google Scholar
7. 1. Cerri DH
   2. Saddoris MP
   3. Carelli RM

   (2014) Nucleus accumbens core neurons encode value-independent associations necessary for sensory preconditioning

   Behavioral Neuroscience 128:567–578.

   https://doi.org/10.1037/a0037797

   • PubMed
   • Google Scholar
8. 1. Doll BB
   2. Daw N

   (2016) Prediction error: the expanding role of dopamine

   eLife 5:e15963.

   https://doi.org/10.7554/eLife.15963

   • Google Scholar
9. 1. Engineer CT
   2. Perez CA
   3. Chen YH
   4. Carraway RS
   5. Reed AC
   6. Shetake JA
   7. Jakkamsetti V
   8. Chang KQ
   9. Kilgard MP

   (2008) Cortical activity patterns predict speech discrimination ability

   Nature Neuroscience 11:603–608.

   https://doi.org/10.1038/nn.2109

   • PubMed
   • Google Scholar
10. 1. Esber GR
    2. Haselgrove M

    (2011) Reconciling the influence of predictiveness and uncertainty on stimulus salience: a model of attention in associative learning

    Proceedings of the Royal Society B: Biological Sciences 278:2553–2561.

    https://doi.org/10.1098/rspb.2011.0836

    • Google Scholar
11. 1. Feierstein CE
    2. Quirk MC
    3. Uchida N
    4. Sosulski DL
    5. Mainen ZF

    (2006) Representation of spatial goals in rat orbitofrontal cortex

    Neuron 51:495–507.

    https://doi.org/10.1016/j.neuron.2006.06.032

    • Google Scholar
12. 1. Fournier DI
    2. Monasch RR
    3. Bucci DJ
    4. Todd TP

    (2020) Retrosplenial cortex damage impairs unimodal sensory preconditioning

    Behavioral Neuroscience 134:198–207.

    https://doi.org/10.1037/bne0000365

    • PubMed
    • Google Scholar
13. 1. Gallagher M
    2. McMahan RW
    3. Schoenbaum G

    (1999) Orbitofrontal cortex and representation of incentive value in associative learning

    The Journal of Neuroscience 19:6610–6614.

    https://doi.org/10.1523/JNEUROSCI.19-15-06610.1999

    • PubMed
    • Google Scholar
14. 1. Gardner MPH
    2. Conroy JS
    3. Shaham MH
    4. Styer CV
    5. Schoenbaum G

    (2017) Lateral orbitofrontal inactivation dissociates Devaluation-Sensitive behavior and economic choice

    Neuron 96:1192–1203.

    https://doi.org/10.1016/j.neuron.2017.10.026

    • PubMed
    • Google Scholar
15. 1. Gardner MP
    2. Conroy JC
    3. Styer CV
    4. Huynh T
    5. Whitaker LR
    6. Schoenbaum G

    (2018) Medial orbitofrontal inactivation does not affect economic choice

    eLife 7:e38963.

    https://doi.org/10.7554/eLife.38963

    • PubMed
    • Google Scholar
16. 1. Garrido MI
    2. Kilner JM
    3. Stephan KE
    4. Friston KJ

    (2009) The mismatch negativity: a review of underlying mechanisms

    Clinical Neurophysiology 120:453–463.

    https://doi.org/10.1016/j.clinph.2008.11.029

    • PubMed
    • Google Scholar
17. 1. Giustino TF
    2. Fitzgerald PJ
    3. Maren S

    (2016) Fear expression suppresses medial prefrontal cortical firing in rats

    PLOS ONE 11:e0165256.

    https://doi.org/10.1371/journal.pone.0165256

    • PubMed
    • Google Scholar
18. 1. Giustino TF
    2. Fitzgerald PJ
    3. Ressler RL
    4. Maren S

    (2019) Locus coeruleus toggles reciprocal prefrontal firing to reinstate fear

    PNAS 116:8570–8575.

    https://doi.org/10.1073/pnas.1814278116

    • PubMed
    • Google Scholar
19. 1. Gradinaru V
    2. Thompson KR
    3. Deisseroth K

    (2008) eNpHR: a Natronomonas halorhodopsin enhanced for optogenetic applications

    Brain Cell Biology 36:129–139.

    https://doi.org/10.1007/s11068-008-9027-6

    • PubMed
    • Google Scholar
20. 1. Gremel CM
    2. Costa RM

    (2013) Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions

    Nature Communications 4:2264.

    https://doi.org/10.1038/ncomms3264

    • PubMed
    • Google Scholar
21. 1. Halladay LR
    2. Kocharian A
    3. Piantadosi PT
    4. Authement ME
    5. Lieberman AG
    6. Spitz NA
    7. Coden K
    8. Glover LR
    9. Costa VD
    10. Alvarez VA
    11. Holmes A

    (2020) Prefrontal regulation of punished ethanol Self-administration

    Biological Psychiatry 87:967–978.

    https://doi.org/10.1016/j.biopsych.2019.10.030

    • PubMed
    • Google Scholar
22. 1. Haselgrove M
    2. Esber GR
    3. Pearce JM
    4. Jones PM

    (2010) Two kinds of attention in pavlovian conditioning: evidence for a hybrid model of learning

    Journal of Experimental Psychology: Animal Behavior Processes 36:456–470.

    https://doi.org/10.1037/a0018528

    • PubMed
    • Google Scholar
23. 1. Headley DB
    2. Weinberger NM

    (2015) Relational associative learning induces Cross-Modal plasticity in early visual cortex

    Cerebral Cortex 25:1306–1318.

    https://doi.org/10.1093/cercor/bht325

    • Google Scholar
24. 1. Holmes NM
    2. Parkes SL
    3. Killcross AS
    4. Westbrook RF

    (2013) The basolateral amygdala is critical for learning about neutral stimuli in the presence of danger, and the perirhinal cortex is critical in the absence of danger

    Journal of Neuroscience 33:13112–13125.

    https://doi.org/10.1523/JNEUROSCI.1998-13.2013

    • PubMed
    • Google Scholar
25. 1. Howard JD
    2. Reynolds R
    3. Smith DE
    4. Voss JL
    5. Schoenbaum G
    6. Kahnt T

    (2020) Targeted stimulation of human orbitofrontal networks disrupts Outcome-Guided behavior

    Current Biology 30:490–498.

    https://doi.org/10.1016/j.cub.2019.12.007

    • PubMed
    • Google Scholar
26. 1. Izquierdo A
    2. Suda RK
    3. Murray EA

    (2004) Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency

    Journal of Neuroscience 24:7540–7548.

    https://doi.org/10.1523/JNEUROSCI.1921-04.2004

    • PubMed
    • Google Scholar
27. 1. Jones JL
    2. Esber GR
    3. McDannald MA
    4. Gruber AJ
    5. Hernandez A
    6. Mirenzi A
    7. Schoenbaum G

    (2012) Orbitofrontal cortex supports behavior and learning using inferred but not cached values

    Science 338:953–956.

    https://doi.org/10.1126/science.1227489

    • PubMed
    • Google Scholar
28. 1. Kennerley SW
    2. Behrens TE
    3. Wallis JD

    (2011) Double dissociation of value computations in orbitofrontal and anterior cingulate neurons

    Nature Neuroscience 14:1581–1589.

    https://doi.org/10.1038/nn.2961

    • PubMed
    • Google Scholar
29. 1. Kringelbach ML

    (2005) The human orbitofrontal cortex: linking reward to hedonic experience

    Nature Reviews Neuroscience 6:691–702.

    https://doi.org/10.1038/nrn1747

    • PubMed
    • Google Scholar
30. 1. Levy DJ
    2. Glimcher PW

    (2012) The root of all value: a neural common currency for choice

    Current Opinion in Neurobiology 22:1027–1038.

    https://doi.org/10.1016/j.conb.2012.06.001

    • PubMed
    • Google Scholar
31. 1. Lopatina N
    2. Sadacca BF
    3. McDannald MA
    4. Styer CV
    5. Peterson JF
    6. Cheer JF
    7. Schoenbaum G

    (2017) Ensembles in medial and lateral orbitofrontal cortex construct cognitive maps emphasizing different features of the behavioral landscape

    Behavioral Neuroscience 131:201–212.

    https://doi.org/10.1037/bne0000195

    • PubMed
    • Google Scholar
32. 1. Mackintosh NJ

    (1975) A theory of attention: variations in the associability of stimuli with reinforcement

    Psychological Review 82:276–298.

    https://doi.org/10.1037/h0076778

    • Google Scholar
33. 1. Mainen ZF
    2. Kepecs A

    (2009) Neural representation of behavioral outcomes in the orbitofrontal cortex

    Current Opinion in Neurobiology 19:84–91.

    https://doi.org/10.1016/j.conb.2009.03.010

    • PubMed
    • Google Scholar
34. 1. McDannald MA
    2. Saddoris MP
    3. Gallagher M
    4. Holland PC

    (2005) Lesions of orbitofrontal cortex impair rats' differential outcome expectancy learning but not conditioned stimulus-potentiated feeding

    Journal of Neuroscience 25:4626–4632.

    https://doi.org/10.1523/JNEUROSCI.5301-04.2005

    • PubMed
    • Google Scholar
35. 1. McGinty VB
    2. Rangel A
    3. Newsome WT

    (2016) Orbitofrontal cortex value signals depend on fixation location during free viewing

    Neuron 90:1299–1311.

    https://doi.org/10.1016/j.neuron.2016.04.045

    • PubMed
    • Google Scholar
36. 1. Milad MR
    2. Quirk GJ

    (2002) Neurons in medial prefrontal cortex signal memory for fear extinction

    Nature 420:70–74.

    https://doi.org/10.1038/nature01138

    • PubMed
    • Google Scholar
37. 1. Nguyen DP
    2. Lin SC

    (2014) A frontal cortex event-related potential driven by the basal forebrain

    eLife 3:e02148.

    https://doi.org/10.7554/eLife.02148

    • PubMed
    • Google Scholar
38. 1. O'Neill M
    2. Schultz W

    (2010) Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value

    Neuron 68:789–800.

    https://doi.org/10.1016/j.neuron.2010.09.031

    • PubMed
    • Google Scholar
39. 1. Ogawa M
    2. van der Meer MA
    3. Esber GR
    4. Cerri DH
    5. Stalnaker TA
    6. Schoenbaum G

    (2013) Risk-responsive orbitofrontal neurons track acquired salience

    Neuron 77:251–258.

    https://doi.org/10.1016/j.neuron.2012.11.006

    • PubMed
    • Google Scholar
40. 1. Padoa-Schioppa C

    (2011) Neurobiology of economic choice: a good-based model

    Annual Review of Neuroscience 34:333–359.

    https://doi.org/10.1146/annurev-neuro-061010-113648

    • PubMed
    • Google Scholar
41. 1. Padoa-Schioppa C
    2. Assad JA

    (2006) Neurons in the orbitofrontal cortex encode economic value

    Nature 441:223–226.

    https://doi.org/10.1038/nature04676

    • PubMed
    • Google Scholar
42. 1. Panayi MC
    2. Killcross S

    (2014) Orbitofrontal cortex inactivation impairs between- but not within-session pavlovian extinction: an associative analysis

    Neurobiology of Learning and Memory 108:78–87.

    https://doi.org/10.1016/j.nlm.2013.08.002

    • PubMed
    • Google Scholar
43. 1. Pearce JM
    2. Hall G

    (1980) A model for pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli

    Psychological Review 87:532–552.

    https://doi.org/10.1037/0033-295X.87.6.532

    • Google Scholar
44. 1. Pickens CL
    2. Saddoris MP
    3. Setlow B
    4. Gallagher M
    5. Holland PC
    6. Schoenbaum G

    (2003) Different roles for orbitofrontal cortex and basolateral amygdala in a reinforcer devaluation task

    The Journal of Neuroscience 23:11078–11084.

    https://doi.org/10.1523/JNEUROSCI.23-35-11078.2003

    • PubMed
    • Google Scholar
45. 1. Port RL
    2. Beggs AL
    3. Patterson MM

    (1987) Hippocampal substrate of sensory associations

    Physiology & Behavior 39:643–647.

    https://doi.org/10.1016/0031-9384(87)90167-3

    • PubMed
    • Google Scholar
46. 1. Reber J
    2. Feinstein JS
    3. O'Doherty JP
    4. Liljeholm M
    5. Adolphs R
    6. Tranel D

    (2017) Selective impairment of goal-directed decision-making following lesions to the human ventromedial prefrontal cortex

    Brain 140:1743–1756.

    https://doi.org/10.1093/brain/awx105

    • PubMed
    • Google Scholar
47. 1. Rizley RC
    2. Rescorla RA

    (1972) Associations in second-order conditioning and sensory preconditioning

    Journal of Comparative and Physiological Psychology 81:1–11.

    https://doi.org/10.1037/h0033333

    • PubMed
    • Google Scholar
48. 1. Robinson S
    2. Keene CS
    3. Iaccarino HF
    4. Duan D
    5. Bucci DJ

    (2011) Involvement of retrosplenial cortex in forming associations between multiple sensory stimuli

    Behavioral Neuroscience 125:578–587.

    https://doi.org/10.1037/a0024262

    • PubMed
    • Google Scholar
49. 1. Robinson S
    2. Todd TP
    3. Pasternak AR
    4. Luikart BW
    5. Skelton PD
    6. Urban DJ
    7. Bucci DJ

    (2014) Chemogenetic silencing of neurons in retrosplenial cortex disrupts sensory preconditioning

    Journal of Neuroscience 34:10982–10988.

    https://doi.org/10.1523/JNEUROSCI.1349-14.2014

    • Google Scholar
50. 1. Sadacca BF
    2. Wied HM
    3. Lopatina N
    4. Saini GK
    5. Nemirovsky D
    6. Schoenbaum G

    (2018) Orbitofrontal neurons signal sensory associations underlying model-based inference in a sensory preconditioning task

    eLife 7:e30373.

    https://doi.org/10.7554/eLife.30373

    • PubMed
    • Google Scholar
51. 1. Schoenbaum G
    2. Nugent SL
    3. Saddoris MP
    4. Setlow B

    (2002) Orbitofrontal lesions in rats impair reversal but not acquisition of go, no-go odor discriminations

    Neuroreport 13:885–890.

    https://doi.org/10.1097/00001756-200205070-00030

    • PubMed
    • Google Scholar
52. 1. Schoenbaum G
    2. Takahashi Y
    3. Liu TL
    4. McDannald MA

    (2011) Does the orbitofrontal cortex signal value?

    Annals of the New York Academy of Sciences 1239:87–99.

    https://doi.org/10.1111/j.1749-6632.2011.06210.x

    • PubMed
    • Google Scholar
53. 1. Sharpe MJ
    2. Batchelor HM
    3. Schoenbaum G

    (2017) Preconditioned cues have no value

    eLife 6:e28362.

    https://doi.org/10.7554/eLife.28362

    • PubMed
    • Google Scholar
54. 1. Sidak Z

    (1967) Rectangular confidence regions for the means of multivariate normal distributions

    Journal of the American Statistical Association 62:626–633.

    https://doi.org/10.2307/2283989

    • Google Scholar
55. 1. Singh T
    2. Jones JL
    3. McDannald MA
    4. Haney RZ
    5. Cerri DH
    6. Schoenbaum G

    (2011) Normal aging does not impair Orbitofrontal-Dependent reinforcer devaluation effects

    Frontiers in Aging Neuroscience 3:4.

    https://doi.org/10.3389/fnagi.2011.00004

    • PubMed
    • Google Scholar
56. 1. Takahashi YK
    2. Roesch MR
    3. Stalnaker TA
    4. Haney RZ
    5. Calu DJ
    6. Taylor AR
    7. Burke KA
    8. Schoenbaum G

    (2009) The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes

    Neuron 62:269–280.

    https://doi.org/10.1016/j.neuron.2009.03.005

    • PubMed
    • Google Scholar
57. 1. Takahashi YK
    2. Chang CY
    3. Lucantonio F
    4. Haney RZ
    5. Berg BA
    6. Yau HJ
    7. Bonci A
    8. Schoenbaum G

    (2013) Neural estimates of imagined outcomes in the orbitofrontal cortex drive behavior and learning

    Neuron 80:507–518.

    https://doi.org/10.1016/j.neuron.2013.08.008

    • PubMed
    • Google Scholar
58. 1. van Duuren E
    2. van der Plasse G
    3. Lankelma J
    4. Joosten RN
    5. Feenstra MG
    6. Pennartz CM

    (2009) Single-cell and population coding of expected reward probability in the orbitofrontal cortex of the rat

    Journal of Neuroscience 29:8965–8976.

    https://doi.org/10.1523/JNEUROSCI.0005-09.2009

    • PubMed
    • Google Scholar
59. 1. Wang F
    2. Schoenbaum G
    3. Kahnt T

    (2020a) Interactions between human orbitofrontal cortex and Hippocampus support model-based inference

    PLOS Biology 18:e3000578.

    https://doi.org/10.1371/journal.pbio.3000578

    • PubMed
    • Google Scholar
60. Preprint

    1. Wang F
    2. Howard JD
    3. Voss JL
    4. Schoenbaum G
    5. Kahnt T

    (2020b) Human orbitofrontal cortex is necessary for behavior based on inferred, not experienced outcomes

    bioRxiv.

    https://doi.org/10.1101/2020.04.24.059808

    • Google Scholar
61. 1. West EA
    2. DesJardin JT
    3. Gale K
    4. Malkova L

    (2011) Transient inactivation of orbitofrontal cortex blocks reinforcer devaluation in macaques

    Journal of Neuroscience 31:15128–15135.

    https://doi.org/10.1523/JNEUROSCI.3295-11.2011

    • PubMed
    • Google Scholar
62. 1. Wong FS
    2. Westbrook RF
    3. Holmes NM

    (2019) 'Online' integration of sensory and fear memories in the rat medial temporal lobe

    eLife 8:e47085.

    https://doi.org/10.7554/eLife.47085

    • PubMed
    • Google Scholar
63. 1. Zhou J
    2. Gardner MPH
    3. Stalnaker TA
    4. Ramus SJ
    5. Wikenheiser AM
    6. Niv Y
    7. Schoenbaum G

    (2019) Rat orbitofrontal ensemble activity contains multiplexed but dissociable representations of value and task structure in an odor sequence task

    Current Biology 29:897–907.

    https://doi.org/10.1016/j.cub.2019.01.048

    • PubMed
    • Google Scholar

Thank you for submitting your article "Devaluation-sensitive responding to preconditioned cues requires orbitofrontal cortex during initial cue-cue learning" for consideration by eLife. Your article has been reviewed by Michael Frank as the Senior Editor, a Reviewing Editor, and two reviewers. The reviewers have opted to remain anonymous.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

We would like to draw your attention to changes in our revision policy that we have made in response to COVID-19 (https://elifesciences.org/articles/57162). Specifically, we are asking editors to accept without delay manuscripts, like yours, that they judge can stand as eLife papers without additional data, even if they feel that they would make the manuscript stronger. Thus the revisions requested below only address clarity and presentation.

Summary:

Sensory preconditioning has been a powerful behavioral tool for exploring how, when and where model based associative structures are stored in the brain. In the present experiment, a natural follow-up to a recent study by Sadacca et al., (2018), the authors explore a single outstanding issue relating to whether neural encoding of valueless associations at the time of preconditioning are actually necessary for subsequent associative value transfer to non-conditioned cues. The broader issue here is whether OFC is a specialized economic structure or whether it plays a more general role in linking information about outcomes to options. Here they use the temporal specificity of optogenetics, activating halorhodopsin to inhibit the activity of OFC neurons only when the two unconditioned cues are presented. They report that inhibition during this period abolishes subsequent responding to the unconditioned cue, indicating that OFC inhibition blocked cue-cue learning. A subsequent experiment using only the control animals, showed that devaluing the US was associated with reductions in responding confirming that model-based associations formed during sensory preconditioning are dependent on OFC.

Essential revisions:

This is an elegant albeit short report that will be of interest to many in the field but mostly as an important control to a larger body of work. There are, however, a couple of places where the manuscript could use some work, most notably in clarifying what the authors think the mechanisms of sensory preconditioning are and confounds relating to the design.

1) The authors only delivered laser light to inhibit OFC neurons during the 10 seconds where the two cues were presented together. This makes perfect sense as they were aiming to understand how disrupting OFC at this point impacts learning. A question then logically follows; is it the case that disrupting OFC for this length of time, irrespective of when during the preconditioning phase is sufficient to disrupt associative learning? Put simply, if 10 seconds of laser light was delivered during the ITI in the preconditioning phase, is this sufficient to disrupt learning/retention of the cue-cue associations? I'm not asking for an extra experiment here, but this should be acknowledged in the discussion as it speaks to the role of OFC in both learning AND/OR retention of cue-cue associations.

2) The discussion of attention in the learning of cue-cue value associations should be expanded as this is the key take away from the paper. Cues that the rats did not expect to be presented are likely learned about as they are attentionally salient. More of a discussion of the role of OFC in attentionally salient events and interaction with other areas feels more than a little appropriate here. For instance, OFC is known to be modulated during odd-ball tasks through input from the basal forebrain (Nguyen and Lin, 2014) and more recently work in primates has started to reveal that OFC encoding of value is attentionally gated (McGinty, et al., 2015). Similarly, how does this speak to learning models based on attention such as the Pearce-Hall model?

3) The paper is brief and relies on a small amount of data. From what I can tell, the key analysis is Figure 2C, which is described in subsection “Orbitofrontal cortex activity is necessary for forming sensory associations”. The thing I don't totally get is that the finding relies on a positive effect in the control condition and a null effect in the experimental condition (subsection “Orbitofrontal cortex activity is necessary for forming sensory associations”). Strictly speaking, this is invalid – the difference between significant and non-significant is not in itself significant (it can be and often is, but isn’t necessarily so, Nieuwenhuis, Forstmann and Wagenmakers, 2011). This is especially important to address because the control effect is marginally significant (p=0.046). Anyway, it looks like it probably is significant, but that ought to be reported. (Another reviewer during consultation surmised that you did do this analysis but they weren't quite sure "It looks like they have computed the interaction term and reported it 'however, there was a significant cue x group interaction (F(1,41)=4.57, p=0.04).'".

4) The second issue is that a disruption of effect, while it does demonstrate causality, has multiple possible interpretations. It could be that the optogenetic effect is, like, super-distracting, or causes a headache or something, and that causes the animal to get distracted and not do very well on the task. There are probably ways to deal with this concern, like showing other things that aren't disrupted. But in this case, I want to see that because we don't want to assume causality from any abolition of effect.

Essential revisions:

This is an elegant albeit short report that will be of interest to many in the field but mostly as an important control to a larger body of work. There are, however, a couple of places where the manuscript could use some work, most notably in clarifying what the authors think the mechanisms of sensory preconditioning are and confounds relating to the design.

We thank the editors and reviewers for their constructive feedback, which has greatly improved the manuscript. In response to this feedback, we have added considerable discussion of caveats, experimental design, elaboration on statistics, and expanded discussion of psychological processes that could mediate sensory preconditioning. We have also made minor changes to improve clarity – namely a change in the Title and added description of the experimental and control groups. In addition to this we have softened some of the language to reflect that it is now more established that OFC is not only engaged in value-guided learning and behavior, as a reviewer noted. Below we give our point-by-point responses.

1) The authors only delivered laser light to inhibit OFC neurons during the 10 seconds where the two cues were presented together. This makes perfect sense as they were aiming to understand how disrupting OFC at this point impacts learning. A question then logically follows; is it the case that disrupting OFC for this length of time, irrespective of when during the preconditioning phase is sufficient to disrupt associative learning? Put simply, if 10 seconds of laser light was delivered during the ITI in the preconditioning phase, is this sufficient to disrupt learning/retention of the cue-cue associations? I'm not asking for an extra experiment here, but this should be acknowledged in the discussion as it speaks to the role of OFC in both learning AND/OR retention of cue-cue associations.

We appreciate this concern and have added more discussion of this and other caveats to the Discussion section. Although we realize the ITI control is typical, we did not include it in the current study because we did not believe it was necessary. Briefly there are several reasons we felt this way. For starters, as the reviewers note, we chose our time period based on the identification of very specific neural correlates; the most direct way to disrupt these is to inactivate during these correlates. While it is straightforward to propose that inactivating in other time periods should not have any effect if our hypothesis is correct, it seems to us that this is not really correct. That is, inactivating in other periods – say right before or after the cues? – might also affect consolidation or other processes required for the learning. Yet this result would not invalidate the importance of our positive result from inhibition during the cue period. For this reason, we felt it was not critical to testing our hypothesis.

We also viewed the standard middle-of-the-ITI inhibition as a relatively trivial negative control in this particular situation, since the ITIs are several minutes long, and we already know from other studies that similar short periods of inactivation of OFC can be conducted without affecting learning and behavior. For example, in a Pavlovian over-expectation task we showed that inactivation during a period where OFC showed key neural correlates affected learning and behavior, whereas inactivation a couple of minutes later in the ITI did not.

Nevertheless, we realize that this concern will be shared by other readers, so we have added the below to the Discussion section to better explain the rationale behind this aspect of our experimental design:

“A second consideration to discuss is the timing of our manipulation and its specificity. We inhibited OFC only during cue presentation, precisely when we previously observed single unit correlates of cue-cue encoding; our design did not include control conditions in which similar inhibition was done before or after the cues or during the intertrial intervals. Thus, while we can conclude that OFC must be online at the time the neural correlates were observed, we cannot conclude that it does not need to be online at any other time during the first phase. Although OFC can be inactivated briefly between trials without affecting OFC-dependent learning and behavior (Takahashi et al., 2013), we would not be surprised if processing outside the strictly defined cue period – say immediately before or after – were also necessary for normal learning. Such a result would not invalidate what we have shown here, but it would show that learning is not fully completed during the external sensory events. While defining this critical period would be very interesting, it would require a succession of control groups that we did not deem practical (just after the cues, 20s after the cues, 40s after the cues, and so on).”

2) The discussion of attention in the learning of cue-cue value associations should be expanded as this is the key take away from the paper. Cues that the rats did not expect to be presented are likely learned about as they are attentionally salient. More of a discussion of the role of OFC in attentionally salient events and interaction with other areas feels more than a little appropriate here. For instance, OFC is known to be modulated during odd-ball tasks through input from the basal forebrain (Nguyen and Lin, 2014) and more recently work in primates has started to reveal that OFC encoding of value is attentionally gated (McGinty, et al., 2015). Similarly, how does this speak to learning models based on attention such as the Pearce-Hall model?

We agree this is a potentially important point, and we have expanded our Discussion section to try to address it more fully. As we note in the below, we recognize the evidence that the OFC is involved in and perhaps contributes to cue salience. However while we cannot rule disruption of this function out as the primary cause of our effect, it seems more parsimonious to us to propose that any such function is secondary to the associative learning role that OFC performs, as evidenced in the neural correlates that normally occur contemporaneous with our period of inhibition. This idea is consistent with ideas from Mackintosh as well as the Esber-Haselgrove successor to the Pearce-Hall and Mackintosh models, which proposes that cue salience varies with associative strength. Thus, our current results could be related to the role of OFC in regulating cue salience, but we would suggest it is because OFC is important for learning the sensory-sensory associations, not the other way around.

“A final consideration is that all rats received food cup approach shaping prior to preconditioning. It is possible that this changes the salience of the conditioning chamber context and causes increased attention to the cues during preconditioning, which may facilitate learning. If this is the case, then one interpretation of the current findings would be that OFC is necessary for normal acquisition of the associations during preconditioning because it contributes to this boost in attention, perhaps secondary to a role in processing the value of the context, rather than contributing to the encoding itself. This idea would be consistent with results showing correlates of risk (O'Neill and Schultz, 2010; Ogawa et al., 2013), uncertainty (van Duuren et al., 2009) and salience (Ogawa et al., 2013) in OFC neurons, as well as the finding that OFC value coding is attentionally gated (McGinty et al., 2016) and active in auditory oddball task (Nguyen and Lin, 2014). However while it is possible that inactivating OFC affected learning of the sensory-sensory associations by interfering with a primary role for OFC in supporting the context or even cue salience, it seems more parsimonious to us to propose that the role OFC plays in supporting salience is secondary to the associative learning role that OFC clearly supports. This idea is consistent with attentional models, both old and new, that link cue salience to associative strength – changes in cue associative strength drive changes in salience (Mackintosh, 1975; Pearce and Hall, 1980; Haselgrove et al., 2010; Esber and Haselgrove, 2011).”

3) The paper is brief and relies on a small amount of data. From what I can tell, the key analysis is Figure 2C, which is described in subsection “Orbitofrontal cortex activity is necessary for forming sensory associations”. The thing I don't totally get is that the finding relies on a positive effect in the control condition and a null effect in the experimental condition (subsection “Orbitofrontal cortex activity is necessary for forming sensory associations”). Strictly speaking, this is invalid – the difference between significant and non-significant is not in itself significant (it can be and often is, but isn’t necessarily so, Nieuwenhuis, Forstmann, and Wagenmakers, 2011). This is especially important to address because the control effect is marginally significant (p=0.046). Anyway, it looks like it probably is significant, but that ought to be reported. (Another reviewer during consultation surmised that you did do this analysis but they weren't quite sure "It looks like they have computed the interaction term and reported it 'however, there was a significant cue x group interaction (F(1,41)=4.57, p=0.04).'".

We regret that we did not make it clearer that we do report the interaction, as the reviewers surmised. The text has been updated to reflect this:

“There was a significant cue x group interaction (F_(1,41)=4.57, p=0.04); ergo, the effect of cue during the probe test (A-C) did depend on group (YFP control – NpHR OFC inhibition)”.

4) The second issue is that a disruption of effect, while it does demonstrate causality, has multiple possible interpretations. It could be that the optogenetic effect is, like, super-distracting, or causes a headache or something, and that causes the animal to get distracted and not do very well on the task. There are probably ways to deal with this concern, like showing other things that aren't disrupted. But in this case, I want to see that because we don't want to assume causality from any abolition of effect.

We appreciate this issue but feel it is mitigated by several findings: (1) Identical stimulation in both lateral and medial OFC had no effect on economic choice behavior in our lab. These citations have been added. (2) If the stimulation was painful/aversive, then it would be expected to impair reward learning about these cues, since light occurred during each cue presentation. We observed intact reward conditioning in phase 2, and there was no difference between groups. We also did not see any effect of group during preconditioning when light was delivered. (3) Inactivation of lateral OFC is not sufficient to produce state-dependent learning. These have been added to the Discussion section:

“A third consideration is whether the effect of optogenetic inhibition could reflect something other than the inhibition of the neural correlates demonstrated in our earlier study. While we know from our control group, that simply delivering light during the cue period is not sufficient, perhaps inhibiting OFC neurons during this period distracts the rats, because it is rewarding or punishing or for some other reason. If this were the case, then we would have expected to see an impact of this on learning and responding to these cues in the later periods of the task. We did not see any evidence of this. Further we know from a number of prior studies that brief periods of OFC inhibition, similar to that employed here, often have minimal or no effects on ongoing behavior (Takahashi et al., 2013; Gardner et al., 2017; Gardner et al., 2018). These results suggest this is not, by itself, rewarding, painful or otherwise distracting. Finally, it has been shown that lateral OFC inactivation is not sufficient to produce state-dependent learning, which provides very strong evidence that it does not create a tangible experience in and of itself (Panayi and Killcross, 2014).”

Author details

1. Evan E Hart

   National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, United States

   Contribution

   Conceptualization, Formal analysis, Investigation, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   evan.hart@nih.gov

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-1487-5628

2. Melissa J Sharpe

   1. National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, United States
   2. Department of Psychology, University of California, Los Angeles, Los Angeles, United States

   Contribution

   Conceptualization, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5375-2076

3. Matthew PH Gardner

   National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, United States

   Contribution

   Conceptualization, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-9146-5043

4. Geoffrey Schoenbaum

   1. National Institute on Drug Abuse Intramural Research Program, National Institutes of Health, Baltimore, United States
   2. Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, United States
   3. Department of Psychiatry, University of Maryland School of Medicine, Baltimore, United States
   4. Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, United States

   Contribution

   Conceptualization, Supervision, Funding acquisition, Writing - original draft, Writing - review and editing

   For correspondence

   geoffrey.schoenbaum@nih.gov

   Competing interests

   Reviewing editor, eLife

   "This ORCID iD identifies the author of this article:" 0000-0001-8180-0701

• 1,500

  Page views

• 195

  Downloads

• 15

  Citations

Article citation count generated by polling the highest count across the following sources: PubMed Central, Crossref, Scopus.