On the normative advantages of dopamine and striatal opponency for learning and choice
Abstract
The basal ganglia (BG) contribute to reinforcement learning (RL) and decision making, but unlike artificial RL agents, it relies on complex circuitry and dynamic dopamine modulaton of opponent striatal pathways to do so. We develop the OpAL* model to assess the normative advantages of this circuitry. In OpAL*, learning induces opponent pathways to differentially emphasize the history of positive or negative outcomes for each action. Dynamic DA modulation then amplifies the pathway most tuned for the task environment. This efficient coding mechanism avoids a vexing explore-exploit tradeoff that plagues traditional RL models in sparse reward environments. OpAL* exhibits robust advantages over alternative models, particularly in environments with sparse reward and large action spaces. These advantages depend on opponent and nonlinear Hebbian plasticity mechanisms previously thought to be pathological. Finally, OpAL* captures risky choice patterns arising from DA and environmental manipulations across species, suggesting that they result from a normative biological mechanism.
Data availability
The current manuscript is a computational study, so no data have been generated for this manuscript. Simulation code is available on the authors' GitHub repositories https://github.com/amjaskir/opal-star
Article and author information
Author details
Funding
National Institute of Mental Health (P50MH119467)
- Michael J Frank
National Institute of Mental Health (R01 MH084840)
- Michael J Frank
National Institutes of Health (S10OD025181)
- Michael J Frank
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Reviewing Editor
- Mimi Liljeholm, University of California, Irvine, United States
Version history
- Preprint posted: March 13, 2022 (view preprint)
- Received: November 22, 2022
- Accepted: March 14, 2023
- Accepted Manuscript published: March 22, 2023 (version 1)
- Version of Record published: May 19, 2023 (version 2)
Copyright
© 2023, Jaskir & Frank
This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.
Metrics
-
- 1,960
- views
-
- 276
- downloads
-
- 14
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading
-
- Neuroscience
Mammals have evolved sex-specific adaptations to reduce energy usage in times of food scarcity. These adaptations are well described for peripheral tissue, though much less is known about how the energy-expensive brain adapts to food restriction, and how such adaptations differ across the sexes. Here, we examined how food restriction impacts energy usage and function in the primary visual cortex (V1) of adult male and female mice. Molecular analysis and RNA sequencing in V1 revealed that in males, but not in females, food restriction significantly modulated canonical, energy-regulating pathways, including pathways associated waith AMP-activated protein kinase, peroxisome proliferator-activated receptor alpha, mammalian target of rapamycin, and oxidative phosphorylation. Moreover, we found that in contrast to males, food restriction in females did not significantly affect V1 ATP usage or visual coding precision (assessed by orientation selectivity). Decreased serum leptin is known to be necessary for triggering energy-saving changes in V1 during food restriction. Consistent with this, we found significantly decreased serum leptin in food-restricted males but no significant change in food-restricted females. Collectively, our findings demonstrate that cortical function and energy usage in female mice are more resilient to food restriction than in males. The neocortex, therefore, contributes to sex-specific, energy-saving adaptations in response to food restriction.
-
- Neuroscience
Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether (‘invariance’), represented in non-interfering subspaces of population activity (‘factorization’) or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters – lighting, background, camera viewpoint, and object pose – in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI, and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not as consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is often preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.