On the normative advantages of dopamine and striatal opponency for learning and choice

  1. Alana Jaskir  Is a corresponding author
  2. Michael J Frank  Is a corresponding author
  1. Brown University, United States

Abstract

The basal ganglia (BG) contribute to reinforcement learning (RL) and decision making, but unlike artificial RL agents, it relies on complex circuitry and dynamic dopamine modulaton of opponent striatal pathways to do so. We develop the OpAL* model to assess the normative advantages of this circuitry. In OpAL*, learning induces opponent pathways to differentially emphasize the history of positive or negative outcomes for each action. Dynamic DA modulation then amplifies the pathway most tuned for the task environment. This efficient coding mechanism avoids a vexing explore-exploit tradeoff that plagues traditional RL models in sparse reward environments. OpAL* exhibits robust advantages over alternative models, particularly in environments with sparse reward and large action spaces. These advantages depend on opponent and nonlinear Hebbian plasticity mechanisms previously thought to be pathological. Finally, OpAL* captures risky choice patterns arising from DA and environmental manipulations across species, suggesting that they result from a normative biological mechanism.

Data availability

The current manuscript is a computational study, so no data have been generated for this manuscript. Simulation code is available on the authors' GitHub repositories https://github.com/amjaskir/opal-star

Article and author information

Author details

  1. Alana Jaskir

    Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, United States
    For correspondence
    alana_jaskir@brown.edu
    Competing interests
    No competing interests declared.
  2. Michael J Frank

    Department of Cognitive, Linguistic and Psychological Sciences, Brown University, Providence, United States
    For correspondence
    Michael_Frank@brown.edu
    Competing interests
    Michael J Frank, Senior editor, eLife.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-8451-0523

Funding

National Institute of Mental Health (P50MH119467)

  • Michael J Frank

National Institute of Mental Health (R01 MH084840)

  • Michael J Frank

National Institutes of Health (S10OD025181)

  • Michael J Frank

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Reviewing Editor

  1. Mimi Liljeholm, University of California, Irvine, United States

Version history

  1. Preprint posted: March 13, 2022 (view preprint)
  2. Received: November 22, 2022
  3. Accepted: March 14, 2023
  4. Accepted Manuscript published: March 22, 2023 (version 1)
  5. Version of Record published: May 19, 2023 (version 2)

Copyright

© 2023, Jaskir & Frank

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 1,769
    Page views
  • 259
    Downloads
  • 9
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Alana Jaskir
  2. Michael J Frank
(2023)
On the normative advantages of dopamine and striatal opponency for learning and choice
eLife 12:e85107.
https://doi.org/10.7554/eLife.85107

Share this article

https://doi.org/10.7554/eLife.85107

Further reading

    1. Developmental Biology
    2. Neuroscience
    Tariq Zaman, Daniel Vogt ... Michael R Williams
    Research Article

    The cell-type-specific expression of ligand/receptor and cell-adhesion molecules is a fundamental mechanism through which neurons regulate connectivity. Here, we determine a functional relevance of the long-established mutually exclusive expression of the receptor tyrosine kinase Kit and the trans-membrane protein Kit Ligand by discrete populations of neurons in the mammalian brain. Kit is enriched in molecular layer interneurons (MLIs) of the cerebellar cortex (i.e., stellate and basket cells), while cerebellar Kit Ligand is selectively expressed by a target of their inhibition, Purkinje cells (PCs). By in vivo genetic manipulation spanning embryonic development through adulthood, we demonstrate that PC Kit Ligand and MLI Kit are required for, and capable of driving changes in, the inhibition of PCs. Collectively, these works in mice demonstrate that the Kit Ligand/Kit receptor dyad sustains mammalian central synapse function and suggest a rationale for the affiliation of Kit mutation with neurodevelopmental disorders.

    1. Neuroscience
    Hideo Hagihara, Hirotaka Shoji ... Tsuyoshi Miyakawa
    Research Article

    Increased levels of lactate, an end-product of glycolysis, have been proposed as a potential surrogate marker for metabolic changes during neuronal excitation. These changes in lactate levels can result in decreased brain pH, which has been implicated in patients with various neuropsychiatric disorders. We previously demonstrated that such alterations are commonly observed in five mouse models of schizophrenia, bipolar disorder, and autism, suggesting a shared endophenotype among these disorders rather than mere artifacts due to medications or agonal state. However, there is still limited research on this phenomenon in animal models, leaving its generality across other disease animal models uncertain. Moreover, the association between changes in brain lactate levels and specific behavioral abnormalities remains unclear. To address these gaps, the International Brain pH Project Consortium investigated brain pH and lactate levels in 109 strains/conditions of 2294 animals with genetic and other experimental manipulations relevant to neuropsychiatric disorders. Systematic analysis revealed that decreased brain pH and increased lactate levels were common features observed in multiple models of depression, epilepsy, Alzheimer’s disease, and some additional schizophrenia models. While certain autism models also exhibited decreased pH and increased lactate levels, others showed the opposite pattern, potentially reflecting subpopulations within the autism spectrum. Furthermore, utilizing large-scale behavioral test battery, a multivariate cross-validated prediction analysis demonstrated that poor working memory performance was predominantly associated with increased brain lactate levels. Importantly, this association was confirmed in an independent cohort of animal models. Collectively, these findings suggest that altered brain pH and lactate levels, which could be attributed to dysregulated excitation/inhibition balance, may serve as transdiagnostic endophenotypes of debilitating neuropsychiatric disorders characterized by cognitive impairment, irrespective of their beneficial or detrimental nature.