Signed and unsigned reward prediction errors dynamically enhance learning and memory

  1. Nina Rouhani  Is a corresponding author
  2. Yael Niv
  1. Princeton University, United States

Abstract

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory, and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

Data availability

All data files and code for models, analysis and figures are publicly available at https://github.com/ninarouhani/2021_RouhaniNiv

The following data sets were generated

Article and author information

Author details

  1. Nina Rouhani

    Princeton Neuroscience Institute, Princeton University, Princeton, United States
    For correspondence
    nrouhani@princeton.edu
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2814-0462
  2. Yael Niv

    Princeton Neuroscience Institute, Princeton University, Princeton, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-0259-8371

Funding

Army Research Office (W911NF-14-1-0101)

  • Yael Niv

National Institute of Mental Health (R01MH098861)

  • Yael Niv

National Science Foundation (Graduate Student Fellowship)

  • Nina Rouhani

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Human subjects: We obtained informed consent online; procedures were approved by Princeton University's Institutional Review Board (IRB #4452).

Reviewing Editor

  1. Thorsten Kahnt, Northwestern University, United States

Publication history

  1. Received: July 14, 2020
  2. Accepted: February 26, 2021
  3. Accepted Manuscript published: March 4, 2021 (version 1)
  4. Version of Record published: April 12, 2021 (version 2)

Copyright

© 2021, Rouhani & Niv

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 3,398
    Page views
  • 508
    Downloads
  • 9
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Nina Rouhani
  2. Yael Niv
(2021)
Signed and unsigned reward prediction errors dynamically enhance learning and memory
eLife 10:e61077.
https://doi.org/10.7554/eLife.61077

Further reading

    1. Neuroscience
    Orie T Shafer et al.
    Research Article

    The circadian clock orchestrates daily changes in physiology and behavior to ensure internal temporal order and optimal timing across the day. In animals, a central brain clock coordinates circadian rhythms throughout the body and is characterized by a remarkable robustness that depends on synaptic connections between constituent neurons. The clock neuron network of Drosophila, which shares network motifs with clock networks in the mammalian brain yet is built of many fewer neurons, offers a powerful model for understanding the network properties of circadian timekeeping. Here we report an assessment of synaptic connectivity within a clock network, focusing on the critical lateral neuron (LN) clock neuron classes within the Janelia hemibrain dataset. Our results reveal that previously identified anatomical and functional subclasses of LNs represent distinct connectomic types. Moreover, we identify a small number of non-clock cell subtypes representing highly synaptically coupled nodes within the clock neuron network. This suggests that neurons lacking molecular timekeeping likely play integral roles within the circadian timekeeping network. To our knowledge, this represents the first comprehensive connectomic analysis of a circadian neuronal network.

    1. Developmental Biology
    2. Neuroscience
    Mariah L Hoye et al.
    Research Article

    Mutations in the RNA helicase, DDX3X, are a leading cause of Intellectual Disability and present as DDX3X syndrome, a neurodevelopmental disorder associated with cortical malformations and autism. Yet, the cellular and molecular mechanisms by which DDX3X controls cortical development are largely unknown. Here, using a mouse model of Ddx3x loss-of-function we demonstrate that DDX3X directs translational and cell cycle control of neural progenitors, which underlies precise corticogenesis. First, we show brain development is sensitive to Ddx3x dosage; complete Ddx3x loss from neural progenitors causes microcephaly in females, whereas hemizygous males and heterozygous females show reduced neurogenesis without marked microcephaly. In addition, Ddx3x loss is sexually dimorphic, as its paralog, Ddx3y, compensates for Ddx3x in the developing male neocortex. Using live imaging of progenitors, we show that DDX3X promotes neuronal generation by regulating both cell cycle duration and neurogenic divisions. Finally, we use ribosome profiling in vivo to discover the repertoire of translated transcripts in neural progenitors, including those which are DDX3X-dependent and essential for neurogenesis. Our study reveals invaluable new insights into the etiology of DDX3X syndrome, implicating dysregulated progenitor cell cycle dynamics and translation as pathogenic mechanisms.