Reward-based training of recurrent neural networks for cognitive and value-based tasks

  1. H Francis Song
  2. Guangyu R Yang
  3. Xiao-Jing Wang  Is a corresponding author
  1. New York University, United States

Abstract

Trained neural network models, which exhibit features of neural activity recorded from behaving animals, may provide insights into the circuit mechanisms of cognitive functions through systematic analysis of network activity and connectivity. However, in contrast to the graded error signals commonly used to train networks through supervised learning, animals learn from reward feedback on definite actions through reinforcement learning. Reward maximization is particularly relevant when optimal behavior depends on an animal's internal judgment of confidence or subjective preferences. Here, we implement reward-based training of recurrent neural networks in which a value network guides learning by using the activity of the decision network to predict future reward. We show that such models capture behavioral and electrophysiological findings from well-known experimental paradigms. Our work provides a unified framework for investigating diverse cognitive and value-based computations, and predicts a role for value representation that is essential for learning, but not executing, a task.

Article and author information

Author details

  1. H Francis Song

    Center for Neural Science, New York University, New York, United States
    Competing interests
    The authors declare that no competing interests exist.
  2. Guangyu R Yang

    Center for Neural Science, New York University, New York, United States
    Competing interests
    The authors declare that no competing interests exist.
  3. Xiao-Jing Wang

    Center for Neural Science, New York University, New York, United States
    For correspondence
    xjwang@nyu.edu
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3124-8474

Funding

Office of Naval Research (N00014-13-1-0297)

  • H Francis Song
  • Guangyu R Yang
  • Xiao-Jing Wang

Google

  • H Francis Song
  • Guangyu R Yang
  • Xiao-Jing Wang

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Reviewing Editor

  1. Timothy EJ Behrens, University College London, United Kingdom

Version history

  1. Received: September 13, 2016
  2. Accepted: January 12, 2017
  3. Accepted Manuscript published: January 13, 2017 (version 1)
  4. Version of Record published: February 6, 2017 (version 2)

Copyright

© 2017, Song et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 10,528
    views
  • 1,878
    downloads
  • 117
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. H Francis Song
  2. Guangyu R Yang
  3. Xiao-Jing Wang
(2017)
Reward-based training of recurrent neural networks for cognitive and value-based tasks
eLife 6:e21492.
https://doi.org/10.7554/eLife.21492

Share this article

https://doi.org/10.7554/eLife.21492

Further reading

    1. Neuroscience
    Sandra P Cárdenas-García, Sundas Ijaz, Alberto E Pereda
    Research Article

    Most nervous systems combine both transmitter-mediated and direct cell-cell communication, known as 'chemical' and 'electrical' synapses, respectively. Chemical synapses can be identified by their multiple structural components. Electrical synapses are, on the other hand, generally defined by the presence of a 'gap junction' (a cluster of intercellular channels) between two neuronal processes. However, while gap junctions provide the communicating mechanism, it is unknown whether electrical transmission requires the contribution of additional cellular structures. We investigated this question at identifiable single synaptic contacts on the zebrafish Mauthner cells, at which gap junctions coexist with specializations for neurotransmitter release and where the contact unequivocally defines the anatomical limits of a synapse. Expansion microscopy of these single contacts revealed a detailed map of the incidence and spatial distribution of proteins pertaining to various synaptic structures. Multiple gap junctions of variable size were identified by the presence of their molecular components. Remarkably, most of the synaptic contact's surface was occupied by interleaving gap junctions and components of adherens junctions, suggesting a close functional association between these two structures. In contrast, glutamate receptors were confined to small peripheral portions of the contact, indicating that most of the synaptic area functions as an electrical synapse. Thus, our results revealed the overarching organization of an electrical synapse that operates with not one, but multiple gap junctions, in close association with structural and signaling molecules known to be components of adherens junctions. The relationship between these intercellular structures will aid in establishing the boundaries of electrical synapses found throughout animal connectomes and provide insight into the structural organization and functional diversity of electrical synapses.

    1. Neuroscience
    Alexandra L Jellinger, Rebecca L Suthard ... Steve Ramirez
    Research Article

    Negative memories engage a brain and body-wide stress response in humans that can alter cognition and behavior. Prolonged stress responses induce maladaptive cellular, circuit, and systems-level changes that can lead to pathological brain states and corresponding disorders in which mood and memory are affected. However, it is unclear if repeated activation of cells processing negative memories induces similar phenotypes in mice. In this study, we used an activity-dependent tagging method to access neuronal ensembles and assess their molecular characteristics. Sequencing memory engrams in mice revealed that positive (male-to-female exposure) and negative (foot shock) cells upregulated genes linked to anti- and pro-inflammatory responses, respectively. To investigate the impact of persistent activation of negative engrams, we chemogenetically activated them in the ventral hippocampus over 3 months and conducted anxiety and memory-related tests. Negative engram activation increased anxiety behaviors in both 6- and 14-month-old mice, reduced spatial working memory in older mice, impaired fear extinction in younger mice, and heightened fear generalization in both age groups. Immunohistochemistry revealed changes in microglial and astrocytic structure and number in the hippocampus. In summary, repeated activation of negative memories induces lasting cellular and behavioral abnormalities in mice, offering insights into the negative effects of chronic negative thinking-like behaviors on human health.