1. Neuroscience

Humans perseverate on punishment avoidance goals in multigoal reinforcement learning

  1. Paul B Sharp  Is a corresponding author
  2. Evan M Russek
  3. Quentin JM Huys
  4. Raymond J Dolan
  5. Eran Eldar
  1. The Hebrew University of Jerusalem, Israel
  2. University College London, United Kingdom
https://doi.org/10.7554/eLife.74402
Share this article
Cite this article
  1. Paul B Sharp
  2. Evan M Russek
  3. Quentin JM Huys
  4. Raymond J Dolan
  5. Eran Eldar
(2022)
Humans perseverate on punishment avoidance goals in multigoal reinforcement learning
eLife 11:e74402.
https://doi.org/10.7554/eLife.74402
  2. Download BibTeX
  3. Download .RIS

Abstract

Managing multiple goals is essential to adaptation, yet we are only beginning to understand computations by which we navigate the resource-demands entailed in so doing. Here, we sought to elucidate how humans balance reward seeking and punishment avoidance goals, and relate this to variation in its expression within anxious individuals. To do so, we developed a novel multigoal pursuit task that includes trial-specific instructed goals to either pursue reward (without risk of punishment) or avoid punishment (without the opportunity for reward). We constructed a computational model of multigoal pursuit to quantify the degree to which participants could disengage from the pursuit goals when instructed to, as well as devote less model-based resources towards goals that were less abundant. In general, participants (n=192) were less flexible in avoiding punishment than in pursuing reward. Thus, when instructed to pursue reward, participants often persisted in avoiding features that had previously been associated with punishment, even though at decision time these features were unambiguously benign. In a similar vein, participants showed no significant downregulation of avoidance when punishment avoidance goals were less abundant in the task. Importantly, we show preliminary evidence that individuals with chronic worry may have difficulty disengaging from punishment avoidance when instructed to seek reward. Taken together, the findings demonstrate that people avoid punishment less flexibly than they pursue reward. Future studies should test in larger samples whether a difficulty to disengage from punishment avoidance contributes to chronic worry.

Data availability

All data are available in the main text or the supplementary materials. All code and analyses can be found at: github.com/pq1289/multigoal_RL

Article and author information

Author details

  1. Paul B Sharp

    The Hebrew University of Jerusalem, Jerusalem, Israel
    For correspondence
    paul.sharp@mail.huji.ac.il
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-4949-1501

  2. Evan M Russek

    Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.

  3. Quentin JM Huys

    Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.

  4. Raymond J Dolan

    The Wellcome Trust Centre for Neuroimaging, University College London, London, United Kingdom
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-9356-761X

  5. Eran Eldar

    The Hebrew University of Jerusalem, Jerusalem, Israel
    Competing interests
    The authors declare that no competing interests exist.

Funding

Fulbright Association (PS00318453)

  • Paul B Sharp

NIH Blueprint for Neuroscience Research (R01MH124092)

  • Eran Eldar

Wellcome Trust (098362/Z/12/Z)

  • Paul B Sharp

2Max Planck UCL Centre for Computational Psychiatry and Ageing Research (Open Access Funding)

  • Paul B Sharp

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Human subjects: Participants gave written informed consent before taking part in the study, which was approved by the university's ethics review board (project ID number 16639/001).

Copyright

© 2022, Sharp et al.

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 2,378
    views
  • 370
    downloads
  • 8
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Paul B Sharp
  2. Evan M Russek
  3. Quentin JM Huys
  4. Raymond J Dolan
  5. Eran Eldar
(2022)
Humans perseverate on punishment avoidance goals in multigoal reinforcement learning
eLife 11:e74402.
https://doi.org/10.7554/eLife.74402

Share this article

https://doi.org/10.7554/eLife.74402

Categories and tags

Research organism

Further reading

    1. Neuroscience

    The neural correlates of novelty and variability in human decision-making under an active inference framework

    Shuo Zhang, Yan Tian ... Haiyan Wu
    Research Article

    Active inference integrates perception, decision-making, and learning into a united theoretical framework, providing an efficient way to trade off exploration and exploitation by minimizing (expected) free energy. In this study, we asked how the brain represents values and uncertainties (novelty and variability), and resolves these uncertainties under the active inference framework in the exploration-exploitation trade-off. Twenty-five participants performed a contextual two-armed bandit task, with electroencephalogram (EEG) recordings. By comparing the model evidence for active inference and reinforcement learning models of choice behavior, we show that active inference better explains human decision-making under novelty and variability, which entails exploration or information seeking. The EEG sensor-level results show that the activity in the frontal, central, and parietal regions is associated with novelty, while the activity in the frontal and central brain regions is associated with variability. The EEG source-level results indicate that the expected free energy is encoded in the frontal pole and middle frontal gyrus and uncertainties are encoded in different brain regions but with overlap. Our study dissociates the expected free energy and uncertainties in active inference theory and their neural correlates, speaking to the construct validity of active inference in characterizing cognitive processes of human decisions. It provides behavioral and neural evidence of active inference in decision processes and insights into the neural mechanism of human decisions under uncertainties.

    1. Genetics and Genomics
    2. Neuroscience

    Zinc finger homeobox-3 (ZFHX3) orchestrates genome-wide daily gene expression in the suprachiasmatic nucleus

    Akanksha Bafna, Gareth Banks ... Patrick M Nolan
    Research Article

    The mammalian suprachiasmatic nucleus (SCN), situated in the ventral hypothalamus, directs daily cellular and physiological rhythms across the body. The SCN clockwork is a self-sustaining transcriptional-translational feedback loop (TTFL) that in turn coordinates the expression of clock-controlled genes (CCGs) directing circadian programmes of SCN cellular activity. In the mouse, the transcription factor, ZFHX3 (zinc finger homeobox-3), is necessary for the development of the SCN and influences circadian behaviour in the adult. The molecular mechanisms by which ZFHX3 affects the SCN at transcriptomic and genomic levels are, however, poorly defined. Here, we used chromatin immunoprecipitation sequencing to map the genomic localization of ZFHX3-binding sites in SCN chromatin. To test for function, we then conducted comprehensive RNA sequencing at six distinct times-of-day to compare the SCN transcriptional profiles of control and ZFHX3-conditional null mutants. We show that the genome-wide occupancy of ZFHX3 occurs predominantly around gene transcription start sites, co-localizing with known histone modifications, and preferentially partnering with clock transcription factors (CLOCK, BMAL1) to regulate clock gene(s) transcription. Correspondingly, we show that the conditional loss of ZFHX3 in the adult has a dramatic effect on the SCN transcriptome, including changes in the levels of transcripts encoding elements of numerous neuropeptide neurotransmitter systems while attenuating the daily oscillation of the clock TF Bmal1. Furthermore, various TTFL genes and CCGs exhibited altered circadian expression profiles, consistent with an advanced in daily behavioural rhythms under 12 h light–12 h dark conditions. Together, these findings reveal the extensive genome-wide regulation mediated by ZFHX3 in the central clock that orchestrates daily timekeeping in mammals.