Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior

  1. Iris IA Groen  Is a corresponding author
  2. Michelle R Greene
  3. Christopher Baldassano
  4. Li Fei-Fei
  5. Diane M Beck
  6. Chris I Baker
  1. National Institutes of Health, United States
  2. Bates College, United States
  3. Princeton University, United States
  4. Stanford University, United States
  5. University of Illinois at Urbana-Champaign, United States

Abstract

Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid- and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.

Article and author information

Author details

  1. Iris IA Groen

    Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, United States
    For correspondence
    iris.groen@nih.gov
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-5536-6128
  2. Michelle R Greene

    Neuroscience Program, Bates College, Lewiston, United States
    Competing interests
    The authors declare that no competing interests exist.
  3. Christopher Baldassano

    Princeton Neuroscience Institute, Princeton University, Princeton, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3540-5019
  4. Li Fei-Fei

    Stanford Vision Lab, Stanford University, Stanford, United States
    Competing interests
    The authors declare that no competing interests exist.
  5. Diane M Beck

    Department of Psychology, University of Illinois at Urbana-Champaign, Urbana-Champaign, United States
    Competing interests
    The authors declare that no competing interests exist.
  6. Chris I Baker

    Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-6861-8964

Funding

National Institutes of Health (ZIAMH002909)

  • Iris IA Groen
  • Chris I Baker

Netherlands Organization for Scientific Research (Rubicon Fellowship)

  • Iris IA Groen

Office of Naval Research (Multidisciplinary Research Initiative Grant N000141410671)

  • Li Fei-Fei
  • Diane M Beck

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Human subjects: All participants had normal or corrected-to-normal vision and gave written informed consent as part of the study protocol (93 M-0170, NCT00001360) prior to participation in the study. The study was approved by the Institutional Review Board of the National Institutes of Health and was conducted according to the Declaration of Helsinki.

Copyright

This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Metrics

  • 6,010
    views
  • 766
    downloads
  • 134
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Iris IA Groen
  2. Michelle R Greene
  3. Christopher Baldassano
  4. Li Fei-Fei
  5. Diane M Beck
  6. Chris I Baker
(2018)
Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior
eLife 7:e32962.
https://doi.org/10.7554/eLife.32962

Share this article

https://doi.org/10.7554/eLife.32962

Further reading

    1. Neuroscience
    Marie-Eve Picard, Miriam Kunz ... Pierre Rainville
    Short Report

    Pain is a private experience observable through various verbal and non-verbal behavioural manifestations, each of which may relate to different pain-related functions. Despite the importance of understanding the cerebral mechanisms underlying those manifestations, there is currently limited knowledge of the neural correlates of the facial expression of pain. In this functional magnetic resonance imaging (fMRI) study, noxious heat stimulation was applied in healthy volunteers and we tested if previously published brain signatures of pain were sensitive to pain expression. We then applied a multivariate pattern analysis to the fMRI data to predict the facial expression of pain. Results revealed the inability of previously developed pain neurosignatures to predict the facial expression of pain. We thus propose a facial expression of pain signature (FEPS) conveying distinctive information about the brain response to nociceptive stimulations with minimal or no overlap with other pain-relevant brain signatures associated with nociception, pain ratings, thermal pain aversiveness, or pain valuation. The FEPS may provide a distinctive functional characterization of the distributed cerebral response to nociceptive pain associated with the socio-communicative role of non-verbal pain expression. This underscores the complexity of pain phenomenology by reinforcing the view that neurosignatures conceived as biomarkers must be interpreted in relation to the specific pain manifestation(s) predicted and their underlying function(s). Future studies should explore other pain-relevant manifestations and assess the specificity of the FEPS against simulated pain expressions and other types of aversive or emotional states.

    1. Neuroscience
    Shan Jiang, Lu Chen ... Chang-Rui Chen
    Research Article

    General anesthesia (GA) is an indispensable procedure necessary for safely and compassionately administering a significant number of surgical procedures and invasive diagnostic tests. However, the undesired stress response associated with GA causes delayed recovery and even increased morbidity in the clinic. Here, a core hypothalamic ensemble, corticotropin-releasing hormone neurons in the paraventricular nucleus of the hypothalamus (PVHCRH neurons), is discovered to play a role in regulating sevoflurane GA. Chemogenetic activation of these neurons delay the induction of and accelerated emergence from sevoflurane GA, whereas chemogenetic inhibition of PVHCRH neurons accelerates induction and delays awakening. Moreover, optogenetic stimulation of PVHCRH neurons induce rapid cortical activation during both the steady and deep sevoflurane GA state with burst-suppression oscillations. Interestingly, chemogenetic inhibition of PVHCRH neurons relieve the sevoflurane GA-elicited stress response (e.g., excessive self-grooming and elevated corticosterone level). These findings identify PVHCRH neurons modulate states of anesthesia in sevoflurane GA, being a part of anesthesia regulatory network of sevoflurane.