Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior

  1. Iris IA Groen  Is a corresponding author
  2. Michelle R Greene
  3. Christopher Baldassano
  4. Li Fei-Fei
  5. Diane M Beck
  6. Chris I Baker
  1. National Institutes of Health, United States
  2. Bates College, United States
  3. Princeton University, United States
  4. Stanford University, United States
  5. University of Illinois at Urbana-Champaign, United States

Abstract

Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid- and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.

Article and author information

Author details

  1. Iris IA Groen

    Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, United States
    For correspondence
    iris.groen@nih.gov
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-5536-6128
  2. Michelle R Greene

    Neuroscience Program, Bates College, Lewiston, United States
    Competing interests
    The authors declare that no competing interests exist.
  3. Christopher Baldassano

    Princeton Neuroscience Institute, Princeton University, Princeton, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-3540-5019
  4. Li Fei-Fei

    Stanford Vision Lab, Stanford University, Stanford, United States
    Competing interests
    The authors declare that no competing interests exist.
  5. Diane M Beck

    Department of Psychology, University of Illinois at Urbana-Champaign, Urbana-Champaign, United States
    Competing interests
    The authors declare that no competing interests exist.
  6. Chris I Baker

    Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, United States
    Competing interests
    The authors declare that no competing interests exist.
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-6861-8964

Funding

National Institutes of Health (ZIAMH002909)

  • Iris IA Groen
  • Chris I Baker

Netherlands Organization for Scientific Research (Rubicon Fellowship)

  • Iris IA Groen

Office of Naval Research (Multidisciplinary Research Initiative Grant N000141410671)

  • Li Fei-Fei
  • Diane M Beck

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Human subjects: All participants had normal or corrected-to-normal vision and gave written informed consent as part of the study protocol (93 M-0170, NCT00001360) prior to participation in the study. The study was approved by the Institutional Review Board of the National Institutes of Health and was conducted according to the Declaration of Helsinki.

Copyright

This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Metrics

  • 6,146
    views
  • 776
    downloads
  • 135
    citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Iris IA Groen
  2. Michelle R Greene
  3. Christopher Baldassano
  4. Li Fei-Fei
  5. Diane M Beck
  6. Chris I Baker
(2018)
Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior
eLife 7:e32962.
https://doi.org/10.7554/eLife.32962

Share this article

https://doi.org/10.7554/eLife.32962

Further reading

    1. Neuroscience
    Walter Senn, Dominik Dold ... Mihai A Petrovici
    Research Article

    One of the most fundamental laws of physics is the principle of least action. Motivated by its predictive power, we introduce a neuronal least-action principle for cortical processing of sensory streams to produce appropriate behavioral outputs in real time. The principle postulates that the voltage dynamics of cortical pyramidal neurons prospectively minimizes the local somato-dendritic mismatch error within individual neurons. For output neurons, the principle implies minimizing an instantaneous behavioral error. For deep network neurons, it implies the prospective firing to overcome integration delays and correct for possible output errors right in time. The neuron-specific errors are extracted in the apical dendrites of pyramidal neurons through a cortical microcircuit that tries to explain away the feedback from the periphery, and correct the trajectory on the fly. Any motor output is in a moving equilibrium with the sensory input and the motor feedback during the ongoing sensory-motor transform. Online synaptic plasticity reduces the somatodendritic mismatch error within each cortical neuron and performs gradient descent on the output cost at any moment in time. The neuronal least-action principle offers an axiomatic framework to derive local neuronal and synaptic laws for global real-time computation and learning in the brain.

    1. Neuroscience
    Mengqiao Cui, Xiaoyuan Pan ... Jun-Li Cao
    Research Article

    Memory impairment in chronic pain patients is substantial and common, and few therapeutic strategies are available. Chronic pain-related memory impairment has susceptible and unsusceptible features. Therefore, exploring the underlying mechanisms of its vulnerability is essential for developing effective treatments. Here, combining two spatial memory tests (Y-maze test and Morris water maze), we segregated chronic pain mice into memory impairment-susceptible and -unsusceptible subpopulations in a chronic neuropathic pain model induced by chronic constrictive injury of the sciatic nerve. RNA-Seq analysis and gain/loss-of-function study revealed that S1P/S1PR1 signaling is a determinant for vulnerability to chronic pain-related memory impairment. Knockdown of the S1PR1 in the dentate gyrus (DG) promoted a susceptible phenotype and led to structural plasticity changes of reduced excitatory synapse formation and abnormal spine morphology as observed in susceptible mice, while overexpression of the S1PR1 and pharmacological administration of S1PR1 agonist in the DG promoted an unsusceptible phenotype and prevented the occurrence of memory impairment, and rescued the morphological abnormality. Finally, the Gene Ontology (GO) enrichment analysis and biochemical evidence indicated that downregulation of S1PR1 in susceptible mice may impair DG structural plasticity via interaction with actin cytoskeleton rearrangement-related signaling pathways including Itga2 and its downstream Rac1/Cdc42 signaling and Arp2/3 cascade. These results reveal a novel mechanism and provide a promising preventive and therapeutic molecular target for vulnerability to chronic pain-related memory impairment.