Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior

Abstract
Article and author information
Metrics

Abstract

Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid- and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.

Article and author information

Author details

Iris IA Groen

Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, United States

For correspondence
iris.groen@nih.gov

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0002-5536-6128
Michelle R Greene

Neuroscience Program, Bates College, Lewiston, United States

Competing interests
The authors declare that no competing interests exist.
Christopher Baldassano

Princeton Neuroscience Institute, Princeton University, Princeton, United States

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0003-3540-5019
Li Fei-Fei

Stanford Vision Lab, Stanford University, Stanford, United States

Competing interests
The authors declare that no competing interests exist.
Diane M Beck

Department of Psychology, University of Illinois at Urbana-Champaign, Urbana-Champaign, United States

Competing interests
The authors declare that no competing interests exist.
Chris I Baker

Laboratory of Brain and Cognition, National Institutes of Health, Bethesda, United States

Competing interests
The authors declare that no competing interests exist.

"This ORCID iD identifies the author of this article:" 0000-0001-6861-8964

Funding

National Institutes of Health (ZIAMH002909)

Iris IA Groen
Chris I Baker

Netherlands Organization for Scientific Research (Rubicon Fellowship)

Iris IA Groen

Office of Naval Research (Multidisciplinary Research Initiative Grant N000141410671)

Li Fei-Fei
Diane M Beck

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Ethics

Human subjects: All participants had normal or corrected-to-normal vision and gave written informed consent as part of the study protocol (93 M-0170, NCT00001360) prior to participation in the study. The study was approved by the Institutional Review Board of the National Institutes of Health and was conducted according to the Declaration of Helsinki.

Copyright

This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.