Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior
Abstract
Inherent correlations between visual and semantic features in real-world scenes make it difficult to determine how different scene properties contribute to neural representations. Here, we assessed the contributions of multiple properties to scene representation by partitioning the variance explained in human behavioral and brain measurements by three feature models whose inter-correlations were minimized a priori through stimulus preselection. Behavioral assessments of scene similarity reflected unique contributions from a functional feature model indicating potential actions in scenes as well as high-level visual features from a deep neural network (DNN). In contrast, similarity of cortical responses in scene-selective areas was uniquely explained by mid- and high-level DNN features only, while an object label model did not contribute uniquely to either domain. The striking dissociation between functional and DNN features in their contribution to behavioral and brain representations of scenes indicates that scene-selective cortex represents only a subset of behaviorally relevant scene information.
Article and author information
Author details
Funding
National Institutes of Health (ZIAMH002909)
- Iris IA Groen
- Chris I Baker
Netherlands Organization for Scientific Research (Rubicon Fellowship)
- Iris IA Groen
Office of Naval Research (Multidisciplinary Research Initiative Grant N000141410671)
- Li Fei-Fei
- Diane M Beck
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Ethics
Human subjects: All participants had normal or corrected-to-normal vision and gave written informed consent as part of the study protocol (93 M-0170, NCT00001360) prior to participation in the study. The study was approved by the Institutional Review Board of the National Institutes of Health and was conducted according to the Declaration of Helsinki.
Copyright
This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Metrics
-
- 6,226
- views
-
- 785
- downloads
-
- 137
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading
-
- Neuroscience
Studying infant minds with movies is a promising way to increase engagement relative to traditional tasks. However, the spatial specificity and functional significance of movie-evoked activity in infants remains unclear. Here, we investigated what movies can reveal about the organization of the infant visual system. We collected fMRI data from 15 awake infants and toddlers aged 5–23 months who attentively watched a movie. The activity evoked by the movie reflected the functional profile of visual areas. Namely, homotopic areas from the two hemispheres responded similarly to the movie, whereas distinct areas responded dissimilarly, especially across dorsal and ventral visual cortex. Moreover, visual maps that typically require time-intensive and complicated retinotopic mapping could be predicted, albeit imprecisely, from movie-evoked activity in both data-driven analyses (i.e. independent component analysis) at the individual level and by using functional alignment into a common low-dimensional embedding to generalize across participants. These results suggest that the infant visual system is already structured to process dynamic, naturalistic information and that fine-grained cortical organization can be discovered from movie data.
-
- Neuroscience
Mesolimbic dopamine encoding of non-contingent rewards and reward-predictive cues has been well established. Considerable debate remains over how mesolimbic dopamine responds to aversion and in the context of aversive conditioning. Inconsistencies may arise from the use of aversive stimuli that are transduced along different neural paths relative to reward or the conflation of responses to avoidance and aversion. Here, we made intraoral infusions of sucrose and measured how dopamine and behavioral responses varied to the changing valence of sucrose. Pairing intraoral sucrose with malaise via injection of lithium chloride (LiCl) caused the development of a conditioned taste aversion (CTA), which rendered the typically rewarding taste of sucrose aversive upon subsequent re-exposure. Following CTA formation, intraoral sucrose suppressed the activity of ventral tegmental area dopamine neurons (VTADA) and nucleus accumbens (NAc) dopamine release. This pattern of dopamine signaling after CTA is similar to intraoral infusions of innately aversive quinine and contrasts with responses to sucrose when it was novel or not paired with LiCl. Dopamine responses were negatively correlated with behavioral reactivity to intraoral sucrose and predicted home cage sucrose preference. Further, dopamine responses scaled with the strength of the CTA, which was increased by repeated LiCl pairings and weakened through extinction. Thus, the findings demonstrate differential dopamine encoding of the same taste stimulus according to its valence, which is aligned to distinct behavioral responses.