Distinct contributions of functional and deep neural network features to representational similarity of scenes in human brain and behavior
- National Institutes of Health, United States
- New York University, United States
- Bates College, United States
- Princeton University, United States
- Stanford University, United States
- University of Illinois, United States
Figures
Figure 1
Models and predicted stimulus dissimilarity.
(A) Stimuli were characterized in three different ways: functions (derived using human-generated action labels), objects (derived using human-generated object labels) and DNN features (derived using …
-
Figure 1—source data 1
- https://doi.org/10.7554/eLife.32962.003
Figure 2
Behavioral multi-arrangement paradigm and results.
(A) Participants organized the scenes inside a large white circle according to their perceived similarity as determined by their own judgment, without receiving instructions as to what information …
Figure 3
RDMs and model comparisons for fMRI Experiment 1 (n = 20).
(A) RDMs displaying average dissimilarity between categories in multi-voxel patterns in PPA, OPA and MPA, ordered as in Figure 1B (rank-ordered for visualization only). (B) Average (bar) and …
Figure 4
Correlations and variance partitioning of behavioral measurements of scene categorization and similarity of fMRI responses.
(A) Correlations of three measures of behavioral categorization (see Results section for details) with fMRI response patterns in PPA, OPA and MPA. See legend of Figure 2B for explanation of the …
Figure 5
RDMs and model comparisons for Experiment 2 (n = 8, covert naming task).
(A) Average dissimilarity between categories in multi-voxel patterns measured in PPA, OPA and MPA (rank-ordered as in Figure 1B). (B) Correlations between the ROIs in A) and the model RDMs for …
Figure 6
Medial (left) and lateral (right) views of group-level searchlights for (A) the DNN and (B) function model, overlaid on surface reconstructions of both hemispheres of one participant.
Each map was created by submitting the partial correlation maps for each model and hemisphere to one-sample tests against a mean of zero, cluster-corrected for multiple comparisons using …
Figure 7
Multi-arrangement behavior searchlights and post-hoc analysis of functional clusters.
(A) Searchlight result for behavioral scene categorization. Maps reflect correlation (Pearson’s r) of the group-average behavior in the multi-arrangement task from the participants of Experiment 1. …
Figure 8 with 2 supplements
DNN layer and DNN training comparisons in terms of correlation with fMRI responses in scene-selective cortex.
Panels show convolutional and fully-connected (FC) layer-by-layer RDM correlations between (A) an object-trained (ReferenceNet) and a scene-trained (Places) DNN; (B) both DNNs and the a priori …
Figure 8—video 1
Layer-by-layer searchlight results for the object-trained DNN (ReferenceNet).
The first half of the movie shows group-average correlation maps for layer 1–8, cluster-corrected for multiple comparisons using Threshold-Free Cluster Enhancement (thresholded on z = 1.64, …
Figure 8—video 2
Layer-by-layer searchlight results for the scene-trained DNN (Places).
See legend of Figure 8—video 1 for details.
Additional files
-
Transparent reporting form
- https://doi.org/10.7554/eLife.32962.013
