1. Neuroscience
Download icon

Integrative and distinctive coding of visual and conceptual object features in the ventral visual stream

  1. Chris B Martin  Is a corresponding author
  2. Danielle Douglas
  3. Rachel N Newsome
  4. Louisa LY Man
  5. Morgan D Barense  Is a corresponding author
  1. University of Toronto, Canada
  2. Mount Allison University, Canada
  3. Baycrest, Canada
  4. Queen's University, Canada
Research Article
Cite as: eLife 2018;7:e31873 doi: 10.7554/eLife.31873
11 figures, 1 table and 1 additional file

Figures

Behavior-based RDMs.

(A) Visual similarity rating task (top) and corresponding 40 × 40 behavior-based visual RDM (bottom). (B) Conceptual feature generation task (top), abridged feature matrix depicting the feature frequencies across participants for each concept (middle), and corresponding 40 × 40 behavior-based conceptual RDM (bottom). The dashed horizontal arrow between behavior-based RDMs denotes a second-level RSA that compared these similarity models with one another. All object concepts are listed in Figure 1—source data 1 - Object concepts and targeted pairs. Behavior-based RDMs (together with the word2vec RDM) are contained in Figure 1—source data 2 - Behavior-based RDMs and word2vec RDM.

https://doi.org/10.7554/eLife.31873.003
Figure 1—source data 1

Object concepts and targeted pairs.

https://doi.org/10.7554/eLife.31873.004
Figure 1—source data 2

Behavior-based RDMs and word2vec RDM.

https://doi.org/10.7554/eLife.31873.005
Brain-based RDMs.

(A) Example of object-evoked neural activity patterns obtained across all eight probes in the visual task context (top), mean object-specific activity patterns averaged across repetitions (middle), and corresponding 40 × 40 brain-based visual task RDM derived from a first-level RSA (bottom). (B) Example of object-evoked neural activity patterns obtained across all eight probes in the conceptual task context (top), mean object-specific activity patterns averaged across repetitions (middle), and corresponding 40 × 40 brain-based conceptual task RDM derived from a first-level RSA (bottom).

https://doi.org/10.7554/eLife.31873.006
fMRI feature verification task performance.

Percentage of trials on which all participants (i.e. 16/16) provided the same ‘yes/no’ response for each property verification probe.

https://doi.org/10.7554/eLife.31873.007
ROIs in a representative participant.

Cortical regions examined in the ROI-based RSAs, including lateral occipital cortex (green), parahippocampal cortex (pink), perirhinal cortex (purple), and the temporal pole (cyan).

https://doi.org/10.7554/eLife.31873.008
Second-level RSAs.

Solid vertical and diagonal arrows reflect second-level RSA in which behavior-based RDMs were compared with brain-based RDMs (ROI-based results in Figure 6, searchlight-based results in Figures 9, 10 and 11). The dashed horizontal arrow between brain-based RDMs reflects second-level RSA in which neural pattern similarities from each task context were directly compared with each other (results in Figure 7).

https://doi.org/10.7554/eLife.31873.009
Figure 6 with 1 supplement
Comparison of behavior-based and brain-based RDMs.

Similarities between behavior-based and brain-based RDMs are plotted for (A) LOC, (B) parahippocampal cortex, (C) PRC, and (D) the temporal pole. These comparisons are denoted by the solid vertical and diagonal arrows in Figure 5. Similarity was quantified as the ranked correlation coefficient (Kendall’s tau-a) between behavior-based RDMs and the brain-based RDMs. Error bars indicate standard error of the mean. ***p<0.001, **p<0.01, *p<0.05 (Bonferroni corrected). Participant-specific Kendall’s tau-a co-efficients are contained in Figure 6—source data 1 - Comparison of similarity models and brain-based RDMs.

https://doi.org/10.7554/eLife.31873.010
Figure 6—source data 1

Comparison of similarity models and brain-based RDMs.

https://doi.org/10.7554/eLife.31873.012
Figure 6—figure supplement 1
Comparison of word2vec RDM with brain-based RDMs.

Similarities between the word2vec RDM and brain-based RDMs are plotted for (A) LOC, (B) parahippocampal cortex, (C) PRC, and (D) the temporal pole. Similarity was quantified as the ranked correlation coefficient (Kendall’s tau-a) between behavior-based RDMs and the brain-based RDMs. Error bars indicate standard error of the mean. **p<0.01, *p<0.05 (Bonferroni corrected). Participant-specific Kendall’s tau-a co-efficients are contained in Figure 6—source data 1 - Comparison of similarity models and brain-based RDMs.

https://doi.org/10.7554/eLife.31873.011
Comparison of brain-based RDMs.

(A) Similarities between brain-based visual task RDMs and brain-based conceptual task RDMs within lateral occipital cortex (LOC), parahippocampal cortex (PHC), perirhinal cortex (PRC), and the temporal pole (TP). These comparisons are denoted by the dashed horizontal arrow in the bottom of Figure 5. (B) Similarities between brain-based visual task RDMs across different ROIs. Labels on the x-axis denote the ROIs being compared. (C) Similarities between brain-based conceptual task RDMs across different ROIs. Similarity was quantified as the ranked correlation coefficient (Kendall’s tau-a) between behavior-based RDMs and the brain-based RDMs. Error bars indicate standard error of the mean. ***p<0.001 (Bonferroni corrected),~p < 0.05 (uncorrected). Participant-specific Kendall’s tau-a co-efficients are contained in Figure 7—source data 1 - Comparison of brain-based RDMs.

https://doi.org/10.7554/eLife.31873.013
Figure 7—source data 1

Comparison of brain-based RDMs.

https://doi.org/10.7554/eLife.31873.014
Comparison of within-object multi-voxel activity patterns across different task contexts.

(A) Depiction of first-level RSA procedure for quantifying within-object multi-voxel activity patterns across the visual and conceptual task contexts. (B) Mean similarities between within-object multi-voxel activity patterns across different task contexts within each region of interest. Similarity was quantified as the linear correlation coefficient (Pearson’s r) between object-evoked multi-voxel activity patterns. Lateral occipital cortex (LOC), parahippocampal cortex (PHC), perirhinal cortex (PRC), and the temporal pole (TP). Error bars indicate standard error of the mean. **p<0.01, *p<0.05 (Bonferroni corrected). Participant-specific Pearson’s r co-efficients are contained in Figure 8—source data 1 - Comparison of within-object similarity across task contexts.

https://doi.org/10.7554/eLife.31873.015
Figure 8—source data 1

Comparison of within-object similarity across task contexts.

https://doi.org/10.7554/eLife.31873.016
Visual task context representational similarity searchlight mapping results.

(A) Cortical regions in which the brain-based visual task RDMs were significantly correlated with the behavior-based visual RDM. (B) Cortical regions in which the brain-based visual task RDMs were significantly correlated with the behavior-based conceptual RDM. The correlation coefficients (Kendall’s tau-a) obtained between behavior-based RDMs and brain-based RDMs were Fisher-z transformed and mapped to the voxel at the centre of each searchlight. Similarity maps were corrected for multiple comparisons using threshold-free cluster enhancement with a corrected statistical threshold of p<0.05 on the cluster level (Smith and Nichols, 2009). Outlines are shown for the lateral occipital cortex (green), parahippocampal cortex (pink), perirhinal cortex (purple), and the temporal pole (cyan).

https://doi.org/10.7554/eLife.31873.017
Conceptual task context representational similarity searchlight mapping results.

(A) Cortical regions in which the brain-based conceptual task RDMs were significantly correlated with the behavior-based visual RDM. (B) Cortical regions in which the brain-based conceptual task RDMs were significantly correlated with the behavior-based conceptual RDM. The correlation coefficients (Kendall’s tau-a) obtained between behavior-based RDMs and brain-based RDMs were Fisher-z transformed and mapped to the voxel at the centre of each searchlight. Similarity maps were corrected for multiple comparisons using threshold-free cluster enhancement with a corrected statistical threshold of p<0.05 on the cluster level (Smith and Nichols, 2009). Outlines are shown for the lateral occipital cortex (green), parahippocampal cortex (pink), perirhinal cortex (purple), and the temporal pole (cyan).

https://doi.org/10.7554/eLife.31873.018
Overlap of searchlight similarity maps.

(A) Overlap between similarity maps obtained in the visual task context (i.e. overlapping voxels from Figure 9A and B). (B) Overlap between similarity maps obtained in the conceptual task context (i.e. overlapping voxels from Figure 10A and B). (C) Overlap across brain-behavior similarity maps across both task contexts (i.e. overlapping voxels from Figures 9A, B, 10A and B). Outlines are shown for the lateral occipital cortex (green), parahippocampal cortex (pink), perirhinal cortex (purple), and the temporal pole (cyan).

https://doi.org/10.7554/eLife.31873.020

Tables

Table 1
Clusters in which behavior-based RDMs were significantly correlated with brain-based RDMs as revealed using representational similarity searchlight analyses, with corresponding cluster extent, peak z-values, and MNI co-ordinates1.
https://doi.org/10.7554/eLife.31873.019
RegionCluster extentPeak z-valueXYZ
Visual task context
Behavior-Based Visual RDM – Brain-Based Visual Task RDM
Mid calcarine16605.79-2−7412
R lateral occipital cortex4553.8950−664
R perirhinal cortex1123.6434−12−34
L superior parietal lobule1103.21−32−4044
L perirhinal cortex762.85−30−12−36
R superior parietal lobule482.6438−5454
R fusiform gyrus452.7740−46−20
R precuneus292.6612−7648
R Inferior Temporal Gyrus92.5244−22−28
Behavior-Based Conceptual RDM – Brain-Based Visual Task RDM
L Perirhinal Cortex3683.96−242−38
R Perirhinal Cortex2323.26222−36
Overlap
L Perirhinal Cortex22−30-8−38
Conceptual task context
Behavior-Based Conceptual RDM – Brain-Based Conceptual Task RDM
L Perirhinal Cortex792.88−30−10−34
R Parahippocampal Cortex642.9430−24−24
L Temporal Pole612.89−344−26
R Temporal Pole252.702412−36
Behavior-Based Visual RDM – Brain-Based Conceptual Task RDM
L Perirhinal Cortex984.87−26-4−10
R Perirhinal Cortex263.0128−12−34
Overlap
L Perirhinal Cortex31−26-8−42
Overlap across all Behavior-Based RDMs and Brain-Based RDMs
L Perirhinal Cortex16−30-8−36
  1. 1MNI co-ordinates are reported for the peak voxel in individual clusters and the centre of mass for cluster overlap.

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)