The transformation of sensory to perceptual braille letter representations in the visually deprived brain
- Department of Education and Psychology, Freie Universität Berlin, Germany
- Berlin School of Mind and Brain, Faculty of Philosophy, Humboldt-Universität zu Berlin, Germany
- Smith-Kettlewell Eye Research Institute, United States
- Bernstein Center for Computational Neuroscience Berlin, Germany
Figures
Figure 1 with 4 supplements
Stimuli, experimental design, pattern extraction, and multivariate analysis framework.
(A) We presented eight braille letters (B, C, D, L, M, N, V, and Z) to participants on braille cells. Two additional letters (E and O) served as catch trials and were excluded from all analyses. (B) Top: During the fMRI session, braille letters were presented for 500 ms with an inter-stimulus interval (ISI) of 2500 ms. Participants were instructed to respond to catch trials by pressing a button with their foot. Bottom: During a separate EEG session, braille letters were presented for 500 ms. The ISI following regular trials lasted 500 ms, the ISI following catch trials lasted 1100 ms to avoid movement confounds. Participants were instructed to respond to catch trials by pressing a foot pedal. (C) In fMRI, we extracted voxel-wise activations for every region of interest (ROI). In EEG, we extracted channel-wise activations for every time point. In both cases, this resulted in one response vector per letter and per experimental run. (D) For both fMRI and EEG, we divided pattern vectors into training (four pseudo-runs) and test (one pseudo-run) sets. For every pair of braille letters (e.g., B and V), we trained a support vector machine (SVM) to classify between pattern vectors related to the presentation of both letters read with the same hand. We then tested the SVM on the left-out pattern vectors related to the presentation of the same two letters read with the same hand (within-hand classification) or with the other hand (across-hand classification). The resulting pairwise decoding accuracies were aggregated in a decoding accuracy matrix that is symmetric along the diagonal, with the diagonal itself being undefined. We interpret the within-hand matrix (black) as a measure of sensory and perceptual braille letter representations. We interpret the across-hand matrix (green) as a measure of perceptual braille letter representations. We derive the measure of sensory braille letter representations (blue) by subtracting one matrix from the other.
Figure 1—figure supplement 1
Confusion matrices for region of interest (ROI) decoding within-hand.
Values represent pairwise decoding accuracies minus chance level.
Figure 1—figure supplement 2
Confusion matrices for region of interest (ROI) decoding across-hand.
Values represent pairwise decoding accuracies minus chance level.
Figure 1—figure supplement 3
Confusion matrices for EEG time decoding at indicated time points within-hand.
Values represent pairwise decoding accuracies minus chance level.
Figure 1—figure supplement 4
Confusion matrices for EEG time decoding at indicated time points across-hand.
Values represent pairwise decoding accuracies minus chance level.
Figure 2 with 1 supplement
Spatial dynamics of braille letter representations.
(A) Rendering of regions of interests associated with tactile processing (S1: primary somatosensory cortex, S2: secondary somatosensory cortex, aIPS: anterior intra-parietal sulcus, pIPS: posterior intra-parietal sulcus, insula) and sighted reading (EVC: early visual cortex, V4: visual area 4, LOC: lateral occipital complex, LFA: letter form area, VWFA: visual word form area). (B) Sensory and perceptual (left), sensory (middle), and perceptual (right) braille letters representations in tactile processing and sighted reading regions of interest (ROIs) N = 15, two-tailed Wilcoxon signed-rank test, p < 0.05, false discovery rate (FDR) corrected; stars below bars indicate significance above chance. Error bars represent 95% confidence intervals. Dots represent single subject data. (C) fMRI searchlight results for sensory (blue) and perceptual (green) braille letter representations (N = 15, height threshold p < 0.001, cluster-level family-wise error (FWE) corrected p < 0.05, colored voxels indicate significance). Results for combined sensory and perceptual representations are in Figure 2—figure supplement 1.
Figure 2—figure supplement 1
fMRI searchlight results for within-hand braille letter classification.
Results index mixed sensory and perceptual braille letter representations (N = 15, height threshold p < 0.001, cluster-level family-wise error (FWE) corrected p < 0.05, colored voxels indicate significance).
Figure 3 with 3 supplements
Temporal dynamics of braille letter representations.
(A) EEG results for sensory and perceptual (black), sensory (blue), and perceptual (green) braille letter representations in time. Shaded areas around curves indicate standard error of the mean. Significance onsets and their 95% confidence intervals are indicated by dots and horizontal lines below curves (color-coded as curves, N = 11, 1000 bootstraps). Significant time points are indicated by x below curves (N = 11, one-tailed Wilcoxon signed-rank test, p < 0.05, false discovery rate [FDR] corrected). (B) EEG searchlight results for sensory and perceptual (top), sensory (middle), and perceptual (bottom) braille letter representations in EEG channel space (sampled down to 10 ms resolution) in 100-ms intervals. Significant electrodes are marked with black circles (N = 11, one-tailed Wilcoxon signed-rank test, p <0 .05, FDR corrected across electrodes and time points).
Figure 3—figure supplement 1
EEG time decoding results with empirical baseline (see Figure 3A for comparison).
For within- and across-hand decoding, we subtracted the chance level (50%) from the data. Shaded areas around the curves indicate standard error of the mean. Significance onset and its 95% confidence interval are indicated by a dot and horizontal line below curve (N = 11, 1000 bootstraps). Significant time points are indicated by x below curves (N = 11, sign permutation test, 10,000 permutations, p < 0.05, false discovery rate [FDR] corrected).
Figure 3—figure supplement 2
EEG results for decoding of stimulated hand in time.
The shaded area around the curve indicates standard error of the mean. Significance onset and its 95% confidence interval are indicated by a dot and horizontal line below curve (N = 11, 1000 bootstraps, one-tailed Wilcoxon signed-rank test, p < 0.05, false discovery rate [FDR] corrected).
Figure 3—figure supplement 3
EEG time decoding results when only using occipital and parieto-occipital electrodes.
Analysis is limited to eight electrodes instead of all electrodes (n = 63, see Figure 3A for comparison). For within- and across-hand decoding, we subtracted the chance level (50%) from the data. The area around curve indicates standard error of the mean. Significance onset and its 95% confidence interval are indicated by a dot and horizontal line below curve (N = 11, 1000 bootstraps). Significant time points are indicated by x below curves (N = 11, one-tailed Wilcoxon signed-rank test, p < 0.05, false discovery rate [FDR] corrected).
Figure 4 with 2 supplements
Representational similarity of braille letters in neural and behavioral measures.
(A) For fMRI (top) and EEG (bottom), we formed representational dissimilarity matrices (RDMs) from pairwise decoding accuracies. In the behavioral experiment (middle), we presented two braille letters (e.g., B and V) on two braille cells. For all combinations, we asked participants to read both braille letters with the same hand (e.g., right) and verbally rate their similarity on a scale from 1 to 7. We formed the behavioral RDM from those perceptual similarity judgements for every letter pair. We then correlated the behavioral RDM (averaged over participants) with neural RDMs (subject specific) using Spearman’s R. (B) Results of representational similarity analysis (RSA) relating fMRI and behavior in regions of interest (ROIs) that showed significant sensory (left) and perceptual (right) braille letter representations in fMRI (see Figure 2) (N = 15, two-tailed Wilcoxon signed-rank test, p < 0.05, false discovery rate [FDR] corrected). Stars below bars indicate significance above chance. Error bars represent 95% confidence intervals. Dots represent single subject data. (C) Results of RSA relating EEG and behavior for sensory (blue) and perceptual (green) representations. Shaded areas around curves indicate standard error of the mean. Significance onsets and their 95% confidence intervals are indicated by dots and lines below curves (N = 11, 1000 bootstraps, one-tailed Wilcoxon signed-rank test, p < 0.05, FDR corrected, color-coded as result curves).
Figure 4—figure supplement 1
Representational similarity of braille letters in neural and behavioral measures using Jaccard similarity.
(A) Results of representational similarity analysis (RSA) relating fMRI and behavior in regions of interest (ROIs) that showed significant sensory (left) and perceptual (right) braille letter representations in fMRI (see Figure 2) (N = 15, two-tailed Wilcoxon signed-rank test, p < 0.05, false discovery rate [FDR] corrected). Error bars represent 95% confidence intervals. Dots represent single subject data. (B) Results of RSA relating EEG and behavior for sensory (blue) and perceptual (green) representations. Shaded areas around the curves indicate standard error of the mean. Significance onsets and their 95% confidence intervals are indicated by dots and lines below curves (N = 11, 1000 bootstraps, one-tailed Wilcoxon signed-rank test, p < 0.05, FDR corrected, color-coded as result curves).
Figure 4—figure supplement 2
EEG-behavior representational similarity analysis (RSA) decoding results when only using occipital and parieto-occipital electrodes.
Analysis is limited to eight electrodes instead of all electrodes (n = 63, see Figure 4C for comparison). Shaded areas around the curves indicate standard error of the mean. Significance onset and its 95% confidence interval are indicated by a dot and horizontal line below curve (N = 11, 1000 bootstraps, one-tailed Wilcoxon signed-rank test, p < 0.05, false discovery rate [FDR] corrected).
Additional files
-
Supplementary file 1
Participant information.
- https://cdn.elifesciences.org/articles/98148/elife-98148-supp1-v1.docx
-
MDAR checklist
- https://cdn.elifesciences.org/articles/98148/elife-98148-mdarchecklist1-v1.pdf
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
The transformation of sensory to perceptual braille letter representations in the visually deprived brain
eLife 13:RP98148.
https://doi.org/10.7554/eLife.98148.3
