DeepRetinotopy toolbox: an application for predicting retinotopic organization from brain structure.

a, DeepRetinotopy toolbox integrates standard neuroimaging software for anatomical MRI data preprocessing (left) and pre-trained deep learning models (middle) for predicting retinotopic maps at the individual level (right). b, The prediction pipeline requires FreeSurfer-reconstructed cortical meshes derived from a T1w structural MRI image as input. Note that our toolbox can perform both the preprocessing and prediction steps, or users can provide preprocessed data directly. The prediction pipeline consists of registering curvature maps from the native space to a template space (32k_fs_LR). These registered curvature maps are then used as input for retinotopic map prediction using pre-trained models. Finally, predicted maps are then registered to the individual’s native space.

Benchmarking models for predicting retinotopic maps of the visual cortex from underlying anatomy.

a, Polar angle, eccentricity, and pRF size maps from left (LH) and right (RH) hemispheres are shown from a representative participant in the HCP test set (#680957). b, Bar plots represent the average correlation between predicted and empirically derived maps across all participants in the HCP test set (n = 10). Correlation score was determined as either the Pearson correlation (for eccentricity and pRF size maps) or the circular correlation (polar angle maps). Retinotopic maps were vectorized, and only vertices within early visual areas (V1-3) and above a 10% variance explained threshold were used to estimate the correlation. Error bars correspond to the 95% confidence interval. The gray shaded area represents the noise ceiling, i.e., the 95% confidence interval of the square root of the Spearman-Brown corrected variance explained between split-half pRF fits.

Cross-dataset generalizability.

a, Diagram of the main characteristics of the datasets used to assess model generalizability. From top to bottom, we show: dataset name/acronym, sample size, visual stimuli used for retinotopic mapping experiments, scanner type, and fMRI data resolution. In grey, we highlight the HCP test set and, in purple, four distinct new datasets. b, Empirically derived and predicted maps were vectorized, filtered to include only vertices from V1-3 and above a variance explained threshold of 10% of the empirical data, concatenated across participants, and represented as hexbin plots. In each plot, empirically derived parameters are shown along the y-axis and the predicted ones along the x-axis. The top row shows polar angle values, while the middle and bottom rows show the equivalent plots for eccentricity and pRF size maps. Black diagonal lines illustrate the ‘perfect’ match between empirically derived and predicted parameters.

Automated visual area segmentation.

a, Diagram shows our automated visual area segmentation pipeline, in which deepRetinotopy’s predicted retinotopic maps are used in combination with the Bayesian model of retinotopy to infer early visual area boundaries. b, Segmentation performance is shown across visual areas. Performance was estimated as the degree of overlap (Dice) between manually drawn and automatically generated early visual area labels, for which data from both hemispheres were combined. Then, each individual’s Dice scores were normalized by the corresponding mean Dice score between all pairs of manual annotations derived from four expert annotators. Error bars correspond to the 95% confidence interval.

Cortical horizontal-vertical anisotropy (HVA) and vertical-meridian asymmetry (VMA) differences between children and adults.

a, Diagram shows how the internal brain representation (middle) of the visual field (left) varies across eccentricity and polar angle. A greater surface area is devoted to central versus peripheral vision (note the expanded disc florets versus contracted ray florets in the brain representations). Moreover, more cortical surface area is dedicated to representing the horizontal meridian than the vertical meridian (HVA; represented as the difference between the combined pink and green edges, and yellow and blue edges), and more surface area is dedicated to representing the lower than the upper vertical meridian (VMA; represented as the difference between the yellow and blue edges). Cortical polar angle asymmetries also vary between adults (top) and children (bottom). Specifically, while HVA is similar between groups, children have reduced cortical VMA. b, We applied deepRetinotopy to over 11,000 brain scans and estimated both cortical VMA and HVA. Bar plots show the magnitudes of HVA (c) and VMA (d) indices for children and adults. Individual data points are also shown. Error bars correspond to ± standard error. Statistical comparisons using two-tailed independent samples t-test revealed significant group difference in cortical VMA (t(11,058) = 5.72, p < 0.001, d = 0.18) as well as cortical HVA (t(11,058) = 2.74, p < 0.01, d = 0.09).

Measures of computational efficiency in two computing environments.

Summary of relevant information from datasets used in this work.

Model benchmarking with an error measure.

Bar plots represent th average error between predicted and empirically derived maps across all participants in the HCP test set. Retinotopic maps werevectorized and only vertices within early visual areas (V1-3) and above a 10% variance explained threshold were used to estimate the error given as the smallest angular difference for polar angle maps or the absolute difference for eccentricity and pRF size maps. The gray shaded area represents the 95% confidence interval of the error floor, i.e., the difference between the retinotopic maps derived from each split half.

Cross-dataset comparison of polar angle maps of the visual cortex

Scatter plots comparing vertex-wise median parameters across participants from each test dataset as in Figure 2. Each data point represents a vertex in the fs_LR_32k surface space within V1-3. Data was aggregated across hemispheres, and we also applied a variance explained threshold of 15% based on datasets along the Y axis.

Cross-dataset comparison of eccentricity maps of the visual cortex.

Scatter plots comparing vertex-wise median parameters across participants from each test dataset as in Figure 2. Each data point represents a vertex in the fs_LR_32k surface space within V1-3. Data was aggregated across hemispheres, and we also applied a variance explained threshold of 15% based on datasets along the Y axis.

Cross-dataset comparison of pRF size maps of the visual cortex

Scatter plots comparing vertex-wise median parameters across participants from each test dataset a in Figure 2. Each data point represents a vertex in the fs_LR_32k surface space within V1-3. Data was aggregated across hemispheres, and we also applied a variance explained threshold of 15% based on datasets along the Y axis.

Automated visual area segmentation performance (non-normalized).

Performance is shown across visual areas. Performance was estimated as the degree of overlap between manually drawn and automatically generated early visual area labels, for which data from both hemispheres were combined. Error bars correspond to the 95% confidence interval. The gray shaded area represents the noise ceiling, i.e., the 95% confidence interval of the DICE scores between all pairs of manual annotations, across anatomists and participants.

Segmentation performance across visual areas per hemisphere.

Plots show the Dice score across visual areas for right (a) and left (b) hemispheres separately. The gray shaded area represents the noise ceiling, i.e., the 95% confidence interval of the Dice scores between all pairs of manual annotations, across anatomists and participants.

Polar angle asymmetries for V1 surface area.

a, Diagram shows visual field partitioning used for determining wedge-ROIs to estimate the cortical surface area dedicated to representing the left (pink) and right(green) horizontal meridians, and upper (blue) and lower (yellow) vertical meridians. Group-level V1 surface area measures from wedge-ROIs are shown for empirically derived (b) and predicted retinotopic maps (c) using the HCP 7T Retinotopy dataset (n=181), and for predicted retinotopic maps using the HCP young adult dataset (d; n = 932, excluding the individuals with retinotopic mapping data). Black data points indicate individual measurements. The top and bottom bounds of each box represent the 75th and 25th percentiles, respectively. LHM: left horizontal meridian; RHM: right horizontal meridian; UVM: upper vertical meridian; LVM: lower vertical meridian; HVA: horizontal-vertical anisotropy; VMA: vertical-meridian asymmetry.

Model performance in the development set (n = 10) across different random initializations.

We trained 5 distinct models per retinotopic map (polar angle, eccentricity, and pRF size) and hemisphere (left: LH; right: RH) using different random initializations, totaling 30 models. Owing to the large model weights’ files (∼465MB each) and slow inference speed using CPU, we made our toolbox available with a single instance model per retinotopic map and hemisphere, totaling 6 models, which minimizes the software container size and inference speed. The green rectangles highlight the selected models. Note we based our selection on both the individual variability and error scores.

Impact of weighting on automated visual area segmentation performance.

Segmentation performance is shown across visual areas, with data from both hemispheres combined. We compared two weighting approaches for the Bayesian model: the mean variance explained across all deepRetinotopy training participants versus uniform maximum weighting (ones everywhere, giving maximum weight to the predicted observations). The variance explained- based weighting achieved higher segmentation performance (V1: mean Dice score = 0.85; V2: 0.69; V3: 0.59) compared to uniform weighting (V1: 0.84; V2: 0.68; V3: 0.57). Statistical significance was assessed using two-tailed paired t-tests. The gray shaded area represents the noise ceiling, .e., the 95% confidence interval of the Dice scores between all pairs of manual annotations, averaged across anatomists and participants.

Mean correlation scores between the empirically derived and predicted maps across datasets and hemispheres.

Correlation scores were determined as the Pearson correlation for eccentricity and pRF size maps and the circular correlation for polar angle maps. LH: left hemisphere; RH: right hemisphere.