Examples of inputs and outputs from the MADRC dataset. (a) 3D surface scan of left human hemisphere, acquired prior to dissection. (b) Routine dissection photography of coronal slabs, after pixel calibration, with digital rulers overlaid. (c) 3D reconstruction of the photographs into an imaging volume. (d) Sagittal cross-section of the volume in (c) with the machine learning segmentation overlaid; note that the input has low, anisotropic resolution due to the large thickness of the slices (i.e., rectangular pixels in sagittal view), whereas the 3D segmentation has high, isotropic resolution (squared pixels in any view). (e) 3D rendering of the 3D segmentation into the different brain regions, including hippocampus (yellow), amygdala (light blue), thalamus (green), putamen (pink), caudate (darker blue), lateral ventricle (purple), white matter (white, transparent), and cortex (red, transparent). (f) Distribution of hippocampal volumes in post mortem confirmed Alzheimer’s disease vs controls in the MADRC dataset, corrected for age and gender.

Area under the receiver operating characteristic curve (AUROC) and p-value of a non-parametric Wilcoxon rank sum test comparing the volumes of brain regions for Alzheimer’s cases vs controls. The volumes were corrected by age and sex using a general linear model. We note that the AUROC is bounded between 0 and 1 (0.5 is chance) and is the non-parametric equivalent of the effect size (higher AUROC corresponds to larger differences). The sample size is N = 33.

Qualitatively comparison of SAMSEG vs Photo-SynthSeg: Coronal (top) and sagittal (bottom) views of the reconstruction and automated segmentation of a sample whole brain from the UW-ADRC dataset.

Dice scores of automated vs manual segmentations on select slices. Box plots are shown for SAMSEG, Photo-SynthSeg, and two ablations: use of probabilistic atlas, and targeted simulation with 4 mm slice spacing. Dice is computed in 2D, using manual segmentations on select slices. We also note that the absence of extracerebral tissue in the images contributes to high Dice for the cortex.

Correlations of volumes of brains regions estimated by SAMSEG and Photo-SynthSeg from the photographs against the ground truth values derived from the MRI. The p-values are for Steiger tests comparing the correlations achieved by the two methods (accounting for the common sample).

Reconstruction error (in mm) in synthetically sliced HCP data. The figure shows box plots for the mean reconstruction error as a function of spacing and thickness jitter. A jitter of i means that the nth slice is randomly extracted from the interval [n − i, n + i] (rather than exactly n). The center of each box represents the median; the edges of the box represent the first and third quartile; and the whiskers extend to the most extreme data points not considered outliers (not shown, in order not to clutter the plot).

(a) Dissection photograph with brain slices on black board with fiducials. (b) SIFT features for fiducial detection. (c) Photograph from (a) corrected for pixel size and perspective, with digital ruler overlaid. Segmentation against the background, grouping pieces of tissue from the same slice. (e) Sagittal slice of the initialization of a 3D reconstruction. (f) Corresponding slice of the final 3D reconstruction, obtained with a surface as reference (overlaid in yellow). (g) Corresponding slice of the 3D reconstruction provided by a probabilistic atlas (overlaid as a heat map); the real surface is overlaid in light blue for comparison.

Intermediate steps in the generative process: (a) Randomly sampled input label map from the training set. (b) Spatially augmented input label map; imperfect 3D reconstruction is simulated with a deformation jitter across the coronal plane. (c) Synthetic image obtained by sampling from a Gaussian Mixture Model conditioned on the segmentation, with randomized means and variances. (d) Slice spacing is simulated by downsampling to low resolution. This imaging volume is further augmented with a bias field and intensity transforms (brightness, contrast, gamma). (e) The final training image is obtained by resampling (d) to high resolution. The neural network is trained with pairs of images like (e) (input) and (b) (target).

Simulation and reconstruction of synthetic data. Top row: skull stripped T1 scan and (randomly translated and rotated) binary mask of the cerebrum, in yellow. Second row: original T2 scan. Third row: randomly sliced and linearly deformed T2 images. Bottom row: output of the 3D reconstruction algorithm, i.e., reconstructed T2 slices and registered reference mask overlaid in yellow.

Reconstruction with surface scan vs probabilistic atlas. (a) Initialization, with contour of 3D surface scan superimposed. (b) Reconstruction with 3D surface scan. (c) Reconstruction with probabilistic atlas (overlaid as heat map with transparency); the contour of the surface scan is overlaid in light blue, for comparison. Even though the shape of the reconstruction in (c) is plausible, it is clearly inaccurate in light of the surface scan.