Machine learning of dissection photographs and surface scanning for quantitative 3D neuropathology

Reviewer #1 (Public Review):

Gazula and co-workers presented in this paper a software tool for 3D structural analysis of human brains, using slabs of fixed or fresh brains. This tool will be included in Freesurfer, a well-known neuroimaging processing software. It is possible to reconstruct a 3D surface from photographs of coronal sliced brains, optionally using a surface scan as a model. A high-resolution segmentation of 11 brain regions is produced, independent of the thickness of the slices, interpolating information when needed. Using this method, the researcher can use the sliced brain to segment all regions, without the need for ex vivo MRI scanning.

The software suite is freely available and includes 3 modules. The first accomplishes preprocessing steps, for correction of pixel sizes and perspective. The second module is a registration algorithm that registers a 3D surface scan obtained prior to sectioning (reference) to the multiple 2D slices. It is not mandatory to scan the surface - a probabilistic atlas can also be used as a reference - however, the accuracy is lower. The third module uses machine learning to perform the segmentation of 11 brain structures in the 3D reconstructed volume. This module is robust, dealing with different illumination conditions, cameras, lenses, and camera settings. This algorithm ("Photo-SynthSeg") produces isotropic smooth reconstructions, even in high anisotropic datasets (when the in-plane resolution of the photograph is much higher than the thickness), interpolating the information between slices.

To verify the accuracy and reliability of the toolbox, the authors reconstructed 3 datasets, using real and synthetic data. Real data of 21 postmortem confirmed Alzheimer's disease cases from the Massachusetts Alzheimer's Disease Research Center (MADRC) and 24 cases from the AD Research at the University of Washington (who were MRI scanned prior to processing) were employed for testing. These cases represent a challenging real-world scenario. Additionally, 500 subjects of the Human Connectome project were used for testing error as a continuous function of slice thickness. The segmentations were performed with the proposed deep-learning new algorithm ("Photo-SynthSeg") and compared against MRI segmentations performed to "SAMSEG" (an MRI segmentation algorithm, computing Dice scores for the segmentations. The methods are sound and statistically showed correlations above 0.8, which is good enough to allow volumetric analysis. The main strengths of the methods are the datasets used (real-world challenging and synthetic) and the statistical treatment, which showed that the pipeline is robust and can facilitate volumetric analysis derived from brain sections and conclude which factors can influence the accuracy of the method (such as using or not 3D scan and using constant thickness).

Although very robust and capable of handling several situations, the researcher has to keep in mind that processing has to follow some basic rules in order for this pipeline to work properly. For instance, fiducials and scales need to be included in the photograph, and the slabs must be photographed against a contrasting background. Also, only coronal slices can be used, which can be limiting for certain situations.

The authors achieved their aims, and the statistical analysis confirms that the machine learning algorithm performs segmentations comparable to the state-of-the-art of automated MRI segmentations.

Those methods will be particularly interesting to researchers who deal with post-mortem tissue analysis and do not have access to ex vivo MRI. Quantitative measurements of specific brain areas can be performed in different pathologies and even in the normal aging process. The method is highly reproducible, and cost-effective since it allows the pipeline to be applied by any researcher with small pre-processing steps.

The paper is very interesting and well structured, adding an important tool for fixed and fresh brain analysis. The software tool is robust and demonstrated good and consistent results in the hard task of managing automated segmentation from brain slices. In the future, segmentation of the histological slices could be developed and histological structures added (such as small brainstem nuclei, for instance). Also, dealing with axial and sagittal planes can be useful to some labs.

Reviewer #2 (Public Review):

Summary:
The authors developed a tool-set Photo-SynthSeg for the software FreeSurfer which performs 3D reconstruction and high-resolution 3D segmentation on a stack of dissection photographs of brain tissues. The tool-set consists of three modules: the pre-processing module, which performs dissection photography correction; the registration module, which registers corrected dissection photographs based on 3D surface scan, ex vivo MRI or probabilistic atlas; the segmentation module based on U-Net. To prove the performance of the tools, three experiments were conducted, including a volumetric comparison of brain tissues on AD and HC groups from MADRC, a quantitative evaluation of segmentation on UW-ADRC and a quantitative evaluation of 3D reconstruction on HCP digitally sliced MRI data.

Strengths:
The quantitative evaluation of segmentation and reconstruction on synthetic and real data demonstrates the accuracy of the methodology. Also, the successful application of this toolset on two brain banks with different slice thicknesses, tissue processing, and photograph settings demonstrates its robustness. The toolset also benefits from its adaptability of different 3D references, such as surface scans, ex vivo MRI, and even probabilistic atlas, suiting the needs of different brain banks.

Weaknesses:

1. The current method could only perform accurate segmentation on subcortical tissues. It is of more interest to accurately segment cortical tissues, whose morphometrics are more predictive of neuropathology. The authors also mentioned that they would extend the toolset to allow for cortical tissue segmentation in the future.

2. Brain tissues are not rigid bodies, so dissected slices could be stretched or squeezed to some extent. Also, dissected slices that contain temporal poles may have several disjoined tissues. Therefore, each pixel in dissected photographs may go through slightly different transformations. The authors constrain that all pixels in each dissected photograph go through the same affine transform in the reconstruction step probably due to concerns of computational complexity. But ideally, dissected photographs should be transformed with some non-linear warping or locally linear transformations. Or maybe the authors could advise how to place different parts of dissected slices when taking dissection photographs to reduce such non-linearity of transforms.

3. For the quantitative evaluation of the segmentation on UW-ARDC, the authors calculated 2D Dice scores on a single slice for each subject. Could the authors specify how this single slice is chosen for each subject? Is it randomly chosen or determined by some landmarks? It's possible that the chosen slice is between dissected slices so SAMSEG cannot segment accurately. Also from Figure 3, it seems that SAMSEG outperforms Photo-SynthSeg on large tissues, WM/Cortex/Ventricle. Is there an explanation for this observation?

4. In the third experiment, quantitative evaluation of 3D reconstruction, each digital slice went through random affine transformations and illumination fields only. However, it's better to deform digital slices using random non-linear warping due to the non-rigidity of the brain as mentioned in 2). So, the reconstruction errors estimated here are quite optimistic. It would be more realistic if digital slices were deformed using random non-linear warping.

Overall, this is quite useful a toolset that could be widely used in many brain banks without MRI scanners.

Author Response

We would like to thank the reviewers for their encouraging comments and useful feedback, which will enable us to improve the manuscript. We would like to briefly comment on some of the points they raised.

1. We agree this is a fairly specialized pipeline that has some requirements in terms of photographic setup. We are working hard to make these requirements as minimal as possible. However, given the huge variability in camera angles, backgrounds, arrangement of brain slices, etc., making the pipeline fully automated for unconstrained photos is extremely challenging.

2. In principle, it should be possible to extend our method to sagittal slices of the cerebellum or axial slices f the brainstem, but this would require collecting and labeling additional training data and thus remains as future work.

3. Producing accurate surfaces with sparse photographs is a very challenging problem and also remains as future work. We have a conference article producing surfaces on MRI scans with sparse slices (https://doi.org/10.1007/978-3-031-43993-3_4) but we haven’t gotten it to work well on photographs yet.

4. Another challenging issue that remains as future work is getting the pipeline to work well with nonlinear deformations, e.g., slices of fresh tissue. While incorporating nonlinear deformation into the model is trivial from the coding perspective, we have not been able to make it work at the level of robustness that we achieve with affine transformations. This is because the nonlinear model introduces huge ambiguity in the space of solutions: for example, if one adds identical small nonlinear deformations to every slice, the objective function barely changes.

5. As we acknowledge in the manuscript, the validation of the reconstruction error (in mm) with synthetic data is indeed optimistic, but informative in the sense that they reflect the trends of the error as a function of slice thickness and its variability (“jitter”).

6. Since we use a single central coronal slice in the direct evaluation, SAMSEG yields very high Dice scores for large structures with strong contrast (e.g., the lateral ventricles). However, Photo-SynthSeg provides better average results across the board, particularly when considering 3D analysis out of the coronal plane (see qualitative results in Figure 2 and results on volume correlations).