Evidence for embracing normative modeling

  1. Saige Rutherford  Is a corresponding author
  2. Pieter Barkema
  3. Ivy F Tso
  4. Chandra Sripada
  5. Christian F Beckmann
  6. Henricus G Ruhe
  7. Andre F Marquand
  1. Department of Cognitive Neuroscience, Radboud University Nijmegen Medical Centre, Netherlands
  2. Donders Institute, Radboud University Nijmegen, Netherlands
  3. Department of Psychiatry, University of Michigan-Ann Arbor, United States
  4. Department of Psychology, University of Michigan-Ann Arbor, United States
  5. Department of Philosophy, University of Michigan-Ann Arbor, United States
  6. Center for Functional MRI of the Brain (FMRIB), Nuffield Department for Clinical Neuroscience, Welcome Centre for Integrative Neuroimaging, Oxford University, United Kingdom
  7. Department of Psychiatry, Radboud University Nijmegen Medical Centre, Netherlands
5 figures, 2 tables and 5 additional files

Figures

Overview of workflow.

(A) Datasets included the Human Connectome Project (young adult) study, the University of Michigan schizophrenia study, and the Center for Biomedical Research Excellence (COBRE) schizophrenia study. (B) Openly shared, pre-trained on big data, normative models were estimated for large-scale resting-state functional brain networks and cortical thickness. (C) Deviation (Z) scores and raw data, for both functional and structural data, were input into three benchmarking tasks: 1. group difference testing, 2. support vector machine (SVM) classification, and 3. regression (predicting cognition). (D) Evaluation metrics were calculated for each benchmarking task. These metrics were calculated for the raw data models and the deviation score models. The difference between each models’ performance was calculated for both functional and structural modalities.

Functional brain network normative modeling.

(A) Age distribution per scanning site in the train, test, and transfer data partitions and across the full sample (train +test). (B) The Yeo-17 brain network atlas is used to generate connectomes. Between network connectivity was calculated for all 17 networks, resulting in 136 unique network pairs that were each individually input into a functional normative model. (C) The explained variance in the controls test set (N=7244) of each of the unique 136 network pairs of the Yeo-17 atlas. Networks were clustered for visualization to show similar variance patterns.

Functional normative model evaluation metrics.

(A) Explained variance per network pair across the test set (top), and both transfer sets (patients – middle, controls – bottom). Networks were clustered for visualization to show similar variance patterns. (B) The distribution across all models of the evaluation metrics (columns) in the test set (top row) and both transfer sets (middle and bottom rows). Higher explained variance (closer to one), more negative MSLL, and normally distributed skew and kurtosis correspond to better model fit.

Group difference testing evaluation.

(A) Significant group differences in the deviation score models, (top left) functional brain network deviation, and (top right) cortical thickness deviation scores. The raw data, either cortical thickness or functional brain networks (residualized of sex and linear/ quadratic effects of age and motion (mean framewise displacement)) resulted in no significant group differences after multiple comparison corrections. Functional networks were clustered for visualization to show similar variance patterns. (B) There are still individual differences observed that do not overlap with the group difference map, showing the benefit of normative modeling, which can detect both group and individual differences through proper modeling of variation. Functional networks were clustered for visualization to show similar variance patterns. (C) There are significant group differences in the summaries (count) of the individual difference maps (panel B).

Benchmark task two multivariate prediction Classification evaluation.

(A) Support vector classification (SVC) using cortical thickness deviation scores as input features (most accurate model). (B) SVC using cortical thickness (residualized of sex and linear/quadratic effects of age) as input features. (C) SVC using functional brain network deviation scores as input features. (D) SVC using functional brain networks (residualized of sex and linear/ quadratic effects of age and motion (mean framewise displacement)) as input features.

Tables

Table 1
Dataset inclusion and sample overview.
Cortical ThicknessFunctional Networks
StudyBenchmark TaskNAge
(m, s.d.)
F, M (%)NAge
(m, s.d.)
F, M (%)
HCPRegression – predicting cognition52928.8, 3.653.4, 46.649928.9, 3.654.3, 45.6
COBREClassification & Group Difference12437.0, 12.724.2, 75.812135.4, 12.423.1, 76.9
UMichClassification & Group Difference8932.6, 9.650.6, 49.38733.0, 10.150.6, 49.3
Table 2
Benchmarking results.

Deviation (Z) score column shows the performance using deviation scores (AUC for classification, the total number of regions with significant group differences FDR-corrected p<0.05 for case versus control, mean squared error for regression), Raw column represents the performance when using the raw data, and Difference column shows the difference between the deviation scores and raw data (Deviation - Raw). Higher AUC, higher count, and lower MSE represent better performance. Positive values in the Difference column show that there is better performance when using deviation scores as input features for classification and group difference tasks, and negative performance difference values for the regression task show there is a better performance using the deviation scores. *=statistically significant difference between Z and Raw established using permutation testing (10 k perms).

BenchmarkModalityNormative ModelingDeviation Score DataRaw DataPerformanceDifference
Group DifferenceCortical thickness117/1870/187117*
Group DifferenceFunctional Networks50/1360/13650*
ClassificationCortical thickness0.870.430.44*
ClassificationFunctional Networks0.690.680.01
RegressionCortical thickness0.6990.708–0.008
RegressionFunctional Networks0.8770.890–0.013

Additional files

Supplementary file 1

Functional Normative Model Demographics.

Description: For each included site, we show the sample size (N), age (mean, standard deviation), and sex distribution (Female/Male percent) in the training set (shown in blue) and testing set (shown in green) of the normative models of functional connectivity between large scale resting-state brain networks from the Yeo 17 network atlas.

https://cdn.elifesciences.org/articles/85082/elife-85082-supp1-v2.xlsx
Supplementary file 2

Surface Area Normative Model Demographics.

Description: For each included site, we show the sample size (N), age (mean, standard deviation), and sex distribution (Female/Male percent) of the normative models of surface area extracted for all regions of interest in the Destrieux Freesurfer atlas.

https://cdn.elifesciences.org/articles/85082/elife-85082-supp2-v2.xlsx
Supplementary file 3

Structural Group Difference Testing Statistics.

Description: We show for all cortical thickness and subcortical volume from the Destrieux and aseg Freesurfer atlases regions of interest (ROIs from a two-sample t-test between Schizophrenia versus Healthy Controls) the t-statistic (T-stat), False Discovery Rate corrected p-value (FDRcorr_pvalue), and uncorrected p-value (uncorr_pvalue) for both the raw data (shown in green) and the deviation scores (shown in blue).

https://cdn.elifesciences.org/articles/85082/elife-85082-supp3-v2.xlsx
Supplementary file 4

Functional Connectivity Group Difference Testing Statistics.

Description: We show for all Yeo-17 between network connectivity regions of interest (ROIs) from a two-sample t-test between Schizophrenia versus Healthy Controls the t-statistic (T-stat), False Discovery Rate corrected p-value (FDRcorr_pvalue), and uncorrected p-value (uncorr_pvalue) for both the raw data (shown in green) and the deviation scores (shown in blue).

https://cdn.elifesciences.org/articles/85082/elife-85082-supp4-v2.xlsx
MDAR checklist
https://cdn.elifesciences.org/articles/85082/elife-85082-mdarchecklist1-v2.pdf

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Saige Rutherford
  2. Pieter Barkema
  3. Ivy F Tso
  4. Chandra Sripada
  5. Christian F Beckmann
  6. Henricus G Ruhe
  7. Andre F Marquand
(2023)
Evidence for embracing normative modeling
eLife 12:e85082.
https://doi.org/10.7554/eLife.85082