1. Neuroscience
Download icon

Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED)

  1. Andreas M Brandmaier  Is a corresponding author
  2. Elisabeth Wenger
  3. Nils C Bodammer
  4. Simone Kühn
  5. Naftali Raz
  6. Ulman Lindenberger
  1. Max Planck Institute for Human Development, Germany
  2. Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Germany
  3. University Clinic Hamburg-Eppendorf, Germany
  4. Wayne State University, United States
Research Article
Cite this article as: eLife 2018;7:e35718 doi: 10.7554/eLife.35718
7 figures, 2 tables and 2 data sets

Figures

Bullseye charts representing precision and bias of a measurement instrument.

The center of each bullseye represents ground truth and the black dots represent repeated measurements. Ideal measurement instruments have minimal bias and maximal precision (as illustrated in the top-left panel).

https://doi.org/10.7554/eLife.35718.002
Each plot shows simulated data of 10 repeated measurements of a fictitious outcome X for five persons.

Within-person variance is identical over persons and panels. Left panel: Between-person variance is identical to within-person variance. In that case, ICC-indexed reliability is estimated at a rather low ICC = 0.48, whereas CV-indexed precision is fair, with averaged CV = 0.07. Right panel: Between-person standard deviation is larger than within-person standard deviation by a factor of five. In this condition, estimated ICC-indexed reliability is high (ICC = 0.96), but CV-indexed precision remains identical with averaged CV = 0.07 as the within-person standard deviation is still fairly small with respect to the sample mean.

https://doi.org/10.7554/eLife.35718.004
A study design with four brain imaging scans per person spread across three sessions on two days.

The start of each session (blue) implies that the person is moved into the scanner. On the first day, there is only a single session, that is, between scans 1 and 2, the person remains in the scanner whereas on day 2, the person is removed from the scanner after the first session and, after a short break, placed back in. This allows separating the session-specific and the day-specific variance contributions to total variance.

https://doi.org/10.7554/eLife.35718.005
Left: Measurement model of four repeated measures according to the study design protocol.

The measurement model includes day-specific effects that are separated by the error variance sources (in green), session-specific effects (represented by error variance sources in blue) and orthogonal residual error variance sources (represented in red). Right: A minimal, power-equivalent model that represents a direct measurement of the construct of interest. The true score (representing the outcome of interest) is measured with only a single error construct, whose variance is the effective error variance representing the combined influence of the complete error (co)variance structure shown in the left model.

https://doi.org/10.7554/eLife.35718.006
A Structural Equation Model of a repeated measures design in which each participant is scanned three times.

Each person is scanned the first time (T1), followed by a back-to-back immediate re-acquisition (T2), and, finally, the person is moved out of the scanner, positioned back in the scanner, and scanned the third time (T3). Parameters in the SEM correspond to estimates of true score variation (σT2), session-specific error variance component (σS2), and a residual error variance component (σE2). The standardized estimates are based on myelin water fraction data in the anterior limb of the internal capsule acquired from Arshad et al. (2017).

https://doi.org/10.7554/eLife.35718.008
Path-diagram of a joint cross-sectional and longitudinal model to estimate link-wise resting state functional connectivity.

The latent variable X (yellow) represents the outcome of interest (e.g., a particular link-wise connectivity coefficient) and is longitudinally measured by 10 measures T1 to T10 (blue rectangles). A second cross-sectional measurement, Tx (green rectangle), augments the estimation of the between-person differences. The respective data set is organized with a mutually missing data scheme, that is, all cells of the longitudinal measurements are missing for cross-sectional data rows and vice versa.

https://doi.org/10.7554/eLife.35718.009
Link-wise reliability based on combined cross-sectional and longitudinal samples from the day2day study.
https://doi.org/10.7554/eLife.35718.010

Tables

Table 1
A list of terms describing the concepts of variability across repetitions and average distance from ground truth over measurements across different disciplines and knowledge domains.
https://doi.org/10.7554/eLife.35718.003
Knowledge domainVariability across repetitionsAverage distance from ground truth
PsychologyReliabilityValidity
PhysicsPrecisionAccuracy
StatisticsVarianceBias
Measurement TheoryRandom errorSystematic error
ISO 5725 (‘Accuracy’)PrecisionTrueness
Table 2
Model-implied covariance matrix.

Rows and columns correspond to the four measurement occasions (brain scans) distributed over two days. Parameters of the covariance matrix are the individual differences of the construct of interest, σT2, the session-specific error variance, σS2, the day-specific error variance, σD2, and the residual error variance, σE2.

https://doi.org/10.7554/eLife.35718.007
Scan 1Scan 2Scan 3Scan 4
Scan 1

σT2+σE2+σD2+σS2

σT2+σD2+σS2

σT2

σT2

Scan 2

σT2+σD2+σS2

σT2+σE2+σD2+σS2

σT2

σT2

Scan 3

σT2

σT2

σT2+σE2+σD2+σS2

σT2+σD2

Scan 4

σT2

σT2

σT2+σD2

σT2+σE2+σD2+σS2

Data availability

The dataset on myelin water fraction measurements is freely available at https://osf.io/t68my/ and the link-wise resting state functional connectivity data are available at https://osf.io/8n24x/.

The following previously published data sets were used
  1. 1
  2. 2
    Reliability of Myelin Water Fraction in ALIC
    1. Arshad M
    2. Stanley J A
    3. Raz N
    (2018)
    Publicly available at Open Science Framework.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)