Limitations of principal components in quantitative genetic association models for human studies

  1. Yiqi Yao
  2. Alejandro Ochoa  Is a corresponding author
  1. Department of Biostatistics and Bioinformatics, Duke University, United States
  2. Duke Center for Statistical Genetics and Genomics, Duke University, United States


Principal Component Analysis (PCA) and the Linear Mixed-effects Model (LMM), sometimes in combination, are the most common genetic association models. Previous PCA-LMM comparisons give mixed results, unclear guidance, and have several limitations, including not varying the number of principal components (PCs), simulating simple population structures, and inconsistent use of real data and power evaluations. We evaluate PCA and LMM both varying number of PCs in realistic genotype and complex trait simulations including admixed families, subpopulation trees, and real multiethnic human datasets with simulated traits. We find that LMM without PCs usually performs best, with the largest effects in family simulations and real human datasets and traits without environment effects. Poor PCA performance on human datasets is driven by large numbers of distant relatives more than the smaller number of closer relatives. While PCA was known to fail on family data, we report strong effects of family relatedness in genetically diverse human datasets, not avoided by pruning close relatives. Environment effects driven by geography and ethnicity are better modeled with LMM including those labels instead of PCs. This work better characterizes the severe limitations of PCA compared to LMM in modeling the complex relatedness structures of multiethnic human data for association studies.

Editor's evaluation

This is an important paper that presents compelling arguments (based on simulation and comprehensively reviewed background theory) that Linear Mixed Models generally should perform better at correcting for genetic and environmental confounding in GWAS than more commonly used Principal Components methods.


The goal of a genetic association study is to identify loci whose genotype variation is significantly correlated to given trait. Naive association tests assume that genotypes are drawn independently from a common allele frequency. This assumption does not hold for structured populations, which includes multiethnic cohorts and admixed individuals (ancient relatedness), and for family data (recent relatedness; Astle and Balding, 2009). Association studies of admixed and multiethnic cohorts, the focus of this work, are becoming more common, are believed to be more powerful, and are necessary to bring more equity to genetic medicine (Rosenberg et al., 2010; Hoffman and Dubé, 2013; Coram et al., 2013; Medina-Gomez et al., 2015; Conomos et al., 2016a; Hodonsky et al., 2017; Martin et al., 2017a; Martin et al., 2017b; Hindorff et al., 2018; Hoffmann et al., 2018; Mogil et al., 2018; Roselli et al., 2018; Wojcik et al., 2019; Peterson et al., 2019; Zhong et al., 2019; Hu et al., 2020; Simonin-Wilmer et al., 2021; Kamariza et al., 2021; Lin et al., 2021; Mahajan et al., 2022; Hou et al., 2023a). When insufficient approaches are applied to data with relatedness, their association statistics are miscalibrated, resulting in excess false positives and loss of power (Devlin and Roeder, 1999; Voight and Pritchard, 2005; Astle and Balding, 2009). Therefore, many specialized approaches have been developed for genetic association under relatedness, of which PCA and LMM are the most popular.

Genetic association with PCA consists of including the top eigenvectors of the population kinship matrix as covariates in a generalized linear model (Zhang et al., 2003; Price et al., 2006; Bouaziz et al., 2011). These top eigenvectors are a new set of coordinates for individuals that are commonly referred to as PCs in genetics (Patterson et al., 2006), the convention adopted here, but in other fields PCs instead denote what in genetics would be the projections of loci onto eigenvectors, which are new independent coordinates for loci (Jolliffe, 2002). The direct ancestor of PCA association is structured association, in which inferred ancestry (genetic cluster membership, often corresponding with labels such as “European”, “African”, “Asian”, etc.) or admixture proportions of these ancestries are used as regression covariates (Pritchard et al., 2000). These models are deeply connected because PCs map to ancestry empirically (Alexander et al., 2009; Zhou et al., 2016) and theoretically (McVean, 2009; Zheng and Weir, 2016; Cabreros and Storey, 2019; Chiu et al., 2022), and they work as well as global ancestry in association studies but are estimated more easily (Patterson et al., 2006; Zhao et al., 2007; Alexander et al., 2009; Bouaziz et al., 2011). Another approach closely related to PCA is nonmetric multidimensional scaling (Zhu and Yu, 2009). PCs are also proposed for modeling environment effects that are correlated to ancestry, for example, through geography (Novembre et al., 2008; Zhang and Pan, 2015; Lin et al., 2021). The strength of PCA is its simplicity, which as covariates can be readily included in more complex models, such as haplotype association (Xu and Guan, 2014) and polygenic models (Qian et al., 2020). However, PCA assumes that the underlying relatedness space is low dimensional (or low rank), so it can be well modeled with a small number of PCs, which may limit its applicability. PCA is known to be inadequate for family data (Patterson et al., 2006; Zhu and Yu, 2009; Thornton and McPeek, 2010; Price et al., 2010), which is called ‘cryptic relatedness’ when it is unknown to the researchers, but no other troublesome cases have been confidently identified. Recent work has focused on developing more scalable versions of the PCA algorithm (Lee et al., 2012; Abraham and Inouye, 2014; Galinsky et al., 2016; Abraham et al., 2017; Agrawal et al., 2020). PCA remains a popular and powerful approach for association studies.

The other dominant association model under relatedness is the LMM, which includes a random effect parameterized by the kinship matrix. Unlike PCA, LMM does not assume that relatedness is low-dimensional, and explicitly models families via the kinship matrix. Early LMMs used kinship matrices estimated from known pedigrees or using methods that captured recent relatedness only, and modeled population structure (ancestry) as fixed effects (Yu et al., 2006; Zhao et al., 2007; Zhu and Yu, 2009). Modern LMMs estimate kinship from genotypes using a non-parametric estimator, often referred to as a genetic relationship matrix, that captures the combined covariance due to family relatedness and ancestry (Kang et al., 2008; Astle and Balding, 2009; Ochoa and Storey, 2021). Like PCA, LMM has also been proposed for modeling environment correlated to genetics (Vilhjálmsson and Nordborg, 2013; Wang et al., 2022). The classic LMM assumes a quantitative (continuous) complex trait, the focus of our work. Although case-control (binary) traits and their underlying ascertainment are theoretically a challenge (Yang et al., 2014), LMMs have been applied successfully to balanced case-control studies (Astle and Balding, 2009; Kang et al., 2010) and simulations (Price et al., 2010; Wu et al., 2011; Sul and Eskin, 2013), and have been adapted for unbalanced case-control studies (Zhou et al., 2018). However, LMMs tend to be considerably slower than PCA and other models, so much effort has focused on improving their runtime and scalability (Aulchenko et al., 2007; Kang et al., 2008; Kang et al., 2010; Zhang et al., 2010; Lippert et al., 2011; Yang et al., 2011; Listgarten et al., 2012; Zhou and Stephens, 2012; Svishcheva et al., 2012; Loh et al., 2015; Zhou et al., 2018).

An LMM variant that incorporates PCs as fixed covariates is tested thoroughly in our work. Since PCs are the top eigenvectors of the same kinship matrix estimate used in modern LMMs (Astle and Balding, 2009; Janss et al., 2012; Hoffman and Dubé, 2013; Zhang and Pan, 2015), then population structure is modeled twice in an LMM with PCs. However, some previous work has found the apparent redundancy of an LMM with PCs beneficial (Price et al., 2010; Tucker et al., 2014; Zhang and Pan, 2015), while others did not (Liu et al., 2011; Janss et al., 2012), and the approach continues to be used (Zeng et al., 2018; Mbatchou et al., 2021), although not always (Matoba et al., 2020). Recall that early LMMs used kinship to model family relatedness only, so population structure had to be modeled separately in those models, in practice as admixture fractions instead of PCs (Yu et al., 2006; Zhao et al., 2007; Zhu and Yu, 2009). The LMM with PCs (vs no PCs) is also believed to help better model loci that have experienced selection (Price et al., 2010; Vilhjálmsson and Nordborg, 2013) and environment effects correlated with genetics (Zhang and Pan, 2015).

LMM and PCA are closely related models (Astle and Balding, 2009; Janss et al., 2012; Hoffman and Dubé, 2013; Zhang and Pan, 2015), so similar performance is expected particularly under low-dimensional relatedness. Direct comparisons have yielded mixed results, with several studies finding superior performance for LMM, notably from papers promoting advances in LMMs, while many others report comparable performance (Table 1). No papers find that PCA outperforms LMM decisively, although PCA occasionally performs better in isolated and artificial cases or individual measures, often with unknown significance. Previous studies generally used either only simulated or only real genotypes, with only two studies using both. The simulated genotype studies, which tended to have low model dimensions and FST, were more likely to report ties or mixed results (6/8), whereas real genotypes tended to clearly favor LMMs (9/11). Similarly, 10/12 papers with quantitative traits favor LMMs, whereas 6/9 papers with case-control traits gave ties or mixed results—the only factor we do not explore in this work. Additionally, although all previous evaluations measured type I error (or proxies such as genomic inflation factors Devlin and Roeder, 1999 or QQ plots), a large fraction (6/17) did not measure power (or proxies such as ROC curves), and only four used more than one number of PCs for PCA. Lastly, no consensus has emerged as to why LMM might outperform PCA or vice versa (Price et al., 2010; Sul and Eskin, 2013; Price et al., 2013; Hoffman and Dubé, 2013), or which features of the real datasets are critical for the LMM advantage other than family relatedness, resulting in unclear guidance for using PCA. Hence, our work includes real and simulated genotypes with higher model dimensions and FST matching that of multiethnic human cohorts (Ochoa and Storey, 2021; Ochoa and Storey, 2019), we vary the number of PCs, and measure robust proxies for type I error control and calibrated power.

Table 1
Previous PCA-LMM evaluations in the literature.
Sim. GenotypesGeneral
PublicationType*KFSTReal §Trait PowerPCs(r)Best
Zhao et al., 2007Q8LMM
Zhu and Yu, 2009I, A, F3, 8≤0.15Q1–22LMM
Astle and Balding, 2009I30.10CC10Tie
Kang et al., 2010Both2–100LMM
Price et al., 2010I, F20.01CC1Mixed
Wu et al., 2011I, A2–40.01CC10Mixed
Liu et al., 2011S, A2–3RQ10Tie
Sul and Eskin, 2013I20.01CC1Tie
Tucker et al., 2014I20.05Both5Tie
Yang et al., 2014CC5Tie
Song et al., 2015S, A2–3RQ3LMM
Loh et al., 2015Q10LMM
Zhang and Pan, 2015Q20–100LMM
Liu et al., 2016Q3–6LMM
Sul et al., 2018Q100LMM
Loh et al., 2018Both20LMM
Mbatchou et al., 2021Both1LMM
This workA, T, F10–243≤0.25Q0–90LMM
  1. *

    Genotype simulation types. I: Independent subpopulations; S: subpopulations (with parameters drawn from real data); A: Admixture; T: Subpopulation Tree; F: Family.

  2. Model dimension (number of subpopulations or ancestries).

  3. R: simulated parameters based on real data, FST not reported.

  4. §

    Evaluations using unmodified real genotypes.

  5. Q: quantitative; CC: case-control.

In this work, we evaluate the PCA and LMM association models under various numbers of PCs, which are included in LMMs too. We use genotype simulations (admixture, family, and subpopulation tree models) and three real datasets: the 1000 Genomes Project (Abecasis et al., 2010; Abecasis et al., 2012), the Human Genome Diversity Panel (HGDP) (Cann et al., 2002; Rosenberg et al., 2002; Bergström et al., 2020), and Human Origins (Patterson et al., 2012; Lazaridis et al., 2014; Lazaridis et al., 2016; Skoglund et al., 2016). We simulate quantitative traits from two models: fixed effect sizes (FES) construct coefficients inverse to allele frequency, which matches real data (Park et al., 2011; Zeng et al., 2018; O’Connor et al., 2019) and corresponds to high pleiotropy and strong balancing selection (Simons et al., 2018) and strong negative selection (Zeng et al., 2018; O’Connor et al., 2019), which are appropriate assumptions for diseases; and random coefficients (RC), which are drawn independent of allele frequency, and corresponds to neutral traits (Zeng et al., 2018; Simons et al., 2018). LMM without PCs consistently performs best in simulations without environment, and greatly outperforms PCA in the family simulation and in all real datasets. The tree simulations, which model subpopulations with the tree but exclude family structure, do not recapitulate the real data results, suggesting that family relatedness in real data is the reason for poor PCA performance. Lastly, removing up to 4th degree relatives in the real datasets recapitulates poor PCA performance, showing that the more numerous distant relatives explain the result, and suggesting that PCA is generally not an appropriate model for real data. We find that both LMM and PCA are able to model environment effects correlated with genetics, and LMM with PCs gains a small advantage in this setting only, but direct modeling of environment performs much better. All together, we find that LMMs without PCs are generally a preferable association model, and present novel simulation and evaluation approaches to measure the performance of these and other genetic association approaches.


Overview of evaluations

We use three real genotype datasets and simulated genotypes from six population structure scenarios to cover various features of interest (Table 2). We introduce them in sets of three, as they appear in the rest of our results. Population kinship matrices, which combine population and family relatedness, are estimated without bias using popkin (Ochoa and Storey, 2021; Figure 1). The first set of three simulated genotypes are based on an admixture model with 10 ancestries (Figure 1A; Ochoa and Storey, 2021; Gopalan et al., 2016; Cabreros and Storey, 2019). The ‘large’ version (1000 individuals) illustrates asymptotic performance, while the ‘small’ simulation (100 individuals) illustrates model overfitting. The ‘family’ simulation has admixed founders and draws a 20-generation random pedigree with assortative mating, resulting in a complex joint family and ancestry structure in the last generation (Figure 1B). The second set of three are the real human datasets representing global human diversity: Human Origins (Figure 1D), HGDP (Figure 1G), and 1000 Genomes (Figure 1J), which are enriched for small minor allele frequencies even after MAF <1% filter (Figure 1C). Last are subpopulation tree simulations (Figure 1F, I, L) fit to the kinship (Figure 1E, H and K) and MAF (Figure 1C) of each real human dataset, which by design do not have family structure.

Table 2
Features of simulated and real human genotype datasets.
DatasetTypeLoci(m)Ind. (n)Subpops.* (K)Causal loci (m1)FST
Admix. Large sim.Admix.100 0001000101000.1
Admix. Small sim.Admix.100 00010010100.1
Admix. Family sim.Admix.+Pedig.100 0001000101000.1
Human OriginsReal190 394292211–2432920.28
HGDPReal771 3229297–54930.28
1000 GenomesReal1 111 26625045–262500.22
Human Origins sim.Tree190 39429222432920.23
HGDP sim.Tree771 32292954930.25
1000 Genomes sim.Tree1 111 2662504262500.21
  1. *

    For admixed family, ignores additional model dimension of 20 generation pedigree structure. For real datasets, lower range is continental subpopulations, upper range is number of fine-grained subpopulations.

  2. m1=round(nh2/8) to balance power across datasets, shown for h2=0.8 only.

  3. Model parameter for simulations, estimated value on real datasets.

Population structures of simulated and real human genotype datasets.

First two columns are population kinship matrices as heatmaps: individuals along x- and y-axis, kinship as color. Diagonal shows inbreeding values. (A) Admixture scenario for both Large and Small simulations. (B) Last generation of 20-generation admixed family, shows larger kinship values near diagonal corresponding to siblings, first cousins, etc. (C) Minor allele frequency (MAF) distributions. Real datasets and subpopulation tree simulations had MAF0.01 filter. (D) Human Origins is an array dataset of a large diversity of global populations. (G) Human Genome Diversity Panel (HGDP) is a WGS dataset from global native populations. (J) 1000 Genomes Project is a WGS dataset of global cosmopolitan populations. (F, I, L) Trees between subpopulations fit to real data. (E, H, K). Simulations from trees fit to the real data recapitulate subpopulation structure.

All traits in this work are simulated. We repeated all evaluations on two additive quantitative trait models, fixed effect sizes (FES) and random coefficients (RC), which differ in how causal coefficients are constructed. The FES model captures the rough inverse relationship between coefficient and minor allele frequency that arises under strong negative and balancing selection and has been observed in numerous diseases and other traits (Park et al., 2011; Zeng et al., 2018; Simons et al., 2018; O’Connor et al., 2019), so it is the focus of our results. The RC model draws coefficients independent of allele frequency, corresponding to neutral traits (Zeng et al., 2018; Simons et al., 2018), which results in a wider effect size distribution that reduces association power and effective polygenicity compared to FES.

We evaluate using two complementary measures: (1) SRMSDp (p-value signed root mean square deviation) measures p-value calibration (closer to zero is better), and (2) AUCPR (precision-recall area under the curve) measures causal locus classification performance (higher is better; Figure 2). SRMSDp is a more robust alternative to the common inflation factor λ and type I error control measures; there is a correspondence between λ and SRMSDp, with SRMSDp>0.01 giving λ>1.06 (Figure 2—figure supplement 1) and thus evidence of miscalibration close to the rule of thumb of λ>1.05 (Price et al., 2010). There is also a monotonic correspondence between SRMSDp and type I error rate (Figure 2—figure supplement 2). AUCPR has been used to evaluate association models (Rakitsch et al., 2013), and reflects calibrated statistical power (Figure 2—figure supplement 3) while being robust to miscalibrated models (Appendix 2).

Figure 2 with 3 supplements see all
Illustration of evaluation measures.

Three archetypal models illustrate our complementary measures: M1 is ideal, M2 overfits slightly, M3 is naive. (A) QQ plot of p-values of “null” (non-causal) loci. M1 has desired uniform p-values, M2/M3 are miscalibrated. (B)SRMSDp (p-value Signed Root Mean Square Deviation) measures signed distance between observed and expected null p-values (closer to zero is better). (C) Precision and Recall (PR) measure causal locus classification performance (higher is better). (D) AUCPR (Area Under the PR Curve) reflects power (higher is better).

Both PCA and LMM are evaluated in each replicate dataset including a number of PCs r between 0 and 90 as fixed covariates. In terms of p-value calibration, for PCA the best number of PCs r (minimizing mean |SRMSDp| over replicates) is typically large across all datasets (Table 3), although much smaller r values often performed as well (shown in following sections). Most cases have a mean |SRMSDp|<0.01, whose p-values are effectively calibrated. However, PCA is often miscalibrated on the family simulation and real datasets (Table 3). In contrast, for LMM, r=0 (no PCs) is always best, and is always calibrated. Comparing LMM with r=0 to PCA with its best r, LMM always has significantly smaller |SRMSDp| than PCA or is statistically tied. For AUCPR and PCA, the best r is always smaller than the best r for |SRMSDp|, so there is often a tradeoff between calibrated p-values versus classification performance. For LMM, there is no tradeoff, as r=0 often has the best mean AUCPR, and otherwise is not significantly different from the best r. Lastly, LMM with r=0 always has significantly greater or statistically tied AUCPR than PCA with its best r.

Table 3
Overview of PCA and LMM evaluations for high heritability simulations.
LMM r=0 vs best rPCA vs LMM r=0
DatasetMetricTrait*Cal.Best rP-value §Best rCal.P-value §Best model
Admix. Large sim.|SRMSDp|FESTrue0112True0.036Tie
Admix. Small sim.|SRMSDp|FESTrue014True0.055Tie
Admix. Family sim.|SRMSDp|FESTrue0190False3.9e-10*LMM
Human Origins|SRMSDp|FESTrue0189False3.9e-10*LMM
1000 Genomes|SRMSDp|FESTrue0190False3.9e-10*LMM
Human Origins sim.|SRMSDp|FESTrue0188True0.017Tie
HGDP sim.|SRMSDp|FESTrue0147True0.046Tie
1000 Genomes sim.|SRMSDp|FESTrue0178True9.6e-10*LMM
Admix. Large sim.|SRMSDp|RCTrue0126True0.11Tie
Admix. Small sim.|SRMSDp|RCTrue014True0.00097Tie
Admix. Family sim.|SRMSDp|RCTrue0190False3.9e-10*LMM
Human Origins|SRMSDp|RCTrue0190True0.00065Tie
1000 Genomes|SRMSDp|RCTrue0176True3.9e-10*LMM
Human Origins sim.|SRMSDp|RCTrue0185True0.14Tie
HGDP sim.|SRMSDp|RCTrue0144True8.8e-07*LMM
1000 Genomes sim.|SRMSDp|RCTrue0190True3.9e-10*LMM
Admix. Large sim.AUCPRFES0135.9e-06*LMM
Admix. Small sim.AUCPRFES0120.025Tie
Admix. Family sim.AUCPRFES10.35223.9e-10*LMM
Human OriginsAUCPRFES01343.9e-10*LMM
1000 GenomesAUCPRFES10.1183.9e-10*LMM
Human Origins sim.AUCPRFES01363.9e-10*LMM
HGDP sim.AUCPRFES01171.7e-05*LMM
1000 Genomes sim.AUCPRFES01105e-10*LMM
Admix. Large sim.AUCPRRC0131.4e-05*LMM
Admix. Small sim.AUCPRRC0110.095Tie
Admix. Family sim.AUCPRRC01343.9e-10*LMM
Human OriginsAUCPRRC30.4369.6e-10*LMM
1000 GenomesAUCPRRC50.00490.00043Tie
Human Origins sim.AUCPRRC01374.1e-10*LMM
HGDP sim.AUCPRRC30.087170.0014Tie
1000 Genomes sim.AUCPRRC30.37108.5e-10*LMM
  1. *

    FES: Fixed Effect Sizes, RC: Random Coefficients.

  2. Calibrated: whether mean |SRMSDp|<0.01 over 50 replicates.

  3. Value of r (number of PCs) with minimum mean |SRMSDp| or maximum mean AUCPR.

  4. §

    Wilcoxon paired 1-tailed test of distributions (|SRMSDp| or AUCPR) between models in header. Asterisk marks significant value using Bonferroni threshold (p<α/ntests with α=0.01 and ntests=72 is the number of tests in this table).

  5. Tie if no significant difference using Bonferroni threshold.

Evaluations in admixture simulations

Now we look more closely at results per dataset. The complete SRMSDp and AUCPR distributions for the admixture simulations and FES traits are in Figure 3. RC traits gave qualitatively similar results (Figure 3—figure supplement 1).

Figure 3 with 5 supplements see all
Evaluations in admixture simulations with FES traits, high heritability.

PCA and LMM models have varying number of PCs (r{0,,90} on x-axis), with the distributions (y-axis) of SRMSDp (top subpanel) and AUCPR (bottom subpanel) for 50 replicates. Best performance is zero SRMSDp and large AUCPR. Zero and maximum median AUCPR values are marked with horizontal gray dashed lines, and |SRMSDp|<0.01 is marked with a light gray area. LMM performs best with r=0, PCA with various r. (A) Large simulation (n=1,000 individuals). (B) Small simulation (n=100) shows overfitting for large r. (C) Family simulation (n=1,000) has admixed founders and large numbers of close relatives from a realistic random 20-generation pedigree. PCA performs poorly compared to LMM: SRMSDp>0 for all r and large AUCPR gap.

In the large admixture simulation, the SRMSDp of PCA is largest when r=0 (no PCs) and decreases rapidly to near zero at r=3, where it stays for up to r=90 (Figure 3A). Thus, PCA has calibrated p-values for r3, smaller than the theoretical optimum for this simulation of r=K-1=9. In contrast, the SRMSDp for LMM starts near zero for r=0, but becomes negative as r increases (p-values are conservative). The AUCPR distribution of PCA is similarly worst at r=0, increases rapidly and peaks at r=3, then decreases slowly for r>3, while the AUCPR distribution for LMM starts near its maximum at r=0 and decreases with r. Although the AUCPR distributions for LMM and PCA overlap considerably at each r, LMM with r=0 has significantly greater AUCPR values than PCA with r=3 (Table 3). However, qualitatively PCA performs nearly as well as LMM in this simulation.

The observed robustness to large r led us to consider smaller sample sizes. A model with large numbers of parameters r should overfit more as r approaches the sample size n. Rather than increase r beyond 90, we reduce individuals to n=100, which is small for typical association studies but may occur in studies of rare diseases, pilot studies, or other constraints. To compensate for the loss of power due to reducing n, we also reduce the number of causal loci (see Trait Simulation), which increases per-locus effect sizes. We found a large decrease in performance for both models as r increases, and best performance for r=1 for PCA and r=0 for LMM (Figure 3B). Remarkably, LMM attains much larger negative SRMSDp values than in our other evaluations. LMM with r=0 is significantly better than PCA (r=1 to 4) in both measures (Table 3), but qualitatively the difference is negligible.

The family simulation adds a 20-generation random family to our large admixture simulation. Only the last generation is studied for association, which contains numerous siblings, first cousins, etc., with the initial admixture structure preserved by geographically biased mating. Our evaluation reveals a sizable gap in both measures between LMM and PCA across all r (Figure 3C). LMM again performs best with r=0 and achieves mean |SRMSDp|<0.01. However, PCA does not achieve mean |SRMSDp|<0.01 at any r, and its best mean AUCPR is considerably worse than that of LMM. Thus, LMM is conclusively superior to PCA, and the only calibrated model, when there is family structure.

Evaluations in real human genotype datasets

Next, we repeat our evaluations with real human genotype data, which differs from our simulations in allele frequency distributions and more complex population structures with greater FST, numerous correlated subpopulations, and potential cryptic family relatedness.

Human Origins has the greatest number and diversity of subpopulations. The SRMSDp and AUCPR distributions in this dataset and FES traits (Figure 4A) most resemble those from the family simulation (Figure 3C). In particular, while LMM with r=0 performed optimally (both measures) and satisfies mean |SRMSDp|<0.01, PCA maintained SRMSDp>0.01 for all r and its AUCPR were all considerably smaller than the best AUCPR of LMM.

Figure 4 with 5 supplements see all
Evaluations in real human genotype datasets with FES traits, high heritability.

Same setup as Figure 3, see that for details. These datasets strongly favor LMM with no PCs over PCA, with distributions that most resemble the family simulation. (A) Human Origins. (B) Human Genome Diversity Panel (HGDP). (C) 1000 Genomes Project.

HGDP has the fewest individuals among real datasets, but compared to Human Origins contains more loci and low-frequency variants. Performance (Figure 4B) again most resembled the family simulations. In particular, LMM with r=0 achieves mean |SRMSDp|<0.01 (p-values are calibrated), while PCA does not, and there is a sizable AUCPR gap between LMM and PCA. Maximum AUCPR values were lowest in HGDP compared to the two other real datasets.

1000 Genomes has the fewest subpopulations but largest number of individuals per subpopulation. Thus, although this dataset has the simplest subpopulation structure among the real datasets, we find SRMSDp and AUCPR distributions (Figure 4C) that again most resemble our earlier family simulation, with mean |SRMSDp|<0.01 for LMM only and large AUCPR gaps between LMM and PCA.

Our results are qualitatively different for RC traits, which had smaller AUCPR gaps between LMM and PCA (Figure 4—figure supplement 1). Maximum AUCPR were smaller in RC compared to FES in Human Origins and 1000 Genomes, suggesting lower power for RC traits across association models. Nevertheless, LMM with r=0 was significantly better than PCA for all measures in the real datasets and RC traits (Table 3).

Evaluations in subpopulation tree simulations fit to human data

To better understand which features of the real datasets lead to the large differences in performance between LMM and PCA, we carried out subpopulation tree simulations. Human subpopulations are related roughly by trees, which induce the strongest correlations, so we fit trees to each real dataset and tested if data simulated from these complex tree structures could recapitulate our previous results (Figure 1). These tree simulations also feature non-uniform ancestral allele frequency distributions, which recapitulated some of the skew for smaller minor allele frequencies of the real datasets (Figure 1C). The SRMSDp and AUCPR distributions for these tree simulations (Figure 5) resembled our admixture simulation more than either the family simulation (Figure 3) or real data results (Figure 4). Both LMM with r=0 and PCA (various r) achieve mean |SRMSDp|<0.01 (Table 3). The AUCPR distributions of both LMM and PCA track closely as r is varied, although there is a small gap resulting in LMM (r=0) besting PCA in all three simulations. The results are qualitatively similar for RC traits (Figure 5—figure supplement 1, Table 3). Overall, these subpopulation tree simulations do not recapitulate the large LMM advantage over PCA observed on the real data.

Figure 5 with 1 supplement see all
Evaluations in subpopulation tree simulations fit to human data with FES traits, high heritability.

Same setup as Figure 3, see that for details. These tree simulations, which exclude family structure by design, do not explain the large gaps in LMM-PCA performance observed in the real data. (A) Human Origins tree simulation. (B) Human Genome Diversity Panel (HGDP) tree simulation. (C) 1000 Genomes Project tree simulation.

Numerous distant relatives explain poor PCA performance in real data

In principle, PCA performance should be determined by the dimension of relatedness, or kinship matrix rank, since PCA is a low-dimensional model whereas LMM can model high-dimensional relatedness without overfitting. We used the Tracy-Widom test (Patterson et al., 2006) with p<0.01 to estimate kinship matrix rank as the number of significant PCs (Figure 6—figure supplement 1A). The true rank of our simulations is slightly underestimated (Table 2), but we confirm that the family simulation has the greatest rank, and real datasets have greater estimates than their respective subpopulation tree simulations, which confirms our hypothesis to some extent. However, estimated ranks do not separate real datasets from tree simulations, as required to predict the observed PCA performance. Moreover, the HGDP and 1000 Genomes rank estimates are 45 and 61, respectively, yet PCA performed poorly for all r90 numbers of PCs (Figure 4). The top eigenvalue explained a proportion of variance proportional to FST (Table 2), but the rest of the top 10 eigenvalues show no clear differences between datasets, except the small simulation had larger variances explained per eigenvalue (expected since it has fewer eigenvalues; Figure 6—figure supplement 1). Comparing cumulative variance explained versus rank fraction across all eigenvalues, all datasets increase from their starting point almost linearly until they reach 1, except the family simulation has much greater variance explained by mid-rank eigenvalues (Figure 6—figure supplement 1). We also calculated the number of PCs that are significantly associated with the trait, and observed similar results, namely that while the family simulation has more significant PCs than the non-family admixture simulations, the real datasets and their tree simulated counterparts have similar numbers of significant PCs (Figure 6—figure supplement 2). Overall, there is no separation between real datasets (where PCA performed poorly) and subpopulation tree simulations (where PCA performed relatively well) in terms of their eigenvalues or kinship matrix rank estimates.

Local kinship, which is recent relatedness due to family structure excluding population structure, is the presumed cause of the LMM to PCA performance gap observed in real datasets but not their subpopulation tree simulation counterparts. Instead of inferring local kinship through increased kinship matrix rank, as attempted in the last paragraph, now we measure it directly using the KING-robust estimator (Manichaikul et al., 2010). We observe more large local kinship in the real datasets and the family simulation compared to the other simulations (Figure 6). However, for real data this distribution depends on the subpopulation structure, since locally related pairs are most likely in the same subpopulation. Therefore, the only comparable curve to each real dataset is their corresponding subpopulation tree simulation, which matches subpopulation structure. In all real datasets, we identified highly related individual pairs with kinship above the 4th degree relative threshold of 0.022 (Manichaikul et al., 2010; Conomos et al., 2016b). However, these highly related pairs are vastly outnumbered by more distant pairs with evident non-zero local kinship as compared to the extreme tree simulation values.

Figure 6 with 2 supplements see all
Local kinship distributions.

Curves are complementary cumulative distribution of lower triangular kinship matrix (self kinship excluded) from KING-robust estimator. Note log x-axis; negative estimates are counted but not shown. Most values are below 4th degree relative threshold. Each real dataset has a greater cumulative than its subpopulation tree simulations.

To try to improve PCA performance, we followed the standard practice of removing 4th degree relatives, which reduced sample sizes between 5% and 10% (Table 4). Only r=0 for LMM and r=20 for PCA were tested, as these performed well in our earlier evaluation, and only FES traits were tested because they previously displayed the large PCA-LMM performance gap. LMM significantly outperforms PCA in all these cases (Wilcoxon paired 1-tailed p<0.01; Figure 7). Notably, PCA still had miscalibrated p-values two of the three real datasets (|SRMSDp|>0.01), the only marginally calibrated case being HGDP which is also the smallest of these datasets. Otherwise, AUCPR and SRMSDp ranges were similar here as in our earlier evaluation. Therefore, the removal of the small number of highly related individual pairs had a negligible effect in PCA performance, so the larger number of more distantly related pairs explain the poor PCA performance in the real datasets.

Figure 7 with 1 supplement see all
Evaluation in real datasets excluding 4th degree relatives, FES traits, high heritability.

Each dataset is a column, rows are measures. Boxplot whiskers are extrema over 50 replicates. First row has |SRMSDp|<0.01 band marked as gray area.

Table 4
Dataset sizes after 4th degree relative filter.
DatasetLoci (m)Ind. (n)Ind. removed (%)
Human Origins189 72226369.8
HGDP758 0098478.8
1000 Genomes1 097 41523904.6

Low heritability and environment simulations

Our main evaluations were repeated with traits simulated under a lower heritability value of h2=0.3. We reduced the number of causal loci in response to this change in heritability, to result in equal average effect size per locus compared to the previous high heritability evaluations (see Trait Simulation). Despite that, these low heritability evaluations measured lower AUCPR values than their high heritability counterparts (Figure 3—figure supplement 2, Figure 3—figure supplement 3, Figure 4—figure supplement 2, Figure 4—figure supplement 3, Figure 7—figure supplement 1). The gap between LMM and PCA was reduced in these evaluations, but the main conclusion of the high heritability evaluation holds for low heritability as well, namely that LMM with r=0 significantly outperforms or ties LMM with r>0 and PCA in all cases (Table 5).

Table 5
Overview of PCA and LMM evaluations for low heritability simulations.
LMM r=0 vs best rPCA vs LMM r=0
DatasetMetricTrait*Cal.Best rp-value §Best rCal.p-value §Best model
Admix. Large sim.|SRMSDp|FESTrue0162True0.00012*LMM
Admix. Small sim.|SRMSDp|FESTrue013True0.27Tie
Admix. Family sim.|SRMSDp|FESTrue0190False3.9e-10*LMM
Human Origins|SRMSDp|FESTrue0181True3.9e-10*LMM
1000 Genomes|SRMSDp|FESTrue0184True3.9e-10*LMM
Admix. Large sim.|SRMSDp|RCTrue0135True0.00094Tie
Admix. Small sim.|SRMSDp|RCTrue013True0.087Tie
Admix. Family sim.|SRMSDp|RCTrue0190False4.1e-10*LMM
Human Origins|SRMSDp|RCTrue0175True0.00016*LMM
1000 Genomes|SRMSDp|RCTrue0141True6.7e-10*LMM
Admix. Large sim.AUCPRFES0130.11Tie
Admix. Small sim.AUCPRFES0100.58Tie
Admix. Family sim.AUCPRFES0172.2e-06*LMM
Human OriginsAUCPRFES01168e-10*LMM
1000 GenomesAUCPRFES60.3442.3e-07*LMM
Admix. Large sim.AUCPRRC0130.14Tie
Admix. Small sim.AUCPRRC0100.1Tie
Admix. Family sim.AUCPRRC0151.9e-06*LMM
Human OriginsAUCPRRC40.16120.003Tie
1000 GenomesAUCPRRC0140.078Tie
  1. *

    FES: Fixed Effect Sizes, RC: Random Coefficients.

  2. Calibrated: whether mean |SRMSDp|<0.01 over 50 replicates.

  3. Value of r (number of PCs) with minimum mean |SRMSDp| or maximum mean AUCPR.

  4. §

    Wilcoxon paired 1-tailed test of distributions (|SRMSDp| or AUCPR) between models in header. Asterisk marks significant value using Bonferroni threshold (p<α/ntests with α=0.01 and ntests=48 is the number of tests in this table).

  5. Tie if no significant difference using Bonferroni threshold.

Lastly, we simulated traits with both low heritability and large environment effects determined by geography and subpopulation labels, so they are strongly correlated to the low-dimensional population structure. For that reason, PCs may be expected to perform better in this setting (in either PCA or LMM). However, we find that both PCA and LMM (even without PCs) increase their AUCPR values compared to the low-heritability evaluations (Figure 8—figure supplement 1; Figure 8 also shows representative numbers of PCs, which performed optimally or nearly so in individual simulations shown in Figure 3—figure supplement 4, Figure 3—figure supplement 5, Figure 4—figure supplement 4, Figure 4—figure supplement 5). p-Value calibration is comparable with or without environment effects, for LMM for all r and for PCA once r is large enough (Figure 8—figure supplement 1). These simulations are the only where we occasionally observed for both metrics a significant, though small, advantage of LMM with PCs versus LMM without PCs (Table 6). Additionally, on RC traits only, PCA significantly outperforms LMM in the three real human datasets (Table 6), the only cases in all of our evaluations where this is observed. For comparison, we also evaluate an ‘oracle’ LMM without PCs but with the finest group labels, the same used to simulate environment, as fixed categorical covariates (‘LMM lab.’), and see much larger AUCPR values than either LMM with PCs or PCA (Figure 8, Figure 3—figure supplement 4, Figure 3—figure supplement 5, Figure 4—figure supplement 4, Figure 4—figure supplement 5, Table 6). However, LMM with labels is often more poorly calibrated than LMM or PCA without labels, which may be since these numerous labels are inappropriately modeled as fixed rather than random effects. Overall, we find that association studies with correlated environment and genetic effects remain a challenge for PCA and LMM, that addition of PCs to an LMM improves performance only marginally, and that if the environment effect is driven by geography or ethnicity then use of those labels greatly improves performance compared to using PCs.

Figure 8 with 1 supplement see all
Evaluation in real datasets excluding 4th degree relatives, FES traits, environment.

Traits simulated with environment effects, otherwise the same as Figure 7. ‘LMM lab.’ includes as fixed effects true groups from which environment was simulated.

Table 6
Overview of PCA and LMM evaluations for environment simulations.
LMM r=0 vs best rPCA vs LMM r=0LMM lab. r=0 vs PCA/LMM
DatasetMetricTrait*Cal.rp-value §rCal.p-value §Best Cal.p-value §Best
Admix. Large sim.|SRMSDp|FESTrue0183True0.38TieTrue1.8e-14*PCA/LMM
Admix. Small sim.|SRMSDp|FESTrue0190True0.001TieFalse1.4e-14*PCA/LMM
Admix. Family sim.|SRMSDp|FESTrue40.1890False3.9e-10*LMMTrue0.066LMM/LMM lab.
Human Origins|SRMSDp|FESTrue93.9e-05*90False1.4e-08*LMMFalse3.9e-10*LMM
1000 Genomes|SRMSDp|FESFalse88.8e-08*85True0.053TieTrue3.9e-10*LMM lab.
Admix. Large sim.|SRMSDp|RCTrue0160True0.033TieTrue6.3e-10*PCA/LMM
Admix. Small sim.|SRMSDp|RCTrue019True0.85TieFalse1.4e-14*PCA/LMM
Admix. Family sim.|SRMSDp|RCTrue50.1490False3.9e-10*LMMTrue0.011LMM/LMM lab.
Human Origins|SRMSDp|RCFalse91.1e-08*90True2.3e-07*PCAFalse3.9e-10*PCA
1000 Genomes|SRMSDp|RCFalse81.6e-08*88True4.9e-09*PCATrue0.09PCA/LMM lab.
Admix. Large sim.AUCPRFES42.4e-06*60.0021Tie1.8e-15*LMM lab.
Admix. Small sim.AUCPRFES30.05540.033Tie0.28Tie
Admix. Family sim.AUCPRFES127e-04633.9e-10*LMM3.9e-10*LMM lab.
Human OriginsAUCPRFES203.7e-06*901.4e-05*LMM3.9e-10*LMM lab.
HGDPAUCPRFES124.3e-06*450.0044Tie3.9e-10*LMM lab.
1000 GenomesAUCPRFES91.9e-08*550.028Tie3.9e-10*LMM lab.
Admix. Large sim.AUCPRRC40.0008550.0018Tie5e-10*LMM lab.
Admix. Small sim.AUCPRRC20.1350.093Tie0.0028Tie
Admix. Family sim.AUCPRRC90.01861.7e-09*LMM3.9e-10*LMM lab.
Human OriginsAUCPRRC220.0039901e-06*PCA3.9e-10*LMM lab.
HGDPAUCPRRC190.0057642.8e-05*PCA3e-07*LMM lab.
1000 GenomesAUCPRRC98.7e-05*871.2e-09*PCA4.4e-10*LMM lab.
  1. *

    FES: Fixed Effect Sizes, RC: Random Coefficients.

  2. Calibrated: whether mean |SRMSDp|<0.01 over 50 replicates.

  3. Value of r (number of PCs) with minimum mean |SRMSDp| or maximum mean AUCPR.

  4. §

    Wilcoxon paired 1-tailed test of distributions (|SRMSDp| or AUCPR) between models in header. Asterisk marks significant value using Bonferroni threshold (p<α/ntests with α=0.01 and ntests=72 is the number of tests in this table).

  5. Tie if no significant difference using Bonferroni threshold; in last column, pairwise ties are specified and “Tie” is three-way tie.


Our evaluations conclusively determined that LMM without PCs performs better than PCA (for any number of PCs) across all scenarios without environment effects, including all real and simulated genotypes and two trait simulation models. Although the addition of a few PCs to LMM does not greatly hurt its performance (except for small sample sizes), they generally did not improve it either (Table 3, Table 5), which agrees with previous observations (Liu et al., 2011; Janss et al., 2012) but contradicts others (Zhao et al., 2007; Price et al., 2010). Our findings make sense since PCs are the eigenvectors of the same kinship matrix that parameterized random effects, so including both is redundant.

The presence of environment effects that are correlated to relatedness presents the only scenario where occasionally PCA and LMM with PCs outperform LMM without PCs (Table 6). It is commonly believed that PCs model such environment effects well (Novembre et al., 2008; Zhang and Pan, 2015; Lin et al., 2021). However, we observe that LMM without PCs models environment effects nearly as well as with PCs (Figure 8), consistent with previous findings (Vilhjálmsson and Nordborg, 2013; Wang et al., 2022) and with environment inflating heritability estimates using LMM (Heckerman et al., 2016). Moreover, modeling the true environment groups as fixed categorical effects always substantially improved AUCPR compared to modeling them with PCs (Figure 8, Table 6). Modeling numerous environment groups as fixed effects does result in deflated p-values (Figure 8, Table 6), which we expect would be avoided by modeling them as random effects, a strategy we chose not to pursue here as it is both a circular evaluation (the true effects were drawn from that model) and out of scope. Overall, including PCs to model environment effects yields limited power gains if at all, even in an LMM, and is no replacement for more adequate modeling of environment whenever possible.

Previous studies found that PCA was better calibrated than LMM for unusually differentiated markers (Price et al., 2010; Wu et al., 2011; Yang et al., 2014), which as simulated were an artificial scenario not based on a population genetics model, and are otherwise believed to be unusual (Sul and Eskin, 2013; Price et al., 2013). Our evaluations on real human data, which contain such loci in relevant proportions if they exist, do not replicate that result. Family relatedness strongly favors LMM, an advantage that probably outweighs this potential PCA benefit in real data.

Relative to LMM, the behavior of PCA fell between two extremes. When PCA performed well, there was a small number of PCs with both calibrated p-values and AUCPR near that of LMM without PCs. Conversely, PCA performed poorly when no number of PCs had either calibrated p-values or acceptably large AUCPR. There were no cases where high numbers of PCs optimized an acceptable AUCPR, or cases with miscalibrated p-values but high AUCPR. PCA performed well in the admixture simulations (without families, both trait models), real human genotypes with RC traits, and the subpopulation tree simulations (both trait models). Conversely, PCA performed poorly in the admixed family simulation (both trait models) and the real human genotypes with FES traits.

PCA assumes that genetic relatedness is restricted to a low-dimensional subspace, whereas LMM can handle high-dimensional relatedness. Thus, PCA performs well in the admixture simulation, which is explicitly low-dimensional (see Genotype simulation from the admixture model), and our subpopulation tree simulations, which are likely well approximated by a few dimensions despite the large number of subpopulations because there are few long branches. Conversely, PCA performs poorly under family structure because its kinship matrix is high-dimensional (Figure 6—figure supplement 1). However, estimating the latent space dimensions of real datasets is challenging because estimated eigenvalues have biased distributions (Hayashi et al., 2018). Kinship matrix rank estimated using the Tracy-Widom test (Patterson et al., 2006) did not fully predict the datasets that PCA performs well on. In contrast, estimated local kinship finds considerable cryptic family relatedness in all real human datasets and better explains why PCA performs poorly there. The trait model also influences the relative performance of PCA, so genotype-only parameters (eigenvalues or local kinship) alone cannot tell the full story. There are related tests for numbers of dimensions that consider the trait which we did not consider, including the Bayesian information criterion for the regression with PCs against the trait (Zhu and Yu, 2009). Additionally, PCA and LMM goodness of fit could be compared using the coefficient of determination generalized for LMMs (Sun et al., 2010).

PCA is at best underpowered relative to LMMs, and at worst miscalibrated regardless of the numbers of PCs included, in real human genotype tests. Among our simulations, such poor performance occurred only in the admixed family. Local kinship estimates reveal considerable family relatedness in the real datasets absent in the corresponding subpopulation tree simulations. Admixture is also absent in our tree simulations, but our simulations and theory show that admixture is well handled by PCA. Hundreds of close relative pairs have been identified in 1000 Genomes (Gazal et al., 2015; Al Khudhair et al., 2015; Fedorova et al., 2016; Schlauch et al., 2017), but their removal does not improve PCA performance sufficiently in our tests, so the larger number of more distantly related pairs are PCA’s most serious obstacle in practice. Distant relatives are expected to be numerous in any large human dataset (Henn et al., 2012; Shchur and Nielsen, 2018; Loh et al., 2018). Our FES trait tests show that family relatedness is more challenging when rarer variants have larger coefficients. Overall, the high relatedness dimensions induced by family relatedness is the key challenge for PCA association in modern datasets that is readily overcome by LMM.

Our tests also found PCA robust to large numbers of PCs, far beyond the optimal choice, agreeing with previous anecdotal observations (Price et al., 2006; Kang et al., 2010), in contrast to using too few PCs for which there is a large performance penalty. The exception was the small sample size simulation, where only small numbers of PCs performed well. In contrast, LMM is simpler since there is no need to choose the number of PCs. However, an LMM with a large number of covariates may have conservative p-values, as observed for LMM with large numbers of PCs, which is a weakness of the score test used by the LMM we evaluated that may be overcome with other statistical tests. Simulations or post hoc evaluations remain crucial for ensuring that statistics are calibrated.

There are several variants of the PCA and LMM analyses, most designed for better modeling linkage disequilibrium (LD), that we did not evaluate directly, in which PCs are no longer exactly the top eigenvectors of the kinship matrix (if estimated with different approaches), although this is not a crucial aspect of our arguments. We do not consider the case where samples are projected onto PCs estimated from an external sample (Privé et al., 2020), which is uncommon in association studies, and whose primary effect is shrinkage, so if all samples are projected then they are all equally affected and larger regression coefficients compensate for the shrinkage, although this will no longer be the case if only a portion of the sample is projected onto the PCs of the rest of the sample. Another approach tests PCs for association against every locus in the genome in order to identify and exclude PCs that capture LD structure (which is localized) instead of ancestry (which should be present across the genome; Privé et al., 2020); a previous proposal removes LD using an autocorrelation model prior to estimating PCs (Patterson et al., 2006). These improved PCs remain inadequate models of family relatedness, so an LMM will continue to outperform them in that setting. Similarly, the leave-one-chromosome-out (LOCO) approach for estimating kinship matrices for LMMs prevents the test locus and loci in LD with it from being modeled by the random effect as well, which is called ‘proximal contamination’ (Lippert et al., 2011; Yang et al., 2014). While LOCO kinship estimates vary for each chromosome, they continue to model family relatedness, thus maintaining their key advantage over PCA. The LDAK model estimates kinship instead by weighing loci taking LD into account (Speed et al., 2012). LD effects must be adjusted for, if present, so in unfiltered data we advise the previous methods be applied. However, in this work, simulated genotypes do not have LD, and the real datasets were filtered to remove LD, so here there is no proximal contamination and LD confounding is minimized if present at all, so these evaluations may be considered the ideal situation where LD effects have been adjusted successfully, and in this setting LMM outperforms PCA. Overall, these alternative PCs or kinship matrices differ from their basic counterparts by either the extent to which LD influences the estimates (which may be a confounder in a small portion of the genome, by definition) or by sampling noise, neither of which are expected to change our key conclusion.

One of the limitations of this work include relatively small sample sizes compared to modern association studies. However, our conclusions are not expected to change with larger sample sizes, as cryptic family relatedness will continue to be abundant in such data, if not increase in abundance, and thus give LMMs an advantage over PCA (Henn et al., 2012; Shchur and Nielsen, 2018; Loh et al., 2018). One reason PCA has been favored over classic LMMs is because PCA’s runtime scales much better with increasing sample size. However, recent approaches not tested in this work have made LMMs more scalable and applicable to biobank-scale data (Loh et al., 2015; Zhou et al., 2018; Mbatchou et al., 2021), so one clear next step is carefully evaluating these approaches in simulations with larger sample sizes. A different benefit for including PCs were recently reported for BOLT-LMM, which does not result in greater power but rather in reduced runtime, a property that may be specific to its use of scalable algorithms such as conjugate gradient and variational Bayes (Loh et al., 2018). Many of these newer LMMs also no longer follow the infinitesimal model of the basic LMM (Loh et al., 2015; Mbatchou et al., 2021), and employ novel approximations, which are features not evaluated in this work and worthy of future study.

Another limitation of this work is ignoring rare variants, a necessity given our smaller sample sizes, where rare variant association is miscalibrated and underpowered. Using simulations mimicking the UK Biobank, recent work has found that rare variants can have a more pronounced structure than common variants, and that modeling this rare variant structure (with either PCA and LMM) may better model environment confounding, reduce inflation in association studies, and ameliorate stratification in polygenic risk scores (Zaidi and Mathieson, 2020). Better modeling rare variants and their structure is a key next step in association studies.

The largest limitation of our work is that we only considered quantitative traits. Previous evaluations involving case-control traits tended to report PCA-LMM ties or mixed results, an observation potentially confounded by the use of low-dimensional simulations without family relatedness (Table 1). An additional concern is case-control ascertainment bias and imbalance, which appears to affect LMMs more severely, although recent work appears to solve this problem (Yang et al., 2014; Zhou et al., 2018). Future evaluations should aim to include our simulations and real datasets, to ensure that previous results were not biased in favor of PCA by not simulating family structure or larger coefficients for rare variants that are expected for diseases by various selection models.

Overall, our results lead us to recommend LMM over PCA for association studies in general. Although PCA offer flexibility and speed compared to LMM, additional work is required to ensure that PCA is adequate, including removal of close relatives (lowering sample size and wasting resources) followed by simulations or other evaluations of statistics, and even then PCA may perform poorly in terms of both type I error control and power. The large numbers of distant relatives expected of any real dataset all but ensures that PCA will perform poorly compared to LMM (Henn et al., 2012; Shchur and Nielsen, 2018; Loh et al., 2018). Our findings also suggest that related applications such as polygenic models may enjoy gains in power and accuracy by employing an LMM instead of PCA to model relatedness (Rakitsch et al., 2013; Qian et al., 2020). PCA remains indispensable across population genetics, from visualizing population structure and performing quality control to its deep connection to admixture models, but the time has come to limit its use in association testing in favor of LMM or other, richer models capable of modeling all forms of relatedness.

Materials and methods

The complex trait model and PCA and LMM approximations

Request a detailed protocol

Let xij{0,1,2} be the genotype at the biallelic locus i for individual j, which counts the number of reference alleles. Suppose there are n individuals and m loci, X=(xij) is their m×n genotype matrix, and y is the length-n column vector of individual trait values. The additive linear model for a quantitative (continuous) trait is:

(1) y=1α+Xβ+Zη+ϵ,

where 1 is a length-n vector of ones, α is the scalar intercept coefficient, β is the length-m vector of locus coefficients, Z is a design matrix of environment effects and other covariates, η is the vector of environment coefficients, ϵ is a length-n vector of residuals, and the superscript prime symbol () denotes matrix transposition. The residuals follow ϵjNormal(0,σϵ2) independently per individual j, for some σϵ2.

The full model of Equation 1, which has a coefficient for each of the m loci, is underdetermined in current datasets where mn. The PCA and LMM models, respectively, approximate the full model fit at a single locus i:

(2) PCA:y=1α+xiβi+Urγr+Zη+ϵ,
(3) LMM:y=1α+xiβi+s+Zη+ϵ,sNormal(0,2σs2ΦT),

where xi is the length-n vector of genotypes at locus i only, βi is the locus coefficient, Ur is an n×r matrix of PCs, γr is the length-r vector of PC coefficients, s is a length-n vector of random effects, ΦT=(φjkT) is the n×n kinship matrix conditioned on the ancestral population T, and σs2 is a variance factor. Both models condition the regression of the focal locus i on an approximation of the total polygenic effect Xβ with the same covariance structure, which is parameterized by the kinship matrix. Under the kinship model, genotypes are random variables obeying

(4) E[xi|T]=2piT1,Cov(xi|T)=4piT(1-piT)ΦT,

where piT is the ancestral allele frequency of locus i (Malécot, 1948; Wright, 1949; Jacquard, 1970; Astle and Balding, 2009). Assuming independent loci, the covariance of the polygenic effect is


which is readily modeled by the LMM random effect s, where the difference in mean is absorbed by the intercept. Alternatively, consider the eigendecomposition of the kinship matrix ΦT=UΛU where U is the n×n eigenvector matrix and Λ is the n×n diagonal matrix of eigenvalues. The random effect can be written as


which follows from the affine transformation property of multivariate normal distributions. Therefore, the PCA term Urγr can be derived from the above equation under the additional assumption that the kinship matrix has approximate rank r and the coefficients γr are fit without constraints. In contrast, the LMM uses all eigenvectors, while effectively shrinking their coefficients γLMM as all random effects models do, although these parameters are marginalized (Astle and Balding, 2009; Janss et al., 2012; Hoffman and Dubé, 2013; Zhang and Pan, 2015). PCA has more parameters than LMM, so it may overfit more: ignoring the shared terms in Equation 2 and Equation 3, PCA fits r parameters (length of γ), whereas LMMs fit only one (σs2).

In practice, the kinship matrix used for PCA and LMM is estimated with variations of a method-of-moments formula applied to standardized genotypes XS, which is derived from Equation 4:

(5) XS=(xij2p^iT4p^iT(1p^iT)),Φ^T=1mXSXS,

where the unknown piT is estimated by p^iT=12nj=1nxij (Price et al., 2006; Kang et al., 2008; Kang et al., 2010; Yang et al., 2011; Zhou and Stephens, 2012; Yang et al., 2014; Loh et al., 2015; Sul et al., 2018; Zhou et al., 2018). However, this kinship estimator has a complex bias that differs for every individual pair, which arises due to the use of this estimated p^iT(Ochoa and Storey, 2021; Ochoa and Storey, 2019). Nevertheless, in PCA and LMM these biased estimates perform as well as unbiased ones (Hou et al., 2023b).

We selected fast and robust software implementing the basic PCA and LMM models. PCA association was performed with plink2 (Chang et al., 2015). The quantitative trait association model is a linear regression with covariates, evaluated using the t-test. PCs were calculated with plink2, which equal the top eigenvectors of Equation 5 after removing loci with minor allele frequency MAF<0.1.

LMM association was performed using GCTA (Yang et al., 2011; Yang et al., 2014). Its kinship estimator equals Equation 5. PCs were calculated using GCTA from its kinship estimate. Association significance is evaluated with a score test. In the small simulation only, GCTA with large numbers of PCs had convergence and singularity errors in some replicates, which were treated as missing data.


Every simulation was replicated 50 times, drawing anew all genotypes (except for real datasets) and traits. Below we use the notation fAB for the inbreeding coefficient of a subpopulation A from another subpopulation B ancestral to A. In the special case of the total inbreeding of A, fAT, T is an overall ancestral population, which is ancestral to every individual under consideration, such as the most recent common ancestor (MRCA) population.

Genotype simulation from the admixture model

Request a detailed protocol

The basic admixture model is as described previously (Ochoa and Storey, 2021) and is implemented in the R package bnpsd. Both Large and Family simulations have n=1,000 individuals, while Small has n=100. The number of loci is m=100,000. Individuals are admixed from K=10 intermediate subpopulations, or ancestries. Each subpopulation Su (u{1,,K}) is at coordinate u and has an inbreeding coefficient fSuT=uτ for some τ. Ancestry proportions qju for individual j and Su arise from a random walk with spread σ on the 1D geography, and τ and σ are fit to give FST=0.1 and mean kinship θ¯T=0.5FST for the admixed individuals (Ochoa and Storey, 2021). Random ancestral allele frequencies piT, subpopulation allele frequencies piSu, individual-specific allele frequencies πij, and genotypes xij are drawn from this hierarchical model:


where this Beta is the Balding-Nichols distribution (Balding and Nichols, 1995) with mean piT and variance piT(1-piT)fSuT. Fixed loci (i where xij=0 for all j, or xij=2 for all j) are drawn again from the model, starting from piT, iterating until no loci are fixed. Each replicate draws a genotypes starting from piT.

As a brief aside, we prove that global ancestry proportions as covariates is equivalent in expectation to using PCs under the admixture model. Note that the latent space of X, which is the subspace to which the data is constrained by the admixture model, is given by (πij), which has K dimensions (number of columns of Q=(qju)), so the top K PCs span this space. Since associations include an intercept term (1α in Equation 2), estimated PCs are orthogonal to 1 (note Φ^T1=0 because XS1=0), and the sum of rows of Q sums to one, then only K-1 PCs plus the intercept are needed to span the latent space of this admixture model.

Genotype simulation from random admixed families

Request a detailed protocol

We simulated a pedigree with admixed founders, no close relative pairings, assortative mating based on a 1D geography (to preserve admixture structure), random family sizes, and arbitrary numbers of generations (20 here). This simulation is implemented in the R package simfam. Generations are drawn iteratively. Generation 1 has n=1000 individuals from the above admixture simulation ordered by their 1D geography. Local kinship measures pedigree relatedness; in the first generation, everybody is locally unrelated and outbred. Individuals are randomly assigned sex. In the next generation, individuals are paired iteratively, removing random males from the pool of available males and pairing them with the nearest available female with local kinship <1/43 (stay unpaired if there are no matches), until there are no more available males or females. Let n=1000 be the desired population size, nm=1 the minimum number of children per family and nf the number of families (paired parents) in the current generation, then the number of additional children (beyond the minimum) is drawn from Poisson(n/nf-nm). Let δ be the difference between desired and current population sizes. If δ>0, then δ random families are incremented by 1. If δ<0, then |δ| random families with at least nm+1 children are decremented by 1. If |δ| exceeds the number of families, all families are incremented or decremented as needed and the process is iterated. Children are assigned sex randomly, and are reordered by the average coordinate of their parents. Children draw alleles from their parents independently per locus. A new random pedigree is drawn for each replicate, as well as new founder genotypes from the admixture model.

Genotype simulation from a subpopulation tree model

Request a detailed protocol

This model draws subpopulations allele frequencies from a hierarchical model parameterized by a tree, which is also implemented in bnpsd and relies on the R package ape for general tree data structures and methods (Paradis and Schliep, 2019). The ancestral population T is the root, and each node is a subpopulation Sw indexed arbitrarily. Each edge between Sw and its parent population Pw has an inbreeding coefficient fSwPw. PiT are drawn from a given distribution, which is constructed to mimic each real dataset in Appendix 1. Given the allele frequencies piPw of the parent population, Sw’s allele frequencies are drawn from:


Individuals j in Sw draw genotypes from its allele frequency: xij|piSwBinomial(2,piSw). Loci with MAF<0.01 are drawn again starting from the piT distribution, iterating until no such loci remain.

Fitting subpopulation tree to real data

Request a detailed protocol

We developed new methods to fit trees to real data based on unbiased kinship estimates from popkin, implemented in bnpsd. A tree with given inbreeding coefficients fSwPw for its edges (between subpopulation Sw and its parent Pw) gives rise to a coancestry matrix ϑuvT for a subpopulation pair (Su,Sv), and the goal is to recover these edge inbreeding coefficients from coancestry estimates. Coancestry values are total inbreeding coefficients of the MRCA population of each subpopulation pair. Therefore, we calculate fSwT for every Sw recursively from the root as follows. Nodes with parent Pw=T are already as desired. Given fPwT, the desired fSwT is calculated via the ‘additive edge’ δw (Ochoa and Storey, 2021):

(6) fSwT=fPwT+δw,δw=fSwPw(1fPwT).

These δw0 because 0fSwPw,fPwT1 for every w. Edge inbreeding coefficients can be recovered from additive edges: fSwPw=δw/(1-fPwT). Overall, coancestry values are sums of δw over common ancestor nodes,

(7) ϑuvT=wδwIw(u,v),

where the sum includes all w, and Iw(u,v) equals 1 if Sw is a common ancestor of Su,Sv, 0 otherwise. Note that Iw(u,v) reflects tree topology and δw edge values.

To estimate population-level coancestry, first kinship (φ^jkT) is estimated using popkin (Ochoa and Storey, 2021). Individual coancestry (θ^jkT) is estimated from kinship using

(8) θ^jkT={φ^jkTifkj,f^jT=2φ^jjT1ifk=j.

Lastly, coancestry ϑ^uvT between subpopulations are averages of individual coancestry values:


Topology is estimated with hierarchical clustering using the weighted pair group method with arithmetic mean (Sokal and Michener, 1958), with distance function d(Su,Sv)=max{ϑ^uvT}-ϑ^uvT, which succeeds due to the monotonic relationship between node depth and coancestry (Equation 7). This algorithm recovers the true topology from the true coancestry values, and performs well for estimates from genotypes.

To estimate tree edge lengths, first δw are estimated from ϑ^uvT and the topology using Equation 7 and non-negative least squares linear regression (Lawson and Hanson, 1974) (implemented in nnls; Mullen, 2012) to yield non-negative δw, and fSwPw are calculated from δw by reversing Equation 5. To account for small biases in coancestry estimation, an intercept term δ0 is included (I0(u,v)=1 for all u,v), and when converting δw to fSwPw, δ0 is treated as an additional edge to the root, but is ignored when drawing allele frequencies from the tree.

Trait simulation

Request a detailed protocol

Traits are simulated from the quantitative trait model of Equation 1, with novel bias corrections for simulating the desired heritability from real data relying on the unbiased kinship estimator popkin (Ochoa and Storey, 2021). This simulation is implemented in the R package simtrait. All simulations have a fixed narrow-sense heritability of h2, a variance proportion due to environment effects ση2, and residuals are drawn from ϵjNormal(0,σϵ2) with σϵ2=1-h2-ση2. The number of causal loci m1, which determines the average coefficient size, is chosen with the heuristic formula m1=round(nh2/8), which empirically balances power well with varying n and h2. The set of causal loci C is drawn anew for each replicate, from loci with MAF0.01 to avoid rare causal variants, which are not discoverable by PCA or LMM at the sample sizes we considered. Letting viT=piT(1-piT), the effect size of locus i equals 2viTβi2, its contribution of the trait variance (Park et al., 2010). Under the fixed effect sizes (FES) model, initial causal coefficients are


for known piT; otherwise viT is replaced by the unbiased estimator (Ochoa and Storey, 2021) v^iT=p^iT(1-p^iT)/(1-φ¯T), where φ¯T is the mean kinship estimated with popkin. Each causal locus is multiplied by –1 with probability 0.5. Alternatively, under the random coefficients (RC) model, initial causal coefficients are drawn independently from βiNormal(0,1). For both models, the initial genetic variance is σ02=iC2viTβi2, replacing viT with v^iT for unknown piT (so σ02 is an unbiased estimate), so we multiply every initial βi by hσ0 to have the desired heritability. Lastly, for known piT, the intercept coefficient is α=-iC2piTβi. When piT are unknown, p^iT should not replace piT since that distorts the trait covariance (for the same reason the standard kinship estimator in Equation 5 is biased), which is avoided with


Simulations optionally included multiple environment group effects, similarly to previous models (Zhang and Pan, 2015; Wang et al., 2022), as follows. Each independent environment i has predefined groups, and each group g has random coefficients drawn independent from ηgiNormal(0,σηi2) where σηi2 is a specified variance proportion for environment i. Z has individuals along columns and environment-groups along rows, and it contains indicator variables: 1 if the individual belongs to the environment-group, 0 otherwise.

We performed trait simulations with the following variance parameters (Table 7): high heritability used h2=0.8 and no environment effects; low heritability used h2=0.3 and no environment effects; lastly, environment used h2=0.3,ση12=0.3,ση22=0.2 (total ση2=ση12+ση22=0.5). For real genotype datasets, the groups are the continental (environment 1) and fine-grained (environment 2) subpopulation labels given (see next subsection). For simulated genotypes, we created these labels by grouping by the index j (geographical coordinate) of each simulated individual, assigning group g=ceiling(jki/n) where ki is the number of groups in environment i, and we selected k1=5 and k2=25 to mimic the number of groups in each level of 1000 Genomes (Table 2).

Table 7
Variance parameters of trait simulations.
Trait variance typeh2ση2σϵ2
High heritability0.80.00.2
Low heritability0.30.00.7

Real human genotype datasets

Request a detailed protocol

The three datasets were processed as before (Ochoa and Storey, 2019; summarized below), except with an additional filter so loci are in approximate linkage equilibrium and rare variants are removed. All processing was performed with plink2 (Chang et al., 2015), and analysis was uniquely enabled by the R packages BEDMatrix (Grueneberg and de Los Campos, 2019) and genio. Each dataset groups individuals in a two-level hierarchy: continental and fine-grained subpopulations. Final dataset sizes are in Table 2.

We obtained the full (including non-public) Human Origins by contacting the authors and agreeing to their usage restrictions. The Pacific data (Skoglund et al., 2016) was obtained separately from the rest (Lazaridis et al., 2014; Lazaridis et al., 2016), and datasets were merged using the intersection of loci. We removed ancient individuals, and individuals from singleton and non-native subpopulations. Non-autosomal loci were removed. Our analysis of both the whole-genome sequencing (WGS) version of HGDP (Bergström et al., 2020) and the high-coverage NYGC version of 1000 Genomes (Fairley et al., 2020) was restricted to autosomal biallelic SNP loci with filter “PASS”.

Since our evaluations assume uncorrelated loci, we filtered each real dataset with plink2 using parameters “--indep-pairwise 1000kb 0.3”, which iteratively removes loci that have a greater than 0.3 squared correlation coefficient with another locus that is within 1000 kb, stopping until no such loci remain. Since all real datasets have numerous rare variants, while PCA and LMM are not able to detect associations involving rare variants, we removed all loci with MAF<0.01. Lastly, only HGDP had loci with over 10% missingness removed, as they were otherwise 17% of remaining loci (for Human Origins and 1000 Genomes they were under 1% of loci so they were not removed). Kinship matrix rank and eigenvalues were calculated from popkin kinship estimates. Eigenvalues were assigned p-values with twstats of the Eigensoft package (Patterson et al., 2006), and kinship matrix rank was estimated as the largest number of consecutive eigenvalue from the start that all satisfy p<0.01 (p-values did not increase monotonically). For the evaluation with close relatives removed, each dataset was filtered with plink2 with option “--king-cutoff” with cutoff 0.02209709 (=2-11/2) for removing up to 4th degree relatives using KING-robust (Manichaikul et al., 2010), and MAF<0.01 filter is reapplied (Table 4).

Evaluation of performance

Request a detailed protocol

All approaches are evaluated using two complementary metrics: SRMSDp quantifies p-value uniformity, and AUCPR measures causal locus classification performance and reflects power while ranking miscalibrated models fairly. These measures are more robust alternatives to previous measures from the literature (Appendix 2), and are implemented in simtrait.

P-values for continuous test statistics have a uniform distribution when the null hypothesis holds, a crucial assumption for type I error and FDR control (Storey, 2003; Storey and Tibshirani, 2003). We use the Signed Root Mean Square Deviation (SRMSDp) to measure the difference between the observed null p-value quantiles and the expected uniform quantiles:


where m0=m-m1 is the number of null (non-causal) loci, here i indexes null loci only, p(i) is the i th ordered null p-value, ui=(i-0.5)/m0 is its expectation, pmedian is the median observed null p-value, umedian=12 is its expectation, and sgn is the sign function (1 if umedianpmedian, –1 otherwise). Thus, SRMSDp=0 corresponds to calibrated p-values, SRMSDp>0 indicate anti-conservative p-values, and SRMSDp<0 are conservative p-values. The maximum SRMSDp is achieved when all p-values are zero (the limit of anti-conservative p-values), which for infinite loci approaches


The same value with a negative sign occurs for all p-values of 1.

Precision and recall are standard performance measures for binary classifiers that do not require calibrated p-values (Grau et al., 2015). Given the total numbers of true positives (TP), false positives (FP) and false negatives (FN) at some threshold or parameter t, precision and recall are


Precision and Recall trace a curve as t is varied, and the area under this curve is AUCPR. We use the R package PRROC to integrate the correct non-linear piecewise function when interpolating between points. A model obtains the maximum AUCPR=1 if there is a t that classifies all loci perfectly. In contrast, the worst models, which classify at random, have an expected precision (=AUCPR) equal to the overall proportion of causal loci: m1/m.

Data and code availability

Request a detailed protocol

The data and code generated during this study are available on GitHub at (copy archived at Ochoa, 2023). The public subset of Human Origins is available on the Reich Lab website at; non-public samples have to be requested from David Reich. The WGS version of HGDP was downloaded from the Wellcome Sanger Institute FTP site at The high-coverage version of the 1000 Genomes Project was downloaded from

Web resources

Request a detailed protocol

plink2, ; GCTA, ; Eigensoft, ; bnpsd, ; simfam, ; simtrait, ; genio, ; popkin, ; ape, ; nnls, ; PRROC, ; BEDMatrix,

Appendix 1

Fitting ancestral allele frequency distribution to real data

We calculated p^iT distributions of each real dataset. However, population structure increases the variance of these sample p^iT relative to the true piT(Ochoa and Storey, 2021). We present a new algorithm for constructing a new distribution based on the input data but with the lower variance of the true ancestral distribution. Suppose the piT distribution over loci i satisfies E[piT]=12 and Var(piT)=VT. The sample allele frequency p^iT, conditioned on piT, satisfies


where φ¯T=1n2j=1nk=1nφjkT is the mean kinship over all individual (Ochoa and Storey, 2021). The unconditional moments of p^iT follow from the laws of total expectation and variance: E[p^iT]=12 and


Since VT14 and φ¯T0, then WTVT. Thus, the goal is to construct a new distribution with the original, lower variance of

(9) VT=WT14φ¯T1φ¯T.

We use the unbiased estimator W^T=1mi=1m(p^iT-12)2, while φ¯T is calculated from the tree parameters: the subpopulation coancestry matrix (Equation 7), expanded from subpopulations to individuals, the diagonal converted to kinship (reversing Equation 5), and the matrix averaged. However, since our model ignores the MAF filters imposed in our simulations, φ¯T was adjusted. For Human Origins the true model φ¯T of 0.143 was used. For 1000 Genomes and HGDP the true φ¯T are 0.126 and 0.124, respectively, but 0.4 for both produced a better fit.

Lastly, we construct new allele frequencies,


by a weighted average of p^iT and q(0,1) drawn independently from a different distribution. E[q]=12 is required to have E[p*]=12. The resulting variance is


which we equate to the desired VT (Equation 9) and solve for w. For simplicity, we also set Var(q)=VT, which is achieved with:


Although w=0 yields Var(p*)=VT, we use the second root of the quadratic equation to use p^iT:


Appendix 2

Comparisons between SRMSDp, AUCPR, and evaluation measures from the literature

2.1 The inflation factor λ

Test statistic inflation has been used to measure model calibration (Astle and Balding, 2009; Price et al., 2010). The inflation factor λ is defined as the median χ2 association statistic divided by theoretical median under the null hypothesis (Devlin and Roeder, 1999). To compare p-values from non-χ2 tests (such as t-statistics), λ can be calculated from p-values using


where pmedian is the median observed p-value (including causal loci), umedian=12 is its null expectation, and F is the χ2 cumulative density function (F-1 is the quantile function).

To compare λ and SRMSDp directly, for simplicity assume that all p-values are null. In this case, calibrated p-values give λ=1 and SRMSDp=0. However, non-uniform p-values with the expected median, such as from genomic control (Devlin and Roeder, 1999), result in λ=1, but SRMSDp0 except for uniform p-values, a key flaw of λ that SRMSDp overcomes. Inflated statistics (anti-conservative p-values) give λ>1 and SRMSDp>0. Deflated statistics (conservative p-values) give λ<1 and SRMSDp<0. Thus, λ1 always implies SRMSDp0 (where λ-1 and SRMSDp have the same sign), but not the other way around. Overall, λ depends only on the median p-value, while SRMSDp uses the complete distribution. However, SRMSDp requires knowing which loci are null, so unlike λ it is only applicable to simulated traits.

2.2 Empirical comparison of SRMSDp and λ

There is a near one-to-one correspondence between λ and SRMSDp in our data (Figure 2—figure supplement 1). PCA tended to be inflated (λ>1 and SRMSDp>0) whereas LMM tended to be deflated (λ<1 and SRMSDp<0), otherwise the data for both models fall on the same contiguous curve. We fit a sigmoidal function to this data,

(10) SRMSDp(λ)=aλb1λb+1,

which for a,b>0 satisfies SRMSDp(λ=1)=0 and reflects log(λ) about zero (λ=1):


We fit this model to λ>1 only since it was less noisy and of greater interest, and obtained the curve shown in Figure 2—figure supplement 1 with a=0.564 and b=0.619. The value λ=1.05, a common threshold for benign inflation (Price et al., 2010), corresponds to SRMSDp=0.0085 according to Equation 10. Conversely, SRMSDp=0.01, serving as a simpler rule of thumb, corresponds to λ=1.06.

2.3 Type I error rate

The type I error rate is the proportion of null p-values with pt. Calibrated p-values have type I error rate near t, which may be evaluated with a binomial test. This measure may give different results for different t, for example be significantly miscalibrated only for large t (due to lack of power for smaller t), and it requires large simulations to estimate well as it depends on the tail of the distribution. In contrast, SRMSDp uses the entire distribution so it is easier to estimate, SRMSDp=0 guarantees calibrated type I error rates at all t, while large |SRMSDp| indicates incorrect type I errors for a range of t. Empirically, we find the expected agreement and monotonic relationship between SRMSDp and type I error rate (Figure 2—figure supplement 2).

2.4 Statistical power and comparison to AUCPR

Power is the probability that a test is declared significant when the alternative hypothesis H1 holds. At a p-value threshold t, power equals


F(t) is a cumulative function, so it is monotonically increasing and has an inverse. Like type I error control, power may rank models differently depending on t, and it is also harder to estimate than AUCPR because power depends on the tail of the distribution.

Power is not meaningful when p-values are not calibrated. To establish a clear connection to AUCPR, assume calibrated (uniform) null p-values: Pr(p<t|H0)=t. TPs, FPs, and FNs at t are


where π0=Pr(H0) is the proportion of null cases and π1=1-π0 of alternative cases. Therefore,


Noting that t=F1(Recall), precision can be written as a function of recall, the power function, and constants:


This last form leads most clearly to AUCPR=01Precision(Recall)dRecall .

Lastly, consider a simple yet common case in which model A is uniformly more powerful than model B:FA(t)>FB(t) for every t. Therefore FA1(Recall)<FB1(Recall) for every recall value. This ensures that the precision of A is greater than that of B at every recall value, so AUCPR is greater for A than B. Thus, AUCPR ranks calibrated models according to power.

Empirically, we find the predicted positive correlation between AUCPR and calibrated power (Figure 2—figure supplement 3). The correlation is clear when considered separately per dataset, but the slope varies per dataset, which is expected because the proportion of alternative cases π1 varies per dataset.

Data availability

The current manuscript is a computational study, so no data have been generated for this manuscript. Code is available at (copy archived at Ochoa, 2023).

The following previously published data sets were used
    1. Bergstrom A
    (2020) Wellcome Sanger Institute
    ID wgs.20190516/. Human Genome Diversity Panel, whole-genome sequencing version.


  1. Book
    1. Hayashi K
    2. Yuan KH
    3. Liang L
    (2018) On the bias in Eigenvalues of sample covariance matrix
    In: Wiberg M, Culpepper S, Janssen R, González J, Molenaar D, editors. Quantitative Psychology Springer Proceedings in Mathematics & Statistics. Cham: Springer International Publishing. pp. 221–233.
  2. Book
    1. Jacquard A
    Structures Génétiques Des Populations
    Paris: Masson et Cie.
  3. Book
    1. Jolliffe IT
    Principal Component Analysis
    New York: Springer-Verlag.
  4. Book
    1. Lawson CL
    2. Hanson RJ
    Solving Least Squares Problems
    Englewood Cliffs: Prentice Hall.
    1. Mahajan A
    2. Spracklen CN
    3. Zhang W
    4. Ng MCY
    5. Petty LE
    6. Kitajima H
    7. Yu GZ
    8. Rüeger S
    9. Speidel L
    10. Kim YJ
    11. Horikoshi M
    12. Mercader JM
    13. Taliun D
    14. Moon S
    15. Kwak S-H
    16. Robertson NR
    17. Rayner NW
    18. Loh M
    19. Kim B-J
    20. Chiou J
    21. Miguel-Escalada I
    22. Della Briotta Parolo P
    23. Lin K
    24. Bragg F
    25. Preuss MH
    26. Takeuchi F
    27. Nano J
    28. Guo X
    29. Lamri A
    30. Nakatochi M
    31. Scott RA
    32. Lee J-J
    33. Huerta-Chagoya A
    34. Graff M
    35. Chai J-F
    36. Parra EJ
    37. Yao J
    38. Bielak LF
    39. Tabara Y
    40. Hai Y
    41. Steinthorsdottir V
    42. Cook JP
    43. Kals M
    44. Grarup N
    45. Schmidt EM
    46. Pan I
    47. Sofer T
    48. Wuttke M
    49. Sarnowski C
    50. Gieger C
    51. Nousome D
    52. Trompet S
    53. Long J
    54. Sun M
    55. Tong L
    56. Chen W-M
    57. Ahmad M
    58. Noordam R
    59. Lim VJY
    60. Tam CHT
    61. Joo YY
    62. Chen C-H
    63. Raffield LM
    64. Lecoeur C
    65. Prins BP
    66. Nicolas A
    67. Yanek LR
    68. Chen G
    69. Jensen RA
    70. Tajuddin S
    71. Kabagambe EK
    72. An P
    73. Xiang AH
    74. Choi HS
    75. Cade BE
    76. Tan J
    77. Flanagan J
    78. Abaitua F
    79. Adair LS
    80. Adeyemo A
    81. Aguilar-Salinas CA
    82. Akiyama M
    83. Anand SS
    84. Bertoni A
    85. Bian Z
    86. Bork-Jensen J
    87. Brandslund I
    88. Brody JA
    89. Brummett CM
    90. Buchanan TA
    91. Canouil M
    92. Chan JCN
    93. Chang L-C
    94. Chee M-L
    95. Chen J
    96. Chen S-H
    97. Chen Y-T
    98. Chen Z
    99. Chuang L-M
    100. Cushman M
    101. Das SK
    102. de Silva HJ
    103. Dedoussis G
    104. Dimitrov L
    105. Doumatey AP
    106. Du S
    107. Duan Q
    108. Eckardt K-U
    109. Emery LS
    110. Evans DS
    111. Evans MK
    112. Fischer K
    113. Floyd JS
    114. Ford I
    115. Fornage M
    116. Franco OH
    117. Frayling TM
    118. Freedman BI
    119. Fuchsberger C
    120. Genter P
    121. Gerstein HC
    122. Giedraitis V
    123. González-Villalpando C
    124. González-Villalpando ME
    125. Goodarzi MO
    126. Gordon-Larsen P
    127. Gorkin D
    128. Gross M
    129. Guo Y
    130. Hackinger S
    131. Han S
    132. Hattersley AT
    133. Herder C
    134. Howard A-G
    135. Hsueh W
    136. Huang M
    137. Huang W
    138. Hung Y-J
    139. Hwang MY
    140. Hwu C-M
    141. Ichihara S
    142. Ikram MA
    143. Ingelsson M
    144. Islam MT
    145. Isono M
    146. Jang H-M
    147. Jasmine F
    148. Jiang G
    149. Jonas JB
    150. Jørgensen ME
    151. Jørgensen T
    152. Kamatani Y
    153. Kandeel FR
    154. Kasturiratne A
    155. Katsuya T
    156. Kaur V
    157. Kawaguchi T
    158. Keaton JM
    159. Kho AN
    160. Khor C-C
    161. Kibriya MG
    162. Kim D-H
    163. Kohara K
    164. Kriebel J
    165. Kronenberg F
    166. Kuusisto J
    167. Läll K
    168. Lange LA
    169. Lee M-S
    170. Lee NR
    171. Leong A
    172. Li L
    173. Li Y
    174. Li-Gao R
    175. Ligthart S
    176. Lindgren CM
    177. Linneberg A
    178. Liu C-T
    179. Liu J
    180. Locke AE
    181. Louie T
    182. Luan J
    183. Luk AO
    184. Luo X
    185. Lv J
    186. Lyssenko V
    187. Mamakou V
    188. Mani KR
    189. Meitinger T
    190. Metspalu A
    191. Morris AD
    192. Nadkarni GN
    193. Nadler JL
    194. Nalls MA
    195. Nayak U
    196. Nongmaithem SS
    197. Ntalla I
    198. Okada Y
    199. Orozco L
    200. Patel SR
    201. Pereira MA
    202. Peters A
    203. Pirie FJ
    204. Porneala B
    205. Prasad G
    206. Preissl S
    207. Rasmussen-Torvik LJ
    208. Reiner AP
    209. Roden M
    210. Rohde R
    211. Roll K
    212. Sabanayagam C
    213. Sander M
    214. Sandow K
    215. Sattar N
    216. Schönherr S
    217. Schurmann C
    218. Shahriar M
    219. Shi J
    220. Shin DM
    221. Shriner D
    222. Smith JA
    223. So WY
    224. Stančáková A
    225. Stilp AM
    226. Strauch K
    227. Suzuki K
    228. Takahashi A
    229. Taylor KD
    230. Thorand B
    231. Thorleifsson G
    232. Thorsteinsdottir U
    233. Tomlinson B
    234. Torres JM
    235. Tsai F-J
    236. Tuomilehto J
    237. Tusie-Luna T
    238. Udler MS
    239. Valladares-Salgado A
    240. van Dam RM
    241. van Klinken JB
    242. Varma R
    243. Vujkovic M
    244. Wacher-Rodarte N
    245. Wheeler E
    246. Whitsel EA
    247. Wickremasinghe AR
    248. van Dijk KW
    249. Witte DR
    250. Yajnik CS
    251. Yamamoto K
    252. Yamauchi T
    253. Yengo L
    254. Yoon K
    255. Yu C
    256. Yuan J-M
    257. Yusuf S
    258. Zhang L
    259. Zheng W
    260. FinnGen
    261. eMERGE Consortium
    262. Raffel LJ
    263. Igase M
    264. Ipp E
    265. Redline S
    266. Cho YS
    267. Lind L
    268. Province MA
    269. Hanis CL
    270. Peyser PA
    271. Ingelsson E
    272. Zonderman AB
    273. Psaty BM
    274. Wang Y-X
    275. Rotimi CN
    276. Becker DM
    277. Matsuda F
    278. Liu Y
    279. Zeggini E
    280. Yokota M
    281. Rich SS
    282. Kooperberg C
    283. Pankow JS
    284. Engert JC
    285. Chen Y-DI
    286. Froguel P
    287. Wilson JG
    288. Sheu WHH
    289. Kardia SLR
    290. Wu J-Y
    291. Hayes MG
    292. Ma RCW
    293. Wong T-Y
    294. Groop L
    295. Mook-Kanamori DO
    296. Chandak GR
    297. Collins FS
    298. Bharadwaj D
    299. Paré G
    300. Sale MM
    301. Ahsan H
    302. Motala AA
    303. Shu X-O
    304. Park K-S
    305. Jukema JW
    306. Cruz M
    307. McKean-Cowdin R
    308. Grallert H
    309. Cheng C-Y
    310. Bottinger EP
    311. Dehghan A
    312. Tai E-S
    313. Dupuis J
    314. Kato N
    315. Laakso M
    316. Köttgen A
    317. Koh W-P
    318. Palmer CNA
    319. Liu S
    320. Abecasis G
    321. Kooner JS
    322. Loos RJF
    323. North KE
    324. Haiman CA
    325. Florez JC
    326. Saleheen D
    327. Hansen T
    328. Pedersen O
    329. Mägi R
    330. Langenberg C
    331. Wareham NJ
    332. Maeda S
    333. Kadowaki T
    334. Lee J
    335. Millwood IY
    336. Walters RG
    337. Stefansson K
    338. Myers SR
    339. Ferrer J
    340. Gaulton KJ
    341. Meigs JB
    342. Mohlke KL
    343. Gloyn AL
    344. Bowden DW
    345. Below JE
    346. Chambers JC
    347. Sim X
    348. Boehnke M
    349. Rotter JI
    350. McCarthy MI
    351. Morris AP
    (2022) Multi-ancestry genetic study of type 2 diabetes highlights the power of diverse populations for discovery and translation
    Nature Genetics 54:560–572.
  5. Book
    1. Malécot G
    Mathématiques de l’hérédité
    Paris: Masson et Cie.
    1. Roselli C
    2. Chaffin MD
    3. Weng LC
    4. Aeschbacher S
    5. Ahlberg G
    6. Albert CM
    7. Almgren P
    8. Alonso A
    9. Anderson CD
    10. Aragam KG
    11. Arking DE
    12. Barnard J
    13. Bartz TM
    14. Benjamin EJ
    15. Bihlmeyer NA
    16. Bis JC
    17. Bloom HL
    18. Boerwinkle E
    19. Bottinger EB
    20. Brody JA
    21. Calkins H
    22. Campbell A
    23. Cappola TP
    24. Carlquist J
    25. Chasman DI
    26. Chen LY
    27. Chen YDI
    28. Choi EK
    29. Choi SH
    30. Christophersen IE
    31. Chung MK
    32. Cole JW
    33. Conen D
    34. Cook J
    35. Crijns HJ
    36. Cutler MJ
    37. Damrauer SM
    38. Daniels BR
    39. Darbar D
    40. Delgado G
    41. Denny JC
    42. Dichgans M
    43. Dörr M
    44. Dudink EA
    45. Dudley SC
    46. Esa N
    47. Esko T
    48. Eskola M
    49. Fatkin D
    50. Felix SB
    51. Ford I
    52. Franco OH
    53. Geelhoed B
    54. Grewal RP
    55. Gudnason V
    56. Guo X
    57. Gupta N
    58. Gustafsson S
    59. Gutmann R
    60. Hamsten A
    61. Harris TB
    62. Hayward C
    63. Heckbert SR
    64. Hernesniemi J
    65. Hocking LJ
    66. Hofman A
    67. Horimoto A
    68. Huang J
    69. Huang PL
    70. Huffman J
    71. Ingelsson E
    72. Ipek EG
    73. Ito K
    74. Jimenez-Conde J
    75. Johnson R
    76. Jukema JW
    77. Kääb S
    78. Kähönen M
    79. Kamatani Y
    80. Kane JP
    81. Kastrati A
    82. Kathiresan S
    83. Katschnig-Winter P
    84. Kavousi M
    85. Kessler T
    86. Kietselaer BL
    87. Kirchhof P
    88. Kleber ME
    89. Knight S
    90. Krieger JE
    91. Kubo M
    92. Launer LJ
    93. Laurikka J
    94. Lehtimäki T
    95. Leineweber K
    96. Lemaitre RN
    97. Li M
    98. Lim HE
    99. Lin HJ
    100. Lin H
    101. Lind L
    102. Lindgren CM
    103. Lokki ML
    104. London B
    105. Loos RJF
    106. Low SK
    107. Lu Y
    108. Lyytikäinen LP
    109. Macfarlane PW
    110. Magnusson PK
    111. Mahajan A
    112. Malik R
    113. Mansur AJ
    114. Marcus GM
    115. Margolin L
    116. Margulies KB
    117. März W
    118. McManus DD
    119. Melander O
    120. Mohanty S
    121. Montgomery JA
    122. Morley MP
    123. Morris AP
    124. Müller-Nurasyid M
    125. Natale A
    126. Nazarian S
    127. Neumann B
    128. Newton-Cheh C
    129. Niemeijer MN
    130. Nikus K
    131. Nilsson P
    132. Noordam R
    133. Oellers H
    134. Olesen MS
    135. Orho-Melander M
    136. Padmanabhan S
    137. Pak HN
    138. Paré G
    139. Pedersen NL
    140. Pera J
    141. Pereira A
    142. Porteous D
    143. Psaty BM
    144. Pulit SL
    145. Pullinger CR
    146. Rader DJ
    147. Refsgaard L
    148. Ribasés M
    149. Ridker PM
    150. Rienstra M
    151. Risch L
    152. Roden DM
    153. Rosand J
    154. Rosenberg MA
    155. Rost N
    156. Rotter JI
    157. Saba S
    158. Sandhu RK
    159. Schnabel RB
    160. Schramm K
    161. Schunkert H
    162. Schurman C
    163. Scott SA
    164. Seppälä I
    165. Shaffer C
    166. Shah S
    167. Shalaby AA
    168. Shim J
    169. Shoemaker MB
    170. Siland JE
    171. Sinisalo J
    172. Sinner MF
    173. Slowik A
    174. Smith AV
    175. Smith BH
    176. Smith JG
    177. Smith JD
    178. Smith NL
    179. Soliman EZ
    180. Sotoodehnia N
    181. Stricker BH
    182. Sun A
    183. Sun H
    184. Svendsen JH
    185. Tanaka T
    186. Tanriverdi K
    187. Taylor KD
    188. Teder-Laving M
    189. Teumer A
    190. Thériault S
    191. Trompet S
    192. Tucker NR
    193. Tveit A
    194. Uitterlinden AG
    195. Van Der Harst P
    196. Van Gelder IC
    197. Van Wagoner DR
    198. Verweij N
    199. Vlachopoulou E
    200. Völker U
    201. Wang B
    202. Weeke PE
    203. Weijs B
    204. Weiss R
    205. Weiss S
    206. Wells QS
    207. Wiggins KL
    208. Wong JA
    209. Woo D
    210. Worrall BB
    211. Yang PS
    212. Yao J
    213. Yoneda ZT
    214. Zeller T
    215. Zeng L
    216. Lubitz SA
    217. Lunetta KL
    218. Ellinor PT
    (2018) Multi-ethnic genome-wide Association study for atrial fibrillation
    Nature Genetics 50:1225–1233.
    1. Sokal RR
    2. Michener CD
    A statistical method for evaluating systematic relationships
    Univ Kansas, Sci Bull 38:1409–1438.

Article and author information

Author details

  1. Yiqi Yao

    Department of Biostatistics and Bioinformatics, Duke University, Durham, United States
    Present address
    BenHealth Consulting, Shanghai, Shanghai, China
    Software, Formal analysis, Investigation, Visualization, Writing – original draft, Writing – review and editing
    Competing interests
    is affiliated with BenHealth Consulting. The author has no financial interests to declare
  2. Alejandro Ochoa

    1. Department of Biostatistics and Bioinformatics, Duke University, Durham, United States
    2. Duke Center for Statistical Genetics and Genomics, Duke University, Durham, United States
    Conceptualization, Resources, Data curation, Software, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing
    For correspondence
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-4928-3403


Whitehead Foundation

  • Alejandro Ochoa

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.


Thanks to Tiffany Tu, Ratchanon Pornmongkolsuk, and Zhuoran Hou for feedback on this article. This work was funded in part by the Duke University School of Medicine Whitehead Scholars Program, a gift from the Whitehead Charitable Foundation. The 1000 Genomes data were generated at the New York Genome Center with funds provided by NHGRI Grant 3UM1HG008901-03S1.

Version history

  1. Preprint posted: March 27, 2022 (view preprint)
  2. Received: April 4, 2022
  3. Accepted: May 4, 2023
  4. Accepted Manuscript published: May 4, 2023 (version 1)
  5. Version of Record published: June 1, 2023 (version 2)


© 2023, Yao and Ochoa

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.


  • 1,123
  • 121
  • 4

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

  1. Yiqi Yao
  2. Alejandro Ochoa
Limitations of principal components in quantitative genetic association models for human studies
eLife 12:e79238.

Share this article

Further reading

    1. Genetics and Genomics
    2. Neuroscience
    Donghui Yan, Bowen Hu ... Qiongshi Lu
    Research Article

    Rich data from large biobanks, coupled with increasingly accessible association statistics from genome-wide association studies (GWAS), provide great opportunities to dissect the complex relationships among human traits and diseases. We introduce BADGERS, a powerful method to perform polygenic score-based biobank-wide association scans. Compared to traditional approaches, BADGERS uses GWAS summary statistics as input and does not require multiple traits to be measured in the same cohort. We applied BADGERS to two independent datasets for late-onset Alzheimer’s disease (AD; n=61,212). Among 1738 traits in the UK biobank, we identified 48 significant associations for AD. Family history, high cholesterol, and numerous traits related to intelligence and education showed strong and independent associations with AD. Furthermore, we identified 41 significant associations for a variety of AD endophenotypes. While family history and high cholesterol were strongly associated with AD subgroups and pathologies, only intelligence and education-related traits predicted pre-clinical cognitive phenotypes. These results provide novel insights into the distinct biological processes underlying various risk factors for AD.

    1. Genetics and Genomics
    2. Microbiology and Infectious Disease
    Jason E Stajich, Brian Lovett ... Carolyn Elya
    Research Article

    Despite over a century of observations, the obligate insect parasites within the order Entomophthorales remain poorly characterized at the genetic level. In this manuscript, we present a genome for a laboratory-tractable Entomophthora muscae isolate that infects fruit flies. Our E. muscae assembly is 1.03 Gb, consists of 7810 contigs and contains 81.3% complete fungal BUSCOs. Using a comparative approach with recent datasets from entomophthoralean fungi, we show that giant genomes are the norm within Entomophthoraceae owing to extensive, but not recent, Ty3 retrotransposon activity. In addition, we find that E. muscae and its closest allies possess genes that are likely homologs to the blue-light sensor white-collar 1, a Neurospora crassa gene that has a well-established role in maintaining circadian rhythms. We uncover evidence that E. muscae diverged from other entomophthoralean fungi by expansion of existing families, rather than loss of particular domains, and possesses a potentially unique suite of secreted catabolic enzymes, consistent with E. muscae’s species-specific, biotrophic lifestyle. Finally, we offer a head-to-head comparison of morphological and molecular data for species within the E. muscae species complex that support the need for taxonomic revision within this group. Altogether, we provide a genetic and molecular foundation that we hope will provide a platform for the continued study of the unique biology of entomophthoralean fungi.