Apart from ancestry, personal or environmental covariates may contribute to differences in polygenic score (PGS) performance. We analyzed the effects of covariate stratification and interaction on body mass index (BMI) PGS (PGS_BMI) across four cohorts of European (N = 491,111) and African (N = 21,612) ancestry. Stratifying on binary covariates and quintiles for continuous covariates, 18/62 covariates had significant and replicable R² differences among strata. Covariates with the largest differences included age, sex, blood lipids, physical activity, and alcohol consumption, with R² being nearly double between best- and worst-performing quintiles for certain covariates. Twenty-eight covariates had significant PGS_BMI–covariate interaction effects, modifying PGS_BMI effects by nearly 20% per standard deviation change. We observed overlap between covariates that had significant R² differences among strata and interaction effects – across all covariates, their main effects on BMI were correlated with their maximum R² differences and interaction effects (0.56 and 0.58, respectively), suggesting high-PGS_BMI individuals have highest R² and increase in PGS effect. Using quantile regression, we show the effect of PGS_BMI increases as BMI itself increases, and that these differences in effects are directly related to differences in R² when stratifying by different covariates. Given significant and replicable evidence for context-specific PGS_BMI performance and effects, we investigated ways to increase model performance taking into account nonlinear effects. Machine learning models (neural networks) increased relative model R² (mean 23%) across datasets. Finally, creating PGS_BMI directly from GxAge genome-wide association studies effects increased relative R² by 7.8%. These results demonstrate that certain covariates, especially those most associated with BMI, significantly affect both PGS_BMI performance and effects across diverse cohorts and ancestries, and we provide avenues to improve model performance that consider these effects.