Bias in nutrition-health associations is not eliminated by excluding extreme reporters in empirical or simulation studies
Abstract
Self-reported nutrition intake (NI) data are prone to reporting bias that may induce bias in estimands in nutrition studies; however, they are used anyway due to high feasibility. We examined whether applying Goldberg cutoffs to remove ‘implausible’ self-reported NI could reliably reduce bias compared to biomarkers for energy, sodium, potassium, and protein. Using the Interactive Diet and Activity Tracking in the American Association of Retired Persons (IDATA) data, significant bias in mean NI was removed with Goldberg cutoffs (120 among 303 participants excluded). Associations between NI and health outcomes (weight, waist circumference, heart rate, systolic/diastolic blood pressure, and VO2 max) were estimated, but sample size was insufficient to evaluate bias reductions. We therefore simulated data based on IDATA. Significant bias in simulated associations using self-reported NI was reduced but not completely eliminated by Goldberg cutoffs in 14 of 24 nutrition-outcome pairs; bias was not reduced for the remaining 10 cases. Also, 95% coverage probabilities were improved by applying Goldberg cutoffs in most cases but underperformed compared with biomarker data. Although Goldberg cutoffs may achieve bias elimination in estimating mean NI, bias in estimates of associations between NI and outcomes will not necessarily be reduced or eliminated after application of Goldberg cutoffs. Whether one uses Goldberg cutoffs should therefore be decided based on research purposes and not general rules.
Editor's evaluation
This important study estimated the bias in nutrition-health associations and the reduction of bias using Goldberg cut-offs. The evidence supporting the study claims is solid, and the findings will be of interest to epidemiologists and nutrition scientists who are concerned with the effects of measurement error in health-diet research.
https://doi.org/10.7554/eLife.83616.sa0Introduction
Measuring nutrition intake plays a pivotal role in obesity and nutrition studies. However, objectively measuring nutrition intake is not easy. Under natural settings with less restriction, two approaches are often used: self-report and biomarkers. Biomarker approaches are considered more accurate compared with self-report; however, they are typically expensive and require special equipment for each nutrient. Self-report approaches are less expensive and do not require special laboratory equipment or techniques, and can be applied to the measurement of any nutrition components from macronutrients (i.e. carbohydrates, fat, and protein) to micronutrients (i.e. minerals and vitamins); however, self-reported data are known to be fraught with bias (Schoeller et al., 1990; Dhurandhar et al., 2015).
Consider the example of biomarker versus self-report energy intake. Energy intake (EI) is defined as a sum of the total metabolizable energy content of consumed foods and beverages (Astrup, 2005). EI can be measured using biomarkers, especially by energy balance approaches in which respiratory gas analysis or doubly labeled water (DLW) are used for the calculation of energy expenditure (EE) (Ferrannini, 1988). EI can be calculated using EE accounting for change in energy stores (i.e. body composition) during the period of measurement. The respiratory gas analysis typically requires the participants to wear a gas mask or stay in a metabolic chamber (Leonard, 2012), which restricts participants’ behavior; thus, the measured EI using the respiratory gas analysis may not reflect EI from habitual eating behavior. DLW requires the participants to visit the laboratory to drink labeled water enriched with stable hydrogen and oxygen isotopes and collect urine for measurement of isotope levels. It is considered to reflect habitual eating behavior and widely accepted as a standard and reliable approach in measuring EI in free-living individuals (Schoeller et al., 1986; Speakman, 1997; Schoeller and van Santen, 1982). However, given the high cost of isotopes and laboratories to analyze collected urine, DLW is not practical in large-scale epidemiological surveillance (Banna et al., 2017).
EI measured using self-report approaches (EISR) is computed by summing up the energy intake from foods and beverages reportedly consumed during the period of measurement, which can commonly range from a day to a year depending on approaches (e.g. 24 hr recall, food diary, and food frequency questionnaire; Leung et al., 2017; Sugiyama et al., 2014; Oftedal et al., 2017). Given that self-report approaches are low cost and scale-up relatively easily, EISR is still widely used in epidemiological studies. However, EISR is biased compared with the EI measured by biomarker such as DLW (EIBIO) (Schoeller et al., 1990) and is discouraged from use to estimate actual EI (Dhurandhar et al., 2015). The mechanism behind the reporting bias in EISR has been argued elsewhere. For example, EISR is computed from the information of reported foods and beverages using tables of food and beverage nutrients, composed of weights, calories, and the amount of macro- and micronutrients. Compositions of foods and beverages are different among brands, stores, and seasons (Martelli et al., 2018), and thus nutrient databases cannot be exhaustive. Consequently, EISR using such databases may not be accurate (McHenry et al., 1945). Further factors, including social desirability (i.e. people do not want to be seen as a ‘big eater’ in general) (Heymsfield et al., 1995) and weight/social status (Heymsfield et al., 1995; Johansson et al., 1998; Macdiarmid and Blundell, 1998; Johansson et al., 2001; Klesges et al., 1995), are known to be associated with reporting bias. Thus, the difference between EIBIO and EISR is due to systemic biases rather than a random error.
Despite concerns about the accuracy of self-report approaches, they have been predominantly used in large-scale nutrition epidemiology due to their high feasibility. Approaches to mitigate the bias in EISR have been argued, including commonly used approaches to exclude the data of EISR that are considered unreliable. For example, one approach is to exclude those reporting EISR out of the range considered by the researchers to be plausible, such as from 500 to 3500 kcal/day (Banna et al., 2017). However, this rule does not consider individual variability and is not appropriate for those whose intake is truly above 3500 kcal/day, such as elite athletes or those whose body size is large and requires more than 3500 kcal/day to maintain their weight (Santos et al., 2014; DeLany et al., 2013).
Other approaches are to use EE prediction equations to account for individual variability in EI (McCrory et al., 2002; Huang et al., 2005; Vinken et al., 1999; Livingstone and Robson, 2000; Xie et al., 2003; Briefel et al., 1995; Huang et al., 2004). If the EE predicted considering various factors (e.g. age, weight, sex) is far from EISR, those individuals are considered under/overreporting their actual EI. Indeed, such an approach has evidently reduced misclassification (Tooze et al., 2012). Among proposed approaches, the Goldberg cutoffs Goldberg et al., 1991 have been used in nutrition epidemiology to assess nutritional status in specific populations and investigate the association between nutritional status and health and socioeconomic outcomes (Livingstone and Robson, 2000; Xie et al., 2003; Briefel et al., 1995; Huang et al., 2004; Martin et al., 2018). Note that nutritional status in this context means dietary intakes in general; not only EI but also nutritional constituents (vitamin, fish oil, etc.) and dietary styles (Mediterranean, vegan, etc.; Grimes et al., 2013; Molina-Montes et al., 2017; New et al., 2000; Shaheen et al., 2001; O’Sullivan et al., 2011). Therefore, the rationale of using the Goldberg cutoffs (and other similar approaches) to exclude data from individuals considered to have unreliable EISR is based on an assumption, ‘if total EI is underestimated, it is probable that the intakes of other nutrients are also underestimated’ (Livingstone and Black, 2003). The corollary for overestimation may also be true.
Bias-correcting approaches like the Goldberg cutoffs are frequently used in nutrition studies. By bias, we are referring to the difference between the estimates and their true values. Here, we focus on two typical estimands in nutrition epidemiology: the mean of nutrition intake, and the associations between nutrition intake and health outcomes in a population. While reducing bias in the mean nutrition intake is useful for determining the true value of population nutrition intake from self-reported data, it is also important to reduce bias in the associations between nutrition intake and health outcomes. However, whether the Goldberg cutoffs reduce bias has not been proved theoretically or empirically. In our previous study, we demonstrated that the Goldberg cutoffs do not necessarily eliminate the bias in estimating the associations between EI and various health outcomes (Ejima et al., 2019). Extending our previous study, we consider nutritional intake beyond EI to include biomarkers of sodium, potassium, and protein with the primary goal to examine if the Goldberg cutoffs eliminate or reduce the bias using a single dataset from the Interactive Diet and Activity Tracking in the American Association of Retired Persons (IDATA) study.
Evaluating the performance of the Goldberg cutoffs solely depending on a single empirical dataset is limited because the dataset is a single realization from an unobserved data-generation process. Further, if the dataset is too small, conclusions may be limited because of issues related to power to detect bias in associations or the power to detect reductions in bias. Therefore, in addition to the empirical data analyses above, we also generated data through simulation and analyzed those generated data that preserve the characteristics of the empirical data. The simulation further enables us to assess the impact of sample size on the performance of the Goldberg cutoffs.
Results
Statistical data analysis on IDATA
The list of variables, equations, and metrics is available in Table 1. Abbreviations are summarized in Supplementary file 1.
The impact of the Goldberg cutoffs on the reporting bias in nutrition intake
Table 2 summarizes the data included in the analyses (n=303). More than 90% of the participants are non-Hispanic whites, and their mean age was 63. The mean BMI was 27.7 kg/m2, and 30% had obesity (BMI ≥30). There was significant mean underreporting in which the biomarker nutrition intakes (NIBIO) was greater than the self-reported nutrition intakes (NISR) in all four types of nutrition intake (Table 2).
By the Goldberg cutoffs, 120 among 303 participants were excluded (40%) (Table 3). There was significant underreporting in the rejected cases, whereas reporting bias (NISR – NIBIO) was not significant in the accepted cases in all four NI. Further, the difference in the biases between the accepted and the rejected cases was significant. The mean NISR in the accepted cases was significantly larger than that in the rejected cases in all four NI. NIBIO was significantly larger in the rejected cases in sodium intake (SI), whereas it was comparable between those groups in energy intake (EI), potassium intake (PoI), and protein intake (PrI). To further understand the dependency of the bias on NIBIO and the impact of the Goldberg cutoffs on the bias, the individual data were plotted (Figure 1). The bias was significantly and negatively correlated with NIBIO (Figure 1 top panels). The means of accepted cases of NISR correspond to the mean of NIBIO better than the mean of NISR in the rejected cases (Figure 1 bottom panels).
These findings suggest that the Goldberg cutoffs reduce (and may eliminate) the bias in mean NISR by identifying predominantly underreporting individuals. The difference in self-reported nutrition intake between accepted and rejected cases is predominantly from differences in reporting and less so from differences in actual biomarker-measured intake. However, statistically significant differences in biomarker-measured intake remained between accepted and rejected cases in SI.
Associations between nutrition intake and health outcomes
The associations between EIBIO, SIBIO, and PrIBIO and WC were statistically significant (Figure 2—figure supplements 1 and 2 and Supplementary file 2). PoIBIO was significantly associated with VO2. Assuming that the biomarker-based health outcome associations () and self-report-based health outcome associations () are different, the associations of health outcomes with self-report after applying the Goldberg rule () are expected to be between raw self-report and biomarker-based exposures. In other words, the Goldberg cutoffs are supposed to reduce the bias: ( or ). We first tested whether bias exists for each nutrition-health outcome pair, and, if so, whether the bias is overestimation or underestimation, using percent bias (). The percent bias () was significantly below 0% (underestimation) in only limited cases (EI and BW, and EI and WC) and we could not conclude whether the bias exists or not for the rest of the cases. Significant overestimation () was not observed in any cases (the left panel in Figure 2 and Supplementary file 2). In the two cases of underestimation from self-report (EI and BW, and EI and WC), there were also significant associations between NIBIO and the outcomes. In all cases, the nutrition-health outcome pairs with significant associations between NIBIO and health outcomes (HO) had smaller confidence intervals for the percent bias than those pairs without significant associations between NIBIO and HO (EI and BW, EI and WC, SI and BW, SI and WC, PoI and VO2, PrI and BW, and PrI and WC, shown in italic bold in Figure 2). Second, we tested whether the bias remains after applying the Goldberg cutoffs, and, if so, whether it is overestimation or underestimation, using the percent remaining bias (). The bias was reduced () but remained () for the association between EI and BW, and between EI and WC. We could not again conclude whether the bias exists or not after applying the Goldberg cutoffs for the rest of the cases.
From the above data analyses, we could not conclude that the Goldberg cutoffs do or do not remove the bias in most cases, because of the wide confidence intervals, which could be partially due to the sample size of the data and weak associations between some combinations of NI and HO.
Simulation
To strengthen the arguments on the performance of the Goldberg cutoffs, we also generated and analyzed data which preserve the characteristics of the IDATA. The simulation was designed to overcome the sample size limitations of IDATA, but keep the structure and nature of the IDATA.
Performance metrics (bias, MSE, and coverage probability) for all the combinations between NI and HO are shown in Figure 3. For each performance metric, the mean across the 1000 simulation replicates is shown. We confirmed the generated self-reported data were consistent with the IDATA (Figure 4—figure supplement 1). The bias for all biomarker simulations was close to 0 (red squares in the bias column of Figure 3). For all four NI, the associations with HO were negative and those with the other outcomes were positive (Figure 2—figure supplements 1 and 2). For the cases with positive correlation, the positive bias in was observed, and for the cases with negative correlation, the negative bias in was observed, suggesting that the self-reported data cause attenuation bias (i.e. ) as observed in many studies (Kipnis et al., 2003). After application of the Goldberg cutoffs, the bias was reduced in 14 among 24 combinations of NI and HO, which can be seen in Figure 3 where the green circles are closer to 0 than the blue triangles. Specifically, . Notably, the bias was reduced in all six outcomes and reduction was large when EI is used as a predictor. When SI, PoI, and PrI are used as predictors, the bias was not much changed by applying the Goldberg cutoffs (green circles and blue triangles are close to each other in the bias column of Figure 3). In most cases, the MSE was also smaller for NIBIO compared with NISR, and the MSE for NISR became smaller after applying the Goldberg cutoffs (green circles are often closer to 0 than the blue triangles in the MSE column of Figure 3). The coverage probabilities (CP) were close to 95% when NIBIO was used as a predictor, as was expected, whereas it was mostly lower than 95% when NISR was used. The coverage probability increased by applying Goldberg cutoffs (green circles are often closer to 95 than the blue triangles in the CP column of Figure 3); however, it is still lower than 95% in most cases. We repeated the simulation for other values (=50, 100, 200, and 300) of sample size and all cases showed similar results (Figure 3—figure supplement 1). We ran sensitivity analyses varying , an error term for the regression between NI and HO, and , the standard deviation of the reporting bias (Figure 3—figure supplement 2). Biases were not affected by varying for the range of values we considered, suggesting that error in health outcomes (or variance in health outcomes which cannot be explained by variance in biomarker-based nutrition intake) does not influence NI-HO association estimations where NI is used as an independent variable. Bias was negative when self-reported data ( or ) were used under the default value of σ, and the magnitude of bias increased as σ increased (i.e., the association was attenuated). However, when the error is extremely small (e.g. σ=0.25 in Figure 3—figure supplement 2), the bias becomes positive, which is because the mean error function (µ) was negative at low NI and positive at high NI, thus the association was intensified. Whether or not the bias becomes positive or negative is determined by the balance between the mean error function and random error.
Discussion
Dietary self-reporting is a basic component of nutrition epidemiology, particularly for studies of nutrition-health relationships. However, reporting bias in nutrition intake is perhaps one of the greatest impediments to understanding the true effect of nutrition on disease. Failure to account for this bias in self-reports can affect the analysis and interpretation of studies designed to assess the influences of nutrition on health. Correlation or regression coefficients estimating associations between self-reported nutrition intakes and health outcomes can be subject to substantial error (Freedman et al., 2011; Gibson et al., 2017). To address such reporting bias issues, there has been increased interest in the use of statistical models in conjunction with biomarker data (Freedman et al., 2010; Brown et al., 2021).
In this study, reporting bias in self-reported nutrition intake was demonstrated using both empirical data analysis and a simulation study. We first examined whether the bias in mean nutrition intake and the bias in the associations between nutrition intake and health outcomes are reduced by applying the Goldberg cutoffs to an empirical dataset, IDATA. We confirmed the bias in mean self-reported nutrition intake was eliminated by applying Goldberg cutoffs to exclude extreme reporters (mostly under-reporters). This bias can be mostly explained by differences in reporting between the accepted and rejected cases rather than a difference in actual nutrition intake between the accepted and rejected cases. We further tested whether the associations between health outcomes and nutrition intake are biased when self-reported data are used, and whether the bias is reduced by applying the Goldberg cutoffs. The significant bias for the associations between EI and BW and the association between EI and WC were reduced but remained after applying Goldberg cutoffs. We did not observe significant bias in associations for other nutrition-outcome pairs, and therefore cannot make conclusions regarding the utility of applying the Goldberg cutoffs to reduce bias for these associations in the empirical data.
To overcome the limitations of the data (i.e. sample size and weak associations among many nutrition-outcome pairs), we generated and analyzed data preserving the characteristics of the IDATA for the assessment of the Goldberg cutoffs. We confirmed the existence of bias in estimated associations when the self-reported data were used, and these biases were reduced, but not eliminated, by applying the Goldberg cutoffs in general. We also confirmed large MSE and low coverage probability when self-reported data were used; MSE and low coverage probability were improved by using Goldberg cutoffs, but were not recovered to the levels of biomarker data. Overall, the empirical data analyses and the simulation study suggest that the Goldberg cutoffs improve the estimates in nutrition-health outcome associations; however, the biases were not completely removed.
This study has several strengths. First, although energy intake has traditionally been studied more with respect to Goldberg cutoffs, herein we also investigated three other nutrition intakes: sodium, potassium, and protein. In our previous study, we used only energy intake as a predictor (Ejima et al., 2019). However, given that the Goldberg cutoffs are believed to reduce the bias in nutrition intake broadly, we expanded the approach to these other nutrition intakes, which can be measured by biomarkers (Livingstone and Black, 2003). Second, we conducted the simulation in addition to empirical data analyses. There are two advantages of the simulation design. (1) The data generation process was based on the IDATA dataset. Thus, we were able to simulate datasets reflective of those that would be seen in the real world. Both the sample size of the generated dataset and the variables that we considered are reflective of what is often observed in nutrition studies. (2) Simulation can consider different models for the data-generating process. Although our objective was not to disentangle the underlying data generation mechanisms, we were able to generate data that preserve the characteristics of the IDATA.
The limitations of this study are worth noting. In the analysis of IDATA, we used only bivariate associations for nutrition-health outcomes. It is possible that including various covariates or different models may influence the existence or lack of bias in nutrition-health associations. In the data-generation process for the simulations, assumptions were made on parameter settings. The reporting bias was set to be normally distributed. Modeling the bias using other distributions (such as a heavier tailed distribution that may be more consistent with a zero-bounded intake) may improve model fit, and thus generated more realistic data. We simulated the bias with constant variance, consistent with the apparent stability of variance across the intake quantiles in Figure 4—figure supplement 2 and Supplementary file 3; however, modeling the variance as a function of intake quantile may alter the data-generating process. We used the self-reported nutrition intake reported by ASA24, as it reflects recent dietary intake. The bias we observed in this study may be different for other self-report methods. Finally, the simulation approach, although based on the IDATA, was nonetheless generated from a limited sample size. We are not aware of thresholds for sufficient sample size for empirical-data-based simulation, and we have had success with smaller plasmode based sampling in the past (Ejima et al., 2020; Alfaras et al., 2021); however, it is possible a larger, more representative, or more robust empirical data set may influence both the empirical and simulated results. Further as the data was from a single population in which white and relatively older people are predominant, generalizability of the study finding should be carefully considered.
In conclusion, our study demonstrated that the use of Goldberg cutoffs can effectively reduce or eliminate bias in mean nutrition intake measurements in the IDATA dataset. However, our analysis also revealed that these cutoffs have limited efficacy in addressing bias in the associations between nutrition intake and health outcomes, as both empirical and simulated results showed. Investigators considering excluding extreme reporters in epidemiological studies should consider whether such approaches are likely to be helpful in answering their specific research questions, as our study has shown that such cutoffs are more useful in estimating mean nutrition intakes than in studies of associations between nutrition intake and health outcomes.
Materials and methods
We conducted a set of empirical analyses to assess the reporting bias due to self-report and examine whether the Goldberg cutoffs reduce reporting bias using the data from The Interactive Diet and Activity Tracking in the American Association of Retired Persons (IDATA) study (National Cancer Institute, 2018a). The reporting bias in the mean of nutrition intakes and the associations (regression coefficients) between nutrition intakes and health outcomes were assessed. We also conducted a simulation study to investigate the performance of Goldberg cutoffs using the results from the empirical analyses as ground truth. Note that in data analyses, we assumed the estimates from biomarkers are the true values, since the true values (i.e. population parameters) are unobservable.
Among various nutrition intakes (NI), we considered sodium intake (SI), potassium intake (PoI), and protein intake (PrI) in addition to energy intake (EI). These NI were selected because they can be measured by urine samples as well as self-reports. We denote self-reported NI by NISR: EISR, SISR, PoISR, and PrISR. We denote biomarker-based NI by NIBIO: EIBIO, SIBIO, PoIBIO, and PrIBIO. The NISR data remaining after applying the Goldberg cutoffs (hereafter, we call those ‘accepted cases’) are denoted by NIG: EIG, SIG, PoIG, and PrIG. Health outcomes (HO) that may be associated with NI were selected from the data dictionaries; body weight (BW), waist circumference (WC), heart rate after fitness test (HR), resting systolic blood pressure (SBP), resting diastolic blood pressure (DBP), and VO2 max (VO2). The estimated regression coefficients are denoted by the subscript of the explanatory variables and the health outcomes (e.g., represents the association between self-reported EI and body weight).
Goldberg cutoffs
Request a detailed protocolThe Goldberg cutoffs are an approach to identify those who report low or high energy intake that, if sustained across long periods of time, would be implausible to sustain life (for underreporting) or present weight (for overreporting), by comparing the reported EI and the energy expenditure (EE) predicted from body composition and physical activity level (Goldberg et al., 1991). First, EE is predicted as a product of physical activity level (PAL) and basal metabolic rate (BMR): EE =PAL*BMR. PAL is assumed to be 1.75 (Dugas et al., 2011). BMR is predicted using fat-free mass (Cunningham, 1991): BMR = 370 + 21.6 * FFM. Second, the ratio of EISR and predicted EE is computed and examined whether it is in a reasonable range: < , where is defined as: . , , and are the within-subject variation in EISR, the within-subject variation in BMR, and the total variation in PAL, and is the number of days of diet assessment. We set following the previous study (Black, 2000). Finally, if individuals are not in the range, they were considered under- or overreporters and were excluded from further analyses.
Data
Overview of IDATA
Request a detailed protocolThe IDATA study was designed ‘to evaluate and compare the measurement error structure of diet and physical activity assessment tools against reference biomarkers’. The study participants ‘were recruited from a list of AARP [American Association of Retired Persons] members aged 50–74 years residing in and around Pittsburgh, Pennsylvania’ and were screened for eligibility by phone interview or clinic visit. The eligibility criteria were speaking English, having internet access, having no major medical issues (such as diabetes, renal/heart failure, and any conditions affecting fluid balance), having no mobility issues, and having BMI between 18.5 and 40.0 (kg/m2). The participants ‘visited the study center three times and also completed assigned activities at home over a 12-month period’ (National Cancer Institute, 2018a) from early 2012 through late 2013 (National Cancer Institute, 2018a; Matthews et al., 2018). Self-reported dietary/physical activity information, biomarker data to estimate dietary intake, and the data from physical objective monitors to estimate physical activity were collected. In addition, demographic information (age, sex, race/ethnicity) and anthropometric measurements (weight, height, and waist circumference) were obtained at a clinical visit or phone screening.
Self-reported nutrition intake: NISR
Request a detailed protocolFor self-report dietary assessment (EISR, SISR, PoISR, and PrISR), four different approaches were used in IDATA: Automated Self-Administered 24 Hour Dietary Assessment Tool (ASA24) (National Cancer Institute, 2018b), Diet History Questionnaire (DHQ-II) (Diet History Questionnaire, 2022), 7-day food checklist, and 4-day food record. These SR approaches were conducted at different times during the IDATA study. We used the data of ASA24 in our analyses because they were collected closest in time to the DLW measurements. ASA24 is an online self-administered recall system asking about a 24-hr dietary recall for the previous day, from midnight to midnight, using a dynamic user interface.
Biomarker-based nutrition intake: NIBIO
Request a detailed protocolThree approaches were used for biomarker assessment of NI: DLW, urine, and saliva (not used herein). EIBIO was computed from DLW data, and SIBIO, PoIBIO, and PrIBIO were calculated from 24-hr urine samples. EIBIO was estimated as a sum of daily total energy expenditure estimated from DLW and daily change in energy stores during the DLW period. Total energy expenditure was calculated using the approach proposed by Schoeller et al., 1986, where the respiratory quotient was assumed as 0.86. Daily change in energy stores during the DLW period (about 2 weeks) was computed from the average daily weight change during the period assuming the energy density of body weight was 2380 kcal/kg (Bhutani et al., 2017). The urinary values obtained from 24-hr urine samples were converted into nutrition intakes assuming that 81%, 80%, and 86% of consumed nitrogen, potassium, and sodium are excreted in the urine, respectively. Dietary protein was calculated assuming that 16% of protein is nitrogen. A more comprehensive explanation of measuring EIBIO, SIBIO, PoIBIO, and PrIBIO is reported elsewhere (Park et al., 2018).
Health outcomes
Request a detailed protocolFrom health outcomes (HO) available in IDATA, we selected objective variables that were potentially associated with nutrition intake. BW (kg) and WC (cm) were obtained at clinical visits; HR (beat/min after fitness test), SBP (mmHg), DBP (mmHg), and VO2 max (L/min) were obtained by the modified Canadian Aerobic Fitness Test (mCAFT).
Summary of the data
Request a detailed protocolIn IDATA, NISR, NIBIO, and HO were repeatedly measured at different time points. For fair comparison among them, we selected the data measured in the same month; thus, the data of Month 0 and Month 11 were used from Groups 1&3 and Groups 2&4, respectively (National Cancer Institute, 2018a). From 1,082 participants, we excluded 779 participants because we did not have information necessary to apply the Goldberg cutoffs (no fat-free mass information [n=725] or NISR [n=240]) or there was no urine-based nutrition intake data [n=231]. The remaining 303 participants’ data were analyzed.
IDATA data were accessible through the Cancer Data Access System (https://biometry.nci.nih.gov/cdas/idata/; downloaded 11/22/2017) after our project proposal was reviewed and approved by the National Cancer Institute (https://biometry.nci.nih.gov/cdas/approved-projects/1702/).
Statistical analysis of IDATA
Request a detailed protocolWhether the bias in the mean NISR compared to NIBIO (e.g. calculated as EISR - EIBIO) would be reduced by the Goldberg cutoffs was tested. We first tested whether the mean bias in each NISR is significantly different from zero. Then, we repeated the same test for the data of the accepted cases for each NIG (e.g., EIG - EIBIO). We further tested whether the mean NISR are different between those who were removed (rejected cases; NISR excluding NIG) and the accepted cases (NIG).
Further, we tested whether the bias in the estimates of associations between NI and HO would be reduced by the Goldberg cutoffs. We computed regression coefficients () between HO and each of the three different NI measuring approaches: NIBIO, NISR, and NIG. Note that we did not adjust the analyses for any other covariates, because the purpose of this study is to understand if the Goldberg cutoffs could reliably reduce bias rather than to refine associations accounting for covariates. To comprehensively assess the reporting bias and bias reduction in different outcomes with different units, we used standardized metrics for reporting bias and bias reduction, which was originally proposed in our previous study (Ejima et al., 2019). In brief, we used the percent bias () for the assessment of the magnitude of the bias and the percent remaining bias () for the assessment of the magnitude of the bias remaining after applying the Goldberg cutoffs, which are defined as follows:
A more in-depth interpretation of the metric is available in our previous study (Ejima et al., 2019). Jackknife estimation (leave-one-out) (Efron and Stein, 1981) was used to compute 95% CIs of those metrics. All analyses were performed separately for each combination of nutrition intakes and outcomes. Two-tailed Student’s t test was used to test whether mean of a single variable is different from zero. Two-tailed independent Welch’s t test was used to test the mean difference of a single variable from two independent groups. The jackknife method was used to test the mean difference of a single variable from two non-independent groups (i.e. whole data vs the data of accepted cases). The type I error rate was fixed at 0.05 (2-tailed).
Simulation
Request a detailed protocolTo strengthen the arguments on the performance of the Goldberg cutoffs, we also generated and analyzed data that preserve the characteristics of the IDATA.
Data generation
Request a detailed protocolWe generated data composed of four variables: NIBIO (biomarker), NISR (self-reported), HO (health outcomes: BW, WC, HR, SBP, DBP, and VO2), and FFM (fat-free mass). NIBIO and FFM were resampled from the IDATA. HO is computed using a linear model using NIBIO as a predictive variable:
where and are the model parameters estimated from the IDATA. NISR is generated by adding an error term (i.e., reporting error), , to NIBIO, adapting the approach proposed by Ward et al., 2019:
where is determined by the percentile of NIBIO in the empirical distribution, . The reporting error is assumed to follow a normal distribution parametrized by percentile-specific mean, , and a constant standard deviation, : . µ and were estimated by fitting polynomial models to the IDATA: , where is the order of the polynomial. The order of the polynomial function was varied from 1 to 5 and we selected the value of that yields the smallest MSE. We used k-fold cross-validation with 10 folds to calculate the test MSE for each model. This resulted in for EI, SI, PoI, and PrI, respectively (Figure 4—figure supplement 2 and Supplementary file 4). For example, assuming EIBIO is 3000 kcal, which corresponds to 90.4 percentile, the mean of the reporting error, μ, was –749.6 and the standard deviation of the reporting error, , was 759.1. Therefore his/her is randomly sampled from the normal distribution: . We further confirmed our assumption of homoskedastic is reasonable by the Goldfeld Quandt test (Goldfeld and Quandt, 1965; Supplementary file 3). The Goldfeld Quandt test compares variance in high and low values of a variable; we discarded the middle 20% of the total observations (eliminated the center 61 observations among 303 in total) to define the high and low groups. The simulated data were truncated so that NISR >0. The variable NIG was generated by applying the Goldberg cutoffs to EISR using FFM (thus, NIG is a subset of NISR). The above process was repeated for each NI. We generated individuals, and the data generation was repeated 1000 times, leading to 1000 replicates. The data generating process is summarized in Figure 4.
Measurement of the performance of the Goldberg cutoffs
Request a detailed protocolThe point estimates of the regression coefficients for an NI and an HO from the ith simulation are denoted as (e.g., is the ith regression coefficient for self-reported energy intake and body weight). The performance of the Goldberg cutoffs was assessed by three metrics: bias, mean squared error (MSE), and coverage probability:
where , is an indicator function, and represent the lower and upper 95% confidence intervals, and is the true parameter value, which we define as the point estimate from the IDATA. Note that bias is used differently here than in the empirical analysis because the parameters remain unknown in the empirical analysis whereas we define the parameters in the simulation.
Sensitivity analyses were performed varying the sample size and two other parameters: error term for the regression between health outcome and biomarker-based nutrition intakes (η), and the standard deviation of the reporting error (σ). All simulations and analyses were performed using the statistical computing software R (version 4.0.1); code book and analytic code are publicly and freely available without restriction at http://doi.org/10.5281/zenodo.7783842. The variables, data generation models, and metrics of performance of the Goldberg cutoffs are summarized in Table 1. STROBE guideline was followed, and the statement is available as supplementary material.
Data availability
The data can be obtained from NCI after permission, and we are not allowed to redistribute the data. Researchers interested in accessing the original data should submit a project proposal. Specifically, "To gain access to available IDATA data and/or biospecimens, you must submit a project proposal. These are reviewed by NCI. If your project is approved, you will be required to complete a Data Transfer Agreement (and a Material Transfer Agreement, if applicable) before you will be granted access". (https://cdas.cancer.gov/idata/). All simulations and analyses were performed using the statistical computing software R (version 4.0.1). Code book and analytic code will be made publicly and freely available without restriction at http://doi.org/10.5281/zenodo.7783842.
-
ZenodoBias in nutrition-health associations is not eliminated by excluding extreme reporters in empirical or simulation studies.https://doi.org/10.5281/zenodo.7783842
References
-
Book3 - energy metabolism and obesityIn: Mela DJ, editors. Food, diet and obesity. Woodhead Publishing. pp. 58–75.
-
Critical evaluation of energy intake using the goldberg cut-off for energy intake: Basal metabolic rate A practical guide to its calculation, use and limitationsInternational Journal of Obesity and Related Metabolic Disorders 24:1119–1130.https://doi.org/10.1038/sj.ijo.0801376
-
Total energy intake of the US population: The third national health and nutrition examination survey, 1988-1991The American Journal of Clinical Nutrition 62:1072S–1080S.https://doi.org/10.1093/ajcn/62.5.1072S
-
Toward more rigorous and informative nutritional epidemiology: The rational space between dismissal and defense of the status quoCritical Reviews in Food Science and Nutrition 2021:1–18.https://doi.org/10.1080/10408398.2021.1985427
-
Body composition as a determinant of energy expenditure: A synthetic review and a proposed General prediction equationThe American Journal of Clinical Nutrition 54:963–969.https://doi.org/10.1093/ajcn/54.6.963
-
High energy expenditure masks low physical activity in obesityInternational Journal of Obesity 37:1006–1011.https://doi.org/10.1038/ijo.2012.172
-
Energy balance measurement: When something is not better than nothingInternational Journal of Obesity 39:1109–1113.https://doi.org/10.1038/ijo.2014.199
-
Energy expenditure in adults living in developing compared with industrialized countries: A meta-analysis of doubly labeled water studiesThe American Journal of Clinical Nutrition 93:427–441.https://doi.org/10.3945/ajcn.110.007278
-
The jackknife estimate of varianceThe Annals of Statistics 9:1176345462.https://doi.org/10.1214/aos/1176345462
-
Does exclusion of extreme reporters of energy intake (the goldberg cutoffs) reliably reduce or eliminate bias in nutrition studies? analysis with illustrative associations of energy intake with health outcomesThe American Journal of Clinical Nutrition 110:1231–1239.https://doi.org/10.1093/ajcn/nqz198
-
Murine genetic models of obesity: Type I error rates and the power of commonly used analyses as assessed by plasmode-based simulationInternational Journal of Obesity 44:1440–1449.https://doi.org/10.1038/s41366-020-0554-2
-
Can we use biomarkers in combination with self-reports to strengthen the analysis of nutritional epidemiologic studies?Epidemiologic Perspectives & Innovations 7:2.https://doi.org/10.1186/1742-5573-7-2
-
Dealing with dietary measurement error in nutritional cohort studiesJournal of the National Cancer Institute 103:1086–1092.https://doi.org/10.1093/jnci/djr189
-
Critical evaluation of energy intake data using fundamental principles of energy physiology: 1. derivation of cut-off limits to identify under-recordingEuropean Journal of Clinical Nutrition 45:569–581.
-
Some tests for homoscedasticityJournal of the American Statistical Association 60:539–547.https://doi.org/10.1080/01621459.1965.10480811
-
The calorie: Myth, measurement, and realityThe American Journal of Clinical Nutrition 62:1034S–1041S.https://doi.org/10.1093/ajcn/62.5.1034S
-
Under- and overreporting of energy intake related to weight status and lifestyle in a nationwide sampleThe American Journal of Clinical Nutrition 68:266–274.https://doi.org/10.1093/ajcn/68.2.266
-
Structure of dietary measurement error: Results of the open biomarker studyAmerican Journal of Epidemiology 158:14–21.https://doi.org/10.1093/aje/kwg091
-
Who underreports dietary intake in a dietary recall? Evidence from the second National health and nutrition examination surveyJournal of Consulting and Clinical Psychology 63:438–444.https://doi.org/10.1037//0022-006x.63.3.438
-
Laboratory and field methods for measuring human energy expenditureAmerican Journal of Human Biology 24:372–384.https://doi.org/10.1002/ajhb.22260
-
Measurement of dietary intake in childrenThe Proceedings of the Nutrition Society 59:279–293.https://doi.org/10.1017/s0029665100000318
-
Markers of the validity of reported energy intakeThe Journal of Nutrition 133 Suppl 3:895S–920S.https://doi.org/10.1093/jn/133.3.895S
-
Assessing dietary intake: Who, what and why of under-reportingNutrition Research Reviews 11:231–253.https://doi.org/10.1079/NRR19980017
-
Understanding seasonal changes to improve good practices in livestock managementFrontiers in Public Health 6:175.https://doi.org/10.3389/fpubh.2018.00175
-
Measurement of active and sedentary behavior in context of large epidemiologic studiesMedicine and Science in Sports and Exercise 50:266–276.https://doi.org/10.1249/MSS.0000000000001428
-
Procedures for screening out inaccurate reports of dietary energy intakePublic Health Nutrition 5:873–882.https://doi.org/10.1079/PHN2002387
-
Sources of error in dietary surveysCanadian Journal of Public Health / Revue Canadienne de Sante’e Publique 36:355–361.
-
Dietary influences on bone mass and bone metabolism: Further evidence of a positive link between fruit and vegetable consumption and bone health?The American Journal of Clinical Nutrition 71:142–151.https://doi.org/10.1093/ajcn/71.1.142
-
Body composition, diet, and physical activity: A longitudinal cohort study in preschoolers with cerebral palsyThe American Journal of Clinical Nutrition 105:369–378.https://doi.org/10.3945/ajcn.116.137810
-
Dietary intake patterns are reflected in metabolomic profiles: Potential role in dietary assessment studiesThe American Journal of Clinical Nutrition 93:314–321.https://doi.org/10.3945/ajcn.110.000950
-
Comparison of self-reported dietary intakes from the automated self-administered 24-h recall, 4-d food records, and food-frequency questionnaires against recovery biomarkersThe American Journal of Clinical Nutrition 107:80–93.https://doi.org/10.1093/ajcn/nqx002
-
Validity of a combined heart rate and motion sensor for the measurement of free-living energy expenditure in very active individualsJournal of Science and Medicine in Sport 17:387–393.https://doi.org/10.1016/j.jsams.2013.09.006
-
Measurement of energy expenditure in humans by doubly labeled water methodJournal of Applied Physiology 53:955–959.https://doi.org/10.1152/jappl.1982.53.4.955
-
Energy expenditure by doubly labeled water: Validation in humans and proposed calculationThe American Journal of Physiology 250:R823–R830.https://doi.org/10.1152/ajpregu.1986.250.5.R823
-
Inaccuracies in self-reported intake identified by comparison with the doubly labelled water methodCanadian Journal of Physiology and Pharmacology 68:941–949.https://doi.org/10.1139/y90-143
-
Dietary antioxidants and asthma in adults: Population-based case-control studyAmerican Journal of Respiratory and Critical Care Medicine 164:1823–1828.https://doi.org/10.1164/ajrccm.164.10.2104061
-
Equations for predicting the energy requirements of healthy adults aged 18-81 YThe American Journal of Clinical Nutrition 69:920–926.https://doi.org/10.1093/ajcn/69.5.920
-
Projected U.S. state-level prevalence of adult obesity and severe obesityThe New England Journal of Medicine 381:2440–2450.https://doi.org/10.1056/NEJMsa1909301
Article and author information
Author details
Funding
National Heart, Lung, and Blood Institute (R25HL124208)
- Andrew W Brown
Japan Society for the Promotion of Science (KAKENHI grant 18K18146)
- Keisuke Ejima
National Institute of Diabetes and Digestive and Kidney Diseases (R25DK099080)
- Andrew W Brown
National Institute of General Medical Sciences (R25GM141507)
- Andrew W Brown
National Institute of Diabetes and Digestive and Kidney Diseases (1R01DK132385-01)
- Andrew W Brown
National Cancer Institute (U01-CA057030-29S1)
- Roger S Zoh
Meiji Yasuda Life Foundation of Health and Welfare
- Keisuke Ejima
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Copyright
© 2023, Yamamoto, Ejima et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics
-
- 670
- views
-
- 60
- downloads
-
- 2
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading
-
- Computational and Systems Biology
- Developmental Biology
Understanding the principles underlying the design of robust, yet flexible patterning systems is a key problem in developmental biology. In the Drosophila wing, Hedgehog (Hh) signaling determines patterning outputs using dynamical properties of the Hh gradient. In particular, the pattern of collier (col) is established by the steady-state Hh gradient, whereas the pattern of decapentaplegic (dpp), is established by a transient gradient of Hh known as the Hh overshoot. Here we use mathematical modeling to suggest that this dynamical interpretation of the Hh gradient results in specific robustness and precision properties. For instance, the location of the anterior border of col, which is subject to self-enhanced ligand degradation is more robustly specified than that of dpp to changes in morphogen dosage, and we provide experimental evidence of this prediction. However, the anterior border of dpp expression pattern, which is established by the overshoot gradient is much more precise to what would be expected by the steady-state gradient. Therefore, the dynamical interpretation of Hh signaling offers tradeoffs between
-
- Computational and Systems Biology
- Neuroscience
Animal behaviour alternates between stochastic exploration and goal-directed actions, which are generated by the underlying neural dynamics. Previously, we demonstrated that the compositional Restricted Boltzmann Machine (cRBM) can decompose whole-brain activity of larval zebrafish data at the neural level into a small number (∼100-200) of assemblies that can account for the stochasticity of the neural activity (van der Plas et al., eLife, 2023). Here, we advance this representation by extending to a combined stochastic-dynamical representation to account for both aspects using the recurrent temporal RBM (RTRBM) and transfer-learning based on the cRBM estimate. We demonstrate that the functional advantage of the RTRBM is captured in the temporal weights on the hidden units, representing neural assemblies, for both simulated and experimental data. Our results show that the temporal expansion outperforms the stochastic-only cRBM in terms of generalization error and achieves a more accurate representation of the moments in time. Lastly, we demonstrate that we can identify the original time-scale of assembly dynamics by estimating multiple RTRBMs at different temporal resolutions. Together, we propose that RTRBMs are a valuable tool for capturing the combined stochastic and time-predictive dynamics of large-scale data sets.