It is becoming increasingly common for scientists to share their data with other researchers. This makes it possible to independently verify reported results, which increases trust in research. Sometimes it is not possible to share certain datasets because they include sensitive information about individuals. In psychology and medicine, scientists have tried to remove identifying information from datasets before sharing them by, for example, adding minor artificial errors. But, even when researchers take these steps, it may still be possible to identify individuals, and the introduction of artificial errors can make it harder to verify the original results.

One potential alternative to sharing sensitive data is to create ‘synthetic datasets’. Synthetic datasets mimic original datasets by maintaining the statistical properties of the data but without matching the original recorded values. Synthetic datasets are already being used, for example, to share confidential census data. However, this approach is rarely used in other areas of research. Now, Daniel S. Quintana demonstrates how synthetic datasets can be used in psychology and medicine.

Three different datasets were studied to ensure that synthetic datasets performed well regardless of the type or size of the data. Quintana evaluated freely available software that could generate synthetic versions of these different datasets, which essentially removed any identifying information. The results obtained by analysing the synthetic datasets closely mimicked the original results.

These tools could allow researchers to verify each other’s results more easily without jeopardizing the privacy of participants. This could encourage more collaboration, stimulate ideas for future research, and increase data sharing between research groups.

Openly accessible biomedical research data provide enormous utility for science, society, and the economy (Arzberger et al., 2004; Munafò et al., 2017; Murdoch and Detsky, 2013; Piwowar et al., 2011). With open data, scholars can verify results, generate new knowledge, form new hypotheses, and reduce the unnecessary duplication of data collection (Asendorpf et al., 2013; Nosek et al., 2012). However, the benefits of data sharing need to be considered in light of disclosure risk. Researchers who wish to share data while reducing the risk of disclosure have traditionally used data anonymization procedures, in which explicit identifiers such as names, addresses, and national identity numbers are removed (Hrynaszkiewicz et al., 2010). To add additional disclosure protection, particularly sensitive variables (e.g., age) are sometimes aggregated and random noise may be added to the dataset. Despite these anonymization efforts, specific individuals can still be identified in anonymized datasets with high accuracy (Ohm, 2009; Rocher et al., 2019). Data aggregation and random noise can also distort the relationships between variables in the dataset (Purdam and Elliot, 2007), which can interfere with reproducibility and exploratory data analysis.

The creation of synthetic datasets can substantially overcome replicability issues, as this method creates a new dataset that mimics an original dataset by preserving its statistical properties and relationships between variables (Little, 1993; Reiter, 2005b; Reiter, 2005a; Reiter and Raghunathan, 2007; Rubin, 1993). Synthetic datasets also reduce disclosure risk to essentially zero, as no complete casewise record in the new dataset represents a real individual (Duncan and Elliot, 2011). Synthetic datasets also allow researchers to fit exploratory models in the synthetic datasets, which the data custodians can verify in the original data. Finally, synthetic datasets enable readers and reviewers to better understand the data, as they can recreate the reported analyses and explore data distributions, variance, outliers, and means.

Synthetic datasets were originally developed for sharing sensitive population-level data (for a summary, see Bonnéry et al., 2019). The use of synthetic data for sharing sensitive information is beginning to emerge in the biobehavioural sciences (e.g., Arslan et al., 2018; Newbury et al., 2018); however, this approach is not widely known in the field. Given the benefits of synthetic data, the purpose of this article is to introduce this concept using examples and an accompanying R script. The R script and datasets to reproduce the analyses described in this paper are available online at https://github.com/dsquintana/synthpop-primer (Quintana, 2019; copy archived at https://github.com/elifesciences-publications/synthpop-primer). This website also includes a link to a RStudio Server instance of the primary analysis and results, which recreates the complete computational environment used for this manuscript (i.e., the R version and R package versions used) and facilitates straightforward reproducibility of the analysis described in this article via any modern web browser.

Three open datasets will be used to demonstrate how the generation of a synthetic dataset can produce a dataset that mimics the original, via the synthpop R package (version 1.5–1; Nowok et al., 2016) in the R statistical environment (version 3.6.0). Synthetic datasets are created by replacing some or all of the data by sampling from an appropriate probability distribution, in a similar fashion to multiple imputation (Drechsler, 2011; Raghunathan et al., 2003; Reiter, 2005b). This approach preserves the statistical properties and the relationships between variables in the original dataset while safeguarding anonymity as no individual in the new dataset represents a real individual. The default synthesis method in synthpop is the classification and regression tree (CART) procedure (Reiter, 2005b), which will be used in this primer. The CART procedure is more flexible and generally performs better than other synthesis methods, such as random forests and bagging (Drechsler and Reiter, 2011). If data synthesis happens to recreate a replicate of a real individual by chance, these replicates can be easily identified and removed from the dataset to reduce the risk of de-identification.

There are two broad approaches for assessing the utility of a synthesized dataset: general utility and specific utility (Snoke et al., 2018). General utility reflects the overall similarities in the statistical properties and multivariate relationships between the synthetic and original datasets. The first step in assessing general utility is data visualisation (Nowok et al., 2016; Raab et al., 2017). The compare() function in synthpop can be used to construct side-by-side univariate distributions of variables in the synthetic and observed datasets. Visualizing bivariate comparisons between specific variables of interest is also recommended, as two datasets might have similar statistical properties despite different distributions (e.g., Anscombe’s quartet; Anscombe, 1973). Confirming general utility is a necessary step for making inferences from the synthetic dataset and is especially important for data exploration in the synthetic dataset (Snoke et al., 2018).

Specific utility for fitted synthetic models can be assessed by calculating the lack-of-fit against the same model in the original data (Nowok et al., 2016). Specific coefficients in synthetic models can also be compared against coefficients in the original models by assessing the differences in standardized coefficients and confidence interval (CI) overlap (Karr et al., 2006), which is the ratio of the overlap of the intervals to an average of their lengths. Higher overlap is indicative of greater specific utility, which suggests that inference from fitted synthetic models is valid.

Finally, there is a risk that individuals coming across the synthetic data without context (i.e., a direct link to a data file) may believe the data are real. Thus, to remove any possible confusion, synthetic datasets prepared for export will have a single string ‘FAKE_DATA’ variable added to the front of the dataset using the sdc() function in synthpop, as recommended by Nowok et al. (2017).

Example 1: Oxytocin and self-reported spirituality

Van Cappellen et al. (2016) investigated the impact of oxytocin administration on self-reported spirituality and deposited the raw study data online (https://osf.io/rk2x7/). In a between-participants design, volunteers were randomly assigned to self-administer an intranasal oxytocin (N = 41) or intranasal placebo spray (N = 42). Approximately forty minutes after receiving the nasal spray, participants were asked “Right now, would you say that spirituality is important in your life?”. The reported outcome from an ANCOVA suggested that when accounting for religious affiliation, participants who self-administered the oxytocin nasal spray reported that spirituality was more important in their lives compared to those who self-administered the placebo spray. A synthetic version of the original dataset was created using the syn() function from the synthpop package. A comparison of the four main variables of interest revealed similar distributions between the synthetic and the original datasets and no individual extreme values (Figure 1A). There were also no replicated unique sets of values. A bivariate comparison of self-reported spiritualty (Figure 1—figure supplement 1) and religious affiliation (Figure 1—figure supplement 2) between the nasal spray groups suggested that the counts between the synthesized and original datasets were similar. The relationship between age and self-reported spirituality was also similar between datasets (Figure 1—figure supplement 3). Altogether, these visualisations were indicative that the synthetic dataset has good general utility.

Figure 1 with 3 supplements see all

Download asset Open asset

Nasal spray group differences in self-reported spiritualty will be examined using both an independent samples Welch’s t-test and a linear regression model equivalent, with the former required to assess specific utility. This analysis in the original dataset suggested no significant difference in spirituality ratings between the nasal spray groups [t = 1.14, 95% CI (−0.45, 1.63), p=0.26]. An equivalent linear regression model yielded the same outcome, as expected. Estimating this linear model in the synthesized dataset revealed the same p-value outcome (t = −1.12, p=0.26).

The lack-of-fit test comparing the models generated in the original and synthetic datasets was not statistically significant, X² (2)=0.01, p=0.995. This suggests that the method used for synthesis retained all the relationships between variables that influenced the model fitting parameters. The standardized difference of the nasal spray condition coefficient between the synthesized and the observed data for this t-test was 0.003, which was not statistically significant (p=0.998). A comparison of confidence intervals revealed 99.94% CI overlap between the synthetic and original datasets (Figure 1B). Overall, these results indicate that the model from the synthesized data demonstrates high specific utility for this t-test. This particular outcome was not reported in the original article; however, this provides a demonstration of how synthetic data can be used for data exploration. The analysis script underlying exploratory analysis can be shared with the owners of the original dataset, for the easy verification of the analysis. In this case, applying this model to the synthetic data produces almost precisely the same results as the original dataset.

Next, let’s explore the correlation between age and self-reported spirituality. A Pearson correlation test revealed no statistically significant correlation between age and self-reported spirituality [r = 0.04, 95% CI (−0.19, 0.26), p=0.75], which is the same result as the linear model equivalent. Estimating this model using the synthetic dataset revealed a similar Pearson’s r value and a non-significant result (r = 0.08, p=0.5). The lack-of-fit test comparing the models generated in the original and synthetic datasets was not statistically significant, X² (2)=0.13, p=0.94. Moreover, the test of the standardized differences between the synthetic and original data for the relationship coefficient was not statistically significant (p=0.72) and there was 90.8% CI overlap between the synthetic and observed data (Figure 1C), suggesting high specific utility.

Finally, let’s explore the main reported outcomes from this study, that oxytocin increased self-reported spirituality controlling for religious affiliation. First, the original outcome was verified [F(1,75)=4.87, p=0.03] and then the analysis was structured as a linear regression model. This analysis yielded the same p-value of .03, which was associated with a t-statistic of −2.2. Estimating this model using the synthetic dataset revealed similar t-statistic and a p-value that was on the border of statistical significance (t = −1.8, p=0.07).

The lack-of-fit test for the full model was not statistically significant, X² (3)=0.91, p=0.82. The test of the standardized differences between the synthetic and original data for the nasal spray coefficient was not statistically significant (p=0.63) and there was 87.8% overlap between the synthetic and observed data (Figure 1D). Although the synthetic model nasal spray coefficient was not statistically significant, like the original model, this matters little given the considerable overlap between the confidence intervals of the synthetic and original models. It is worth a reminder at this point that just because one coefficient is significant and the other is not, this does not necessarily mean that these coefficients are significantly different from each other (Gelman and Stern, 2006). The primary interest in the comparison of synthetic and original models is effect size estimation and confidence interval overlap. The test of the standardized differences between the synthetic and original data for the religious affiliation coefficient was not statistically significant (p=0.49) and there was 82.3% overlap between the synthetic and observed data (Figure 1D). Overall, these results are indicative of high specific utility for the overall synthesized model and its coefficients.

Example 2: Sociosexual orientation and self-rated attractiveness

Sociosexual orientation is described as an individual’s propensity to participate in uncommitted sexual relationships. Given the personal nature of sociosexual orientation, this a good example of the type of sensitive data that individuals may be hesitant to share in some cases, thus demonstrating the benefit of releasing of synthetic data. Jones and DeBruine, 2019 collected sociosexual orientation data from 9627 individuals using a revised version of the sociosexual orientation inventory (Penke and Asendorpf, 2008) both in the laboratory and online, along with data on self-rated attractiveness and basic demographic information. Fourteen variables were synthesized from the original dataset, which has been archived online (https://osf.io/6bk3w/). The synthetic data demonstrated good general utility, as the distributions of variables were comparable between the original and synthetic datasets (Figure 2A). A model was fitted to examine if self-rated attractiveness, data collection location (laboratory or online), and age predicted the number of times someone has had sexual intercourse on only a single occasion with another individual. Both self-rated attractiveness (t = 13.64, p<0.001) and age (t = 27.69, p<0.001) were statistically significant predictors, whereas location was not a significant predictor (t = 0.05, p=0.96). Estimating this linear model in the synthesized dataset revealed relatively similar t-statistic outcomes (self-rated attractiveness: t = 14, p>0.001; age: t = 27.51, p<0.001; location: t = 1.12, p=0.26).

Figure 2

Download asset Open asset

The lack-of-fit test was not statistically significant, [X² (4)=1.5, p=0.83], suggesting that the method used for synthesis retained all the relationships between variables that influenced the parameters of the fit. The standardized difference of the self-rated attractiveness coefficient between the synthesized and the observed data was 0.32, which was not statistically significant (p=0.75). A comparison of confidence intervals revealed 91.8% CI overlap between the synthetic and original datasets (Figure 2B). The standardized difference of the age coefficient between the synthesized and the observed data for this t-test was −0.28, which was not statistically significant (p=0.78). A comparison of confidence intervals revealed 92.9% CI overlap between the synthetic and original datasets (Figure 2B). The standardized difference of the location coefficient between the synthesized and the observed data for this t-test was 1.08, which was not statistically significant (p=0.28). A comparison of confidence intervals revealed 72.4% CI overlap between the synthetic and original datasets (Figure 2B). Overall, these results indicate that this model from the synthesized data demonstrated high specific utility.

A comparison of the original and synthetic datasets revealed 213 replicated individuals (2.3% of the total sample). To reduce disclosure risk, these replicated values were removed and models were refitted to examine the effects of removal on outcomes. The lack-of-fit test between the original model and synthetic model with replicated individuals was not statistically significant, X² (4)=2.8, p=0.6. The refitted model coefficients yielded similar results to the model derived from the original data (self-rated attractiveness: standardized difference = 0.3, p=0.77, CI overlap = 92.4%; age: standardized difference = −0.74, p=0.46, CI overlap = 81.2%; location: standardized difference = 0.95, p=0.34, CI overlap = 75.8%; Figure 2C). Therefore, reducing the risk of disclosure by removing replicated individuals in the synthetic dataset maintains specific utility, in this case.

Example 3: Heart rate variability and fitness levels in a series of simulated datasets

Heart Rate Variability (HRV) is a non-invasive measure of autonomic cardiac control (Akselrod et al., 1981) and thought to be positively correlated with fitness level (Dixon et al., 1992). The Root Mean Square of Successive Differences (RMSSD) is a commonly used HRV measure, however, its distribution tends to be positively skewed (e.g., Kobayashi et al., 2012). Moreover, missing data are common in HRV investigations due to equipment malfunction. To demonstrate the effects of the distribution pattern of HRV (normal, low skew, high skew), and missing data (none, 5%, 20%) for a range of sample sizes (40, 100, 10000), twenty-seven data sets with four variables (heart rate, weight, fitness level, and HRV) were simulated for the creation of synthetic datasets, which included outliers (Supplementary file 1). For all datasets, HRV and fitness level were modelled to have a relationship that is typically associated with a medium effect size (r = 0.3).

The specific utility of synthetic datasets was examined by comparing the relationship between HRV and fitness in each synthetic dataset to its respective original dataset. None of the lack-of-fit tests were statistically significant (Supplementary file 1), suggesting that the method used for synthesis retained all the relationships between variables that influenced the parameters of the fit. A comparison of confidence intervals revealed strong overlap between the synthetic and original datasets for most (but not all) of the 27 analyses and none of the standardized coefficient differences between the synthesized and the observed datasets were statistically significant (all p’s > 0.05; Figure 3, Figure 3—figure supplements 1–2). However, the overlap between the synthetic and original models were on the border of statistical significance when synthesizing data with a low skew in the simulated samples with 10,000 cases (Supplementary file 1; Figure 3—figure supplement 2). All 27 synthetic datasets also demonstrated good general utility, regardless of the parameters (Figure 3—figure supplements 3–5). Thus, synthetic dataset generation in synthpop seems to be relatively robust against differences in sample size, missingness, and skew in these simulated samples, however, there were indications of poorer performance in some of larger datasets with 10,000 (Supplementary file 1). Altogether, it is crucial that general and specific utility is assessed for each synthesised dataset, as it is difficult to predict before synthesis how well the procedure will perform.

Figure 3 with 5 supplements see all

Download asset Open asset

Researchers need to consider the trade-off between the risk of identification and the utility of open datasets when deciding whether to make their research data openly available. Open data can provide substantial utility, but this may expose research participants to the risk of identification. Conversely, closed datasets decrease the risk of disclosure to essentially zero, but have almost no public utility. The generation of synthetic datasets provides an appealing compromise, as synthetic data can offer comparable levels of utility as the original datasets while substantially reducing disclosure risk. The adoption of synthetic datasets in the biobehavioural sciences will improve reproducibility and secondary data exploration, as it will facilitate the sharing of data that would otherwise not be made available.

Study participants are generally in favour of researchers sharing their deidentified data (Ludman et al., 2010; Mello et al., 2018). Thus, when planning future studies researchers should include data sharing provisions when receiving participant consent (Taichman et al., 2016). Obtaining updated consent to share data from participants who have not provided this when originally participating in a study can be resource intensive. Some have suggested that sharing deidentified datafiles should not require new re-consent from participants (Taichman et al., 2017), but as mentioned above, many commonly used data deidentification approaches may not sufficiently reduce disclosure risk (Ohm, 2009; Rocher et al., 2019). In some circumstances, datasets may include extremely sensitive information that is difficult to anonymise. For instance, Arslan et al. (2018) collected highly sensitive data examining the role of ovulatory changes on sexual desire and behavior in women but did not request consent from participants to share data considering valid privacy concerns. Instead, a synthetic version of the dataset was created using synthpop and made available on a publicly accessible repository. Releasing this synthetic dataset provides considerable utility, as other researchers can verify the analysis and fit novel models using this dataset, which can be confirmed by the data custodians with the original data. An additional step of removing individuals that have been fully replicated in the synthesized data set can further reduce disclosure risk, without necessarily reducing general or specific utility. Therefore, synthetic data can offer a valuable solution for sharing data collected under conditions where participants did not specifically provide consent for sharing data (and where re-consent is impractical), as well as for situations in which a dataset contains especially sensitive information. In addition to the verification of results and hypothesis generation, synthetic datasets can also benefit the training of machine learning algorithms in research areas with a dearth of data, such as rare condition research (Ekbatani et al., 2017; Sabay et al., 2018), via the creation of additional synthetic datasets that closely match real datasets.

One criticism of sharing raw data is that research groups would not have the first opportunity to analyse the data and report outcomes to their satisfaction (Lo, 2015; Ross et al., 2012). It has been recommended that secondary data analysts should seek collaborations with teams that collected the original data in recognition of their investment in collecting the data (Taichman et al., 2016), but this is difficult to enforce in practice. To make meaningful inferences with synthetic data, secondary data analysts need to verify their synthetic models against models from the original data, which is in the possession of the original authors who can verify these inferences (Reiter et al., 2009). This would increase the likelihood of co-authored collaborations, at least compared to the status-quo in which secondary analysts could publish their results without necessarily collaborating with the original authors. Thus, open synthetic data provide an opportunity for secondary analysists scholars to fit models that the original authors may not have considered, while also encouraging them to collaborate with the original authors to verify their models in the real dataset. Of course, secondary analysts could still report results from synthetic datasets without verification from the primary authors, but it would need to be made explicit that analyses were conducted on a synthetic dataset, and generated models may not necessarily mirror the models generated from the original dataset.

Journals have adopted a spectrum of public data archiving (PDA) policies, ranging from the policies that data should be made “available upon request” all the way to mandated data deposition in peer-reviewed journals dedicated to open data (Sholler et al., 2019). While an “available upon request” PDA policy is better than no policy at all (Stodden et al., 2018), such datasets are often difficult to retrieve in practice as corresponding authors can become unreachable or original datasets are lost (Couture et al., 2018; Stodden et al., 2018; Wicherts et al., 2006). Sharing data with published papers would remove these impediments for accessing data, with synthetic data offering a solution for when it is not possible to share the original dataset due to disclosure concerns.

Despite the benefits of synthetic datasets, this approach is not without limitations. First, it is possible for synthetic data to have poor general and specific utility, which would diminish the benefits of sharing in terms of reproducibility and secondary data exploration. While a synthetic dataset with poor utility would still provide a valuable guide for reproducing reported analyses, these are likely to provide substantially different estimates and exploratory analyses may not produce accurate models. Second, current synthetic data methods limit the types of statistical inference that can be performed on synthetic data to linear models. In practice, this means that only linear models can be for comparison in order to demonstrate specific utility. Of course, scholars are free to perform any type of analysis on the synthetic data, which should provide approximately the same outcome as the original data as long as the synthetic data offer good general utility. Third, as mentioned above, the risk of identity disclosure from synthetic datasets is negligible but this only holds under two conditions: that none of the complete synthetic data records match with the original data and that there are no extreme single individual values in the dataset that can be linked to an individual (Drechsler, 2011; Duncan and Elliot, 2011). Therefore, to reduce disclosure risk and the possibility that participants will recognise themselves in the dataset, complete matching records (i.e., when all variables in the original dataset for a case matches a case in the synthetic dataset) should be identified and removed. Additionally, in the case of categorical variables with only a few observations, scholars should consider collapsing these into another category (e.g., if there are only a few observations in an age band of 70–79 years old, this can be collapsed into the previous age band of 60–69 years old). If there are uncommon continuous values above or below a certain threshold, it may be prudent to collapse these into another category or creating a new category (e.g., top-coding a new ‘70+’ age variable for any age equal to or above 70). While recoding may lead to synthetic datasets with less utility, this approach might be required to reduce disclosure risk, something that data synthesizers will have to carefully consider in light of the sensitivity of the dataset along with national laws and guidelines.

When the creation of synthetic datasets for disclosure control was first proposed in the early 1990s, it was considered “rather radical” at the time (pg. 461; Rubin, 1993). Researchers have continued improving this method since these initial proposals (Reiter, 2005b; Reiter, 2005a; Reiter and Raghunathan, 2007), but only more recently has an easy-to-implement tool for creating synthetic data become available. The synthpop R package enables researchers to generate and share synthetic datasets that mimic original datasets with sensitive information. Importantly, the use synthetic data will improve the reproducibility of biobehavioral research and help generate novel hypotheses for future research (Bonnéry et al., 2019).

Data and analysis scripts are available at the article's Open Science Framework webpage https://osf.io/z524n/.

1. 1. Van Cappellen P
   2. Way BM
   3. Isgett SF
   4. Fredrickson BL

   (2016) Open Science Framework

   Effects of oxytocin administration on spirituality and emotional responses to meditation.

   https://osf.io/rk2x7/
2. 1. Jones BC
   2. DeBruine L

   (2019) Open Science Framework

   Sociosexuality and self-rated attractiveness.

   https://osf.io/6bk3w/

1. 1. Akselrod S
   2. Gordon D
   3. Ubel FA
   4. Shannon DC
   5. Berger AC
   6. Cohen RJ

   (1981) Power spectrum analysis of heart rate fluctuation: a quantitative probe of beat-to-beat cardiovascular control

   Science 213:220–222.

   https://doi.org/10.1126/science.6166045

   • PubMed
   • Google Scholar
2. 1. Anscombe FJ

   (1973) Graphs in statistical analysis

   The American Statistician 27:17–21.

   https://doi.org/10.1080/00031305.1973.10478966

   • Google Scholar
3. 1. Arslan RC
   2. Schilling KM
   3. Gerlach TM
   4. Penke L

   (2018) Using 26,000 diary entries to show ovulatory changes in sexual desire and behavior

   Journal of Personality and Social Psychology 26:208.

   https://doi.org/10.1037/pspp0000208

   • Google Scholar
4. 1. Arzberger P
   2. Schroeder P
   3. Beaulieu A
   4. Bowker G
   5. Casey K
   6. Laaksonen L
   7. Moorman D
   8. Uhlir P
   9. Wouters P

   (2004) Promoting access to public research data for scientific, economic, and social development

   Data Science Journal 3:135–152.

   https://doi.org/10.2481/dsj.3.135

   • Google Scholar
5. 1. Asendorpf JB
   2. Conner M
   3. De Fruyt F
   4. De Houwer J
   5. Denissen JJA
   6. Fiedler K
   7. Fiedler S
   8. Funder DC
   9. Kliegl R
   10. Nosek BA
   11. Perugini M
   12. Roberts BW
   13. Schmitt M
   14. van Aken MAG
   15. Weber H
   16. Wicherts JM

   (2013) Recommendations for increasing replicability in psychology

   European Journal of Personality 27:108–119.

   https://doi.org/10.1002/per.1919

   • Google Scholar
6. 1. Bonnéry D
   2. Feng Y
   3. Henneberger AK
   4. Johnson TL
   5. Lachowicz M
   6. Rose BA
   7. Shaw T
   8. Stapleton LM
   9. Woolley ME
   10. Zheng Y

   (2019) The promise and limitations of synthetic data as a strategy to expand access to State-Level Multi-Agency longitudinal data

   Journal of Research on Educational Effectiveness 12:616–647.

   https://doi.org/10.1080/19345747.2019.1631421

   • Google Scholar
7. 1. Couture JL
   2. Blake RE
   3. McDonald G
   4. Ward CL

   (2018) A funder-imposed data publication requirement seldom inspired data sharing

   PLOS ONE 13:e0199789.

   https://doi.org/10.1371/journal.pone.0199789

   • PubMed
   • Google Scholar
8. 1. Dixon EM
   2. Kamath MV
   3. McCartney N
   4. Fallen EL

   (1992) Neural regulation of heart rate variability in endurance Athletes and sedentary controls

   Cardiovascular Research 26:713–719.

   https://doi.org/10.1093/cvr/26.7.713

   • PubMed
   • Google Scholar
9. Book

   1. Drechsler J

   (2011) Synthetic Datasets for Statistical Disclosure Control: Theory and Implementation

   Springer-Verlag.

   https://doi.org/10.1007/978-1-4614-0326-5

   • Google Scholar
10. 1. Drechsler J
    2. Reiter JP

    (2011) An empirical evaluation of easily implemented, nonparametric methods for generating synthetic datasets

    Computational Statistics & Data Analysis 55:3232–3243.

    https://doi.org/10.1016/j.csda.2011.06.006

    • Google Scholar
11. Book

    1. Duncan GT
    2. Elliot M

    (2011) Statistical Confidentiality: Principles and Practice

    Springer-Verlag.

    https://doi.org/10.1007/978-1-4419-7802-8

    • Google Scholar
12. Conference

    1. Ekbatani HK
    2. Pujol O
    3. Segui S

    (2017) Synthetic data generation for deep learning in counting pedestrians

    Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods - Volume 1: ICPRAM. pp. 318–323.

    https://doi.org/10.5220/0006119203180323

    • Google Scholar
13. 1. Gelman A
    2. Stern H

    (2006) The difference between “Significant” and “Not Significant” is not Itself Statistically Significant

    The American Statistician 60:328–331.

    https://doi.org/10.1198/000313006X152649

    • Google Scholar
14. 1. Hrynaszkiewicz I
    2. Norton ML
    3. Vickers AJ
    4. Altman DG

    (2010) Preparing raw clinical data for publication: guidance for journal editors, authors, and peer reviewers

    Trials 11:9.

    https://doi.org/10.1186/1745-6215-11-9

    • PubMed
    • Google Scholar
15. Data

    1. Jones BC
    2. DeBruine LM

    (authors) (2019) Sociosexuality and self-rated attractiveness

    Open Science Framework.

    https://doi.org/10.17605/OSF.IO/6BK3W
16. 1. Karr AF
    2. Kohnen CN
    3. Oganian A
    4. Reiter JP
    5. Sanil AP

    (2006) A framework for evaluating the utility of data altered to protect confidentiality

    The American Statistician 60:224–232.

    https://doi.org/10.1198/000313006X124640

    • Google Scholar
17. 1. Kobayashi H
    2. Park BJ
    3. Miyazaki Y

    (2012) Normative references of heart rate variability and salivary alpha-amylase in a healthy young male population

    Journal of Physiological Anthropology 31:9.

    https://doi.org/10.1186/1880-6805-31-9

    • PubMed
    • Google Scholar
18. 1. Little RJ

    (1993) Statistical analysis of masked data

    Journal of Official Statistics 9:407.

    https://doi.org/10.1016/S0169-7161(01)20020-0

    • Google Scholar
19. 1. Lo B

    (2015) Sharing clinical trial data: maximizing benefits, minimizing risk

    Jama 313:793–794.

    https://doi.org/10.1001/jama.2015.292

    • PubMed
    • Google Scholar
20. 1. Ludman EJ
    2. Fullerton SM
    3. Spangler L
    4. Trinidad SB
    5. Fujii MM
    6. Jarvik GP
    7. Larson EB
    8. Burke W

    (2010) Glad you asked: participants' opinions of re-consent for dbGap data submission

    Journal of Empirical Research on Human Research Ethics 5:9–16.

    https://doi.org/10.1525/jer.2010.5.3.9

    • PubMed
    • Google Scholar
21. 1. Mello MM
    2. Lieou V
    3. Goodman SN

    (2018) Clinical trial participants' Views of the risks and benefits of data sharing

    New England Journal of Medicine 378:2202–2211.

    https://doi.org/10.1056/NEJMsa1713258

    • PubMed
    • Google Scholar
22. 1. Munafò MR
    2. Nosek BA
    3. Bishop DVM
    4. Button KS
    5. Chambers CD
    6. Percie du Sert N
    7. Simonsohn U
    8. Wagenmakers E-J
    9. Ware JJ
    10. Ioannidis JPA

    (2017) A manifesto for reproducible science

    Nature Human Behaviour 1:0021.

    https://doi.org/10.1038/s41562-016-0021

    • Google Scholar
23. 1. Murdoch TB
    2. Detsky AS

    (2013) The inevitable application of big data to health care

    Jama 309:1351–1352.

    https://doi.org/10.1001/jama.2013.393

    • PubMed
    • Google Scholar
24. 1. Newbury DF
    2. Simpson NH
    3. Thompson PA
    4. Bishop DVM

    (2018) Stage 2 registered report: variation in neurodevelopmental outcomes in children with sex chromosome trisomies: testing the double hit hypothesis

    Wellcome Open Research 3:85.

    https://doi.org/10.12688/wellcomeopenres.14677.1

    • PubMed
    • Google Scholar
25. 1. Nosek BA
    2. Spies JR
    3. Motyl M

    (2012) Scientific utopia: ii. restructuring incentives and practices to promote truth over publishability

    Perspectives on Psychological Science : A Journal of the Association for Psychological Science 7:615–631.

    https://doi.org/10.1177/1745691612459058

    • PubMed
    • Google Scholar
26. 1. Nowok B
    2. Raab GM
    3. Dibben C

    (2016) Synthpop : bespoke creation of synthetic data in R

    Journal of Statistical Software 74:1–26.

    https://doi.org/10.18637/jss.v074.i11.

    • Google Scholar
27. 1. Nowok B
    2. Raab GM
    3. Dibben C

    (2017) Providing bespoke synthetic data for the UK longitudinal studies and other sensitive data with the synthpop package for R1

    Statistical Journal of the IAOS 33:785–796.

    https://doi.org/10.3233/SJI-150153

    • Google Scholar
28. 1. Ohm P

    (2009)

    Broken promises of privacy: responding to the surprising failure of anonymization

    UCLA Law Review 57:1701.

    • Google Scholar
29. 1. Penke L
    2. Asendorpf JB

    (2008) Beyond global sociosexual orientations: a more differentiated look at Sociosexuality and its effects on courtship and romantic relationships

    Journal of Personality and Social Psychology 95:1113–1135.

    https://doi.org/10.1037/0022-3514.95.5.1113

    • PubMed
    • Google Scholar
30. 1. Piwowar HA
    2. Vision TJ
    3. Whitlock MC

    (2011) Data archiving is a good investment

    Nature 473:285.

    https://doi.org/10.1038/473285a

    • PubMed
    • Google Scholar
31. 1. Purdam K
    2. Elliot M

    (2007) A case study of the impact of statistical disclosure control on data quality in the individual UK samples of anonymised records

    Environment and Planning A: Economy and Space 39:1101–1118.

    https://doi.org/10.1068/a38335

    • Google Scholar
32. 1. Quintana DS

    (2019) A primer on using the synthpop package for the biobehavioral sciences: An executable R script (Version 1.1.0) [Computer software]

    A primer on using the synthpop package for the biobehavioral sciences: An executable R script (Version 1.1.0) [Computer software], http://doi.org/10.5281/zenodo.3404199.

    http://doi.org/10.5281/zenodo.3404199

    • Google Scholar
33. Preprint

    1. Raab GM
    2. Nowok B
    3. Dibben C

    (2017) Guidelines for producing useful synthetic data

    arXiv.

    https://arxiv.org/abs/1712.04078

    • Google Scholar
34. 1. Raghunathan TE
    2. Reiter JP
    3. Rubin DB

    (2003) Multiple imputation for statistical disclosure limitation

    Journal of Official Statistics 19:1.

    https://doi.org/10.1002/sim.3974

    • Google Scholar
35. 1. Reiter JP

    (2005a) Releasing multiply imputed, synthetic public use microdata: an illustration and empirical study

    Journal of the Royal Statistical Society: Series A 168:185–205.

    https://doi.org/10.1111/j.1467-985X.2004.00343.x

    • Google Scholar
36. 1. Reiter JP

    (2005b)

    Using CART to generate partially synthetic public use microdata

    Journal of Official Statistics 21:441.

    • Google Scholar
37. 1. Reiter JP
    2. Oganian A
    3. Karr AF

    (2009) Verification servers: Enabling analysts to assess the quality of inferences from public use data

    Computational Statistics & Data Analysis 53:1475–1482.

    https://doi.org/10.1016/j.csda.2008.10.006

    • Google Scholar
38. 1. Reiter JP
    2. Raghunathan TE

    (2007) The multiple adaptations of multiple imputation

    Journal of the American Statistical Association 102:1462–1471.

    https://doi.org/10.1198/016214507000000932

    • Google Scholar
39. 1. Rocher L
    2. Hendrickx JM
    3. de Montjoye YA

    (2019) Estimating the success of re-identifications in incomplete datasets using generative models

    Nature Communications 10:3069.

    https://doi.org/10.1038/s41467-019-10933-3

    • PubMed
    • Google Scholar
40. 1. Ross JS
    2. Lehman R
    3. Gross CP

    (2012) The importance of clinical trial data sharing

    Circulation: Cardiovascular Quality and Outcomes 5:238–240.

    https://doi.org/10.1161/CIRCOUTCOMES.112.965798

    • Google Scholar
41. 1. Rubin DB

    (1993)

    Statistical disclosure limitation

    Journal of Official Statistics 9:461–468.

    • Google Scholar
42. 1. Sabay A
    2. Harris L
    3. Bejugama V
    4. Jaceldo-Siegl K

    (2018)

    Overcoming small data limitations in heart disease prediction by using surrogate data

    SMU Data Science Review 1:12.

    • Google Scholar
43. 1. Sholler D
    2. Ram K
    3. Boettiger C
    4. Katz DS

    (2019) Enforcing public data archiving policies in academic publishing: a study of ecology journals

    Big Data & Society 6:205395171983625.

    https://doi.org/10.1177/2053951719836258

    • Google Scholar
44. 1. Snoke J
    2. Raab GM
    3. Nowok B
    4. Dibben C
    5. Slavkovic A

    (2018) General and specific utility measures for synthetic data

    Journal of the Royal Statistical Society: Series A 181:663–688.

    https://doi.org/10.1111/rssa.12358

    • Google Scholar
45. 1. Stodden V
    2. Seiler J
    3. Ma Z

    (2018) An empirical analysis of journal policy effectiveness for computational reproducibility

    PNAS 115:2584–2589.

    https://doi.org/10.1073/pnas.1708290115

    • PubMed
    • Google Scholar
46. 1. Taichman DB
    2. Backus J
    3. Baethge C
    4. Bauchner H
    5. de Leeuw PW
    6. Drazen JM
    7. Fletcher J
    8. Frizelle FA
    9. Groves T
    10. Haileamlak A
    11. James A
    12. Laine C
    13. Peiperl L
    14. Pinborg A
    15. Sahni P
    16. Wu S

    (2016) Sharing clinical trial data: a proposal from the international committee of medical journal editors

    Annals of Internal Medicine 164:505.

    https://doi.org/10.7326/M15-2928

    • PubMed
    • Google Scholar
47. 1. Taichman DB
    2. Sahni P
    3. Pinborg A
    4. Peiperl L
    5. Laine C
    6. James A
    7. Hong ST
    8. Haileamlak A
    9. Gollogly L
    10. Godlee F
    11. Frizelle FA
    12. Florenzano F
    13. Drazen JM
    14. Bauchner H
    15. Baethge C
    16. Backus J

    (2017) Data sharing statements for clinical trials: a requirement of the international committee of medical journal editors

    Jama 317:2491–2492.

    https://doi.org/10.1001/jama.2017.6514

    • PubMed
    • Google Scholar
48. 1. Van Cappellen P
    2. Way BM
    3. Isgett SF
    4. Fredrickson BL

    (2016) Effects of oxytocin administration on spirituality and emotional responses to meditation

    Social Cognitive and Affective Neuroscience 11:1579–1587.

    https://doi.org/10.1093/scan/nsw078

    • PubMed
    • Google Scholar
49. 1. Wicherts JM
    2. Borsboom D
    3. Kats J
    4. Molenaar D

    (2006) The poor availability of psychological research data for reanalysis

    American Psychologist 61:726–728.

    https://doi.org/10.1037/0003-066X.61.7.726

    • PubMed
    • Google Scholar

Acceptance summary:

We feel that the study is important as it touches on an emerging area of data sharing, provenance and reproducibility – a nascent movement towards achieving rigor and transparency that is being adopted by funding agencies, journal editors and industry alike. Applying the concept of synthetic datasets, in this case using SynthpopR, is seen as innovative and applicable to a wide range of investigations, importantly to those where full data and identity disclosure are not possible. The tool is likely to be utilized widely in translational and clinical research.

Decision letter after peer review:

Thank you for submitting your article "Synthetic datasets: A primer for the biobehavioural sciences to promote reproducibility and hypothesis-generation" for consideration by eLife. Your article has been reviewed by Christian Büchel as the Senior Editor, Mone Zaidi as the Reviewing Editor, and two reviewers. The following individual involved in review of your submission has agreed to reveal their identity: Dorothy VM Bishop (Reviewer #1).

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

Summary:

Creation of synthetic datasets has been adopted in some areas of social science where large survey datasets are used, and where full anonymisation is difficult or impossible. Here, the author demonstrates the usefulness of the synthpop R package for generating synthetic data in a different field, where datasets are typically smaller and simpler.

While the paper does not add new empirical observations or analysis approaches, it is, as its title indicates, a primer for others who might wish to try this approach. It highlights the benefits of synthetic datasets as compared to open research data. Importantly, the use of synthetic datasets shows reliable general utility in terms of data reproducibility of original findings. As long as the risk of identity disclosure is eliminated, synthetic datasets can and should be employed given the reproducibility of scientific data in biomedical sciences is a growing issue. The inclusion of oxytocin data re-analyzed in the synthetic datasets is a strength. The data are strong and well presented. The manuscript is clearly written for the most part and delivers a straightforward message that would be useful to a general readership. Nonetheless several areas need improvement and further analysis.

Essential revisions:

1) It would be preferable to have more diversity in the illustrative datasets – perhaps even demonstrating a case when synthpop does not perform as well. In particular, questions were raised regarding data with unusual distributions, evident outliers, or a hierarchical structure, for example, those with a code for family, and then family members.

2) There is mention in the Discussion section about cases where the synthetic dataset has poor utility. Perhaps, this cannot be predicted in advance, but it would be interesting to test this with datasets that may have the kinds of characteristics that are noted in point 1 (above). Indeed, this would all get rather meta, but one could simulate data that has specific features, such as striking outliers or non-normal distributions, to determine how far synthpop could retain its utility in such situations. Maybe this would be a better use of the second demonstration (see point 3). Would it be possible to take the original dataset and throw in some outliers or transform distributions to push synthpop to determine how well it stands up to non-normal data and the likes.

3) The three illustrative cases seem repetitive. Examples 1 and 3 show how the package worked with relatively large and small datasets, respectively, but example 2 seemed unnecessary. It is suggested that this part of an online appendix or is utilized it as noted in point 2 (above).

4) There is the issue about discarding data points that happen to replicate a real individual by chance. It is mentioned in the Introduction, again in Results section and in the Discussion section. A reviewer isn't sure about the logic of removing these. How would the individual – or anyone else – know this was their data? There is a problem with de-identification in a real dataset. For example, one knows that someone is a 55-year-old, left-handed woman, and there is only one of those in the sample. But if one knows that a dataset is synthesized, one could not be confident that the remainder of the data in that row came from this person.

5) It would be critical to model missing data. This might have an impact on fidelity of the synthetic data.

Essential revisions:

1) It would be preferable to have more diversity in the illustrative datasets – perhaps even demonstrating a case when synthpop does not perform as well. In particular, questions were raised regarding data with unusual distributions, evident outliers, or a hierarchical structure, for example, those with a code for family, and then family members.

To increase diversity of the illustrative datasets, I have now removed one of the datasets (example 2) and included a series of simulated datasets instead (subsection “Example 2: Sociosexual orientation and self-rated attractiveness”). There are several approaches for how to simulate data to demonstrate synthpop’s robustness (or where it performs suboptimally). To best represent common characteristics of unusual datasets, I have simulated datasets that differ in sample size, skew, and the percentage of missing data. Outliers are also included among the simulated datasets.

These simulated datasets are modelled off common cardiovascular and demographic variables (heart rate variability, heart rate, fitness level, weight). In total, 3 groups (n = 40, n = 100, n = 10,000) of datasets have been simulated, and each group contains 9 datasets with the following characteristics:

Normal distribution and no missing data

Normal distribution and 5% missing data

Normal distribution and 20% missing data

Low skew and no missing data

Low skew and 5% missing data

Low skew and no missing data

High skew and no missing data

High skew and 5% missing data

High skew and no missing data

Thus, a total of 27 datasets are presented in the new illustrative example.

2) There is mention in the Discussion section about cases where the synthetic dataset has poor utility. Perhaps, this cannot be predicted in advance, but it would be interesting to test this with datasets that may have the kinds of characteristics that are noted in point 1 (above). Indeed, this would all get rather meta, but one could simulate data that has specific features, such as striking outliers or non-normal distributions, to determine how far synthpop could retain its utility in such situations. Maybe this would be a better use of the second demonstration (see point 3). Would it be possible to take the original dataset and throw in some outliers or transform distributions to push synthpop to determine how well it stands up to non-normal data and the likes.

As mentioned above I have created a new illustrative example with 27 simulated datasets. Instead of transforming the original dataset, I created a set of simulated datasets from scratch for more fine-grained control of the parameters. For each dataset, I present general and specific utility diagnostics. These simulations demonstrate that synthpop is remarkably robust in most cases to non-normal data and missing data, at least for these examples. However, the utility of simulated datasets is suboptimal for some of the samples. For example, the model generated for the synthetic version of the dataset with 100 cases, HRV data with a low skew, and 5% missing data overall only had a 60.5% confidence interval overlap with the model generated from the observed data. While the standardized difference between these two estimates was not significantly different, one should not place too much confidence in the synthetic model given the relatively modest confidence interval overlap, as this is indicative of weak specific utility. As suggested by the reviewer, it is difficult to predict which datasets will perform poorly a priori based on the dataset characteristics alone. Thus, it is crucial that each generated synthetic dataset is checked for general and specific utility.

3) The three illustrative cases seem repetitive. Examples 1 and 3 show how the package worked with relatively large and small datasets, respectively, but example 2 seemed unnecessary. It is suggested that this part of an online appendix or is utilized it as noted in point 2 (above).

As mentioned above, this example has been replaced by a set of simulated datasets.

4) There is the issue about discarding data points that happen to replicate a real individual by chance. It is mentioned in the Introduction, again in Results section and in the Discussion section. A reviewer isn't sure about the logic of removing these. How would the individual – or anyone else – know this was their data? There is a problem with de-identification in a real dataset. For example, one knows that someone is a 55-year-old, left-handed woman, and there is only one of those in the sample. But if one knows that a dataset is synthesized, one could not be confident that the remainder of the data in that row came from this person.

To clarify, my intention was to highlight the potential issue of the replication of cases in which all variables (rather than some variables) happen to replicate a real case. I have now made this clearer in the manuscript, as follows:

In the Introduction:

“Synthetic datasets also reduce disclosure risk to essentially zero, as no complete casewise record in the new dataset represents a real individual (Duncan et al., 2011).”

And in the Discussion section:

“Therefore, to reduce disclosure risk and the possibility that participants will recognise themselves in the dataset, complete matching records (i.e., when all variables in the original dataset for a case matches a case in the synthetic dataset) should be identified and removed”

5) It would be critical to model missing data. This might have an impact on fidelity of the synthetic data.

As mentioned above, I have now modelled the effects of missing data on the creation of synthetic datasets. The degree of missingness (none, 5%, or 20%) seems to make little systematic difference in terms of general or specific utility.

Author details

1. Daniel S Quintana

   Norwegian Centre for Mental Disorders Research (NORMENT), Division of Mental Health and Addiction, University of Oslo, and Oslo University Hospital, Oslo, Norway

   Contribution

   Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Visualization, Methodology

   For correspondence

   daniel.quintana@medisin.uio.no

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2876-0004

• 3,873

  Page views

• 324

  Downloads

• 42

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, Scopus, PubMed Central.