Results pertaining to Question 1 are shown in Figure 1A. Comparing the sexes, either statistically or by assertion, was common, occurring in 80% of the articles. A positive finding of a sex difference was reported in 83 articles, or 57%. Of the articles reporting a sex difference, 41 (49% of the 83 articles) mentioned that result in the title or the abstract. Thus, in our sample of articles in which data were reported by sex, a sex difference was reported in more than half of the articles and in half of those, the difference was treated as a major finding by highlighting it in the title or abstract. In 44% of articles, a sex difference was neither stated nor implied.

Figure 1 with 1 supplement see all

Download asset Open asset

Figure 1—source data 1

Data depicted in Figure 1.

All raw data are in Supplementary file 1b and are tabulated in Supplementary file 1c.

https://cdn.elifesciences.org/articles/70817/elife-70817-fig1-data1-v1.xlsx

Download elife-70817-fig1-data1-v1.xlsx

These results are broken down by discipline in Figure 1B. The sexes were most commonly compared in the field of Endocrinology (93%) and least often in the field of Neuroscience (33%). In the field of Reproduction, the sexes were compared 89% of the time and in 100% of those cases, a sex difference was mentioned in the title or abstract. Sex differences were least likely to be emphasized in the title or abstract in the fields of General Biology and Neuroscience (11% each).

Although a sex difference was claimed in a majority of articles (57%), not all of these differences were supported with statistical evidence. In more than a quarter of the articles reporting a sex difference, or 24/83 articles, the sexes were never actually compared statistically. In these cases, the authors claimed that the sexes responded differentially to a treatment when the effect of treatment was not statistically compared across sex. This issue is explored in more detail under Question 2, below. Finally, we noted at least five articles in which the authors claimed that there was no sex difference, but did not appear to have tested statistically for one.

For each article, we asked whether it contained a study with a factorial design in which sex was one of the factors. This design is common when researchers are interested in testing whether the sexes respond differently to a manipulation such as a drug treatment (Figure 2A). Below, we use the term ‘treatment’ to refer to any non-sex factor in a factorial design. Such factors were not limited to treatment, however; they also included variables such as genotype, season, age, exposure to stimuli, etc. Hypothetical results of a study with such a design are shown in Figure 2B. In order to draw a conclusion about whether responses to treatment differed between females and males, the effect of the treatment must be compared across sex. Although there are several ways of making such a comparison (see Cumming, 2012; Gelman and Stern, 2006), it is typically done by testing for an interaction between sex and treatment. If the interaction is significant, then a claim can be made that the sexes responded differently to the treatment. Comparing the treated and control groups within each sex, in other words disaggregating the data by sex and testing for effects of treatment separately in females and males, does not test whether the sexes responded differently; that is, it does not test whether the magnitude of the response differs between females and males (Gelman and Stern, 2006; Makin and Orban de Xivry, 2019; Maney, 2016; Nieuwenhuis et al., 2011; Radke et al., 2021).

Figure 2

Download asset Open asset

Figure 2—source data 1

Data depicted in Figure 2.

All raw data are in Supplementary file 1b and are tabulated in Supplementary file 1c.

https://cdn.elifesciences.org/articles/70817/elife-70817-fig2-data1-v1.xlsx

Download elife-70817-fig2-data1-v1.xlsx

The results pertaining to Question 2 are shown in Figure 2C-F. Out of the 147 articles we analyzed, 92 (63%) contained at least one study with a factorial design in which sex was a factor (Figure 2C). Regardless of whether a sex difference was claimed, we found that the authors explicitly tested for interactions between sex and other factors in only 27 of the 92 articles (29%). That is, authors tested statistically for a sex difference in the responses to other factor(s) less than one-third of the time. Testing for interactions with sex varied by discipline (Figure 2D). Authors were most likely to test for and report the results of interactions in the field of Behavioral Physiology (54% of relevant articles) and least likely in the fields of Physiology (0%) and Reproduction (0%).

Of the studies with a factorial design, 58% reported that the sexes responded differently to one or more other factors. The language used to state these conclusions often included the phrase ‘sex difference’ but could also include ‘sex-specific effect’ or that a treatment had an effect ‘in males but not females’ or vice versa. Of the 53 articles containing such conclusions, the authors presented statistics showing a significant interaction, in other words appropriate evidence that females and males responded differently, in only 16 (30%; Figure 2E, blue color in first column). In an additional article, the authors presented statistical evidence that the interaction was non-significant, yet claimed a sex-specific effect nonetheless. In five other articles, the authors mentioned testing for interactions but presented no results or statistics (e.g., p values) for those interactions. In the remainder of articles containing claims of sex-specific effects, the authors took one of two approaches; neither approach included testing for interactions. Instead, authors proceeded to what would normally be the post hoc tests conducted after finding a significant interaction. In 24 articles (45% of articles with claims of sex-specific effects), authors reported the effect of treatment within each sex and, reaching different conclusions for each sex (e.g., finding a p value below 0.05 in one sex but not the other), inappropriately argued that the response to treatment differed between females and males (see Figure 2B). In seven other articles claiming a sex-specific effect (13%), the sexes were compared within treatment; for example, authors compared the treated males with the treated females, not considering the control animals. Neither approach tests whether the treatment had different effects in females and males. Thus, a substantial majority of articles containing claims of sex-specific effects (70%) did not present statistical evidence to support those claims (Figure 2E, red color in first column); further, in the majority of articles without such evidence (24/37), the sexes were never compared statistically at all.

The omission of tests for interactions was related to whether researchers were claiming sex differences or not. Among the articles that were missing tests for interactions and yet contained conclusions about the presence or absence of sex-specific effects (41 articles), those claims were in favor of sex differences 88% of the time, compared with only 12% claiming that the responses in females and males were similar. Of all of the articles claiming similar responses to treatment, authors tested for interactions in the majority of cases (67%; Figure 2E blue color in second column).

The prevalence of reporting sex-specific effects is broken down by discipline in Figure 2F. The field with the lowest percentage of sex-specific effects was Behavior (18%), and that field also had the highest rate of backing up such claims with statistical evidence (71%). The field most likely to contain claims of sex-specific effects was Reproduction (67%), but this field was among three for which such claims were never backed up with statistical evidence (0% for Reproduction, Physiology, or Pharmacology).

In this study we included only articles in which data were reported by sex as previously determined by Woitowich et al., 2020. Thus, any articles in which the sexes were pooled for all analyses were not included here. We assigned each of the 147 articles to one of three categories, as follows (Figure 3A). In 31 (21%) of the articles, data from males and females were analyzed separately throughout. In 62 (42%) of the articles, males and females were analyzed in the same statistical models, but in those cases sex was included as a fixed factor or a covariate. In most cases when sex was a covariate, authors reported the results of the effect of sex rather than simply controlling for sex. In the remaining 54 (37%) articles, the sexes were pooled for at least some of the analyses.

Figure 3

Download asset Open asset

Figure 3—source data 1

Data depicted in Figure 3.

All raw data are in Supplementary file 1b and are tabulated in Supplementary file 1c.

https://cdn.elifesciences.org/articles/70817/elife-70817-fig3-data1-v1.xlsx

Download elife-70817-fig3-data1-v1.xlsx

Among the articles in which the sexes were pooled, the authors did so without testing for a sex difference almost half of the time (48%; Figure 3B). When authors did test for a sex difference before pooling, they sometimes found a significant difference yet pooled the sexes anyway; this occurred in 17% of the articles that pooled. When the sexes were pooled after finding no significant difference (35% of the articles that pooled), authors presented p values for the sex difference the majority of the time (11 out of 19 articles). Those p values ranged from 0.15 to >0.999. We noted no effect sizes reported in the context of pooling.

Across disciplines, pooling was most prevalent in Immunology (60%) and least prevalent in General Biology (22%). Males and females were most likely to be kept separate in General Biology (56%) and most likely to be included in statistical models in the field of Behavior (53%). When females and males were pooled, authors in the field of Immunology were least likely to have tested for a sex difference before pooling (33%) and most likely to do so in Pharmacology (80%). Pooling after finding a significant difference was most common in the field of Reproduction (67% of articles that pooled).

To refer to the categorical variable comprising male/female or man/woman (all were binary), the term ‘sex’ was used exclusively in 69% of the articles (Figure 4). ‘Gender’ was used exclusively in 9%, and both ‘sex’ and ‘gender’ were used in 19%. When both terms were used, they usually seemed to be used interchangeably. In 4% of the articles, neither term was used.

Figure 4

Download asset Open asset

Figure 4—source data 1

Data depicted in Figure 4.

All raw data are in Supplementary file 1b and are tabulated in Supplementary file 1c.

https://cdn.elifesciences.org/articles/70817/elife-70817-fig4-data1-v1.xlsx

Download elife-70817-fig4-data1-v1.xlsx

Of the articles in which the term ‘gender’ was used, 20% of the time it referred to non-human animals, such as mice, rats, and pigs. In one case, both ‘sex’ and ‘gender’ were used to refer to non-human animals in the title. In another case, ‘gender’ was used to refer to human cells. The majority of articles on non-human species used ‘sex’ (85%).

Woitowich et al., 2020, found that over the past decade, the proportion of biological studies that included both females and males has increased, but the proportion in which sex is treated as a variable has not. Here, we have taken a closer look at the studies determined by those authors to have reported data by sex, that is, to have conformed to NIH guidelines on SABV. We found that in this subset of studies, authors typically also compared the sexes either statistically or by assertion (>80% of cases). Thus, the authors that complied with NIH guidelines to disaggregate data usually went beyond NIH guidelines to explicitly compare the sexes with each other. This finding is consistent with a larger analysis of articles in the field of Neuroscience from 2010 to 2014; when authors disaggregated data by sex, they usually proceeded to compare the sexes as well (Will et al., 2017). It is important to note, however, that both Will et al., 2017, and Woitowich et al., 2020, found that data were not analyzed by sex in the majority of articles that included both sexes (see Figure 1—figure supplement 1). Thus, our current finding that the sexes were usually compared should be interpreted in the context of the subset of articles following NIH guidelines. In the set of articles analyzed here, sex differences were claimed in a majority and were often highlighted in the title or abstract. We therefore found little evidence that researchers—at least those who comply with NIH guidelines—are uninterested in sex differences. We cannot rule out the possibility, however, that the researchers following NIH guidelines are primarily those that are interested in sex differences.

Testing whether the sexes respond differently to a treatment requires statistical comparison between the two effects, which is typically done by testing for a sex × treatment interaction. In our analysis, however, tests for interactions were done only 29% of the time (Figure 2C and D). In the remaining 71%, the most common method for detecting differential effects of treatment was to compare qualitatively the conclusions drawn for each sex; that is, to assert that a p value below 0.05 for one sex but not the other (Figure 2B) represents a meaningful difference between the effects. But null hypothesis significance testing does not allow for such conclusions (Cumming, 2012). This error, and the frequency with which it is made, has been covered in multiple publications; for example Gelman and Stern, 2006, titled their commentary “The difference between ‘significant’ and ‘not significant’ is not itself statistically significant.” Makin and Orban de Xivry, 2019, included the error in their ‘Top ten list of common statistical mistakes’. In an analysis of 520 articles in the field of Neuroscience, Nieuwenhuis et al., 2011, found that the error was committed in about half of articles containing a factorial design. The current analysis showed that, even a decade later, the frequency of this error in the field of Neuroscience has not changed (Figure 2D), at least when sex is one of the factors under consideration. The frequency of the error was high in most of the other disciplines as well, particularly Physiology and Reproduction, for which we found that authors never tested for interactions even though doing so was necessary to test their hypotheses about sex.

Statements such as the following, usually made without statistical evidence, were common: ‘The treatment increased expression of gene X in a sex-dependent manner’; ‘Our results demonstrate that deletion of gene X produces a male-specific increase in the behavior’; ‘Our findings indicate that females are more sensitive to the drug than males’. In some of these cases, the terms ‘sex-specific’, ‘sex-dependent’, or ‘sexual dimorphism’ were used in the title of the article despite a lack of statistical evidence supporting the claim. In many of these articles, some of which stated that finding a sex difference was the major goal of the study, the sexes were not statistically compared at all. Thus, a lack of statistical evidence for sex-specific effects did not prevent authors from asserting such effects. Authors failing to test for interactions were far more likely to claim sex-specific effects than not (88% vs. 12%; Supplementary file 1c); they were also more likely to do so than were authors that did test for interactions (88% vs. 63%; Supplementary file 1c). Statistical analysis of these data showed that, in fact, sex-specific effects were reported significantly more often when no tests for interactions were reported (χ² = 5.84; p = 0.016). Together, these results suggest a bias toward finding sex differences. In the absence of evidence, differences were claimed more often than not. A bias toward finding sex differences, where there are none, could artificially inflate the importance of sex in the reporting of biological data. Given that findings of sex × treatment interactions are rare in the human clinical literature, with false positives outnumbering false negatives (Wallach et al., 2016), and given also that sex differences are often reported in the media and used to shape education and health policy (Maney, 2014), it is especially important to base conclusions from preclinical research on solid statistical evidence.

The set of articles we analyzed was pre-screened by Woitowich et al., 2020, to include only studies in which sex was considered as a variable. Nonetheless, even in this sample, data were often pooled across sex for some of the analyses (Figure 3A). In a majority of these articles, authors did not test for a sex difference before pooling (Figure 3B). Thus, for at least some analyses represented here, the data were not disaggregated by sex, sex was not a factor in those analyses, and we do not know whether there might have been a sex difference. Even when authors did test for a sex difference before pooling, the relevant statistics were often not presented. Finding and reporting a significant sex difference did not seem to reduce the likelihood that the sexes would be pooled. Note that the original sample of 720 articles in the study by Woitowich et al. included 251 articles in which sex was either not specified or the sexes were pooled for all analyses (Figure 1—figure supplement 1). Thus, the issue is more widespread than is represented in the current study. Pooling is not consistent with the NIH mandate to disaggregate data by sex and can prevent detection of meaningful differences. We note further that effect sizes were generally not reported before pooling; in addition to p values, effect sizes would be valuable for any assessment of whether data from males and females can be pooled without masking a potentially important difference (Beltz et al., 2019; Diester et al., 2019).

In their article on ‘Ten statistical mistakes…,’ Makin and Orban de Xivry, 2019, list another issue that is likely to be relevant to the study of sex differences: comparing multiple dependent variables across sex without correcting for multiple comparisons. The omission of such a correction increases the risk of false positives, that is, making a type I error, which would result in over-reporting of significant effects. This risk is particularly important for researchers trying to comply with SABV, who may feel compelled to test for sex differences in every measured variable. In the current study, we found this issue to be prevalent. For example, we noted articles in which researchers measured expression of multiple genes in multiple tissues at multiple time points, resulting in a large number of comparisons across sex. In one such study, authors made 90 separate comparisons in the same set of animals and found five significant differences, which is exactly the number one would expect to find by chance. Although opinions vary about when corrections are necessary, omitting them when they are clearly needed is likely contributing to over-reporting of sex differences broadly across disciplines.

We found that a large majority of studies on non-human animals used ‘sex’ to refer to the categorical variable comprising females and males. In eight articles, we noted usage of the word ‘gender’ for non-human animals. This usage appears to conflict with current recommendations regarding usage of ‘gender’, that is, gender should refer to socially constructed identities or behaviors rather than biological attributes (Clayton and Tannenbaum, 2016; Holmes and Monks, 2019; Woitowich and Woodruff, 2019). We did not, however, investigate the authors’ intended meaning of either term. Although definitions of ‘gender’ vary, the term might be appropriate for non-human animals under certain circumstances, such as when the influence of social interactions is a main point of interest (Cortes et al., 2019). Operational definitions, even for the term ‘sex’, are important and, in our experience conducting this study, almost never included in publications. As others have done (e.g., Duchesne et al., 2020; Cortes et al., 2019; Holmes and Monks, 2019; Johnson et al., 2009), we emphasize the importance of clear operational definitions while recognizing the limitations of binary categories.

The categorization of each article into a particular discipline was defined exclusively by the journal in which it appeared, in order to be consistent with the original categorizations of Beery and Zucker, 2011, and Woitowich et al., 2020. For most disciplines, fewer than a dozen articles were in our starting sample; for Neuroscience and Reproduction, only nine. As a result, after we coded the articles, some categories contained few or no articles in a given discipline (see Supplementary file 1c). The within-discipline analyses, particularly the pie charts in Figure 3B, should therefore be interpreted with caution. Firm conclusions about whether a particular practice is more prevalent in one discipline than another cannot be drawn from the data presented here.

As is the case for any analysis, qualitative or otherwise, our coding was based on our interpretation of the data presentation and wording in the articles. Details of the statistical approach were sometimes left out, leaving the author’s intentions ambiguous. Although our approach was as systematic as possible, a small number of articles may have been coded in a way that did not completely capture those intentions. We believe our sample size, particularly in the overall analyses across disciplines, was sufficient to reveal the important trends.

SABV has been hailed as a game-changing policy that is already bringing previously ignored sex-specific factors to light, particularly for females. In this study, we have shown that a substantial proportion of claimed sex differences, particularly sex-specific effects of experimental manipulations, are not supported by sufficient statistical evidence. Although only a minority of studies that include both sexes actually report data by sex (Woitowich et al., 2020), our findings suggest that when data are reported by sex, critical statistical analyses are often missing and the findings likely to be interpreted in misleading ways. Note that in most cases, our findings do not indicate that the conclusions were inaccurate; they may have been supported by appropriate statistical analyses. Our results emphasize the need for resources and training, particularly those relevant to the study designs and analyses that are commonly used to detect sex differences. Such training would benefit not only the researchers doing the work, but also the peer reviewers, journal editors, and program officers who have the power to hold researchers to a higher standard. Without better awareness of what can and cannot be concluded from separate analysis of males and females, SABV may have the undesired effect of reducing, rather than enhancing, rigor and reproducibility.

Because we were interested in the frequency with which sex differences were found, we first identified articles in which the sexes were explicitly compared. We counted as a comparison any of the following: (1) sex was a fixed factor in a statistical model; (2) sex was included as a covariate in a statistical model and a p value for the effect of sex was reported; (3) a p value for a comparison of means between males and females was presented; (4) the article contained wording suggestive of a comparison, for example, ‘males were larger than females’. We also included articles with wording suggestive of a sex difference in response to a treatment, for example, ‘the treatment affected males but not females’ or ‘the males responded to treatment, whereas the females did not’, or ‘the treatment had a sex-specific effect’. Similarly, we included here articles with language referring to a non-difference, for example, ‘we detected no sex differences in size’ or ‘the response to treatment was similar in males and females’. Articles in which sex was included as a covariate for the purposes of controlling for sex, rather than comparing the sexes, were not coded as having compared the sexes (see Beltz et al., 2019). When the sexes were compared but no results of those comparisons, for example, p values, were reported, that omission was noted and the article was coded accordingly. Each article in which the sexes were compared was then further coded as either reporting a sex difference or not, and if so, whether a sex difference was mentioned in the title or abstract. If mentioned in the title or abstract, the sex difference was coded as a ‘major finding’; otherwise, sex differences mentioned in the body of the paper, figures, or tables were coded as ‘minor’.

We looked for studies with a 2 × 2 factorial design (Figure 2A) in which sex was one of the factors. Sex did not need to be explicitly identified as a fixed factor; we included here all studies comparing across levels of one factor that comprised females and males with each of those levels. In some cases that factor was a manipulation, such as a drug treatment or a gene knockout. Non-sex factors also included variables such as age, season, presentation of a stimulus, etc. For simplicity, we refer to the other factor as ‘treatment’. Any article containing at least one such study was coded as having a factorial design. The other articles were coded as containing no comparisons across sex or as containing only group comparisons across sex. The latter category included studies with sex as a covariate of interest in a model such as a multiple regression, if the authors were not making any claims about potential interactions between sex and other variables.

For studies with a factorial design, we further coded the authors’ strategy of data analysis. First, we noted whether authors tested for an interaction between sex and treatment; that is, they tested whether the effect of treatment depended on sex. We coded as 'yes' one study in which the magnitude of the differences between treated and control groups was explicitly compared across sex. For articles containing tests for interactions, we noted the outcome of that test and the interpretation. Articles containing no tests for interactions were assigned to one of several sub-categories in the following order (coded as the first category on this list for which the description was met for any analysis in the article): tested for effects of treatment within sex, tested for effects of sex within at least one level of treatment, or tested for main effects of sex only. Within each of those categories we further coded the outcome/interpretation, for example, sex difference or no sex difference. Any articles containing statements that the sexes responded differently to treatment or that the response was ‘sex-specific’ were coded as reporting a sex-specific effect. We also noted when authors reported an absence of such a result. Articles not comparing across sex at all, with statistical evidence or by assertion, were coded accordingly.

We assigned articles to one of three categories: analyzed males and females separately throughout, included sex in the statistical model for at least some analyses (with the rest analyzed separately), or pooled for at least some analyses. The second category, included sex in the model, included articles in which AIC or similar statistic was used to choose among models that included sex, although sex may not have been in the model ultimately chosen. This category did not distinguish between analyses including sex as a fixed factor vs. a covariate; this distinction is noted where relevant in Supplementary file 1b. Any article containing pooled data was coded as pooled, even if some analyses were conducted separately or with sex in the model. For articles that pooled, we further noted whether the authors tested for a sex difference before pooling and, if so, whether p values or effect sizes were reported.

We searched the articles for the terms ‘sex’ and ‘gender’ and noted whether the authors used one or the other, both, or neither. Terms such as ‘sex hormones’ or ‘gender role’, which did not refer to sex/gender variables in the study, were excluded from this assessment. For the articles using ‘gender’, we further noted when the term was used for non-human animals.

To visualize the data, we used river plots (Weiner, 2017), stacked bar graphs, and pie charts based on formulae and data presented in Supplementary file 1c.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

We thank Nicole Baran, Isabel Fraccaroli, and Naomi Green for assistance and suggestions in the initial stages of this project, and Chris Goode for assistance with the river plots. We are grateful to Lise Eliot, Chris Goode, Niki Woitowich, Colby Vorland, Chanaka Kahathuduwa, and an anonymous reviewer for providing comments on the manuscript.

© 2021, Garcia-Sifuentes and Maney

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.