The number of PhDs awarded in science, technology, engineering and mathematics (STEM) has increased dramatically over the past three decades (Cyranoski et al., 2011; Ghaffarzadegan et al., 2015), but the number of faculty positions available has essentially remained constant (Schillebeeckx et al., 2013). In the US, for instance, the situation has not changed significantly since 2003, when the National Institutes of Health (NIH) received a major budget increase (Alberts et al., 2014). Given the low numbers of faculty positions compared to the numbers of PhDs produced (Larson et al., 2014; Committee to Review the State of Postdoctoral Experience in Scientists and Engineers, 2014), trainees are limited in their job prospects. Many also emerge from academic training feeling underprepared and under-mentored for any other type of job search (McDowell et al., 2015). This leads to a high number of applicants per academic position, many of whom are uncertain about their chances of obtaining a faculty job (Grinstein and Treister, 2018; Sauermann and Roach, 2016).

Cohorts of new PhDs are also both more diverse than before and more diverse than many current hiring committees (Alberts et al., 2014; White, 2019; Bhalla, 2019). Scientific publishing is also faster-paced than it used to be: for example, evolutionary biologists recruited as "junior researchers" in 2013 had published nearly twice as many articles (22 ± 3.4) as those hired in 2005 (12.5 ± 2.4); the same study also found that the length of time between first publication and recruitment as a faculty member had increased from 3.25 (±0.6) to 8.0 (±1.7) years (Brischoux and Angelier, 2015). Longer training periods have been reported repeatedly in many STEM fields, and are perceived as detrimental to both the greater scientific community and individuals in temporary postdoctoral positions (Committee to Review the State of Postdoctoral Experience in Scientists and Engineers, 2014; Ahmed, 2019; Rockey, 2012; Acton et al., 2019).

Despite these changes, the academic job search has largely remained the same, resulting in academic hiring being perceived as an opaque process with no clear standards or guidelines. Beyond a requirement for a doctoral degree and possibly postdoctoral training, faculty job advertisements rarely contain specific preferred qualifications. Furthermore, the criteria used to evaluate applicants are typically determined by a small departmental or institutional committee and are neither transparent nor made public. The amount of materials required for faculty job applications is also highly variable among hiring institutions, and often places a heavy burden on both applicants and search committees (Lee, 2014).

Previous studies agree on a need to increase transparency in career outcomes and hiring practices (Golde, 2019; Polka et al., 2015; Wright and Vanderford, 2017). The annual pool of faculty job applicants is large and provides a unique opportunity for examining the application process. We performed an anonymous survey, asking applicants for both common components of research and scholarly activity found on an academic CV, as well as information on their success through the 2018–2019 job cycle. We further performed a small-scale, complementary survey of search committee members. Here we present qualitative and quantitative data on the academic job market, including information on the number of successful off-site and on-site interviews, offers, rejections, and the lack of feedback.

Job applicants start by searching for relevant job postings on a variety of platforms (Supplementary file 1). The initial electronic application generally consists of a cover letter addressing the search committee, a teaching philosophy statement, CV, and a research plan (Figure 1). The length and content of these materials can vary drastically based on the application cycle, region, institution, or particular search committee. In the current system, the expectation is that application materials be tailored for each specific institution and/or department to which the applicant is applying. This includes department-specific cover letters (Fox, 2018a), but may also involve a range of changes to the research, teaching, and diversity statements.

Figure 1

Download asset Open asset

The search committee convenes for a few meetings to shortlist the applicants. Applicants are then contacted for interviews somewhere between one to six months after application materials are due. Searches may include an initial off-site (remote) interview, followed by an on-site interview at the hiring university. The on-site interview typically lasts one or two days and consists of a research seminar, possibly a teaching demonstration, and likely a chalk-talk (Rowland, 2016). The on-site interview also usually consists of one-on-one meetings with other faculty members, including a meeting with the hiring department chair, trainees, and the administrative staff.

After the interviews, candidates may be contacted and offered a position, usually in writing. The offer package will include the proposed start date, salary and start-up funds (Macdonald, 2019). The time to offer is also variable, but is usually shorter than the time between application and first contact (based on anecdotal information). Importantly, a single search can result in multiple offers (for instance the department may be able to fund multiple competitive candidates, or the first-choice candidate may decline and the second candidate is given an offer). Searches can also fail if the committee does not find a suitable candidate for their program/department or "go dry" if the applicant(s) deemed qualified by the search committee decline their offer.

We designed a survey for early-career researchers aimed at bringing transparency to the academic job market (see Materials and methods and Supplementary file 41). The survey was distributed via Twitter, the Future PI Slack group, and email listservs of multiple postdoctoral associations, resulting in 322 responses from self-identified early-career researchers who applied for academic positions in the 2018–2019 application cycle. Of these, data from 317 respondents passed simple quality filters and were used for analyses. As all questions were optional, these 317 responses represent the maximum number in our analyses; in cases where respondents chose not to answer the question, we analyzed only the applicant subset with responses and list the number of responses used for each analysis in the appropriate figures and supplementary files.

Demographics of respondents

Respondents reported a large range in the number of submitted applications from a minimum of one to a maximum of 250 (median: 15). The respondent pool was notably enriched in applicants who received at least one off-site interview (70%), at least one on-site interview (78%) and at least one offer (58%); this may represent a significant bias towards successful applicants in our study, as a recent study shows that less than 23% of PhDs eventually secure a tenure-track position (Langin, 2019).

Respondents represented researchers in a wide variety of fields, with 85% from life sciences and related fields, with relatively equal numbers of applications from men and women across this group (Figure 2A). Our survey captured data from an international applicant pool, representing 13 countries (Figure 2B). However, 72% of our respondents reported currently working in the United States, which may reflect the larger circulation of our survey on social media platforms and postdoctoral associations there. Most candidates applied to jobs within the United States (82%), Canada (33%), and the United Kingdom (24%). 96% of respondents entered the job market as postdoctoral researchers (Figure 2C). The applicants spent 1 to 13 years (median: 4 years) in a postdoctoral position. These data are consistent with a recent report suggesting that postdocs in the United States across a variety of fields spend an average of 2.5–3.6 years in their positions (Andalib et al., 2018).

Figure 2

Download asset Open asset

Notably, in our survey population, postdocs in the life sciences spent a median of 5 years in a postdoctoral position, significantly longer than those in other fields, who reported a median postdoc length of 2.75 years prior to applying for a faculty position (Figure 2D), consistent with previous findings on increased training times in the life/biomedical sciences before junior faculty recruitment (Committee to Review the State of Postdoctoral Experience in Scientists and Engineers, 2014; Brischoux and Angelier, 2015; Ahmed, 2019; Powell, 2017; Rockey, 2012). 68% of respondents went on the job market while in their first postdoctoral position (Figure 2E).

Applicants had a large range in their publication records, including number of papers co-authored, h-index, and total citation count. Respondents reported a median of 13 total publications (including co-authorships and lead authorships), with a median of 6 first author papers when entering the job market (Figure 2F).

Publishing metrics by gender

Gender bias in publishing and evaluation is well documented (Aileen Day and Boyle, 2019; Centra and Gaubatz, 2000; Cameron et al., 2016; Witteman et al., 2019). The respondents to our survey were relatively evenly distributed across self-identified genders, with 51% identifying as male, 48% as female, and 1% preferring not to disclose this information (no applicants identified as non-binary; Figure 3A). Men reported significantly more first-author publications, total publications, overall citations, and a higher h-index compared to women (Figure 3B); more men also reported being authors on papers in three journals with high impact factors (Cell, Nature and Science; Figure 3C) than women. The gender differences we observe mirror those seen in other reports on differences in citation counts in STEM fields based on the corresponding author gender (Schiermeier, 2019). Despite popular discussions on a need for papers in Cell, Nature, Science or other journals with a high impact factor (Brock, 2019; McKiernan et al., 2019), 74% of respondents were not authors on a paper in Cell, Nature or Science (CNS), and a greater majority (~84%) did not have a first author publication in these journals (Figure 3C). Of the 51 respondents with papers in these journals, 49 (96%) were in a life science-related field, indicating that the valuation of these journals was highly field-specific (Figure 3C).

Figure 3

Download asset Open asset

While 78% of respondents reported having obtained fellowships at some point in their career, this figure was 87% for women and 72% for men (Figure 3D). Women had better success at receiving both doctoral and postdoctoral fellowships. However, the questions in our survey did not distinguish between the types (e.g. government funded versus privately funded, full versus partial salary support) or number of fellowships applied to; many of these factors are likely critical in better understanding gender differences in fellowship support (Figure 3D).

Applications, interviews and offers

The 317 respondents submitted a total of 7644 job applications in the 2018–2019 application cycle, with a median of 15 applications per respondent (Figure 4A). Applicants were invited for a total of 805 off-site interviews (phone, Zoom or Skype; median: 1) and 832 onsite or campus interviews (median: 2), receiving 359 offers (median: 1; Figure 4A). Although many hiring processes consist of an off-site (remote) interview, we found that this was not standard since the typical applicant received more on-site than off-site interviews. In our dataset, 42% of participants received no offers, 33% received one offer, 14% received two offers, 6% received three offers, and 6% received more than three offers. Candidates who received offers typically submitted more applications than those who received no offers, indicating that some candidates may not have submitted enough applications to have a reasonable chance of getting an offer (Figure 4A,D). According to a recent poll on Twitter (which received over 700 responses), most faculty received between one and three offers when they were applying for faculty positions (Whitehead, 2019; Supplementary file 15).

Figure 4 with 1 supplement see all

Download asset Open asset

Despite the fact that successful candidates submitted more applications, the number of applications per candidate did not correlate with the number of offers, while being only weakly correlated with the number of off-site interviews (Figure 4B). Not surprisingly, the number of on-site interviews strongly correlated with the number of offers received (Figure 4C, bottom). Population medians changed slightly by gender as men submitted slightly more applications, but received slightly fewer off-site interviews. These small differences by gender were not statistically significant (Figure 4A). The median number of offers also did not vary by gender.

We split our population into two groups by application number, one group either at or below the median (<15 applications, n = 162) and the other group above the median (>15 applications, n = 155). These groups had a significant difference in success rates: respondents who submitted more than 15 applications had a significantly higher average number of off-site interviews (Figure 4D). We also asked whether respondents applied for non-faculty positions during this cycle (Supplementary file 16). 71% of applicants did not apply for other jobs and these applicants had a small, but significant increase in offer percentage (Figure 4E).

Taken together, these data seemingly indicate that increasing the number of applications submitted can lead to more interviews, as suggested by others (Jay et al., 2019), with the typical candidate submitting at least 15 applications to achieve one offer. However, the lower correlation between application number and offers (compared to application number and interviews) suggests that while higher application numbers can generate more interview opportunities, other criteria (e.g. the strength of the interview) are important in turning an interview into an offer.

Publication related metrics

The number of papers published, and the impact factors of the journals these papers were published in, can influence the chances of an early-career researcher obtaining an independent position (van Dijk et al., 2014; Powdthavee et al., 2018). As mentioned previously, it is widely believed that you need a paper in Cell, Nature or Science to secure a faculty position in the life sciences (McKiernan et al., 2019; Sheltzer and Smith, 2014; Fox, 2018b). Our data demonstrates that a CNS paper is not essential to an applicant receiving a faculty job offer.

The majority (74%) of our respondents were not an author on a CNS paper (Figure 5A), and yet most participants received at least one offer (58%). However, applicants with a CNS paper did have a higher number of onsite interviews and faculty job offer percentage. Of our respondents, 16% were first author on a CNS paper, and these applicants had a significantly higher percentage of offers per application (p=1.50×10⁻⁴, median offer percentages: 11% with a CNS paper and 2% without a CNS paper) and on-site interviews (p=2.70×10⁻⁴, median onsite interview percentages: 21% with a CNS paper, and 10% without a CNS paper; Figure 5A).

Figure 5 with 2 supplements see all

Download asset Open asset

Since the number of on-site interviews and offers are highly correlated (Figure 4C), it is unclear if this increased success simply represents a higher chance at landing more onsite interviews. It is important to note that this effect is correlative and these candidates likely had other attributes that made them appealing to the search committee(s).

We examined several other publication metrics and found no correlation with the number of offers. Specifically, the total number of publications, the number of first author publications, the number of corresponding author publications, and h-index did not significantly correlate with offer percentage (Figure 4—figure supplement 1). When we separated candidates who were above and below the medians for each of these metrics and compared the distribution of offer percentages, only the total number of citations significantly associated with a higher offer percentage (Figure 5B). Although the offer percentage was generally higher for applicants above the median for the other metrics, none of these differences were statistically significant (Figure 5B).

Levels of teaching experience

Discussions surrounding the academic job market often center on publications and/or funding, while teaching experience generally receives much less attention. However, the level of teaching experience expected from the applicants can vary, but mostly depends on the type of hiring institution.

We asked applicants whether they focused their applications to a specific type of institution (R1, PUI, or both; see Box 1 for definitions), allowing us to examine teaching experience across R1 and/or PUI applicants. Most respondents applied to jobs at R1 institutions (Figure 6A), which may explain the focus on research-centric qualifications. It remains unclear what the emphasis on teaching experience is for search committees at R1 institutions, however the literature suggests that there seems to be a minimal focus (Clement et al., 2019). Additionally, there might be differences in departmental or institutional requirements that are unknown to outsiders. What is commonly accepted is that many applications to an R1 institution require a teaching philosophy statement.

Figure 6

Download asset Open asset

Box 1.

Definition of specific terms used in this study.

Early-career researcher (ECR): For the purpose of this study, we define an ECR to be anyone engaged in research who is not recognized as an independent leader/investigator of a research group. This includes graduate and postdoctoral researchers; junior research assistants, research associates, and staff scientists.

Principal Investigator (PI): A scholar recognized as an independent leader of a research group. This includes full professors, group leaders, and tenure-track, non-tenure-track or tenured faculty.

Faculty Job Applicant: An early-career researcher with a PhD (a recent graduate, postdoctoral fellow or research scientist) who seeks to apply for a PI position (see above), usually at the assistant professorship level.

STEM Fields: STEM is an acronym for degrees in fields related to science, technology, engineering, and mathematics. STEM graduates work in a wide variety of fields including the life sciences, the physical sciences, different areas of engineering, mathematics, statistics, psychology, and computer science.

Research Mentor: A research advisor, usually the PI of a lab who mentors graduate and postdoctoral researchers during their academic training in his/her lab.

Adjunct Lecturer: A teacher or post-PhD scholar who teaches on a limited-term contract, often for one semester at a time. This individual is ineligible for tenure.

Teaching Assistant (TA): An individual who assists a course instructor with teaching-related duties in a lecture-based and/or laboratory-based undergraduate or graduate level course.

Doctoral/Graduate and Postdoctoral Fellowships: Funding mechanisms to support the training of a graduate or postdoctoral researcher: the proposal for this is written by the trainee and contains a mentoring/training plan and request for funding to support the trainee salary and/or part of their research expenses such as equipment, lab supplies and travel expenses typically for 1–3 years.

Career Transition Awards: Funding mechanisms facilitating senior trainees towards independent research careers: Includes core/substantial funds to fully support 1–3 years of postdoctoral salary and additional 2–5 years of independent faculty research and staff salaries as well as support for research expenses such as equipment, lab supplies and travel expenses. As a result, some portion of these funds can transition from the training institute to the hiring institute.

R1 University: There are 131 institutions in the United States that are classified as "R1: Doctoral Universities – very high research activity" in the Carnegie Classification of Institutions of Higher Education (2019 update), can be private or public.

R2 University: There are 135 institutions in the United States that are classified as “R2: Doctoral Universities – high research activity" in the Carnegie Classification of Institutions of Higher Education (2019 update), can be private or public.

R3 University, PUI or Small Liberal Arts College (SLAC): Primarily undergraduate institutions (PUI) are often smaller than large research universities, can be private or public, and offer varying levels of resources for students and faculty. Many faculty at PUIs run a research lab while maintaining significant teaching loads and heavy contact hours with students.

Almost all respondents (99%) had teaching experience (Figure 6B): for roughly half this experience was limited to serving as a Teaching Assistant (TA; Box 1), with the rest reporting experience beyond a TA position, such as serving as an instructor of record (Figure 6B). The degree of teaching experience did not change based on the target institution of the applicant (Figure 6C), nor did the percentage of offers received significantly differ between groups based on teaching experience (Figure 6D).

Research versus teaching-intensive institutions

To our knowledge, there is a lack of systematic evidence describing the process or expected qualifications of a PUI-focused (Box 1) job search (Ramirez, 2016). A subgroup of 25 "PUI Focused" applicants responded to our survey, and, despite this small number, we aimed to describe this important sub-group relative to "R1 Focused" applicants as well as applicants who applied to both types of institutes. The PUI subgroup included a majority of female applicants (60%, Figure 7A) while the R1 subgroup had a majority of male applicants (54%, Figure 7A). Within the PUI subgroup, no differences were seen in the number of first author publications across genders (Figure 7B), although women had a better fellowship history (Figure 7C). The median number of remote interviews, onsite interviews, and offers was also similar to that for the R1 subgroup, although the PUI subgroup submitted fewer applications (Figure 7E). Although both subgroups reported teaching experience (Figure 7D), the PUI subgroup was enriched in adjunct, visiting professor, instructor of record, community college, or contract-based teaching experiences (Figure 7F). Having adjunct experience did not significantly increase the median number of offers received for applicants focused on PUIs, R1s, or both types of institutions (Figure 7G).

Figure 7

Download asset Open asset

A time-consuming and opaque process with little feedback

We asked the applicants to comment on whether any aspect of their training or career was particularly helpful or harmful to their faculty applications (Figure 8A–B). We used word clouds (Supplementary files 27 and 28) to analyze recurrent themes in these open-ended questions. The applicants identified funding as most helpful for their applications, and no-funding as subsequently harmful; this perception agrees with the data presented above (Figure 8A, Figure 5C, Figure 4—figure supplement 1). Additionally, perceptions were also in line with the rest of the data, in that they were unable to largely agree on other measurable aspects of their career that were perceived as helpful. Qualitative aspects that were perceived as particularly helpful included networking and attending/presenting at conferences. Interestingly interdisciplinary-research, which is often highlighted as a strength and encouraged by institutions and funders, was perceived by candidates as a challenge to overcome. Indeed, interdisciplinary candidates may pose an evaluation challenge for committees, given the differences in valuation of research metrics across fields, the extended training time required to master techniques and concepts in multiple fields, as well as valuation of interdisciplinary teams of specialists over interdisciplinary individuals (Eddy, 2005).

Figure 8

Download asset Open asset

Notably, many applicants found the amount of time spent on applications and the subsequent lack of feedback from searches frustrating (Figure 8B–C). Most applicants never received any communication regarding their various submissions. For instance, an applicant who applied for 250 positions only received 30 rejections. Overall, our respondents submitted 7644 applications (Figure 4A) and did not hear anything back in 4365 cases (57% of applications), receiving 2920 formal rejection messages (38% of applications; Supplementary file 19). Application rejection messages (if received at all) most often do not include any sort of feedback. Additionally, a considerable amount of time is spent on writing each application and extensive tailoring is expected for competitive materials. Combining these insights, it is therefore unsurprising that almost all applicants, including applicants that received at least one offer (Supplementary file 29), found the process "time-consuming", a "burden on research", and "stressful" (Figure 8B–C).

44% of respondents had applied for faculty jobs for more than one cycle (Supplementary file 30). Though applicants who applied for more than one cycle had significantly lower offer percentages (p=3.45×10⁻²; Figure 5B), many reported perceived benefits from significant feedback from their current PI through their previous application cycles. Though mentorship was not as often reported as specifically helpful (Supplementary file 17), the lack of mentorship was a commonly cited harmful obstacle (Figure 8B, Supplementary file 18). Lastly, multiple candidates felt that issues pertaining to family, the two-body problem (need for spousal/significant other hire), parental leave, or citizenship status significantly harmed their prospects.

The view from the search committees

To learn more about the characteristics search committees valued in applicants, we performed an exploratory survey of members of such committees. This anonymous survey was distributed in a limited fashion, taking advantage of the professional networks of the authors. Fifteen faculty members responded, with nine having been involved in search committees for over ten years (Figure 9A). As with our survey of applicants, we focused on faculty members at R1 academic centers working in life sciences (14/15 of those polled) and engineering (1/15) within the United States (Figure 9A).

Figure 9

Download asset Open asset

Two-thirds of respondents replied that the search committees they sat on typically received over 200 applicants per job posting, with one-third receiving 100–199 applications per cycle. Between 5 and 8 applicants were typically invited to interview on-site; one-third of respondents replied that off-site interviews (e.g., via phone or Skype) were not performed (Figure 9B). These statistics help demonstrate the challenges that hiring committees face; the sheer volume of applicants is overwhelming, as mentioned explicitly by several search committee respondents (Supplementary file 32).

We asked what factors search committee members found most important, what their perception of the market was, and how they felt it had changed since they first became involved in hiring. We also asked them to weigh specific application criteria in evaluating an application from 1 (not weighted at all) to 5 (heavily weighted; Figure 9C). Criteria such as transition awards were consistently ranked highly, matching applicant perception; however, committee members also placed substantial emphasis on the research proposal. Two-thirds viewed preprints favorably, although their strength may not yet be equivalent to published peer-reviewed work (Figure 9C). In follow-up questions, a number of respondents emphasized that the future potential of the candidate both as a colleague and a scientist was important.

Since this last point was not prominent in our survey of job applicants, we looked for discrepancies in the two sets of responses (Figure 9D). In general, search committees placed greater emphasis on the future potential and scientific character (research proposal, research impact, collegiality), while applicants focused on publication metrics and funding. However, despite the search committees placing less emphasis on papers in CNS, candidates with papers in these journals were more successful.

We also asked if there were additional factors that search committees wished applicants knew when applying (Figure 10). Several emphasized the quality of the research and papers was the most important factor for assessing prior achievement, but added that a compelling and coherent research proposal was also critical, and was sometimes underdeveloped in otherwise competitive candidates. The importance of departmental fit was also emphasized; interpersonal interactions with faculty members at the interview stage were also mentioned. This last sentiment is consistent with a recent Twitter poll which found that "overall attitude/vibe" was the single most important factor for selection at the interview stage (Tye, 2019). Intriguingly, while one faculty respondent noted that they rarely interview any applicant without a career transition award, such as a K99/R00 Pathway to Independence Award from the NIH (a situation they noted as problematic), another lamented that applicants worried too much about metrics/benchmarks anecdotally perceived to be important, such as receiving these awards. Finally, a majority of respondents noted that it was easy to identify good candidates from their submitted application (11/15), that there were too many good applicants (10/15), and that candidates often underperformed at the interview stage (10/15) (Figure 10, Figure 10—figure supplement 1, Supplementary File 35).

Figure 10 with 1 supplement see all

Download asset Open asset

The search process for faculty jobs lacks transparency and data regarding what makes a successful applicant. Here, we began to address this deficiency through a survey targeted at the applicants themselves, and including their perceptions of the application process. Of over 300 responses by job applicants, we did not receive a single positive comment about the process, despite the fact that 58% of our participants received at least one job offer. Our data suggest that baseline thresholds exist for those more likely to receive a faculty job offer, but that many different paths can lead to a job offer. This variety of paths likely reflects both the preparation done by applicants and the different evaluation criteria used by individual search committees. For these reasons, we urge applicants not to conclude that lower than average metrics in any one area are automatically disqualifying. Indeed, we believe that increasing the transparency of the application process through systematic data collection will allow a more detailed study of the many paths to obtaining a faculty offer.

Our data also show the mental strain on applicants during the hiring process. We propose a number of potential solutions with the understanding that hiring faculty is a complex process involving multiple stakeholders. We believe the application process could be improved by simplifying the process, including standardizing application materials (e.g. requirements for research statements are similar for R1 institutions) and requesting references only after candidates are shortlisted, so that the burden of application preparation time can be reduced. Constructive feedback from mentors is vital for success during the application and interview preparation stages. Additionally, if possible, communication from search committees about unsuccessful applications would be helpful. We understand that these points may increase the workload of mentors and search committees but, if put into place, could alleviate some of the stress related to the academic job application process. In addition, applicants need to work to be sure their materials are strong and well-researched as the quality of these materials and demonstrating fit for a job posting are important to faculty on search committees (Clement et al., 2019). Further work into the challenges search committees face is needed to improve their experience of the application process.

It is our hope that this and future work will not only allow all stakeholders to make informed decisions, but will also enable critical examination, discussion, and reassessment of the implicit and explicit values and biases being used to select the next generation of academic faculty. Such discussions are crucial in building an academic environment that values and supports all of its members.

The authors confirm that, for approved reasons, access restrictions apply to the data underlying the findings. Raw data underlying this study cannot be made publicly available in order to safeguard participant anonymity and that of their organizations. Ethical approval for the project was granted on the basis that only aggregated data is provided (as has been provided in the supplementary tables) (with appropriate anonymization) as part of this publication.

1. 1. Abdill RJ
   2. Blekhman R

   (2019) Tracking the popularity and outcomes of all bioRxiv preprints

   eLife 8:e45133.

   https://doi.org/10.7554/eLife.45133

   • PubMed
   • Google Scholar
2. 1. Acton SE
   2. Bell ADJ
   3. Toseland CP
   4. Twelvetrees A

   (2019) A survey of new PIs in the UK

   eLife 8:e46827.

   https://doi.org/10.7554/eLife.46827

   • Google Scholar
3. Website

   1. Ahmed MZ

   (2019) Opinion: the postdoc crisis

   The Scientist Magazine. Accessed August 8, 2019.

   https://www.the-scientist.com/opinion/opinion-the-postdoc-crisis-34259
4. Preprint

   1. Aileen Day PC
   2. Boyle J

   (2019) Is there a gender gap in chemical sciences scholarly communication?

   ChemRxiv.

   https://chemrxiv.org/articles/Is_There_a_Gender_Gap_in_Chemical_Sciences_Scholarly_Communication_/10115936

   • Google Scholar
5. 1. Alberts B
   2. Kirschner MW
   3. Tilghman S
   4. Varmus H

   (2014) Rescuing US biomedical research from its systemic flaws

   PNAS 111:5773–5777.

   https://doi.org/10.1073/pnas.1404402111

   • PubMed
   • Google Scholar
6. 1. Andalib MA
   2. Ghaffarzadegan N
   3. Larson RC

   (2018) The postdoc queue: a labour force in waiting

   Systems Research and Behavioral Science 35:675–686.

   https://doi.org/10.1002/sres.2510

   • Google Scholar
7. Website

   1. Arvanitis L
   2. Cho S

   (2018) Gender imbalance in STEM remains despite diversity initiative

   Yale Daily News. Accessed October 15, 2018.

   https://yaledailynews.com/blog/2018/10/15/gender-imbalance-in-stem-remains-despite-diversity-initiative/
8. 1. Bhalla N

   (2019) Strategies to improve equity in faculty hiring

   Molecular Biology of the Cell 30:2744–2749.

   https://doi.org/10.1091/mbc.E19-08-0476

   • PubMed
   • Google Scholar
9. 1. Brischoux F
   2. Angelier F

   (2015) Academia’s never-ending selection for productivity

   Scientometrics 103:333–336.

   https://doi.org/10.1007/s11192-015-1534-5

   • Google Scholar
10. Website

    1. Brock J

    (2019) The allure of the journal impact factor holds firm, despite its flaws

    Nature Index. Accessed August 29, 2019.

    https://www.natureindex.com/news-blog/allure-journal-impact-factor-holds-firm-despite-flaws
11. 1. Cameron EZ
    2. White AM
    3. Gray ME

    (2016) Solving the productivity and impact puzzle: do men outperform women, or are metrics biased?

    BioScience 66:245–252.

    https://doi.org/10.1093/biosci/biv173

    • Google Scholar
12. 1. Cantwell B

    (2011) Transnational mobility and international academic employment: gatekeeping in an academic competition arena

    Minerva 49:425–445.

    https://doi.org/10.1007/s11024-011-9181-3

    • Google Scholar
13. 1. Ceci SJ
    2. Williams WM

    (2015) Women have substantial advantage in STEM faculty hiring, except when competing against more-accomplished men

    Frontiers in Psychology 6:1532.

    https://doi.org/10.3389/fpsyg.2015.01532

    • PubMed
    • Google Scholar
14. 1. Centra JA
    2. Gaubatz NB

    (2000) Is there gender bias in student evaluations of teaching?

    Journal of Higher Education 71:17.

    https://doi.org/10.2307/2649280

    • Google Scholar
15. 1. Clauset A
    2. Arbesman S
    3. Larremore DB

    (2015) Systematic inequality and hierarchy in faculty hiring networks

    Science Advances 1:e1400005.

    https://doi.org/10.1126/sciadv.1400005

    • PubMed
    • Google Scholar
16. Preprint

    1. Clement L
    2. Dorman JB
    3. McGee R

    (2019) The academic career readiness assessment: clarifying training expectations for future life sciences faculty

    bioRxiv.

    https://doi.org/10.1101/829200

    • Google Scholar
17. Report

    1. Committee to Review the State of Postdoctoral Experience in Scientists and Engineers

    (2014) The postdoctoral experience revisited

    National Academies Press.

    http://www.ncbi.nlm.nih.gov/books/NBK268776/

    • Google Scholar
18. Website

    1. Conroy G

    (2019) Preprints boost article citations and mentions

    Nature Index. Accessed July 9, 2019.

    https://www.natureindex.com/news-blog/preprints-boost-article-citations-and-mentions
19. 1. Cyranoski D
    2. Gilbert N
    3. Ledford H
    4. Nayar A
    5. Yahia M

    (2011) The PhD factory

    Nature 472:276–279.

    https://doi.org/10.1038/472276a

    • PubMed
    • Google Scholar
20. Website

    1. Eastlack S

    (2017) How scarce funding shapes young scientists

    Accessed August 8, 2019.

    https://blogs.plos.org/thestudentblog/2017/03/20/how-scarce-funding-shapes-young-scientists/
21. 1. Eddy SR

    (2005) “Antedisciplinary” science

    PLOS Computational Biology 1:e6.

    https://doi.org/10.1371/journal.pcbi.0010006

    • Google Scholar
22. Website

    1. Fox J

    (2018a) How much do you – and should you – tailor your ecology faculty job application to the hiring institution? Poll results and commentary

    Dynamic Ecology. Accessed July 24, 2018.

    https://dynamicecology.wordpress.com/2018/07/24/how-much-do-you-and-should-you-tailor-your-ecology-faculty-job-application-to-the-hiring-institution-poll-results-and-commentary/
23. Website

    1. Fox J

    (2018b) What proportion of recently-hired tenure-track N. american asst. professors of ecology have nature/Science/PNAS papers? (UPDATED with a second year’s worth of data)

    Dynamic Ecology. Accessed August 8, 2019.

    https://dynamicecology.wordpress.com/2018/03/07/what-proportion-of-recently-hired-tenure-track-n-american-asst-professors-of-ecology-have-nature-science-pnas-papers/
24. Preprint

    1. Fraser N
    2. Momemi F
    3. Mayr P
    4. Peters I

    (2019) The effect of bioRxiv preprints on citations and altmetrics

    bioRxiv.

    https://doi.org/10.1101/673665

    • Google Scholar
25. 1. Fu DY
    2. Hughey JJ

    (2019) Releasing a preprint is associated with more attention and citations for the peer-reviewed article

    eLife 8:e52646.

    https://doi.org/10.7554/eLife.52646

    • PubMed
    • Google Scholar
26. 1. Ghaffarzadegan N
    2. Hawley J
    3. Larson R
    4. Xue Y

    (2015) A note on PhD population growth in biomedical sciences

    Systems Research and Behavioral Science 32:402–405.

    https://doi.org/10.1002/sres.2324

    • Google Scholar
27. Website

    1. Golde CM

    (2019) 5 easy fixes for a broken faculty job market

    Chronicle of Higher Education. Accessed August 8, 2019.

    https://www.chronicle.com/article/5-Easy-Fixes-for-a-Broken/246798
28. 1. Grinstein A
    2. Treister R

    (2018) The unhappy postdoc: a survey based study

    F1000Research 6:1642.

    https://doi.org/10.12688/f1000research.12538.2

    • Google Scholar
29. 1. Gumpertz M
    2. Durodoye R
    3. Griffith E
    4. Wilson A

    (2017) Retention and promotion of women and underrepresented minority faculty in science and engineering at four large land grant institutions

    PLOS ONE 12:e0187285.

    https://doi.org/10.1371/journal.pone.0187285

    • PubMed
    • Google Scholar
30. 1. Hothorn T
    2. Zeileis A

    (2015)

    Partykit: a modular toolkit for recursive partytioning in R

    Journal of Machine Learning Research : JMLR 16:3905–3909.

    • Google Scholar
31. Website

    1. Jay J
    2. Lewis NA
    3. Cunningham WA

    (2019) In the tough academic job market, two principles can help you maximize your chances

    Science. Accessed July 10, 2019.

    https://www.sciencemag.org/careers/2019/07/tough-academic-job-market-two-principles-can-help-you-maximize-your-chances
32. 1. Kahn S
    2. Ginther DK

    (2017) The impact of postdoctoral training on early careers in biomedicine

    Nature Biotechnology 35:90–94.

    https://doi.org/10.1038/nbt.3766

    • PubMed
    • Google Scholar
33. 1. Kuhn M

    (2008) Building predictive models in R using the caret package

    Journal of Statistical Software 28:i05.

    https://doi.org/10.18637/jss.v028.i05

    • Google Scholar
34. Book

    1. Kuhn M
    2. Johnson K

    (2013) Applied Predictive Modeling

    New York: Springer.

    https://doi.org/10.1007/978-1-4614-6849-3

    • Google Scholar
35. Website

    1. Langin K

    (2019) In a first, US private sector employs nearly as many PhDs as schools do

    Science. Accessed March 12, 2019.

    https://www.sciencemag.org/careers/2019/03/first-us-private-sector-employs-nearly-many-phds-schools-do
36. 1. Larson RC
    2. Ghaffarzadegan N
    3. Xue Y

    (2014) Too many PhD graduates or too few academic job openings: the basic reproductive number R₀ in academia

    Systems Research and Behavioral Science 31:745–750.

    https://doi.org/10.1002/sres.2210

    • PubMed
    • Google Scholar
37. 1. Leaper C
    2. Starr CR

    (2019) Helping and hindering undergraduate women’s STEM motivation: experiences with STEM encouragement, STEM-related gender bias, and sexual harassment

    Psychology of Women Quarterly 43:165–183.

    https://doi.org/10.1177/0361684318806302

    • Google Scholar
38. Report

    1. Lee CD

    (2014)

    Search Committees: A Comprehensive Guide to Successful Faculty, Staff, and Administrative Searches

    Stylus and CUPA-HR.

    • Google Scholar
39. 1. Macdonald JE

    (2019) Negotiating details in the academic letter of offer

    Chemistry of Materials 31:9213–9215.

    https://doi.org/10.1021/acs.chemmater.9b04051

    • Google Scholar
40. 1. McDowell GS
    2. Gunsalus KTW
    3. MacKellar DC
    4. Mazzilli SA
    5. Pai VP
    6. Goodwin PR
    7. Walsh EM
    8. Robinson-Mosher A
    9. Bowman TA
    10. Kraemer J
    11. Erb ML
    12. Schoenfeld E
    13. Shokri L
    14. Jackson JD
    15. Islam A
    16. Mattozzi MD
    17. Krukenberg KA
    18. Polka JK

    (2015) Shaping the future of research: a perspective from junior scientists

    F1000Research 3:291.

    https://doi.org/10.12688/f1000research.5878.2

    • Google Scholar
41. 1. McKiernan EC
    2. Schimanski LA
    3. Muñoz Nieves C
    4. Matthias L
    5. Niles MT
    6. Alperin JP

    (2019) Use of the journal impact factor in academic review, promotion, and tenure evaluations

    eLife 8:e47338.

    https://doi.org/10.7554/eLife.47338

    • PubMed
    • Google Scholar
42. 1. Miuccio M
    2. Liu KY
    3. Lau H
    4. Peters MAK

    (2017) Six-fold over-representation of graduates from prestigious universities does not necessitate unmeritocratic selection in the faculty hiring process

    PLOS ONE 12:e0185900.

    https://doi.org/10.1371/journal.pone.0185900

    • PubMed
    • Google Scholar
43. Preprint

    1. Niles MT
    2. Schimanski LA
    3. McKiernan EC
    4. Alperin JP

    (2019) Why we publish where we do: faculty publishing values and their relationship to review, promotion and tenure expectations

    bioRxiv.

    https://doi.org/10.1101/706622

    • Google Scholar
44. 1. Pickett CL

    (2019) The increasing importance of fellowships and career development awards in the careers of early-stage biomedical academic researchers

    PLOS ONE 14:e0223876.

    https://doi.org/10.1371/journal.pone.0223876

    • PubMed
    • Google Scholar
45. 1. Polka JK
    2. Krukenberg KA
    3. McDowell GS

    (2015) A call for transparency in tracking student and postdoc career outcomes

    Molecular Biology of the Cell 26:1413–1415.

    https://doi.org/10.1091/mbc.E14-10-1432

    • Google Scholar
46. 1. Powdthavee N
    2. Riyanto YE
    3. Knetsch JL

    (2018) Lower-rated publications do lower academics’ judgments of publication lists: evidence from a survey experiment of economists

    Journal of Economic Psychology 66:33–44.

    https://doi.org/10.1016/j.joep.2018.04.003

    • Google Scholar
47. Website

    1. Powell JD

    (2017) The price of doing a postdoc

    Science. Accessed August 8, 2019.

    https://www.sciencemag.org/careers/2017/01/price-doing-postdoc
48. 1. Ramirez JJ

    (2016)

    Career advice: finding a job at a predominantly undergraduate institution

    Journal of Undergraduate Neuroscience Education 14:E15–E19.

    • PubMed
    • Google Scholar
49. Website

    1. Rockey S

    (2012) Postdoctoral researchers — Facts Trends, and Gaps

    NIH Extramural Nexus. Accessed August 8, 2019.

    https://nexus.od.nih.gov/all/2012/06/29/postdoctoral-researchers-facts-trends-and-gaps/
50. Website

    1. Rowland A

    (2016) Preparing your academic chalk talk

    American Society for Cell Biology (ASCB) Blog. Accessed May 13, 2016.

    https://www.ascb.org/careers/preparing-academic-chalk-talk/
51. 1. Sarabipour S
    2. Debat HJ
    3. Emmott E
    4. Burgess SJ
    5. Schwessinger B
    6. Hensel Z

    (2019) On the value of preprints: an early career researcher perspective

    PLOS Biology 17:e3000151.

    https://doi.org/10.1371/journal.pbio.3000151

    • PubMed
    • Google Scholar
52. 1. Sauermann H
    2. Roach M

    (2016) Why pursue the postdoc path?

    Science 352:663–664.

    https://doi.org/10.1126/science.aaf2061

    • Google Scholar
53. Website

    1. Saxbe D

    (2019) I’m on a search committee this year, and we just rejected some amazing candidates for our faculty position. Important PSA: it's not personal. A rejection is (truly!) not a referendum on you or your skills/qualifications. More often than not, the decision comes down to fit. 1/

    Accessed August 8, 2019.

    https://twitter.com/darbysaxbe/status/1197091457397612546
54. 1. Schiermeier Q

    (2019) Huge study documents gender gap in chemistry publishing

    Nature 8:03438.

    https://doi.org/10.1038/d41586-019-03438-y

    • Google Scholar
55. 1. Schillebeeckx M
    2. Maricque B
    3. Lewis C

    (2013) The missing piece to changing the university culture

    Nature Biotechnology 31:938–941.

    https://doi.org/10.1038/nbt.2706

    • PubMed
    • Google Scholar
56. Preprint

    1. Sever R
    2. Roeder T
    3. Hindle S
    4. Sussman L
    5. Black K-J
    6. Argentine J
    7. Manos W
    8. Inglis JR

    (2019) bioRxiv: the preprint server for biology

    bioRxiv.

    https://doi.org/10.1101/833400

    • Google Scholar
57. 1. Sheltzer JM
    2. Smith JC

    (2014) Elite male faculty in the life sciences employ fewer women

    PNAS 111:10107–10112.

    https://doi.org/10.1073/pnas.1403334111

    • PubMed
    • Google Scholar
58. Website

    1. Tye K

    (2019) Faculty search committees: once you get to the interview stage, what is the single most important factor in candidate selection?

    Accessed September 18, 2019.

    https://twitter.com/kaymtye/status/1174311929789071360
59. 1. van Dijk D
    2. Manor O
    3. Carey LB

    (2014) Publication metrics and success on the academic job market

    Current Biology 24:R516–R517.

    https://doi.org/10.1016/j.cub.2014.04.039

    • PubMed
    • Google Scholar
60. Website

    1. White M

    (2019) Why science needs more diversity

    Pacific Standard. Accessed August 8, 2019.

    https://psmag.com/education/why-science-needs-more-diversity
61. Website

    1. Whitehead K

    (2019) Faculty, when you accepted your first position, how many offers did you have?

    Accessed August 8, 2019.

    https://twitter.com/kwhiteheadlab/status/1145375772619788289
62. 1. Whittaker JA
    2. Montgomery BL
    3. Martinez Acosta VG

    (2015)

    Retention of underrepresented minority faculty: strategic initiatives for institutional value proposition based on perspectives from a range of academic institutions

    Journal of Undergraduate Neuroscience Education 13:A136–A145.

    • PubMed
    • Google Scholar
63. 1. Witteman HO
    2. Hendricks M
    3. Straus S
    4. Tannenbaum C

    (2019) Are gender gaps due to evaluations of the applicant or the science? A natural experiment at a national funding agency

    The Lancet 393:531–540.

    https://doi.org/10.1016/S0140-6736(18)32611-4

    • Google Scholar
64. Website

    1. WordItOut

    (2020) Make a word cloud

    Accessed July 16, 2020.

    https://worditout.com/word-cloud/create
65. 1. Wright CB
    2. Vanderford NL

    (2017) What faculty hiring committees want

    Nature Biotechnology 35:885–887.

    https://doi.org/10.1038/nbt.3962

    • PubMed
    • Google Scholar

Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by two members of the eLife Features Team. The following individuals involved in review of your submission have agreed to reveal their identity: Adriana Bankston (Reviewer #2).

Summary:

This article (which is based on a survey of more than 300 early-career researchers who are/were on the job market) has the potential to be important. However, the data require further analysis and the presentation and discussion need to be improved.

Essential revisions:

1. The authors should remove the Twitter poll and related analysis from this paper, leaving just the main survey and the faculty survey.

2. Throughout the manuscript, the authors seek to minimize the importance of papers published in Cell, Nature or Science (CNS). However, there are at least two instances in which having a CNS paper appears to be the strongest predictor of job market success (Table S7 and Figure 4B). Additionally, prior studies have also suggested the same (see Pinheiro et al., 2014; and links to multiple peer reviewed studies in Cuff, 2017). It is absolutely critical that the instances where having a CNS paper appears to be the strongest predictor of job market success be noted and described in the text. (See comments from reviewer #1 for a fuller discussion of this issue).

- Cuff, A. J. (2017). An Academic Lottery or a Meritocracy? Inside HigerEd. Retrieved from https://www.insidehighered.com/advice/2017/05/03/phds-need-real-data-how-potential-employers-make-hiring-decisions-essay.

- Pinheiro, D., Melkers, J., & Youtie, J. (2014). Learning to play the game: Student publishing as an indicator of future scholarly success. Technological Forecasting & Social Change, 81, 56-66. doi:10.1016/j.techfore.2012.09.008

3. Men have more first author publications than women particularly in CNS journals - the authors should comment on this.

4. In Figure 4D the authors should identify the factors that differentiate between candidates with one or more offers vs. those who received none.

5. Regarding the logistic regression: rather than conducting an analysis with 16 variables, many of which are collinear, the authors should conduct this analysis one variable at a time to identify the single features that predict job market success. The authors can then test these variables for collinearity and combine uncorrelated variables into models that include two, three, four, etc. independent predictors.

6. 96% of the respondents with CNS publications were in fields in "Biomedical or life sciences" or "Biology (other)" as described in Figure 1A. The authors should therefore repeat several of the key analyses on success predictors on the life science cohort alone, including making a new version of Figure 4 that includes data for the life science cohort alone. The authors should also update the tables S7, S9, S10 and S22 to include statistics about CNS publications done for the life scientists in the cohort alone (alongside the data already presented for the full cohort).

7. The authors conducted a "PUI-only" subgroup analysis (Figure 6), but it would also be informative to conduct an analysis among candidates who only applied to research-intensive positions. The differences between the positions may be noteworthy, and by combining too many unrelated job searches into one analysis, the authors may be missing out on important relationships.

8. The authors should provide some more background on the survey and how it was distributed at the start of the results section. What checks were in place to ensure that only early-career scientists answered it? Similarly, the authors should provide more information in the results section on how the search committee survey was distributed and who answered it.

9. This study really does not represent a cross section of different types of early career researchers from different fields nor "a wide variety of fields." Nor is it really an international applicant pool since data on race/ethnicity or nationality or citizenship status was not asked. It really represents a sample of postdocs (96%) from the biomedical and biological sciences with (72%) currently working within the U.S., Canada and U.K. who are on the job market. This is an important distinction to make because prior research Cantwell (2011), Cantwell & Taylor (2015), and Sauerman & Roach (2016) and others have done research in this area and shown how international postdocs working in the U.S. have had limited success in transitioning into tenure-track faculty positions and provide reasons to suggest why. The authors should discuss this at an appropriate place in the text and consider citing some or all of the following references:

- Cantwell, B., & Taylor, B. J. (2015). Rise of the science and engineering postdoctorate and the restructuring of academic research. Journal of Higher Education, 86(5), p 667-696.

- Cantwell, B. (2011). Transnational mobility and international academic employment: gatekeeping in an academic competition arena. Minerva: A Review of Science, Learning and Policy, 49(4), p 425-445.

- Sauermann, H. & Roach, M. (2016). Why pursue the postdoc path? Science 352:663-664. Doi: 10.1126/science.aaf2061

10. Another concern is an attempt to make correlations between number of applications with both number of interviews and offers, without taking into consideration the quality of the application, nor where the applicants received their training. There have been some large quantitative studies done by others (e.g. Clauset, Arbesman, & Larremore, 2015) that found faculty hiring follows a common and steeply hierarchical structure where doctoral prestige and where an applicant did both their Ph.D. and postdoc appointment better predicts hiring, especially in R1 institutions. The authors should discuss this concern at an appropriate place in the text and consider citing the following reference:

- Clauset, A., Arbesman, S., & Larremore, D. B. (2015). Systematic inequality and hierarchy in faculty hiring networks. Science Advances 1: e1400005. doi:10.1126/sciadv.1400005

11. Under "Statement of Ethics", the authors write that an IRB exemption was obtained on 08/29/2019. However, the survey itself was conducted in April 2019. I believe that retroactive IRB approval is generally prohibited by federal regulations. Was the IRB aware that the study had already been conducted when they granted the exemption? Can the authors comment on this serious discrepancy?

12. Teaching experience was not assessed either from the applicant or the institution perspective. Applicants self-disclosed their presumed teaching experience and then when a number of them did get job offers at R1s, the statement was made that applicants fulfill the teaching requirements for any university type. That is an oversimplification of findings. The authors' data indicates that the majority of applicants applied to R1s. This should be further discussed by the authors.

13. The authors could comment on whether the number of postdoc positions attained correlated with their ability to obtain faculty positions (in other words, are 3 postdoc appointments more desirable as compared to 1 when applying for a faculty position at a prestigious institution?).

14. The authors could address how the application process might be improved so it is a more positive experience.

[We repeat the reviewers’ points here in italic, and include our replies in Roman.]

Essential revisions:

1. The authors should remove the Twitter poll and related analysis from this paper, leaving just the main survey and the faculty survey.

We have now removed the section below from the “Applicants perceive the process to be time-consuming and opaque, with minimal to no feedback” results, including the corresponding tables:

“A separate Twitter poll indicated that applicants in general, not specifically for this cycle, typically spend more than 3 hours tailoring each application (49) (Table S26 Supplementary File 1). Our pooled applicants at minimum then spent a combined 22,932 hours (7,644 applications x 3 hours preparation each), or 2.62 years, on these applications. Individually, this number amounts to 72 hours for each applicant on average, but does not take into account how long the initial creation of “base” application materials takes, which is often a much longer process. In another follow-up Twitter poll, a majority of respondents felt that time spent on preparing faculty job applications impeded their ability to push other aspects of their career forward (Table S27 Supplementary File 1) (50).”

2. Throughout the manuscript, the authors seek to minimize the importance of papers published in Cell, Nature or Science (CNS). However, there are at least two instances in which having a CNS paper appears to be the strongest predictor of job market success (Table S7 and Figure 4B). Additionally, prior studies have also suggested the same (see Pinheiro et al., 2014; and links to multiple peer reviewed studies in Cuff, 2017). It is absolutely critical that the instances where having a CNS paper appears to be the strongest predictor of job market success be noted and described in the text. (See comments from reviewer #1 for a fuller discussion of this issue).

- Cuff, A. J. (2017). An Academic Lottery or a Meritocracy? Inside HigerEd. Retrieved from https://www.insidehighered.com/advice/2017/05/03/phds-need-real-data-how-potential-employers-make-hiring-decisions-essay

- Pinheiro, D., Melkers, J., & Youtie, J. (2014). Learning to play the game: Student publishing as an indicator of future scholarly success. Technological Forecasting & Social Change, 81, 56-66. doi:10.1016/j.techfore.2012.09.008

We thank the reviewers for this comment, and would like to clarify our position. We agree with the reviewers that the discussion of the importance of CNS publications is a sensitive and well-known issue. The relationship between job market success and CNS publications is complex. We have read the references suggested by the reviewers, and we agree that several studies have concluded that CNS publications are generally held in high regard by search committees. Moreover, we discuss in the text that there is a high perceived importance of CNS papers to applicants. However, our data shows 2 clear findings with regards to CNS: 1) Most applicants who get offers do *not* have a CNS paper and 2) Applicants with a CNS paper appear to have a higher chance of getting an offer than those who do not have a CNS paper. We believe that the demonstration of the first point (most candidates do not have a CNS paper) is an important point for postdocs to consider but does not detract from the 2nd point (CNS papers appear to confer an advantage to an applicant). We note that an analysis of the offer association with CNS publications is shown in Figure 4D. We observed a 7-15% increase in offer rate for candidates which had published in CNS journals. We also note that despite this observation, ~60% of our survey respondents still received at least one offer. Moreover, our data (Figure 4C) suggest that those individuals with CNS authorship who also received a job offer were more highly cited than those with CNS authorship without a job offer, suggesting that assessment of the impact of an applicant’s work is a complex blend of many factors.

Our small survey of faculty search committee members, revealed an attitude towards CNS publications that was discordant with that of the applicants and the conventional expectation. While we acknowledge that we do not have extensive information on the decision making process of every single search committee, the volume of current literature around this issue speaks to the culture shift that is occurring around academic hiring. Formal discussions of CNS as a metric and its impact on equity in the professoriate have resulted in the development of guidelines such as DORA, that reflect the burgeoning culture change. While an in-depth analysis of impact is outside the scope of this work, we look forward to future work on this topic.

3. Men have more first author publications than women particularly in CNS journals - the authors should comment on this.

We note that these results are presented in Figure 2B and 2C, as well as discussed in the “applicant scholarly metrics by gender” results subsection. How these results are in line with previous findings is presented within the introductory paragraph to that results section. We additionally revisit this point in the discussion stating “Further, despite the fact that women face numerous challenges in academia, including underrepresentation at the faculty level in most STEM departments (52,53,56,57), and trail men in publication-related metrics (Figure 2B), our data suggest very few differences in outcomes in the May 2018-May 2019 female applicant pool relative to their male counterparts. Both genders received similar numbers of interviews and offers, and gender-based differences in publication-related metrics persisted even when considering only the 185 individuals with offers, suggesting that committees are becoming increasingly aware of gender bias in publication-related metrics and are taking them into account when evaluating applicants (Table S40 Supplementary File 1). We have also now added a further point in the discussion pertaining specifically to the intersection of gender and CNS publications. “First, examination of our data revealed a gender gap in publication metrics, with males reporting more CNS authorship and publications overall, indicating that opportunities for publication are not equally available (52,53). ” Our survey has captured numerous aspects of the academic job market and the factors influencing the success of applicants. We recognize that gender (and other marginalized identities) play a large role. We believe the role of gender is thoroughly discussed in our manuscript while allowing in depth analysis of other factors.

4. In Figure 4D the authors should identify the factors that differentiate between candidates with one or more offers vs. those who received none.

We thank the reviewers for this comment, and for the opportunity to clarify this point within our manuscript. In our study, we performed two different analyses to understand which factors differentiate candidates based on their success in obtaining offers. We performed Wilcoxon tests (presented in Figure 4B and Table S7 Supplementary File 1) comparing the offer percentage (the number of offers/the number of applications) and also performed a logistic regression comparing candidates with offers vs those without (Table S22). The data from these two analyses are summarized in Figure 4D, as noted by the reviewer, but were confusing due to the wording in both the text and the figures. We have clarified this in the text by adding further details regarding the offer status and positive correlations in the following sentence: “When missing values were imputed, significant positive coefficients were observed for having a higher h-index, higher application numbers, career transition awards and identifying as female and obtaining an offer.” Further, we have clarified this in the figure legend for Figure 4D as follows: “Summary of significant results testing criteria associated with 1+ offer (positive) or no offers (negative) with offer outcomes either through Wilcoxon analyses (Table S7 Supplementary File 1) or logistic regression (Table S24 Supplementary File 1) ordered by decreasing effect size.”

5. Regarding the logistic regression: rather than conducting an analysis with 16 variables, many of which are collinear, the authors should conduct this analysis one variable at a time to identify the single features that predict job market success. The authors can then test these variables for collinearity and combine uncorrelated variables into models that include two, three, four, etc. independent predictors.

We thank the reviewer for bringing this to our attention. As suggested by the reviewer, we have undertaken a greedy stepwise variable selection procedure in which we started with the variable that was the most predictive of offer status (1+ offer vs 0 offers) and then incrementally added one variable at a time, selecting for the best pair, triplet, and so on. While doing this we also ensured that no new variable was highly correlated or anticorrelated with a previously included variable (Absolute Spearman correlation coefficient cutoff of 0.5). This resulted in the exclusion of two variables that were highly correlated with citation count: h-index and total number of publications (see correlation plot below).

Author response image 1

Download asset Open asset

Among the remaining 14 variables, the combination of seven variables resulted in the highest accuracy in cross-validation experiments: (Applying to) Other jobs, Application number, Citations, Years on job market, Postdoc fellowships, (transition to independence) Funding and CNS (co-authorship) (see Author response table 1 below).Author response table 1.

We note that while this 7-variable model is more accurate in predicting offer status than the full model presented in the earlier version, there is a possibility of bias typically arising from the subjective decisions made during a variable selection exercise. For instance, the choice of a greedy stepwise approach, the use of Spearman correlation as a measure of collinearity, the correlation coefficient thresholds, the choice of accuracy as the quantity to maximize, among others, are all likely to influence which variables are selected and how many are deemed optimal. Nonetheless, the agreement between the single-variable analyses (Wilcoxon tests) and the more rigorous logistic regression analysis gives us confidence to update the logistic regression analysis in the manuscript to this 7-variables model (see updated Figures 4 and Table S24 Supplementary File 1). We have also modified the text summarizing these results in the “Interplay between metrics” section to read as follows:

"We implemented a rigorous variable selection procedure to maximize accuracy and remove highly correlated variables. This resulted in a model that included only seven variables (Table S24 Supplementary File 1) that was tested on a subset of applicants (n=105) who provided answers across all variables. This regression model revealed that a higher number of applications, a higher citation count and obtaining a postdoctoral fellowship were significantly associated with receipt of an offer. When missing values were imputed and the full applicant pool (n=317) was considered, all previous variables remained significant, and a significant positive coefficient was also observed for having a career transition award. In both versions of the model, the search for non-academic jobs was significantly negatively associated with offer status (Figure 4D). We note that the model with imputed data was more accurate than that with missing values excluded at distinguishing between applicants with and without offers in 10-fold cross-validation experiments. However this accuracy was found to only be 69.6%, which is insufficient to construct a usable classifier of offer status. Due to the predominance of applicants from the life sciences in our dataset, we also repeated these analyses on a subset containing only these applicants. While more variables were included in the model, the general trends remained the same, with the addition of the number of years spent on the job market as a significant negative factor in receiving an offer (Table S25 Supplementary File 1; Figure 4 – Supplement 1)."

6. 96% of the respondents with CNS publications were in fields in "Biomedical or life sciences" or "Biology (other)" as described in Figure 1A. The authors should therefore repeat several of the key analyses on success predictors on the life science cohort alone, including making a new version of Figure 4 that includes data for the life science cohort alone. The authors should also update the tables S7, S9, S10 and S22 to include statistics about CNS publications done for the life scientists in the cohort alone (alongside the data already presented for the full cohort).

We have updated Table S7 in Supplementary File 1 and added new tables S9 and S11 equivalents for Life/Biomedical Sciences applicants (respondents who indicated their field of research as Chemistry, Biology, Bioengineering or Biomedical or Life Sciences) scholarly metrics to the Supplementary file. The new added tables are numbered S10 and S12 in Supplementary File 1. Figure 4 – Supplement 1 shows the same analyses as in Figure 4, but restricted to the life sciences cohort. Table S25 in Supplementary File 1 reports the coefficients, p-values and other related information in the same manner as in Table S24 in Supplementary File 1.

7. The authors conducted a "PUI-only" subgroup analysis (Figure 6), but it would also be informative to conduct an analysis among candidates who only applied to research-intensive positions. The differences between the positions may be noteworthy, and by combining too many unrelated job searches into one analysis, the authors may be missing out on important relationships.

We have updated Figure 6 to include comparisons of candidates who only applied to PUIs (PUI-focused), those who only applied to R1 institutions (R1 Focused) and candidates who applied to both. Our original observations still hold (candidates applying to PUI positions are more likely to have more extensive teaching experience). However the new analysis also reveals some interesting trends (the R1 Focused subgroup is majority male in contrast to the two other groups) that we now comment on in the “Research versus Teaching-intensive institutions” section.

8. The authors should provide some more background on the survey and how it was distributed at the start of the results section. What checks were in place to ensure that only early-career scientists answered it? Similarly, the authors should provide more information in the results section on how the search committee survey was distributed and who answered it.

We have altered the text in two separate places to provide clarity on this point. The first paragraph under “Academic Job Applicant Demographics” has been changed to include statements that participants in the survey self-identified as applicants in the academic job market in 2018-2019. The first paragraph under “Search Committees Value the Future” has been changed to state that the search committee survey was sent to a limited number of faculty, from within the authors’ professional networks, who were known to serve on search committees.

9. This study really does not represent a cross section of different types of early career researchers from different fields nor "a wide variety of fields." Nor is it really an international applicant pool since data on race/ethnicity or nationality or citizenship status was not asked. It really represents a sample of postdocs (96%) from the biomedical and biological sciences with (72%) currently working within the U.S., Canada and U.K. who are on the job market. This is an important distinction to make because prior research Cantwell (2011), Cantwell & Taylor (2015), and Sauerman & Roach (2016) and others have done research in this area and shown how international postdocs working in the U.S. have had limited success in transitioning into tenure-track faculty positions and provide reasons to suggest why. The authors should discuss this at an appropriate place in the text and consider citing some or all of the following references:

- Cantwell, B., & Taylor, B. J. (2015). Rise of the science and engineering postdoctorate and the restructuring of academic research. Journal of Higher Education, 86(5), p 667-696.

- Cantwell, B. (2011). Transnational mobility and international academic employment: gatekeeping in an academic competition arena. Minerva: A Review of Science, Learning and Policy, 49(4), p 425-445.

- Sauermann, H. & Roach, M. (2016). Why pursue the postdoc path? Science 352:663-664. Doi: 10.1126/science.aaf2061

We thank the reviewer for this comment and note that the complete demographic data of survey participants' country of origin, field of scientific research, and current position is included in Figure 1 and detailed in Tables S2, S3 and S5 in Supplementary File 1. We acknowledge that our survey did not fully capture the difficulties faced by non-citizens in obtaining a US faculty position, though these difficulties have been demonstrated in previous literature. We have added that citizenship status was a common issue thought to hinder applicants’ progress in the last line of the last paragraph of the “Applicants perceive the process to be time-consuming and opaque, with minimal to no feedback” section and highlight its prominence in Figure 7B. We have also added the following text to the discussion of the limitations of our study: “As indicated in our open question responses (Figure 7B), international postdocs may be specifically challenged in obtaining faculty job offers in the United States and Europe due to immigration policies as well as how mobility is interpreted by the job market (59).” We note that we had previously referenced Sauermann, H. & Roach, M. (2016) in our introduction, it is reference number 9.

10. Another concern is an attempt to make correlations between the number of applications with both number of interviews and offers, without taking into consideration the quality of the application, nor where the applicants received their training. There have been some large quantitative studies done by others (e.g. Clauset, Arbesman, & Larremore, 2015) that found faculty hiring follows a common and steeply hierarchical structure where doctoral prestige and where an applicant did both their Ph.D. and postdoc appointment better predicts hiring, especially in R1 institutions. The authors should discuss this concern at an appropriate place in the text and consider citing the following reference:

- Clauset, A., Arbesman, S., & Larremore, D. B. (2015). Systematic inequality and hierarchy in faculty hiring networks. Science Advances 1:e1400005. doi:10.1126/sciadv.1400005

While we acknowledge that variables such as training institution and prestige of the applicants’ graduate school and postdoctoral mentors undoubtedly influence who obtains faculty job offers, studying this relationship was beyond the scope of the current study. The current study was not designed to explore these variables because of privacy concerns for our respondents. We did not inquire about respondents’ training institutions, advisors’ prestige, or other metrics associated with their “networks” which may have influenced their job search success. Future work to explore these variables are certainly needed. We have cited Clauset et al in our discussion section when we discuss the potential impact of these training/mentoring/network variables we did not assess in the current study. From “Challenges in the Academic Job Market” in the Discussion:

“Additionally, other unmeasured factors (e.g. applicant pedigree) are likely important considerations, consistent with recent data implicating institutional prestige and non-meritocratic factors in faculty hiring (51). This should be a major consideration for future studies of the academic job market.”

We would like to add that our open-ended question regarding “What was helpful for your application” (Figure 7A) indicated applicants perceived networking methods and pedigree to be valuable in helping them land faculty jobs. So, there is clearly merit to study these metrics in future work.

11. Under "Statement of Ethics", the authors write that an IRB exemption was obtained on 08/29/2019. However, the survey itself was conducted in April 2019. I believe that retroactive IRB approval is generally prohibited by federal regulations. Was the IRB aware that the study had already been conducted when they granted the exemption? Can the authors comment on this serious discrepancy?

We initially conducted the surveys for the Future PI Slack community only. When we decided on data analysis and wider dissemination of the results, we applied for IRB exemption for use of existing data. We have now corrected our statement of ethics to reflect the exemption criteria for publication of the survey data: “The surveys used in this manuscript were designed and implemented by the authors listed above on a voluntary basis outside of their post-doctoral positions at the time. The authors respect the confidentiality and anonymity of all respondents, and no identifiable private information was collected. Participation in both surveys was voluntary and the respondents could stop responding to the surveys at any time. The use of the data collected in these surveys was determined to meet the exemption criteria for secondary use of existing data [45 CRF 46.104 (d)(4)] by the University of North Dakota Institutional Review Board (IRB) on 08/29/2019. IRB project number: IRB-201908-045. Please contact Dr. Amanda Haage (amanda.haage@und.edu) for further inquiries.”

12. Teaching experience was not assessed either from the applicant or the institution perspective. Applicants self-disclosed their presumed teaching experience and then when a number of them did get job offers at R1s, the statement was made that applicants fulfill the teaching requirements for any university type. That is an oversimplification of findings. The authors' data indicates that the majority of applicants applied to R1s. This should be further discussed by the authors.

We have revised the teaching section within the results section to address the reviewers’ comments about oversimplification of the findings. The statement that applicants fulfill the teaching requirement for any university type has been removed, as requested by reviewers. Below is the revised text.

“Levels of teaching experience varied among respondents

Discussion surrounding the academic job market is often centered on applicants' publications and/or funding record, while teaching experience generally receives much less attention. Accordingly, a candidate’s expected teaching credentials and experience vary, largely depending on the type of hiring institution. We asked applicants whether they focused their applications to a specific type of institution (R1, PUI, or both; see Box 1 for definitions), allowing us to examine teaching experience across R1 and/or PUI applicants. Most respondents applied to jobs at R1 institutions (Figure 5A), which may explain the focus on research-centric qualifications. It remains unclear what the emphasis on teaching experience is for search committees at R1 institutions, however the literature suggests that there seems to be a minimal focus (47). Additionally, there might be differences in departmental or institutional requirements that are unknown to outsiders. What is commonly accepted is that many applications to an R1 institution requires a teaching philosophy statement. The majority (99%) of our survey respondents have teaching experience (Figure 5B), with roughly half of applicants’ experience limited to serving as a Teaching Assistant (TA) (Box 1), and half reporting experience beyond a TA position, such as serving as an instructor of record (Figure 5B). The degree of teaching experience did not change based on the target institution of the applicant (Figure 5C), nor did the percentage of offers received significantly differ between groups based on teaching experience (Figure 5D).”

13. The authors could comment on whether the number of postdoc positions attained correlated with their ability to obtain faculty positions (in other words, are 3 postdoc appointments more desirable as compared to 1 when applying for a faculty position at a prestigious institution?).

We did not look for a correlation between the number of postdoc positions and the ability to obtain faculty positions since the length of postdoctoral positions vary widely in our dataset (i.e. 1 postdoctoral position does not necessarily infer a shorter length of training compared to training spanning across more than one postdoctoral positions) and the majority of our respondents had a single postdoctoral appointment (see Figure 1E). In our survey, we did ask respondents for the total number of years (across all appointments) of postdoc training (Figure 1D) and this further reveals field-specific differences in expectations of postdoc appointments and length. Therefore we believe that the relationship between likelihood of receiving a faculty job offer and training extent is complex and likely requires additional considerations (e.g. quality of training over a training timespan, length of graduate training, etc) and is beyond the scope of the analysis in this paper.

14. The authors could address how the application process might be improved so it is a more positive experience.

We note the reviewer’s desire to provide recommendations for hiring institutions and applicants going on the job market. We have added a paragraph to the conclusion of the paper to address how the application process might be improved, so that it is a more positive experience. Additionally, we plan to share our perspective and recommendations for policy changes in a companion piece currently under preparation that provides opinion on data reported here.

The following text has been added to the conclusion of the manuscript:

“Conclusions

The faculty job search process lacks transparency and data regarding what makes a successful applicant. Here, we began to address this need via a job market survey targeted towards the applicants themselves, including their perceptions of the application process. Of over 300 responses by job applicants, we did not receive a single positive comment on the process, despite the fact that 58% of our applicants received at least one job offer. Our data suggest that baseline thresholds exist for those more likely to receive a faculty job offer, but that many different paths can lead to a job offer. This variety of paths likely reflects both the applicant's preparation as well as different evaluation criteria used by individual search committees. For these reasons, we urge applicants not to conclude that lower than average metrics in any one area are automatically disqualifying. Indeed, we believe that increasing the transparency of the application process through systematic data collection will allow a more detailed study of the many paths to obtaining a faculty offer.

Our data show that there is a mental strain on applicants during this process, and we propose a number of potential solutions with the understanding that faculty hiring is a complex process involving multiple stakeholders. We believe the application process could be improved by simplifying the process, including standardizing application materials (e.g. requirements for research statements are similar for R1 institutions) and requesting references only after candidates are shortlisted, so that the burden of application preparation time can be reduced. Constructive feedback from mentors is vital for success during the application and interview preparation stages. Additionally, if possible, communication from search committees about unsuccessful applications would be helpful. We understand that these points may increase the workload of mentors and search committees but, if put into place, could alleviate some of the stress related to the job application process. In addition, applicants need to work to be sure their materials are strong and well-researched as the quality of these materials and demonstrating fit for a job posting are important to faculty on search committees (47). More work is needed to understand the challenges search committees face in order to improve their experience of the application process.

It is our hope that this and future work will not only allow all stakeholders to make informed decisions, but will also enable critical examination, discussion, and reassessment of the implicit and explicit values and biases being used to select the next generation of academic faculty. Such discussions are crucial in building an academic environment that values and supports all of its members.”

Author details

1. Jason D Fernandes

   Jason D Fernandes is in the Department of Biomolecular Engineering, University of California, Santa Cruz, United States and is a member of the eLife Community Ambassadors programme

   Contribution

   Conceptualization, Resources, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

   Contributed equally with

   Sarvenaz Sarabipour

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8625-1796

2. Sarvenaz Sarabipour

   Sarvenaz Sarabipour is in the Institute for Computational Medicine and the Department of Biomedical Engineering, Johns Hopkins University, Baltimore, United States and is a member of the eLife Early-Career Advisory Group

   Contribution

   Conceptualization, Data curation, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   Contributed equally with

   Jason D Fernandes

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5097-5509

3. Christopher T Smith

   Christopher T Smith is in the Office of Postdoctoral Affairs, North Carolina State University Graduate School, Raleigh, United States

   Contribution

   Validation, Investigation, Methodology, Writing - original draft, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8212-7886

4. Natalie M Niemi

   Natalie M Niemi is in the Morgridge Institute for Research, Madison, United States and in the Department of Biochemistry, University of Wisconsin-Madison, Madison, United States

   Contribution

   Conceptualization, Investigation, Methodology, Writing - original draft, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5174-4005

5. Nafisa M Jadavji

   Nafisa M Jadavji is in the Department of Biomedical Sciences Midwestern University, Glendale, United States and is a member of the eLife Community Ambassadors programme

   Contribution

   Conceptualization, Investigation, Methodology, Writing - original draft, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-3557-7307

6. Ariangela J Kozik

   Ariangela J Kozik is in the Division of Pulmonary and Critical Care Medicine, Department of Internal Medicine, University of Michigan, Ann Arbor, United States

   Contribution

   Validation, Investigation, Methodology, Writing - original draft, Project administration, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2322-4085

7. Alex S Holehouse

   Alex S Holehouse is in the Department of Biochemistry and Molecular Biophysics, Washington University School of Medicine, St. Louis, United States

   Contribution

   Conceptualization, Resources, Data curation, Investigation, Methodology, Project administration

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-4155-5729

8. Vikas Pejaver

   Vikas Pejaver is in the Department of Biomedical Informatics and Medical Education and the eScience Institute, University of Washington, Seattle, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Visualization, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-1943-0284

9. Orsolya Symmons

   Orsolya Symmons was at the Department of Bioengineering, University of Pennsylvania, Philadelphia, United States. Current address: Max Planck Institute for Biology of Ageing, Cologne, Germany

   Present address

   Max Planck Institute for Biology of Ageing, Cologne, Germany

   Contribution

   Conceptualization, Data curation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2435-4236

10. Alexandre W Bisson Filho

    Alexandre W Bisson Filho is in the Department of Biology and the Rosenstiel Basic Medical Science Research Center, Brandeis University, Waltham, United States

    Contribution

    Data curation, Formal analysis, Validation, Investigation, Visualization, Methodology

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0002-5940-7230

11. Amanda Haage

    Amanda Haage is in the Department of Biomedical Sciences, University of North Dakota, Grand Forks, United States

    Contribution

    Conceptualization, Resources, Data curation, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing - original draft, Project administration, Writing - review and editing

    For correspondence

    amanda.haage@und.edu

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0001-6305-440X

• 29,172

  views

• 2,103

  downloads

• 47

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.