Webinar Report: Refreshing approaches to researcher evaluation

Miguel Godinho Ferreira was proud to be part of a recruitment call at the Gulbenkian Institute that told applicants NOT to include “numerical indicators, such as the impact factor” on their CVs. Some candidates went further, and didn’t name the journal their work had been published in. Instead, candidates were invited to describe up to three of their top findings in more detail.

The HHMI investigator program funds researchers for up to seven years as they tackle difficult questions that may take years to answer. Janet Shaw explained that HHMI developed an application process that allows applicants to “distill their scientific accomplishments and describe their approach to studying or solving an important biological problem”. As such, and similar to the Gulbenkian Institute, applicants are asked to provide significance statements about 3-5 of their papers that document their most important scientific contributions with a focus on the recent past. These help the people reviewing the application to understand how you think about the work that you’ve done, why you think it’s important, and why you have the potential to make important discoveries in the field.

When Michelle Linterman asked her boss why she got her position as a tenure track Group Leader, she was told that the Babraham Institute assesses applicants in two main areas. First, although a strong track record of publications is important, these papers do not have to be published in high-impact journals. In Linterman’s case, she had published regularly and her work had been well-received and highly cited. The second important factor is research fit. What are your research goals and how do they align with the strategic direction of the institute? What makes you different from other researchers in your field?

Raquel Oliveira says that it is important to think carefully about your answers to these questions as they “can really make or break your chances of getting a position”.

Another way to establish whether you’re right for an institution is to get to know people personally. “The networks that I’ve made […] have been an incredibly important part of people knowing both me and my work”, says Linterman. Talking face to face can be one of the most effective ways of conveying your interests and approaches to research. In Ferreira’s case, he attributes his first postdoc position to the fact that he had the opportunity to have a one on one interview with his advisor. “An interview […] should be part of all selection procedures”, he says.

Nicolas Le Novere has been part of many recruitment, grant and evaluation panels at different levels, and has found that panel members often only glance at an applicant's CV, focussing instead on the journals in which the applicant has published. Part of the problem, he says, is that information about work other than journal publications is often missing from the CVs. Le Novere suggests that applicants should prominently list the other scientific work by which they want to be assessed – for example, non-peer reviewed articles, technical specifications, software tools – and also any non-research activities (such as public engagement or training they have led) that have had a significant impact. Applicants should not, however, just list everything they have ever done: “research the members of the evaluation panels beforehand”, says Le Novere, “and put the emphasis on the types of outcomes that are most likely to impress them”.

How Linterman recruits PhD students and postdocs to her lab has little to do with their publication record. Instead, she evaluates them by “what they’re able to do, what their scientific background is, and ultimately their work ethic and their enthusiasm for research”. By taking steps to improve our own evaluation procedures where we have the power to do so, we can help to create new standards for assessment in science.