In the United Kingdom (UK), 81.4% of white students and 68.0% of Black, Asian, and other minority ethnic (BAME) students are awarded good undergraduate degrees (where good means a first or upper second), which equates to an award gap of 13.4% (see Methods for more details on the use of the term BAME in this article). The existence of an award gap has been known about for over 25 years, and such gaps have been reported for every university in the UK (AdvanceHE, 2020b; Woolf et al., 2011).

Similar gaps have also been reported in the Netherlands, United States (US) and Australia for undergraduate and postgraduate taught degrees, and other college-level courses (Stegers-Jager et al., 2016; Kleshinski et al., 2009; Tekian et al., 2001; Alexander et al., 2009; Harris et al., 2020; Haak et al., 2011; Rath et al., 2012; Farmer, 2017; Kay-Lambkin et al., 2002). In theory, scores in qualifying entrance exams may explain the gap in grades awarded. Indeed, studies in the US, UK and the Netherlands have reported gaps in the marks awarded to different ethnic groups for various qualifying exams for both undergraduate and postgraduate courses (Miller and Stassun, 2014; Miller et al., 2019; Reeves and Halikias, 2017; Stegers-Jager et al., 2016; UK Government, 2020). However, when tested directly in both the UK and the US, the award gap between ethnic groups persists when comparing students entering higher education with the same qualifications (Amos and Doku, 2019; Tekian et al., 2001). Furthermore, despite students from Chinese and Indian ethnic backgrounds outperforming white students in university entrance exams in the UK, an award gap of 5% still exists for both of these minority groups (UK Government, 2020; AdvanceHE, 2020b).

These analyses indicate that institutions themselves are responsible for the award gap between white and BAME students. Within the context of marking students’ work, racism or unconscious (implicit) bias is unlikely since most marking is blind, and when it is not (such as in clinical work) it has been found to be an unlikely major cause (McManus et al., 2013; Woolf et al., 2011; Yeates et al., 2017). Other studies investigating the cause of the award gap have controlled for over 20 other potential factors, such as poverty, age, prior school type and learning style. Yet, while some of these factors have been found to make a contribution, a large, unexplained award gap still remains (Broeke and Nicholls, 2007; Woolf et al., 2013).

The award gap follows BAME students into their careers: the proportion of white graduates in employment or further education one year after graduation is a third higher than the proportion of BAME graduates (UK Government, 2020). Moreover, only 9.1% of professors are BAME (AdvanceHE, 2020a), compared with 14% of the population in England and Wales, and 40% in London (White, 2012). Most of this attrition takes place at the transition from undergraduate to PhD student, and the transition from PhD student to postdoctoral researcher (see Methods for the trajectory; AdvanceHE, 2020a; AdvanceHE, 2020b). In practice, selection for interview for the most competitive and prestigious PhD positions – which appear to increase the chances of progressing further in an academic research career – tend to go to those students with the highest grade (that is, to students with a first, or students with an upper second overall with a first or high upper second in the relevant modules), which fewer BAME students possess (AdvanceHE, 2020b; this report). This in turn affects progression into postdoctoral positions as students on prestigious PhD programs have a higher chance of obtaining the skills and outputs favoured by selection panels.

There is little direct data, but this notion is consistent with the observation that Black Caribbean PhD students are less likely than other PhD students to have a fully-funded UK Research Council Studentship (a pre-doctoral fellowship that is considered to be highly competitive; Williams et al., 2019). It is important to note, however, that other inequalities linked to the undergraduate degree, such as disparities in information on application processes and prior university attended, also likely contribute to reducing the chance of a BAME student progressing to PhD (Williams et al., 2019) or receiving an interview for doctorate positions. Furthermore, previous data have shown that other inequalities also contribute to career outcomes in the UK: for example, considerably fewer school leavers (of all white and minority ethnic groups combined) who live in the most deprived neighbourhoods progress to higher education (Sosu and Ellis, 2014; UCAS, 2019), and Pakistani, Black, or Roma/Gypsy/Irish/other traveller ethnic groups are awarded fewer high grades in end-of-school exams (UK Government, 2020).

A similar effect is also seen in the US, where the ethnicity award gap (and to a lesser extent, gender gaps) in the graduate record exams (GRE) leads to higher rejection rates for these minority groups when applying to PhD positions, or even prevents them from applying (Hodapp and Woodle, 2017; Miller and Stassun, 2014). Efforts to reduce racism and unconscious bias are critical for all stages of recruitment and promotion in academia. However, the transitions to PhD and postdoc positions show by far the largest loss of talent of minority ethnic groups, so focusing resources on these transitions is likely to have the largest impact.

One strategy for reducing the unexplained award gap for undergraduate degrees is to seek quantitative (see Methods for definition) and scientific data on the underlying causes. Although such evidence is recognised as important, it is not widely used (Amos and Doku, 2019). Furthermore, the fact the gap can vary from subject to subject – from 3% to 22% in the UK (AdvanceHE, 2020b) – is often not considered. This suggests that different solutions (or different combinations of solutions) may be needed for different subjects.

Here, data from courses in cell biology at University College London (UCL) are used to explore the origins of the unexplained award gap, and to suggest measures to reduce this difference and lessen the impact of this gap on academic career progression. In addition, conclusions based on discussions with the organisers of PhD programmes are presented in the article along with an analysis of published initiatives that have increased the recruitment and retention of doctoral students from minority ethnic backgrounds.

Undergraduate degree pathways in the biosciences and natural sciences at UCL are complex with many possible routes, which is typical across the life sciences in the UK. The cell biology courses (also termed modules) in this study were part of 17 different degree pathways: 15 bioscience and natural science routes were taken by 96.1% of students and the other two routes were taken by the remaining 3.9% (other life science or medical students). The third and often final year of the degree is fragmented into smaller specialised courses, which means that students do not typically take the same modules across all years of their degree. To accommodate this constraint, two complimentary approaches were taken to study the award gap in cell biology.

The first approach, termed the ‘year 1–2 pathway’, analyses the marks of students entering UCL between 2013–2016 who studied the same cell biology courses in year 1 and 2 (up to 344 students; Table 1). Out of the 344 students, eight did not have a record for having taken year 1 cell biology: based on when they joined UCL, five are likely to have transferred directly into year 2 from another UCL degree, and the other three from another university. In year 1, each student takes one of two courses, and the grades awarded were combined as a single year 1 cell biology output. In year 2 all students study the same course.

The other experimental approach, termed the ‘year 3 study’, looked at four different year 3 courses (448 students, entering UCL between 2012–2016; Table 1). Each course had a different composition of students and was chosen on the basis of their class size and variety of specialisms. From the data, it was possible to approximate how many students from the year 1–2 pathway were present in at least one of the year 3 courses (see Methods). This suggests that ~41% (182/448) of the students in the year 3 study flowed from the year 1–2 pathway and an additional ~59% (266/448) came from other bioscience degrees or another degree route.

In the year 1–2 pathway there are four cohorts (one for each year of entry from 2013 to 2016). For clarity, all years in this study are academic years (which in the UK start in September-October and end May-June the following year) and cohorts are cited by the year they start. As each cohort takes two courses – one in year 1 and one in year 2 – a total of eight sets of marks were analysed (Table 1). The year 3 study encompasses students who entered UCL from 2012 to 2016, equalling five-year groups in total. Due to the small class sizes, the marks of the five-year groups included in the study are aggregated together for each course.

In the UK, most undergraduate degrees are either a three year bachelor’s degree, or a four year undergraduate master’s degree (exceptions include medicine). The fourth year was not studied here, because most of that year is a lab project and the ethnicity of students is visible to the examiners, whereas in the cell biology courses selected, all the marking is done blind or by a machine. This circumvents the possibility of unconscious bias or racism contributing to or explaining any gap identified.

In addition to the 792 students included in the study (across both experimental approaches), an additional 22 students were excluded: 12 withheld their ethnicity and 10 (five BAME and five white) did not complete the course. 47.0% (372/792) of the cell biology students assigned themselves as having BAME ethnicity, and 63.3% (501/792) were UK-domiciled. Out of the 792 students included in the study, 95.5% (756) had a grade returned for all the individual components in their course; all data that was returned for all 792 students were included (see Methods - Student Count for further details). In the UK, the final degree is then classified into bands (grades) according to the percent awarded: first class,70% or higher; upper second (2i), 60–69%; lower second (2ii), 50–59%; third class (40–49%); lower than 40% is further separated into referral and fail. A mark of 60% or higher (i.e., a first or upper second) is considered a good degree or a good mark when looking at components of a degree.

The average award gap in good grades is the difference in the mean proportion of BAME and white students that are awarded good marks. For each course, the final mark awarded is the sum of a weighted exam and weighted coursework mark (with exams typically receiving the higher weighting). Here, to enable a comparison of exam and coursework results, the raw (unweighted) marks are reported for each, along with the final weighted mark.

Entrance qualifications for cell biology students in this study

The most common type of entrance exam (termed advanced level, or A-level) was accessible from UCL records. At UCL about 90% of UK and 30–40% of international students are admitted with A-levels. The university places strict equivalency requirements on those students that enter with other types of admission exams. Thus A-level grades are a reasonable approximation of all students in the study.

UCL released the A-level grades of a sub-set of students in this study. However, these are representative, and include the first three cohorts (out of four) from the year 1–2 pathway (250 students), 54% of which (135) also took one of the courses in the year 3 study. 72% (179/250) were admitted with A-levels. Typically, only three A-levels are required in sciences or sciences and maths for entrance in science-related degrees. Approximately a quarter received four or more A-levels, and the other three-quarters had three A-levels. To make the analysis fair, for students with four or more entrance subjects, their three highest grades in most relevant A-level subjects (sciences and maths) were included. A student with grades in only two A-level subjects was also included in the analysis.

A-level grades awarded are alphabetical with A* the highest and E the lowest. Each grade was then assigned a numerical value based on the UK standard tariff: A* = 56; A = 48; B = 40; C = 32 (no student had lower than a B). An average of the best three grades (as described above) was then calculated for each student.

The mean admission tariff for BAME (50.53 ± 0.15) and white (50.29 ± 0.33) students included in this study were virtually identical (Figure 1A): the equivalent of nearly all students in each of these ethnic groupings gaining one A* grade and two A grades (which would be a mean tariff of 50.7). These data are very similar to the mean entrance tariff for all BAME (50.27 ± 0.06) and all white (50.03 ± 0.01) undergraduate students who entered UCL with A-levels for the same cohort years as these cell biology students (2013, 2014, 2015 entrants).

Figure 1

Download asset Open asset

Figure 1—source data 1

Source data on the entrance qualifications of white and BAME cell biology students entering UCL in 2013–2015 and their subsequent grades for year 2 cell biology.

https://cdn.elifesciences.org/articles/58971/elife-58971-fig1-data1-v1.xlsx

Download elife-58971-fig1-data1-v1.xlsx

An alternative approach for assessing admission qualifications is to look at the distribution of the total number of each individual grade (A*, A, B, C or lower) awarded to each cohort. As expected distributions for BAME and white students were nearly identical (Figure 1B); of the total grades awarded to BAME students 95.45% ± 1.01 were A* or A, and for white students this figure was 95.02% ± 0.62 (Figure 1C).

Looking at the grades awarded to the same three cohorts of students in year 2 of their biosciences, natural sciences or other degree revealed that proportionally fewer of these same BAME students were awarded good grades for cell biology (Figure 1C). The mean of all marks awarded was a low 2i (60–64%) for BAME students and a high 2i (65–69%) for white students (Figure 1C). Students admitted with A-levels performed similarly in cell biology to all students admitted in the study as expected (Figure 1—source data 1).

These data indicate that prior qualifications do not explain the subsequent award gap in cell biology, at least at the resolution that the A-level grading system permits. This is in line with other admission studies at UK universities (Amos and Doku, 2019; Broeke and Nicholls, 2007; Woolf et al., 2013).

Dissecting cell biology course components at the level of good grades

In all year 1, year 2 and year 3 courses in the study, proportionally fewer BAME students were awarded final good grades than white students (Figure 2A, B and E). That the award gap has appeared by year 1 has also been reported for the University of Nottingham (Amos and Doku, 2019) and is the case for all undergraduate degrees at UCL (Figure 2F). The average gap in good grades awarded each year of study in cell biology (~8–13%) and across UCL (~7–11%) were similar (Figure 2, compare E and F, final mark). This suggests that the gaps identified in cell biology are in the range expected for students studying at UCL. Note the UCL award gap each year of study (starting the year 2016–18) were higher (~7–11%; Figure 2F) than the final degree classification (average 5.7%; 13,715 UK and international students starting the graduation year 2016–2018). One likely reason is that for most degrees, a defined number of worse performing courses are automatically excluded when calculating the final classification.

Figure 2

Download asset Open asset

Figure 2—source data 1

Source data for the final course, exam and coursework marks of white and BAME students in the year 1–2 pathway and year three study; and the grades for all UCL undergraduates that took year 1, 2 and 3 in either 2016, 2017 or 2018.

https://cdn.elifesciences.org/articles/58971/elife-58971-fig2-data1-v1.xlsx

Download elife-58971-fig2-data1-v1.xlsx

For 11 out of 12 sets of marks studied, this award gap seemed largely derived from the exam component of the course (Figure 2C–E). The average exam gaps were similar in year 1 and 2 at over 13% each, and most pronounced in year 3 at nearly 17% (Figure 2E). Whether the peak in year 3 is a trend needs to be more widely tested. No exam gap was detected in year 2, cohort 4 (Figure 2C). However, a gap overall was still observed for this group (Figure 2A). This was explained by a longer ‘tail’ of lower exam marks for BAME students which resulted in fewer BAME students (61.3%; 23/36) being able to use their coursework grade to obtain a good final mark compared to white students (88.5%; 23/26).

The coursework award gap in good grades was smaller than the exam gap in all individual cohorts and cell biology courses in year 1 (4/4) and year 3 (4/4); and most of the cohorts in year 2 (3/4) (Figure 2C,D, compare solid and dashed lines). The coursework award gap was 7.1- (year 1), 13.1- (year 2) and 3.1- (year 3) times smaller than the exam award gap on average (Figure 2E, compare exam and coursework values). Taken together this suggests a reproducible and therefore significant pattern.

It is speculated that the small coursework award gap (1–2%) for the year 1–2 pathway in this study reflects small fluctuations in different admission years: this is consistent with 5/8 incidences where BAME students were awarded equal, near equal or slightly higher proportion of good marks for the coursework than white students (observable in Figure 2C, compare dashed lines). In contrast, although different admission years cannot be evaluated in the year three study as they were necessarily aggregated (see An exploration of the award gap in cell biology courses), there is no such fluctuation, suggesting that the coursework effect in year 3 courses in this study (average 5% award gap) is consistently small (Figure 2D, compare dashed lines).

In theory, if most students are awarded a good grade that may mask a gap. However, when the grades awarded for the exam and coursework were similarly high as each other (such as cohorts 1 and 3 in year 1 for white students) a relatively large award gap between white and BAME students was still distinguishable for the exam, but was small or none for the coursework of the same cohort (Figure 2C). Similarly, it is unlikely that larger gaps are only revealed when tasks are harder: for example, in year 3, which is expected to have harder coursework tasks overall, grades were broadly the same for exams and coursework (for white students), yet the gap in grades awarded to BAME and white students were only evident in the exam.

Overall, the method of looking at good marks seems sensitive enough to both detect a gap and to reflect its size if it exists. Cohen’s D (which is a measure of effect size) was then applied to the mean award gap of all four cohorts or courses studied for each component in the year 1–2 pathway and year 3 study. This revealed that the effect for coursework is three to six times smaller than the exam (Figure 2—source data 1).

Testing for potential hidden award gaps across all marks

There is a further possibility that there may be a larger coursework award gap, but it was not detectable because it occurred within a specific range of marks. However, in year 2 and year 3 cell biology a larger coursework gap could not be found when comparing every absolute mark awarded to BAME and white students at the resolution of 5% intervals (which is one-half grade band) from less than 30% to 100% scores (Figure 3; only cohorts 1–3 were included in the analysis for year 2 – see legend for more details) Clearly evident in these frequency distributions, the peak of final marks (Figure 3A and B) and the peak of exam marks (Figure 3C and D) awarded to BAME students were shifted half to more than one grade band lower than white students (Figure 3A–D). The end result is that BAME students are most likely to receive a low 2i (60–64%), whilst white students are more likely to receive a high 2i (65–69%) or a low first (70–74%) for these courses overall (Figure 3A and B).

Figure 3 with 1 supplement see all

Download asset Open asset

In contrast, the histograms of coursework marks for white and BAME students were almost completely superimposed in both studies (Figure 3E and F). Further inspection of the curves demonstrates the method is sensitive enough to visualise the small award gap in good marks between BAME and white students for the coursework in year 3 (Figure 3F, slight broadening in the bell-shaped, blue curve away from the orange curve at 59% to 40%) that was reported in Figure 2E. However, clearly there is no larger visible gap anywhere in the entire spectrum of marks at the resolution of 5% intervals for either year 2 or 3 cell biology coursework (Figure 3E and F).

There was insufficient data to fully assess year 1 cell biology course components by first class marks, though they were found to follow the same general trends as year 2 and 3 (see Limitations of the study).

For the past 10 years, only 0.5% per year has been shaved off the undergraduate degree award gap (AdvanceHE, 2020b). Yet in the same period there have been extensive reviews and reports that have detailed why the unexplained, award gap exists and hundreds of recommendations on how universities can tackle it (Alexander and Arday, 2015; Amos and Doku, 2019; Berry and Loke, 2011; ECU and HEA, 2008; Miller, 2016; NUS, 2011; summarised in Figure 4; for a review of earlier studies (from pre-1990s) see Singh, 2011).

Figure 4

Download asset Open asset

Figure 4—source data 1

Source data for the values used to calculate the percentages of the different recommendation categories.

https://cdn.elifesciences.org/articles/58971/elife-58971-fig4-data1-v1.xlsx

Download elife-58971-fig4-data1-v1.xlsx

A scrutiny of the literature suggests that various factors are reducing the speed at which the award gap is closing: lack of staff time coupled to insufficient funding and resources to implement existing recommendations (see Policy 1); limited scientific approaches and diverse types of evidence on the underlying causes (Policy 2); limited funding for fundamental scientific studies and for applied scientific approaches on what works (Policies 3 and 4). That the gap is only disappearing very slowly also propels the need for alternative, parallel action to reduce impact on academic (and other) career progression, by removing (in the case of academic careers) barriers to accessing PhD and postdoc positions (Policy 5). Overall, combining these five strategies may help eliminate the award and progression gap between white and BAME students.

Many caring academic and other university staff are driven to eliminate the unexplained, undergraduate degree award gap, but it is an immense challenge. What seems missing is sufficient and sustained investment to create enough time, resources and training for them to do so. In addition, there needs to be more funding (such as through an alliance of stakeholders) for scientific approaches to research the fundamental causes of this gap, and implement and identify which of the existing recommendations work.

Whilst the award gap persists, parallel actions on monitoring and changing recruitment practices are more widely needed across society in all jobs that employ graduates, to accelerate equal opportunity. Combining both sets of actions (Policies 1–5) will widen opportunity for increased numbers of BAME individuals to access the trajectory that leads to the more senior academic positions. This may positively feedback on itself, as students consider the lack of ethnic diversity among senior leaders as the number one cause of the gap (Amos and Doku, 2019).

The study looked at individual academic results of human subjects and for legal reasons can only be published as aggregated data. The individual grades of each student are therefore not included. Raw values used to generate the figures in the article have been included in the source data. See the methods section for further details.

1. Report

   1. AdvanceHE

   (2020a) Equality in Higher Education, Staff Statistical Report 2020

   AdvanceHE.

   https://www.advance-he.ac.uk/knowledge-hub/equality-higher-education-statistical-report-2020

   • Google Scholar
2. Report

   1. AdvanceHE

   (2020b) Equality in Higher Education, Student Statistical Report 2020

   AdvanceHE.

   https://www.advance-he.ac.uk/knowledge-hub/equality-higher-education-statistical-report-2020

   • Google Scholar
3. 1. Alexander C
   2. Chen E
   3. Grumbach K

   (2009) How leaky is the health career pipeline? Minority student achievement in college gateway courses

   Academic Medicine 84:797–802.

   https://doi.org/10.1097/ACM.0b013e3181a3d948

   • Google Scholar
4. Report

   1. Alexander C
   2. Arday J

   (2015) Aiming Higher: Race, Inequality and Diversity in the Academy

   The Runnymede Trust.

   http://www.runnymedetrust.org/uploads/Aiming_Higher.pdf

   • Google Scholar
5. Report

   1. Amos V
   2. Doku A

   (2019) Black, Asian and Minority Ethnic Student Attainment at UK Universities: #closingthegap

   Universities UK and National Union of Students.

   http://www.universitiesuk.ac.uk/policy-and-analysis/reports/Documents/2019/bame-student-attainment-uk-universities-closing-the-gap.pdf

   • Google Scholar
6. Report

   1. Berry J
   2. Loke G

   (2011) Improving the Degree Attainment of Black and Minority Ethnic Students

   Equality Challenge Unit and Higher Education Academy.

   http://www.advance-he.ac.uk/knowledge-hub/improving-degree-attainment-black-and-minority-ethnic-students

   • Google Scholar
7. Report

   1. Broeke S
   2. Nicholls T

   (2007) Ethnicity and Degree Attainment

   Department for Education and Skills.

   https://dera.ioe.ac.uk/6846/1/RW92.pdf

   • Google Scholar
8. Report

   1. ECU and HEA

   (2008) Ethnicity, Gender and Degree Attainment Project: Final Report

   Equality Challenge Unit and Higher Education Academy (Now AdvanceHE).

   http://www.ecu.ac.uk/wp-content/uploads/external/ethnicity-gender-and-degree-attainment-project-final-report.pdf

   • Google Scholar
9. Report

   1. Farmer L

   (2017) Success Rates and Examiner Bias in the Testing of International Medical Graduates on High-stakes Postgraduate Clinical Examinations

   Australasian College for Emergency Medicine.

   https://acem.org.au/getmedia/b87ed5ff-7210-450f-b676-9649bd86d6de/EFarmer_Report.aspx

   • Google Scholar
10. 1. Haak DC
    2. HilleRisLambers J
    3. Pitre E
    4. Freeman S

    (2011) Increased structure and active learning reduce the achievement gap in introductory biology

    Science 332:1213–1216.

    https://doi.org/10.1126/science.1204820

    • PubMed
    • Google Scholar
11. 1. Harris RB
    2. Mack MR
    3. Bryant J
    4. Theobald EJ
    5. Freeman S

    (2020) Reducing achievement gaps in undergraduate general chemistry could lift unprecedented students into a “hyperpersistent zone”

    Science Advances 6:eaaz5687.

    https://doi.org/10.1126/sciadv.aaz5687

    • PubMed
    • Google Scholar
12. Report

    1. HESA

    (2020) UK Domiciled First Year Students at Higher Education Providers with Low Medium and High Entry Tariffs by Ethnicity 2013/14 to 2018/19

    HESA.

    http://www.hesa.ac.uk/data-and-analysis/students/table-27

    • Google Scholar
13. 1. Hodapp T
    2. Woodle KS

    (2017) A bridge between undergraduate and doctoral degrees

    Physics Today 70:50–56.

    https://doi.org/10.1063/PT.3.3464

    • Google Scholar
14. 1. Kay-Lambkin F
    2. Pearson SA
    3. Rolfe I

    (2002) The influence of admissions variables on first year medical school performance: a study from Newcastle University, Australia

    Medical Education 36:154–159.

    https://doi.org/10.1046/j.1365-2923.2002.01071.x

    • PubMed
    • Google Scholar
15. 1. Kleshinski J
    2. Khuder SA
    3. Shapiro JI
    4. Gold JP

    (2009) Impact of preadmission variables on USMLE step 1 and step 2 performance

    Advances in Health Sciences Education 14:69–78.

    https://doi.org/10.1007/s10459-007-9087-x

    • Google Scholar
16. 1. Krzywinski M
    2. Altman N

    (2013) Points of significance: error bars

    Nature Methods 10:922.

    https://doi.org/10.1038/nmeths.26

    • Google Scholar
17. 1. Langin K

    (2019) A wave of graduate programs drops the GRE application requirement

    Science 372:aay2093.

    https://doi.org/10.1126/science.caredit.aay2093

    • Google Scholar
18. 1. McManus IC
    2. Elder AT
    3. Dacre J

    (2013) Investigating possible ethnicity and sex bias in clinical examiners: an analysis of data from the MRCP(UK) PACES and nPACES examinations

    BMC Medical Education 13:1–11.

    https://doi.org/10.1186/1472-6920-13-103

    • PubMed
    • Google Scholar
19. Report

    1. Miller M

    (2016) The Ethnicity Attainment Gap: Literature Review

    University of Sheffield.

    https://www.sheffield.ac.uk/polopoly_fs/1.661523!/file/BME_Attainment_Gap_Literature_Review_EXTERNAL_-_Miriam_Miller.pdf

    • Google Scholar
20. 1. Miller CW
    2. Zwickl BM
    3. Posselt JR
    4. Silvestrini RT
    5. Hodapp T

    (2019) Typical physics PhD admissions criteria limit access to underrepresented groups but fail to predict doctoral completion

    Science Advances 5:eaat7550.

    https://doi.org/10.1126/sciadv.aat7550

    • PubMed
    • Google Scholar
21. 1. Miller C
    2. Stassun K

    (2014) A test that fails

    Nature 510:303–304.

    https://doi.org/10.1038/nj7504-303a

    • Google Scholar
22. Book

    1. NASEM

    (2019) Minority Serving Institutions: America's Underutilized Resource for Strengthening the STEM Workforce

    National Academies Press.

    https://doi.org/10.17226/25257

    • Google Scholar
23. Report

    1. NSF

    (2019a) Doctorate Recipients From US Universities: 2018

    National Science Foundation.

    https://ncses.nsf.gov/pubs/nsf20301/report

    • Google Scholar
24. Report

    1. NSF

    (2019b) Women, Minorities and Persons with Disabilities in Science and Engineering: 2019

    National Science Foundation.

    https://ncses.nsf.gov/pubs/nsf21321/report

    • Google Scholar
25. Report

    1. NUS

    (2011) Race for Equality

    National Union of Students.

    https://www.nusconnect.org.uk/resources/race-for-equality-a-report-on-the-experiences-of-black-students-in-further-and-higher-education-2011

    • Google Scholar
26. 1. Rath KA
    2. Peterfreund A
    3. Bayliss F
    4. Runquist E
    5. Simonis U

    (2012) Impact of supplemental instruction in entry-level chemistry courses at a mid-sized public university

    Journal of Chemical Education 89:449–455.

    https://doi.org/10.1021/ed100337a

    • Google Scholar
27. Report

    1. Reeves RV
    2. Halikias D

    (2017) Race Gaps in SAT Scores Highlight Inequality and Hinder Upward Mobility

    Brookings.

    http://www.brookings.edu/research/race-gaps-in-sat-scores-highlight-inequality-and-hinder-upward-mobility/

    • Google Scholar
28. Website

    1. Richardson H

    (2019) University counselling services 'inundated by stressed academics'

    Accessed April 26, 2021.

    https://www.bbc.co.uk/news/education-48353331
29. Report

    1. Singh G

    (2011) Black and Minority Ethnic (BME) Students' Participation in Higher Education: Improving Retention and Success

    Advance HE.

    https://www.advance-he.ac.uk/knowledge-hub/black-and-minority-ethnic-bme-students-participation-higher-education-improving

    • Google Scholar
30. Report

    1. Sosu E
    2. Ellis S

    (2014) Closing the Attainment Gap in Scottish Education

    Joseph Rowntree Foundation.

    http://www.jrf.org.uk/report/closing-attainment-gap-scottish-education

    • Google Scholar
31. 1. Stegers-Jager KM
    2. Brommet FN
    3. Themmen APN

    (2016) Ethnic and social disparities in different types of examinations in undergraduate pre-clinical training

    Advances in Health Sciences Education 21:1023–1046.

    https://doi.org/10.1007/s10459-016-9676-7

    • Google Scholar
32. 1. Tekian A
    2. Han Y
    3. Hruska L
    4. Krainik AJ

    (2001) Do underrepresented minority medical students differ from non-minority students in problem-solving ability?

    Teaching and Learning in Medicine 13:86–91.

    https://doi.org/10.1207/S15328015TLM1302_3

    • PubMed
    • Google Scholar
33. Website

    1. THE

    (2020) Top universities with the best student-to-staff ratio 2020

    Accessed April 26, 2021.

    http://www.timeshighereducation.com/student/best-universities/top-universities-best-student-staff-ratio
34. Website

    1. UCAS

    (2019) Statistical releases: daily clearing analysis 2019

    Accessed April 26, 2021.

    http://www.ucas.com/data-and-analysis/undergraduate-statistics-and-reports/daily-clearing-analysis-2020/statistical-releases-daily-clearing-analysis-2019
35. Report

    1. UCU

    (2016) UCU Workload Survey 2016

    University and College Union.

    https://www.ucu.org.uk/media/8196/Executive-summary-Workload-is-an-education-issue-UCU-workload-survey-report-2016/pdf/ucu_workloadsurvey_summary_jun16.pdf

    • Google Scholar
36. Website

    1. UK Government

    (2020) Ethnicity facts and figures

    UK Government. Accessed April 26, 2021.

    http://www.ethnicity-facts-figures.service.gov.uk
37. 1. Vaux DL

    (2012) Know when your numbers are significant

    Nature 492:180–181.

    https://doi.org/10.1038/492180a

    • Google Scholar
38. Report

    1. White E

    (2012) Ethnicity and National Identity in England and Wales: 2011

    Office for National Statistics.

    http://www.ons.gov.uk/peoplepopulationandcommunity/culturalidentity/ethnicity/articles/ethnicityandnationalidentityinenglandandwales/2012-12-11

    • Google Scholar
39. Report

    1. Williams P
    2. Bath S
    3. Arday J
    4. Lewis C

    (2019) The Broken Pipeline: Barriers to Black Students Accessing Research Council Funding

    Leading Routes.

    https://leadingroutes.org/mdocs-posts/the-broken-pipeline-barriers-to-black-students-accessing-research-council-funding

    • Google Scholar
40. 1. Woolf K
    2. Potts HW
    3. McManus IC

    (2011) Ethnicity and academic performance in UK trained doctors and medical students: systematic review and meta-analysis

    BMJ 342:d901.

    https://doi.org/10.1136/bmj.d901

    • PubMed
    • Google Scholar
41. 1. Woolf K
    2. McManus IC
    3. Potts HWW
    4. Dacre J

    (2013) The mediators of minority ethnic underperformance in final medical school examinations

    British Journal of Educational Psychology 83:135–159.

    https://doi.org/10.1111/j.2044-8279.2011.02060.x

    • Google Scholar
42. 1. Yeates P
    2. Woolf K
    3. Benbow E
    4. Davies B
    5. Boohan M
    6. Eva K

    (2017) A randomised trial of the influence of racial stereotype bias on examiners’ scores, feedback and recollections in undergraduate clinical exams

    BMC Medicine 15:179.

    https://doi.org/10.1186/s12916-017-0943-0

    • Google Scholar

Thank you for submitting your article "Alternative strategies to reduce the race-gap in undergraduate degree scores and career progression" to eLife for consideration as a Feature Article.

Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by two members of the eLife Features Team (Julia Deathridge and Peter Rodgers). The following individuals involved in review of your submission have agreed to reveal their identity: Tamsin Majerus (Reviewer #1); Kaushika Patel (Reviewer #2); Juliet Coates (Reviewer #3).

We have discussed the reviews and drafted this decision letter to help you prepare a revised submission.

Summary

This work provides a detailed analysis of the differences in grades awarded to white and Black, Asian or Minority Ethnic (BAME) students on a cell biology degree at a university in the UK. Unlike previous studies, this paper does not just look at the overall degree mark but at the individual assessments that make up the degree, and finds that exams contribute the most to the 'award-gap' in the degree studied. Although this finding may not be representative of all degrees, the methodology provides a framework that can be used to investigate the award-gap of other courses or programs, beyond the sciences, which may also have multiple exam and coursework components. The paper also provides general policy recommendations that will be of interest to leaders of Higher Education Institutes (HEIs) who are engaged in identifying and implementing strategies for closing the awarding gap. However, there are also a number of points that need to be addressed to make the article suitable for publication.

Essential revisions:

1) In the policy recommendations there is a strong emphasis on research funding and increased staffing. However, much research has already gone into 'why' the attainment-gap exists, and the recommendations should focus more on how HEIs can improve the experiences of BAME students. We, therefore, suggest removing policy 3, and adding the following recommendations to policy 2:

a) To spend more funding and resources on interrogating every process of the academy: the curriculum, appointment processes, assessments, the training delivered around racism (would probably fit into policy 2a).

b) To provide more training for academic staff so they are better at recognizing and calling out the toxic behaviours often experienced by BAME students, such as microaggressions. BAME students acutely feel a lack of similar role models in the academy and lack a sense of belonging. We, therefore, need to work on building supportive communities that really listen to the experiences of BAME students and collaborates with them on the process of making academia more inclusive so that barriers can be removed compassionately (Atkinson 2002).

c) Academic staff should also receive mandatory training in how to assess and grade work, avoiding implicit bias (techniques other than anonymising which is a good start, might include clear, specific rubrics and mark schemes). There is a reason that multiple-choice exams marked by machines tend not to show a bias and in part that is because the answer is either right or wrong, not subjective. This could be added to policy 2a.

2) For policy 4 to work, we need to find people who believe they will benefit from the change and put them in positions of power. Race is an uncomfortable subject for many who are unwilling (consciously or unconsciously) to confront their privilege. It might be worth adding a comment to that effect.

3) Please incorporate the following suggestions into policy 5:

a) For policy 5b to include removing the requirement for a 1st class degree to get a PhD place and to do shortlisting for recruitment based on anonymous CVs.

b) More resource needs to go into the training and diversity of recruitment panels.

c) Encouraging more BAME applicants will require more visible role models at higher levels but there is a worry around overburdening the minority, and this may be worth discussing.

4) Please link policy recommendations to pre-existing charter marks/professional awards. The two that are obvious are the Race Equality Charter (REC) and the Teaching Excellence Framework (TEF). A major step forward would be to require a REC award and clear evidence of action and plans to eliminate award gaps as fundamental criteria for the TEF.

5) Please remove the term 'race-gap' from the title and elsewhere in the text.

Using the term 'race-gap' is problematic for two reasons: first, it implies that there is a gap in race, not award (some places the text uses race award gap). The other issue is with the term 'race' itself. Mostly 'ethnicity' is used and this is a 'better' term. 'Race' has a problematic definition and its use simply muddies waters that need not be muddied! Please, therefore, stick to using 'ethnicity' and 'award-gap' consistently throughout the manuscript.

6) Please can you clarify the description of A level grades obtained by BAME and White students on the cell biology course (lines 233-236).

From Figure 1 A it seems as if the vast majority of students got one A* and 2As. Whereas the text (lines 233-236) suggests that students got a mix of A* and As. On the face of it then this could vary from 3A* grades to 3A grades (via A*A*A, A*AA). This leaves one question unanswered (except possibly by presumption that virtually all students regardless of ethnicity got A*AA) and that is was there a difference if the top grades are divided into the 5 possible combinations? It would help if this detail could be added/explained. This is important because the conclusion (lines 239-241) is only robust if this underlying assumption is correct.

7) In the literature review, please include more detail on how the awarding gaps for BAME students impact their 'academic' career progression. This will help strengthen the overall aim of the manuscript.

8) Please add the following to the Limitations section:

a) The methodology is clear and can be replicated but only where a combination of coursework and examinations make up the assessment components. If a degree, module, or specific subject within a degree only utilises one assessment type then the methodology will either not be relevant or will need adaptation.

b) One limitation of the study is perhaps that the cohorts, degree streams, and course selection is very complex as many different types of students take cell biology courses. Although the data is analysed appropriately using two different approaches, it means that any nuances around achievement for students on different degree programs (who may experience differently supportive learning environments) are masked.

c) The manuscript also aims to identify how the alternative strategy will look to reduce gaps in career progression. Although the outcomes of the study demonstrate the differences in awarding based on component assessments, it is not specific with regards to challenges for career progression for BAME graduates. The narratives around the impact of the awarding gap and the recommendations and policies do provide a direct link to how changing assessment methodologies could improve outcomes and in time career progression but this could be considered as a 'by-product' of the alternative strategy.

9) Another limitation is that the data analysis focuses on 'good honours', a first or 2.1 degree. We have carried out similar (unpublished) analyses at my institution that suggest that the 'first class' awarding gap is much bigger than that for good honours. Would it be possible to analyse firsts-only? The current analysis suggests that marking coursework identity-blind (largely) eliminates the awarding gap but (again based on my own data analyses) I wonder if looking at firsts-only would tell a different story.

If this is not possible, please include it as a limitation of the study in the revised manuscript and explain why you have decided not to address it in your point-by-point response to this letter.

10) To disaggregate the ethnicity data. "BAME" is a blanket term that includes groups in addition to people of colour. Moreover, previous data shows that Black students tend to fare worse than, for example, Asian students. Could you specifically look at the awarding gap for Black students?

If this is not possible, please include it as a limitation of the study in the revised manuscript and explain why you have decided not to address it in your point-by-point response to this letter.

Note: Here for your information are the original reviews, from which the above list of major comments have been taken.

Reviewer #1:

This is a very interesting and timely paper, discussing the important issue of the 'award gap' which is clearly an issue impacting on degree outcomes as well as career progression and opportunity. The paper provides welcome subject-level consideration and generalised policy recommendations to address the issues discussed. I absolutely think it should be published subject to the author addressing a small number of suggested corrections/amendments.Reviewer #2:

This manuscript will be of interest to all academics and leaders of Higher Education Institutes (HEIs) who are engaged in identifying and implementing a ranges of strategies for closing the awarding gap. The data presented in the manuscript clearly provides a basis for the full range of degrees who assess via both examination and coursework to consider the marking differentials between the two methods and their impact on either maintaining, worsening or closing the awarding gap.

The manuscript has the potential of significant impact across a range of degrees, beyond the sciences, due to its exploration of marks at component level and through the comparison of exams and coursework. It does provide a methodology which can be readily replicated across other disciplines and institutions.

A challenge to be considered is the emphasis within the recommendations (policies) around research funding and increased staffing. A great deal of insight has already been achieved with one of the key areas of focus landing on the levels of cultural competence and feelings of identity and belonging which need to be developed and improved within HEI's which although recognised within the recommendations does feature as less significant.

This manuscripts provides a clear overview of the 'race awarding' gap issue within western institutions and provides a clear outline of the range of research undertaken over a 10 year period with little progression in closing the race awarding gap.

The manuscript aims to identify alternative strategies in reducing the gap and certainly the methodology used, as outlined, does provide a different perspective in looking at assessment components. The methodology is clear and can be replicated but only where a combination of course work and examinations make up the assessment components. If a degree, module or specific subject within a degree only utilises one assessment type then the methodology will either not be relevant or will need adaptation.

The manuscript also aims to identify how the alternative strategy will look to reduce gaps in career progression. The outcomes of the study do demonstrate the differences in awarding based on component assessments but what the strategy does not do is be more specific with regards to challenges for career progression for BAME graduates. The narratives around the impact of the awarding gap and the recommendations and policies do provide a direct link to how changing assessment methodologies could improve outcomes and in time career progression but this could be considered as a 'by product' of the alternative strategy.

The HEI sector is currently becoming more aware of issues of racial inequality and the impetus generated by the Black Lives Matter movement both here and across the world has provided a spotlight on these racial inequalities in all sectors and particularly in higher education where the discrimination exacerbates overall life chances of a significant part of the university population. And so although the awarding/attainment gap has been researched, reviewed and action planned for many many years the BLM spotlight has provided a new energy and drive to address this significant inequality with a new and particular focus over the last few years away from the 'student deficit' model and more towards the 'institutional responsibility' model. So in this very current context this work will be particularly relevant in the wider 'race equality' context the sector finds itself in.

The manuscript would be strengthened to deliver on its aim if the connection to awarding gaps for BAME students and their 'academic' career progression were emphasised more in the literature review.Reviewer #3:

This paper provides a detailed breakdown of the awarding gap between BAME students and white students on a cell biology degree programme (with multiple pathways) at a single institution with a large BAME population. This detailed analysis shows that the biggest awarding gap lies in the exam component of the assessment and suggests some policies to close the gap going forward. There are some limitations to the analysis (should include first class marks as well as good honours) and suggested policies (e.g. vagueness around asking for funding) that I will address in detail in my review.In this study, the author examines the awarding gap that exists between BAME students and white students on cell biology degree programs. Unlike previous studies, the research does not just look at overall degree classification but examines individual assessments that make up the degree. The research concludes that it is specifically the exam component of assessment that contributes towards the awarding gap in Cell Biology degrees at UCL. Although these conclusions may not be representative of all programmes and institutions, the study provides a welcome framework to enable others to carry out this kind of detailed analysis if they have not started doing so already and shows that it is possible to carry out these analyses even in complex degree streams and pathways. From the data analysis, clear policy recommendations are put forward although I would caution against advocating more funding into research around 'why' the gap exists and instead repurpose such money for investing in proper resource to tackle the problem including by creating compassionate communities within institutions of BAME students working together with allies who have the power to instigate change.

Specific points – strengths, and issues to address, around data analysis.

1) The introduction sets the context for the pervasive awarding gap (and other factors that may contribute to it) extremely well and raises the issue of variability between subjects.

2) The data is collected from seven student cohorts across all three years of the degrees, including only students who completed the course. All marking for this dataset was carried out by machine or candidate identity-blind, which is a strength of the institution's assessment process as it eliminates human bias, conscious or unconscious.

3) One limitation of the study is perhaps that the cohorts, degree streams, and course selection is very complex as many different types of students take cell biology courses. Given that, the data is analysed appropriately using two different approaches, but that does mean that any nuances around achievement for students on different degree programs (who may experience differently supportive learning environments) is masked.

4) Another limitation is that the data analysis focuses on 'good honours', a first or 2.1 degree. We have carried out similar (unpublished) analyses at my institution that suggest that the 'first class' awarding gap is much bigger than that for good honours. Would it be possible to analyse firsts-only? The current analysis suggests that marking coursework identity-blind (largely) eliminates the awarding gap but (again based on my own data analyses) I wonder if looking at firsts-only would tell a different story. I think that this would be an important addition to the study.

5) I wonder if it is also possible (as this is a large dataset) to disaggregate the ethnicity data. "BAME" is a blanket term that includes groups in addition to people of colour. Moreover, previous data shows that Black students tend to fare worse than, for example, Asian students. Could we specifically look at the awarding gap for Black students is the largest?

6) A thought about the statistical analysis: a t-test is not appropriate for non-parametric (%) data unless normality has been tested for or a correction has been applied. Has this been done? It is not clear from the methods.

7) Showing figure 2c as continuous data points joined by a line is unclear: it would be more consistent to use bars as in Figure

Some thoughts about the discussion and policy recommendations.

I absolutely agree with the suggestions of policy 1 and policy 2. However, Policy 3 needs to be careful and specific otherwise it is not convincing to "Increase Direct Funding to Research the Factors that Cause the Award Gap", in fact I would be tempted to remove this section and expand Policy 2 as outlined in the next couple of paragraphs.

Much "why" research has already been carried out over the last 30 years that all points in the same direction: there are huge systemic, structural barriers that are put in place by people of privilege and power that disadvantage BAME students in education and the problems start before University, although are reinforced in higher education. It is the 'business as usual' approach has to change as this is what causes the problem (see, for example, much of David Gillborn's research including Gillborn et al. (2017)). Racism is subtle, common and can be well intentioned/paternalistic ('we know best how to fix you'). Funding and resource would be better spent on Policy 2 and in addition an interrogation of every process of the academy: the curriculum, appointment processes, assessments, the training delivered around racism – this would probably fit under Policy 2a.

More thoughts on Policy 2a: We need to resource local support that involves the relevant students. Building supportive communities is key. We need to really listen to student experiences of all the academy's processes and involve students in the process of change as collaborators so that barriers can be removed compassionately (Atkinson 2002). My own anecdotal listening plus institutional training shows that BAME students are subject to daily micro-aggressions and more training is needed to increase the number of allies who are confident to call out toxic behaviour. BAME students acutely feel a lack of similar role models in the academy and lack a sense of belonging. Thus, changing recruitment processes is a key step (see policy 5 below).

For policy 4 to work, we need to find people who believe they will benefit from the change and put them in positions of power. Race is an uncomfortable subject for many who are unwilling (consciously or unconsciously) to confront their privilege. It might be worth adding a comment to that effect.

Policy 5: I absolutely agree. I would advocate for policy 5b to include removing the requirement for a 1st class degree to get a PhD place (this is partly why I made my earlier point about extending the data analysis beyond good honours if at all possible) and to do shortlisting for recruitment based on anonymous CVs. More resource needs to go into the training and diversity of recruitment panels. Encouraging more BAME applicants will require more visible role models at higher levels but there is a worry around over burdening the minority with this work perhaps this could be discussed a little too.

References:

Atkinson J (2002) Trauma Trails, recreating song lines: The transgenerational effects of trauma in indigenous Australia p.13-17 (Spinifex Press).

Gillborn D et al. (2017) Moving the goalposts: Education policy and 25 years of the Black/White achievement gap. British Educational Research Journal 43 p.848-874.

[Editor’s note: Further changes were made following revisions to the manuscript in response to queries and discussions with the editor. As a result, some of the comments below may differ from the published version of the article.]

Essential revisions:

1) In the policy recommendations there is a strong emphasis on research funding and increased staffing. However, much research has already gone into 'why' the attainment-gap exists, and the recommendations should focus more on how HEIs can improve the experiences of BAME students. We, therefore, suggest removing policy 3, and adding the following recommendations to policy 2:

a) To spend more funding and resources on interrogating every process of the academy: the curriculum, appointment processes, assessments, the training delivered around racism (would probably fit into policy 2a).

b) To provide more training for academic staff so they are better at recognizing and calling out the toxic behaviours often experienced by BAME students, such as microaggressions. BAME students acutely feel a lack of similar role models in the academy and lack a sense of belonging. We, therefore, need to work on building supportive communities that really listen to the experiences of BAME students and collaborates with them on the process of making academia more inclusive so that barriers can be removed compassionately (Atkinson 2002).

c) Academic staff should also receive mandatory training in how to assess and grade work, avoiding implicit bias (techniques other than anonymising which is a good start, might include clear, specific rubrics and mark schemes). There is a reason that multiple-choice exams marked by machines tend not to show a bias and in part that is because the answer is either right or wrong, not subjective. This could be added to policy 2a.

As suggested these reviewer recommendations around funding (1a), resources (1a) and training (1a-c) have been incorporated into extended Policy 1 (old Policy 2) and elsewhere: addition to Policy 4; new Figure 4; within the main conclusion. In the original manuscript previous recommendations on how to close the award-gap were restricted as a whole to citations of previous studies to enable the current manuscript to focus on new ideas. However, the point is well taken that may have inadvertently given less space to a key anchor “Widespread or sustained delivery of the reported solutions can’t be working sufficiently then…” which is now addressed.

Of note, a critical objective in the revised manuscript was to fully meet these reviewer suggestions on previous recommendations, but also to ensure the manuscript simultaneously remained focused on addressing “why the award-gap is only closing slowly”.

This aim seems met by providing a summary of all previous recommendations distributed by category (new Figure 4). This was achieved by careful assessment and quantification of three, major substantive reports representing the past decade (nearly 200 recommendations). In addition, it was thought important to capture substantive previous work in a balanced way (to represent across all ideas as well as the reviewer ones suggested) and this is reflected in Figure 4.

New Figure 4 explicitly includes the category suggested by the reviewer (comment 1, improve the experiences of BAME students).

In new Figure 4, legend non-exhaustive examples of the top three categories of previous recommendations (infrastructure, student experience, teaching and learning) are listed. The reviewer suggestions to provide resources and training (reviewer comments 1a-c), are included as important examples within this list, and specifically all the ones listed by the reviewer in 1a-c.

The reviewer comment, “Spend more funding…” (start of comment 1) is taken to mean invest in existing recommendations to ensure resources and training (comment 1a-c) is implemented, is now an explicit part of the revised version: raised as an issue in the introduction to the policy section; in the title of Policy 1; in the introduction to Policy 1; in the content of new Policy 1d; and in the title and content of modified Policy 4.

Responding to the other part of reviewer comment 1 “…much research has already gone into 'why' the attainment-gap exists…” and “We, therefore, suggest removing policy 3,..”.

This comment is understood; there has been a lot of research and these were cited for example in the original manuscript and now stated more directly in the revised version “At first glance it may appear there is plenty of evidence on causes of the award-gap… However, on closer inspection there seem two large gaps in the evidence base.”

In retrospect perhaps, more details on how these evidence-gaps were originally identified and more clarity on the concept of ‘unexplained’ ethnic award-gap would have been helpful and both these are now expressed more clearly in the revised version.

Unexplained ethnic degree award-gap: Previous research (cited in the introduction) tested >20 different external factors (for example, poverty, age, study habits, type of previous school) as possible explanations for why the ethnic degree award-gap exists, but found that a large gap remained; this larger gap is known as the unexplained award-gap. It is thought to arise through being at university and is the one that universities have been charged to remove and the focus of the manuscript. This is now made more explicit in the introduction (literature review) of the revised version and in the introduction to the policy section.

Gaps in the evidence base: it is now made explicit, that gaps in the evidence are around those that seek causes of the unexplained (university-driven) award-gap: in the introduction to the policy section; in the title of new Policy 2 (old Policy 3); in the titles of new Figures 4 and 5. These identified gaps in the evidence are limited diverse types of evidence (in particular quantitative as defined by the raw data is numerical) and limited scientific approaches as defined by the principle of conducting a ‘fair test’; both these are reflected in the new title to Policy 2.

How limited diverse types of evidence and limited scientific approaches were identified:

i. Limited quantitative approaches were identified by careful and thorough mapping the types of evidence that underlies ~200 recommendations, representative of the past decade (new Figure 5), which have previously suggested to universities how to reduce the unexplained award-gap (new Figure 4).

Please note, it is acknowledged in the revised manuscript that “Clearly, much has been quantified, particularly the size of award-gaps, however to date this has almost exclusively been used as evidence that the problem exists, as a benchmark or as attempts to identify the size (if any) of a number of potential, contributory external factors […] Yet, these do not in themselves identify the larger, university-driven (unexplained) causes of the gap.”

Note it is also acknowledged in the revision, that whilst this investigation has identified that quantitative approaches seem under-represented in seeking causes of the unexplained gap (it was found to be only ~0.5% of the evidence base), it is also made plain in the text to Policy 2, that the existing main evidence (qualitative: as defined by the raw data is dialogue with students and staff) are critical too. The point is that to close the unexplained gap faster (one of the main aims of the paper) likely requires a balance in quantitative and qualitative approaches.

ii. When this research was started, there did not seem many other scientific approaches in the literature. This was also recently observed by a large collaboration between the National Union of Students, Universities UK and 99 HEIs. Baroness Amos and Amatey Doku conclude “Universities need to take a far more scientific approach to tackling the attainment gap…” as one of the top 5 outstanding actions required to successfully reduce the award-gap (page 2 in Amos and Doku 2019).

There are several benefits to increasing the diversity of approaches and scientific evidence: they have the potential of contributing to reducing the award-gap in ways less amenable by other methods; one example is the findings in this paper (Figure 1-3) another is facilitating impact-evidence on what works, and equally important can guide prioritisation of allocation of valuable resources (for further details, please see Policy 2).

This manuscript might contribute to awareness that these gaps in the evidence base seem to exist; but practically, funding is required to address it. This is highlighted in modified original Policy 3 and it is then suggested that creation of a funding alliance could generate the necessary funds in Policy 4.

Note, it is also suggested in the revised version that this alliance could also be used to fund the implementation of existing recommendations and resources (as raised by the reviewer in comment 1) and scientifically test which ones work.

2) For policy 4 to work, we need to find people who believe they will benefit from the change and put them in positions of power. Race is an uncomfortable subject for many who are unwilling (consciously or unconsciously) to confront their privilege. It might be worth adding a comment to that effect.

Absolutely. In the original manuscript it was proposed that the Nuffield Foundation house such an alliance, exactly for this reason. The revised text now makes plain this point, “Strong leaders who believe what society has to gain from such an alliance are required for success; it is proposed then to take advantage of a pre-existing structure such as the UK’s Nuffield Foundation, which already seems to demonstrate such a belief both in its strategy document 2017-2021 and it its funding of projects widely across education, welfare and justice; their housing of the suggested alliance may speed up its creation.”

3) Please incorporate the following suggestions into policy 5:

a) For policy 5b to include removing the requirement for a 1st class degree to get a PhD place.

This was in the original text, it is now rephrased to make clearer: “…have removed the highest undergraduate degree class (1^st class) as advantage for selection for interview or award of a PhD place (LIDo)”.

And to do shortlisting for recruitment based on anonymous CVs.

This is an important point and has been attempted by some PhD programmes. To address the reviewer’s point, a sentence has been added to policy 5b: “It is also suggested that candidates’ names are removed from CVs and LIDo have had partial success with anonymous CVs; the main issue remaining is how to ensure 100% of referees use applicant number instead of name in their letters of recommendation.”

b) More resource needs to go into the training and diversity of recruitment panels.

This is now added to a new section in policy 5b:

“(iv) In part, the success of these three programmes is that there seems specific training of recruitment panels and at least for some, a common rubic for selection of applicants. Diversity on selection panels is an ongoing concern – it directly relates to the high loss of potential senior academics early in the pipeline to professor (detailed in the introduction) and some universities (including UCL) are implementing initiatives to increase representation of minority ethnic staff on panels.”

c) Encouraging more BAME applicants will require more visible role models at higher levels but there is a worry around overburdening the minority, and this may be worth discussing.

Role models seem inherently part of the success of policy 5bii (establishing relationships between highly competitive PhD programmes and HEIs where minority ethnic undergraduate students are more likely to attend) and appear to work without any sense of overburden. At UCSF, minority (and other) ethnic doctorate students have the opportunity to visit minority serving colleges, some of which were their own undergraduate colleges to reinforce the cycle of success in informal ambassador type roles. At Bridge, there is a cycle of visible mentorship which seems to function in a similar way. At LIDo PhD students serve as ambassadors at interview days and mentor undergraduate students on summer placement schemes, whom are recruited mainly from lower tariff universities. Space limitations probably preclude these details, however the point is noted and encompassed in additional text: “An added benefit of these schemes is that they incorporate opportunities for current minority ethnic doctorate students on the PhD programmes to act as role models in such a way that also contributes to their own career development.”

4) Please link policy recommendations to pre-existing charter marks/professional awards. The two that are obvious are the Race Equality Charter (REC) and the Teaching Excellence Framework (TEF). A major step forward would be to require a REC award and clear evidence of action and plans to eliminate award gaps as fundamental criteria for the TEF.

There is some hesitation here from the experience of working on the Athena Swan (AS) charter marks (our department was one of the first in the UK to receive Gold in 2016 and was awarded Gold again in 2020). What seems to have been discussed at AdvanceHE (who run the AS and RE charter marks) is the acknowledgement of substantial and disproportionate burden on female staff engaging in the AS charter mark work (which is very high). Without explicitly linking contributions to these charter marks to promotion criteria, may result in similar disproportionate overburden for minority ethnic staff working on the REC/TEF. Some (if not all) HEIs include an expectation of citizenship in promotion applications, however there seems variability – both to the weighting/clarity of how much that work counts to promotion and that some HEIs do not appear to permit promotion on contribution to citizenship alone.

Another possible idea is to link the REC as a requirement to grant applications. However, that too has now changed; whereas a silver AS award was required, some participating grant award bodies have now (reported locally in 2020) withdrawn the requirement, no reason was given; possibly connected to disproportionate load or other reasons.

The reviewer suggestion could work very well (gains in reducing the gender professor-gap may have come from similar links in the AS charter marks), however space would be required in the manuscript to discuss these other significant factors.

(Somewhat related, however, the REC and fellow of HEA awards are now linked to a point in Policy 2).

5) Please remove the term 'race-gap' from the title and elsewhere in the text.

Using the term 'race-gap' is problematic for two reasons: first, it implies that there is a gap in race, not award (some places the text uses race award gap). The other issue is with the term 'race' itself. Mostly 'ethnicity' is used and this is a 'better' term. 'Race' has a problematic definition and its use simply muddies waters that need not be muddied! Please, therefore, stick to using 'ethnicity' and 'award-gap' consistently throughout the manuscript.

Ethnic is used throughout the manuscript, other than where it is a title such as Race Equality Charter.

6) Please can you clarify the description of A level grades obtained by BAME and White students on the cell biology course (lines 233-236).

From Figure 1 A it seems as if the vast majority of students got one A* and 2As.

Indeed, the average tariff was the equivalent of A*AA for almost all students. This is now directly shown in new Figure 1A. To be able to determine the average entrance score for each individual student, grades were converted to the UK standard tariff of: A* = 56; A = 48; B = 40 (no student obtained lower than a B), then the cohort mean determined:

“The mean admission tariff for these BAME (50.53+/- 0.15) and White (50.29+/- 0.33) students were virtually identical (Figure 1A): the equivalent of nearly all students in each of these ethnic groupings gaining one A* grade and two A grades (which would be a mean tariff of 50.7). These data are very similar to the mean entrance tariff for all BAME (50.27 +/-0.06) and all white (50.03 +/- 0.01) students at UCL who enter with A levels for the same cohort years as these cell biology students (2013, 2014, 2015 entrants), thus in this context cell biology students in this study are typical for the university.”

Whereas the text (lines 233-236) suggests that students got a mix of A* and As. On the face of it then this could vary from 3A* grades to 3A grades (via A*A*A, A*AA).

It is realised that the description here could have been confusing. What was quantified is the total number of each grade to each cohort (in other words what was the total number of A grades awarded to all the BAME students entering in 2013, then in 2014, 2015 and repeated for all the different grades). The text is now rephrased and the data is now in Figure 1B, as:

“An alternative approach to assess admission qualifications is to look at the distribution of the total number of each individual grade (A*, A, B, C or lower) awarded across each cohort: as expected distributions for BAME and White students were nearly identical (Figure 1B); of the total grades awarded to BAME students 95.45% +/- 1.01 were A* or A, and to White students this figure was 95.02% +/- 0.61 (Figure 1C).”

This leaves one question unanswered (except possibly by presumption that virtually all students regardless of ethnicity got A*AA) and that is was there a difference if the top grades are divided into the 5 possible combinations? It would help if this detail could be added/explained. This is important because the conclusion (lines 239-241) is only robust if this underlying assumption is correct.

A*AA was the equivalent tariff for virtually all students; please see new data in new Figure 1A, described above.

7) In the literature review, please include more detail on how the awarding gaps for BAME students impact their 'academic' career progression. This will help strengthen the overall aim of the manuscript.

More details have been provided, in particular: how loss of potential minority ethnic professors is directly linked to less competitive undergraduate degree scores, is more clearly explained; and more details on how selection criteria for PhD positions (which create barriers for minority ethnic students) is directly linked to data and suggested actions in the paper. Other changes to the title, abstract and introduction to the policy section also help strengthen the overall aim of the manuscript in this context.

8) Please add the following to the Limitations section:

a) The methodology is clear and can be replicated but only where a combination of coursework and examinations make up the assessment components. If a degree, module, or specific subject within a degree only utilises one assessment type then the methodology will either not be relevant or will need adaptation.

The limitations section has been modified to incorporate the reviewer comment:

“The study is restricted to certain courses at one university. The wider applicability of the written exam and coursework findings, require testing in cell biology in other universities and in other subjects across institutions. Most degree subjects (if not all) are composed of different types of activities and assessed components, the method described here is also envisaged to be readily adaptable to look systematically at whichever ones are relevant to any particular subject.”

b) One limitation of the study is perhaps that the cohorts, degree streams, and course selection is very complex as many different types of students take cell biology courses. Although the data is analysed appropriately using two different approaches, it means that any nuances around achievement for students on different degree programs (who may experience differently supportive learning environments) are masked.

This good point seems more appropriate for the Discussion section as relative award-gap sizes are not limited when comparing the exam and coursework components to each other; the same student on their particular degree pathway would be in the same learning environment for both components. However different supportive environments may explain relative gap-sizes when comparing the same component-type between year-1, -2 and year-3 cell biology courses. This possibility has been added to other possibilities in the discussion that might explain relative gap-size in this context:

“It is also not yet possible to distinguish whether precise size of the exam and coursework award-gaps comparing the same component across the different year-1, -2 and year-3 cell biology courses (observable in Figure 2E, compare years; and perhaps also Figure 2D) is a manifestation of a difference in the precise curriculum activity on that course or how it is delivered or for example, other factors such as supportive environments that students may encounter differently on different degree routes or several of these factors combined.”

c) The manuscript also aims to identify how the alternative strategy will look to reduce gaps in career progression. Although the outcomes of the study demonstrate the differences in awarding based on component assessments, it is not specific with regards to challenges for career progression for BAME graduates. The narratives around the impact of the awarding gap and the recommendations and policies do provide a direct link to how changing assessment methodologies could improve outcomes and in time career progression but this could be considered as a 'by-product' of the alternative strategy.

This has been addressed by clarifying the aims of the paper in the title, abstract, introduction and policy section and more clearly describing direct links between the award-gap and academic progression (as explained in response to comment 7). In brief, one of the main purposes of the paper is to provide an account of ‘what works’ to increase academic career progression and to convey the concept of the need for parallel action. In other words, careers matter now; hence it is suggested to i) address the award-gap by asking what could speed progress which, in time would then directly contribute to career progression, as noted by the reviewer; and ii) whilst the award-gap still exists also take parallel (separate) action on addressing the career-gap directly. In the USA funding schemes to promote academic career progression for example seem more widely developed and discussed (cited in the paper).

Please find an extract from the revised introduction, as an example of how these comments have been addressed: “Then by focusing on asking the question of why the UK ethnic award-gap is only shrinking very slowly, some likely causes of the slow pace are identified. […] Given this award-gap directly reduces chance of career progression, dialogue with PhD programme organisers or analysis of published initiatives that have successfully increased proportion and retention of under-represented, minority ethnic doctorial students are also presented.”

9) Another limitation is that the data analysis focuses on 'good honours', a first or 2.1 degree. We have carried out similar (unpublished) analyses at my institution that suggest that the 'first class' awarding gap is much bigger than that for good honours. Would it be possible to analyse firsts-only? The current analysis suggests that marking coursework identity-blind (largely) eliminates the awarding gap but (again based on my own data analyses) I wonder if looking at firsts-only would tell a different story.

If this is not possible, please include it as a limitation of the study in the revised manuscript and explain why you have decided not to address it in your point-by-point response to this letter.

This has been addressed with a substantial new body of work in the year 2 and year 3 cell biology courses (Figure 3 and Figure 3—figure supplement 1). The possibility of potentially hidden larger, coursework award-gaps was tested in the original manuscript, however a considerably deeper and wider test going beyond 1^st class grades, seeking any hidden gaps in the entire range of marks awarded from <30% to 100% in 5% intervals was performed and no larger gap in the coursework could be found (Figure 3, compare histograms).

As expected from these frequency distributions, the coursework award-gap remains small in either the analysis of 1^st class marks-only or all good marks combined and at least 2-fold smaller than the exam award-gap (which is also smaller at 1^st class marks-only) and the coursework award-gap at 1^st class-marks is not significant (Figure 3—figure supplement 1; Figure 3—figure supplement 1-Source data 1).

It was not feasible to fully assess year 1-cell biology though there was also no significant gap in the coursework in 1^st class marks in the available data and this is explained in the limitation of the study.

Overall, these new analyses and the original ones strengthen the view that the award-gap in these 6 cell biology courses is largely derived from the exam.

It was not possible to compare these new data to the unpublished work the reviewer mentioned, however, the UK average is that the award-gap at the level of 1^st class marks-only is smaller than at all good marks (AdvanceHE 2019b) which is consistent with the findings in these cell biology courses (above) (Figure 3 and Figure 3—figure supplement 1). None-the-less an assessment of other UK statistics (also in AdvanceHE 2019b) and other data at UCL, both looking at a finer grain reveals that there are at least 5 different patterns evident on the relative contribution from 1^st class marks and 2i marks to the overall ethnic award-gap in good grades – and one of these other patterns might be consistent with the unpublished data mentioned by the reviewer. This might be expected when looking at a finer grain (and this point and the existence of these 5 other patterns is acknowledged in the new version of the manuscript).

10) To disaggregate the ethnicity data. "BAME" is a blanket term that includes groups in addition to people of colour. Moreover, previous data shows that Black students tend to fare worse than, for example, Asian students. Could you specifically look at the awarding gap for Black students?

If this is not possible, please include it as a limitation of the study in the revised manuscript and explain why you have decided not to address it in your point-by-point response to this letter.

3% of the population has a Black ethnic background (2011 Census) similar to the ~2% of the cell biology students in this study. The study-size would therefore have to be far larger to be able to assess Black ethnic students-only. Some preliminary findings are reported in the limitation section of the paper:

“Preliminary data, which is consistent with the UK pattern, looking at year-3 cell biology courses, the proportion of Black students awarded good grades for the final course mark (~38%) and for the exam (~38%) was up to 2-times lower than all minority ethnic students (BAME) for these two components (72.7% +/- 3.7 and 64.1% +/- 2.6 respectively). However, the proportion of Black students awarded good grades for cell biology coursework (~87%) was at least equally as high as all BAME students (82.1% +/- 2.0) and White students (87.5% +/- 1.5).”

Author details

1. Louise Cramer

   Louise Cramer is in the MRC Laboratory for Molecular Cell Biology and affiliated with the Department of Cell and Developmental Biology, University College London, London, United Kingdom

   Contribution

   Conceptualization, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   l.cramer@ucl.ac.uk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3502-0434

• 4,996

  views

• 348

  downloads

• 6

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.