#159 2015 · Avon and Somerset Police / Behavioural Insights Team · Public sector recruitment / assessment designenter-their-reality
The test wasn't biased on paper, so the police force stopped looking at the test and looked at what happened right before it
the problem
A selection test showed a stark pass-rate gap between groups that content review of the test itself couldn't explain
background
Avon and Somerset Police's situational judgement test, part of its officer recruitment process, was producing a large pass-rate gap between Black and minority ethnic (BME) candidates and other candidates — roughly 30% versus roughly 65% by one account — even though the test's content had been reviewed and showed no overt bias in its questions or scoring. A test that looks fair on paper can still be shaped by psychological factors that have nothing to do with its content.
The Behavioural Insights Team, working with the force, hypothesized the gap was driven by stereotype threat — the well-documented effect where awareness of a negative stereotype about one's group can itself depress performance on a high-stakes evaluation, independent of actual ability. Standard fixes for test bias, like revising question wording or scoring rubrics, would do nothing to address a psychological effect happening before a candidate ever reads the first question.
the move
Rather than change the test itself, the team added two brief additional questions to the email invitation candidates received before sitting the test, asking them to reflect on what would make them a good addition to the police force and what significance that would have in their community — a short self-affirming reflection designed to counter stereotype threat before the assessment began.
the payoff
BME candidates who received the priming questions passed the situational judgement test at rates comparable to non-BME candidates, roughly closing a gap that had previously run from about 30% up to parity around 65%, without any change to the test's content, difficulty, or scoring.
what came after
The intervention is cited in UK public-sector behavioural science circles as a demonstration that fairness problems in assessment can originate in candidates' psychological framing rather than in test design, and the Behavioural Insights Team has since promoted similar priming-based approaches to other recruitment and assessment contexts facing unexplained demographic performance gaps.
references
- [1]Minorities remain missing from officer roles, but behavioural economics may come to the rescueCapX, 2023capx.co
- [2]Better, fairer recruitment with behavioural scienceMoreThanNow, 2018morethannow.co.uk