Are code tests undermining inclusive hiring?

Last month we shared a number of programs as part of our commitment to Black software engineers. One of these initiatives is to use our interview data and research to develop new formats that reduce bias. Today’s research shows that the emphasis code tests place on absolute completeness and correctness is failing to produce a quality hiring signal. Code tests also disproportionately exclude underrepresented minority groups by “failing” qualified software engineers that could receive offers based on live technical interviews.

“We’re getting candidates onsite that can’t code.”

This is something we often hear from our partners before they start working with us. They’re experiencing a lack of signal. These false positives are especially common across interviewing and hiring processes that start with code tests. It is a drain on the productivity of engineers who dedicate their time to interviewing their future coworkers.

Another unfortunate outcome of code tests is the false negative. This results in the rejection of potentially strong candidates at the top of the funnel.

A few weeks ago we shared data that revealed code tests’ requirements of absolute completeness and correctness could be filtering out as many as 1 in 3 developer hires. Conversely, hiring processes that start with a live technical interview show 55% of offers go to candidates with incomplete solutions. Live technical interviews represent a real-work environment and support candidates in a way that code tests, by their nature of being a solo-effort with a pass/fail outcome, cannot.

Digging deeper, the data showed that reliance on absolute completeness and correctness disproportionately impacts underrepresented minorities and women. This should be a red flag for every organization that aims to be inclusive, diverse, and free of systemic barriers to hiring software engineers.

As we enter University Recruiting season, companies are building diverse talent pipelines from HBCUs, HSIs, Grace Hopper Conference, and organizations like the National Society of Black Engineers. Now is the time to ask: Are code tests preventing my hiring process from being as inclusive as it ought to be?

Over indexing on completeness produces false negatives

The incompleteness of a solution doesn’t mean a candidate can’t code. Over the course of a 60-minute technical interview, candidates have opportunities to demonstrate problem-solving and coding skills relevant to a role and hiring bar in a variety of ways. Less than a third of candidates complete all three problems in the Karat technical interview.

Completeness is just one factor that needs to be considered, but in the context of a code test, it becomes a binary pass/fail element. In live interviews, more than half of all job offers go to candidates with incomplete solutions. For URMs and women, that is slightly higher at 57% and 59% respectively.

*For analysis purposes, URM candidates include those who identify as Black or African American, Hispanic or Latino, Native Hawaiian or Pacific Islander, and Two or More Races

Excluding candidates who receive guidance in technical interviews disproportionately affects URMs

Who hasn’t had a simple spelling mistake while coding? Or gotten stuck while solving a problem simply because, let’s face it, interviews tend to be nerve-wracking? Code tests are solo-efforts. They don’t afford candidates the opportunity to ask clarifying questions and receive guidance from an engineer experienced in conducting interviews. Yet, 19% of job offers to women started with a technical interview that included some form of guidance, as did 23% of job offers to URMs.

Reduce the false negatives of code tests and improve team diversity

There is no silver bullet for building an inclusive hiring process. It takes soul-searching and hard analysis to understand why you may not be achieving diversity goals. The answer may be very hard to accept. But diverse software engineering talent does exist, and by taking a more holistic approach to live interviews, your hiring team can achieve a more complete picture of each candidate beyond pass/fail.

In the short term, you can analyze your hiring process and look for places where URMs and women may be falling through the cracks due to false negatives and intentionally give them a bridge to cross them. Evaluating your hiring process by examining how you build your talent pipeline and how you evaluate the skills of future software engineers is a good place to start.