Think you're good at conducting technical interviews?

We’ve all been talking about fairness and bias a lot these past few months, and the depth at which it’s currently being discussed is long overdue. At Karat, we have been conducting research to identify sources of bias, inconsistency, and other challenges that can creep into technical interviews. This includes the Interview Gap survey that we released last month.

We’ve covered the high-level findings in previous blogs, but one interesting trend that stood out to me was the difference between responses based on seniority–in particular, how rapidly the C-suite has shifted its perceptions, exposing some differences between them and the VP, Director, and Manager levels.

Do you see eye to eye with your boss?

The good news is that your C-suite is happy with your performance. 71% of C-level leaders are “very satisfied” with the performance of their software engineers, compared to just 37% other engineering leaders.

The bad news–especially as you are looking to grow as organizations emerge from the economic downturn–is that they think you suck at conducting interviews. 84% of C-level agree or strongly agree that “very few people at my company know how to conduct a technical interview,” compared to just 56% of managers.

Why the difference?

I’ve conducted a lot of technical interviews. I used to think I was pretty good at them. I’m personable. I can remember questions I’ve been asked previously (hit me up if you feel like counting the number of lowercase a’s in a null-terminated string?), and I treat all of my candidates fairly. But when we’re talking about systemic issues of bias, or functional issues of predictiveness, we need to look at the full system, not just our individual opinions. And that’s something the C-suite does.

Do you have a sense of how your interviews impact the company’s bottom line? Do you know how many interviews your company conducts per open software engineering role? Do you track detailed hiring funnel metrics?

The answer for the majority of us, is “no,” or at least “not to the extent of my C-suite.”

• Nearly half of C-level executives (47%) track detailed hiring funnel metrics like the number of interviews, candidates, and onsite to offer ratios compared to just 25% of managers.
• This means that 75% of engineering managers, don’t understand the inner workings of their own hiring system.

Think you know how many interviews you conduct per open SWE role?

• Managers say they conduct an average of 17.9 technical interviews per SWE hire.
• C-level execs, who are looking at fuller data sets and hiring funnel metrics, say that number is 24.7.

So it’s no surprise that 79% of C-level execs agree that “Interviewing software engineering candidates is a financial drain on my company” compared to just. 50% of VPs, directors, and managers.

Planning to ship some major features this fall during the thick of the upcoming university recruiting season?

• 55% of C-level leaders recognized that “reaching hiring targets while delivering product features is challenging,” compared to just 32% across other levels.

How can we get better at technical interviews?

There are three areas that organizations can focus on to right their interviewing ships: capacity, consistency, and measurement.

• Capacity constraints make it difficult to conduct enough interviews to reach hiring targets. 61% of C-level execs agreed that “reaching hiring targets would be easier if we had more people qualified to interview” compared to 37% of VPs, directors, and managers. You can increase capacity by training more engineers to conduct interviews.
• Consistency is another way to improve your hiring program. 55% of the C-suite strongly agree that “the typical technical interview fails to predict the performance of software engineers,” compared to 24% of the rest of us. Create a structured interview process that includes dedicated interviewers and performance reviews. Use standardized, battle-tested questions to evaluate specific competencies. 85% of engineering leaders who were very confident in reaching hiring targets used a structured process compared to just 73% of less-confident leaders.
• Measurement is critical to reducing bias. Tracking detailed hiring funnel metrics is something that 73% of companies that are “very satisfied” with their engineers’ performance do, compared to just 57% of other organizations. In addition to looking across the entire funnel, use a structured scoring rubric for every interview to reduce opportunities for individual interviewer bias to affect your hiring process. One of the most common questions I get at events and meetups is about our structured scoring rubric. Here’s a quick guide on how to create a structured scoring rubric.

If you’re interested in other perspectives on hiring software engineers, here’s a great Q&A from our partners at Plato you can check out.