It’s not uncommon for us to start working with a new client and discover that onsite-to-offer ratios are in the 5:1, 10:1, or even 15:1 range.
For engineering teams, high onsite-to-offer ratios represent a massive tax. Having a 10:1 ratio means that for every new hire, you’re conducting 10 onsite interview loops, each with somewhere between 3-8 people.
That doesn’t even factor in the time spent on technical interviews, phone screens, and candidate sourcing. At that rate, it’s easy to imagine spending well over 100 hours of engineering time per engineering hire, which is an unfair request for software engineers who are already overworked to the point of needing to hire more colleagues.
But what should an onsite-to-offer ratio be?
The exact number depends on the particulars of your market, geography, recruiting and role requirements, but anything over 15:1 sounds unnecessarily high. To start with, if your ratio is over 10:1, try cutting it in half, or even aim for 5:1.
Consider the following steps to improve onsite-to-offer ratios
Look at performance reviews
Performance reviews, improvement plans, and promotion requirements all tell you what actually gets assessed for a particular role. What are the common reasons employees succeed or struggle?
Identify the skills and experiences that are not only most relevant to the role but also important when making a hiring decision. Cross off knowledge or skills that can be learned on the job. Prioritize the competencies that remain, and then build interviews and a decision-making process around the top items. This is your signal.
Look at onsite notes and assessment rubrics
Before changing your process too much, assess the consistency and predictability of your current process.
Characterize the reasons candidates are rejected. Evaluate the relevance and importance of the skill/knowledge/experience behind each reason. In other words, is each competency contributing signal or noise to the hiring decision?
On interview question guides and assessment rubrics, explicitly clarify what is considered noise and what is considered signal. Update how questions are asked and how observations are recorded to make it easier for interviewers to succeed in this task.
Break the onsite-to-offer decision-making process down into four steps
Improve your current onsite process by making each step in the interviewing process more intentional.
- Conducting an interview that gives the candidate the best opportunity to present relevant and important competencies (aka signal). Question guides and communication techniques play a large role here.
- Correctly observing and recording this signal. Question guides and assessment rubrics play a large role here.
- Making a yes/no interview decision based on the specific objectives and bar of this particular interview. A good way to know if this part of the process needs work is if interviewers are coming to debrief meetings confused, uncertain, or wishing they had more time or had asked different questions. After filling out an assessment, interviewers should feel confident about the yes/no result of their conduct and observations.
- Making a yes/no hiring decision based on the results of multiple interviews. You may find that you have multiple “successful engineering manager” personas and that this decision-making algorithm is not simply a single bar across all competencies. Whatever the algorithm is, it should be written down as an explicit guiding force. Hiring decisions are necessarily complex human decisions, but data and guidelines can increase the intentionality, consistency, and quality of these decisions at scale.
Illuminate signal earlier
Once you understand why candidates are being rejected, assess that earlier in the process. For example, if the System Design interview is a common cause of onsite rejection, move some system design questions into an earlier gate.
While you could move the System Design interview into a new gate wholesale, often you’ll want to create a smaller, more efficient module to get at the core competencies, either as part of a phone screen or as part of a 1-hr pre-onsite gate that contains a few different modules.
You can even create smaller interview modules by being specific about the competencies you are assessing. Being specific will allow you to cull extraneous context and tasks to turn a 45-minute interview into a 15-minute foray.
“System Design” is a topic, not a competency. Consider Bloom’s Taxonomy and be specific about the verb and objective.
Do you need your engineering managers to create an architecture from scratch? Or to demonstrate that they can analyze two solutions proposed by a senior architect? Do they need to evaluate the judgement of their architects, or exercise good judgement themselves? Do they need to remember the CAP theorem or do they need to generate insightful conclusions based on known constraints around system consistency and availability?
Beware proxy-based signals
It may be that asking an engineering manager to write code is a good proxy for the problem-solving skills, strategic thinking, and general learning and mastery skills that you are looking for, not to mention exposing experience likely to elicit empathy and understanding for and from their reports.
Similarly, while it may be that someone could learn a particular skill or piece of knowledge on the job, say the CAP theorem, it’s also a proxy for real experience and may suggest a candidate hasn’t yet grappled with the challenges of distributed systems, and may not be ready to set up their team for success on an ambitious redesign.
The problem is that proxies are a double-edged sword. They hide what you’re getting at for both interviewers and candidates, which means things tend to go wrong in the assessment happening on both sides of the table. Without turning this essay into a book, my main advice is this: whenever there are directly relevant and important competencies that you can assess, assess those first.
If you still need a hiring decision to rest atop a proxy, make it as explicit as possible for everyone. “This isn’t a coding interview, this is a strategic problem-solving interview. We’ll first ask behavioral questions to explore how you’ve used strategic thinking to solve problems in the past. Then we’ll ask some greatly simplified toy problems to see how you solve problems and utilize strategy in real-time.”
For the behavioral part, you might try to dig into a technical challenge, or you might be ok with investigating their strategic skills when handling a people management quandary or a surprise challenge impacting a business deadline.
For the skill assessment, maybe it’s more about problem-solving skills. If you’re looking for candidates’ ability to come up with approaches and analyze trade-offs, you may realize the coding is not even important. That is signal that you can uncover in a subsequent code review interview, or in a different module where they explain code they’ve written previously (ideally, this would be a patch you can also look at, too, or else they’ll have to paint you a detailed picture using words).
Or perhaps, you do want to see them spend 5 minutes writing code, but you also tell interviewers not to waste time while the candidate nervously overlooks a missed semicolon or forgets a function name. Just tell them and move on; it’s noise to this particular signal.
Try different approaches and measure your onsite-to-offer ratios until you start seeing the improvement you’re looking for.
Beware of turning candidates away
Senior candidates, especially the ones you want to hire, are likely happily employed, or else considering multiple exciting opportunities. The start of your pipeline could be the start of a long-term relationship; the foundation matters A LOT to candidates. That conversation, ideally with a hiring manager, is a chance for the candidate to:
- Learn if you are worth their time. Excellent senior candidates have the privilege of being choosy, while also having limited free time and time-off.
- Share their experience, strengths, and interests. Having a conversation where you ask some behavioral questions is ok, but giving a senior candidate a test before introductions is deeply unsatisfying.
Regarding the first point, you can still hire for humility, ability to compromise, respect for the less empowered, etc. The point is that the leverage equation is different than when hiring college graduates. In early-career hiring, it’s more common to ask candidates to prove they can code, often with very little veneer.
A better trade-off for senior candidates is not a code test, but rather a written application. Give the candidate has a chance to answer a few questions about their past experience, strengths, and future interests. Replacing the conversation with a coding exam is not a decision about saving time, but about what competencies matter.
For senior candidates, you want to maintain the initial human connection as much as possible. Reduce the onsite-to-offer ratio by making the initial phone screen assess more signal.
You may need a 1-hr gate between a non-technical recruiter phone screen and the 4-5 hr onsite. However, before introducing new competencies into that gate, consider what your current interviews are doing right and what you may want to change. Move the common rejection reasons earlier.
Work backward from job success. Getting the right answers to the wrong questions creates chaotic results, which is near impossible to improve iteratively. Once you know specifically what competencies are relevant and important to a hiring decision, the interview questions (and guides and structured rubrics) practically write themselves.
I hope that helps!