What do Karat technical interviews measure?

Earlier this month, the team and I were reviewing online feedback, which is something we do regularly to refine the Karat technical interviewing solution.

Some of the feedback suggested the Karat interview is too hard. Other feedback stated the Karat interview is too easy. One overarching theme that we did find was that candidates think they do badly if they don’t solve questions fast enough.

Is that true?

Well, before I tell you whether our interviews are too easy or too hard, indulge me in some context. It’ll be as relevant as your last interview question. Probably more relevant if your last interview wasn’t with Karat, I promise.

Additional technical hiring resources

• Technical hiring explained
• What is a technical interview?
• Tech hiring trends report

Assessment and learning

Homo sapiens, Latin for “wise person,” are constantly learning. We start out as messy little balls who do nothing but eat, sleep, and poop, but eventually, we learn how to iterate through arrays and validate user input.

Assessment is integral to learning. Each new piece of knowledge or skill that we learn builds on knowledge and skills that we’re already comfortable with. We constantly and incrementally cycle between assessment and learning, expanding the foundation upon which new understandings and abilities grow.

2,500 years ago Socrates asked questions as a teaching method in itself. We still do that today, albeit with new and improved granularity and color.

https://en.wikipedia.org/wiki/Bloom’s_taxonomy

What does this have to do with speed?

Here are the dirty secrets, the lynchpins of learning and assessment, upon which all interviews are based:

• When a person is asked a question requiring knowledge or skills that they are very comfortable with, they can generally answer those questions correctly and quickly. If a candidate writes unit tests for breakfast without thinking, they can likely rattle off an explanation for “What is a unit test?” without blinking.

• On the other extreme, when a person is asked to explain or do something that they aren’t familiar with or don’t know how to do, they generally know right away that they don’t know. If a candidate has never heard of REST, their explanation to “What is REST?” is a short and sweet  “I don’t know.” Similarly, if they’ve never used a hashmap and now need to use one, they would likely ask to look up syntax.

• The grey area is that tricky middle ground where someone is on the cusp of competency. This is where we need a sensitive and accurate signal. If a candidate understands the general concept of a depth-first search, they might slowly work out the implementation during the interview or get close before running out of time. If they have written and debugged a handful of programs, they might spend the whole interview working through error messages and logic bugs, sometimes getting a working solution in the knick of time and sometimes not.

So interviews are a speed test?

Yes, in the sense that we can assess the coarse level of a specific, fine-grained competency by the correctness and speed of the answer.

But also, no, in the sense that counting minutes is neither a fine-grained nor linear scale. There might be a difference between solving a problem in ten minutes rather than 100 minutes, but a difference of two minutes is hardly meaningful.

The goal of any interview is to assess relevant and important competencies in a limited amount of time. At Karat, we work with hiring managers to define which competencies are relevant to success in a specific role and which are important to assess at a specific point in the hiring pipeline. Then, we identify the interview format and questions that will illuminate those competencies, which we call the signal, and measure them in a structured rubric.

There are a few conclusions that follow from the setup.

One size does not fit all

There isn’t one Karat interview. Each Karat interview is tied to a specific role at a specific company and each candidate’s performance is evaluated by that specific company based on their specific needs. [note: our Interview Engineers do their best to highlight a candidate’s strengths, the Karat Interviewing Infrastructure uses their observations to make a recommendation and the hiring company makes the final decision.]

What questions you get, how many you are asked, and what skills you have to demonstrate to move forward, depend on the company and role. When you schedule an interview, Karat sends you a personalized email explaining what you should expect and how to prepare. If you have additional questions, you can reach out to Karat’s customer experience team, as well as the hiring company’s recruiter.

What is the signal?

While speed is part of assessing mastery, speed is not the signal. Doing well in an interview is not meant to feel like winning a race, so much as working through coding problems in a steady, straightforward way without getting stumped or lost for long stretches of time.

Typing speed, for example, is the opposite of signal, otherwise known as noise. To reduce noise, we accommodate situations where the interface of the interview would otherwise bog down a candidate. Whether it’s giving extra time to a candidate who broke both wrists and typing speed actually was an issue or ensuring that our platform is accessible to screen readers. We’ve even conducted interviews in American Sign Language.

Often the signal that hiring managers are looking for in a coding interview is an indication of specific programming competencies such as:

• Iterating through a list to find an item
• Transforming business logic into if/then/else statements
• Retrieving data from key-value pairs
• Breaking a recursive problem down using inductive logic

Simple coding problems may seem irrelevant to on-the-job development, but they are useful for revealing relevant software engineering competencies, including:

• Problem-solving, such as clarifying assumptions, verifying uncertainties, and risk management
• Abstraction, such as identifying a practical data structure or interface for representing a general problem and solution
• Implementing clear code that neither hides bugs nor gets in the way of debugging
• Familiarity with common bugs and how to avoid them
• Salient technical communication, such as explaining one’s approach out loud, or explaining what is causing a bug and how one plan’s to fix it

Time is an engineering trade-off

Filtering candidates based on resumes and code tests can introduce noise. This especially true for false negatives, rejecting candidates even though they could be great hires.

On the other hand, if every candidate requires hours of interviews with multiple engineers, engineers may find themselves working overtime to keep up, while the engineering team as a whole misses both product deadlines and hiring goals.

We’re engineers. We’re used to trade-offs! One-hour live interviews strike a useful balance because Interview Engineers deliver both improved candidate experience and a stronger hiring signal by assessing interpersonal and decision-making skills that solitary tests cannot, and by maximizing candidate performance to reduce false negatives, such as clarifying silly misunderstandings and providing encouragement during a nerve-wracking brain freeze.

We extensively test Karat interview questions calibrate expectations, ensure they aren’t overly sensitive (overfit), and reduce noise. The goal is that candidate performance demonstrates a true signal on the competencies specifically relevant and important for that particular hiring process.

What does this mean for candidates?

We design Karat interviews to provide some challenges, so just because you aren’t breezing through an interview doesn’t mean you are failing. Maybe grappling with difficult questions is successful behavior. Focus on doing your best, being positive, and let the hiring company make the decision.

Second, if you are not moving forward from early interview stages, see if you can identify what gaps to work on. Keep practicing! Not just alone, but with a friend acting as an interviewer so you can practice explaining your approach and writing code with an onlooker. And keep practicing until you are comfortable during interviews and move steadily forward when brainstorming and implementing solutions.

I hope you found this post insightful. Big thank you to all the candidates who leave us feedback after their Karat interviews. Our interviews are a solution to the problem of assessing technical talent, however, we’re always working hard to find a more optimal solution — more predictive, more fair, more enjoyable. Keep leaving feedback, and we’ll keep you updated with what we learn.