Karat Interview Questions, Explained

Google: What are Karat interview questions?

Chances are, if you’ve taken a Karat interview, you’ve run a search like the example above. In fact, most candidates interviewing at any company will have scoured Leetcode, Reddit, Blind, Glassdoor, and other sites for clues on what questions they might be asked throughout the technical hiring process.

At most companies, there’s no straightforward answer. Historically, most interviewers create their own questions and score interviews on their own scales. Likeability bias plays an outsized role, as does the individual interviewer, how many cups of coffee they’ve had, and whether or not they had a decent night’s sleep. But at Karat, it’s our mission to make every interview predictive, fair, and enjoyable, so we have a structured process for creating, testing, and scoring all of our interview questions.

I sat down with Don Gannon-Jones, Vice President of Content at Karat, to get his insights on how Karat designs technical interview questions. Don is a veteran in the tech industry and has first-hand experience building successful educational training sessions and interview content. Check out the full Q&A below to learn more about how and why we engineer our Karat interview questions to match the competencies organizations are hiring for.

Don is a veteran in the tech industry and has first-hand experience building successful educational training sessions and interview content.

Q: What is “interview content”?

Imagine your team needs to hire a couple of dozen software engineers. Everybody who’s there now gets together, and they kind of work out — what do we want to ask these folks as we’re interviewing them? What kind of problems do we want to talk about? That’s the content.

Figuring out what interview questions and formats to use is a big part of what we do at Karat. And because it’s our full-time job, we are able to dedicate more rigor and science to the interview process than in-house teams. The idea for us is to be fair. And even though I don’t think anybody loves doing job interviews, we want to make them as enjoyable as possible. We want candidates to be comfortable and really be able to shine. Thinking about content is how we achieve consistency, and it’s how we make sure that there’s no unintended bias built into it.

Q: What kind of science goes into Karat interview questions?

Above all, we want to make sure that we’re never asking things that aren’t related to the job. We don’t want to talk about your hobbies, although I hope you have some. We don’t want to talk about your family. We just want to talk about the things we know you’re going to be doing if you get this job.

We start by building out a validated job competency model. We look at a ton of job descriptions. We do panels, we do surveys, and we conduct workshops with technology leaders to come to a relatively short list of core competencies. These are the things you would expect a front-end software engineer to do, or a back-end software engineer. That forms the basis of our content.

Then, we build what we call probes from that. For example, if the competency is “mitigate security concerns in code,” we may build a little code exhibit and ask you some questions about potential security concerns that you might have. And all of the interview content gets built by that model.

Then we run it through a whole bunch of testing: We hire people who aren’t front-end engineers, people who have been doing it for a couple years, people who’ve been doing it for five years, and we make sure that the people who shouldn’t be passing the interview aren’t, and the people who should pass it easily do. So there’s a lot of validation that goes into it. We want to make sure that what we’re putting out is something that’s meaningful and fair.

Q: How does Karat align interview questions to job expectations?

You have to look at your job description and ask questions that align to that, and you have to look at what the existing engineers in that job do all day long. You want to ask questions about those things because we do overload our job descriptions.

For example, I can’t tell you how many so-called full-stack engineers that I’m friends with, who don’t do any front-end work whatsoever. They understand it, and they could do it, but it’s not what they’re actually doing. They’re only called full-stack because that’s what the company wanted to hire. To make a really fair interview, I need to focus on the things that the incumbent engineers actually do – not all the things they might aspirationally do. I want to keep it focused on the actual reality of the job.

We sit down with some folks who are in the role and ask, “what are some of the key things you do today?” We talk to people who are in the job, and then we normalized the language that they used in order to build our competency models.

Q: Does this match up with the specific tech stack a customer uses?

It depends on the organization. Some large enterprises–especially big organizations undergoing digital transformation–will have specialized tech-stack requirements. But others may be more interested in core computer science problem-solving abilities.

We see some job descriptions that are like a wish list of everything. Employers know they’re never going to get all those things. But at the candidate level, it can seem incredibly intimidating. Core competencies need to be deal-breakers.

Q: Are all Karat interviews structured the same way?

We build different interview structures to assess different things. For example, if I’m interviewing someone who’s fresh out of college, I’m obviously not leaning heavily on their business experience. So we interview that person a little bit differently than someone who’s been in the industry for five years.

One approach is the classic FAANG interview question, which is built around algorithms and data structures: “Here’s a data structure. I need you to write some code that will scan this matrix of data and tell me some facts about it.” That’s an algorithm and data structure question that is very appropriate for university graduates, because it’s the type of thing they’ve just spent four years doing, and it’s fresh in their minds. It reflects problem-solving. It reflects logical thinking. And you get to see a little bit of their coding ability without assessing for real-world experience that they might not have yet.

For more senior candidates, a different approach could be to get more into discussion and analysis. We might say, “Here’s a hunk of code. We’re seeing some performance problems with this as it scales. Is there anything in here that you might look at and call to my attention that might be a performance issue?” That is a really great cognitive exercise. It gets to someone who’s had some experience, they’ve been in the trenches, they’ve seen these things.

A candidate who can immediately say, “Oh, yeah, right here. You should not be doing it that way; I would change it to do this other thing.” Fantastic. That tells me they’ve done this for real – and that’s really what I’m trying to get at here.

In other instances, we might probe lightly for conceptual knowledge. “Talk to me about API design. We’ve got this one API and we need to add some fields to the input. How would you go about doing that?” In that case, we’re looking for someone to say something like, “Well, you need to version the API, and create a new version of it, because it’s a contract. You can’t just change it within the version.” And that tells us they’ve been there; they know what they’re doing and they have a good, informed approach.

Most of all, we focus on lining the format and questions up for the right job role. I’m not going to take an entry-level role and then ask the candidates API design questions, because your entry-level folks won’t be doing API design, I hope. You’ll want them to learn that through experience over time. But if someone’s coming in from a more senior role? Yeah, I should be able to ask them that type of question, and they should just be able to nail it right away.

Q: How do you ensure Karat interview questions aren’t too easy or hard?

We write a ton more content than anyone will ever see, and in our testing, we identify the things that really tilt it one way or the other. This one, everybody gets right? Forget it. Everyone gets this one wrong? Forget it. We don’t want to make it artificially hard; we’re looking for the things that help us identify a great candidate versus a not-so-good candidate. And that isn’t always about making it hard; it’s about pinpointing those things that really probe those job competencies the best.

So what you wind up with is roughly an hour’s worth of content (depending on format) that, taken as a whole, really lets candidates demonstrate the skills needed to be a solid software engineer. They may still have some things to learn. They’re going to have to come up to speed on your code base. You may be using a couple of tools they have never used before. But it’s a solid person and they’re going to work well for you.

Q: How does Karat measure candidate performance?

Any question you ask needs an objective scoring rubric – a scorecard. And that doesn’t have to be binary – in fact, it shouldn’t be. Instead of yes or no, it should be “this is a great answer”, or “this is an okay answer” that maybe misses some nuances we wish we had heard, or “this is a very poor answer,” when it isn’t what we’re really looking for, with clear definitions for each level. Any five people on your team should be able to look at it, and know exactly what it means and all have the same answer. That’s one of the ways you remove bias.

Q: Any final thoughts on interviewing?

Structuring interviews that are predictive, fair, and enjoyable is a lot of work! There’s a lot of science to it. The challenge for most companies is that when they hire software engineers, they aren’t being hired for their interviewing abilities. So at a lot of companies, interviews are an afterthought with a manager going in between meetings and thinking, “oh, let me just come up with some trivia.” And that’s why Karat exists. Interview Engineering is a real discipline, it’s a real task. It does take a lot of work to do it really, really well.

For more information about how Karat helps organizations improve technical hiring, check out the latest Total Economic Impact report from Forrester.