“The path to more diverse technical talent” report by Harvard Business Review Analytic Services and sponsored by Karat is now available. Read it here.
How Karat supports leading technical recruiting processes.
Technical interviewing and the technology to make it predictive, fair, and enjoyable.
Our mission is to make interviews fair, predictive, and enjoyable.
What developer candidates need to know about the Karat interview.
Are Karat interviews too easy or too hard? Lusen Mendel breaks down what we measure and why.
Earlier this month, the team and I were reviewing online feedback, which is something we do regularly to refine the Karat technical interviewing solution.
Some of the feedback suggested the Karat interview is too hard. Other feedback stated the Karat interview is too easy. One overarching theme that we did find was that candidates think they do badly if they don’t solve questions fast enough.
Is that true?
Well, before I tell you whether our interviews are too easy or too hard, indulge me in some context. It’ll be as relevant as your last interview question. Probably more relevant if your last interview wasn’t with Karat, I promise.
Homo sapiens, Latin for “wise person,” are constantly learning. We start out as messy little balls who do nothing but eat, sleep, and poop, but eventually, we learn how to iterate through arrays and validate user input.
Assessment is integral to learning. Each new piece of knowledge or skill that we learn builds on knowledge and skills that we’re already comfortable with. We constantly and incrementally cycle between assessment and learning, expanding the foundation upon which new understandings and abilities grow.
2,500 years ago Socrates asked questions as a teaching method in itself. We still do that today, albeit with new and improved granularity and color.
Here are the dirty secrets, the lynchpins of learning and assessment, upon which all interviews are based:
Yes, in the sense that we can assess the coarse level of a specific, fine-grained competency by the correctness and speed of the answer.
But also, no, in the sense that counting minutes is neither a fine-grained nor linear scale. There might be a difference between solving a problem in ten minutes rather than 100 minutes, but a difference of two minutes is hardly meaningful.
The goal of any interview is to assess relevant and important competencies in a limited amount of time. At Karat, we work with hiring managers to define which competencies are relevant to success in a specific role and which are important to assess at a specific point in the hiring pipeline. Then, we identify the interview format and questions that will illuminate those competencies, which we call the signal, and measure them in a structured rubric.
There are a few conclusions that follow from the setup.
There isn’t one Karat interview. Each Karat interview is tied to a specific role at a specific company and each candidate’s performance is evaluated by that specific company based on their specific needs. [note: our Interview Engineers do their best to highlight a candidate’s strengths, the Karat Interviewing Infrastructure uses their observations to make a recommendation and the hiring company makes the final decision.]
What questions you get, how many you are asked, and what skills you have to demonstrate to move forward, depend on the company and role. When you schedule an interview, Karat sends you a personalized email explaining what you should expect and how to prepare. If you have additional questions, you can reach out to Karat’s customer experience team, as well as the hiring company’s recruiter.
While speed is part of assessing mastery, speed is not the signal. Doing well in an interview is not meant to feel like winning a race, so much as working through coding problems in a steady, straightforward way without getting stumped or lost for long stretches of time.
Typing speed, for example, is the opposite of signal, otherwise known as noise. To reduce noise, we accommodate situations where the interface of the interview would otherwise bog down a candidate. Whether it’s giving extra time to a candidate who broke both wrists and typing speed actually was an issue or ensuring that our platform is accessible to screen readers. We’ve even conducted interviews in American Sign Language.
Often the signal that hiring managers are looking for in a coding interview is an indication of specific programming competencies such as:
Simple coding problems may seem irrelevant to on-the-job development, but they are useful for revealing relevant software engineering competencies, including:
Filtering candidates based on resumes and code tests can introduce noise. This especially true for false negatives, rejecting candidates even though they could be great hires.
On the other hand, if every candidate requires hours of interviews with multiple engineers, engineers may find themselves working overtime to keep up, while the engineering team as a whole misses both product deadlines and hiring goals.
We’re engineers. We’re used to trade-offs! One-hour live interviews strike a useful balance because Interview Engineers deliver both improved candidate experience and a stronger hiring signal by assessing interpersonal and decision-making skills that solitary tests cannot, and by maximizing candidate performance to reduce false negatives, such as clarifying silly misunderstandings and providing encouragement during a nerve-wracking brain freeze.
We extensively test Karat interview questions calibrate expectations, ensure they aren’t overly sensitive (overfit), and reduce noise. The goal is that candidate performance demonstrates a true signal on the competencies specifically relevant and important for that particular hiring process.
We design Karat interviews to provide some challenges, so just because you aren’t breezing through an interview doesn’t mean you are failing. Maybe grappling with difficult questions is successful behavior. Focus on doing your best, being positive, and let the hiring company make the decision.
Second, if you are not moving forward from early interview stages, see if you can identify what gaps to work on. Keep practicing! Not just alone, but with a friend acting as an interviewer so you can practice explaining your approach and writing code with an onlooker. And keep practicing until you are comfortable during interviews and move steadily forward when brainstorming and implementing solutions.
I hope you found this post insightful. Big thank you to all the candidates who leave us feedback after their Karat interviews. Our interviews are a solution to the problem of assessing technical talent, however, we’re always working hard to find a more optimal solution — more predictive, more fair, more enjoyable. Keep leaving feedback, and we’ll keep you updated with what we learn.
Your email address will not be published. Required fields are marked *
Over the last couple of years, we’ve seen an industry-wide push to interview software engineers for intern and junior developer positions very early — even...
In March 2021, 309 tech leaders shared observations of their companies’ approaches to improving diversity on their technical teams. The results show that most teams...
Last week, we had the chance to celebrate Pride month by listening and learning from three outstanding leaders across the tech industry: Roz Francuz Harris;...