Your Guide to Technical Interviews

When hiring any employee, it’s customary to evaluate them to make sure they’ll be successful in the role and be a good fit for your company. The hiring process usually includes reviewing the candidate’s resume and interviewing them. For technical roles, such as engineering, data science, and security, candidates need to complete technical interviews that evaluate their technical skills, problem-solving abilities, and domain knowledge. 

While technical interviews can be extremely helpful in selecting the best candidate, there are also many pitfalls that can make them unreliable, create a frustrating experience for candidates, and lead you to the wrong conclusions. To avoid this, it’s important for engineering leaders and talent teams to know how to properly structure technical interviews so that they’re fair and inclusive to generate a reliable, accurate hiring signal. 

In this guide, you’ll gain a deep understanding of technical interviews, including:

• How to create the right questions for a role based on responsibilities and seniority levels
• How to score interviews objectively and consistently
• How AI impacts technical interviews

You’ll also uncover best practices to hire the best possible talent effectively and efficiently.

What Is a Technical Interview?

A technical interview assesses the technical skills of software engineers. During the hiring process for engineering roles, candidates need to complete one or more technical interviews to demonstrate that they have the necessary skills and knowledge to do the job. This approach can include coding interview questions that are completed in an integrated development environment (IDE), problem-solving, talking about technology or computer science topics, and walking through previous work experience. 

An interviewer with relevant experience conducts these interviews so they can have informed discussions with the candidate and assess them. It’s also important to use consistent interview questions to treat candidates fairly and generate an accurate hiring signal. 

Technical interviews should be inclusive and predictive of candidates’ skills. Companies can ensure that their interviews have these qualities by:

• Making the interview content relevant to the job’s seniority and skills
• Using purpose-built technology for the interview
• Having a professional interviewer with software engineer experience conduct the interview

Technical interviews are important because they answer the question: What is the likelihood that a candidate will succeed in this role? The “perfect” interview will always give you the correct answer to this question. 

While hiring each candidate and seeing if they succeed would certainly give you an accurate answer, it’s not cost or time-effective; however, this doesn’t mean that it’s impossible to arrive at the correct answer. You can adapt the hypothetical perfect interview into a technical interview that’s more viable and nearly as effective.

Components of a Technical Interview

Technical interviews can be broken down into five components: interview time, interview relevance, interview questions, interview interactions, and interview scoring. Every component can be tweaked to build an effective and consistent interview process that gets you as close as possible to the perfect interview. 

Interview Time

Filling an open role requires a huge time commitment and a lot of work from both the hiring company and the candidate. It can take months for businesses to find the right person, while candidates typically need to juggle interviewing for a new role with their current job and responsibilities. To reduce the time commitment, it’s best to front-load risk in the technical interview process. If there are tasks or skills that are particularly important to the role, test for these early on so to quickly determine whether the candidate will be able to complete them when hired. 

Interview Relevance

Not all tasks can be feasibly tested during the interview process. For example, if the role is responsible for large-scale software design, the candidate can’t build a complex system in a reasonable amount of time — not to mention that asking them to do so for free is unreasonable — but, there is a way to approximate these situations.

You can ask candidates to complete related tasks and then extrapolate to figure out how the candidate would perform the larger task. When doing this, it’s important to make sure that the smaller task is a good proxy for a task that’s important to the role. 

Interview Questions

Not every question is a good interview question, even if it’s a proxy for a large task that the job entails or asks about a real-world problem. Asking a software engineer to discuss their previous work or how to solve a real-world problem usually results in a lengthy answer. The reason: Software engineering requires understanding the problem being solved, frameworks, protocols, dependencies, and the codebase. You run into the same constraints of an ideal interview — an interview takes an enormous amount of upfront effort for a marginal benefit. 

Effective interview questions are scale models of real-world problems that quickly get at the core of a technical task without requiring much context. When done well, technical interview questions give candidates an opportunity to demonstrate a set of critical programming skills in a brief amount of time. 

Interview Interactions

Providing a great interview experience to the candidate is important, and interviewers play a key role. Good interviewers put candidates at ease, deliver questions clearly and unambiguously, and partner with candidates to bring out their best performance. They also practice asking questions about the core competencies that matter to the organization. 

Interview Scoring

When describing how an interview went, language can be subjective. People can mean different things when using the same word, or they can use the same word but mean something different. If it’s possible to gather all the interviewers together, you can have a discussion to unpack what someone means when they say a candidate was “okay” or figure out whether a candidate who received varying reviews is worth inviting to the next round. However, if interviewers are remote, or you want to make a decision without gathering everyone, you’ll need to compile everyone’s evaluations as data points. Be aware that this can magnify inconsistencies. 

Structured scoring rubrics solve this by creating a consistent scale to measure competencies. They make it possible to aggregate feedback, compare it from interviewer to interviewer, and consistently report on these competencies.

Technical Interviews by Career Level

The format of a technical interview depends on the career level of the engineering role:

• Intern and early-career: For those at the beginning of their careers, technical interviews typically begin with a 10- to 15-minute project discussion where the interviewer provides the question, direction, and context. Then, the candidate spends the next 45 to 50 minutes coding solutions that address the problem. 
• Mid-career: Software engineers with three to five years of experience are asked to solve more complex problems. Their interviews start with 10 minutes of discussing the project, a 30-minute coding exercise, and end with 20 minutes of questions that demonstrate their working knowledge. 
• Senior: Senior roles handle complex business problems and lead projects, so they require a deep understanding of the competencies that are required to succeed. Technical interviews for this typically have three parts: discussions focused on past work, questions that demonstrate their working knowledge, and a business logic complexity or code and architecture review.

Technology and Tools Needed for Technical Interviews

One of the most consistent interview pain points that engineering candidates identify is when they’re asked to write code in a text document, such as Google Docs, instead of an integrated development environment or IDE. IDEs allow engineers to easily develop and debug code and verify its functionality. Providing an IDE for technical interviews dramatically improves fairness and ease for candidates. 

When conducting technical interviews, you need the proper tools that:

• Allow candidates to write and run their code in an IDE
• Connect the interviewer and candidate via video chat
• Support the interviewer with a pre-approved set of interview questions and format that ensures interview consistency
• Removes opinion-based recommendations through a rubric-based structured write-up that informs the hiring recommendation

Typical Technical Interview Structure

How technical interviews are structured largely depends on the size of a company, the interview expertise of the hiring team, and the role you’re hiring for. Typically, they also include the following steps:

• Pre-interview preparation
• Initial introduction
• Questions from the candidate
• Next steps

Pre-Interview Preparation

Successful technical interviews start well before the candidate actually interviews. It starts when you create your interview questions. 

First, review the job description. If it’s written correctly, it will guide question creation. Every skill or responsibility can be turned into a question that tests the candidate’s ability. It’s also important to consider the future of the role and how it will evolve. Add interview questions to see if a candidate will be able to fulfill their duties when there are sudden changes or new technologies. 

Initial Introduction

Interviewers should welcome candidates with enthusiasm and introduce themselves. If the interview is remote, check that the videoconferencing technology works for both parties. The first moments in an interview should be used to make the candidate feel comfortable, share how the technical interview will go, and optionally, what your expectations are. Taking the time to guide candidates through the interview experience creates a fair and inclusive interview process, and it increases the chances of candidates accepting your offer by as much as 75%.

After the introduction, the interview moves on to technical questions, which typically start easy and gradually become more complex. The latter questions are mental exercises that require candidates to use their working knowledge, and they should help you separate candidates from each other. 

Alternatively, the interview can present a more difficult problem as a series of smaller questions. The answers to the first questions will help the candidate solve the final task without putting a lot of pressure on them all at once. 

Questions From the Candidate

Interviews should also give candidates the opportunity to learn more about your organization, the team, and the role they’re interviewing for. Engineering candidates are often curious about the tech stack that you use, your team structure, and company details. Taking questions from the candidate can also help you uncover areas where you can improve your interview process in the future.  

Next Steps

Every interviewer should now have a rough idea of whether the candidate will proceed to the next round. Interviewing notes are shared among the team so that you can reach a clear hiring recommendation. Even if you decide not to move a candidate forward, make sure the candidate hears back and isn’t “ghosted.” 

Ideal Interview Length

There isn’t an ideal interview length. One interview usually lasts between 30 minutes and an hour, while an entire interview loop can take four to six hours. In a Karat engineering survey, we found that respondents averaged 20.7 first-round technical interviews per hire. In comparison, our clients hold about 13 interviews per hire in New York, 16 in San Francisco, and 18 in Seattle. The average across all other markets is 16 interviews per hire. 

While the number of interviews and duration of a single interview doesn’t really matter, the quality of interviews does. Technical interviews should be efficient so that you have a good onsite-to-offer ratio and ultimately hire the right people. 

The History of Technical Interviews

The origin of interviews can be traced back to the early 1900s, when the first tests were developed to evaluate candidates on their knowledge and/or skills. Prior to that time, craftsmen passed their profession onto their children. If they didn’t have children, they would train apprentices. During the Industrial Revolution in the 1800s, most jobs didn’t require much skill, so people were hired and retained depending on their performance. If an employee didn’t work out, they could be easily replaced.

In 1917, Robert Woodworth, a psychologist at Columbia University, developed what is known as the Woodworth Personal Data Sheet or Psychoneurotic Inventory. It consisted of 116 yes or no questions and was meant “to assess intelligence, to determine aptitude for different jobs, to weed out the mentally ‘unfit,’” of recruited soldiers. 

Because Woodworth’s test is considered to be a personality test, rather than an aptitude test, it serves as the precursor to interviews. For the modern interview, most sources attribute it to Thomas Edison.

With more college-educated people entering the workforce in the 1900s, employers realized that they could be more selective and needed a way to evaluate candidates. It’s said that Thomas Edison would get hundreds of applicants and he was becoming increasingly frustrated with employees who lacked the knowledge that he had. As a result, he created a 150-question test in 1921 that asked prospective employees questions related to the specific role and/or general knowledge questions.

While the 1921 test is the most well-known, it wasn’t the first time that Edison created a test for prospective employees. In 1883, he created a number of tests that covered dynamos, steam engines, and meters. Most of the questions were technical, although some were tests of common sense. 

Computer science wasn’t its own discipline until the 1970s and 1980s. Interviews primarily uncovered a candidate’s experience, where interviewers would review a candidate’s resume and ask about the projects they worked on. In some cases, candidates would be asked to write some code on paper. If candidates had experience that was similar to what the job entailed, they would get hired.  

Technology Enables More Convenient Interviews

Starting in the mid-1900s, the invention of the Internet and mobile phones meant that interviews no longer needed to take place in person. Instead, interviews and assessment tests could take place remotely. 

In 2020, the global COVID-19 pandemic forced most interviews to go fully virtual. The first step in the interview process, the recruiter screen, already occurred on the phone, and candidates were used to completing a preliminary skill assessment online. The onsite interview is where the biggest change in the interview process occurred. Onsite interviews had to be conducted remotely through videoconferencing. Writing code with pen and paper had already shifted to whiteboards, which were now replaced with IDEs and collaborative coding tools. 

Andrew Hewitt, an analyst for infrastructure and operations professionals at Forrester Research, said that although companies were already leveraging technologies like artificial intelligence, video interviews, and online skills screenings, these tools shifted from being a nice-to-have to a must-have during the pandemic. 

This change in how onsite interviews are conducted benefited companies in several ways. Because companies were no longer limited to candidates who had the availability and means to travel to their offices, they could access a larger talent pool of candidates located in other cities, states, or even countries. Regardless of where they’re located, companies can tap into the top markets for engineering talent, which can lead to more diverse hiring, perspectives and experiences, and innovative solutions. 

Remote interviews have also benefited candidates. They saved the time and money required to travel to a company’s office. Candidates were relieved of the high-pressure environment of whiteboarding in front of new people for the first time. Now, candidates write code in the comfort and familiarity of their own home; however, there is a trade-off with remote interviews, as they make it more difficult for candidates to get a sense of company culture. Candidates can’t tour the office, which is important if the company eventually decides to bring employees back in person. 

How AI Is Impacting Technical Interviews

AI is the latest technology that’s changing the nature of technical interviews. AI, like ChatGPT, can produce high-quality code from a simple natural language prompt, which is causing many companies to wonder what this means for technical interviews. 

Our R&D team has been monitoring the research and trends on generative AI for a while. We’ve also run experiments, trying out our interview questions on ChatGPT. Based on what we’ve seen, AI will likely disrupt the day-to-day software development process, but it’s less likely to invalidate our current interview methods, especially well-designed interviews.  

The impact that AI has on technical interviews also depends on the interview format and content. Traditional code tests that simply ask a candidate to produce a working code sample are particularly susceptible to cheating, since AI is very good at generating code samples. This isn’t a new problem though, as candidates could always ask someone else to write their code. 

When candidates use AI tools for live interviews, we’ve found that this doesn’t alter or disrupt the results — as long as the interview is well designed and is conducted by a properly trained interviewer. Less-skilled candidates who use AI to write code or solve a problem won’t be able to explain their approach or why they made certain decisions. On the other hand, skilled candidates using AI can write code faster, resulting in more time spent on problem-solving and less on producing code. If this matches what they’ll be doing on the job, then it makes sense to interview them this way.

Given the benefits and disadvantages, should you allow AI in your technical interviews? Our recommendation is to mirror real-life working conditions whenever possible. In Karat interviews, we allow candidates to use StackOverflow and Google to look up documentation, just as they would on the job. ChatGPT and other large language models (LLMs) are similar — they’re tools that software engineers would leverage when working. 

If your organization allows engineers to use AI tools in their work, mirror this by allowing candidates to use those tools during interviews. For organizations that don’t allow the use of AI tools on the job, monitor your interviews for signs of external help. 

At Karat, our interview engineers are trained to ask probing questions to determine what the candidate is doing. If they suspect there’s cheating, they tag the interview and flag it for further review. We’ve observed that this candidate behavior is rare, and the instances of candidates using generative AI are less frequent than candidates receiving help from third parties off camera — a behavior our interviewers are also trained to notice. 

Although AI is changing how technical interviews are conducted and evaluated, the fundamentals of hiring haven’t changed. Companies still need to hire people with strong problem-solving skills and give them enough context and autonomy to solve your business problems. 

How to Conduct Technical Interviews

If you’re new to technical interviews, there are several key steps to take when establishing your technical interview process and executing it.

1. Establish a Standard Interview Process

Once you’ve created a job description for the role you’re hiring, you can use that to map out the interview process. Determine the number of stages you’ll have and what occurs at each stage. Consistent interviewing is crucial for hiring at scale, so every candidate should go through the same interview stages, although they may progress at different speeds. 

Creating an interview process that can be repeated for each candidate isn’t the only way to keep technical interviews consistent. Consistency also comes from identifying and measuring specific competencies, creating structured scoring, and training interviewers — all of which we’ll dive into in the following steps.

2. Design Interview Questions and a Scoring Rubric

If you haven’t already, identify the competencies that are required for success in the role. It’s important to do this because interview questions should stem from these competencies and what the engineer will actually do when hired. Competencies vary based on job responsibilities and seniority level. For example, QA engineers should have a commitment to quality that’s technology agnostic, a strong analytical perspective, and an aptitude for learning, among other skills. For senior engineers, competencies often include specific programming language skills, the ability to understand business logic, and the ability to complete an architecture review. 

Aside from focusing on evaluating core competencies, there are many other best practices for crafting technical interview questions that we cover later on in this guide.

Once you’ve created your interview questions, develop a structured scoring rubric for each question. Rubrics ensure consistency by aligning interviewers around concrete observations that can be made for each competency and helping them record their observations without being impacted by biases or subjectivity. No matter who the interviewer is, candidates will be evaluated on the same rubric that reflects the set of competencies that are relevant to the role.  

Scoring rubrics enable the hiring team to evaluate the candidate using the same rating scale, and they create a shared language for describing the candidate’s performance. This makes it easier to assess candidates and generate a strong hiring signal. You don’t have to decipher what interview write-ups mean and can make a clear distinction between candidates who are described as “great,” “awesome,” and “rock star.”

3. Identify and Train Interviewers

Interviews shouldn’t be conducted by whoever is available. Instead, identify a team of core interviewers who understand that interviewing is an important part of their job, have enough knowledge and experience to assess candidates, and can be trained for consistency. 

Interviewer training is an often overlooked step in the technical interview process. Interviewers should train and practice asking interview questions so that they can deliver them with clarity and empathy, and they should be trained on how to use the scoring rubric to reduce bias and subjectivity. 

You should also set up a quality control (QC) process for interviews to reduce the number of errors that interviewers make. This typically takes the form of video recording and reviewing interviewers’ performance. You can also collect feedback from candidates to identify areas of improvement for each interviewer. 

Although it takes a significant amount of time and effort to train interviewers, it’s worth it in the long run. Experienced interviewers produce fewer errors and you’ll have a more predictive process that doesn’t vary based on how “hard” or “easy” the interviewer performs. At Karat, we QC every interviewer and provide mentorship. Our rigorous process results in the same recommendation for the same candidate skill levels 98.8% of the time. 

4. Conduct the Interview

Now that you’ve got all the pieces in place for a consistent technical interview — a standard interview process, interview questions, a scoring rubric, and trained interviewers — you can begin interviewing candidates.

If you’re conducting remote interviews, it’s best to make them available 24 hours a day, seven days a week. You’ll give candidates the flexibility to schedule interviews when it works best for them, even outside of normal business hours. It also makes it easier to find an available time that works for both the interviewer and candidate, allowing candidates to move forward in the process as quickly as they want. 

For remote technical interviews, don’t forget to provide candidates with an IDE. If you don’t have a code editor, you’ll need to consider other tools that simulate how engineers code. Avoid text editors, such as Google Docs, since they lack important features such as testing, auto brackets, and clarifications. 

Lastly, both the candidate and interviewer should check their technology before the remote interview. Test your microphones to make sure each party can clearly hear the other, make sure there’s good lighting that helps the other person pick up on body language and facial cues, and confirm that your internet connection is stable to avoid wasting precious time. 

5. Evaluate the Candidate and Reach a Decision

At the end of every stage in the interview process, review the information that you’ve gathered about the candidate and consider whether they would be a good fit for the job. If there were multiple people who interviewed the candidate during an interview round, you may want to gather everyone together for a discussion to determine whether to move the candidate forward. When it’s difficult to coordinate a discussion with all interviewers, you can use the scoring rubrics to reach a decision. 

Once you’ve made a decision, follow up with each candidate — even if you’ve decided to pass on them. While a candidate may not be the best person for this role, it doesn’t mean that they won’t be right for another role in the future.

Examples of Technical Interview Assessments

There are several different ways for companies to assess a candidate’s technical skills: code tests, take-home technical assessments, and live technical interviews. Each has a different purpose and is best utilized in certain scenarios.

Code Tests

Code tests are automated assessments where the candidate’s goal is to write and run code that successfully solves a question or set of questions within a time limit. They’re typically used in the first round of technical interviews as a way to filter candidates in or out early on. Technical recruiters usually send candidates an email invitation to complete the test using a platform, such as HackerRank, Codility, or CodeSignal. 

Most code tests result in a binary pass or fail recommendation that’s based on whether the candidate’s solution is correct or not. This can negatively impact candidates from underrepresented and nontraditional backgrounds by “failing” qualified engineers who could receive an offer based on live technical interviews. Our data indicates that code tests could be filtering out as many as one in three hires. 

It’s best to use code tests for high-volume hiring programs targeting junior engineers. They’re not ideal when hiring specialized engineering roles and more senior engineers. These candidates are less likely to complete code tests because they believe the tests aren’t representative of their skills and they associate code tests with junior roles. Engineers are right, as code tests produce a poor hiring signal compared to live interviews. 

Take-Home Technical Assessments

A take-home technical assessment is a coding problem that candidates have multiple days or weeks to complete. Take-home tests might include questions about algorithms and structure knowledge, similar to code tests. Since candidates have more time to complete them, take-home tests might also ask candidates to build or extend applications. 

The test is reviewed and scored by engineers on the hiring team. Some use an interview rubric to score the test. Others evaluate it based on their experience and gut, which is not recommended because it can produce noise and bias. 

Take-home tests are useful when candidates feel nervous about timed code tests and don’t have code that can be shared with the hiring team. By being able to have more time, candidates can show their skills without the pressure of a time limit. Another scenario where take-home tests are useful is when you want to screen for development work where the candidate uses their own tooling and infrastructure or open-source inspired communication best practices.

Live Technical Interviews

Live technical interviews require candidates to solve problems and answer questions with the guidance of a professional interviewer. They’re the most human-centered type of technical assessment. Ideally, a live technical interview should be no longer than 60 minutes and use an IDE, consistent interview questions, and a video chat that allows the interviewer and candidate to talk to each other. 

Interview rubrics, which describe the competencies for the job and what different proficiency levels look like for each, are used to evaluate the interview. The competencies are scored, and the score indicates how to proceed with the candidate. Some candidates aren’t ready for the role or are below the hiring bar, while others exceed the hiring bar and should be moved to the next stage in the hiring process.

Technical Interview Questions and Scoring

Structuring interviews is a science and it requires a lot of work to do it exceptionally well. Our mission at Karat is to make every technical interview predictive, fair, and enjoyable. We have a deliberate process for creating, testing, and scoring the interview questions that we use when helping companies improve the quality, efficiency, and equity of their technical hiring. With the tips below, you can achieve a similar process that results in an accurate hiring recommendation.

How to Create Technical Interview Questions

How you craft interview questions for engineering interviews is incredibly important, as they play a big role in whether the interview is fair and effective or problematic and biased. Interviews should provide candidates with an opportunity to shine, rather than set them up for failure. Follow these tips to avoid ineffective interviewing strategies:

Strike the Balance Between Too Easy and Too Hard

Interview questions that very few people can solve aren’t effective filters because they cause candidates to perform below their actual capabilities. They show you candidates in the worst-case scenario when interview questions should be showing you how candidates deal with everyday work. 

At the same time, your interview questions shouldn’t be too easy. When questions can be easily solved by most, they aren’t an effective way to screen out candidates. 

Avoid Rapid-Fire Questions Covering Multiple Topics

You might think that asking candidates as many questions as possible that cover a wide range of topics is an effective use of the limited time you have with them. Instead of covering a lot of ground quickly, this interview approach results in shallow questions and answers. You won’t be able to have in-depth conversations that truly uncover a candidate’s abilities. 

Test for Ability, Not Knowledge

Many interviews tend to ask knowledge questions, such as “What are the four stages of a CI pipeline?” or, “Name five types of software tests.” Anyone can memorize the answers to these types of questions without knowing what they mean. 

If your interview has any knowledge questions, turn the questions into situations that candidates need to analyze. For example, tell the candidate that you want to make sure not to reintroduce any bugs when checking your code in the source control repo. Ask them what type of software test should be used in this scenario. If they identify that it’s a unit test, this provides a much stronger hiring signal that indicates the candidate understood what a unit test is and when it’s appropriate to use.

Focus on Deal-Breakers

Many software engineers use the interview to weed out candidates based on their own past work experiences. To prevent a bad hire, they turn the problems that they’ve had with past colleagues into interview questions. There’s always a risk in hiring, but you can’t eliminate that risk without being unfair. 

The best way to avoid this is to focus on what truly brings value to the organization. What makes up 80% of the job for 80% of the people are the deal-breakers that your interview questions should cover.

Scoring Rubrics

No matter who is interviewing a candidate, it’s important for companies to be consistent in their candidate assessments because it generates an accurate hiring signal and ensures they’ll hire the right person. Scoring rubrics or interview rubrics are a method that technical hiring teams use to align everyone around the same interview questions, rating scale, and language used to describe a candidate’s performance. By comparing candidates using the same criteria, companies create a fair and inclusive hiring process where every candidate gets an equal chance to succeed. 

The rubric takes the form of a scorecard that lists the criteria and competencies that are being used to assess the candidate’s skills and potential. Every job will have its own rubric that has been adapted to clearly lay out the technical and soft skills to analyze a candidate on, how to spot whether the candidate has the right experience, and a rating scale that describes what a “good” hire looks like. 

Rubrics should be created early on in the hiring process so that it’s applied to every candidate. They’ll help you create the same interview experience for all candidates, reduce bias, and assess each candidate on the same objective criteria. 

How to Create a Scoring Rubric

Although your rubrics should be customized for each engineering role, you can follow this general process to create a structured interview rubric:

1. Figure out which competencies are relevant to the job. There’s no limit to the number of competencies that you can use, but every competency that’s listed on the rubric should produce a hiring signal. Good competencies are relevant to the role; are not biased for things that are irrelevant to the job, such as cultural knowledge or information that can be quickly learned once hired; and, can be demonstrated in the live interview. 
2. Identify observable candidate actions and results that demonstrate a candidate has those competencies. Interview rubrics allow interviewers to quickly check off whether a candidate completed an action during the technical assessment. This is much easier and faster, more consistent, and less biased than creating a write-up after every interview. Translate competencies into a scale of observable behaviors. For example, the “implementation quality” competency can be turned into a list of the actions that the candidate took. If the candidate is confused by their own code, this indicates that the code is confusing. If the code doesn’t confuse its author and it doesn’t use any code-organizing techniques, the interviewer can conclude that the implementation quality is okay but not great. The same approach can be applied to other areas of expertise and behavioral interviews.

3. Translate a completed interview rubric into a hiring recommendation: Interview rubrics can clearly lead teams to a hiring recommendation, but it requires interviewers to first understand how the rubric and interview questions map to each other, how to use the rubric objectively, and how to focus on creating a great candidate experience. There are several ways of structuring your rubric to lead to a quantifiable result. You can weigh the competencies so that interviewers know how much a competency impacts their hiring decision. Another option is to assign points to the behaviors associated with each competency and score the rubric like a test.

Testing Your Interview Rubric

The best way to check whether your rubric is objective is to test for inter-rater reliability, which answers the question, “Between multiple interviewers who are rating the candidate, what is their reliability for giving the same answer?” 

Give the interviewers all the questions you’re going to ask, and have them write down their answers without their names. Get everyone together, have them score each answer, and explain why they gave it that score. If everyone agrees on the score, that’s great. If you don’t agree, this might indicate that you have a bad interview question, a bad rubric, or both. You can either rework the question, adjust the rubric, or simply take out the question in order to remove subjectivity. 

Why Developers Hate Technical Interviews

It’s well-known that many engineers, particularly senior engineers, do not like technical interviews. There are numerous articles written by engineers that detail why technical interviews are hated by the exact people for which they’re designed. These reasons are also why technical interviews may actually be hurting companies. “Technical interviews are feared and hated in the industry, and it turns out that these interview techniques may also be hurting the industry’s ability to find and hire skilled software engineers,” said Chris Parnin, an assistant professor of Computer Science at North Carolina State University and co-author of the research paper “Does Stress Impact Technical Interview Performance?”

By knowing why developers despise technical interviews, you can create interviews that are not only relevant to the role and effective in assessing candidates, but also help you gain respect and positive branding among engineers. 

They Exclude Certain Candidates

Technical interviews can exclude certain groups or favor others. A North Carolina State University and Microsoft study found that technical interviews test whether a candidate has performance anxiety, rather than whether they’re competent at coding.

Half of the study participants completed a traditional technical interview where they were asked to solve a problem on a whiteboard and explain each step of their process to an interviewer. The other half solved the problem on a whiteboard in a private room. They didn’t have an interviewer looking on and they were not required to explain their answers aloud. 

Participants who took the traditional interview performed half as well as participants who interviewed privately. “In short, the findings suggest that companies are missing out on really good programmers because those programmers aren’t good at writing on a whiteboard and explaining their work out loud while coding,” said Parnin. 

Candidates who have performance anxiety is just one example of how technical interviews may intentionally or unintentionally exclude certain candidates or entire groups of candidates. There are many other ways that this can occur. For example, interviewers may give easier questions to candidates who they prefer. The study also found that all of the women who took the public interview failed, while all of the women who took the private interview passed. Our own data shows that code tests that rely on absolute completeness and correctness disproportionately impact underrepresented minorities and women. 

They Require a Lot of Preparation and Time

Technical interviews often require a significant time investment from candidates. Considering that people often apply for multiple jobs at once, the amount of time that they spend on completing take-home assignments quickly adds up. Bennett Garner of Developer Purpose said, “My rule of thumb: If a take-home assignment takes more than an hour (2 hours, max) and is unpaid, then you have better ways you could use your time.”

Many candidates also spend weeks or months preparing for the technical interview. “[The technical interview process] gives a particularly large advantage to people who can afford to take the time to focus solely on preparing for an interview process that has very little to do with the nature of the work itself,” said Mahnaz Behroozi, a Ph.D. student at North Carolina State University and first co-author of the research paper “Does Stress Impact Technical Interview Performance?”

They Don’t Test for Relevant Skills

Some technical interviews don’t test for the skills that are relevant to the job, or they don’t provide candidates with enough time to correctly solve the problem. These problems with technical interviews can often mean “you hire people who are good at interviewing, not necessarily people who are good at the job,” said Garner. 

In particular, coding tests only capture a small sliver of what senior engineers will do in their jobs. Senior engineers often help and mentor junior engineers, identify systemic problems, debug complex issues, and code within complex systems and intricate work. None of these things are tested for in a coding interview. 

Best Practices for Technical Interviews

This guide is already filled with lots of tips on how to effectively and fairly interview and evaluate candidates. These additional best practices will help you further optimize your technical interview process and increase your hiring yield.

Create Interview Content That’s Researched, Tested, and Continuously Updated

Strong interview content gives you a true hiring signal that’s based on the candidate’s abilities and decreases cheating in interviews. As AI becomes powerful and prevalent, it’s increasingly important for your interview content to be designed with AI in mind. For example, put your interview questions through ChatGPT and see how it does. Questions that are easily solved by AI are unlikely to help you separate strong candidates from weak ones. To fix this, focus your interview content on logic and critical thinking problems where the candidate needs to generate a solution and then translate it into code. 

As technology and the field of software engineering advances, your interview content needs to keep up too. No interview question remains effective forever, so you’ll need to regularly review your questions and remove any that get leaked or are deeply flawed.

Allow Candidates to Use Common Tools, But With Limitations

Some companies may want to prevent candidates from using ChatGPT, Stack Overflow, Google, or other tools that could help them during the interview. What’s important though is not whether candidates can use them, but rather what they can use these tools for. 

For Karat interviews, we tell candidates that they can use reference materials to answer syntax questions, look up language details, and interpret error outputs from compilers. Developers do this regularly in their work, so why should they have to struggle without these resources during an interview? However, we ask that they not look for a full solution or copy and paste code from elsewhere directly into their solution. 

It’s more important that candidates can explain the code they’re writing and why they’ve written it than producing the right code. Referring to other code as a refresher doesn’t invalidate their understanding of the problem and their solution.

Offer Second Chances

Traditionally, interviewing is based on the belief that a single interview should allow candidates to adequately display their skills. In reality, there are many factors that can impact how well a candidate performs. They may not be feeling well, they may experience technical difficulties, or they may have never experienced a technical interview before. 

Giving candidates the opportunity to redo their interview not only results in better performance, but it also helps companies hire faster and progress on their diversity, equity, and inclusion goals. Redo interviews are a cost-effective way of making more hires while also supporting underrepresented talent who may be less familiar or comfortable with technical interviews. They also reduce candidate stress and put candidates at ease, resulting in a better candidate experience. 

Karat offers every candidate the option to redo their technical interview if they’re not happy with how they did in the initial interview. We’ve seen over 1,000 candidates hired after completing a redo interview, and our clients have increased their overall hiring yield by 17% due to the redo. The impact is even more significant with underrepresented talent. Black candidates are 30% more likely to take a redo interview than White candidates, and nearly 60% of Black candidates improve their scores in the redo. 

Redo interviews can make a big impact toward creating a more efficient and equitable hiring process, and the results make it clear that every company should offer redos.

Be Transparent With Candidates

Help candidates succeed and showcase their best abilities and work by setting clear expectations as to what the interview process will entail and the competencies that are required for the job. This gives candidates time to prepare, puts them at ease, and allows them to confidently go into the interview. 

Being transparent also ensures that there’s a mutual fit if the candidate is hired. When candidates know what to expect in the role, what growth trajectory is available, and what it’s like working at your organization, they can opt to exclude themselves. This is better than hiring someone, putting in the time and effort to train them, and then having them quit after a few months. 

Make Sure Your Interview Process Is Human-Centered

Technical recruiting relies on human connection. Companies don’t just hire developers based on their technical skills. They also take into account the candidate’s ability to collaborate and communicate. On the other hand, job seekers choose to join a company for many reasons, including the work culture and the team members they’ll work with daily. 

It can be easy to lose sight of this, but human connection continues to be paramount and companies should keep a human-centered experience at the heart of their interview process. Technical assessments can feel cold and impersonal, which can negatively impact the diversity of your talent pipeline. Our findings support this, as software engineers actually prefer live interviews over take-home tests.

By prioritizing human connection throughout your interview process, you can make it easier for candidates to say “yes” when you give them an offer to join.

Resources for Technical Interviews

In addition to this technical interview guide, Karat offers many more resources for talent acquisition and engineering leaders to help you create the best interview experience for candidates and hire great engineers:

• Understand the pros, cons, and best practices of using AI in technical interviews with our guide to Interviewing in the Age of AI. 
• Get The Interviewer’s Playbook to upgrade your team’s interviewing skills and build a solid interview program from scratch.
• Learn how to structure technical interviews to account for the latest developments in AI in our How to Build AI-Enabled Engineering Interviews webinar.  
• Understand why there’s an interview access gap for Black engineers and how you can change this with The Interview Access Gap for Black Engineers report. 
• Discover the top cities to hire software engineers, based on your hiring bar, resources, and geography. 

Conclusion

This guide sets you up to create technical interviews that generate a strong hiring signal, give all candidates a fair chance and a wonderful interview experience, and ultimately extend an offer to candidates who will be most successful in the role. If you’d like some more hands-on help in setting up or improving your technical interview process, Karat partners with organizations to confidently hire top talent. 

Our innovative human-tech Karat Interview solution combines thoroughly researched interview content, trained and experienced interviewers, flexible 24/7 scheduling, and a consistent interview process. As the world leader in technical interviewing, we’ve helped countless companies reclaim over a million hours in engineering time, reduce their time-to-hire by 50%, and achieve 60% more inclusive hires. Learn more about Karat Interview and how we can help.