Hiring software engineers is notoriously challenging. Whether it’s a lack of time to conduct technical assessments, inability to find the right candidates in the first place, or lack of reliability in the recommendations — it’s hard to find an approach that works.
Most technical hiring processes start first-round technical assessment:
- Code tests
- Take-home technical assessment
- Live technical interviews
Below you’ll find a guide to each type of technical assessment, including how to use them to hire software engineers and determine final recommendations.
1. Code tests
What is a code test?
Code tests are timed, automated assessments used by technical recruiters to filter candidates in or out at the very beginning of the hiring process. They require candidates to write and run working code that solves a single question or set of questions.
Technical recruiters will typically invite candidates to complete the test via an email invitation. The most common technical code tests are Hackerrank, Codilty, and CodeSignal.
How are recommendations determined?
Candidates are on their own to write and run solutions to questions presented to them in a code test. Most code tests produce a binary pass or fail recommendation based on absolute completeness and correctness.
These limitations can negatively impact candidates from underrepresented and nontraditional backgrounds. For example, if a candidate’s solution uses the correct algorithm and data structures but includes a minor error like a typo, it will fail many code tests.
When should teams use a code test for a technical assessment?
Code tests are useful in high-volume hiring programs that focus on junior developers. University Recruiting programs that recruit new grads frequently use code tests. Some use a combination of a code test and technical interview to determine which candidates to advance or reject.
Code tests are not an ideal technical assessment for specialized engineering roles and more senior candidates. These candidates are less likely to sign up for the code test because they associate it with junior candidates and feel it will not represent their skills. And they’re right — code tests produce less reliable recommendations. This is because the nature of code tests relies on a coding exercise rather than a discussion on business logic complexity or code review. The latter is more relevant to specialized and senior software engineering roles, like software architects and DevOps.
2. Take-home technical assessment
What is a take-home technical assessment?
A take-home test is a coding problem that candidates have multiple days or weeks to complete.
Candidates are allowed to use their own programming environments. Sometimes take-home tests include similar questions as code tests to explore algorithms and data structure knowledge. Other times, because of the additional time and flexibility given to candidates, they ask candidates to build or extend applications with tools they use on the job.
How are recommendations determined?
Take-home tests are reviewed and scored by the in-house software engineering team. Some teams use an interview rubric aligned to specific competencies to score the test and make a recommendation. Others go by their own experience and gut. The latter creates a haphazard approach that can produce noise and bias.
When should teams use a take-home technical assessment?
Take-home tests can be handy when:
- Candidates need an alternative assessment. Candidates (often) find interviews nerve-wracking and don’t have other code available to share with the hiring team. Take-home tests allow them to show what they can do without a time limit. Though, you’ll still want to have the candidate do a live technical interview in the next round.
- Hiring teams want to screen candidates for development work using personal tooling and infrastructure or open-source inspired communication best practices. They may aim to screen for dedication to a hiring process that can require as many as 10 or 20 hours before a live interview.
Take-home tests may not be useful if you are concerned about:
- Noise: Interview rubrics are seldom shared with candidates. Candidates are often told to do X and then marked down because they didn’t do Y and Z.
- Bias: Interview reviewers often use the best take-home test results as the gold standard they apply to all other candidates. This becomes noise, like in the example above.
- Lack of time: The time needed to score take-home tests manually can dramatically slow down the hiring process and prevent software engineers from doing their day jobs.
- Candidates dropping out: Candidates frequently report that take-home tests are too time-consuming and feel that they are giving away their code. Many candidates fail to return take-home tests and drop out of the hiring process.
- Fraud: Take-home tests can lead to fraud because they are easy to share and discuss. Fraud increases false positives and harms onsite-to-offer ratios.
3. Technical interviews
What is a technical interview?
Live technical interviews are the most human-centered technical assessments. Not to be confused with a phone screen, technical interviews should include questions and problems that are relevant to the role and are solved with the guidance of a professional interviewer.
The ideal technical interview:
- Is no more than 60-minutes in length
- Uses an Interactive Developer Environment (IDE)
- Uses consistent interview questions
- Uses video chat for the interviewer and candidate to talk to each other while coding
How are recommendations determined?
Interview rubrics mapped to the technical interview are used to produce recommendations. They describe competencies relevant to the role and what different levels of mastery look like. Candidates can receive partial credit, too.
The score drives the recommendation. Each indicates the likelihood of the candidate getting an offer. For example, here are Karat’s recommendations:
- Fast Track: Top candidate with a strong chance of receiving an offer
- Invite to Next Round: Candidate exceeded the technical bar for this role and should be moved to the next step in the hiring process
- Requires Further Review: Candidate may be below the technical bar. Seek input from an engineering hiring manager
- Not Ready Yet: Candidates has a low chance of receiving an offer and may not be ready for the role
When should teams use a technical interview?
Hiring teams will need a technical interview during their first-round technical assessment to increase candidate completion rate, eliminate false negatives and false positives, and save software engineering time throughout the process.
Here are a few tips to be successful with live technical interviews:
- Competencies: Determine the necessary competencies and skill level for the role. Then build your job description, interview questions, and interview rubric.
- Alignment: Engineering leaders should share the interview rubric with the technical recruiting team. This will help them in sourcing and developing a sustainable talent pipeline.
- Interviewer training: Leveraging software engineering team members who want to interview and are trained to do so will improve candidate experience and hiring outcomes.
For more information about the pitfalls, pros, and cons of different technical assessments, check out “Technical Recruiting: The Good, The Bad, and The Ugly.”
Final considerations for using technical assessments to make your next developer hire
- Beyond keeping the assessment relevant for each role, consider every possible task a new hire will have. This ensures you’ll have the right competencies for the role so you can evaluate their performance completely.
- Let candidates know what kind of assessment they’ll need to do. Particularly when it comes to live interviews or real-time coding tests, candidates should have enough time to prepare.
- Use both a code test and a live technical assessment for junior developers. This will improve efficiency of high volume technical recruiting processes, like University Recruiting.
- Consider working with technical interviewing experts. At Karat, experienced Interview Engineers take care of this task on your behalf. This gives you access to developers who use interview questions and rubrics calibrated to your technical bar so you can hire the best developers for you — no matter the seniority level.
