1200 Technical Interviews and Counting: John Scully’s Journey from Software Engineer to Interview Engineer

Over two years ago John Scully stumbled upon Karat’s Interview Engineer job listing on Stack Overflow. At first, he couldn’t believe that interviewing software engineers was actually a job. Then he decided to apply to learn more.

John was looking for a role he would not only find rewarding, but as the father of 10 kids, could also provide work-life harmony. He found that the role of an Interview Engineer offered flexibility, but also a huge opportunity to play a pivotal role in increasing access to a structured, consistent, and fair technical interview.

Since John joined Karat, he has conducted over 1,200 technical interviews — and continues to conduct over 20 every week. And, as a senior Interview Engineer and mentor, he has completed over 190 quality checks of other technical interviews.

We chatted with John about why he became a Karat Interview Engineer and what he feels software engineering teams should know about the technical interview. Here’s what he had to say.

What motivated you to become an Interview Engineer?

The benefits to software engineers are huge. I thought that conducting technical interviews on behalf of companies was a great idea. It’s an incredible offering — removing the burden of interviewing from overworked development teams is something developers really appreciate. They get professional, consistent, technical interviewing without affecting their development time. Karat is a 24/7 operation, which is helpful for candidates because they can be interviewed from home. For me, the flexibility of the Interview Engineer role is special and much appreciated.

“If you have 10 candidates interviewed by five different people, how do you know who is the best fit?”

What do you appreciate most about the Karat interview approach?

Structure and consistency. Without it you might interview a few candidates with different interviewers. The interviewers might ask different questions and have different insights into the candidate’s performance. Then it becomes a decision based on personality rather than technical ability. It becomes hard to compare. If you have 10 candidates interviewed by five different people, how do you know who is the best fit?

Having a live coding exercise is helpful. The video aspect of Karat interviews lets me see how a candidate talks about their projects and problems. It’s better than a take-home homework assignment test because I get more information than just the code they submitted. I see how they approach problems and if they can articulate their approach.

Some of the people I’ve interviewed through Karat have tons of experience but struggle to write code that works. With Karat, we’re able to arrive at a consistent comparison between candidates and their abilities.

How has the role of Interview Engineer helped you grow professionally?

I watched a few of my first interviews, and I said “um” and “uh” a lot more than I do now. My delivery has definitely gotten smoother over time. When I was being on-boarded to become an Interview Engineer, I received feedback on my starter interviews from seasoned interviewers. They helped me learn how to score questions using a rubric and how to use a consistent, calm in-interview communication style. I also learned what is most important to mention in write-ups following the interview.

It’s always interesting to see new questions for new roles and organizations. It’s helping me keep up with technology as well. When a candidate and I discuss their approach to solving a problem or completing a project, I talk with them about what technology they’re using, what problems they’re trying to solve and what tools and modalities are popular. I’ve learned a lot about machine learning through Karat interviews. When the question qualification came up, I took a course about it, studied the questions and listened to candidates responses.

I’m cross-training myself on a regular basis. A part of that involves looking at which technologies are becoming more common, which ones are solid to use and trying to find innovative ways to tackle problems. As I learn something new, I think, “How can I apply this to another problem?” For example, I started using React, a JavaScript library after talking to some of the candidates.

“One trap that some companies fall into is carrying on with unstructured and inconsistent technical interviews that, in the long run, can become adversarial.”

Why do you think professionalizing and structuring the technical interview is important?

One trap that some companies fall into is carrying on with unstructured and inconsistent technical interviews that, in the long run, can become adversarial. Developers are working on a project and when they need help, they end up looking for a clone of themselves. Many are prone to taking the following point of view: “I know my problem very well and you, the candidate, are coming here to help me with my problem.” What you end up with is an unprofessional interview process that is very anchored around a subjective perspective, rather than the team’s. When software engineers get hired, their interviewing skills aren’t taken into account, although they’re often put in the position of interviewing.

What are some ways companies can make their technical interviews consistent across interviewers and teams?

I always tell people that three things are necessary for ensuring consistency in interviewing:

1. The same questions must be used across all interviews and interviewers and they must be specific to a role.
2. Groom your data and analyze it to help make better hiring decisions. Use insights from your interview process to gain a wider perspective of your decision-making process.
3. Do a retrospective. Ask yourself, “Did you make a good hiring decision? Is there a way we could make interviewing better?” Because creating a consistent and predictive interview requires constant reflection.

When you use these best practices, you’ll be much more likely to conduct interviews that can accurately identifying a candidate’s strengths. You’ll be setting up your team and the candidate for success at the onsite and on the job.