Webinar insights: harnessing the power of genAI in tech hiring

The recent webinar, “Harnessing AI and the future of tech hiring,” featured an enlightening discussion about the future of generative AI, or genAI, in software development and engineering hiring. Moderated by Scott Bonneau, EVP of Product and Operations at Karat, the webinar brought together industry leaders Sagnik Nandy, CTO at Okta, and Eddie Zhou, Founding Engineer at Glean, to share their insights on the applications, challenges, and implications of incorporating genAI into engineering hiring and software development processes. This article recaps the highlights from the webinar, shedding light on how generative AI is poised to reshape the landscape of engineering and beyond.

Understanding genAI’s role in software development

The conversation kicked off with a succinct introduction to generative AI, highlighting its ability to process vast amounts of information and simulate human-level reasoning. 

Both panelists delved into their organizations’ experiences with integrating generative AI into practical workloads. Sagnik Nandy discussed pilot programs leveraging tools like Copilot and context-driven information to streamline onboarding and operational efficiency. Eddie Zhou shared how his team used their own product, Glean, to accelerate tasks, particularly in templated or batch projects, highlighting AI’s role as a significant accelerant.

Nandy also noted that today’s generative AI systems can serve as the backbone for agents that mimic human reasoning, making them more advanced than language models of the past. This fundamental difference is paving the way for innovative applications across various domains, including software development.

Balancing AI-specific roles with foundational skills

A noteworthy poll question presented to the 300-plus webinar attendees revealed a balanced split regarding hiring for AI-specific roles. This result underscored the ongoing evolution of AI in the engineering sector and raised questions about how AI’s role intersects with traditional skill sets.

In fact, Eddie Zhou emphasized his team’s focus on hiring strong software engineers and thinking about machine learning and AI engineering as supplemental skills they need on top of strong fundamentals — which he assesses without the use of AI coding tools.

“All of our ML engineers are all software engineers in title. Everyone is held to the same software engineering bar. We also have additional AI-specific interviews including ML systems design or ML foundations,” shared Zhou. “For us, those are additive skill sets, and that’s because we’re focused on hiring horizontal ML engineers who are capable of building both the model and systems that incorporate those models. This horizontal persona can be a lot more flexible in terms of finding people that can integrate intelligence into a product. So from a hiring space, AI and ML is very similar to traditional hiring with an additional set of evaluation that involves a little more statistical and mathematical foundations in addition to the problem solving and algorithm side.”

Nandy highlighted the need to evaluate candidates’ depth of AI knowledge and problem-solving skills, especially in scenarios that demand thoughtful design and creativity. 

“We have subject matter experts who go a lot deeper, but also AI-aware generalists. People who are injecting intelligence and machine learning into the software development process,” Nandy said. “For that group, we’re making sure people can deliver what they claim to. Everyone claims to have some form of AI experience today, and filtering out how relevant and real it is. AI can be more of an art. It’s easy to get programming feedback in terms of will the code work versus how do you solve a feature problem where the outcome wasn’t what you expected. Teasing that out to understand the level of depth is something we are injecting into the process.” 

Human interviewers can assess problem-solving beyond genAI

While acknowledging the utility of AI tools like Copilot for day-to-day tasks, Zhou shared that Glean discourages the use of ChatGPT during interviews. Instead, the company focuses on assessing candidates’ core skill sets without relying on AI. This is done through human interactions, rather than any kind of experimental AI-detection software. This also echoes what Karat researchers have found in their test interviews with genAI tools like ChatGPT.

Having strong interview content that is researched, tested, and continuously updated based on the latest developments is one way to ensure you’re getting a true hiring signal on a candidate’s abilities. As Scott Bonneau shared, “ChatGPT is really good at taking an English language description of a problem and turning it into code. And if all you’re asking someone to do is translate from English to Python or JavaScript or C, ChatGPT can be really good at that. The interview content and types of questions you’re putting in front of candidates needs to evolve. If that’s what you’re seeing in your process, try focus more on problems that are about logic and critical thinking where you generate a solution and then translate that solution into code.”

Rather than analyzing static code samples, Zhou stressed a human-plus-technology approach that involves coaching interviewers to increase the frequency of their check-ins with candidates to probe their approach and problem-solving abilities, ensuring a more accurate evaluation of candidates’ true capabilities. 

“Ultimately that process isn’t as difficult as it sounds,” noted Zhou. “The reality is having a human probing how candidates are solving problems at any kind of depth will be very apparent. It’s just like knowing if a candidate has seen a problem before and is just regurgitating an answer. We’ve been able to acknowledge it and continue hiring at a healthy pace.”

The future of genAI and software development

The webinar concluded with a forward-looking perspective on the future of genAI in software development. The panelists acknowledged the ongoing evolution of AI technology and the potential for disruption and innovation. They emphasized the need to stay adaptable and reevaluate strategies regularly, given the rapid pace of change in this dynamic field.

As AI continues to advance, it’s clear that embracing its potential while maintaining a balanced approach will be key to realizing its benefits in the engineering landscape and beyond. The full webinar is accessible on demand. 

The conversation continues

Join us for Part Two of our series on genAI and the future of tech hiring, featuring Jocelyn Lai, Global Head of Talent Acquisition at Duolingo; Chad MacRae, VP of Talent Acquisition at Tinder; and Elena Stefanopol, Director of Recruiting at Labelbox on September 13. Sign up for the September 13 event today.