Mercor’s “AI Job Interview” is interesting, but has some ways to go

I recently enrolled on Mercor, which does an “AI Job Interview”. This article is a blend of review, opinion, and perspective.

Photo by charlesdeluvio on Unsplash

Some light sleuthing over the Internet (aka Reddit) disambiguated a few things.

1. It was actually an AI “Job Interview”, the interviewer would be an AI voice agent.
2. The job was largely focused on general software engineering.
3. The interview would be based on the resume and would be alright.

With this information, I signed up to just experience it and curious see how it goes. I had two AI interviews, the first was a General interview and focused on resume and software engineering skills. The second interview as DSA (Data Structures and Algorithms) which is the mainstream primary topic for testing software engineers on some of the fundamentals for programming and computer science.

Strong technology, clear scope, and runway for improvements

Photo by Andrea De Santis on Unsplash

• The voice, whether generated with intermediate text or through deep encoding, was coherent, this is not a game changer, it is clearly above the level of old school IVRS you still hear in most of the customer helplines, but not much above the level of Google Duplex demo from few years ago. The needle has been moving on this, so they can keep improving this.
• The interview was just 20 minutes and hence pretty short, I had some mic connection trouble over Firefox, same as the ones many of us faced in the beginning of the pandemic, the interview orchestration should be able to handle such things better, with visual prompts etc.
• The first interview was a resume interview and hence the agent was required to be able to use information from my resume effectively to achieve a good interview. Sadly, the interview was solely focused on my most recent experience item, which in my case was a research engineering experience at Siemens, which is off the beaten path for software engineering applicants. The agent failed to catch the nuance, this is expected and something to improve on.
• There was a nice surprise in this interview, specifically when the agent oriented software engineering design questions off my research project, spinning-off onto standard questions of scale, security, versioning, etc. that one expects any senior engineer to be grilled in later rounds. The caveat was, my experience was not conventionally software engineering, and hence had the interview had a little less fidelity than I would have preferred. I am going to call this a grounding challenge, the AI interviewer has a limited understanding of the breadth of industry and professional experiences and hence cannot be expected to handle these nuances yet.
• The DSA interview however was better, the AI Interviewer led the discussion with data structure question, and hence was somewhat “more” in control and did not have to worry about pesky nuances. The AI Interviewer did a pretty good job of working down a standard recipe of data-structure based question, querying my understanding of a specific data structure, its working, the advantages and disadvantages, utility, and even alternatives. This is a place I would say it did shine.
• The AI interviewer's ability to hold a conversation is limited and non-robust. Mercor did a good job of warning me about it prior to starting the interview, which is an easy win on the candidate experience side.

Candidate experience needs urgent focus

Considering how new this process is, there are quite a few things that need to be improved on the candidate experience side, but I think they are doing a good job and moving in the right direction.

• Initial Sign-up needs to be better, I was not forced to get a good understanding before using it, I just had to upload a resume and use my Google account to do so. There was an almost critical lack of disclaimers and explanations. The understanding that it starts with “AI Interviewer” was not explicit or the nature of it, the onus of it is on the candidate, which is a friction that could be resolved.
• The privacy and usage rights of the recordings is effectively unknown. I had to schedule and attend the interview without a clear expectation of the privacy and usage conditions they plan to uphold and enforce, this is a problem we have seen time and time again in tech, especially AI, but does not excuse Mercor because of it. This is a clear are for them to act on, and hopefully act on fast.
• The talent pool is what you get added to after going through these interviews, the which is largely opaque to the candidate. They are doing a good job with sending a weekly blast about it, but other than that, there is neither much agency about your membership in the pool and ability to see your information on the other side (but it is open to the public if you dig just a little — another privacy and usage violation) easily.

A mixed bag, but interesting nonetheless.

Mercor is trying something truly innovative and interesting in the recruiting and sourcing area of tech. I am genuinely curious to see how it unfolds this year, if not beyond. They seem to have strong technology on their side, with clear direction of improvement. Their high-level candidate experience needs an urgent focus, especially an effort to establish a baseline of privacy for the candidates.