Software Developer Skills Assessment Is Broken in the AI Era. Here’s What We Do Instead.

QUICK ANSWER

A software developer skills assessment in 2026 can’t be a coding test anymore. AI made any online test easy to cheat, and writing code stopped being the skill worth screening for. The thing to assess now is product thinking, judgment, and communication, and you find it through real conversations. We even use AI to help grade those conversations.

For years, assessing a developer was simple. A candidate came into the office, we sat down, we had a good conversation. Sometimes we went to the whiteboard and had them sketch an architecture or walk through some code. In an hour I knew most of what I needed to know. I could see how they thought, not just what they had memorized.

Then everything went remote, and the room went away.

Then came scale. At Full Scale, there were months when a single job posting drew close to a thousand applicants. You cannot have a screening call with a thousand people. So we did what almost everyone did. We reached for the coding test.

That whole system held up for a long time. In the AI era, it fell apart.

How developer skills assessment used to work, before AI

The in-person version worked because it tested the right thing without anyone calling it a test. You weren’t grading syntax. You were watching a person reason out loud, ask questions, get stuck, and find their way out.

The whiteboard was the real assessment. The code on it barely mattered; what counted was the thinking around it. Did they ask what the system was for before they started drawing boxes? Did they push back when the requirements didn’t make sense? Did they explain their choices in a way another human could follow?

Remote work flattened all of that. A video call is fine for a conversation, but you lose the room, the body language, the easy back-and-forth at the board. And once hiring went global and remote, the volume went up too. The old hour-long, in-person read on a candidate didn’t scale to a thousand applicants a month. Assessment is only one piece of the bigger question of hiring software engineers, from where you source them to how you make the offer.

When a thousand people apply, you reach for the coding test

We used coding tests to screen at volume, the same as everyone else.

It started with multiple-choice coding quizzes. Then platforms like HackerRank and Codility added small online coding projects that the platform graded for you. You could put hundreds of applicants through the same test without a human in the loop, then look at who passed.

I’ll be honest about what that was actually good for. It was never great at finding the best developer. It was good at one thing: weeding out the genuinely bad ones. And it is crazy how many bad applicants you get when a thousand people apply for one role. The test was a sieve for the bottom of the pile, and at that volume, a sieve is worth a lot.

A coding test was always a filter for the worst applicants, not a spotlight on the best ones.

That distinction matters, because it’s the part most companies got backwards. They started treating the test score as the signal, when it was only ever a way to clear out the noise.

AI broke the coding test in two different ways

The coding test had its problems, but it did a job. AI took the job away. It did it in two separate ways, and the second one is the bigger deal.

Anyone can cheat any online test now

This is the obvious one. A candidate alone at home with a coding test and a second browser tab is going to use AI, and most of them will. HackerRank’s own research found that 83% of candidates said they would use AI on an assessment if they thought they wouldn’t get caught. That isn’t a fringe problem you can ignore. That’s most of your applicant pool.

The platforms are in an arms race now, adding proctoring and AI-detection to catch it. You can spend your budget trying to win that race. Or you can ask why you’re testing the one skill a machine is best at in the first place.

Code stopped being the skill you’re hiring for

Here’s the part that actually changed my mind. Even a perfectly proctored, impossible-to-cheat coding test would be testing the wrong thing.

AI writes the code now. Most developers already use AI tools every day, per the Stack Overflow Developer Survey. On a lot of teams, including ours, none of us really sit and write code by hand the way we used to. So screening for raw coding speed is like hiring a driver by testing how fast they can saddle a horse. It is also why a software engineer competency matrix now has to weight judgment and review over raw output.

What I’ve watched happen is that software engineering turned into software development. The job is no longer “here’s a ticket, write the code.” A developer now has to be part of the product direction: what are we building, and why. Then they build it with AI, test it, get it in front of a customer, and react to the feedback. They’re involved in more of the whole process than ever.

That takes communication. It takes understanding how people think and how they solve problems. As I’ve said before, pure coders will be replaced by AI, and problem solvers will run technology organizations. The developer shortage everyone keeps writing about was never really about people who can write code. The scarce ones are the people who can run the whole process, from a fuzzy problem to a shipped product.

The best senior developers won’t take your coding test anyway

There’s one more problem with the coding-test approach, and it has nothing to do with AI. It’s about who the test scares off.

The best senior developers already have good jobs. They are not going to do your weekend homework project, or sit a two-hour timed test, on the chance that they might want to work for you. They have nothing to prove and a lot to lose. So they opt out.

Which means a heavy test process quietly filters out the exact people you most want to hire.

A bad assessment process doesn’t just pick the wrong person. It scares off the right one.

That’s the risk we decided we weren’t willing to take. The last thing I want is to lose a shot at a great developer because we put them through a process they found insulting. A junior fresh out of school will jump through hoops. A senior with fifteen years of shipped product will just say no, and you’ll never know what you missed.

What a software developer skills assessment should test now

If code isn’t the thing, what is? For us it comes down to product thinking, judgment, communication, and problem solving. Can this person understand a problem, weigh the tradeoffs, work with other humans, and make a good call when the spec is fuzzy?

This is the whole argument of my book, Product Driven: the hard part was never the code. It maps to what I call the Three C’s of software development going forward, which are communication, curiosity, and courage. Those are the human skills that decide whether a developer succeeds once the machine handles the typing. As I keep telling our own engineers, the hard part of software development isn’t writing code. It’s understanding the problem you’re trying to solve.

Here’s the shift in plain terms.

What we used to test	Why it’s weak now	What we test instead
Can they write correct code fast	AI writes it; the test is easy to cheat	Can they decide what to build and why
Algorithms and data-structure trivia	Rarely the daily job; memorizable	How they reason through a real, messy problem
Framework and syntax recall	AI fills the gaps instantly	Curiosity: how they’re adapting to AI in their own work
Solo performance on a timed test	The job is collaborative	Communication: can they explain a decision to a human

None of this means technical depth stopped mattering. It means depth shows up in judgment now, in the choices someone makes on a messy problem. How many syntax questions they can answer from memory tells you almost nothing about that. A developer who can reason about the difference between a junior, mid-level, and senior engineer on a real problem tells you more than any auto-graded score ever did. The same goes for the soft skills worth looking for, which now carry most of the weight in how we evaluate someone.

Now half the interview is making sure the person is real

There’s a newer problem, and it’s the one that still surprises me. We hear horror stories of companies interviewing someone, hiring them, and finding out the person wasn’t who they said they were at all. Sometimes it’s a North Korean operative using a stolen American identity to land a remote dev job. The FBI has warned this is happening at scale, with hundreds of U.S. companies caught up in fraudulent remote IT worker schemes. It sounds like a movie, but it’s a normal thing to have to screen for now.

Half the job of an interview now is just verifying the person on the call is who they say they are.

On top of that, there are tools built to feed someone answers during a live interview. The candidate is on the call with you, having what feels like a real conversation, while a tool listens to your questions and quietly tells them what to say. So even the back-and-forth I just spent this whole article recommending can be gamed.

The good news is that a real conversation is also the best defense against both. An impostor and an AI-fed answer share the same tell: the words come out sounding rehearsed instead of real, a beat too slow, or weirdly disconnected from what you actually asked. So go off-script, interrupt, and ask a follow-up they couldn’t have prepared for. A canned test can’t do that, but a person in a real conversation can. That’s exactly why we lean on the conversation.

How we assess developers at Full Scale now

So we went back to basics, with one modern twist.

The basics: we review the resume like humans, then we have a couple of rounds of online meetings and real conversation. There’s no homework project and no timed gauntlet. We talk to people, because the entire point is to find good developers. A process that scares the good ones off is worse than no process at all. Our hiring process is built to give a strong candidate a fair shot.

The twist is the AI part, and it’s the opposite of what you’d expect. We use a tool called Metaview to analyze the interview conversation. It helps us judge how well the candidate answered the questions, and it organizes what they said by the things we actually care about, like communication and problem solving.

It also scores our side of the table. Metaview helps us see how well our own interviewer asked the questions, so we can get better at interviewing too.

We don’t expect candidates to use AI to cheat our test. Instead, we use AI to grade the conversation.

That flip is the whole point. The candidate is being assessed as a human, on human skills, in a real conversation. The AI is on our side, making us sharper and more consistent about what we heard. It’s the same approach behind every developer we place. We’ve assessed and placed hundreds of engineers, and our retention sits at 93%. You only get a number like that when you read people right at the start. Every developer we put on a client team has already cleared this process. That’s the whole point of our staff augmentation model: you get a pre-vetted offshore team instead of hiring blind.

Where coding tests still earn their place

I don’t want to oversell the death of the coding test. It still has a use, as long as you’re honest about which one.

At real volume, when a thousand people apply and most of them are not qualified, an automated screen still helps you clear the obvious no’s. For high-volume junior screening, it’s a reasonable first sieve. It can be a quick sanity check that someone can write basic code at all.

What it cannot be is the thing that decides your senior hire. Let it filter out the obviously unqualified if you must, but don’t let a test score make your most important hiring call. The moment it does, you’ve handed that call to a machine. And writing code is the one part of the job a machine already does better than the person you’re trying to hire.

How we assess skills has completely changed, and it changed because the job did. Software development is back, and the companies that remember what that ever meant are going to win.