How We Interview Developers at Full Scale: The Rubric We Use

In the first six months of this year, 4,869 people applied for a job at Full Scale. We only hired 43 of them.

That’s under 1%, and honestly, the 1% doesn’t make much sense as a number. It includes a lot of junk applications: people who live in countries we don’t even hire from, applicants for roles we aren’t hiring for, and applicants that don’t meet our most basic job requirements for the roles we are hiring for. Only 430 of the 4,869 matched a job we had open and got through a basic phone screen. That means about 90% failed on the basic filters before they reached an in-depth interview.

You can see why companies like ours say they only hire 1% of applicants. It is kind of easy to tell whatever story you want from the numbers. Especially when bots are spamming every job opening there is.

The 430 is the part that matters, and it is a number our team can actually manage. We can’t possibly interview 4,869 people. The interviews happen with the few hundred who pass the basic requirement filters.

Here’s what happens to those people, including the rubric we score them against for 2026.

The tests worked, and that was the problem

Like a lot of companies, we used to use online screening tests and live coding challenges. In the past we have had more than a thousand job applications and it was a useful tool to screen out the worst applicants.

The problem is that the best developers don’t want to take them.

I say this a lot: the best people usually aren’t looking for a job. They already have one. It isn’t that there aren’t good people out there looking, but the very best ones somehow always already have a job. They’re not the ones getting fired, and they’re not the ones getting laid off. They’re overachievers at their current job at the top of every force rank.

Think about what you’re asking that person to do. They already have a job and they already have recruiters in their inbox every week. Now you want them to spend a Saturday afternoon taking a timed test to prove themselves to a company they’ve never heard of. The reaction our recruiters kept getting from senior candidates was some version of why are you making me take this, why not just interview me.

Fair question and we agree.

Kristine, who runs our recruiting team, said something similar when we looked at putting an AI filter on resumes. She told me she doesn’t actually like filtering, because the majority of the senior developer candidates structure their resumes in a way that’s very old, and she doesn’t think the AI would get the details right. What she was really saying is that software developers are not good at marketing, and they’re especially bad at marketing themselves. Her conclusion: “If we are going to add the filtering part, probably the majority of the candidates would actually fail the first phase.”

Build a filter tuned to catch resumes like that and you’ve built a filter tuned to reject the people you most want.

The test was doing its job. It was just doing it to the wrong end of the pool.

I’ll give the other side of this its due, because there’s a real version of the argument. If you’re getting fifty thousand applications a year with two recruiters, you can’t talk to everybody, and an automated filter is the only practical option you have. I’d probably run one too. Our situation is different in one specific way: most of the people we hire never came out of the application pile at all. They came from a recruiter finding somebody on LinkedIn who wasn’t looking that morning. When your best channel is outbound, a filter on the inbound pile mostly just repels the people you were trying to recruit.

Then AI finished the argument

Another thing that AI has done is render the tests almost pointless.

Consider the value of a take-home code challenge in 2026. Actually, your 2026 challenge will show whether your candidate knows how to use AI, a skill we expect from everyone we hire. Your screening challenge is just a tedious way of verifying something you can gather from a candidate in under a minute.

I’ve written about the impact of AI on technical hiring in the past, so I won’t go through all of it again. In this case, it is important to talk about what you are left with in the aftermath. Once the part of the interview where candidates write code is effectively faked, then the only competencies of the candidate that are left that are worth measuring are the ones that a LLM cannot provide for a candidate while a human is observing and interacting with the candidate in real time.

The things that always mattered now matter more

Interviewing developers now is mostly about asking questions that AI can’t really answer for them anyway.

We’re looking for problem-solving. Their actual experience. Can they tell us about something they solved before, and why did they solve it that way? Do they understand why they’ve been doing the things they’ve been doing? Do they have the courage to ask questions and push back when the requirements don’t make sense? Do they know how to talk to a customer and understand the customer’s problem? Do they have empathy for the customer? Do they make sure their work actually works all the way through to production, and do they take ownership when it doesn’t?

Those are the things we have to interview for now. And none of them are new. They’re the things that have always mattered. But because of AI and everything else that’s changed, they matter way more than they ever did before.

The only one I’ll pull out here is courage. I’ve been hiring engineers for twenty years, across two companies I helped build and sell before this one, and the developers I’ve valued most are the ones who came back and told me the requirement didn’t make sense. Courage is one of the five pillars in Product Driven for that reason.

The rest of them show up on their own in the interview answers further down. I’ve made the broader case for why communication, curiosity, and courage now beat raw coding talent elsewhere. This post is about the question that follows it: how do you actually test for these things, the same way, across hundreds of interviews a year?

What we actually score

We came up with six categories to score applicants by and weighted them. Most companies would probably never share this, but you might find it interesting!

Dimension	Weight	Hard rule
Communication and async presence	25%	A 1 disqualifies
Technical fundamentals	20%
Practical ability	20%	A 1 or 2 disqualifies
Project experience and ownership	15%
Delivery reliability and ownership	10%
AI proficiency	10%

Communication is the most important category, weighted at 25%. Our rubric enforces the key thing that we know: communication is the number one failure mode in offshore roles, and a technically strong engineer who communicates poorly struggles on the job. Their async remote presence is also critical.

If a candidate can’t show practical ability with basic software development concepts, they will be disqualified. If someone scores a 1 or 2 in that category, it doesn’t matter how well they performed in any other area. So dropping the coding test didn’t mean we stopped checking whether people can write code. We moved that check into a live conversation where a real person is watching, and made failing it fatal.

These six categories came from analysis of real client interview transcripts: what our clients actually probe for when they interview our candidates during final interviews. Every dimension gets scored 1 to 5 on demonstrated behavior, not potential. The full rubric, all thirty scoring anchors, is on our how-we-hire page.

The same question, two very different developers

The most useful part of scoring this way: you can ask two developers the identical question and tell almost immediately whether you’re talking to a junior developer or a senior one, purely from the shape of the answer. Not whether they got it right, because usually there isn’t a right.

The pattern is consistent. Ask a junior developer a question and they’re mostly concerned with the technical implementation: how to build it, how to make it work. Ask a senior developer the same question and they focus on scale, edge cases, user experience, trade-offs, and what happens if it fails.

Take a boring one: “How do you handle errors in your code?”

A junior developer answers the question you asked:

“I wrap it in a try/catch and log the error.”

That’s not wrong. It’s a 2. It’s the textbook answer, and it tells me they’ve written code but haven’t been on call for it.

A mid-level developer answers the question and one layer past it:

“Try/catch, log it with enough context to trace it later, and return something useful to the caller instead of a stack trace.”

That’s a 3, and a 3 is fine for plenty of roles. They’ve been burned by a useless log message before, which is its own kind of experience.

A senior developer stops answering and starts asking:

“What kind of error? If a payment provider times out, I don’t want a retry loop that double-charges somebody, so that path has to be idempotent before I touch anything else. If it’s a validation error, that’s not exceptional, that’s Tuesday, and it shouldn’t be sitting in the error log burying the ones that matter. Then the real question is what the user sees and whether anyone gets paged. Half the alerts I’ve inherited were noise nobody acted on, which is worse than no alert, because the team learns to ignore the channel.”

That’s a 5. Same question. Look at everything that showed up in the answer without anybody asking for it: money, the customer’s experience, the on-call engineer at 3am, and the fact that an alert everyone ignores is worse than none.

Nobody teaches that in a bootcamp. It comes from having shipped things that broke.

And the irony is that even that answer is still mostly about the how. The most important thing of all is figuring out whether the work delivers any value to the customer in the first place, and that’s where Product Driven really comes in. The developer who asks what the customer is actually trying to accomplish, before touching the retry logic, is the one everybody wants on their team. “Product thinkers don’t just ship code, they ask better questions.” You can’t get any of that from a screening test, because a screening test has a correct answer and these questions don’t.

A human runs the interview and AI checks the human

The process itself is straightforward. One of our engineering managers runs the technical interview, in the discipline they actually work in. Before the call, an internal tool builds a structured question set for that specific role and seniority, so two people interviewing for the same job get a comparable set of questions instead of whatever the interviewer happened to think of that morning.

The interview gets recorded in MetaView, and the rubric is matched against the transcript, and instantly processed. The AI score is compared to the human score. Everything comes back into Zoho Recruit, where our candidate database contains 15,000 people.

Using tools like this helps us take quicker action. The scored assessment is available at the conclusion of the call, allowing us to advance a good candidate to the next stage the same day instead of waiting a week.

Additionally, the analysis excels at determining whether or not an answer is internally consistent. A candidate claims they led a migration, while the transcript reveals they described three decisions made by someone else and no decisions of their own. A human would identify these inconsistencies on occasion. A model reading the transcript, however, always identifies these inconsistencies and flags them for the interviewer to investigate, rather than deciding anything by itself.

Human-only interviewing has real problems nobody advertises, and it’s worth being honest about them. Our engineering managers cover every discipline we staff, so one of them is regularly interviewing slightly outside their own stack. Interviews happen on good days and bad days. Two candidates for the same role get asked different questions by different people and then get compared as though the inputs were the same. That’s just what happens when a judgment call has no second reader and no record.

The interviewer still has the final call, always. Nobody at Full Scale gets hired or rejected by a machine, and nobody wants to be hired by a chatbot.

The rubric can’t see potential

I want to be clear about something, because it’s easy to read all of this and think we’ve turned hiring into arithmetic.

The rubric is just a guide. It’s a recommendation. It isn’t a hard and fast rule. We hire fresh college graduates who would fail nearly all of it, completely, because they’re fresh college graduates and they haven’t had the chance to do any of the things it measures yet. Somebody has to hire those people, and some of the strongest engineers on our team walked in exactly that way.

AI and rubrics like this one are a tool. You still need a human to make the final decision and to see the potential in somebody, which is the one thing an instrument built on demonstrated behavior can never do.

That’s also why we rely on our clients to do a final interview. You get people you believe in as long-term members of your own team, and it’s one of the things that makes us different. With some of our competitors you may never get to interview the talent at all. You just get assigned somebody.

We want to make sure it’s a good long-term fit, so it’s a win for the client and a win for our employee. We also want to make sure our employee wants to work for this client. Our whole staff augmentation model depends on people staying, so a match only one side is happy about does us no good either.

None of this made hiring easier

I don’t want to make this sound like we simplified our lives, because we did not.

We moved the work from filtering applications to interviewing people. Five technical recruiters run two to three interviews a day, every day. That’s a lot more human hours than an automated screen that rejects most of a pile overnight while everybody sleeps.

It costs us more and it finds better engineers, which for a company whose product is people isn’t a close call.

The rubric also does what a rubric is supposed to do, which is reject somebody who looks fine on the total. We had a candidate come in at 59% overall, which lands in our hold band rather than the no-hire band, and he was still recorded as a no hire, because practical ability came in at a 2 and that’s automatic. He explained concepts fluently and then stalled completely on the live exercise. The interviewer’s note read: talks well but could not perform when it mattered.

The point of publishing any of this

Everybody says they’re selective. It’s easy to say and hard to prove, which is why we’d rather just show you the thing we actually use.

The whole rubric, all thirty scoring anchors and two real scorecards, is on our how-we-hire page. And if you’d rather skip building all of this yourself and borrow the engineers it produces, tell us what your team needs.