The AI Skills Gap Nobody Talks About: Judgment

Most AI skills advice is about getting more out of the model: better prompts, more context, more agents running at once. That advice works. The problem is what happens next. You end up with more drafts, more code and more options than you can properly assess, and the quality of the final result depends on the assessing, not the generating.

Why is judgment the real AI skills gap?

Because output is no longer the bottleneck, and the work has moved to deciding what is good. When anyone can produce a reasonable draft, a working function or ten campaign ideas in a minute, the person who adds value is the one who can say which of them is right, which is subtly wrong, and which should never have been asked for.

The numbers on software teams show the shift clearly. Faros AI's AI Productivity Paradox report, based on telemetry from over 10,000 developers across 1,255 teams, found that developers on teams with heavy AI use completed 21% more tasks and merged 98% more pull requests, while pull request review time went up 91%. Average pull request size grew 154%, and bugs per developer rose 9%. Faros describes the slowdown as a human approval bottleneck. More output arrived; the capacity to judge it did not grow with it.

Developers already sense this. In the Stack Overflow 2025 Developer Survey, 84% of respondents use or plan to use AI tools, but only 3.1% highly trust the accuracy of the output, and the most common frustration, cited by 66%, is AI solutions that are "almost right, but not quite." Almost right is precisely the zone where judgment matters, because a quick look will not separate it from right.

What does judgment actually mean in AI work?

Judgment is four separate abilities that tend to get lumped together as "being careful". Splitting them helps, because people are often strong at one and weak at another.

Microsoft researchers saw the same shift from the other side. Their survey of 319 knowledge workers, covering 936 first-hand examples of generative AI use at work, concluded that AI shifts the nature of critical thinking toward information verification, response integration and task stewardship. In other words, less effort on producing, more on checking, combining and owning the result.

Why does AI make weak judgment more costly?

Because AI is excellent at some tasks and quietly poor at others that look similar, and you only find out which is which by judging. The clearest evidence is still the Harvard and BCG field experiment with 758 consultants, published as Navigating the Jagged Technological Frontier. On tasks inside the model's capability, consultants using AI completed more subtasks and produced responses rated more than 40% higher in quality. On a task deliberately chosen to sit outside that capability, the control group got the answer right about 84.5% of the time, while the AI groups scored 60% and 70%, an average drop of 19 percentage points.

The people in that study were highly skilled. What hurt them was not a lack of knowledge but trusting the tool in the one place it did not deserve trust. That is a judgment failure, and it is the expensive kind, because the output looked as good as everything else.

The Microsoft study points at the mechanism. It found that higher confidence in generative AI was associated with less critical thinking, while higher confidence in one's own abilities was associated with more. The more you lean on the tool's apparent competence, the less you check, and the less you check, the less practice your judgment gets. I see this as the real risk behind the worry about junior developers, which I covered in junior developers in the AI era: the issue is not that AI does the work, it is that nobody learns to judge the work.

How do you build judgment deliberately?

You build it the same way you build any skill: repetitions with feedback. These are the exercises I use myself and suggest to people I work with. None of them takes long, and all of them work with whatever AI tool you already use.

• Predict before you prompt. Before asking the model, write down in one or two lines what you expect a good answer to contain. Then compare. Over time the gap between your prediction and the output becomes the most useful signal you have, because it shows where you and the model disagree.
• Generate three, rank them, defend the ranking. Ask for three different approaches to the same problem instead of one. Rank them and write one sentence on why the winner wins. This trains evaluation, and it stops the first draft becoming the default.
• Find the flaw before reading the explanation. When an agent hands back code or a document, look for the weakest point before you read its summary of what it did. The summary is written to persuade, and reading it first anchors you.
• Keep a "confidently wrong" log. Every time the AI was wrong in a way that looked right, note the task type and what gave it away. Patterns appear quickly, and they tell you where the jagged frontier sits for your own work.
• Do some reps without AI. Write the occasional function, email or analysis by hand, then compare with the AI version. Judgment needs a reference for what good looks like, and you only keep that reference by doing the work sometimes.
• Write the decision down. For anything that ships, record in a line why it shipped and what would make you reverse it. Decisions you can explain are decisions you made; the rest were made for you.

Where does judgment show up in day-to-day AI work?

It shows up at the hand-offs: before a task goes to the AI, and when the result comes back. The habits that matter most in practice are small.

• Choosing what to hand off at all. Not every task belongs with an AI, and the decision depends on risk and how easy the result is to check. I set out a simple matrix for this in delegating to AI.
• Sizing the review to the risk. A copy change and a payment flow do not deserve the same scrutiny. Triage is a judgment call, and my system for reviewing AI-written code is built around making it explicit.
• Saying no. Some of the most valuable calls on client projects are not to build something: an AI feature with no way to measure whether it works, or an automation for a process that should be removed rather than automated.
• Knowing when to throw output away. If a result needs heavy repair, regenerating from a better brief is often the better call. Sunk effort is a weaker argument when generation is cheap.

Can AI help with judgment, or only with output?

It can help a lot, as long as it supports your judgment rather than replacing it. A second model reviewing a diff against a spec, a checklist run by an agent, or an evaluation set that scores outputs automatically all catch things a tired human misses. I use all three.

What AI cannot take over is accountability. A reviewer agent can tell you a test was weakened; it cannot decide whether the trade-off in a data migration is acceptable to the business, because it does not carry the consequences. The research on developer speed, which I summarised in does AI make developers faster, keeps arriving at the same point: the gains depend on the system around the tool, and judgment is the part of that system only people can own.

That is why I think of judgment as a career skill rather than a soft one. Clients, employers and teams increasingly have access to the same models. The difference between two people with the same tools is which of them can be trusted to say "this is right" and be correct.

Work with me

I help teams put AI into their products and workflows with the review steps, checks and decision points that keep quality high as output grows. See AI integration and AI agent development, browse my work, and book a scoping call if you want a second opinion on where judgment belongs in your process.