Why AI Agent Projects Fail, and the Evals That Prevent It

Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, because of escalating costs, unclear business value or inadequate risk controls. As that date approaches, I expect a lot of teams to be asking why their agent project stalled. This post is the answer I would give, and the fix.

Quality is the main blocker, and most teams do not measure it

LangChain's State of Agent Engineering survey of 1,340 practitioners, run in late 2025, found that 57.3% have agents in production, and quality is the top barrier to production at 32%, ahead of latency at 20%. In the same survey, only 52.4% run offline evaluations on test sets.

So quality is the main problem, and about half of teams have no systematic way to measure it. Developers feel this directly: in Stack Overflow's 2025 survey, the most cited frustration with AI tools was solutions that are "almost right, but not quite," at 66%.

"Almost right" is exactly what an agent looks like without evals. It passes the five examples the team tried by hand and fails on the sixth one a customer sends.

The four ways agent projects actually fail

These map closely to Gartner's three reasons.

• No definition of "good." The team never wrote down what a correct outcome is, so every review turns into opinion. This is "unclear business value" in practice.
• Silent regressions. Someone improves the prompt for one case and breaks three others. Nobody notices for weeks.
• Uncontrolled actions. The agent can take consequential actions, such as sending emails, changing records or spending money, without checks. This is "inadequate risk controls."
• Runaway cost. Loops, retries and oversized contexts push the cost per task past the value of the task. This is "escalating costs."

Evals address the first two directly and expose the other two early.

What an eval actually is

Anthropic's engineering guide on agent evals, published in January 2026, gives clear definitions that I now use with clients (source):

• Task: one test with defined inputs and success criteria.
• Trial: one attempt at a task. You run several, because outputs vary.
• Grader: logic that scores some aspect of the result. A task can have several.
• Transcript: the full record of a trial, including tool calls and intermediate results.

It also distinguishes two kinds of eval suites. Capability evals ask what the agent can do well and should start with a low pass rate, giving you something to climb. Regression evals check the agent still does what it used to and, in the guide's words, "should have a nearly 100% pass rate".

Three kinds of graders

Same source for the trade-offs: Anthropic. My default is to use code-based graders wherever the property can be checked mechanically, model-based graders for tone and completeness, and human review to calibrate the model-based graders.

Measure consistency, not just capability

One idea from the Anthropic guide changed how I report results. pass@k is the chance that at least one of k attempts succeeds. pass^k is the chance that all k succeed. For a customer-facing agent, pass^k is the honest metric, because the customer gets one attempt. The guide's example: an agent with a 75% per-trial success rate has roughly a 42% chance of succeeding on all three of three trials. An agent that "usually works" can still be unreliable for your users.

How to build your first eval suite

Start with 20 to 50 real cases

The guide's advice is that "20-50 simple tasks drawn from real failures is a great start". Take them from support tickets, past mistakes, and the checks you already do by hand before a release. You do not need hundreds to begin.

Write a reference answer for each

For every task, write what a correct outcome looks like, or the properties it must have: the right tool was called, the answer cites a source, no write action happened without approval. If two people on your team would disagree on whether an output passes, the task is too vague.

Include cases where the agent should not act

Test the refusals and handoffs too: a question outside scope, a request the user is not permitted to make, a document that does not contain the answer. An agent that always tries to help is an agent that will eventually do something it should not.

Run it on every change

Prompt edits, model upgrades, tool changes and retrieval changes all go through the suite. Treat a drop in the regression suite like a failing unit test.

Read the transcripts

When a case fails, read what the agent actually did. Sometimes the agent is wrong; sometimes the grader is. You only find out by reading.

Guardrails that complement evals

Evals tell you how the agent behaves on known cases. Guardrails limit the damage on unknown ones:

• schema validation on every tool input and output,
• human approval on consequential actions,
• caps on steps, retries and tokens per task,
• permission checks on the server for every tool,
• logging of every run with a replayable transcript.

At Avenuor, the AI document processing I built validates extracted invoice data against a schema before anything is written, so a bad extraction becomes a visible error instead of a quiet wrong record. In AgentOps, a person approves each stage's output before the next agent acts on it; I explain why those checkpoints matter in multi-agent systems: when one agent isn't enough. Neither replaces evals; both reduce what evals have to catch.

Evals as compliance evidence

If you serve the EU, evals have a second job. The AI Act's high-risk obligations, which apply from 2 December 2027 for stand-alone high-risk systems, include risk management and testing across the system's lifecycle. A versioned eval suite with results per release is the kind of evidence that work requires. More in EU AI Act 2027: what AI builders must do.

Work with me

If you have an agent that works in demos but not reliably in production, I can help you build the eval suite and guardrails that make it shippable, or build the agent properly from the start. See AI agent development and AI integration, or book a scoping call.