From Prototype to Production: Hardening an AI Feature

A prototype answers one question: can the model do this at all? Production asks different ones. What happens on the thousandth request, the malformed one, the one during a provider outage, the one from a user trying to break it? This is the checklist I work through on client projects between "the demo worked" and "it is live for everyone".

Why do AI features break between demo and production?

AI features break after the demo because a demo is tested on a handful of friendly inputs, while production sees every input, at volume, with real money and real users attached. The model's behaviour also varies: the same question can get different answers.

The New York City MyCity chatbot is a public example. The Markup reported that it told business owners "Yes, you can make your restaurant cash-free", which conflicts with a 2020 city law, and that on housing vouchers it told one reporter landlords had to accept them while telling ten other staffers they did not. A demo with three questions would not have caught either problem. A test set with a hundred real questions run on every change would have had a fair chance.

Hamel Husain, who consults on AI products, puts the pattern bluntly in Your AI Product Needs Evals: "unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems."

What does the production checklist look like?

The checklist has six stages, and each one closes a specific gap the prototype left open.

How do you build evals for an AI feature?

Start with a set of real inputs and the result you expect for each, and run it automatically whenever the prompt, model or retrieval changes. Anthropic's guide to developing tests gives three principles worth pinning above the desk: "Be task-specific", "Automate when possible", and "Prioritize volume over quality", meaning more automatically graded questions beat a few hand-graded ones.

In practice:

• Collect inputs from reality. Pull them from the prototype's logs, support tickets or the client's own examples. Include the awkward ones: empty input, very long input, other languages, hostile input.
• Grade with code first. Exact match, a JSON schema, "contains the right ID". Use a second model as a grader only where code cannot judge, such as tone; the same Anthropic guide notes it is "generally best practice to evaluate with a different model from the one used to generate the evaluated output."
• Track a score, not a feeling. A number per run lets you compare prompts and models.
• Add every production failure. Each bug becomes a test case, so it cannot quietly return.

I go deeper on letting agents check their own work in give AI agents a way to check, and on scoring agents in AI agent evaluation.

How should the feature handle bad output and provider failures?

Treat every model response as untrusted input and every provider call as something that can fail. Two layers handle this.

Output checks. If the feature needs structured data, ask for structured output and validate it against a schema before using it. If it answers from documents, refuse when retrieval finds nothing relevant. If it writes something a customer will see, check for the things that must never appear, such as internal IDs or another customer's data.

Fallbacks. Anthropic's API errors page lists the cases to plan for: 429 for rate limits, 500 for internal errors, 529 when "the API is temporarily overloaded", and a 400 when usage reaches a spend limit you set. It also notes that the official SDK "automatically retries transient failures" with exponential backoff, "twice by default", configurable with maxRetries in TypeScript. Two details catch people out:

• When streaming, "an error can occur after the API returns a 200 response", so the UI needs a failed state mid-stream, not only before it. My post on streaming LLM responses in Next.js covers those states.
• Retries do not help with a spend-cap error; it keeps failing until access resumes. Show a clear message instead of retrying.

Beyond retries, decide what the user sees when the AI path is down: a cached answer, a simpler model, a non-AI version of the feature, or an honest "try again in a few minutes". Pick one before launch, not during the outage.

How do you stop one user or one bug running up the bill?

Put limits at three levels: per request, per user and per account. Per request, cap max_tokens and input size. Per user, meter the cost of each call and enforce a budget by plan; the design is in how to cap LLM costs per user. Per account, set a spend limit with the provider so a runaway loop stops at a number you chose.

Then cut the bill you do have. Prompt caching, smaller models for simple steps and batching for work that can wait are covered in how to reduce LLM costs.

What should you monitor once it is live?

Monitor four things: cost, latency, errors and output quality. The first three are ordinary engineering. The fourth is what makes AI features different, because a model can return a fast, cheap, well-formed answer that is wrong.

• Cost per request and per user, with an alert when either jumps.
• Latency, including time to first token for streamed responses.
• Errors by type, so a rise in 529s reads differently from a rise in validation failures.
• Sampled outputs reviewed by a person each week, with failures added to the eval set.
• User signals: thumbs down, retries, edits to the AI output, abandoned flows.

Log the provider's request ID with each call. Every Anthropic response includes a request-id header, and the TypeScript SDK exposes it as _request_id on response objects, according to the errors page. It turns "the AI did something odd yesterday" into a specific request you can look up.

How do you roll it out safely?

Ship behind a feature flag, turn it on for a small group first, and keep a switch that turns it off without a deploy. A small cohort gives you real traffic for the monitoring above without exposing everyone to an unknown failure. Widen in steps, watching cost and quality at each step.

Write down who owns the feature after launch, how often outputs are reviewed and what triggers the kill switch. An AI feature with no owner degrades quietly as content, users and models change. That handover is part of the work, not an extra.

Work with me

Hardening AI features for production is most of my client work: evals, fallbacks, limits, monitoring and a clean handover. See AI integration and full-stack development, or browse my work and book a call to scope yours.