Structured Outputs: Getting Reliable JSON From LLMs

Most LLM features end in code, not in a chat window. The model reads an invoice and your database needs a row; it triages a ticket and your router needs a category; it plans an action and your agent needs a tool call with typed arguments. Every one of those hand-offs breaks if the output is a paragraph of prose with some JSON in the middle.

For a long time the fix was a pile of regular expressions and a retry loop. Structured outputs have made most of that unnecessary, but not all of it. This post covers what the native features guarantee, what they do not, and the validation and retry patterns I use on client projects.

What are LLM structured outputs?

Structured outputs are a provider feature that constrains a model's response to match a JSON schema you supply. Instead of asking nicely for JSON and hoping, you pass the schema with the request, and the model's generation is restricted so that it can only produce output that fits.

Anthropic's structured outputs documentation describes this as constrained decoding, so responses are "always valid, parseable, and type-safe". It offers two related features: JSON outputs, set with output_config.format, which shape the model's answer, and strict tool use, set with strict: true on a tool definition, which validates tool names and inputs.

OpenAI's Structured Outputs guide draws an important distinction with its older JSON mode: "While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence." Valid JSON is not the same as the JSON you asked for. A response of {"note": "I could not find a total"} is valid JSON and useless to an invoice parser expecting a number.

How do you request structured output from OpenAI and Claude?

You define a schema, pass it in the request, and read the parsed result, and both major SDKs will build the schema for you from a typed model. In practice I define the schema once in code and generate everything else from it.

• TypeScript with Zod. Define the shape with z.object({ ... }). OpenAI's SDK accepts it through zodTextFormat, and Anthropic's TypeScript SDK through zodOutputFormat passed to client.messages.parse(), according to their respective docs.
• Python with Pydantic. Define a BaseModel subclass. Anthropic's docs show client.messages.parse(output_format=YourModel) returning the result in response.parsed_output, and OpenAI's Python SDK accepts Pydantic models directly.
• Raw JSON Schema. Both accept a plain schema too. On OpenAI you set text.format to json_schema with strict set to true. On Claude you pass output_config with format.type set to json_schema.

A few schema habits make life easier with both providers:

• Set additionalProperties to false on every object. Anthropic's docs say this is required for objects, and OpenAI's examples use it throughout.
• Mark every field as required, and model "unknown" explicitly with a nullable type rather than an optional field.
• Use enum for categories, and give each property a description, because the model reads it.
• Keep the schema shallow. Deeply nested, recursive structures are harder for the model and, in some cases, unsupported.

What does structured output not guarantee?

Structured output guarantees the shape of a completed response, not that every response completes or that the values are correct. There are four gaps to plan for.

Refusals. If the model declines a request for safety reasons, the response may not match your schema. Anthropic's docs say a refusal returns stop_reason: "refusal" with a 200 status and billed tokens, and "the output may not match the schema". OpenAI surfaces refusals in a separate refusal field and recommends checking for it before parsing.

Truncation. If the response hits the token limit, it may be cut off mid-object. Anthropic's docs note that with stop_reason: "max_tokens" the output "may be incomplete" and suggest retrying with a higher limit.

Unsupported constraints. Not every JSON Schema keyword is enforced. Anthropic's docs list numeric constraints such as minimum and maximum and string constraints such as minLength as unsupported in the schema itself; its SDK helpers move those constraints into field descriptions and still enforce them locally when validating. The docs also note that capitalisation of enum values "isn't guaranteed", and recommend comparing case-insensitively.

Wrong but valid values. The schema can force total to be a number. It cannot force it to be the right number. Line items that do not add up, a date in the future, a category that is plausible but wrong: all pass schema validation.

How should you validate and retry LLM JSON?

Validate twice: once by the provider's constrained decoding, and once in your own code against the same schema plus your business rules. Then retry with the error, a limited number of times, before falling back.

The pattern I use:

• Check the stop reason first. If it is a refusal or a token-limit cut-off, do not parse. Route refusals to a fallback message or a person; retry truncations once with a higher limit.
• Parse and validate with the same schema. Zod's `.safeParse()` returns a result object rather than throwing, so you can branch on result.success and read the error details. In Python, validate against the same Pydantic model.
• Apply business rules. Totals match line items, dates fall in a sensible range, referenced IDs exist in your database.
• Retry once with the error. Send the original input, the invalid output and the specific validation message back to the model and ask for a corrected version. One retry fixes most problems; more retries mostly burn tokens.
• Escalate the rest. Anything still failing goes to a review queue rather than being saved. Silent bad data is worse than a visible backlog.

This is the design behind invoice extraction in Avenuor: a model "constrained to a schema, with validation and retry on mismatch", and users confirm the extraction before it is saved. The same principle applies to agents. A schema-checked tool call is the cheapest kind of feedback an agent can get, which is the idea behind giving AI agents a way to check their work.

Which approach fits which situation?

The last row is the one I ship. The provider feature removes most format errors for free; the code-level validation is what lets you trust the data.

What else improves structured output quality?

Structure helps the model as well as your parser, but a few prompt and schema choices still matter.

• Ask for reasoning before the answer, when it helps. A reasoning string field placed before the answer field gives the model room to work, and Anthropic's docs note that required properties are output in schema order.
• Allow "not found". If a field might be missing from the source, make it nullable and say so in the description. Otherwise the model will invent something to satisfy the schema.
• Keep schemas stable. Anthropic's docs mention that compiled schemas are cached for 24 hours and that changing the schema structure invalidates the cache, which affects first-request latency.
• Test with ugly inputs. Blurry scans, empty documents, two invoices in one PDF. Your eval set should include them, with the correct expected output.

If you are wiring this into a web app, my guide to integrating Claude with Next.js covers the server-side setup.

Work with me

I build LLM integrations where model output flows straight into databases and workflows, with schemas, validation and review paths designed in. See AI integration and full-stack development, browse my work, and book a scoping call.