Cutting LLM App Costs: Caching, Routing, Smaller Models

Once an AI feature has real users, the model bill stops being a rounding error. The good news is that most of it is waste you can remove without touching answer quality: repeated context, an expensive model doing easy work, and urgent pricing on jobs that are not urgent. This post walks through each lever with arithmetic based on Anthropic's published prices as of late 2026, so you can redo it with your own numbers.

Start by measuring where the tokens go

You cannot cut a bill you have not broken down. Before changing anything, log four numbers per request for each feature: stable input tokens (instructions, tools, shared documents), variable input tokens (the user's message, retrieved context, history), output tokens, and which model served it. The usage ledger I use for per-user cost caps records exactly this.

That breakdown usually tells you which lever matters most:

• Large stable input points to caching.
• Many short, simple requests on a large model point to routing.
• Big offline jobs point to batching.
• Long answers point to output limits, since output tokens usually cost several times more than input.

If the feature is wired through a gateway, much of this logging is already there. Vercel's AI Gateway, for example, records token usage and cost per request and supports budgets, which I covered in running AI agents on Vercel.

One trap is worth knowing before you compare models on price alone. Tokens are not a fixed unit across models. Anthropic's pricing page notes that Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text". A model with a lower per-token price can still cost more per request, so compare cost per request on your own traffic, not price per million tokens.

The worked example: a support assistant

Here is the scenario I use for the maths. It is illustrative, not a client's figures. A support assistant handles 100,000 requests a month. Each request sends 5,000 stable tokens (instructions, tool definitions and product docs), 1,000 variable tokens, and gets 400 tokens back. Everything runs on Claude Sonnet 5.5, which Anthropic's pricing page lists at $2 per million input tokens and $10 per million output tokens.

• Input: 600 million tokens at $2 per million = $1,200.
• Output: 40 million tokens at $10 per million = $400.
• Total: $1,600 a month.

Lever 1: cache the stable part of every prompt

Prompt caching bills repeated input at a fraction of the normal price, so it is the first thing to turn on. For Sonnet 5.5, Anthropic lists cache reads at $0.10 per million tokens, 0.05x the input price. I explain the mechanics, and the prompt structure it needs, in prompt caching explained.

In the example, the 5,000 stable tokens per request are the cacheable prefix. Assuming steady traffic keeps the cache warm, so cache writes are a negligible share:

• Cached stable input: 500 million tokens at $0.10 per million = $50.
• Variable input: 100 million tokens at $2 per million = $200.
• Output: unchanged at $400.
• Total: $650 a month, about 59 percent less, with identical answers.

Notice what is left: output is now the biggest line. Caching cannot help with that, which is why the next lever matters.

Lever 2: route easy requests to a smaller model

Most products have a mix of easy and hard requests, and the easy ones do not need the largest model. Routing sends each request to the cheapest model that can handle it. Research backs the idea: the RouteLLM paper reported that learned routers cut costs "by over 2 times in certain cases" without compromising response quality (Ong et al., 2024).

The price gap between tiers is large. Anthropic lists Claude Haiku 5.5, for prompts up to 100,000 tokens, at $0.10 per million input tokens, $0.01 for cache reads and $0.50 per million output tokens. That is 20 times cheaper than Sonnet 5.5 on both input and output.

Suppose your evaluations show that 60 percent of requests, such as order status, password help and simple how-to questions, get equally good answers from Haiku 5.5. A small Haiku call classifies each request first.

That is 82 percent below the original $1,600. The 60 percent share is an assumption you must test, not a target. Route on evidence: build a set of real requests with expected answers, run both models over it, and only route a category once the smaller model matches. My post on AI agent evaluation covers how I set those tests up.

A simpler variant skips the classifier: try the small model first and escalate when a cheap check fails, such as invalid JSON or a low-confidence flag. It costs a second call on the hard cases but needs no router.

Lever 3: send work that can wait to a batch API

Anything that does not need an answer within seconds should go through a batch API, which every provider I checked prices at half. Anthropic's Batch API gives "a 50 percent discount on both input and output tokens", Amazon Bedrock offers batch inference for select models "at a 50% lower price compared to on-demand inference pricing", and Microsoft's Foundry deployment types list Global Batch at 50 percent less than Global Standard with a 24-hour target turnaround.

In the example, suppose a nightly job summarises 10,000 tickets for the support team, each with 3,000 input and 300 output tokens on Sonnet 5.5.

• On demand: 30 million input at $2 plus 3 million output at $10 = $90.
• Batch: the same tokens at $1 and $5 per million = $45.

Good candidates are tagging, summarising backlogs, generating embeddings or descriptions, and running evaluation suites. Caching discounts stack with the batch discount on Claude, according to the same pricing page, so a batch job with a shared prefix gets both.

Lever 4: cap output and trim the context

Output tokens cost five times the input price on the models above, so long answers are expensive. Three cheap controls:

• Set a maximum output length on every call. It stops runaway generations and makes cost per request predictable.
• Ask for the format you need: a short answer, a JSON object or a list, not an essay.
• Trim the input: send the last few turns of a conversation rather than all of it, and retrieve fewer, better chunks, as I describe in agentic RAG. Lower variable input also makes caching more effective.

For narrow, high-volume tasks such as classification or extraction, a small open model you host yourself can be cheaper still, at the cost of running it. I weigh that trade-off in small language models and on-device AI.

Before and after

The order matters. Caching is safe and changes no answers, so it goes first. Routing changes which model answers, so it needs evaluations. Batching changes when answers arrive, so it only suits offline work. Output limits are worth adding at any point.

Work with me

I help teams bring LLM bills under control without losing quality, from token logging and caching to routing backed by evaluations. See AI integration and full-stack development, browse my work, and book a scoping call.