Open-Source LLMs vs API Models: When Each Wins

The question I get from CTOs is usually phrased as "should we just run our own model?" It sounds like a cost question, and sometimes it is. More often it is a privacy, control or vendor-risk question wearing a cost question's clothes. The answer depends on which one you are really asking.

Open-weight models have become genuinely capable. OpenAI's own open-weight release, gpt-oss-120b, is published under the Apache 2.0 licence and, per its model card, is designed to "fit into a single 80GB GPU" such as an NVIDIA H100. That makes self-hosting a realistic option for far more teams than a few years ago. Realistic is not the same as sensible, though, so this post sets out when each approach wins and gives you a way to calculate your own break-even.

What is the difference between open-source LLMs and API models?

An API model runs on the provider's infrastructure and you pay per token; an open-weight model is one whose weights you can download and run yourself, paying for the hardware instead. "Open source" is used loosely here. Many popular models publish weights under licences with conditions, so "open-weight" is the more accurate term.

Licences matter more than people expect. Meta's Llama 4 licence, for example, requires companies with more than "700 million monthly active users in the preceding calendar month" to request a separate licence from Meta, and requires anyone distributing products built on it to "prominently display 'Built with Llama'". Apache 2.0 models have no such conditions. Read the licence before you build, not after.

There is also a middle path: open-weight models served by a hosting provider through an API. You get per-token pricing without the infrastructure, and you can move to self-hosting later without changing models.

Open-weight vs API: comparison table

The version-control row is underrated. API providers retire models, and Anthropic's deprecations page commits to "at least 60 days' notice" before retiring a publicly released model. For most products that is fine. For a regulated workflow validated against one exact model, owning the weights can be worth a lot.

How do you calculate the break-even for self-hosting?

The break-even is the monthly token volume at which an always-on GPU costs the same as paying per token. The arithmetic is simple; the hard part is the throughput assumption.

Step 1: price the GPU. Lambda's pricing page lists a single NVIDIA H100 SXM (80 GB) at $4.89 per GPU-hour, for prices effective from 16 November 2026. Running one around the clock for a 730-hour month comes to about $3,570.

Step 2: price the API. Anthropic's pricing page lists, per million tokens, Claude Haiku 4.5 at $1 input and $5 output, Claude Sonnet 5.5 at $2 input and $10 output, and Claude Haiku 5.5 at $0.10 input and $0.50 output for prompts up to 100,000 tokens. Assume, for illustration, three input tokens for every output token. The blended price is then $2 per million tokens for Haiku 4.5, $4 for Sonnet 5.5 and $0.20 for Haiku 5.5.

Step 3: divide. $3,570 divided by the blended price gives the monthly volume at which one always-on GPU matches the API bill.

Step 4: check that one GPU can actually serve that volume. This is where most break-even spreadsheets go wrong. Throughput depends on the model, quantisation, serving software, prompt length, batch size and your latency target. Measure it with your real prompts before trusting any number, including mine.

Why is the real break-even usually higher than the spreadsheet says?

The real break-even is higher because a single GPU is not a production deployment. Several costs sit on top of the GPU price:

• Redundancy. One GPU is one point of failure. Production usually means at least two, often in separate zones, which doubles the floor.
• Idle time. Traffic is not flat. If your GPUs are busy eight hours a day, you pay for twenty-four.
• Engineering time. Someone has to run the serving stack, patch it, scale it, monitor it and answer the pager. That is a recurring cost even when nothing breaks.
• Model quality gap. If the open model needs longer prompts, more retries or a second pass to match an API model's quality, your effective token count rises.

Meanwhile, the API side keeps getting cheaper in ways a flat comparison misses. Anthropic's pricing page lists a 50% discount on both input and output tokens through the Batch API, and cache hits at 10% of the standard input price, lower still on some newer models. My guide to reducing LLM costs covers these levers. Apply them first, then compare.

The honest conclusion from the arithmetic: for most products I work on, self-hosting only wins on cost at volumes far beyond what they actually run.

When does self-hosting an open-weight model win?

Self-hosting wins when at least one of these is true, and usually when two are.

• Data cannot leave your infrastructure. Some contracts, regulators or internal policies forbid sending certain data to any third party, regardless of DPAs. Running the model inside your own network removes the question. For EU clients, though, check first whether a provider's data residency options already satisfy the requirement; my post on GDPR-safe LLM integration walks through those options.
• Volume is very high and steady. Batch processing millions of documents, continuous classification of a large event stream, or an internal tool used all day across a large company can keep GPUs busy enough to beat per-token pricing.
• You need a frozen model for years. Validated or certified workflows sometimes cannot accept a model change every year or two.
• You need deep customisation. Full fine-tuning, custom decoding or serving-level changes that APIs do not expose.
• Offline or edge use. On-device and air-gapped deployments, where small models shine. I cover that end of the spectrum in small language models and on-device AI.

My own Flo voice assistant is an example of the hybrid pattern: speech is transcribed on device with Whisper, and users can bring their own key for Claude, OpenAI or Gemini, or run fully local models through Ollama and LM Studio. The privacy-sensitive step stays local; the user chooses where the rest runs.

When do API models win?

API models win in most other cases, which covers most products I see. Choose an API when:

• You are still finding product fit. Volume is uncertain, and paying per token scales to zero when usage does.
• You need the provider's strongest model. The top proprietary models are only available through their providers' APIs and cloud platforms.
• Traffic is spiky. APIs absorb bursts without you provisioning for the peak.
• You do not have infrastructure people to spare. The engineering time is better spent on the product.
• You need several models. Routing between a cheap model and a strong one is a configuration change on an API, and a capacity-planning exercise when self-hosted.

A sensible path for many teams: start on an API, design the integration so the model is swappable, track cost per task from day one, and revisit self-hosting only when the numbers or a hard data requirement say you should.

Work with me

I help teams choose between API and self-hosted models with their own numbers, and build integrations that keep the model swappable either way. See AI integration and AI agent development, browse my work, and book a scoping call.