Small Language Models and On-Device AI: When They Win

For the last few years, the default answer to any AI feature was "call the biggest model." That is changing because of cost, latency and privacy, and because small models got good enough for a large share of real tasks. Here is what the evidence says and how I decide which size of model goes where.

What counts as a small language model

There is no official cut-off. NVIDIA researchers, in their 2025 paper arguing that SLMs are the future of agentic AI, define an SLM as a model that can fit onto a common consumer device and respond fast enough for one user's agentic requests, and say they would consider most models below 10 billion parameters to be SLMs.

Some real examples:

• Apple's on-device foundation model has about 3 billion parameters, compressed to 2 bits per weight, and Apple's Foundation Models framework gives developers access to it in Swift with inference that is free of cost.
• Google's Gemma 3n models have raw parameter counts of 5B and 8B but run with a memory footprint of about 2GB and 3GB, designed for phones.

Why small models are rising

Enterprises expect to use more of them

Gartner predicts that by 2027, organisations will use small, task-specific AI models at least three times more than general-purpose large language models. Its reasoning: general-purpose LLMs lose accuracy on tasks that need specific business context, while smaller task-specific models respond faster and use less compute.

Agent workloads are mostly repetitive

The NVIDIA paper estimates that serving a 7-billion-parameter SLM is 10 to 30 times cheaper in latency, energy and compute than a 70 to 175-billion-parameter LLM. In case studies of three open-source agent frameworks, the authors estimate that roughly 40% to 70% of LLM calls could be handled by specialised SLMs. Those are the authors' estimates, not measurements, but the logic matches what I see: much of an agent's work is formatting, classifying and choosing the next tool, not hard reasoning.

Agent costs are rising

Gartner predicts inference costs per agentic workflow will increase more than fivefold through 2028, because agents make many more model calls per task. Routing routine calls to smaller models is one of the most direct ways to contain that.

The hardware is shipping

Gartner's forecast puts AI PCs, laptops and desktops with dedicated AI processors, at 31% of the worldwide PC market by the end of 2025, and says they will become the norm by 2029. More devices can run a capable model locally every year.

Small vs large: when each wins

Tasks small models do well

• Classification and routing: is this email a complaint, an order or spam? Which tool should the agent call next?
• Extraction into a fixed schema: pulling dates, amounts and names from a document type you know well.
• Short, constrained generation: summaries of a known format, suggested replies, autocomplete.
• Privacy-sensitive features: processing personal notes, messages or health data on the device itself.

Tasks to keep on large models

• multi-step reasoning across many documents,
• open-ended questions with no fixed format,
• code generation across a large codebase,
• anything where a wrong answer is expensive and hard to detect.

The design pattern to use: route by task

The architecture I recommend is a router, not a single model:

• Classify the request with a small, fast model or a rule.
• Send routine work to a small model with a strict output schema.
• Escalate to a large model when the task is complex, the small model's output fails validation, or confidence is low.
• Validate every output against a schema or rules before it is used. At Avenuor, extracted invoice data is validated against a schema before anything is saved; that check works the same whichever model produced the data.
• Measure both paths with the same eval set, so you know what the small model gets wrong. My guide to AI agent evaluation covers how.

This pattern also makes you less dependent on any one provider. If a model's price or terms change, you swap one route, not the whole system.

Fine-tune or prompt?

Small models benefit more from task-specific tuning than large ones, because they have less general capability to fall back on. If you have hundreds of good examples of a narrow task, a fine-tuned small model can be a strong option. If you do not, start with a large model, collect real examples, and move the task down to a small model once you know what "correct" looks like. I compare the options in RAG vs fine-tuning.

What changes by 2028

• More features run on the device by default, especially in mobile apps, where Apple and Google already expose on-device models to developers.
• Privacy becomes a product feature. "Your data never leaves your phone" is a selling point that small models make possible.
• Agent systems become multi-model. Gartner's own forecast of task-specific models outnumbering general-purpose ones points this way.
• Cost discipline becomes architecture. Choosing the smallest model that passes your evals becomes routine engineering practice.

Work with me

I design AI systems that use the right model for each step, with validation and evals so you know what each one gets wrong. See AI integration and AI agent development, or book a scoping call.