- A small language model (SLM) does language tasks with far fewer parameters than an LLM, usually a few billion or less.
- SLMs get strong on narrow tasks through better training data, distillation from larger models, and fine-tuning.
- SLMs cost less, answer faster and run on a single GPU, a laptop or on-premises; LLMs still lead on open-ended reasoning.
- Many AI systems route routine work to SLMs and save large models for the hardest questions.
What is a small language model?
A small language model is a language model built to do a narrower job with far less compute than a large language model. It reads and writes text the same way, but with fewer parameters, so it can run on a single GPU server, a laptop or even a phone, answer faster and cost less per answer.
There is no fixed cutoff for "small". Definitions put the line anywhere from under 1 billion parameters to around 10 billion, and the line moves as hardware improves. A rough guide:
| Size | Parameters | Runs on | Examples |
|---|---|---|---|
| Tiny | Under 1 billion | Phones, laptops, edge devices | H2O Danube3 0.5B |
| Small | 1 to 4 billion | A single GPU or a laptop | H2O Danube2 1.8B, Microsoft Phi-3-mini (3.8B), H2O-Lightning-4B |
| Mid-size | About 7 to 14 billion | One or two data center GPUs | Mistral 7B, Llama 3 8B |
| Large | Tens of billions to trillions | Clusters of data center GPUs | Frontier models |
How small language models work
A small language model uses the same transformer design and attention mechanism as a large one, just with fewer layers and parameters. It makes up for its size in four ways:
- Better training data. Carefully chosen data can beat sheer size. Microsoft's Phi-3 report says the 3.8-billion-parameter phi-3-mini "rivals that of models such as Mixtral 8x7B and GPT-3.5" and still runs on a phone.
- Distillation. A large model teaches a small one by generating examples and answers, and the small model learns to match them.
- Fine-tuning. Training a small model further on one task, with transfer learning, makes it an expert at that task.
- Quantization. Storing weights at lower precision shrinks the model so it fits on smaller hardware, at a small cost in accuracy.
SLM vs LLM
| Small language model | Large language model | |
|---|---|---|
| Parameters | Up to roughly 10 billion | Tens of billions to trillions |
| Best at | Focused, repeated tasks: classify, extract, summarize, route, decide | Open-ended reasoning and broad knowledge |
| Hardware | One GPU, a laptop or a phone | Clusters of data center GPUs |
| Cost per answer | Low | High |
| Speed | Fast | Slower |
| Where it can run | Anywhere, including on-premises and air-gapped, if its weights are open | Usually a provider's cloud |
Most systems use both. A router sends routine work to small models and saves the large model for the hard questions. The 2025 position paper Small Language Models are the Future of Agentic AI argues that small models are "sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems," since most steps an AI agent takes are narrow and repetitive.
When to use a small language model
Use a small model when the task is narrow and repeated, and a large one when it's open-ended. A small language model is the better fit when:
- The task has a clear input and output: classify a ticket, extract fields from a form, summarize a call, route a request, approve or flag a transaction.
- It runs at high volume, so the cost of each answer adds up.
- Answers have to come back fast, inside an application or an agent's loop.
- The data can't leave your environment.
- You have, or can collect, good examples of the right answer to fine-tune on.
Reach for a large model when the questions are open-ended, need broad world knowledge or chain many reasoning steps, or when the task isn't defined yet. A common path is to prototype with a large model, then move the steady, high-volume parts to a fine-tuned small one.
Small language models in AI agents
An AI agent breaks a goal into many small steps: pick a tool, fill in its arguments, read the result, decide what's next. Most of those steps are narrow and repeated, which is exactly what a small model does well. A practical pattern is a router: small models handle the routine steps, and a large model is called only when a step needs broad reasoning. That keeps an agent fast and its running cost predictable, even when it takes thousands of steps a day.
Why organizations use small language models
- Cost. A large North American telecom operator cut its token costs 10x by moving to a small language model on its own hardware.
- Speed. Fewer parameters mean faster answers, which matters when a model runs thousands of times inside a workflow.
- Control. An open-weight SLM fits on hardware you own, so it can run on-premises or fully air-gapped.
- Accuracy on the task. A small model fine-tuned on your own examples often matches or beats a general large model on that one job.
How to choose a small language model
| Question | What to check |
|---|---|
| Can you use it commercially? | The license. Apache 2.0 and MIT allow commercial use; some licenses don't |
| Is it good at your task? | Benchmarks close to your task, then a test on your own examples |
| Will it fit your hardware? | The parameter count, and the memory it needs at the precision you'll run |
| How much text can it read at once? | The context window, if you work with long documents |
| Does it handle your languages? | The languages in its training data |
| Can you trust where it came from? | The model card: who built it, how, and on what data |
How to fine-tune a small language model
- Define the task and how you'll score it, with a test set the model never trains on.
- Collect examples: real inputs paired with the outputs you want, cleaned and checked.
- Pick a base model that already does reasonably well on your test set.
- Fine-tune efficiently. Methods such as LoRA train a small set of added weights instead of the whole model; the LoRA paper reports it "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times."
- Evaluate against the test set, and against the large model you would otherwise use.
- Deploy it where the data lives, and monitor its answers in production.
How to measure a small language model
A small model earns its place on four numbers, measured on your own task rather than a public leaderboard:
- Accuracy on your test set, compared with the large model you would otherwise use. The goal is the same answer, not a perfect score.
- Latency, the time to the first word and to a full answer, under realistic load.
- Cost per request, from the hardware it needs and how many requests it serves at once.
- Memory, which decides the smallest GPU, laptop or phone it fits on.
If the small model matches the large one on the first number and wins on the other three, it should take that job.
Where small language models run
- On a device. Phones and laptops, for offline or private use. Phi-3-mini runs on a phone.
- On-premises. A single GPU server in your own data center, as an on-premises LLM.
- Air-gapped. Networks with no outside connection at all, as air-gapped AI.
- In your cloud account. A small GPU instance you control, at a fraction of a large model's cost.
Limits of small language models
- Less general knowledge. A small model knows less about the world, so it does worse on questions outside its training.
- Weaker long reasoning. Multi-step problems and open-ended planning still favor large models.
- Needs good data. Fine-tuning only helps as much as the examples you give it.
- Narrow by design. A model tuned for one task can be brittle on a different one.
Common myths about small language models
- "Small means weak." On a narrow task with good training data, a small model can match much larger ones. Phi-3-mini's report compares it with Mixtral 8x7B and GPT-3.5.
- "Small means a phone app." Many small models run in data centers, serving thousands of requests inside business systems.
- "You have to train one from scratch." Most teams fine-tune an existing open-weight model on their own examples, in hours or days.
- "One model should do everything." Production systems mix several small models with a large one, each on the job it does best.
Small language model examples
- Microsoft Phi-3-mini, 3.8 billion parameters, small enough to run on a phone.
- H2O Danube, a family of open-weight small language models under the Apache 2.0 license. H2O Danube2-1.8B ranked first among models under 2 billion parameters on the Hugging Face Open LLM Leaderboard in April 2024.
- H2O Mississippi, an open-weight vision-language model at 2 billion parameters that reads documents and images.
- H2O Lightning, a family of open-weight decision models. H2O-Lightning-4B ranked first among 104 open-weight systems on JevBench as of October 7, 2026.
Small language models in H2O AI
H2O Cascade fine-tunes and distills small models on your own data, and the H2O AI Super Agent routes each job to the right model, small or large. H2O.ai's own H2O Danube, H2O Mississippi and H2O Lightning models are open-weight and run on H2O AI Hybrid Cloud, on-premises or air-gapped. AT&T cut costs up to 90% with fine-tuned small language models.