How Small Language Models Became Genuinely Useful
Compact models now handle summarisation, extraction and routing well enough to run on a phone. That changes where inference happens.
The interesting work in applied AI has moved down-market. Models small enough to run locally are now competent at the narrow tasks that make up most production workloads.
The tasks small models do well
Classification, extraction, routing and short summarisation are all bounded problems. They reward precision and consistency far more than broad world knowledge.
Latency is a feature
A local model that answers in 80 milliseconds changes interface design. You can afford to run it on every keystroke.
- No network round trip, so no jitter.
- Data never leaves the device, which simplifies privacy review.
- Cost per call approaches zero.
The question is no longer which model is smartest, but which model is small enough to sit inside the interaction.
Where they still fail
Long multi-step reasoning, rare domain knowledge and anything requiring current information remain weak spots. Treat small models as fast, cheap components in a larger system rather than as replacements for frontier models.
More from ByteScope
The ByteScope Brief
Technology explained, once a week.
One email with the reporting and analysis worth your time. No hype, no daily noise.
No spam. Unsubscribe in one click.