AI in practice · Advanced

Small, local and edge models

In one line: When volume is large, latency is critical, or the data can't leave — a small model running on your side beats a big one in the cloud.

When it pays off

A narrow, recurring task with a fixed format; a privacy or regulatory requirement; volume that makes pay-per-use expensive; or a need for an instant response with no network dependency.

What you pay in return

Operations: hardware, availability, updates, monitoring, and a team that knows how to maintain it. And: the quality gap on open tasks still exists, even if it's narrowed.

A middle path

Routing by difficulty: the small model handles most requests, and hard ones go to a strong model. That's usually the optimum — low cost with quality preserved where it matters.

You can also distill: use a strong model to generate quality examples on which you fine-tune a small model for a specific task.

Going deeper

Measure throughput in concurrent requests, not just a single response time. A local model behaves entirely differently under load, and capacity planning done by a single-user test always leads to a surprise at launch.

Cost Performance

Next: Process automation: where a model adds and where it's redundant · Previous: Bias, fairness and accountability in using models · Back to library