AI in practice · Mixed

Cost and latency in model-based systems

In one line: Most of the cost comes from the text going in, and most of the latency from the text going out — so the two fixes differ.

Lowering cost

Attach less context; route simple requests to a small model; exploit caching for fixed parts; and batch what isn't urgent.

Compute cost per business unit — per request, per user, per order — not just a monthly sum. That's how you see whether growth pays off.

Speeding up

Streaming the answer to the user shortens the perceived wait more than any other optimization. Shortening the output length helps directly. And tasks that can run in parallel — run in parallel.

Protecting the budget

Daily caps, alerts on threshold crossings, and a step limit in automated processes. A wrong loop can burn a monthly budget in hours.

Going deeper

Tag every call by user, capability and purpose. Without tagging, the question “what caused the spike” is answered by guessing, and you can't turn off the expensive, less-useful capability — which almost always exists and only surfaces when you look.

CostPerformance