AI in practice · Mixed

Operating an AI system over time

In one line: Models get replaced, documents change and users learn to ask differently — a system untouched for six months is almost always worse.

A maintenance cadence

Running the evaluation set once a month even with no change; a weekly review of a real conversation sample; and refreshing the knowledge base at the rate the sources change.

Moving between model versions

Vendors change and retire models. Keep an abstraction layer, run the evaluation set on the new version before the move, and transition gradually with a live comparison.

Don't assume a new version is always better on your task — it sometimes changes output style and breaks an analysis that relies on it.

What you watch

The share of answers marked negative, the rate of handoff to a human, the “no answer” rate, cost per request, and response time. Five numbers on one board.

Going deeper

Assign an owner to every model-based system, like any service. Such systems tend to be left ownerless after the project ends, and then the decline continues for months until someone complains — and the investigation starts from scratch.

Long-term maintenance Monitoring

Next: Tests: how many, which, and what isn't worth it · Previous: Architecture of a model-based system · Back to library