The Dashboard Is Green and the System Is Wrong

Here is the honest answer up front: in an LLM system, the only signal that really matters is output quality, and it is the one thing your infrastructure cannot measure for you. Uptime, latency, and error rate — everything a normal monitoring stack gives you for free — are all measuring the plumbing. A model that has quietly started citing a superseded document is, from your server's point of view, a perfectly healthy request.

So monitoring AI is not a matter of pointing Grafana at the right endpoint. It is a matter of manufacturing ground truth on purpose, because nothing produces it for you. If you skip that step, you get the worst possible state: a wall of green metrics that makes you feel informed while the thing degrades. Flying blind is survivable. Flying blind and believing your instruments is not.

The rule of thumb. If your AI monitoring would look identical whether the model was answering brilliantly or answering nonsense, you are not monitoring the AI. You are monitoring the web server it happens to run on.

The Five Signals Worth Monitoring

You do not need a large observability programme. You need five numbers, tracked over time, with someone who looks at them.

Quality, via a golden set

Take 50 to 200 real cases from your actual traffic, agree the correct answer for each once, and re-run them on a schedule and before every change. That is it. It is unglamorous and it works, and it is the only one of the five that measures the thing you actually care about. The hard part is not the tooling; it is that someone with domain knowledge has to sit down and decide what "correct" means for each case. Nobody enjoys that afternoon. It is the highest-leverage afternoon in the project. How to build and score that set is its own subject — I have written about evaluating LLM systems separately.

Retrieval hit rate — the cheapest signal nobody logs

If you run RAG, log whether the chunk that should have answered the question was in the context window at all. This is measurable without judging the answer, which makes it cheap, deterministic, and immune to the usual argument about whether an output was good. It is also decisive: if the right passage never reached the model, no amount of prompt work will save the answer. Most teams log the answer and not the retrieval, which means they debug the model when the corpus was at fault.

Cost per case, latency, and refusals

Track cost per resolved case, not cost per token. Per-token cost is an accounting fact and it barely moves; an agent that has quietly started retrying three times shows a perfectly normal per-token cost and triple the bill. Only the per-case number catches that. For latency, watch p95 rather than the average, and watch it for a human reason: when a system gets slow, people do not complain, they stop using it — or worse, they stop reading the output and start clicking accept. And track the refusal rate: the share of answers that hedge or decline. A refusal rate that climbs without any change on your side is one of the earliest visible fingerprints of a provider-side model update.

How Drift Actually Shows Up

"Drift" sounds like a slow, natural process. In practice it arrives in four distinct flavours, and only one of them is the one people expect.

Uptime · latency · error rate: all green throughout Provider update same prompt, new behaviour → golden set Corpus drift a new revision lands beside the old → retrieval hit rate User drift new phrasing, new questions → refusal rate Cost drift the agent quietly retries → cost per case Each kind of drift has exactly one signal that sees it first.
Four kinds of drift, four different early-warning signals. Not one of them trips an infrastructure alert — which is why a green dashboard tells you nothing about whether the system still works.

The provider ships a better model and your system gets worse. This is the one that feels unfair, and it is real. You call the same endpoint with the same prompt and the behaviour changes, because a model behind that endpoint was updated. The counterintuitive part: the new model is usually genuinely better on average and still worse for you. Prompt engineering is partly overfitting — the instructions you tuned over three weeks are shaped around one model's specific quirks, its habit of over-explaining, the phrasing that reliably stopped it inventing a part number. Fix those quirks upstream and your carefully shaped workaround is now a workaround for a problem that no longer exists, applied to a model that responds to it differently. "Better on the vendor's benchmarks" and "better on your task" are separate claims, and only one of them is being tested by your golden set.

Your Corpus Drifts, and It Is Almost Always This One

Here is the failure I see most, and it has nothing to do with the model. A machine builder runs a service assistant over its manuals. For eight months it is excellent. Then the answers get subtly worse — not wrong enough for anyone to file a ticket, just worse. No code changed. No model changed. What happened is that someone uploaded the 2026 revision of a manual without removing the 2023 one, and retrieval now returns the superseded torque spec about half the time. The model is behaving perfectly: it is faithfully summarising the wrong document.

Nothing in a conventional monitoring stack sees this. Retrieval hit rate sees it the day it happens. A golden set sees it at the next scheduled run. Uptime will never see it, and neither will your users until a technician torques something to a 2023 specification. The lesson generalises: in a RAG system, your document pipeline is a production dependency, and it is mutated by people who have no idea they are deploying to production.

Pinning a Model Version Is a Deadline, Not a Fix

The obvious response to provider drift is to pin the version, and you should — never point a production system at a floating "latest" alias. But be clear about what you have bought, because it is not stability. Pinned model snapshots get deprecated on the provider's schedule, with notice typically measured in months. Pinning does not remove drift; it converts a continuous, small, observable drift into a single discrete migration on a date somebody else picks. You will re-qualify everything at once, under time pressure, probably in a quarter that already has plans.

That trade is usually still worth it. But it means the pinned system needs the golden set more than the unpinned one, not less — it is the only thing that turns that forced migration from a fortnight of anxious manual spot-checking into an afternoon of running a suite and reading a diff. This is also the strongest operational argument for self-hosting an open-weight model, and it is a different argument from the data-protection one people usually reach for: a model sitting on your own hardware cannot be deprecated out from under you. The cost is that you now own the upgrade decision forever, including the one where you decide not to upgrade and slowly fall behind. That is a real cost. It is just a cost you control the timing of.

What I would not build. An LLM-as-judge scoring every production call in real time. It puts a second model call on every request, adds latency to the user's request, and — the part that kills it — the judge is itself a model that drifts. You would be monitoring an instrument you are not monitoring. Sample a slice of traffic, judge it offline, and keep the human-verified golden set as the thing you actually trust.

What an Honest Maintenance Budget Looks Like

Maintenance for an AI system is not a contingency line. It is a standing monthly item, and the useful question is not "what percentage should we reserve?" — it is "whose name is on it, and is the time actually in their week?" A budget with money but no named person is a system that degrades on schedule.

Concretely, a month of upkeep on a well-built system is: re-run the golden set (automated, minutes); read a sample of real traffic (about an hour, and this is the part everyone skips and the part where you learn the most — you will find questions nobody anticipated); triage retrieval misses; apply dependency and model updates. Add a reserve for the forced migration when a pinned model is deprecated, because that one is not optional and not small. This is the running-cost side of what I cover in what an AI project actually costs, and it is the line that decides whether the thing is a good investment over three years rather than three months.

Two things make this cheaper than it sounds. Instrument with OpenTelemetry from day one rather than bolting it on after the first incident — its GenAI semantic conventions mean traces of prompts, tool calls, and token counts land in tooling you may already run, instead of in a bespoke logging layer you now maintain forever. And if the system falls under the EU AI Act's high-risk regime, logging and post-market monitoring stop being engineering preferences you can defer and become obligations — the text is more specific about them than most teams expect. Building the monitoring in is cheaper than retrofitting it for an audit.

Where Tippel Fits

Most of the monitoring work above is decided at build time, not after. Whether you can measure retrieval hit rate at all depends on how the pipeline was assembled; whether the golden set exists depends on whether someone insisted on it before launch. Retrofitting observability into a system that was shipped without it is doable and it is never the cheap path.

If you have an AI system in production that nobody is quite sure is still working, that is a concrete, answerable question and the AI Readiness Check is a sensible way to answer it: a fixed-price, timeboxed look at what the system is actually doing on your real traffic, ending in a plain assessment rather than a dashboard nobody reads. If you are still building, the cheaper move is to design the five signals in now. Either way, get in touch.