Model quality degrades without touching a single infrastructure metric
Classical dependencies and models can both return incorrect results without an error. A model fails by returning something plausible and wrong, which every status-code-based metric records as a success. Quality needs its own signals, its own threshold and its own budget, or it is unobservable until a customer complains.
TL;DR: Availability built on HTTP status codes reports 100 percent through an outage where every answer was useless. Detect quality with three layers: a scheduled offline suite for changes you made, online proxy signals for changes you did not, and a sampled review to keep the proxies honest. Then give the primary quality metric a threshold and an error budget so it behaves like any other SLO.
Successful transport can carry a wrong answer
A database can return stale rows, a cache can contain the wrong value and a queue consumer can process a message incorrectly. Availability metrics do not establish business correctness. Models make this distinction especially visible because open-ended output often needs statistical or human evaluation.
A model returns a confident, well-formed, wrong answer. It returns a refusal to a reasonable question. It returns prose where JSON was required. It answers a question nobody asked. All of that is an HTTP 200 with normal latency, and no dashboard moves.
What causes it, and why deploy gates are not enough
Check these possible changes:
- a model version changed, yours or a provider's
- a prompt template was edited three services away
- the retrieval index fell behind, so the model is reasoning over stale context
- someone shipped a quantised build to save money
- a tokeniser or truncation change silently dropped the end of long prompts
- the questions users ask moved away from what the system was tuned for
Only the first few are under your change control. A provider-side change arrives with no deploy of yours, which is why detection has to run on a schedule rather than only in a pipeline.
Three layers
A golden set, run on a schedule. A fixed, versioned collection of representative inputs with known-good expectations, scored automatically and stratified by the segments that matter: long context, each language, each customer tier. Aggregate-only scoring hides the regression that matters, which is usually confined to one stratum.
Online proxies, from real traffic. These move before anyone files a ticket, and the sharpest of them need no labels at all:
- structured-output parse and schema-validation rate
- regeneration or retry rate, which is a user saying the answer was not good enough
- tool-call validity, for anything agentic
- fallback-to-a-larger-model rate, if you cascade
- refusal and empty-answer rate
A sampled review. A small daily sample scored against a rubric, by a person or by a grading model validated against human labels. This keeps the proxies honest, because users adapt to a worse product without complaining.
Establish the noise band before the threshold
Put the unchanged system through the same suite several times and watch how far the score wanders by itself. Choose a sample size and uncertainty estimate for the smallest meaningful change, including dataset and generation variability. A score inside a measured range is inconclusive rather than proof of no regression.
Make it an SLO
For a defined per-request quality success criterion, set a target and budget. A noisy mean grader score needs a sampling model and an actionable threshold before it can drive paging; it is not automatically an availability-style burn rate. That converts a vague argument about whether the model feels worse into the same mechanics as availability, and it gives a platform team the standing to block a model change, which is otherwise an argument they lose.
The organisational half
The golden set belongs to whoever owns the product outcome; the pipeline belongs to the platform. A golden set curated by infrastructure engineers drifts away from what the product cares about within a quarter, and a quality gate nobody in product recognises gets overruled the first time it blocks a launch.
Self-check
An unchanged grader reports a one-point quality drop on a small sample. What would justify paging? Check sample uncertainty, affected segments and user impact, and identify an action the responder can take. One uncalibrated score alone is insufficient.