Prometheus recording rules: useful aggregates and reliable evaluation
Design Prometheus recording rules that preserve useful labels and correct ratios. Test counter resets, missing series and rule health before dashboards depend on them.
TL;DR: A recording rule stores the result of a PromQL expression as new time series. Choose its labels and units as an interface, test the expression, and monitor evaluation health before relying on it for alerts or dashboards.
A stored calculation has a contract
Suppose several dashboards repeatedly calculate the request rate for each service and status code. A recording rule computes that expression on its evaluation schedule and saves the resulting samples. Consumers can then query the smaller result instead of repeating the original aggregation. The saved series still consumes storage, and a poorly bounded output label set can create substantial additional cardinality.
Prometheus evaluates rules within a group sequentially at the same evaluation timestamp. A later rule can use a result from an earlier rule in that group. Separate groups do not offer the same ordering contract. The recording-rule reference describes group intervals, result limits and failed evaluation behavior.
The branching matters: preserve the numerator and denominator so other consumers can aggregate correctly. A ratio alone throws away the weight needed to combine groups.
Rate first, then aggregate
Save this rule file as requests.rules.yml. It assumes the application exposes a counter named http_requests_total with service, instance and code labels. The error definition is HTTP 5xx, chosen for this teaching example; an actual SLI may classify outcomes differently.
groups:
- name: service-requests
interval: 1m
rules:
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))
- record: service:http_errors:rate5m
expr: sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
- record: service:http_error_ratio:rate5m
expr: service:http_errors:rate5m / service:http_requests:rate5m
Calculating rate before summing lets Prometheus detect resets in each original counter. After aggregation, independent resets can be hidden by growth in another instance. The names describe the retained level, measured quantity and calculation; the Prometheus naming guidance also explains why ratios should be calculated from aggregated numerators and denominators.
In a hypothetical five-minute window, service A receives 100 requests per second with 1 error per second. Service B receives 1 request per second and every request fails. Their error ratios are 1% and 100%. Averaging those percentages gives 50.5%, while the combined request-weighted ratio is 2 / 101, about 1.98%. Neither figure explains both services on its own. Retain the service label for service-level response, and use weighted totals only when the combined population is the intended contract.
Missing data is part of the interface
If no 5xx series exists, the error-rate expression can return no series for that service. The ratio then disappears too. That differs from an explicit zero-valued error series. Decide whether instrumentation should initialize expected label combinations, or whether a query should fill zero only for a known, healthy request population. A bare or vector(0) does not attach the missing service labels and can disguise collection failures.
| Condition | Consequence | Review action |
|---|---|---|
| Instance restarts | Counter decreases | Test per-series reset handling |
| A label is removed by aggregation | Later queries cannot recover it | Keep dimensions required by consumers |
| Error series never exists | Ratio can be absent | Define initialization or guarded zero handling |
| Request rate is zero | Ratio can be undefined | Treat no traffic separately from good traffic |
| Rule evaluation fails | Derived samples stop arriving | Monitor evaluator health and freshness |
A rule change affects future samples. It does not rewrite already stored history, so reusing a metric name for a different error definition mixes two meanings in one graph. Introduce a new name or coordinate a documented migration, including dependent alert expressions and comparison windows.
Test the query before wiring the alert
Run promtool check rules requests.rules.yml for syntax. Then use rule unit tests with synthetic counter samples and expected output labels and values. Syntax success alone cannot catch the percentage-averaging error above. Include a reset, no traffic and a missing error series alongside the steady-state case.
Also inspect rule duration, missed evaluations and failures under realistic input cardinality. If evaluation takes longer than its interval, a fast dashboard query can still be showing delayed evidence. Faster evaluation increases work; first remove unnecessary dimensions or expensive repeated expressions.
Self-check: an engineer aggregates away region to save storage, then wants a regional error ratio from the recorded service totals. Can a join with region metadata recover it? No. Metadata can attach a label, but it cannot reconstruct how the original traffic was divided. Keep region in the numerator and denominator when regional decisions are required, or query the retained raw series for that purpose.