Prometheus metric types: counters, gauges, histograms and useful queries
Choose Prometheus counters, gauges and histograms correctly. Work through request rates, error ratios, cumulative buckets and fleet-wide percentiles with explicit PromQL assumptions.
TL;DR: Count completed events with counters, measure values that rise and fall with gauges, and record distributions with histograms or supported summary instrumentation. Apply rate calculations before aggregating counters, and preserve the dimensions required by the histogram representation you query.
The metric type encodes the measurement
A counter accumulates events and can reset, commonly when its process restarts. A gauge represents a value that can increase or decrease. Histograms describe distributions; summaries can expose client-computed quantiles along with count and sum. The Prometheus metric-types guide explains these models, including the distinction between classic and native histograms.
| Measurement | Appropriate model | Reason |
|---|---|---|
| Requests completed | Counter | Each completion adds one event |
| Requests currently executing | Gauge | The value falls as requests finish |
| Request duration distribution | Histogram | Tail behavior and threshold fractions matter |
| Available queue capacity | Gauge | Capacity can rise or fall |
| Bytes transmitted since process start | Counter | Throughput comes from change over time |
Do not apply rate() to a gauge merely because it is a numeric series. Its decreases are legitimate measurements, while counter-rate logic interprets decreases as resets. Choose a query that matches the instrumented meaning.
Rate each counter before summing
Assume http_requests_total is a counter with labels service, instance, route and status, and all examples use a five-minute range containing enough scrape samples. For a fleet-wide request rate:
sum by (service) (rate(http_requests_total[5m]))
rate() handles observed counter resets within each series. Summing counters from different processes before computing a rate can hide individual resets behind another process's increase. The PromQL function reference documents the required ordering and extrapolation behavior.
For an illustrative service that defines server-side 5xx responses as bad requests:
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (service) (rate(http_requests_total[5m]))
That ratio is a fraction, so multiply by 100 only when you need a percentage display. Decide how your real service counts client cancellations, rejected requests and business failures before using this as an SLI. Missing series and zero traffic need deliberate handling; an absent result is not evidence of zero errors.
Classic histogram buckets are cumulative
Suppose 100 observed requests produce the following illustrative bucket counts. Each bucket includes every observation less than or equal to its boundary.
| Upper bound in seconds | Cumulative count | Interpretation |
|---|---|---|
| 0.1 | 40 | Forty requests completed within 100 ms |
| 0.3 | 85 | Eighty-five completed within 300 ms |
| 1.0 | 98 | Ninety-eight completed within one second |
| +Inf | 100 | Total observations |
The count between 0.1 and 0.3 seconds is 45, obtained by subtraction. Summing all bucket counts would count the same requests multiple times. For a 300 ms threshold, this sample has 85% of observations at or below the threshold.
The diagram illustrates selected classic buckets. The le=0.1 bucket is unchanged for this observation. Native histograms use a different representation and different query details.
Aggregate the distribution, then estimate the quantile
For classic histograms named http_request_duration_seconds, with compatible bucket boundaries across instances, a service-level p95 can be estimated as:
histogram_quantile(
0.95,
sum by (service, le) (
rate(http_request_duration_seconds_bucket[5m])
)
)
Keep the le label because it identifies the classic bucket boundary. The estimate depends on bucket placement and interpolation. Put useful boundaries near thresholds you must evaluate, and understand the error before interpreting a small p95 movement as a real regression.
Averaging per-instance p95 values does not produce the fleet p95. An instance with ten requests would receive the same weight as one with ten thousand, and even a request-weighted average of percentiles does not reconstruct the combined distribution. Aggregate compatible histogram observations first. The histograms and summaries guidance explains aggregation and quantile tradeoffs; check its version-specific notes when adopting native histograms.
Cardinality is part of the schema
Each distinct combination of labels creates another series. Adding user_id to a metric with route, region and status can produce unbounded growth. For a classic histogram, each label combination also produces multiple bucket series plus sum and count. Budget the dimensions before instrumenting a high-volume service.
Use cardinality discipline to select bounded labels. Keep detailed request identity in logs or traces and use supported correlation mechanisms where available. Metrics should remain economical enough to query during an incident.
Check the arithmetic and the meaning
A service records 20 bad requests and 980 good requests during the same observation window. The error fraction is 20/1,000, or 2%. Dividing bad by good yields about 2.04%, which answers a different question.
Self-check: two instances report p95 values of 100 ms and 900 ms. Can you say the service p95 is 500 ms?
No. You need the combined distribution and its observation counts, with comparable measurement definitions. Use aggregatable histograms or another suitable distribution representation. Then verify the query includes the intended service, route set and time range.
Practice Prometheus metric types, then connect a correctly defined bad-event fraction to error budgets and burn rates.
When several consumers repeat a query, recording rules can save derived series. Preserve the labels and weighting that later decisions need.
Native histogram migration develops the representation change, query differences and coverage checks needed before retiring classic buckets.