Prometheus scraping and staleness: missing is not zero
Diagnose failed scrapes, removed targets and disappearing metric series. Understand up, lookback and stale markers before treating dashboard gaps as recovery.
TL;DR: A failed scrape, a removed target and a successfully scraped exporter that stops emitting a metric are different events. Inspect target discovery and scrape health before assigning meaning to a missing series.
A time series starts with a discovered target
Prometheus obtains targets through configuration or service discovery, applies target relabeling and attempts scrapes. Each accepted sample belongs to a series identified by its metric name and complete label set. Changing an instance label or adding a deployment label creates different series, even if the application is logically the same service.
Prometheus generates up for scraped instances: one for a successful scrape and zero for a failed scrape. Its jobs and instances guide describes the generated scrape metrics. up = 1 means the scrape succeeded, not that the business operation works. A payments exporter can return metrics while every payment fails.
The diagram explains why a single dashboard panel cannot identify all three failure paths.
Read a rollout as a sequence of observations
Consider an illustrative worker scraped every 15 seconds. At 12:00:00, up is one and jobs_waiting is 7. At 12:00:15, the endpoint times out. At 12:00:30, scraping succeeds again, but a new exporter version omits jobs_waiting when the queue is empty.
The failed scrape produces up = 0. The later successful scrape returns up = 1, yet the application metric is still missing. If the old exporter emitted an explicit zero for an empty queue, this deployment changed the measurement contract. A graph that converts every gap to zero conceals that change.
| Evidence | Likely path to investigate | What it does not establish |
|---|---|---|
Target listed, up = 0 | Scrape timeout, TLS, authorization or parse error | That the entire service is down |
Target listed, up = 1, application metric absent | Exporter behavior or metric relabeling | That the measured quantity equals zero |
| Target disappeared from discovery | Discovery source, selectors and target relabeling | That no workload should exist |
| Same metric under a new label set | Deployment identity or relabeling change | Continuity of the old series |
Inspect the target's last scrape error and the exporter response from an authorized environment. A scrape timeout can be caused by the measurement endpoint itself becoming expensive, so increasing the timeout without checking duration and sample volume can worsen collection pressure.
Lookback is not a universal grace period
Instant-vector selection normally looks back for a recent sample. Prometheus's default lookback is five minutes, but staleness markers can make a series disappear from instant results earlier. Therefore, “Prometheus always keeps the last value for five minutes after a target fails” is an incorrect operating rule. The querying basics reference explains both mechanisms and the special considerations for exporter-supplied timestamps.
Range queries and range-vector functions answer different questions from a current instant selector. A five-minute range can contain earlier samples even after the current series has disappeared. Examine the query and evaluation time before comparing a dashboard graph with an instant API result.
Avoid encoding staleness timing as a fixed outage timer without testing your target-removal path and configuration. Discovery changes, scrape failures and exporter timestamps do not all have identical observation behavior.
Monitor absence against an expected population
These expressions inspect different conditions for an illustrative billing-worker job:
up{job="billing-worker"} == 0
absent(up{job="billing-worker"})
The first selects failed discovered instances. The second reports that the entire selected up vector is absent at evaluation time. It cannot enumerate which of several expected instances disappeared while others remain. For that, compare against an authoritative expected-target inventory or an appropriate workload availability signal.
Likewise, alerting on an application queue depth needs a policy for missing telemetry. Establish whether the exporter always emits the gauge and how collection failure is reported. Keep monitoring availability distinct from the application's service-level indicator.
Check a misleading recovery graph
Self-check: an alert based on jobs_waiting > 100 stops firing immediately after a deployment, while no worker has completed a job. Has the queue recovered?
Not necessarily. The metric may have vanished, changed labels or stopped matching the expression. Verify the expected target and metric, then compare queue state with processing evidence. An absent vector usually provides no element for a threshold expression to select. Review PromQL vector matching when the alert combines that gauge with another metric; a failed match can also remove the result.