Grafana dashboards: operational questions, query correctness, and provisioning

TL;DR: A useful Grafana dashboard lets an operator decide what to inspect next. Start with a question such as “Is checkout failing for customers in one region after this release?” Then choose panels that establish the affected population and time boundary. A wall of host metrics may be accurate while giving no answer to that question.

For an illustrative checkout service, the opening view should show request volume and customer-visible failures over the same interval. Add latency with its unit and percentile explicitly named, then a deployment annotation and links to regional or dependency detail. Grafana's dashboard guidance recommends a consistent hierarchy that lets readers move from an overview into more specific evidence.

Make the first screen a decision point

A link to logs should preserve the service and environment filters and, where supported, the incident's time range. A reader investigating yesterday's spike should not land on today's logs. Keep a visible indication of the current time range, especially when an incident link opens a dashboard with a fixed interval.

Do not put every dimension into the first view. Provide focused drill-downs for route, zone, or deployment. High-cardinality variables can make the dashboard slow exactly when the service is under pressure. Review query latency and data-source load as part of dashboard maintenance.

Weight errors by the traffic they represent

Assume a Prometheus counter named http_requests_total has service and status labels. This example treats HTTP 5xx responses as failures and sums across instances:

sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{service="checkout"}[5m]))

Apply a dashboard percentage unit that expects a ratio from zero to one. The exact label names and the definition of a failed request must match the instrumentation. For a customer-facing SLI, a successful HTTP response containing an application error may also need to count as a failure.

In a small arithmetic example, instance A handles 90 requests with 9 failures, while B handles 10 requests with 5 failures. Their percentages are 10% and 50%. The service ratio is 14 / 100 = 14%; averaging their percentages gives a misleading 30%. The query calculates rates before aggregation so counter resets can be handled per series. See Prometheus rate semantics.

No traffic and no telemetry require separate treatment. Division by a zero denominator can produce NaN; absent series can produce no result at all. Decide how the panel should display each case and inspect an independent telemetry-health signal. Filling every gap with zero can turn a failed collection path into a reassuring picture. A graph is also separate from an alert rule: verify alert evaluation, routing, and no-data behavior in their own configuration.

Keep the reviewed definition authoritative

For classic file provisioning, store dashboard JSON beside a provider configuration in the deployed Grafana instance. This provider example points to a directory inside that instance:

apiVersion: 1
providers:
  - name: service-dashboards
    orgId: 1
    folder: Services
    type: file
    disableDeletion: true
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

A stable dashboard UID keeps links usable across updates. allowUiUpdates: false prevents saving edits to provisioned dashboards through the UI. With disableDeletion: true, removing the source file does not automatically remove the dashboard; retirement needs an intentional cleanup step. These choices suit a reviewed repository workflow, but neither setting validates whether the panel queries are useful.

If UI updates are allowed, a later provisioning update can overwrite them. Export intended edits and review the source change before the next rollout. Do not rely on the JSON version field to protect a UI edit: file provisioning can overwrite the stored dashboard regardless of that number. Grafana provisioning documents this precedence.

Check the interpretation

A dashboard shows zero errors while its request-rate panel has no data. Is the service healthy? The evidence is insufficient. Verify the data source and scrape path, then distinguish genuinely idle traffic from missing telemetry.

A dashboard edit disappears after a release. Where should the fix go? Find the provisioning source and make the reviewed change there. Also verify the dashboard UID and provider so the edit reaches the intended object instead of creating a second dashboard with a similar name.

When a dashboard supports a load-test decision, JMeter workload models helps distinguish genuine service capacity from a generator that reduced its own demand.