Dashboards with Grafana

Start from questions, not metrics

Before adding a panel, write down who will look at the dashboard and what they need to decide. For an on-call engineer who has just been paged, the questions come in a fixed order.

1. Is it really broken, and for whom? Are we meeting the SLO right now? How much error budget is left?
2. What are users experiencing? Request rate, error rate and latency for the service as a whole.
3. Where is it? Which route, which instance, which zone, which version, which dependency?
4. Why? Is a resource saturated: CPU, memory, connection pool, queue, disk?
5. What changed? Was there a deploy, a config push, a traffic spike, an autoscaling event?

Lay the dashboard out in the same order, top to bottom. The top row should be readable in five seconds and answer question one, so that someone glancing at it knows whether to worry. Detail goes lower down, in rows that can be collapsed. Keep one question per panel and give the panel a title that states it, such as "Error ratio by route" and not "Errors".

Build a small hierarchy instead of one giant board. An overview dashboard shows the health of every service on one screen, mostly as SLO status. Each service has its own dashboard, reached by a link from the overview. From there, links lead to the dashboards of its dependencies and resources: database, queue, nodes. An engineer should be able to move from "something is wrong" to "it is the connection pool on the orders database" in three or four clicks.

What goes on a service dashboard

The RED method gives you the core of any dashboard for a request-driven service: Rate, Errors, Duration. It is the user's view, and it should come first. The USE method covers every resource the service depends on: Utilisation, Saturation, Errors. It is the machine's view, and it explains the symptoms that RED reveals.

# Request rate by route
sum by (route) (rate(http_requests_total{job="$job"}[$__rate_interval]))

# Error ratio (a percentage when the unit is set to 'Percent (0.0-1.0)')
sum(rate(http_requests_total{job="$job", code=~"5.."}[$__rate_interval]))
/
sum(rate(http_requests_total{job="$job"}[$__rate_interval]))

# Latency percentiles: one query per percentile on the same panel
histogram_quantile(0.99, sum by (le) (
  rate(http_request_duration_seconds_bucket{job="$job"}[$__rate_interval])))

# CPU throttling: share of CPU periods in which the container was throttled
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{pod=~"$job-.*"}[$__rate_interval]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{pod=~"$job-.*"}[$__rate_interval]))

$__rate_interval is a Grafana variable designed for rate(). It picks a window that is always at least four scrape intervals and that grows as you zoom out, so graphs stay correct whether you look at the last fifteen minutes or the last thirty days. Use it in place of a hard-coded [5m] in dashboard queries. Alerting rules in Prometheus still need a fixed window.

Visualisation that does not mislead

• Set the unit on every panel: seconds, bytes, percent, requests per second. Grafana then formats 0.0042 as 4.2 ms and picks sensible axis labels. A graph without a unit cannot be read under pressure.
• Start ratio and count axes at zero, and fix the range of percentages to 0 to 100. An automatically scaled axis makes a change from 0.01% to 0.02% look like a catastrophe.
• Draw the SLO threshold on the graph, as a line or a shaded region, so that "is this bad?" needs no mental arithmetic.
• Use colour for meaning, not decoration. Reserve red for bad. Keep the same colour for the same series on every panel. Do not rely on red and green alone, since a meaningful share of people cannot tell them apart.
• Limit the number of series. A panel with sixty lines conveys nothing. Show the top five with topk, or aggregate, and put the detail on a drill-down.
• Prefer percentiles or a heatmap to averages for latency, and show p50 alongside p99, because the gap between them is itself informative.
• Use a shared crosshair across panels, so that hovering over a spike in errors shows the same moment on the latency and CPU graphs.

Variables, annotations and links

Variables turn one dashboard into a template. Instead of forty near-identical service dashboards that drift apart, you maintain one, with drop-downs at the top for service, environment and instance. A query variable fills its options from the data source, so new services appear automatically.

Name: job         Type: Query   Query: label_values(http_requests_total, job)
Name: route       Type: Query   Query: label_values(http_requests_total{job="$job"}, route)
                  Multi-value: on     Include All option: on
Name: percentile  Type: Custom  Values: 0.5,0.95,0.99

Used in a query:
  sum by (route) (rate(http_requests_total{job="$job", route=~"$route"}[$__rate_interval]))

For a multi-value variable, match with =~ and not =, because Grafana expands the selection into a regular expression such as (a|b|c). Variables can be chained, as route depends on job above, and repeating a row or panel over a variable draws one copy per selected value.

Annotations mark events as vertical lines across every graph. Deploys are the essential ones, because most incidents start with a change. When the line sits exactly where the error rate rises, the investigation is nearly over. Have your pipeline post an annotation at the end of each deploy.

curl -fsS -X POST "$GRAFANA_URL/api/annotations" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{\"tags\": [\"deploy\", \"orders\"], \"text\": \"orders ${GIT_SHA} deployed\"}"

Annotations can also be driven by a query, for example marking every moment when a build-info metric changes version. Links complete the workflow: dashboard links to go from the overview to the service, and data links on a panel that carry the clicked series and time range to a drill-down dashboard, to the matching logs, or to the traces for that route. Every alert should in turn link to the dashboard that explains it, so that the path from page to picture is one click.

Dashboards as code

Dashboards edited by hand in the UI share the problems of infrastructure changed by hand in a console: no review, no history, no way to recreate them, and differences between environments that nobody can explain. A dashboard is a JSON document, and Grafana can load data sources and dashboards from files at start-up. This is provisioning.

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

apiVersion: 1
providers:
  - name: services
    folder: Services
    type: file
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

• Keep the dashboard JSON in Git, review changes in pull requests, and let the pipeline deliver them. Design in the UI if you like, then export the JSON and commit it.
• With allowUiUpdates: false, changes made in the UI cannot be saved over the provisioned version, so Git remains the source of truth, which is the GitOps idea applied to dashboards.
• On Kubernetes, the kube-prometheus-stack chart runs a sidecar that loads any ConfigMap carrying a specific label as a dashboard, so a service's dashboard can ship in the same Helm chart as the service.
• For large estates, generate dashboards from code with Grafonnet, the Grafana Foundation SDK or Terraform's Grafana provider, so that every service gets the same standard layout.
• Refer to data sources by a variable or a stable name, not by an internal ID, so that the same JSON works in every environment.

Manage sprawl deliberately. Use folders and tags, give each dashboard an owner, and delete what is unused. Grafana's usage statistics show which dashboards nobody has opened in months. Ten dashboards that people trust are worth more than three hundred that nobody can find their way around.

Dashboards are not alerts

A dashboard only works when someone is looking at it, and nobody watches graphs at three in the morning. Anything that needs a human response must be an alert. A dashboard is what the human opens after the alert, to understand and diagnose. If your team finds problems by noticing them on a wall-mounted screen, the alerting has a gap.

Grafana has its own alerting engine, which can evaluate queries against any data source and route notifications. Whether you use it or Prometheus with Alertmanager matters less than keeping the rules in version control and applying the principles of the next module: alert on symptoms that users feel, tie alerts to SLOs, and make every page actionable.

Grafana is also not limited to metrics. The same dashboard can query logs in Loki or Elasticsearch, traces in Tempo or Jaeger, and SQL databases. Placing a log panel beneath the error graph, filtered by the same variables, saves a great deal of switching between tools during an incident.