Metrics with Prometheus

How Prometheus works

Prometheus pulls. Each application exposes its current metric values as plain text on an HTTP endpoint, conventionally /metrics. On a fixed scrape interval, commonly 15 to 60 seconds, Prometheus fetches that page from every target, timestamps the values and stores them in its local time series database. Software you cannot instrument, such as a Linux host, PostgreSQL or an HTTP endpoint you want to probe, is covered by an exporter: a small process that translates its state into the same format.

# HELP http_requests_total Total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="GET",route="/orders",code="200"} 184223
http_requests_total{method="GET",route="/orders",code="500"} 37
http_requests_total{method="POST",route="/orders",code="201"} 9120
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.2843008e+08

Pulling has practical advantages. Prometheus knows what it expects to scrape, so a target that stops answering is itself a signal: the automatic up metric becomes 0. You can open any /metrics page in a browser to debug it. And the monitoring system, not each application, controls the load.

global:
  scrape_interval: 30s
  evaluation_interval: 30s

rule_files:
  - rules/*.yml

scrape_configs:
  - job_name: orders
    static_configs:
      - targets: ["orders-1:8000", "orders-2:8000"]

  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

In dynamic environments you do not list targets by hand. Service discovery asks Kubernetes, a cloud API or Consul what exists, and relabelling decides which of those to scrape and which labels to attach. In Kubernetes most teams install the Prometheus Operator, usually through the kube-prometheus-stack Helm chart, and describe targets with ServiceMonitor objects instead of editing this file.

Every sample belongs to a time series, identified by a metric name plus a set of labels. http_requests_total{method="GET",code="200"} and http_requests_total{method="GET",code="500"} are two different series. Prometheus adds job and instance labels to everything it scrapes.

Metric types, names and labels

Prefer a histogram to a summary. Histogram buckets from many instances can be added together and a quantile computed over the whole service, while quantiles precomputed by a summary cannot be meaningfully averaged. Choose bucket boundaries around the values you care about, and put one exactly at your SLO threshold, so that "proportion of requests under 500 ms" is an exact count and not an estimate.

import time

from prometheus_client import Counter, Gauge, Histogram, start_http_server

REQUESTS = Counter(
    "http_requests_total", "Total HTTP requests handled.", ["method", "route", "code"]
)
LATENCY = Histogram(
    "http_request_duration_seconds", "Request latency in seconds.", ["route"],
    buckets=[0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
)
IN_FLIGHT = Gauge("http_requests_in_flight", "Requests currently being served.")


def handle(method: str, route: str) -> None:
    IN_FLIGHT.inc()
    start = time.perf_counter()
    code = "500"
    try:
        code = do_work(route)                  # returns the status code as a string
    finally:
        LATENCY.labels(route=route).observe(time.perf_counter() - start)
        REQUESTS.labels(method=method, route=route, code=code).inc()
        IN_FLIGHT.dec()


start_http_server(9000)                        # serves /metrics on port 9000

• Names state what is measured and end with the base unit: _seconds, _bytes, and _total for counters. Never milliseconds or megabytes; conversion is the dashboard's job.
• Labels are dimensions you will want to filter or group by: method, route, code.
• Label the route template, such as /orders/{id}, never the raw path /orders/48213.

The four golden signals

The Google SRE book names four signals that, if you could measure only four things about a user-facing service, you should choose. They line up with the SLIs from the previous stage.

Two related checklists are worth knowing. RED, for request-driven services, is Rate, Errors and Duration, which is the first three golden signals. USE, for resources such as CPUs, disks and network links, is Utilisation, Saturation and Errors. Use RED for every service and USE for every resource it depends on.

Latency needs care in two ways. A fast error is still an error: if failing requests return in five milliseconds, a wave of failures makes your average latency look better, so track the latency of successful requests separately. And averages hide the experience of the users who suffer most, which is why you work with percentiles from histograms.

PromQL you will use every day

A selector such as http_requests_total{job="orders", code=~"5.."} returns an instant vector: the latest value of each matching series. Adding a range, [5m], returns a range vector: all samples from the last five minutes. Functions like rate() take a range vector and return an instant vector. Matchers are =, !=, =~ for a regular expression match and !~ for its negation.

# TRAFFIC: requests per second, per route
sum by (route) (rate(http_requests_total{job="orders"}[5m]))

# ERRORS: share of requests failing with a 5xx
sum(rate(http_requests_total{job="orders", code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="orders"}[5m]))

# LATENCY: 99th percentile across all instances
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="orders"}[5m]))
)

# LATENCY as an SLI: proportion of requests served within 500 ms
sum(rate(http_request_duration_seconds_bucket{job="orders", le="0.5"}[5m]))
/
sum(rate(http_request_duration_seconds_count{job="orders"}[5m]))

# SATURATION: memory used as a share of the container limit
container_memory_working_set_bytes{container="orders"}
/
container_spec_memory_limit_bytes{container="orders"}

• Always rate() a counter. The raw value is a large number that depends on when the process last restarted. rate() gives the per-second increase over the window and handles counter resets for you. increase() is the same thing multiplied by the window length.
• rate first, then sum. Take the rate of each series and then aggregate. Summing first destroys the reset detection, so one restarting pod produces a huge false spike.
• Keep le when aggregating histogram buckets. histogram_quantile needs the bucket boundaries, so write sum by (le), adding any other labels you want to break down by, such as sum by (le, route).
• Make the range at least four times the scrape interval, so that each window always contains enough samples. With a 30 second scrape, [2m] is the minimum and [5m] is a comfortable default.
• Never average percentiles. The average of each pod's 99th percentile is not the service's 99th percentile. Aggregate the buckets, then compute the quantile once.
• sum by (label) keeps the labels you name. sum without (label) drops the ones you name and keeps the rest.

up{job="orders"} == 0                                   # targets that are failing to scrape
topk(5, sum by (route) (rate(http_requests_total[5m])))  # the five busiest routes
absent(up{job="orders"})                                # returns 1 if the series does not exist at all
predict_linear(node_filesystem_avail_bytes[6h], 4 * 3600) < 0   # disk full within 4 hours?
rate(http_requests_total[5m] offset 1w)                 # the same query, one week ago

Recording rules

Some expressions are expensive, and some are used in many places: dashboards, alerts, SLO reports. A recording rule evaluates an expression on a schedule and stores the result as a new time series. Queries then read one precomputed series instead of aggregating thousands. Just as important, the definition of "error ratio" exists in exactly one place, so the dashboard and the alert cannot disagree.

groups:
  - name: orders-sli
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_requests_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

The naming convention is level:metric:operations. level lists the labels that remain after aggregation, metric is the original metric name, and operations lists what was applied, newest first. job:http_requests_errors:ratio_rate5m therefore reads as: aggregated to the job level, from the requests metric, a ratio of five-minute rates.

promtool check config prometheus.yml          # validates the config and every rule file it references
promtool check rules rules/orders.yml
promtool test rules tests/orders_test.yml     # unit tests for rules and alerts
curl -s -X POST http://localhost:9090/-/reload   # reload; needs --web.enable-lifecycle

Treat rules as code: keep them in Git, validate them with promtool in CI, and review changes. The alerting module builds burn-rate alerts directly on top of recording rules like these.

Running it in production

• Local storage is not long-term storage. A single Prometheus keeps data on local disk, with a default retention of 15 days, and is not clustered. For high availability, run two identical servers scraping the same targets. For long retention and a global view across clusters, add a system built for it, such as Thanos, Grafana Mimir or Cortex, fed through remote_write.
• Watch your own cardinality. prometheus_tsdb_head_series shows the number of active series. The query topk(10, count by (__name__) ({__name__=~".+"})) lists the metrics with the most series. Drop unwanted series or labels at scrape time with metric_relabel_configs.
• Monitor the monitor. Alert on up == 0, on scrape failures, and on rule evaluation failures. Add a dead man's switch: an alert that always fires, routed to an external service that raises the alarm when it stops arriving.
• Short-lived jobs cannot be scraped reliably. Batch jobs push their final metrics to the Pushgateway, which Prometheus then scrapes. Use it only for that purpose, never as a general way to push metrics.
• Metrics tell you that something is wrong and where. They rarely tell you why for one specific request. That is the job of logs and traces, covered in the OpenTelemetry module.