Traces & logs with OpenTelemetry

Three signals, one investigation

A typical investigation uses all three, in order. A metric-based alert says the error budget is burning. The dashboard narrows it to one route. You open traces for slow or failed requests on that route and see that the time is spent in a call to the inventory service. You then read the logs for that exact span and find the cause: a timeout connecting to a replica. Each signal hands you to the next, provided they share identifiers.

Monitoring answers questions you thought of in advance: is the error rate above the threshold? Observability is the ability to ask new questions about a system's behaviour from the outside, without shipping new code, which matters most for the failures nobody predicted. High-cardinality detail, such as customer ID, build version and feature flag state, is what makes that possible, and it lives in traces and logs because it cannot live in metric labels.

OpenTelemetry, usually shortened to OTel, is a CNCF project that provides a specification, APIs and SDKs for most languages, a wire protocol called OTLP, and the Collector. It is not a backend: it does not store or display anything. You send its output to Jaeger, Tempo, Prometheus, Loki, Elasticsearch or a commercial vendor, and you can change your mind later without touching your application code. That freedom from lock-in is the reason it became the standard.

Traces, spans and context propagation

A trace represents one request end to end. It is made of spans, each representing one unit of work: handling an HTTP request, running a database query, calling another service. A span has a name, a start time and a duration, a status, a set of key-value attributes, optional timestamped events, and the ID of its parent span. The parent links assemble the spans into a tree, usually drawn as a waterfall.

trace 4bf92f3577b34da6a3ce929d0e0e4736                          total 1,240 ms

GET /checkout                 [==============================================] 1240 ms
  auth.verify_token            [==]                                              38 ms
  cart.load                       [====]                                         95 ms
    SELECT cart_items                [==]                                        41 ms
  POST inventory/reserve               [==============================]          870 ms  ERROR
    SELECT stock ... FOR UPDATE           [==========================]           812 ms
  payment.authorize                                                  [=======]  190 ms

One glance shows what no metric could: the request spent 870 of its 1,240 milliseconds reserving inventory, nearly all of it waiting on one locking query. The waterfall also reveals structure, such as calls made one after another that could run in parallel, or a query executed forty times in a loop.

For this to work across processes, the trace identity must travel with the request. This is context propagation. When a service makes an outgoing call, the SDK injects the current trace and span IDs into the request headers, and the receiving service extracts them and continues the same trace. The standard header is W3C Trace Context:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             |  |                                |                |
             |  trace-id (32 hex chars)          |                trace-flags (01 = sampled)
             version                             parent span-id (16 hex chars)

Instrumenting a service

OpenTelemetry separates the API, which your code calls to create spans, from the SDK, which decides what happens to them. Libraries can depend on the lightweight API safely: if no SDK is configured, the calls do nothing. Automatic instrumentation gets you most of the value with no code changes, by patching common frameworks and clients, such as web servers, HTTP clients, database drivers and messaging libraries, to create spans and propagate context.

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install          # adds instrumentation for the libraries it finds

export OTEL_SERVICE_NAME=checkout
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=staging,service.version=1.4.2
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1

opentelemetry-instrument python app.py

The OTEL_* environment variables are standard across languages, which fits the twelve-factor approach to configuration. service.name is the most important resource attribute: it labels every span, metric and log from this process, and without it your backend shows a service called unknown_service. Java has an equivalent agent attached with -javaagent, and Node.js and .NET have their own auto-instrumentation packages. On Kubernetes, the OpenTelemetry Operator can inject instrumentation into pods by annotation.

Automatic instrumentation knows about HTTP and SQL, not about your business. Add manual spans around the operations that matter, and attach the attributes you will want to search by.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)


def reserve_stock(order_id: str, items: list[dict]) -> None:
    with tracer.start_as_current_span("inventory.reserve_stock") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.item_count", len(items))
        try:
            for item in items:
                lock_and_decrement(item["sku"], item["qty"])
            span.add_event("stock reserved")
        except StockUnavailable as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, "stock unavailable"))
            raise

• Follow the semantic conventions for attribute names, such as http.request.method, http.response.status_code, db.system and server.address. Backends build their views on those names, and consistent names make queries work across services written in different languages.
• Attributes can be high cardinality, which is the whole point. Customer ID, order ID and feature-flag state are exactly what you will want to filter by.
• Never put secrets, tokens, passwords or unnecessary personal data in attributes. Trace data is widely readable and kept for a long time.
• Name spans after the operation, not the instance: GET /orders/{id}, not GET /orders/48213.
• Do not create a span for every function call. Spans cost memory and money; instrument boundaries and meaningful units of work.

Logs that correlate

Logs are the oldest signal and the most abused. Three rules make them useful. Write structured logs, one JSON object per line on stdout, as the twelve-factor module described, so that they can be filtered by field and not by regular expression. Use levels consistently. And include the trace ID and span ID in every line written while handling a request.

{
  "timestamp": "2026-09-17T14:02:11.482Z",
  "level": "error",
  "message": "stock reservation failed",
  "service": "inventory",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "order_id": "o-1842",
  "sku": "SKU-9917",
  "error": "lock wait timeout exceeded",
  "duration_ms": 812
}

With the trace ID present, your tooling can jump from a span straight to the log lines written during it, and from any log line to the full trace of the request it belongs to. That single field turns a log search across six services into one click. OpenTelemetry's logging instrumentation can inject these IDs into your existing logger's records automatically, so you usually do not have to pass them around by hand.

import json
import logging

from opentelemetry import trace


class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        ctx = trace.get_current_span().get_span_context()
        event = {
            "timestamp": self.formatTime(record),
            "level": record.levelname.lower(),
            "message": record.getMessage(),
            "logger": record.name,
        }
        if ctx.is_valid:
            event["trace_id"] = format(ctx.trace_id, "032x")
            event["span_id"] = format(ctx.span_id, "016x")
        return json.dumps(event)

Log volume is a real cost, as the FinOps module noted. Log an event once, at the place that handles it, not at every layer it passes through. Do not log inside tight loops. And do not use logs to count things: that is a metric, and it is thousands of times cheaper.

The Collector

Applications could export straight to a backend, but in practice you run the OpenTelemetry Collector in between. It is a standalone process that receives telemetry, processes it, and exports it to one or more destinations. It takes batching, retries, credentials and vendor-specific formats out of your applications, and gives operators one place to filter, redact and route. Its configuration is a set of components wired into pipelines, one pipeline per signal.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  batch:
    timeout: 5s
  attributes/redact:
    actions:
      - key: http.request.header.authorization
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/redact, batch]
      exporters: [otlp/tempo, debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

• A component does nothing until it is listed in a pipeline under service. Defining an exporter and forgetting to reference it is the most common configuration mistake.
• Processor order matters. Put memory_limiter first, so the Collector sheds load instead of crashing, and batch last, so that data is grouped after all filtering.
• The type/name form, as in otlp/tempo, lets you define several instances of one component type.
• OTLP uses port 4317 for gRPC and 4318 for HTTP.
• Two deployment patterns are common, often together. An agent runs next to the application, as a DaemonSet or sidecar, and collects locally. A gateway is a central, scaled pool that does the heavy processing and holds the backend credentials.

The core distribution contains only the most stable components. The contrib distribution adds hundreds more, including receivers for Prometheus scraping, host metrics, log files and Kubernetes events, so check which distribution a component ships in before you copy a configuration.

Sampling: you cannot keep everything

A busy service produces millions of traces a day, and the great majority are identical, uneventful successes. Storing them all is expensive and pointless. Sampling keeps a subset, and the strategy determines which.

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

With head sampling, use a parent-based sampler, as in the environment variables earlier. The first service makes the decision, records it in the traceparent flags, and every downstream service honours it. Without that, each service decides independently and you collect fragments of traces with no complete one among them.