The golden signals of the SRE track will tell you that your LLM feature is up, fast and free of HTTP errors. They will not tell you that it has begun giving wrong answers, because a confident hallucination returns status 200 in 800 milliseconds, just as a correct answer does. With LLM systems, the failure that matters is the quality of the output, and ordinary monitoring cannot see it. This module extends the observability of the SRE track to cover it: what to record for every request, how to trace an agent's chain of calls, how to measure quality in production, and how to turn every bad answer into a permanent test.
- Explain why conventional monitoring misses the main failure mode of LLM systems
- Decide what to capture for each request, and handle the privacy implications of logging prompts
- Trace multi-step LLM applications and agents with OpenTelemetry spans
- Measure quality in production with explicit feedback, implicit signals and sampled LLM judges
- Detect drift, and close the loop from production failures back into the eval dataset
Status 200, and wrong
Conventional observability rests on an assumption: a request that fails announces itself, through an error code, an exception or a timeout. LLM applications break that assumption. Look at what can go wrong without any of those occurring.
- The model hallucinates a command, a policy or a figure.
- Retrieval returns the wrong documents, so the answer is fluent and grounded in irrelevant text.
- The provider updates the model, and the tone or the format shifts, breaking a parser downstream for one request in fifty.
- An edit to the prompt, meant to fix one case, degrades another category that nobody checked.
- An agent loops through fifteen tool calls where three would have done, and succeeds at five times the cost.
- Users discover a kind of question that the system handles badly, and quietly stop using it.
Each of these returns 200. None of them moves the error rate. Latency may even improve. You therefore need a second layer of signals on top of the first, not in place of it.
| Layer | Question | Signals |
|---|---|---|
| System health | Is it up, fast and affordable? | Request rate, errors, latency, time to first token, tokens, cost, rate-limit responses |
| Behaviour | What did it actually do? | Traces of each step: retrieval, model calls, tool calls, with their inputs and outputs |
| Quality | Was the output good? | User feedback, implicit signals, sampled judge scores, rates of guard-rail triggers |
| Drift | Is it changing over time? | Distributions of inputs, of retrieval scores, and of output length and format, against a baseline |
The first layer is the SRE track, applied without alteration, and it still matters: a provider outage or a rate limit is an ordinary incident. The rest of this module is about the other three layers.
What to capture for every request
The unit of LLM observability is a structured record of each model call, rich enough to reproduce and explain it later. Metrics are derived from these records. The reverse is impossible: you cannot recover what the model said from a latency histogram.
| Group | Fields |
|---|---|
| Identity | Trace ID, span ID, timestamp, feature name, environment, a pseudonymous user or session ID |
| Versions | Provider, the exact model identifier, the prompt version, the retrieval configuration version, the application version |
| Request | The messages or the rendered prompt, the tool definitions offered, the sampling parameters |
| Response | The output text, the tool calls requested, the stop reason |
| Usage | Input tokens, cached tokens, output tokens, the computed cost |
| Timing | Total latency, time to first token, time spent queued, retries |
| Context | The IDs and scores of retrieved documents, and the results of tools, or references to them |
| Outcome | The result of validation, guard-rail flags, user feedback, the judge's score, when they arrive |
The stop reason deserves its own metric. A rise in responses that ended because they reached the token limit means truncated answers. A rise in refusals means that a change to a prompt, or to a model, has made the system over-cautious, or that someone is probing it.
Prompts and responses are the most sensitive data that you will ever log. Users paste in passwords, customer records and medical details, and retrieved documents may be confidential. Decide this deliberately: redact known patterns such as keys, emails and card numbers before storing; restrict who can read the raw text; set a short retention period for it, and keep the metadata for longer; honour requests for deletion; and check what your contracts and your privacy policy actually allow. In regulated settings, store the metadata, and only a sample, or nothing, of the content.
Logging everything in full becomes expensive at scale, and the usual answer is a tiered scheme. Keep the metadata for every request, since it is small and drives all the metrics. Keep the full content for a random sample, together with every request that failed validation, triggered a guard rail, received negative feedback, or was unusually slow or costly. That is the same reasoning as tail sampling in the OpenTelemetry module: keep what is interesting.
Tracing chains and agents
A single user request to a RAG assistant, or to an agent, fans out into many operations: rewrite the query, embed it, search, rerank, call the model, call a tool, call the model again. When the answer is wrong, the question is which step failed, and a flat log cannot answer it. Distributed tracing, from the OpenTelemetry module, fits exactly: one trace per user request, and one span per step, nested.
trace 7d1c... POST /assistant/ask total 6,840 ms cost 0.0214
assistant.ask [=================================================] 6840 ms
rag.rewrite_query (llm, small) [==] 310 ms in 412 out 28
rag.embed_query [=] 45 ms
rag.vector_search k=50 [=] 38 ms top score 0.71
rag.rerank 50 -> 5 [===] 260 ms
llm.generate turn 1 [===========] 1900 ms in 3,180 out 96 stop=tool_use
tool.get_error_rate service=checkout [==] 210 ms
tool.get_recent_deploys [==] 190 ms
llm.generate turn 2 [===================] 3700 ms in 3,610 out 402 stop=endOne look shows where the time and the money went, what was retrieved, which tools ran and with which arguments, and how many turns the agent took. When an answer is wrong, you open its trace and read the retrieved passages and the tool results that the model actually saw. In most cases the fault is visible at once: the right document was never retrieved, or a tool returned an error that the model papered over.
import time
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
tracer = trace.get_tracer(__name__)
def traced_llm_call(messages: list[dict], *, model: str, prompt_version: str, **params) -> dict:
"""call_llm is a stand-in returning {'text', 'stop_reason', 'input_tokens',
'cached_tokens', 'output_tokens'}. Attribute names here are illustrative."""
with tracer.start_as_current_span("llm.generate") as span:
span.set_attribute("llm.model", model)
span.set_attribute("llm.prompt_version", prompt_version)
span.set_attribute("llm.temperature", params.get("temperature", 1.0))
started = time.perf_counter()
try:
reply = call_llm(messages, model=model, **params)
except Exception as exc:
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR, type(exc).__name__))
raise
span.set_attribute("llm.latency_ms", round((time.perf_counter() - started) * 1000))
span.set_attribute("llm.input_tokens", reply["input_tokens"])
span.set_attribute("llm.cached_tokens", reply["cached_tokens"])
span.set_attribute("llm.output_tokens", reply["output_tokens"])
span.set_attribute("llm.stop_reason", reply["stop_reason"])
return reply- OpenTelemetry defines semantic conventions for generative AI, with attribute names beginning
gen_ai., for the model, the token usage and the operations. They are still evolving, so check the current specification, and prefer an instrumentation library that tracks it to hard-coded names of your own. - Put large payloads, meaning the full prompt and the response, in span events, or in a separate store that is referred to by ID. They are not suitable as attributes, which are meant to be small and searchable.
- Propagate the trace ID into your logs and your feedback events, so that a thumbs-down can be joined to the exact trace that produced it.
- Several open-source and commercial platforms specialise in tracing LLM applications. The model above is common to all of them. Choose on the basis of data residency, cost and how well each integrates with what you already run.
Measuring quality in production
Offline evals measure the cases that you thought of. Production measures reality. There are three sources of signals about quality, and you want all three, because each is weak where another is strong.
| Source | Examples | Strength | Weakness |
|---|---|---|---|
| Explicit feedback | Thumbs up or down, a rating, a correction, "report a problem" | Direct and unambiguous | A tiny share of users respond, and mostly the unhappy ones |
| Implicit signals | The user rephrases and asks again; abandons the session; copies the answer; accepts or rejects a suggestion; escalates to a person | Available for every interaction | Indirect; needs interpretation for each product |
| Automated judges | An LLM judge that scores faithfulness, relevance or tone on a sample of live traffic | Scalable, consistent, and chosen by you | Costs money; must be validated against people |
The implicit signals are underrated. For a coding assistant, the rate at which suggestions are accepted is the best quality metric that exists. For a support assistant, it is the rate at which conversations end without being escalated to a person. For a search or RAG product, it is whether the user asked again in different words within a minute. Find the behaviour in your own product that reveals satisfaction, and instrument it.
import random
JUDGE_SAMPLE_RATE = 0.05 # judge 5% of normal traffic...
def should_judge(record: dict) -> bool:
"""...and everything that looks suspicious."""
if record["feedback"] == "negative" or record["validation_failed"]:
return True
if record["retrieval_top_score"] < 0.35: # weak retrieval: a likely hallucination
return True
return random.random() < JUDGE_SAMPLE_RATE
def judge_online(record: dict) -> None:
"""Runs asynchronously, off the request path. judge_faithfulness is from the evals module."""
verdict = judge_faithfulness(record["passages"], record["question"], record["answer"])
emit_metric("llm_judge_faithful", 1 if verdict["verdict"] == "pass" else 0,
labels={"feature": record["feature"], "prompt_version": record["prompt_version"],
"model": record["model"]})
store_verdict(record["trace_id"], verdict)- Run the judges asynchronously, never in the request path. The user does not wait for the grading.
- Label every quality metric with the prompt version and the model version. The whole point is to see that version 12 is worse than version 11.
- Treat quality as an SLI, in the SRE sense: "at least 95% of sampled answers are judged faithful, over 7 days". Give it an error budget and an alert on its burn rate.
- Guard rails are signals too. Count how often the output validation fails, how often a retry was needed, and how often the checks on input or output fired. A rising rate is an early warning.
- Look at the numbers by segment: language, category of user, topic, length of input. An average conceals the group for whom it is broken.
Drift
An LLM application can degrade with no change to your code, because four things around it move.
| What drifts | Example | How to detect it |
|---|---|---|
| Inputs | A new product launches, and users ask questions that the documentation does not cover. A new market brings a new language. | Embed a sample of the queries and watch the clusters; track the language, the length and the topic mix; watch the rate of "not found" answers |
| Retrieval | The corpus grows stale, or a re-index changes the chunking, and the top scores fall | Track the distribution of top-k similarity scores, and the share of queries that have no passage above a threshold |
| The model | The provider updates a model behind an alias, or you upgrade on purpose | Pin versions; record the model identifier on every call; run the eval suite on a schedule, as a canary |
| Outputs | Answers grow longer, the format changes, refusals increase | Track output tokens, the stop reasons, the rate of validation failures, and the rate of refusals |
The cheapest and most effective detector of drift is a scheduled run of your offline eval suite against the production configuration, every night. If the score moves and you changed nothing, something outside your control has moved. It is the LLM equivalent of a synthetic probe.
# share of answers that ended by hitting the token limit (truncated), per prompt version
sum by (prompt_version) (rate(llm_responses_total{stop_reason="max_tokens"}[1h]))
/
sum by (prompt_version) (rate(llm_responses_total[1h]))
# sampled faithfulness, as an SLI, over 7 days
sum(rate(llm_judge_faithful_sum[7d])) / sum(rate(llm_judge_faithful_count[7d]))
# cost per request, by feature: catches a prompt that has quietly grown
sum by (feature) (rate(llm_cost_total[1h])) / sum by (feature) (rate(llm_requests_total[1h]))
# median output length has risen by more than 30% against last week
histogram_quantile(0.5, sum by (le) (rate(llm_output_tokens_bucket[1d])))
> 1.3 * histogram_quantile(0.5, sum by (le) (rate(llm_output_tokens_bucket[1d] offset 7d)))Closing the loop
Observability earns its cost when it changes what you do next. The loop that turns production experience into a better system has six steps, and it should become a weekly routine.
- Detect. A thumbs-down, a failed judge, a guard-rail trigger, a drift alert, or a complaint from a person.
- Inspect the trace. What did the user ask? What was retrieved? What did the tools return? What did the model see, and what did it say?
- Classify the failure: retrieval missed; the prompt was ambiguous; a limitation of the model; a bug in a tool; bad source data; outside the system's scope; an attempt at abuse.
- Add it to the eval dataset, as a permanent case, tagged with its failure class and its origin.
- Fix the right layer, which the classification tells you, and confirm the fix against the whole eval suite, not only against this case.
- Ship it through the gated rollout, and watch the same signal that first alerted you.
The classification step repays the effort. After a month, the tally tells you where to invest. If most of the failures are "retrieval missed", then better prompts will not help, and work on chunking or on hybrid search will. If most are "outside the scope", the product needs to say more clearly what it is for.
Set aside time to read real conversations, a random sample, every week. Dashboards show the failures that you already knew to look for. Reading shows the ones that you did not: the question that everyone asks, and that you never anticipated; the answer that is technically correct and useless; the user who worked around a limitation in a way that reveals what they truly wanted. No metric replaces it.
Build the dashboard in the layout of the Grafana module. The top row answers "is it healthy, and is it good?": the request rate, the error rate, p95 latency, the cost per request, the rate of positive feedback, and the sampled judge score, each of them split by prompt version. Below that come the breakdowns by feature and by segment, and then the drift panels. Annotate every change of prompt and of model as a deploy, because that is what it is.