Evaluating LLM output

Why vibes do not scale

Checking by eye fails for predictable reasons. You try the handful of inputs that you thought of, which are the ones the system already handles. You remember the impressive answers and forget the mediocre ones. You cannot tell whether version seven is better than version six, because you did not run the same inputs through both. And when the provider updates the model, you have no means of knowing what changed for you.

An eval has three parts: a dataset of inputs, a task that runs your system on each of them, and a grader that scores each output. Run it, and you get a number that you can compare across versions. That number turns arguments into experiments: should we use the cheaper model, a shorter prompt, more retrieved passages? Run the eval and look.

Because the outputs vary, a single case tells you little, and even a whole run carries noise. With 50 cases, each case is worth two percentage points, so a move from 84% to 86% is one case, and could easily be chance. Use more cases where you can, run the noisy evals several times, and be suspicious of small differences.

Building the golden dataset

The dataset is the asset. Graders and prompts come and go, and a well-chosen set of cases keeps its value for years. It must resemble reality, which means most of it should come from reality.

• Real traffic. Sample actual user inputs from logs, tickets and search queries, with personal data removed. They are messier, shorter and stranger than anything you would invent.
• Every failure you have seen. Each bug report and each bad answer becomes a permanent case. This is the rule that stops regressions from coming back.
• Edge cases. Empty input, very long input, another language, several questions in one, a question the system should decline, a question whose answer is not in your data.
• Adversarial cases. Attempts to override the instructions, extract the system prompt, or provoke harmful output.
• Coverage. Include every category and every kind of user in realistic proportions, so that an improvement for the common case does not conceal a collapse in a rare, important one.

{
  "id": "rag-0042",
  "input": "how do I roll back the checkout service?",
  "expected": {
    "must_mention": ["kubectl rollout undo", "deployment/checkout"],
    "must_not_mention": ["kubectl delete"],
    "relevant_chunks": ["kb-12#3"],
    "reference_answer": "Run kubectl rollout undo deployment/checkout -n shop, then confirm with kubectl rollout status."
  },
  "tags": ["runbook", "kubernetes", "from-incident-2304"],
  "added": "2026-09-02"
}

Start small. Twenty to fifty good cases are enough to catch the large problems and to make the habit stick, and a small eval that you run is worth more than a large one that you never finish. Write the expected results down yourself, or with a domain expert. Doing so forces you to define what "good" means, which is often the most valuable result of the whole exercise. You can use a model to help draft variations, provided that a person reviews them, since purely synthetic cases tend to be tidier and easier than real ones.

Choosing a grader

Use the cheapest, most deterministic grader that can judge the property you care about. Code-based checks are fast, free and perfectly consistent, so use them wherever they apply, and keep model-based judging for what truly needs judgement.

import json


def grade_structured(output: str, expected: dict) -> dict:
    """Code-based grading: deterministic, free, and run on every case."""
    checks = {}
    try:
        data = json.loads(output)
        checks["valid_json"] = True
    except json.JSONDecodeError:
        return {"valid_json": False, "passed": False}

    checks["category_correct"] = data.get("category") == expected["category"]
    checks["urgency_in_range"] = isinstance(data.get("urgency"), int) and 1 <= data["urgency"] <= 5
    checks["reason_present"] = bool(data.get("reason", "").strip())
    checks["passed"] = all(checks.values())
    return checks


def grade_text(output: str, expected: dict) -> dict:
    text = output.lower()
    missing = [s for s in expected.get("must_mention", []) if s.lower() not in text]
    forbidden = [s for s in expected.get("must_not_mention", []) if s.lower() in text]
    return {"missing": missing, "forbidden": forbidden, "passed": not missing and not forbidden}

Break quality down into separate, specific properties, and grade each one by itself: is it valid JSON, is the category right, is every claim supported by the source, does it cite a real passage, is it under 100 words? A single "quality score from 1 to 10" conceals which of those went wrong, and is much noisier than several narrow checks.

LLM-as-judge, done properly

Using a model to grade another model's output sounds circular, and it works, because judging is easier than generating: checking whether a summary is supported by a document is a simpler task than writing the summary. But a judge is a model too, with failure modes of its own, so it has to be built with care and checked against people.

import json

JUDGE_SYSTEM = """You are grading the answer given by a documentation assistant.
You will receive the source passages, the question, and the answer.

Grade ONE property only: faithfulness.
An answer is faithful if every factual claim and every command in it is directly
supported by the passages. Information that is true, but is not in the passages,
counts as unfaithful.

First write your reasoning: list each claim in the answer, and say whether a passage
supports it, quoting the supporting text. Then give the verdict.

Respond with a JSON object: {"reasoning": "...", "unsupported_claims": [...], "verdict": "pass" or "fail"}"""


def judge_faithfulness(passages: str, question: str, answer: str) -> dict:
    """call_llm is a stand-in: wire it to whichever provider you use."""
    user = (f"<passages>\n{passages}\n</passages>\n\n"
            f"<question>\n{question}\n</question>\n\n<answer>\n{answer}\n</answer>")
    raw = call_llm(
        [{"role": "system", "content": JUDGE_SYSTEM}, {"role": "user", "content": user}],
        temperature=0, max_tokens=800,
    )
    return json.loads(raw)

• One property per judge. Ask a precise question with a clear definition. "Is this good?" gives noise.
• Binary or a small scale. Pass or fail, or three to five levels, each one described. Models do not use a 1 to 100 scale consistently.
• Reasoning before the verdict. Having the judge write out its analysis first improves accuracy, and gives you an explanation to read when you disagree with it.
• Give it what it needs. A faithfulness judge needs the source, and a correctness judge needs the reference answer. A judge that has neither is merely checking whether the answer sounds good.
• Use a capable model for judging, and low temperature, so that the grades are consistent.

Most important of all, validate the judge against human labels. Have a person grade 50 to 100 outputs, run the judge on the same ones, and measure how often the two agree. Read the disagreements: sometimes the judge is wrong, and sometimes it has caught something the human missed. If agreement is poor, improve the rubric before you trust any number that the judge produces. Repeat the check from time to time, and whenever you change the judge's model or its prompt.

A harness, and a gate in CI

import json
import sys
from pathlib import Path

THRESHOLD = 0.90            # minimum pass rate
MAX_DROP = 0.03             # largest allowed fall against the stored baseline


def run_eval(cases_path: str, system_under_test, grade) -> dict:
    cases = [json.loads(line) for line in Path(cases_path).read_text().splitlines() if line.strip()]
    results = []
    for case in cases:
        output = system_under_test(case["input"])
        verdict = grade(output, case["expected"])
        results.append({"id": case["id"], "tags": case.get("tags", []),
                        "output": output, **verdict})
    passed = sum(r["passed"] for r in results)
    return {"pass_rate": passed / len(results), "n": len(results), "results": results}


if __name__ == "__main__":
    report = run_eval("cases.jsonl", system_under_test=my_app, grade=grade_text)
    Path("eval_report.json").write_text(json.dumps(report, indent=2))

    baseline = json.loads(Path("eval_baseline.json").read_text())["pass_rate"]
    print(f"pass rate {report['pass_rate']:.1%} (baseline {baseline:.1%}, n={report['n']})")
    for r in report["results"]:
        if not r["passed"]:
            print(f"  FAIL {r['id']} {r['tags']}")

    if report["pass_rate"] < THRESHOLD or report["pass_rate"] < baseline - MAX_DROP:
        sys.exit(1)             # non-zero exit fails the pipeline

This is the pipeline idea from the DevOps track, applied to AI. A change to a prompt, a model version, a retrieval parameter or a tool definition is a change to production, and it should pass through a gate before it ships. A few practical points follow.

• Tier the suites. Run a small, fast, code-graded suite on every pull request, and the full suite with LLM judges every night and before each release, because judges cost money and time.
• Report by tag, not only in total. An overall pass rate of 92% can conceal a collapse from 95% to 60% on Spanish inputs.
• Store the full outputs, not only the scores, so that you can compare two versions side by side and read what changed.
• Record the versions: the prompt, the model, the retrieval configuration and the dataset. A score without them cannot be reproduced.
• Cache the unchanged parts. If only the generation prompt changed, there is no need to run retrieval again.
• Track cost and latency alongside quality. A change that gains one point of quality for three times the cost is rarely a win.
• Pin the model version and re-run the suite before adopting a new one. This is how you decide about an upgrade using evidence.

Evals never finish

An offline eval measures the cases that you thought of. Production supplies the ones you did not. The two form a loop.

1. Log inputs, outputs, the context retrieved, versions, latency and cost, with sensitive data removed.
2. Collect signals. Explicit ones, such as thumbs up or down, and implicit ones, such as the user rephrasing the question, abandoning the session, copying the answer, or escalating to a person.
3. Sample and review. Read a random sample of real conversations every week. There is no substitute for it, and it is where you discover kinds of failure that no metric was watching for.
4. Run judges online, on a sample of live traffic, to follow faithfulness or tone over time, and alert on a fall.
5. Promote failures into the dataset. Every confirmed bad output becomes a permanent eval case, with a tag recording where it came from.
6. Fix, re-run and ship, and start the loop again.

Beware of optimising the number instead of the product. If the judge rewards long answers, your prompts will drift towards padding. If the dataset grows stale, scores rise while users become less happy. Refresh the cases, re-validate the judges against people, and keep reading real outputs yourself. The LLM observability module in the final stage continues this loop in production.