Cost, latency & caching

Where the money and the time go

Pricing is per token, and quoted separately for input and output. Output tokens usually cost several times as much as input tokens, because they are generated one at a time, while the input is processed in parallel. Most providers also sell cached input at a steep discount, and asynchronous batch processing at a substantial one. Prices change often, so put them in configuration, and never in code or in your head.

cost per request  = input_tokens  x input_price
                  + cached_tokens x cached_price        (much cheaper than input_price)
                  + output_tokens x output_price        (usually the most expensive per token)

cost per task     = the sum over every model call that the task needed
                    (an agent that takes 15 turns resends the growing history 15 times)

latency           = time to first token  +  output_tokens / tokens per second
                    (queueing + reading the input)   (generation: usually the larger part)

from dataclasses import dataclass


@dataclass
class Prices:
    """Per million tokens. Load these from configuration; they change."""
    input: float
    cached_input: float
    output: float


def request_cost(prices: Prices, input_tokens: int, cached_tokens: int, output_tokens: int) -> float:
    uncached = input_tokens - cached_tokens
    return (uncached * prices.input
            + cached_tokens * prices.cached_input
            + output_tokens * prices.output) / 1_000_000


def monthly_cost(prices: Prices, requests_per_day: int, input_tokens: int,
                 cached_tokens: int, output_tokens: int) -> float:
    return 30 * requests_per_day * request_cost(prices, input_tokens, cached_tokens, output_tokens)

Before optimising anything, measure. Every API response reports its token usage. Log it for every call, tagged with the feature, the prompt version and the model, exactly as the FinOps module tagged cloud resources. You will nearly always find that a small number of features, or one bloated prompt, account for most of the spend, and that the expensive part is not the part you expected.

Free wins first

Some savings cost nothing in quality. Take every one of them before you consider a trade-off.

• Trim the input. Remove boilerplate, duplicated instructions, unused examples and tool definitions that are never called. Retrieve five relevant passages and not twenty. Every token in a system prompt is paid for on every single request.
• Keep tool results small, as the tool-use module said. In an agent, a bloated result is paid for again on every later turn.
• Bound the output. Ask for the length you need, and set a maximum. Output is the expensive side, and it is also the slow one. "Answer in two sentences" is both a cost optimisation and a latency optimisation.
• Do not ask for what you will throw away. If your code needs a label, do not request a label together with a paragraph of explanation, unless that explanation measurably improves the label.
• Stop wasteful loops. Cap the turns of an agent, detect repeated tool calls, and do not retry failures that cannot succeed.
• Cache the prompt prefix, covered next. For many applications it is the largest saving available.
• Use batch processing for anything that nobody is waiting for.
• Do not call the model at all where a regular expression, a database lookup or a rule gives the right answer. The cheapest call is the one you never make.

Prompt caching

Most applications send the same long opening on every request: the system prompt, the tool definitions, a reference document, the few-shot examples. Prompt caching lets the provider store its processed form of that prefix, and reuse it. Cached tokens are billed at a fraction of the normal input price, and they are faster too, since the work of reading them has already been done. Some providers cache automatically, and others need you to mark what should be cached. In either case, the same rule governs whether it works.

A cache is a prefix match. The provider can reuse the cached computation only for a request whose beginning is identical, byte for byte, to a previous one, up to the point that was cached. A single changed character early in the prompt invalidates everything that follows it. The whole technique follows from that: stable content first, variable content last.

GOOD: the stable prefix is reused on every request
  [tool definitions] [system prompt] [reference document] [few-shot examples] | [the user's question]
   ^------------------------- identical every time -------------------------^   ^--- varies ---^

BAD: a changing value near the top destroys the cache for everything after it
  [system prompt: "Current time: 14:02:11 ..."] [tool definitions] [reference document] ...
                   ^ changes every second, so nothing after it can ever be reused

• Verify it. The usage data in each response reports how many input tokens were read from the cache. If that number stays at zero across repeated requests, something in your prefix is changing. Compare two consecutive requests byte for byte to find it.
• Caches expire after a short period without use, typically minutes. They pay off for steady traffic, for multi-turn conversations, and for agent loops, where every turn sends the whole earlier history again.
• Writing to the cache may cost slightly more than ordinary input. The saving comes from the reads, so caching a prefix that is used once gains nothing.
• Caches are specific to a model. Switching models part of the way through a conversation throws away the accumulated cache.
• Caching makes a long, detailed system prompt affordable. Without it, there is a constant pressure to cut useful context.

Choosing and routing models

Providers offer families of models, from small, fast and cheap to large, slower and much more capable, with prices that can differ by an order of magnitude or more. No model is right for everything. Match the model to the task, and let your evaluation set make the decision, not your intuition.

Routing sends each request to the cheapest model that can handle it. A small model, or a simple rule, classifies the request, and only the hard ones reach the expensive model. A cascade tries the cheap model first, and escalates when a check fails: low confidence, a failed validation, or an "unsure" answer.

def answer(question: str, context: str) -> dict:
    """A cascade: cheap model first, escalating only when a check fails.
    call_llm is a stand-in, and the model names come from configuration."""
    draft = call_llm(build_prompt(question, context), model=CONFIG["small_model"], max_tokens=400)

    if passes_checks(draft, context):                 # cites the context, valid format, not "unsure"
        return {"answer": draft, "model": "small"}

    final = call_llm(build_prompt(question, context), model=CONFIG["large_model"], max_tokens=400)
    return {"answer": final, "model": "large"}

• A cascade helps only if most requests pass at the cheap tier. If half of them escalate, you pay for both calls half of the time, and add latency as well.
• Many current models have a reasoning effort or thinking setting. Before building a multi-model cascade, test the simpler alternative: one capable model at a lower effort setting. It is often competitive, and it keeps one cache and one set of prompts.
• In multi-agent designs, run the orchestrator on a capable model, and the narrow, reading-heavy sub-tasks on a cheaper one.
• Re-run the evals whenever you change a model. "It seemed fine" is not a measurement.

Latency: streaming, parallelism and caching responses

• Streaming returns tokens as they are generated. The total time is unchanged, and the perceived latency falls from many seconds to a fraction of one, because the user starts reading almost at once. Use it for anything that a person watches. It also avoids HTTP timeouts on long generations. Streaming does not help when code has to have the whole response before it can act.
• Shorter output is faster output. Generation time is roughly proportional to output length, so tightening the requested length is the most direct lever on latency.
• Parallelise independent calls. Three classifications that do not depend on one another should run concurrently, not one after another.
• Reduce round trips. Each turn of an agent is a full request. Tools that return what is needed in one call, and models that call several tools in parallel, reduce the number of turns.
• Pick the model for the latency you need. Smaller models are much faster. An autocomplete needs a fast model, and a nightly report does not.
• Warm the cache before a known burst of traffic, where your provider supports it, so that the first real user does not pay for the cache write.

A response cache avoids the call altogether. If the same question recurs, keep the answer.

import hashlib
import json


def cache_key(model: str, messages: list[dict], params: dict) -> str:
    """An exact-match key. Every input that affects the output must be part of it."""
    payload = json.dumps({"model": model, "messages": messages, "params": params},
                         sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()


def cached_call(store, model: str, messages: list[dict], **params) -> str:
    key = cache_key(model, messages, params)
    hit = store.get(key)
    if hit is not None:
        return hit
    result = call_llm(messages, model=model, **params)
    store.set(key, result, ttl_seconds=3600)
    return result

• Exact-match caching suits deterministic tasks at low temperature: classification, extraction, and frequently asked questions that are normalised before hashing.
• Include the prompt version and the model in the key, or a change to the prompt will go on serving stale answers.
• Semantic caching matches questions that are similar, using embeddings. It catches more repeats, and it carries a real risk: "how do I enable X?" and "how do I disable X?" are close together in embedding space. Use a strict similarity threshold, and do not use it where a wrong answer is costly.
• Never share cached responses between users if the answer depends on private data, or on the user's permissions.

Batch work, budgets and rate limits

A great deal of LLM work is not interactive: classifying last month's tickets, generating embeddings for a corpus, running evals, writing summaries overnight. Batch APIs accept a file of requests, process them asynchronously, typically within a day, and charge substantially less than the interactive price. Give every request in a batch an identifier of your own, because results come back in arbitrary order, and must be matched by that identifier and never by position. Batches also do not count against your interactive rate limits in the same way, so a large job does not starve live traffic.

Providers enforce rate limits in requests and in tokens per minute. When you exceed them you receive an HTTP 429, usually with a Retry-After header. Handle it with the retry discipline of the failure-modes module: exponential backoff with jitter, respect for Retry-After, a bounded number of attempts, and a queue that smooths out bursts in place of a stampede.

• Set budgets at every level: a maximum output per call, a token and turn budget per task, a daily limit per user, and a monthly limit per feature, with alerts on the forecast as well as on the total.
• Protect yourself against abuse. A public endpoint backed by an LLM is a way for strangers to spend your money. Require authentication, apply rate limits per user, and cap the length of input.
• Degrade gracefully when a budget or a limit is reached: fall back to a smaller model, to a cached answer, or to an honest "try again shortly", in preference to an error.
• Track the unit economics: cost per conversation, per resolved ticket, per active user. Compare the figure with the value of that outcome, which is the only way to tell whether the feature pays for itself.
• Watch the trend. Prompts grow, histories lengthen, and someone adds five tools. Put cost per task on a dashboard, with an alert when it drifts.