Serving models in production

Hosted API or self-hosted?

The first decision is whether to run the model at all. Calling a provider's API and operating your own inference fleet are very different commitments, and for most teams, most of the time, the API is the right answer.

Self-hosting is justified by specific needs: data that may not leave your environment for legal or contractual reasons, very high and steady volume where your own GPUs work out cheaper, strict latency requirements, operation without internet access, or a small fine-tuned model that does one narrow job well. It is not justified by a vague wish for control. Be honest about the full cost, which includes idle GPUs at night and the engineers who keep the stack running.

Classical models are another matter. A gradient-boosted tree, or a small neural network that scores a fraud risk, is cheap to run on CPUs, has no hosted equivalent, and is nearly always served by your own team. The serving principles in the rest of this module apply to both kinds, and the GPU material applies to large models.

Why inference is not web serving

A typical web service is limited by CPU and I/O, starts in seconds, and handles each request independently. Large-model inference breaks all three of those assumptions.

• GPU memory is the constraint. The model's weights have to fit in GPU memory, and they are large: roughly two bytes per parameter at 16-bit precision, so a model with 7 billion parameters needs about 14 GB for its weights alone. On top of that comes the KV cache, the memory that holds the attention state of every token of every request being processed. It grows with the length of the context and with the number of concurrent requests, and it is what usually limits how many users one GPU can serve.
• Two phases with different characters. Prefill processes the whole prompt in parallel, and is limited by computation. It determines the time to first token. Decode then generates one token at a time, and is limited by memory bandwidth. It determines the tokens per second that a user sees.
• Batching is essential. A GPU processing one request at a time is mostly idle. Serving engines group requests, so that a single pass through the model advances many of them together. Continuous batching goes further: requests join and leave the batch at every step, without waiting for the others to finish, which raises throughput enormously.
• Throughput trades against latency. Bigger batches serve more users per GPU, and make each user's tokens arrive a little more slowly. You choose a point on that curve, guided by your latency SLO.
• Start-up is slow. Pulling an image of several gigabytes, loading tens of gigabytes of weights into GPU memory and warming up can take minutes. That changes how you scale, how you roll out, and how much headroom you keep.

GPU memory needed  ~  weights + KV cache + activations and overhead

weights            ~  parameters x bytes per parameter
                      7B params x 2 bytes (16-bit)  = about 14 GB
                      7B params x 1 byte  (8-bit)   = about  7 GB
                      7B params x 0.5 byte (4-bit)  = about  3.5 GB

KV cache           grows with: concurrent requests x tokens per request x layers x hidden size
                   -> long contexts and high concurrency are what exhaust memory

Quantisation stores the weights at lower precision, in 8 or 4 bits instead of 16. It cuts the memory and often raises the speed, at some cost in quality, which you must measure on your own evals instead of assuming. It is frequently what allows a model to fit on a smaller, cheaper GPU at all. Models too large for one GPU are split across several with tensor or pipeline parallelism, which the serving engine handles, at a cost in complexity and in communication between GPUs.

You rarely write this machinery yourself. Open-source inference servers such as vLLM, Hugging Face Text Generation Inference, and NVIDIA Triton implement continuous batching, efficient management of the KV cache, quantisation and streaming, and several of them expose an HTTP API compatible with the common chat-completion format, so that the client code barely changes.

Packaging a model as a service

Whatever the model, the service around it follows the same rules. Load the model once, at start-up, never on each request. Validate the inputs. Expose health endpoints that tell the truth about whether the model is ready. Report the model version with every response. This example serves a classical scikit-learn pipeline of the kind built in the core ML module.

import os
import time
from contextlib import asynccontextmanager

import joblib
import pandas as pd
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

MODEL_PATH = os.environ.get("MODEL_PATH", "/models/churn-pipeline.joblib")
MODEL_VERSION = os.environ.get("MODEL_VERSION", "unknown")
state = {"model": None}


@asynccontextmanager
async def lifespan(app: FastAPI):
    state["model"] = joblib.load(MODEL_PATH)      # load ONCE, before serving traffic
    yield
    state["model"] = None


app = FastAPI(lifespan=lifespan)


class Features(BaseModel):
    tenure_months: int = Field(ge=0, le=600)
    monthly_spend: float = Field(ge=0)
    support_tickets_90d: int = Field(ge=0)
    plan: str


@app.get("/healthz")
def liveness():
    return {"status": "alive"}                    # the process is up; nothing more


@app.get("/ready")
def readiness():
    if state["model"] is None:
        raise HTTPException(status_code=503, detail="model not loaded")
    return {"status": "ready", "model_version": MODEL_VERSION}


@app.post("/predict")
def predict(features: Features):
    started = time.perf_counter()
    frame = pd.DataFrame([features.model_dump()])
    probability = float(state["model"].predict_proba(frame)[0, 1])
    return {"churn_probability": probability, "model_version": MODEL_VERSION,
            "latency_ms": round((time.perf_counter() - started) * 1000, 1)}

• Ship the whole pipeline, with its preprocessing, as a single artifact, so that serving transforms the inputs exactly as training did. A mismatch between the two is training-serving skew, a classic and silent cause of bad predictions.
• Validate the inputs with a schema. Values that are out of range or missing should be rejected, or at least counted, since they are often the first sign of a fault upstream.
• Return the model version with every prediction, and log it. When the behaviour changes, you need to know which model produced which answer.
• Instrument it with the golden signals from the SRE track, together with model-specific ones: the distribution of predictions, the distributions of the inputs, and, for LLMs, the time to first token, tokens per second, queue time and tokens per request.
• Keep the weights out of the image when they are large. Pull them at start-up from object storage or a model registry, onto a cached volume, so that images stay small and one image can serve many versions of a model.
• Treat model files as untrusted code. Pickle-based formats, including joblib files, can execute arbitrary code when they are loaded. Load only artifacts that your own pipeline produced, and prefer safe formats such as safetensors for neural network weights.

GPUs on Kubernetes

Kubernetes schedules GPUs as an extended resource, which the vendor's device plugin advertises. A pod requests whole GPUs under limits, and the scheduler places it on a node that has one free. GPUs are not overcommitted: a GPU assigned to one pod is unavailable to every other, unless you deliberately configure a sharing mechanism.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-server
  namespace: ml
spec:
  replicas: 2
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0            # never reduce serving capacity during a rollout
  selector:
    matchLabels:
      app: llm-server
  template:
    metadata:
      labels:
        app: llm-server
    spec:
      terminationGracePeriodSeconds: 120      # let in-flight generations finish
      nodeSelector:
        accelerator: nvidia-gpu
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: server
          image: registry.example.com/ml/llm-server:1.8.0
          env:
            - name: MODEL_VERSION
              value: "support-assistant-2026-09-01"
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "4"
              memory: 32Gi
            limits:
              nvidia.com/gpu: 1
              memory: 32Gi
          startupProbe:                        # loading weights takes minutes
            httpGet:
              path: /ready
              port: 8000
            periodSeconds: 10
            failureThreshold: 60               # allow up to 10 minutes to start
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8000
            periodSeconds: 20
          volumeMounts:
            - name: model-cache
              mountPath: /models
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: model-cache

• The startup probe is essential. Without one, the liveness probe kills the container while it is still loading its weights, and the pod never becomes ready. This is the most common failure when a model is first deployed.
• Readiness must mean that the model is loaded, not merely that the process has started, or traffic will arrive at a pod that cannot serve it.
• Taints and node selectors keep GPU nodes for GPU workloads, so that expensive hardware is not filled with ordinary pods.
• maxUnavailable: 0 matters more than usual, because a replacement takes minutes to become ready, and removing a pod early leaves a long gap in capacity.
• A generous termination grace period lets streaming responses complete before the pod is removed.
• GPU nodes are expensive, and they are the natural target of the FinOps module: tag them, watch their utilisation, and scale them down when they are idle.

Scaling on the right signal

The autoscaling habit of targeting CPU utilisation fails for inference. The CPU of a GPU server tells you nothing, and GPU utilisation is misleading as well, because a GPU can report itself as fully busy at very different levels of real load. Scale on what reflects demand against capacity: queue depth, the number of requests in flight, or the time that requests spend waiting.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-server
  namespace: ml
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-server
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Pods
      pods:
        metric:
          name: inference_requests_in_flight     # exposed through a custom metrics adapter
        target:
          type: AverageValue
          averageValue: "12"                     # tuned from a load test against the latency SLO
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
    scaleDown:
      stabilizationWindowSeconds: 600            # scale in slowly: a replacement costs minutes

• Find the target with a load test, as the capacity module described. Raise the concurrency until the time to first token, or the tokens per second, breaches your SLO. The target is somewhat below that point.
• Cold starts dominate. If a new replica takes five minutes to serve, the autoscaler cannot rescue you from a spike that arrives in thirty seconds. Keep more headroom than you would for a web service, pre-scale before known peaks, and cache the weights on the node or on a fast volume.
• Bound the queue, and shed load. When you are saturated, reject quickly with 429 or 503, in preference to letting the queue grow without limit. A request that waits two minutes helps nobody. This is the backpressure of the failure-modes module.
• Limit the work per request: a maximum input length, a maximum number of output tokens, and a timeout. One pathological request should not be able to occupy a GPU for minutes.
• Scale to zero only for workloads that can tolerate a cold start of several minutes, such as internal batch tools. For interactive use, keep a warm minimum.
• Separate traffic by kind. Interactive requests and bulk batch jobs have opposite needs. Give each its own pool, or a priority queue, so that a batch job cannot starve live users.

Rolling out model changes

A new model is a release that changes behaviour, and it is more dangerous than a code release because the failure is quiet. Nothing crashes, and no error rate rises. The answers simply become worse for some kinds of input. The same applies to a changed prompt, a new version of a provider's model, a different quantisation, or an upgraded serving engine. Apply the progressive delivery of the SRE track, with checks adapted to quality.

1. Pin versions. Refer to an exact version of the model, or of the provider's snapshot, and never to a moving alias such as "latest". An unannounced change in a model is an unannounced production deploy.
2. Pass the offline gate. The candidate must meet the thresholds of your eval suite, on quality, safety, latency and cost, before it goes near any traffic.
3. Shadow it. Send a copy of real production requests to the candidate, discard its responses, and compare them offline with those of the current model. Users are not affected, and the traffic is real. This suits models well, because most predictions have no side effects.
4. Canary it. Route a small share of users to the new model. Watch the system metrics, together with the quality signals: user feedback, the rate at which users retry or rephrase, the escalation rate, scores from a sampled judge, and the distribution of outputs.
5. A/B test when the question is whether it is better for the business, and not only whether it is safe. Assign users consistently to one arm, and measure the outcome that you care about.
6. Keep rollback instant. Leave the previous model loaded, or quick to load, and make switching a change of configuration. With start-up times of several minutes, a rollback that needs a cold start is too slow.

Record the model version, the prompt version and the serving configuration with every prediction. When someone asks why the assistant said something last Tuesday, that record is the only route to an answer, and it is the starting point for the observability module that follows.