Traditional software has one thing that changes: the code. A machine learning system has three, namely the code, the data and the model, and an LLM application adds a fourth, the prompts. Any one of them can change the system's behaviour, and most teams version only the first. The result is the model that nobody can reproduce, the "improvement" that cannot be traced to its cause, and the silent regression that ships because nothing tested for it. MLOps applies the DevOps discipline of this site to all four. This module shows what to version, how to make training repeatable, and how to build a pipeline whose gates are evaluations.
- Identify everything that determines an ML system's behaviour, and put each of them under version control
- Make training runs reproducible, with pinned environments, seeds and tracked experiments
- Use a model registry to promote models through stages, with their lineage attached
- Design a CI/CD pipeline for ML and for prompts, with data validation and evaluation gates
- Recognise training-serving skew and the feedback loops that degrade models in production
More things change than the code
In ordinary software, the same code always behaves in the same way, so versioning the code is enough. In an ML system, the behaviour is a function of several artifacts, and a change to any of them is a change to production.
| Artifact | Examples | How to version it |
|---|---|---|
| Code | Feature engineering, training scripts, the serving application | Git |
| Data | The training set, the labels, the evaluation set | Immutable snapshots with content hashes; data versioning tools; dated partitions |
| Configuration | Hyperparameters, feature lists, thresholds | Files in Git, never command-line arguments that are lost |
| Environment | Library versions, CUDA, the base image | A lockfile, together with a container image pinned by digest |
| Model | The trained weights or the fitted pipeline | A model registry, with lineage back to everything above |
| Prompts | System prompts, templates, few-shot examples, tool definitions | Files in Git, with a version identifier logged on every call |
| External models | The provider's model, and its exact version | Pinned identifiers in configuration; never a moving alias |
The test of whether you have this under control is lineage. For any prediction that your system made last month, can you name the exact model, and for that model, the exact code, data, configuration and environment that produced it? If not, you cannot reproduce a bug, cannot audit a decision, and cannot tell which of five simultaneous changes caused an improvement.
This matters for LLM applications that train nothing at all. An application built on a hosted model still has prompts, retrieval settings, tool definitions, an evaluation set and a pinned model version, and a change to any of them alters what users see. MLOps for such an application is mostly the discipline of treating those as code.
Git is the wrong tool for large files. Keep datasets and model weights in object storage, and have Git track a small pointer file that holds the content hash. Data versioning tools such as DVC do exactly this, and so does a plain convention of immutable, dated paths, recorded in the experiment's metadata.
Reproducible training
A training run should be a deterministic function of versioned inputs: given the same commit, the same snapshot of the data, the same configuration and the same environment, it produces an equivalent model. Four habits get you most of the way there.
- Pin the environment. A lockfile with exact versions, installed into a container image that is referred to by digest. A silent upgrade of a numerical library can change results.
- Fix the seeds of every source of randomness: Python, NumPy, the ML framework, and the splitting of the data. Record the seed along with the run. Bit-for-bit reproducibility on GPUs is not always achievable, and results that fall within a small tolerance are usually enough.
- Snapshot the data. Train from an immutable, versioned snapshot, never from a live table that changes beneath you. Record its hash.
- Put configuration in files. Every hyperparameter lives in a versioned configuration file. Nothing that matters is typed at a prompt and then forgotten.
Experiment tracking records each run: its parameters, its metrics, its artifacts, and the versions of code and data. It turns "I think the run on Tuesday was better" into a table that you can sort. MLflow is a widely used open-source tracker, and the pattern is the same in all of them.
import hashlib
import json
import random
import subprocess
from pathlib import Path
import mlflow
import numpy as np
def file_hash(path: str) -> str:
return hashlib.sha256(Path(path).read_bytes()).hexdigest()[:16]
config = json.loads(Path("config/train.json").read_text())
random.seed(config["seed"])
np.random.seed(config["seed"])
with mlflow.start_run(run_name="churn-gbm"):
# lineage: what exactly produced this model?
mlflow.log_param("git_commit", subprocess.check_output(
["git", "rev-parse", "HEAD"], text=True).strip())
mlflow.log_param("data_hash", file_hash(config["train_path"]))
for key, value in config["hyperparameters"].items():
mlflow.log_param(key, value)
model, metrics = train_and_evaluate(config) # your own training code
for name, value in metrics.items(): # e.g. val_f1, val_pr_auc, latency_ms
mlflow.log_metric(name, value)
mlflow.sklearn.log_model(model, "model")Run training in the pipeline, not on a laptop. A model trained by hand on someone's machine has an environment that nobody can reconstruct. A model trained by a pipeline job, from a commit, has its lineage by construction.
The model registry
A model registry is to models what the artifact registry of the DevOps track is to container images: the hand-off point between whoever produces them and whoever deploys them. It stores immutable, numbered versions of each model, with their metadata, and records which version is approved for which purpose.
- Every registered version links back to the run that produced it, and through that to the code, the data and the configuration. This is the lineage, made navigable.
- Versions carry a stage or an alias, such as candidate, staging and production. Deployment refers to the alias, and promotion moves the alias. Rolling back means moving it back again.
- Promotion is a controlled action, with a record of who did it and why, and ideally one that the pipeline performs after the gates have passed, not something a person does by hand.
- Store the evaluation results with the model: the metrics on the standard evaluation set, broken down by segment, together with latency and size.
- Attach a short model card: what the model is for, what data it was trained on, its known limitations, the groups on which it performs worse, and what it must not be used for. It is the README that the next engineer, or an auditor, will need.
The serving layer of the previous module pulls the artifact that the production alias points to. That is the "build once, promote everywhere" principle of the DevOps track: the artifact that was evaluated is, byte for byte, the artifact that serves traffic.
CI/CD for machine learning
An ML pipeline has the stages of a software pipeline, with two additions: the data is tested as well as the code, and the quality gate is an evaluation, not a unit test that passes or fails.
commit / new data / schedule
-> lint + unit tests (the code is correct)
-> validate the data (schema, ranges, nulls, volume, drift against the last snapshot)
-> train (pinned environment, seeded, tracked)
-> evaluate (standard eval set, by segment; latency; size)
-> GATE: beats the threshold AND does not regress against production
-> register as a candidate (with lineage and a model card)
-> deploy to staging -> shadow / canary in production (the serving module)
-> promote the alias -> monitor (the next module)import json
import sys
from pathlib import Path
THRESHOLDS = {"val_pr_auc": 0.80, "recall_at_precision_90": 0.55}
MAX_REGRESSION = 0.01 # allowed drop against the production model, per metric
MAX_LATENCY_MS = 50.0
MIN_SEGMENT_RECALL = 0.45 # no customer segment may fall below this
candidate = json.loads(Path("reports/candidate.json").read_text())
production = json.loads(Path("reports/production.json").read_text())
failures = []
for metric, floor in THRESHOLDS.items():
if candidate[metric] < floor:
failures.append(f"{metric} {candidate[metric]:.3f} is below the threshold {floor}")
if candidate[metric] < production[metric] - MAX_REGRESSION:
failures.append(f"{metric} regressed: {production[metric]:.3f} -> {candidate[metric]:.3f}")
if candidate["p99_latency_ms"] > MAX_LATENCY_MS:
failures.append(f"p99 latency {candidate['p99_latency_ms']:.1f} ms exceeds {MAX_LATENCY_MS}")
for segment, recall in candidate["recall_by_segment"].items():
if recall < MIN_SEGMENT_RECALL:
failures.append(f"recall for segment '{segment}' is {recall:.3f}")
if failures:
print("GATE FAILED:\n - " + "\n - ".join(failures))
sys.exit(1)
print("gate passed")name: train-and-gate
on:
push:
paths: ["src/**", "config/**", "data.dvc"]
workflow_dispatch:
jobs:
train:
runs-on: ubuntu-latest
steps:
- name: Check out the code
run: git clone --depth 1 "$REPO_URL" . && git checkout "$COMMIT_SHA"
- name: Install the pinned environment
run: pip install --require-hashes -r requirements.txt
- name: Fetch the versioned data
run: dvc pull
- name: Unit tests
run: pytest tests/unit
- name: Validate the data
run: python -m pipeline.validate_data --snapshot data/train.parquet
- name: Train
run: python -m pipeline.train --config config/train.json
- name: Evaluate the candidate and the current production model
run: python -m pipeline.evaluate --out reports/
- name: Quality gate
run: python gate.py
- name: Register the candidate
run: python -m pipeline.register --stage candidate- Validate the data before you train. Check the schema, the types, the ranges, the rates of missing values, the volume of rows, and the distributions of the labels and features against the previous snapshot. Most bad models come from bad data, and a model trained on a broken join will pass every test of the code.
- Evaluate by segment, not only in total. A model that improves on average, and collapses for one region or one group of customers, must fail the gate.
- Compare with what is in production, not only with a fixed threshold. The question is whether this model is better than the one that you have.
- Include the non-functional metrics: inference latency, the size of the model, and its memory. A model that is more accurate and too slow for its SLO is not shippable.
- Retrain on a trigger: a schedule, the arrival of new labelled data, or an alert about drift from monitoring. Automated retraining still goes through every gate.
Prompts and LLM applications as code
For an application built on a hosted LLM, the pipeline is the same, with the training step removed. The artifacts are prompts, retrieval configuration, tool definitions and a pinned model version, and the gate is the eval suite from the evals module.
prompts/
triage/
system.md <- the prompt, reviewed in pull requests like code
examples.jsonl <- few-shot examples
config.json <- {"version": "triage-v12", "model": "PINNED-MODEL-ID",
"temperature": 0, "max_tokens": 300}
evals/
triage/cases.jsonl <- the golden dataset, also versioned
triage/baseline.json <- the scores of the version now in production- A pull request that changes a prompt runs the eval suite, and shows the difference in scores in the review, exactly as a change to code runs the tests.
- The prompt version is logged with every call, so that a change in behaviour in production can be traced to the edit that caused it.
- Upgrading the provider's model is a release. Change the pinned identifier in a branch, run the full eval suite, and compare. A prompt tuned for the old model is often wrong for the new one.
- The evaluation set is itself an artifact, with a history. When you add cases, record the change, because scores before and after it are not comparable.
- Roll out changes to prompts with the same canary and rollback mechanisms as anything else. Feature flags work well: the new prompt version is a flag that can be switched off in seconds.
- Keep prompts out of the database, and out of an admin screen that bypasses review. An unreviewed edit to a prompt is an unreviewed deployment to production.
What goes wrong after deployment
A deployed model is correct for the world as it was when the training data was collected. The world moves on, and the pipeline has to notice.
| Problem | What it is | Defence |
|---|---|---|
| Training-serving skew | Features are computed in one way for training and in another way in production: a different library, a different default for missing values, a different time zone | One shared implementation of the features; ship the whole pipeline as one artifact; log the features used in serving, and compare them with those used in training |
| Data drift | The distribution of the inputs changes: new kinds of customer, a new product, a changed format upstream | Monitor the distributions of the inputs against the training baseline, and alert |
| Concept drift | The relationship between the inputs and the outcome changes: fraud patterns evolve, and behaviour shifts | Monitor real outcomes when the labels arrive; retrain on recent data |
| Label delay | The true outcome is known only weeks later, so that accuracy cannot be measured in real time | Watch proxy signals and the distributions of predictions in the meantime |
| Feedback loops | The model's own decisions shape the data on which it is next trained, as when a recommender only ever learns about what it chose to show | Keep a small randomised sample; be careful when training on data that the model influenced |
| Breakage upstream | A source system changes a field, a unit or an encoding, and the model quietly receives nonsense | Validation of the inputs at serving time; checks of data contracts in the pipeline |
A feature store is one structural answer to skew. It is a system that computes each feature once, from a single definition, and serves the same values to training, from an offline store with correct historical values, and to production, from a low-latency online store. It is worth its considerable complexity when many models share their features. A single model is usually better served by one shared library of features, and a pipeline that ships the preprocessing together with the model.
Start smaller than the diagrams of tooling suggest. Git for the code, the configuration and the prompts, dated immutable paths for the data, a tracker for the runs, a registry with aliases, and a pipeline with a single evaluation gate will take a small team a long way. Add a feature store, automated retraining and an orchestration platform when a specific pain demands them, and not before.