Core ML concepts

Learning from examples

In supervised learning you have examples for which the right answer is known. Each example has features, the inputs, usually written X, and a label or target, the output, written y. Training finds a function that maps X to y well enough to be useful on examples it has never seen. That last clause is the entire point. Reproducing the training data is easy, and it is called a lookup table.

Most of the effort, and most of the gain, lies outside the model. Deciding what to predict, obtaining reliable labels, and building informative features usually matter far more than which algorithm you pick. A simple model on good features beats a sophisticated model on poor ones.

1. Frame the problem. What decision will the prediction drive? What would a wrong answer cost, in each direction?
2. Get and understand the data, using the tools from the previous two modules. Look hard at how the labels were produced.
3. Establish a baseline. How well does a trivial rule do: always predict the majority class, or predict the same value as yesterday?
4. Split the data before doing anything else with it.
5. Train a simple model first, then more complex ones if they are needed.
6. Evaluate on data the model has never seen, with a metric that reflects the real cost.
7. Deploy and monitor, because the world changes and the model does not.

Train, validation, test

You cannot judge a model on the data it learned from, for the same reason you cannot assess a student with the exact questions they revised. Hold some data back.

The validation set exists because tuning is itself a form of learning. If you try fifty configurations and keep whichever scores best on the test set, you have fitted your choices to that set, and its score is no longer an honest estimate. Keep the test set locked away until you have finished making decisions.

from sklearn.model_selection import train_test_split

# first carve off the test set, then split the remainder
X_rest, X_test, y_rest, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.25, stratify=y_rest, random_state=42
)
# result: 60% train, 20% validation, 20% test

• stratify=y keeps the class proportions the same in every split, which is essential when one class is rare.
• random_state fixes the shuffle, so that results can be reproduced.
• Time-ordered data must be split by time, not at random. Train on the past, test on the future. A random split lets the model look ahead, and produces wonderful scores that vanish in production.
• Keep groups together. If one customer, host or patient has many rows, all of them belong in the same split. Otherwise the model recognises the individual instead of learning the pattern. Use GroupKFold or GroupShuffleSplit.

With limited data, a single validation split is noisy. k-fold cross-validation divides the training data into k parts, trains k times, each time validating on a different part, and averages the scores. It costs k times the computation and gives a much more stable estimate, along with a sense of its variance.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

scores = cross_val_score(LogisticRegression(max_iter=1000), X_rest, y_rest, cv=5, scoring="f1")
print(f"F1: {scores.mean():.3f} +/- {scores.std():.3f}")

Underfitting and overfitting

A model can fail in two opposite ways. It underfits when it is too simple to capture the pattern, like a straight line through curved data. It is poor on the training data and poor on new data. It overfits when it is flexible enough to memorise the training examples, noise included. It is excellent on the training data and poor on new data. The goal is generalisation, which lies between the two.

This is the bias-variance trade-off. Simple models are consistently wrong in the same way, which is bias. Flexible models are sensitive to the particular sample they were given, which is variance. The most reliable cure for overfitting is more data. Where that is not possible, regularisation penalises complexity: L2 regularisation shrinks weights towards zero, L1 drives some to exactly zero, limiting the depth of a tree stops it from carving out one leaf per example, and dropout and early stopping do the equivalent job for neural networks.

from sklearn.metrics import f1_score
from sklearn.tree import DecisionTreeClassifier

for depth in (2, 5, 10, None):
    model = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_train, y_train)
    train_f1 = f1_score(y_train, model.predict(X_train))
    val_f1 = f1_score(y_val, model.predict(X_val))
    print(f"depth={depth!s:>4}  train={train_f1:.3f}  val={val_f1:.3f}")

# typical pattern: training score climbs to 1.000 as depth grows,
# while the validation score peaks and then falls. The peak is the depth to use.

Always compare the two scores side by side. A training score alone tells you nothing, and the gap between the two tells you nearly everything.

Data leakage

Leakage means that information which will not be available at prediction time has found its way into training. The model learns to exploit it, scores brilliantly in evaluation, and fails in production. It is the most common reason that impressive results do not survive deployment, and it is hard to spot because nothing looks broken.

• Target leakage. A feature that is a consequence of the label. Predicting whether an incident will be severe, using a postmortem_written column. Predicting churn, using account_closed_date.
• Preprocessing leakage. Calculating a scaler's mean, or an imputer's fill value, on the whole dataset before splitting it. The test set's statistics have leaked into training.
• Temporal leakage. Using data from after the moment of prediction, or splitting a time series at random.
• Duplicate and group leakage. The same record, or the same customer, appearing in both the training and the test set.
• Selection leakage. Choosing features by their correlation with the label, using all the data, before cross-validating.

The test for any feature is a single question: would I know this value at the moment I need to make the prediction? The structural defence against preprocessing leakage is a pipeline, which bundles the preprocessing with the model. Calling fit learns the preprocessing statistics from the training data only, and the identical transformation is then applied to validation, test and production data.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["latency_ms", "payload_kb", "requests_last_hour"]
categorical = ["plan", "region"]

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

model = Pipeline([
    ("preprocess", preprocess),
    ("classifier", LogisticRegression(max_iter=1000, class_weight="balanced")),
])

model.fit(X_train, y_train)            # statistics learned from training data ONLY
val_predictions = model.predict(X_val) # the same transformation is applied, not re-fitted

A pipeline has an operational benefit too. It is a single object that you save and deploy, so the preprocessing in production is guaranteed to match training. A mismatch between the two, known as training-serving skew, is a classic production failure, and the MLOps module returns to it.

Choosing the metric

Accuracy, the share of predictions that are correct, is the natural first metric and frequently the wrong one. When classes are imbalanced it rewards ignoring the rare class, which is usually the one you care about. Look instead at the confusion matrix, which counts each kind of outcome.

                    predicted positive     predicted negative
actually positive   TP  true positive      FN  false negative  (missed)
actually negative   FP  false positive     TN  true negative
                        (false alarm)

precision = TP / (TP + FP)      of everything flagged, how much was real?
recall    = TP / (TP + FN)      of everything real, how much was caught?
F1        = 2 x precision x recall / (precision + recall)
accuracy  = (TP + TN) / total

Example: 1,000 transactions, 20 of them fraud. The model flags 30, of which 15 are fraud.
  TP = 15   FP = 15   FN = 5   TN = 965
  precision = 15 / 30 = 0.50      recall = 15 / 20 = 0.75
  F1 = 2 x 0.50 x 0.75 / 1.25 = 0.60      accuracy = 980 / 1000 = 0.98

Accuracy of 98% sounds excellent, and "never flag anything" would also score 98%. Precision and recall tell the real story: half the alarms are false, and a quarter of the fraud gets through. This is the base-rate effect from the maths module, and it is the same trade-off as alert precision and recall in the SRE track.

Precision and recall pull against each other. Most classifiers output a probability, and you choose the threshold above which to call something positive. Lower it and you catch more, which raises recall, while raising more false alarms, which lowers precision. Which one matters more is a business decision about the cost of each kind of error. Missing a cancer is worse than a false alarm, so favour recall. Blocking a legitimate customer's payment is costly, so favour precision.

from sklearn.metrics import (average_precision_score, classification_report,
                             confusion_matrix, roc_auc_score)

proba = model.predict_proba(X_val)[:, 1]          # probability of the positive class

print(confusion_matrix(y_val, proba >= 0.5))
print(classification_report(y_val, proba >= 0.5, digits=3))

print("ROC AUC:", roc_auc_score(y_val, proba))              # threshold-free ranking quality
print("PR AUC: ", average_precision_score(y_val, proba))    # better when positives are rare

for threshold in (0.3, 0.5, 0.7):                 # choose the operating point deliberately
    print(threshold, confusion_matrix(y_val, proba >= threshold).ravel())

Which model?

Start simple, and add complexity only when validation results justify it.

For structured, tabular data, which is most business and operational data, gradient-boosted trees remain the usual winner, and deep learning rarely helps. For images, audio and language, neural networks dominate, and in practice you will seldom train one from scratch. You will adapt a pretrained model, which is where the next module begins.