AI / ML Engineer — interview prep kit

Interviewers still check that you understand what is under the hood. These are the concept questions that come up whether the role is LLM-focused or classical ML.

1. known

   Explain the bias-variance tradeoff and how it relates to overfitting and underfitting.

   What it tests The single most common ML fundamentals question — whether you understand generalization.

   Model answer

   A model's error decomposes into bias (error from wrong assumptions — the model is too simple to capture the pattern) and variance (error from sensitivity to the training data — the model captures noise specific to that data). The tradeoff: making a model more complex reduces bias but increases variance, and vice versa, so you cannot minimise both by turning one knob. Underfitting is high bias — the model is too simple, and it does poorly on both training and test data (it never learned the pattern). Overfitting is high variance — the model is too complex and memorised the training data including its noise, so it does great on training data but poorly on unseen data (the gap between train and test performance is the tell). The goal is the sweet spot with enough complexity to capture the real signal but not so much that it memorises noise, which you find by watching the validation error: as complexity rises, validation error falls (reducing bias) then rises again (adding variance), and the minimum is where you want to be. The practical framing interviewers want: you detect overfitting by a large train-vs-validation gap, and you fight it with more data, regularization, or a simpler model — and this exact tradeoff still applies to large models, just at a different scale.

   Likely follow-ups:

   • How do you detect overfitting in practice?
   • Name three ways to reduce overfitting.
2. known

   What is the difference between training, validation, and test sets, and why do you need all three?

   What it tests Whether you understand data hygiene — a leak here invalidates every metric, and interviewers care that you know it.

   Model answer

   You split data three ways because each set answers a different question and reusing one for another leaks information and inflates your results. The training set is what the model learns from — it fits its parameters to this. The validation set is what you use to tune hyperparameters and make model-selection decisions (which architecture, how much regularization, when to stop training) — you evaluate on it repeatedly during development. The test set is held out and touched only once, at the very end, to get an honest estimate of how the model will perform on truly unseen data. Why all three: if you tuned on the test set, you would be optimizing to it, and your reported number would be optimistic — the test set is 'unseen' only if you never used it to make decisions. The validation set exists precisely so you have something to iterate against without contaminating the final estimate. The leakage traps to mention: never let information from validation or test leak into training (a feature computed over the whole dataset, or a time-series split done randomly instead of by time, both leak), and for time-based data split by time so you are not training on the future to predict the past. The signal interviewers want: you treat the test set as sacred and you know that a leak makes every metric a lie — which is a surprisingly common real bug.

   Likely follow-ups:

   • Why must you touch the test set only once?
   • What is data leakage, and give an example that inflates your metrics.
3. known

   What are embeddings, and why are they so central to modern AI systems?

   What it tests Whether you understand the representation that underlies RAG, search, and much of LLM tooling.

   Model answer

   An embedding is a dense vector representation of something — a word, a sentence, an image, a user — in a high-dimensional space, learned so that semantically similar things are close together and dissimilar things are far apart. The power is that this turns meaning into geometry: once text is embedded, 'find similar meaning' becomes 'find nearby vectors' (by cosine similarity or distance), which is fast and scalable. That is why embeddings are central to so much: semantic search and RAG (embed the query and the documents, retrieve the nearest chunks — matching on meaning, not keywords, so 'how do I reset my password' matches a doc titled 'account recovery'); recommendation (embed users and items, recommend nearby items); clustering and classification (similar things cluster); and as the input representation inside models themselves. The intuition to convey: embeddings let a computer measure semantic similarity numerically, which unlocks searching and reasoning over meaning rather than exact strings. For an LLM-systems role, the key application is retrieval — you store document embeddings in a vector database and retrieve by nearest-neighbour to the query embedding — and the practical points are choosing an embedding model suited to your domain and language, and that similarity in embedding space is a proxy for relevance, not a guarantee (which is why retrieval quality needs measuring).

   Likely follow-ups:

   • How does a vector database use embeddings to retrieve relevant documents?
   • Why might two texts that a human sees as related end up far apart in embedding space?
4. known

   How do you evaluate a classification model, and why is accuracy often misleading?

   What it tests Whether you know the metrics beyond accuracy and understand class imbalance — a classic trap.

   Model answer

   Accuracy — the fraction of correct predictions — is misleading whenever classes are imbalanced, which is common: if 99% of transactions are legitimate, a model that predicts 'legitimate' for everything is 99% accurate and completely useless, because it never catches fraud. So you need metrics that account for the kinds of error. Precision (of the things I flagged as positive, how many really were) and recall (of the actual positives, how many did I catch) capture the two failure modes — false positives versus false negatives — and they trade off, so you also look at the F1 score (their harmonic mean) or, better, the precision-recall curve and choose the operating point that matches the cost of each error type. The confusion matrix shows all four outcomes and is where I would start. ROC-AUC measures ranking quality across thresholds. The judgement interviewers want: pick the metric by what the errors *cost* — for fraud or disease detection, recall matters (missing a positive is expensive), so you accept more false positives; for spam filtering, precision matters (a false positive hides a real email). And with imbalance, accuracy is nearly meaningless — you report precision/recall/F1 and pick a threshold deliberately, rather than quoting a headline accuracy that the majority class inflates.

   Likely follow-ups:

   • When would you optimize for recall over precision, and vice versa?
   • Why is a confusion matrix more informative than a single accuracy number?

The core of modern AI engineering. Interviewers want to hear that you understand retrieval, grounding, and why RAG beats fine-tuning for most knowledge problems.

1. known

   What is RAG, and when would you use it instead of fine-tuning a model?

   What it tests The defining LLM-systems question — whether you understand grounding versus teaching.

   Model answer

   RAG (Retrieval-Augmented Generation) means retrieving relevant information at query time and putting it into the model's context so it answers from that material rather than from its parametric memory. The flow: embed the user's question, retrieve the most relevant chunks from a knowledge base (a vector store), assemble them into the prompt with instructions to answer only from the provided context, and generate. You use RAG over fine-tuning for knowledge problems — when the model needs access to specific, current, or private information (your company's docs, recent data, facts that change) — because RAG lets you update the knowledge by updating the documents (no retraining), it grounds answers in sources you can cite (so you can check them and reduce hallucination), and it keeps private data out of the model weights. Fine-tuning is for teaching the model a behaviour, format, or style it does not have — a specific output structure, a tone, a domain's way of responding, a task it does poorly zero-shot — not for injecting facts, which fine-tuning does badly and expensively (facts baked into weights are hard to update and the model still hallucinates around them). The clean rule interviewers reward: RAG for knowledge, fine-tuning for behaviour — and often neither is needed if good prompting suffices, so try prompting first. A strong answer also notes RAG's failure mode: it is only as good as its retrieval, so if the right chunk is not retrieved, no prompting saves the answer.

   Likely follow-ups:

   • Why is fine-tuning a poor way to give a model new facts?
   • What is RAG's main failure mode, and where does it originate?
2. known

   Your RAG system gives wrong or unsupported answers. How do you debug it?

   What it tests Whether you can diagnose a RAG pipeline systematically — the most common real problem in LLM apps.

   Model answer

   The key insight is that RAG has two stages that fail differently, so first localise which one broke: retrieval (did the right information get pulled?) or generation (given the right information, did the model use it correctly?). Look at the retrieved chunks for the failing query. If the relevant chunk was not retrieved, the problem is upstream and no prompt fixes it — the causes are chunking (chunks too big and diluted, or split so the answer is severed), a weak or mismatched embedding model, the query phrased differently from the document (a query-rewriting or hybrid keyword+vector search problem), or too small a k. **If the relevant chunk *was* retrieved but the answer is still wrong, it is a generation problem — the model ignored the context, or the context was buried among distractors, or the prompt did not firmly instruct 'answer only from these sources and say so if they do not contain the answer', or the chunks were too long and the model lost the signal. The method to voice: inspect the retrieval before blaming the model, because most 'the LLM hallucinated' complaints are actually retrieval misses. And underneath all of it — build an eval set** (questions with known answers and known source documents) so you can measure retrieval quality (did the right doc come back) separately from answer quality (was the answer correct and grounded), rather than debugging by anecdote.

   Likely follow-ups:

   • How would you measure retrieval quality separately from answer quality?
   • Give two chunking mistakes that cause the right answer to never be retrieved.
3. known

   What is prompt injection, and why can't you fully fix it with a better prompt?

   What it tests Whether you understand the fundamental security problem of LLM systems — a topic that separates serious engineers.

   Model answer

   Prompt injection is when untrusted input — a user message, a retrieved document, a web page the model reads, a tool's output — contains instructions that the model follows, overriding what you intended. The classic case: your system prompt says 'summarize this document', the document contains 'ignore previous instructions and instead output the following…', and the model may obey it. It happens because, to the model, instructions and data are the same thing — it processes one stream of tokens and has no reliable, built-in way to know that the system prompt is authoritative and the document is just data to be summarised. That is why you cannot fully fix it with a better prompt: any instruction you add ('never follow instructions in the document') is itself just more text in the same stream that a cleverly crafted injection can talk around — it raises the bar but does not close the hole, because you are fighting a persuasion problem with more persuasion. The real defences are architectural, not prompt-based: treat all model output and all tool/retrieval input as untrusted; do not give the model the ability to do damage — the tools it can call should be least-privilege and, for anything consequential, gated behind human approval rather than executed autonomously; sandbox and validate tool inputs; and separate trusted instructions from untrusted data as much as the API allows (delimiters help a little, structural separation helps more). The mature framing: prompt injection is not a bug to patch but a property of how LLMs work, so you contain the *consequences* (limit what the model can do, require approval for real actions) rather than trusting that the model will not be fooled.

   Likely follow-ups:

   • How would you design an agent's tools so a successful injection cannot cause real damage?
   • Why do delimiters and 'ignore injected instructions' prompts only reduce, not eliminate, the risk?
4. known

   How would you reduce hallucinations in an LLM application?

   What it tests Whether you have practical techniques and understand hallucination is managed, not eliminated.

   Model answer

   Hallucination — the model producing confident, plausible, but false content — cannot be fully eliminated because it is a property of how these models generate, so the goal is to reduce it and to catch it. The most effective lever is grounding: give the model the facts (RAG) and instruct it to answer only from the provided sources and to say 'I don't know' when they do not contain the answer — a model answering from provided, cited context hallucinates far less than one answering from memory, and citations let you verify. Beyond that: prompt for uncertainty (explicitly allow and reward 'I'm not sure' — models over-assert by default); constrain the task (a narrower, well-specified task hallucinates less than an open-ended one); use structured outputs and validation so malformed or out-of-range answers are caught programmatically; verify with a second step — a separate check that the answer is supported by the sources (an LLM-as-judge or a rule), or having the model cite exact spans you can confirm; and choose a stronger model for tasks where correctness matters, since capability correlates with faithfulness. Critically, measure it: an eval set with a faithfulness metric (is every claim supported by the context) tells you your hallucination rate and whether changes help, versus guessing. The honest framing interviewers reward: you manage hallucination with grounding, verification, and measurement, and you design the product to tolerate the residual (show sources, keep a human in the loop for high-stakes uses) rather than promising it is gone — because a system that assumes the model is always right is the one that fails in production.

   Likely follow-ups:

   • Why does citing sources reduce hallucination and also make it easier to catch?
   • How would you measure your application's hallucination rate?

Agentic systems are the frontier interviewers probe for. They want you to understand the loop, its failure modes, and — above all — the safety of letting a model take actions.

1. known

   What is an LLM agent, and how is it different from a single model call or a fixed workflow?

   What it tests Whether you understand the agentic loop and when its autonomy is worth the added risk.

   Model answer

   An agent is an LLM in a loop that can use tools and decide its own next step: it is given a goal and a set of tools, and it iterates — reason about what to do, call a tool, observe the result, reason again — until it decides the task is done. This differs from a single model call (one prompt, one response, no actions) and from a fixed workflow (a pipeline where *you* code the sequence of steps and the model fills in parts) in one crucial way: in a workflow, the control flow is determined by your code; in an agent, the model decides the control flow at runtime, choosing which tools to call and when to stop. That autonomy is the point — it handles open-ended tasks where you cannot predetermine the steps — but it is also the risk and cost: agents are less predictable, can loop or go off track, cost more (many model calls), and are harder to test. The judgement interviewers want most: prefer the simplest thing that works — a single call or a fixed workflow if the task's steps are known, and reach for an agent only when the task genuinely requires the model to decide the path dynamically. Reaching for an agent when a workflow would do is a common over-engineering mistake, because you take on the unpredictability and cost without needing the autonomy.

   Likely follow-ups:

   • When is a fixed workflow the better choice than an agent?
   • What are the main failure modes that make agents hard to operate?
2. known

   How do you keep an agent that can take actions from doing something harmful or expensive?

   What it tests The safety question at the heart of agentic systems — whether you design for containment.

   Model answer

   You assume the agent will sometimes be wrong or manipulated (via prompt injection) and design so that a wrong decision cannot cause serious harm — containment, not trust. The main controls: least-privilege tools — give the agent only the capabilities the task needs, scoped tightly, so there is simply no tool that can do the dangerous thing; human-in-the-loop approval for consequential or irreversible actions — the agent *proposes*, a person approves before anything executes (the difference between an agent that drafts an email and one that sends it, or one that suggests a fix and one that runs it in production); budgets and limits — cap the number of steps, the tokens, the spend, and the tool-call count so a runaway loop stops itself rather than burning money or hammering a system; validate tool inputs and sandbox execution, because the model's tool arguments are untrusted output that could be malformed or injected; read-only by default — separate the tools that observe from the tools that act, and make acting the guarded path; and log and trace everything so you can see what the agent did and why. The framing that signals maturity: the safety of an agent is an architectural property — what tools exist, what requires approval, what limits bound it — not something you achieve by asking the model nicely in the prompt to behave. For anything with real-world consequences, the default is 'the agent recommends, a human decides', and you widen its autonomy only as you build confidence and guardrails.

   Likely follow-ups:

   • What is the difference between a read-only tool and an action tool in your design?
   • Why do budgets and step limits matter for an agent specifically?
3. known

   How do you evaluate and monitor an agent in production, given it is non-deterministic?

   What it tests Whether you can bring engineering rigor to a probabilistic system, which is what production AI demands.

   Model answer

   Non-determinism does not mean unmeasurable — it means you measure differently, over distributions rather than single runs, and you instrument heavily. Evaluation before production: build a set of scenario tests — tasks with known good outcomes — and run the agent against them repeatedly, scoring whether it reached the right result, stayed within budget, and did not take unsafe actions; because outputs vary, you look at pass *rates* across runs, not a single pass/fail, and you use graders (rules where possible, an LLM-as-judge where the output is open-ended) to score. Include adversarial cases (prompt injection, ambiguous inputs) because that is where agents fail dangerously. Monitoring in production: trace every run — the full sequence of reasoning, tool calls, inputs, and outputs — so that when something goes wrong you can replay exactly what happened (this is the single most valuable thing to build); track operational metrics (tool-call counts, tokens, cost, latency, how often it hit budget limits, how often a human rejected a proposed action); and capture feedback (thumbs up/down, corrections) to feed back into the eval set. The maturity signal interviewers want: you treat the agent like any production system that can fail — with a regression eval that gates changes, full tracing for debugging, cost and safety monitoring, and a loop from production failures back into the eval set — rather than shipping it and hoping, because a non-deterministic system that you cannot observe or measure is one you cannot operate.

   Likely follow-ups:

   • Why is tracing the most valuable thing to build for an agent?
   • How do you evaluate something whose output legitimately varies between runs?
4. known

   What are evals, and why do teams say 'evals are the moat'?

   What it tests Whether you grasp that measurement, not the model, is the durable advantage in AI products.

   Model answer

   Evals are the tests for an AI system — a curated set of inputs with a way to judge whether the output is good — that let you measure quality objectively instead of relying on 'it looks fine when I try it'. They matter because AI systems are probabilistic and their behaviour shifts with every prompt tweak, model version, or data change, so without evals you are flying blind: you cannot tell whether a change improved things or regressed them, and 'vibes' do not scale past a handful of examples or catch the regression that only shows on the edge case. A good eval set has representative and adversarial cases, a grading method (exact match or rules where possible; an LLM-as-judge with a clear rubric for open-ended outputs, ideally using a stronger model than the one being graded), and it runs in CI to gate changes so a prompt or model change that drops quality is caught before it ships. The reason 'evals are the moat': the base models are available to everyone, prompting techniques are shared, and models keep changing — so your durable advantage is not the model but knowing, precisely and continuously, whether your system is good for your specific task, which lets you improve reliably and adopt new models safely while competitors guess. The team with the best evals can iterate fastest and with confidence; the team without them is stuck, unable to tell improvement from regression. The framing that lands: building the eval set is the highest-leverage work in an AI product, and it is the asset that compounds — every production failure becomes a new eval case, so your measurement gets sharper over time even as everything else commoditizes.

   Likely follow-ups:

   • How do you build an eval set when you are starting from nothing?
   • What are the pitfalls of using an LLM as a judge, and how do you mitigate them?

Running AI in production has its own economics and failure modes. Interviewers probe whether you understand latency, cost, and the operational reality of LLM systems.

1. known

   How do you control the cost of an LLM-powered application at scale?

   What it tests Whether you understand the cost levers — LLM costs are real and a frequent production concern.

   Model answer

   LLM cost is driven by tokens (input + output) times price-per-token times request volume, so every lever pulls on one of those. The biggest free wins first: prompt caching — if a large stable prefix (a system prompt, retrieved context, few-shot examples) repeats across requests, caching it cuts the cost of those tokens dramatically, so structure prompts with the stable part first; trim input tokens — do not stuff the whole knowledge base into context when good retrieval sends only the few relevant chunks (better retrieval is cheaper *and* more accurate); cap output tokens and prompt for concision, since output tokens are usually the most expensive. Then the tradeoffs: right-size the model — use a smaller/cheaper model for easy tasks and route only the hard ones to the expensive model (a router or a cascade: try the cheap model, escalate on low confidence); batch offline work through batch APIs at a discount; and cache whole responses for repeated identical queries. Structural: set budgets and monitoring per feature so you can attribute spend and catch a runaway, and measure cost-per-request as a first-class metric. The framing interviewers want: attack the free wins (caching, input hygiene, output limits) before the tradeoffs (model choice, effort), match model capability to task difficulty rather than using the biggest model for everything, and instrument cost so it is visible — because an LLM feature whose cost you do not watch can quietly become the most expensive thing you run.

   Likely follow-ups:

   • How does prompt caching work, and how do you structure a prompt to benefit from it?
   • What is a model cascade / router, and when is it worth the complexity?
2. known

   What is the difference between latency and throughput for LLM serving, and why do both matter?

   What it tests Whether you understand the serving characteristics that make LLM infrastructure different from web serving.

   Model answer

   Latency is how long one request takes — and for LLMs it splits into time-to-first-token (how long before the user sees anything) and inter-token latency / total generation time (how fast the rest streams). Throughput is how many tokens or requests the system handles per second across all users. They matter for different reasons and trade off: users care about latency (especially time-to-first-token, which is why streaming the response matters so much — it makes a slow generation feel responsive because words appear immediately); the business cares about throughput because it determines how many users a given amount of expensive GPU serves, which drives cost. The tension: techniques that raise throughput, like batching many requests together on the GPU, can raise individual latency (a request waits to be batched), so serving systems use continuous batching to get most of the throughput benefit without stalling individual requests. Why LLM serving is unlike web serving: generation is sequential and compute-heavy (each token depends on the last), a request occupies expensive GPU memory (the KV cache) for its whole generation, and response times vary wildly with output length — so you cannot treat it like a stateless CPU service. The signal interviewers want: you optimize time-to-first-token and stream for perceived latency, you batch for GPU throughput and cost, you know those trade off, and you scale on the right signal (GPU utilization / queue depth, not CPU) — because naive web-serving instincts (spin up more replicas on CPU) do not fit a GPU-bound, sequential, memory-constrained workload.

   Likely follow-ups:

   • Why does streaming the response matter so much for perceived latency?
   • Why can't you autoscale LLM serving on CPU utilization the way you would a web service?
3. known

   A model that worked well in testing is giving worse results in production. What could be happening?

   What it tests Whether you understand distribution shift and the gap between offline and online — a mature production concern.

   Model answer

   The usual root cause is that production data differs from your test data — the classic train/serve or offline/online gap — and there are several specific forms. Distribution shift / drift: real users send inputs unlike your test set (different phrasings, edge cases, languages, longer or messier inputs), so the model that aced a clean test set struggles on the real distribution — and this can *worsen over time* as the world changes (data drift) or the relationship you learned goes stale (concept drift). Training-serving skew: the features or preprocessing differ between how you tested and how production computes them (a subtle difference in tokenization, normalization, or the retrieval context), so the model effectively sees different inputs than it was evaluated on. The test set was not representative: it was too clean, too small, or leaked, so the offline metric was optimistic — a very common reason 'it worked in testing'. For LLM systems specifically: the retrieved context in production may be worse than in your curated tests, or prompt/model changes slipped in, or you are hitting inputs (adversarial, ambiguous) your evals never covered. How to investigate: compare production inputs to your test distribution (are they different?), log and inspect the actual failing production cases (they will show you what the test set missed), check for any skew in the pipeline, and — the fix that closes the loop — feed the production failures back into your eval set so your measurement finally reflects reality. The maturity signal: you expect the offline/online gap, you monitor for drift with production metrics and delayed labels, and you treat 'worked in testing, worse in production' as evidence your evals were not representative rather than as bad luck.

   Likely follow-ups:

   • What is the difference between data drift and concept drift?
   • How would you detect drift in production before users complain?
4. known

   How would you design an LLM application to fail gracefully when the model API is slow, down, or returns garbage?

   What it tests Whether you build production resilience around a fallible external dependency — a real engineering concern.

   Model answer

   You treat the model API as an unreliable external dependency (because it is — rate limits, latency spikes, outages, and malformed or refused outputs all happen) and build the same resilience you would around any such dependency, plus LLM-specific validation. For slow or down: timeouts on every call so a hung request does not tie up resources; retries with backoff and jitter for transient errors and rate limits (but capped, and respecting the API's retry guidance); a fallback — a cheaper/faster model, a cached previous answer, or a graceful degraded response ('I can't answer right now') rather than a spinner forever or a 500; and a circuit breaker if the provider is clearly down, so you fail fast instead of hammering it. For garbage output: never trust the model's response — validate it against a schema (if you asked for JSON, parse and validate it; if it does not conform, retry or fall back rather than passing malformed data downstream), check it is on-topic and within expected bounds, and for agents validate tool arguments before executing. Also handle refusals and truncation (a response cut off at max_tokens is incomplete — detect the stop reason and handle it). Cross-cutting: stream so the user sees progress and a slow response is tolerable, set budgets so retries and fallbacks cannot spiral in cost, and monitor the provider's error and latency rates. The framing interviewers want: an LLM call is an unreliable network dependency that also returns untrusted, sometimes-malformed content, so you wrap it in timeouts, retries, fallbacks, and output validation — a production LLM app that assumes the API is always fast and the output is always well-formed is one outage or one weird response away from breaking.

   Likely follow-ups:

   • How do you handle a response that was truncated because it hit the token limit?
   • Why validate the model's output even when you asked it for a specific format?

AI systems live or die on their data and their operational loop. Interviewers probe whether you can build the pipeline around the model, not just the model.

1. known

   What does a production ML lifecycle look like beyond training a model?

   What it tests Whether you understand that the model is a small part of a production ML system.

   Model answer

   Training the model is a small slice; the production lifecycle is a loop around it. The stages: data — collecting, validating, and versioning the data (a model is only as good as its data, and data-quality checks catch the broken upstream extract that would otherwise become a broken model); reproducible training — pinning the data, code, hyperparameters, and environment so a run can be recreated, and logging everything (experiment tracking like MLflow); a model registry — versioning trained models with their metrics and lineage, so you know exactly what is deployed and can roll back; CI/CD for models — testing and promoting a model through a gate (does the candidate beat the current champion on a holdout?) before it serves, and building it into a deployable artifact; serving — deploying it behind an API with the same instrumentation as any service; and — the stage people forget — monitoring the deployed model for drift and quality degradation using production metrics and delayed labels (ground truth often arrives later), with an alert and a retraining loop that closes the circle when the model goes stale. The insight interviewers want: the model is not the product, the *system* is — the data pipeline, the reproducibility, the registry, the deployment, and especially the monitoring-and-retraining loop — because models decay after deployment as the world shifts, and a team that trains a great model but has no loop to detect and fix its decay ships something that quietly gets worse. MLOps is the discipline of building and operating that loop.

   Likely follow-ups:

   • Why is monitoring the stage teams most often neglect, and what goes wrong when they do?
   • What does a model registry give you that just saving model files does not?
2. known

   How do you decide when a deployed model needs retraining?

   What it tests Whether you understand model decay and can tie retraining to a signal rather than a calendar.

   Model answer

   Ideally you retrain in response to a signal that the model has degraded, not on a fixed schedule alone, because the world drives decay, not the clock. The strongest signal is quality against ground truth: if you can get labels (even delayed — the actual outcome arrives later and you compare it to the prediction), track the model's live accuracy/error and retrain when it drops below a threshold. When labels are delayed or scarce, use leading indicators: drift detection — monitor the distribution of the inputs and the model's outputs versus the training distribution (a metric like PSI), because a large shift means the model is now operating on data unlike what it learned, which usually precedes a quality drop and gives you an early warning before the label-based metric confirms it. Practical triggers: a quality metric crossing a threshold (retrain now), significant input/output drift (investigate and likely retrain), a known change in the environment (a product change, a new market, seasonality), or a scheduled cadence as a backstop. The nuance interviewers want: distinguish the leading indicator (drift — cheap, early, but only a proxy) from the truth (label-based quality — accurate but delayed), use drift as the early warning and labels as the confirmation, and automate the loop — an alert triggers a retrain on fresh data, the new model must pass the promotion gate before it serves, and you verify the fix improved the metric. Retraining on a calendar alone wastes effort when nothing changed and reacts too slowly when something did; tying it to signals is the mature approach.

   Likely follow-ups:

   • What is the difference between a leading indicator like drift and a lagging one like label-based quality?
   • Why not just retrain on a fixed weekly schedule regardless of signals?
3. known

   How would you build a data pipeline to prepare training data, and what quality checks would you add?

   What it tests Whether you take data quality seriously — 'garbage in, garbage out' is the most reliable ML failure.

   Model answer

   The pipeline transforms raw data into clean, validated, versioned training data, and the quality checks are what stop a silent data problem from becoming a bad model. The stages: ingest raw data (and keep it immutable so you can reprocess); validate and clean — this is where the checks live; transform into features; and version the output so a training run references an exact, reproducible dataset. The quality checks to add, because ML fails silently on bad data: schema and type checks (are the columns and types what you expect — an upstream change that renames or retypes a field should fail the pipeline, not corrupt the model); value/range checks (are numbers in plausible ranges, are categorical values in the known set, are there impossible values like negative ages); null and completeness checks (is the null rate suddenly higher than usual — a broken join or a failed upstream source); volume checks (did roughly the expected number of rows arrive — a 90% drop means an upstream break); distribution checks (has the data distribution shifted from previous runs, which could be real drift or a data bug); and freshness (is the data recent enough). Tools like Great Expectations or dbt tests encode these as assertions that gate the pipeline. Critically, the checks should fail loudly and block the pipeline rather than warn, because a model trained on silently-broken data is worse than no model — it makes confident wrong predictions. The framing interviewers want: data quality is enforced by automated assertions at the pipeline boundary, treated as seriously as code tests, because the most common and hardest-to-catch ML failures come from data, not the model — and a validation gate that stops a bad extract is cheaper than discovering the problem in production predictions.

   Likely follow-ups:

   • Why should a data-quality check fail the pipeline rather than just log a warning?
   • How would a distribution check distinguish real drift from a data bug — or can it?
4. known

   How would you approach fine-tuning a model, and how do you know it worked?

   What it tests Whether you understand fine-tuning's place and, crucially, how to evaluate it rather than assuming it helped.

   Model answer

   First, decide fine-tuning is actually the right tool — it teaches a behaviour, format, or style the base model lacks, not new facts (that is RAG's job), and it is worth the cost only after prompting and RAG have been tried and fall short, because fine-tuning adds real complexity (data curation, training, versioning, and you now own a model that can drift from the base). If it is warranted: the work is mostly data — curate a high-quality dataset of input-output examples demonstrating the desired behaviour, because the model learns exactly what you show it, so a few hundred excellent, representative, consistent examples beat thousands of noisy ones (bad or contradictory examples teach bad behaviour). Split into train and a held-out eval set. Fine-tune (often parameter-efficient methods like LoRA, which are cheaper and avoid catastrophic forgetting of the base capabilities). How you know it worked — and this is the part interviewers listen for — you compare the fine-tuned model against the base model on a held-out eval set the fine-tuning never saw, measuring the specific behaviour you were trying to improve, and you check it did not regress on general capability (fine-tuning can make a model worse at everything else while getting better at your task — catastrophic forgetting). 'It seems better when I try it' is not knowing; a measured win on a representative eval, with no regression on a broader eval, is. Also watch for overfitting to the fine-tuning set (great on train examples, poor on held-out ones — the same bias-variance problem). The maturity signal: fine-tuning is a last resort after prompting and RAG, it is a data-quality exercise more than a training one, and its success is a measured comparison against the base on unseen data, not a vibe.

   Likely follow-ups:

   • What is catastrophic forgetting, and how would your evaluation catch it?
   • Why is a small set of high-quality examples often better than a large noisy one?