A large language model can write code, summarise a contract and explain a stack trace, and it can also state something false with complete confidence. Both behaviours come from the same mechanism, and once you understand that mechanism, LLMs stop being magic and become a component you can engineer with. This module explains what a model actually does when you call it, and what that means for cost, latency, limits and reliability in the applications you build.
- Explain next-token prediction, and how pretraining and instruction tuning produce an assistant
- Reason about tokens as the unit of cost, speed and limits
- Treat the context window as the model's only working memory, and the API as stateless
- Control output with temperature, top-p, maximum tokens and stop sequences
- Explain why models hallucinate, and use embeddings to find relevant text
One thing, done repeatedly: predict the next token
An LLM is the decoder-only transformer from the previous module, at very large scale. Given a sequence of tokens, it outputs a probability for every token in its vocabulary as the one that comes next. To generate text, the application picks one token from that distribution, appends it to the sequence, and runs the model again. It repeats until the model emits a special end token, or a limit is reached. This loop is called autoregressive generation.
input: "The capital of France is"
model -> " Paris" 0.92
" the" 0.03
" located" 0.01
...every other token in the vocabulary...
pick " Paris" -> input becomes "The capital of France is Paris" -> run the model again -> ...Everything else follows from this. The model has no database of facts and no separate reasoning engine. It has billions of numerical weights that encode statistical patterns of language, and those patterns happen to capture a remarkable amount of knowledge and problem-solving ability. It also means that output is produced one token at a time, strictly in order, which is why generation is slow compared with reading, why responses can be streamed as they are produced, and why the model cannot revise an earlier sentence once it has been written.
A model becomes an assistant in stages.
| Stage | Data | Result |
|---|---|---|
| Pretraining | A very large corpus of text and code; the task is simply to predict the next token | A base model: fluent and knowledgeable, but it only continues text. Ask it a question and it may reply with more questions. |
| Instruction tuning | Curated examples of instructions paired with good responses | A model that follows instructions and holds a conversation |
| Preference tuning | Human or AI judgements about which of two responses is better (RLHF and related methods) | Responses that are more helpful, more honest and safer |
Two practical facts follow. A model's knowledge stops at its training cutoff: it knows nothing about later events, or about your private data, unless you put that information into the prompt. And the model you call through an API is a fixed snapshot. It does not learn from your conversations.
Tokens: the unit of everything
Models read and write tokens, not characters or words. A tokeniser splits text into pieces from a fixed vocabulary, using an algorithm such as byte-pair encoding that gives short codes to common sequences. Frequent English words are usually one token. Rare words, names, code, numbers and text in many other languages break into several.
"Hello world" -> ["Hello", " world"] 2 tokens
"Kubernetes" -> ["Kub", "ernetes"] 2 tokens (illustrative)
"antidisestablishment" -> ["ant", "idis", "establish", "ment"] 4 tokens (illustrative)
"2026-09-17T14:02:11Z" -> many small pieces 10+ tokens
Rule of thumb for English prose: 1 token is about 4 characters, or 0.75 words.
1,000 words is about 1,300 tokens.
The exact split differs between model families. Count with the provider's tokeniser.- Cost is charged per token, separately for input and output. Output tokens usually cost several times more than input tokens.
- Latency depends mostly on the number of output tokens, because they are generated one after another. A long prompt adds a little; a long answer adds a lot.
- Limits are in tokens: the context window, the maximum output, and your rate limits per minute.
- Non-English text and structured data such as JSON, logs and UUIDs can use far more tokens than their length suggests, which affects cost and how much fits.
- Character-level tasks are hard. The model sees
["Kub", "ernetes"], not letters, so counting the letters in a word, reversing a string, or exact arithmetic on long numbers is unreliable. Have it write code that does such work, or use a tool.
def estimate_tokens(text: str) -> int:
"""Rough estimate for English prose. Use the provider's tokeniser for real budgeting."""
return max(1, round(len(text) / 4))
def fits(prompt: str, context_window: int, reserve_for_output: int) -> bool:
return estimate_tokens(prompt) + reserve_for_output <= context_windowThe context window, and statelessness
The context window is the maximum number of tokens the model can consider at once, counting everything: system instructions, conversation history, documents you pasted in, tool definitions and results, and the response being generated. It is the model's only working memory. Whatever is not in the window does not exist for the model on that call.
The most important architectural fact for application builders is that the API is stateless. The model remembers nothing between calls. A chat application creates the appearance of memory by sending the whole conversation again with every request.
# call_llm is a stand-in: wire it to whichever provider you use.
# It takes a list of messages and returns the assistant's reply as a string.
messages = [
{"role": "system", "content": "You are a concise assistant for on-call engineers."},
{"role": "user", "content": "What does exit code 137 mean in Kubernetes?"},
]
reply = call_llm(messages)
messages.append({"role": "assistant", "content": reply})
# The follow-up only makes sense because the WHOLE history is sent again
messages.append({"role": "user", "content": "How do I find out which container it was?"})
reply = call_llm(messages)- Each turn costs more than the last, because the history grows. A fifty-turn conversation pays for the first message fifty times.
- When the history would overflow the window, you must do something: truncate the oldest turns, summarise them, or retrieve only the relevant parts. That decision is yours, not the model's.
- "Memory" features in products are built the same way: something stores notes outside the model, and places the relevant ones into the prompt.
- A larger window is not a substitute for selecting well. Models attend less reliably to material buried in the middle of a very long context, more tokens mean more cost and latency, and irrelevant text dilutes the relevant. Put in what is needed, clearly organised.
- Roles matter. The system message sets behaviour for the whole conversation. User and assistant messages alternate. Providers differ in the details, and the pattern is universal.
Sampling: controlling the output
The model outputs a probability distribution, and the sampling settings decide how a token is chosen from it. These are the main controls you have at request time.
| Parameter | Effect | Guidance |
|---|---|---|
| Temperature | Divides the logits before the softmax. Low values sharpen the distribution; high values flatten it. | Around 0 for extraction, classification and code. Higher for brainstorming and creative writing. |
| Top-p (nucleus) | Samples only from the smallest set of tokens whose probabilities add up to p | Leave at the default, or adjust temperature or top-p, not both |
| Top-k | Samples only from the k most likely tokens | Offered by some providers; rarely needs changing |
| Max output tokens | A hard cap on the length of the response | Always set it. It bounds cost and latency, and stops runaway output. |
| Stop sequences | Strings at which generation halts | Useful for structured formats and for ending at a delimiter |
Temperature 0 makes the model pick the most likely token every time, which is called greedy decoding. It makes output far more repeatable, and still not perfectly deterministic, because of floating-point behaviour on parallel hardware and because providers update models. Do not build anything that requires byte-identical output from one call to the next. If a response ends abruptly, check the stop reason that the API returns: a value indicating that the token limit was reached means the output was cut off, not that the model had finished.
# Deterministic-leaning task: extract fields from a log line
extraction = call_llm(messages, temperature=0, max_tokens=300)
# Open-ended task: propose names for an internal tool
ideas = call_llm(messages, temperature=1.0, max_tokens=500)Some recent models that reason at length before answering fix or restrict certain sampling parameters. Check the documentation for the model you are using instead of assuming that every setting is available.
Why models hallucinate
A hallucination is output that is fluent, confident and false: a function that does not exist, an invented citation, a plausible configuration flag. It is not a malfunction. It is the normal mechanism operating where it lacks grounding. The model is trained to produce a likely continuation. It has no internal marker that separates "I know this" from "this is the kind of thing that usually comes next", and a text that says "I am not sure" is, in most training data, less common than a text that answers.
- Hallucination is most likely for specifics: exact names, numbers, quotations, URLs, API signatures, version details, anything recent or obscure, and anything about your private systems.
- It is least likely where the answer is in the context. A model asked to answer from a document you have supplied is far more reliable than one asked from memory. This is the basis of retrieval-augmented generation, two modules ahead.
- Give it a way out. Instruct the model to say when the provided material does not contain the answer, and hallucination falls noticeably.
- Ask for evidence. Have it quote the passage that supports each claim, then check that the quote really appears in the source.
- Give it tools for what it is bad at: a calculator, a code interpreter, a search, a database query. A later module covers tool use.
- Verify mechanically whatever can be verified: compile the code, run the tests, validate the JSON against a schema, check that the URL exists.
- Keep a human in the loop in proportion to the stakes.
Fluency is not evidence of correctness. Models write with the same assured tone whether they are right or wrong, and people are strongly inclined to trust well-written text. Design your application on the assumption that any unverified specific may be wrong.
Embeddings
An embedding model is a related kind of model that, instead of generating text, turns a piece of text into a single vector which represents its meaning. Texts with similar meanings produce vectors that point in similar directions, so cosine similarity, from the maths module, measures how closely two texts are related, even when they share no words at all.
import numpy as np
# embed is a stand-in: it takes a list of strings and returns an array (n, dimension)
docs = [
"Pod is in CrashLoopBackOff after the deploy",
"How to request annual leave",
"Container keeps restarting with exit code 137",
]
doc_vecs = embed(docs)
query_vec = embed(["my application restarts over and over"])[0]
def normalise(v: np.ndarray) -> np.ndarray:
return v / np.linalg.norm(v, axis=-1, keepdims=True)
scores = normalise(doc_vecs) @ normalise(query_vec) # cosine similarities, shape (3,)
for i in np.argsort(-scores):
print(f"{scores[i]:.3f} {docs[i]}")
# the two restart-related documents rank first, despite sharing few words with the query- Uses: semantic search, retrieval for RAG, clustering similar tickets or incidents, removing near-duplicates, recommendations, and classification with very few examples.
- Embedding calls are far cheaper and faster than generation, and the vectors can be calculated once and stored.
- Vectors from different embedding models cannot be compared with each other. If you change the model, you must embed everything again.
- Embedding models have their own input limits, so long documents are split into chunks first. How to chunk is a design decision that the RAG module covers.
- Similar does not mean correct, or relevant to the question. "How do I enable feature X?" and "How do I disable feature X?" are close together in embedding space.