Models & architecture
Distillation
Definition
Distillation trains a smaller model to imitate a larger one, transferring much of the capability at a fraction of the inference cost. Most small production models are distilled from bigger teachers.
The student model trains on outputs from the teacher rather than only on raw text. Because the teacher provides richer signal than a single correct label — including its uncertainty across options — the student learns more efficiently than it would from the original data alone.
Distillation is why the small-model tier has improved so quickly. A distilled 8B model can approach the quality of a much larger model on the specific distribution of tasks it was distilled for, while costing a small fraction to run.
The limitation is generality: distilled models tend to match their teacher on covered task types and fall off more sharply outside them.
Related terms
Parameters
Parameters are the learned numerical weights inside a model, adjusted during training. Parameter count — 7B, 70B, 405B — is a rough proxy for capacity, but not a reliable predictor of quality.
Fine-tuning
Fine-tuning continues training a pre-trained model on your own examples to specialise its behaviour. It is the right tool for teaching consistent style, format or classification behaviour — not for adding knowledge.
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Large language model (LLM)
A large language model is a neural network trained on vast amounts of text to predict the next token in a sequence. That single objective, at sufficient scale, produces the ability to write, summarise, translate, reason and code.
Transformer
The transformer is the neural network architecture behind virtually all modern language models. Its key innovation is self-attention, which lets every token directly consider every other token in the input.
Attention
Attention is the mechanism that lets a model weigh how much each token should influence the interpretation of every other token. It is how a model resolves what a pronoun refers to, or which clause modifies which noun.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.