Cost & performance
Latency
Definition
Latency is the delay before and during a response. Two measures matter: time to first token, which governs perceived responsiveness, and tokens per second, which governs how fast the rest arrives.
Time to first token is what users experience as speed. Streaming responses lets you display text as it generates, which is why a streamed answer feels dramatically faster than an equivalent buffered one even though total time is identical.
Latency is driven by model size, output length, prompt length, and whether reasoning is involved. Reasoning models are notably slower because they generate substantial hidden deliberation before their first visible token.
Common mitigations: stream everything, route easy requests to smaller models, cache stable prefixes, and cap output length. For agent loops, latency compounds per step — a five-step agent inherits five round trips.
Related terms
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Prompt caching
Prompt caching stores the processed form of a repeated prompt prefix so subsequent requests reusing it are cheaper and faster. Cached input tokens typically cost a fraction of normal input tokens.
Reasoning model
A reasoning model is trained to generate extended internal deliberation before answering. It trades latency and cost for substantially better performance on maths, logic, coding and multi-step problems.
Streaming
Streaming sends a response token by token as it is generated rather than waiting for completion. It does not reduce total generation time but dramatically improves how fast the response feels.
Determinism
A deterministic system returns identical output for identical input. Language models are only approximately deterministic even at temperature 0, because floating-point and infrastructure variation introduce small differences.
Evaluation (evals)
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.