Cost & performance
Prompt caching
Definition
Prompt caching stores the processed form of a repeated prompt prefix so subsequent requests reusing it are cheaper and faster. Cached input tokens typically cost a fraction of normal input tokens.
Many applications send a large, stable prefix on every request — a long system prompt, a document, a set of few-shot examples — followed by a small variable part. Caching lets the provider skip reprocessing the stable portion.
Discounts are substantial, often 75-90% on cached tokens, with latency improvements as well. The requirement is that the prefix be byte-identical, which has a design implication: put stable content first and variable content last. A timestamp at the top of your prompt defeats caching entirely.
Cache lifetimes are short — typically minutes — so this helps high-frequency workloads much more than occasional requests.
Related terms
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Token
A token is the unit of text an AI model reads and writes — roughly four characters or three-quarters of a word in English. Models price by the token and cap context by the token, so token count determines both what you pay and what fits.
Latency
Latency is the delay before and during a response. Two measures matter: time to first token, which governs perceived responsiveness, and tokens per second, which governs how fast the rest arrives.
Streaming
Streaming sends a response token by token as it is generated rather than waiting for completion. It does not reduce total generation time but dramatically improves how fast the response feels.
Determinism
A deterministic system returns identical output for identical input. Language models are only approximately deterministic even at temperature 0, because floating-point and infrastructure variation introduce small differences.
Evaluation (evals)
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.