Cost & performance
Determinism
Definition
A deterministic system returns identical output for identical input. Language models are only approximately deterministic even at temperature 0, because floating-point and infrastructure variation introduce small differences.
Setting temperature to 0 makes the model always select the highest-probability token, which gets you close to deterministic behaviour. It does not guarantee it. GPU non-determinism, batching effects and backend changes can all alter output.
This has testing implications. Assertions on exact model output are fragile. Test for properties instead: valid JSON, required fields present, values within range, correct classification.
It also has product implications. If your users expect the same input to give the same result, cache the result rather than relying on the model to reproduce it.
Related terms
Temperature
Temperature controls how random a model's word choices are. Low values (0-0.3) make output focused and repeatable; high values (0.8-1.2) make it more varied and creative but less reliable.
Top-p (nucleus sampling)
Top-p limits the model to choosing from the smallest set of tokens whose combined probability exceeds p. At top-p 0.9, the model samples only from options making up the top 90% of probability mass, ignoring the unlikely tail.
Evaluation (evals)
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Prompt caching
Prompt caching stores the processed form of a repeated prompt prefix so subsequent requests reusing it are cheaper and faster. Cached input tokens typically cost a fraction of normal input tokens.
Latency
Latency is the delay before and during a response. Two measures matter: time to first token, which governs perceived responsiveness, and tokens per second, which governs how fast the rest arrives.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.