Cost & performance
Evaluation (evals)
Definition
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Without evals, you are guessing. A prompt change that fixes the case in front of you may quietly break five others, and you will not find out until users do.
A workable eval set starts small: 20-50 representative inputs with expected outputs or grading criteria. Run it after every meaningful change. Grade automatically where you can (exact match, schema validity, presence of required elements) and use an LLM judge or human review for subjective quality.
Include the failures you have already fixed. An eval set that only contains easy cases will pass forever while quality drifts.
Related terms
Determinism
A deterministic system returns identical output for identical input. Language models are only approximately deterministic even at temperature 0, because floating-point and infrastructure variation introduce small differences.
LLM as judge
LLM as judge uses one model to grade another's output against criteria you define. It makes evaluating subjective quality practical at scale, with known biases to control for.
Prompt engineering
Prompt engineering is the practice of designing model inputs that reliably produce the output you want. It combines clear instruction-writing, structured formatting, worked examples, and systematic testing.
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Prompt caching
Prompt caching stores the processed form of a repeated prompt prefix so subsequent requests reusing it are cheaper and faster. Cached input tokens typically cost a fraction of normal input tokens.
Latency
Latency is the delay before and during a response. Two measures matter: time to first token, which governs perceived responsiveness, and tokens per second, which governs how fast the rest arrives.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.