Cost & performance
LLM as judge
Definition
LLM as judge uses one model to grade another's output against criteria you define. It makes evaluating subjective quality practical at scale, with known biases to control for.
Human grading is accurate but slow and expensive. LLM judges are fast and cheap enough to run on every change, which is what makes continuous evaluation feasible.
Judges have documented biases: they prefer longer answers, favour their own family's outputs, and are sensitive to option order in comparisons. Counter these by supplying an explicit rubric, randomising position in pairwise comparisons, and requiring a justification before the score.
Validate the judge against human labels on a sample before trusting it. If judge and human agree on most of a hundred examples, the judge is usable for that task.
Related terms
Evaluation (evals)
Evaluation is systematically measuring AI output quality against a test set, rather than judging from a handful of examples. It is what makes prompt and model changes safe to ship.
Determinism
A deterministic system returns identical output for identical input. Language models are only approximately deterministic even at temperature 0, because floating-point and infrastructure variation introduce small differences.
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Prompt caching
Prompt caching stores the processed form of a repeated prompt prefix so subsequent requests reusing it are cheaper and faster. Cached input tokens typically cost a fraction of normal input tokens.
Latency
Latency is the delay before and during a response. Two measures matter: time to first token, which governs perceived responsiveness, and tokens per second, which governs how fast the rest arrives.
Streaming
Streaming sends a response token by token as it is generated rather than waiting for completion. It does not reduce total generation time but dramatically improves how fast the response feels.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.