Types
The main types of AI evals
There's no single 'eval' — there are four families, each with its own cost, speed, and blind spots. Strong teams mix them instead of picking one.
1. Exact match and programmatic checks
The output is compared to a known answer: string match, JSON schema validation, code that compiles and passes tests, a number within tolerance. Cheap, fast, deterministic — and only possible when there's a verifiable right answer.
Best for: classification, extraction into structured formats, code generation, math.
2. LLM-as-a-judge
A strong model grades another model's output against a rubric: "Does the answer cite a source? Is it under 100 words? Is the tone professional?" This scales to open-ended tasks where exact match is impossible.
The catch: judges have biases. They favor longer answers, their own style, and confident tone. Always anchor a judge to a small set of human-labeled examples before trusting it.
Best for: summaries, chat responses, creative or support writing, RAG answers.
3. Human evaluation
People rate outputs directly — side-by-side comparisons, Likert scales, or free-form review. Slow and expensive, but it's the ground truth that calibrates every automated grader. Even a few hundred human labels can validate whether your LLM judge agrees with your users.
Best for: defining 'good', calibrating judges, high-stakes launches.
4. Regression suites
Not a grader but a habit: re-run your evals on every prompt change, model upgrade, and data refresh. A regression suite turns evals from a one-time audit into a living safety net wired into CI.
Best for: anything in production, forever.