Types of evals

Question 1

The eval families

Accepted Answer

Deterministic checks, reference-based scoring, model-graded judges, and human review. Most teams blend cheap automation with a small human set.

Question 2

Exact match & assertions

Accepted Answer

Does the output equal the answer, match a regex, parse as valid JSON, or contain the required field? Cheap, instant, zero ambiguity — start here.

Question 3

Reference-based scoring

Accepted Answer

Compare against a gold answer with similarity measures or embeddings. Useful for summaries and translation, weak when many answers are valid.

Question 4

LLM-as-a-judge

Accepted Answer

A strong model grades outputs against a rubric — fast, cheap, scalable. Anchor it to human labels first and watch for self-preference bias.

Question 5

Pairwise comparison

Accepted Answer

Ask 'A or B?' instead of 'score 1–5'. Humans and judges are far more reliable at ranking two options than at assigning an absolute number.

Question 6

Human evaluation

Accepted Answer

People rate quality, tone and correctness. Slow and costly, but still the gold standard that every automated grader must be calibrated against.

Question 7

Criteria grading

Accepted Answer

Score each output on separate criteria — factual, complete, on-tone, safe — instead of one blurred 'quality' number. You learn where it broke.

Question 8

Regression suites

Accepted Answer

Every bug you fix becomes a permanent test case. The suite only grows, and it's the cheapest insurance against re-shipping yesterday's failure.

Question 9

Adversarial & red team

Accepted Answer

Deliberately attack your system: jailbreaks, prompt injection, toxic bait, nonsense inputs. Safety is a capability you have to test, not assume.

Question 10

RAG evals

Accepted Answer

Score retrieval and generation separately: did it fetch the right chunks, and did the answer stay grounded in them? Most RAG bugs are retrieval bugs.

Guides in this deck