Build your first eval

Question 1

Your first eval

Accepted Answer

Pick 20 real prompts, write a clear pass/fail rubric, run it, track the score. Small and honest beats big and vague every single time.

Question 2

Collect real inputs

Accepted Answer

Mine logs, support tickets and complaints. Invented test cases are too polite — real users are messy, and messy is what you must survive.

Question 3

Label a golden set

Accepted Answer

Write the ideal answer for each case, with a note on why. Aim for coverage — common cases, edge cases, and the failures that burned you.

Question 4

Write the rubric

Accepted Answer

Define good concretely: criteria, weights, examples of pass and fail. 'Cites a source' beats 'is helpful' because two people score it the same.

Question 5

Choose your grader

Accepted Answer

Use code when the answer is checkable, a judge model when it's subjective, a human when the stakes are high. Cheapest reliable option wins.

Question 6

Validate the judge

Accepted Answer

Run your judge on human-labelled cases and measure how often it agrees. Below ~80%, fix the rubric — you're grading with a broken ruler.

Question 7

Set a threshold

Accepted Answer

Decide in advance what score is shippable and what blocks release. A metric with no threshold is a chart nobody acts on.

Question 8

Stress the edges

Accepted Answer

Empty inputs, other languages, huge documents, contradictory instructions, questions you should refuse. Edge cases are where trust is lost.

Question 9

Version everything

Accepted Answer

Pin the prompt, model, parameters and dataset for every run. If you can't say what changed between two scores, you learned nothing.

Question 10

Close the loop

Accepted Answer

Fix, re-run, compare, keep the winner. One change at a time, always against the same dataset — that's the whole discipline in one line.

Guides in this deck