Foundations

Automated tests that score a model's outputs on your tasks — unit tests for AI. An eval runs prompts against a dataset and grades the results.

Not vibes, not a demo, not a leaderboard rank. An eval you can't re-run, or whose result you can't explain, is an anecdote wearing a number.

Four parts: a dataset of inputs, the system under test, a grader that scores each output, and a reported metric. Change one, hold the rest fixed.

A small, hand-checked set of inputs with ideal answers. It's your source of truth — guard it, version it, and never let a model write it alone.

On day one, with 20 examples in a spreadsheet. The cost of starting is an hour; the cost of not starting is never knowing if you improved.