Metrics
Which eval metrics actually matter?
The best metric is the one that maps to user value. Here's the short list that covers almost every AI product — and the traps in each.
Accuracy and pass rate
The share of eval cases your system gets right. Simple, universal, and the first number everyone asks for. Just make sure 'right' is defined by a rubric your users would agree with — a 95% score on the wrong criterion is worse than no score.
Pass@k
Give the model k attempts per case and count a win if any attempt succeeds. Standard for code generation, where a user will happily retry. pass@1 measures reliability; pass@10 measures capability. Report both.
Latency and cost per call
Quality that arrives too slowly or costs too much isn't quality. Track p50/p95 latency and cost per successful answer alongside accuracy. Most real-world wins come from hitting the same score faster or cheaper.
Rubric scores
For open-ended tasks, break quality into criteria — correctness, completeness, tone, citation of sources — and score each. Aggregate rubric scores tell you not just whether you regressed, but which dimension regressed.
Drift: the metric over time
A single score is a snapshot; the time series is the story. Models get updated, prompts evolve, user behavior shifts. Chart every eval run and set alerts on drops — silent regression is the most common production AI failure.