Metrics & scoring

Question 1

Which metrics matter?

Accepted Answer

Accuracy, latency, cost per call and rubric scores. Pick the few that map to user value — measure what you ship, not what's easy to compute.

Question 2

Pass rate

Accepted Answer

The share of cases your system gets right. Simple, honest, comparable across runs — and the first number anyone will ask you for.

Question 3

pass@k

Accepted Answer

Give the model k attempts and count a win if any one succeeds. Essential for code and agent tasks where retries are part of the product.

Question 4

Precision & recall

Accepted Answer

Precision: of what you flagged, how much was right. Recall: of what mattered, how much you caught. A single accuracy number hides the trade-off.

Question 5

Groundedness

Accepted Answer

What share of claims is supported by the retrieved source? It's the practical way to measure hallucination without arguing about the word.

Question 6

Latency & cost

Accepted Answer

Track p50 and p95 latency and cost per successful task. A model that's two points better and four times slower is usually a downgrade.

Question 7

Rater agreement

Accepted Answer

If two humans disagree on a case, no judge can be right about it. Measure agreement first — it's the ceiling on every score you'll report.

Question 8

Is the gain real?

Accepted Answer

On 50 cases, a 4% jump is noise. Report confidence intervals, re-run with a fixed seed, and grow the dataset before you celebrate.

Question 9

Eval drift

Accepted Answer

Scores shift as models, prompts and traffic change. Re-run on a schedule and chart the trend so regressions never ship silently.

Question 10

Goodhart's law

Accepted Answer

When a metric becomes a target, it stops being a good metric. Rotate held-out cases and keep reading raw outputs, not just the dashboard.

Guides in this deck