Tools & shipping

Question 1

Tools & frameworks

Accepted Answer

Open-source harnesses plus observability platforms that score live traffic. Pick one, keep it boring, and spend your effort on the dataset.

Question 2

promptfoo

Accepted Answer

Config-driven eval runs from a YAML file: prompts, providers, assertions, side-by-side diffs. The fastest way to go from spreadsheet to suite.

Question 3

DeepEval & Ragas

Accepted Answer

Python libraries with ready-made metrics — faithfulness, answer relevance, context recall — that plug straight into pytest and RAG pipelines.

Question 4

Evals & Inspect

Accepted Answer

OpenAI Evals and UK AISI's Inspect give you registries of graded tasks and a standard format — handy for capability and safety-style testing.

Question 5

Tracing & observability

Accepted Answer

Log every prompt, tool call and response with a trace id. You cannot evaluate what you never recorded — instrumentation comes before scoring.

Question 6

Evals in CI/CD

Accepted Answer

Run the suite on every prompt or model change, like unit tests, and block the deploy when the score drops below your threshold.

Question 7

Scoring production

Accepted Answer

Sample a slice of live traffic and grade it automatically. Offline scores tell you what should happen; online scores tell you what did.

Question 8

User feedback signals

Accepted Answer

Thumbs, edits, retries, abandonment. Implicit signals are noisy but free, and they surface failure types your dataset never imagined.

Question 9

Keep evals cheap

Accepted Answer

Full suite nightly, fast smoke subset on every commit, cached results for unchanged cases. An eval too slow to run is an eval nobody runs.

Question 10

Build an eval culture

Accepted Answer

Show the score in the room where decisions happen. When 'did the eval move?' becomes the default question, quality stops being an opinion.

Guides in this deck