Tools
Tools & frameworks for AI evals
The ecosystem is crowded, but the categories are few. Pick one harness, one place to track scores, and one habit — running evals in CI.
Open-source eval harnesses
- promptfoo — YAML-defined test cases, assertions, and LLM-graded rubrics. Great first harness; runs locally and in CI.
- DeepEval — pytest-style evals in Python with metrics for RAG, hallucination, and faithfulness.
- OpenAI Evals — the original open-source eval framework and registry, useful for reference patterns even if you don't adopt it wholesale.
LLM-as-judge tooling
Most harnesses support judge prompts out of the box. What matters isn't the tool but the setup: a tight rubric, few-shot examples, and periodic checks that judge scores still agree with human reviewers.
Observability platforms
Tools like LangSmith, Langfuse, and Braintrust score real production traffic alongside your offline evals. Offline evals tell you how the system should behave; production scoring tells you how it actually behaves. You want both views.
Evals in CI/CD
The endgame: your eval suite runs on every pull request that touches a prompt, model version, or retrieval pipeline — exactly like unit tests. Set a threshold, block merges below it, and let the scoreboard guard your releases.