Skip to content

Guides and resources for AI engineers: an agents course, five handbooks, a hands-on second-brain project and a vetted reading list.

2.9 KB
Raw Blame

Eval Resources

A short list, deliberately. Everything here earns its place; everything it links to is optional. The concepts they teach map onto why evals and the rest of this handbook.

The canon

  • Your AI Product Needs Evals — Hamel Husain. The essay that made evals a discipline: unit tests, human review, and A/B tests as three levels, plus the case for looking at your data. Start here.
  • AI Evals FAQ — Husain and Shreya Shankar's running answers to the questions every team asks: binary vs Likert, how many labels, who should annotate, build vs buy. Dense and free.
  • AI Evals for Engineers & PMs — the Maven course behind the FAQ; several thousand engineers and PMs through it, cohorts still running in 2026. Paid, and the one course worth the money in this space.
  • Who Validates the Validators? — Shankar et al. The research grounding for aligning LLM judges with human labels; explains why rubric criteria drift as humans grade more outputs.
  • Demystifying Evals for AI Agents — Anthropic's engineering guide (January 2026): task selection, trajectory vs outcome grading, judge calibration, evals as CI. The closest thing to an official playbook for agent evals.
  • Evaluation Best Practices — OpenAI's guide; complements the above and pairs with the evals cookbook section for worked judge examples.

Repos worth reading

  • inspect_evals — 200+ evaluations implemented under UK AISI's Inspect. Read a few scorers and solvers to see what production-grade eval code looks like.
  • terminal-bench — hand-audited tasks in Docker with executable graders. The cleanest reference for building environment-based evals of your own; more context in agent evals.

Start here: three steps

  1. Read 30 traces. Pull thirty real (or realistic) transcripts from your product tonight and write one sentence per failure. No tooling, just a spreadsheet. This is error analysis, and it will reshape your roadmap.
  2. Name the failures, encode the top three. Cluster your notes into a taxonomy, then turn the three most frequent failure modes into checks — code assertions where possible, one narrow calibrated judge where not. The mechanics are in designing evals and llm as judge.
  3. Put the suite in CI and re-run weekly. Wire the checks into your pipeline with a threshold, fix the biggest failure mode, and return to step 1 with fresh traces. That loop, repeated, is the entire discipline.