Module 6 of 6 · Advanced · 8 min read
Evaluating agents
How to know whether your agent works, and whether a change made it better.
- 1Evaluate outcomes in the environment, not just the final message.
- 2A small set of realistic tasks beats a large synthetic one.
- 3Trace every run so failures can be diagnosed.
01Outcome-based evals
Agents often claim success when they have not succeeded. Check the actual end state: do the tests pass, is the record in the database, does the page render? Write graders in code where possible.
For subjective outputs, use model graders with specific rubrics and spot-check them against human judgement.
02Build the eval set
Start small and realistic.
- 20 to 50 tasks drawn from real usage.
- Include hard and adversarial cases.
- Record expected outcomes and acceptable variations.
- Run several trials per task: agents are non-deterministic.
03Tracing and iteration
Log every model call, tool call and result. When an eval fails, read the trace to find where it went wrong: a misunderstood instruction, a bad tool output, lost context.
Re-run the full eval on every prompt, tool or model change. Regressions are common and silent without it.
Finished reading?
Mark it complete to track your progress, then test yourself with the Benchmarks & Evals quiz.