Phase 14 · Lesson 30 of 54

Eval-Driven Agent Development

Anthropic's guidance: "start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when needed." Evaluation is not the last step. It's the outer loop that drives every other choice in Phase 14.

Lesson 30 of 54Learn + Build~60 minutesPython (stdlib)

Before this: All of Phase 14.

On this page

Learning Objectives

  • Name the three evaluation layers — static benchmarks, custom offline, online production — and what each is for.
  • Explain the evaluator-optimizer tight loop.
  • Describe the 2026 best practice: evals live next to code, run in CI, gate PRs.
  • Connect every Phase 14 lesson to the eval case it generates.

The Problem

Agents pass demos. They fail in production in ways demos cannot predict. Benchmarks answer "is this model broadly capable?" not "is this agent shipping the right patches for my product?" The answer: evaluation at three layers, running continuously, with every guardrail and learned rule mapped to an eval case.

The Concept

Three evaluation layers

  1. Static benchmarks — SWE-bench Verified for code (Lesson 19), WebArena/OSWorld for browsing / desktop (Lesson 20), GAIA for generalist (Lesson 19), BFCL V4 for tool use (Lesson 06). Use for cross-model comparison and regression gating. Contamination is real: SWE-bench+ found 32.67% solution leakage. Always report scores on SWE-bench Verified or on a leakage-audited set such as SWE-bench+.

  2. Custom offline evals — your product's shape:

    • LLM-as-judge (Langfuse, Phoenix, Opik — Lesson 24).
    • Execution-based (run the patch, check tests).
    • Trajectory-based (compare action sequences against gold; OSWorld-Human shows top agents 1.4-2.7x over gold).

  3. Online evals — production:

    • Session replays (Langfuse).
    • Guardrail-triggered alerts (Lesson 16, 21).
    • Per-step cost / latency tracking (Lesson 23 OTel spans).

Evaluator-optimizer (Anthropic)

The tight loop:

  1. Proposer generates output.
  2. Evaluator judges.
  3. Refine until evaluator passes.

This is Self-Refine (Lesson 05) generalized. Any agent flow you care about can wrap in evaluator-optimizer for reliability.

2026 best practice

  • Evals live next to code.
  • Run in CI on every PR.
  • Gate merge on eval scores (e.g. "no regression > 5% vs main").
  • Every guardrail maps to an eval case.
  • Every learned rule (Reflexion, pro-workflow learn-rule) maps to a failure case.

Tying Phase 14 together

Most lessons in Phase 14 generate eval cases. The table lists the lessons with the clearest one:

Lesson Eval case it generates
01 Agent Loop Budget-exhausted, infinite-loop guard
02 ReWOO Planner replans correctly when a tool fails
03 Reflexion Learned reflections apply on retry
05 Self-Refine/CRITIC Judge passes refined output
06 Tool Use Argument coercion works; unknown tools rejected
07-10 Memory Retrieval citations match sources; stale facts invalidate
12 Workflow Patterns Each pattern produces correct output
13 LangGraph Resume reproduces state exactly
14 AutoGen Actors DLQ catches crashed handlers
16 OpenAI Agents SDK Guardrail trips on the right inputs
17 Claude Agent SDK Subagent results return to orchestrator
19-20 Benchmarks SWE-bench Verified score, WebArena success rate, OSWorld efficiency
21 Computer Use Per-step safety catches injected DOM
23 OTel Spans emit required attributes
26 Failure Modes Detectors tag known failures
27 Prompt Injection PVE refuses poisoned retrievals
28 Orchestration Supervisor routes to the right specialist
29 Runtime Shapes DLQ handles N% failure

If your eval suite has a case for each row, you have covered the core of Phase 14.

Where eval-driven development fails

  • No baseline. Evals without a last-known-good are unreadable. Store baselines.
  • LLM-judge without grounding. Judges hallucinate too. CRITIC pattern (Lesson 05) — judge grounds on external tools.
  • Over-fitting to evals. Optimizing for the eval diverges from production usefulness. Rotate cases.
  • Flaky evals. Non-deterministic cases cause false alarms. Pin seeds, snapshot state.
ae-eval-three-layers

Build It

code/main.py is a stdlib eval harness:

  • Case registry with categories (benchmark, custom, online).
  • A scripted agent under test.
  • Evaluator-optimizer loop: propose, judge, refine until pass or max rounds.
  • CI gate: aggregate pass rate + regression against baseline.

Run it:

python3 code/main.py

Output: per-case pass/fail, regression flag, CI gate verdict.

Use It

  • Write eval cases in the same repo as your agent code.
  • Run them on every PR via CI.
  • Fail the build on regression.
  • Track pass rate over time.
  • Tie every production failure to a new case.

Ship It

outputs/skill-eval-suite.md builds a three-layer eval suite for an agent product with CI gates and regression tracking.

Exercises

  1. Take one of your production failures. Write an eval case that reproduces it. Does your agent pass it now?
  2. Build an LLM-judge rubric for your domain with three dimensions (factual, tone, scope). Score 50 sessions.
  3. Wire the eval suite into CI. Fail the build on >=5% regression.
  4. Add a trajectory-efficiency metric: how many steps did the agent take vs a gold trajectory?
  5. Map every Phase 14 lesson to an eval case in your suite. Any missing? That's a gap to close.

Key Terms

Term What people say What it actually means
Static benchmark "Off-the-shelf eval" SWE-bench, GAIA, AgentBench, WebArena, OSWorld
Custom offline eval "Domain eval" LLM-as-judge / exec / trajectory on your product shape
Online eval "Production eval" Session replay, guardrail alerts, cost/latency tracking
Evaluator-optimizer "Propose-judge-refine" Iterate until judge passes
CI gate "Merge blocker" Fail the build on eval regression
Baseline "Last-known-good" Reference score to detect regression
Trajectory efficiency "Steps over gold" Agent step count divided by human expert minimum

Further Reading

Check yourself

Quick quiz

  1. What are the three evaluation layers the lesson names?

    1. Unit, integration, end-to-end
    2. Smoke, regression, acceptance
    3. Pre, check, post
    4. Static benchmarks, custom offline evals, online production evals
    Show the answer

    D. Static benchmarks, custom offline evals, online production evals Static (SWE-bench, GAIA), custom offline (LLM-judge, exec, trajectory), online (replays, alerts, cost/latency).

  2. What is Anthropic's recommended starting point?

    1. Start with multi-agent debate
    2. Start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when needed
    3. Start with a frontier model only
    4. Start with hierarchical orchestration
    Show the answer

    B. Start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when needed Anthropic explicitly says evaluation is the outer loop that drives every other choice.

  3. What is the evaluator-optimizer tight loop?

    1. Proposer generates output, evaluator judges, refine until evaluator passes (Self-Refine generalized)
    2. Sample, sort, deduplicate
    3. Cache, retry, fail
    4. Train, evaluate, deploy
    Show the answer

    A. Proposer generates output, evaluator judges, refine until evaluator passes (Self-Refine generalized) It is Self-Refine generalized: any flow can wrap in propose-judge-refine.

  4. What is the 2026 best practice for where evals live?

    1. Owned exclusively by the QA team
    2. Only run quarterly
    3. Next to code, run in CI on every PR, gate merges on eval scores
    4. In a separate vendor dashboard only
    Show the answer

    C. Next to code, run in CI on every PR, gate merges on eval scores Co-located with code, CI-gated, regression-tracked is the standard.

  5. Why does the lesson warn against an LLM-judge without grounding?

    1. It violates Apache 2.0
    2. Judges hallucinate too; pair with the CRITIC pattern so judgment grounds on external tools
    3. It is too slow
    4. It only works on GPUs
    Show the answer

    B. Judges hallucinate too; pair with the CRITIC pattern so judgment grounds on external tools CRITIC (Lesson 5) applies: tool-grounded verification keeps the judge honest.

  6. What is the danger of over-fitting to evals?

    1. Vector indices fragment
    2. Compute cost rises
    3. Latency drops too far
    4. Optimizing for the eval set diverges from production usefulness; rotate cases
    Show the answer

    D. Optimizing for the eval set diverges from production usefulness; rotate cases Eval set rotation keeps the optimization aligned with production reality.

  7. Why do flaky evals cause problems?

    1. They exceed the context window
    2. They cannot reach the database
    3. They double inference cost
    4. Non-deterministic cases produce false alarms; pin seeds and snapshot state
    Show the answer

    D. Non-deterministic cases produce false alarms; pin seeds and snapshot state Flake makes regressions unreadable; determinism (seeds, state snapshots) is required.

Questions people ask

How do you test an AI agent?

Build a test set of realistic tasks with expected outcomes, including hard and edge cases. Run the agent on it after each change and score results with code checks, model graders or people. Track the scores over time.

What is an eval in AI?

An eval is a repeatable test that measures how well an AI system does a task. It pairs inputs with a way to judge outputs, such as an exact answer, a rubric or a human rating, so changes can be compared fairly.

Run the code Read it on GitHub

Next lessonAgent Workbench Engineering: Why Capable Models Still FailPrevious: Production Runtimes: Queue, Event, Cron

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes