Phase 19 · Lesson 27 of 85

Capstone Lesson 29: End-to-End Coding Agent on the Harness

Track A's payoff. This lesson stitches the gate chain, the sandbox, the eval harness, and the OTel spans into one working coding agent that fixes a real (small, fixture-scale) bug in a multi-file Python project. The agent is a deterministic policy, not an LLM; the substitution makes the lesson reproducible and shows that the harness was the interesting part all along. The contract is identical: a real model plugs in at the policy seam.

Lesson 27 of 85Build~90 minutesPython (stdlib)

Before this: Lesson 23 of Phase 19 (verification gates), lesson 24 of Phase 19 (sandbox), lesson 25 of Phase 19 (eval harness), lesson 26 of Phase 19 (observability), lesson 38 of Phase 14 (verification gates), lesson 41 of Phase 14 (workbench for real repos), lesson 42 of Phase 14 (agent workbench capstone)

On this page

Learning Objectives

  • Compose the gate chain, sandbox, eval harness, and span builder into a single agent loop.
  • Implement a deterministic policy that uses read_file, run_tests, and write_file to fix a fixture bug.
  • Enforce a global step budget plus an observation token budget across an end-to-end run.
  • Emit complete OTel GenAI traces and Prometheus metrics for the full run.
  • Verify the agent solves the fixture in fewer than 12 steps with zero gate trips on legal tools.

The Problem

Most agent demos work in isolation: a sandbox by itself, an eval harness by itself, a span emitter by itself. They look fine. Compose them and the seams show.

The gate chain says ALLOW but the sandbox refuses for a reason the chain did not anticipate. The eval harness records a pass but the OTel spans say the gate refused a tool the agent claims it used. The Prometheus counter is incremented twice when it should be incremented once. The observation budget is exceeded but the agent kept going because the budget was tracked in the chain and the sandbox didn't know.

This lesson is the integration test for the whole track. The agent has to do six things in order: read the project, run the tests, identify the bug from the test failure, write the fix, rerun the tests, and stop. Every operation goes through the gate chain. Every tool execution goes through the sandbox. Every step is wrapped in a span. The eval harness scores the whole thing at the end.

The Concept

Diagram (opens full size)

The agent's policy is a state machine. Five states.

SURVEY: the agent reads the project listing. The next state is RUN_TESTS.

RUN_TESTS: the agent runs the test command. If the tests pass, the state machine halts with success. Otherwise the next state is INSPECT.

INSPECT: the agent reads the failing source file. The next state is FIX.

FIX: the agent writes the corrected file. The next state is VERIFY.

VERIFY: the agent runs the test command again. If the tests pass, halt success. Otherwise halt with failure.

Each state corresponds to a tool call. Each tool call passes through the gate chain. If a tool call is denied, the agent reports the refusal in the trace and halts.

The fixture bug is an off-by-one in fizz.py. The deterministic policy detects the bug from the test failure message via a regex and emits the corrected file. Replacing the policy with an LLM does not change the harness contract.

cg-harness-weave

Architecture

Diagram (opens full size)

The lesson is self-contained. Each prior-lesson primitive is reimplemented at minimal scale in main.py (gate, sandbox, ledger, span) so the lesson runs without importing siblings. The names match lessons 25-28 exactly so the conceptual mapping is unambiguous.

What you will build

main.py ships:

  1. The minimal harness primitives, copied with the same names as lessons 25-28: GateChain, Sandbox, ObservationLedger, SpanBuilder, MetricsRegistry.
  2. CodingAgentPolicy class: state machine with five states.
  3. Repo helper: prepares a scratch dir with the bundled buggy fixture.
  4. AgentRun class: drives the policy, dispatches through the harness, returns an AgentRunReport.
  5. A bundled fixture (fixture_repo/) with src/fizz.py, tests/test_fizz.py, and an expected/ tree for the eval harness.
  6. Demo: runs the policy end-to-end, prints the step-by-step trace, asserts pass, prints metrics.

The bundled fixture is the same shape as lesson 27's task structure: a buggy file and a tests file. The test failure message contains enough information for the deterministic policy to identify the fix. A real LLM would do the same job, slower and with broader recall, but it would not change the harness's expectations.

Why the policy is not an LLM

A real LLM requires an API key, a network call, and unverifiable stochasticity. The harness is the part the lesson cares about. Subbing in a deterministic policy lets the lesson run on any developer laptop with zero external dependencies and lets the test suite assert exact-step counts.

The lesson's policy is a strict subset of what an LLM agent does. The policy reads the repo, sees the failing test, identifies the line, and emits a fix. An LLM goes through the same loop with the same harness contract; the bookkeeping is identical.

What the demo asserts

The end-to-end demo asserts five things at exit time, and the test suite reasserts them programmatically.

The policy solved the fixture in fewer than 12 steps.

The observation budget was never exceeded.

Zero gate denials fired on legal tools. (The agent never invented a denied tool name.)

Every step has a corresponding span in the traces.jsonl.

The Prometheus exposition contains a tools_called_total{tool="read_file"} entry and a tool_latency_ms histogram.

How this composes with the rest of Track A

This lesson is the integration. Lesson 25 wrote the gate chain. Lesson 26 wrote the sandbox. Lesson 27 wrote the eval harness. Lesson 28 wrote the observability. Lesson 29 proves they work as a system. A real agent harness extends from here: swap the deterministic policy for a model, swap the bundled fixture for a real-repo task, swap the JSONL exporter for OTLP.

Running it

cd phases/19-capstone-projects/29-end-to-end-coding-task-demo
python3 code/main.py
python3 -m pytest code/tests/ -v

The demo prints a per-step trace, the final eval report, and the Prometheus exposition. Exit code is zero. The tests cover the policy state transitions, the gate refusals on synthetic tool calls, the end-to-end run on the bundled fixture, and the step-budget invariants.

Check yourself

Quick quiz

  1. Before reading the lesson: why does the end-to-end demo replace the LLM with a deterministic policy?

    1. Deterministic policies always outperform LLMs on coding tasks.
    2. The lesson cannot afford the inference cost.
    3. The LLM is too slow to run in tests.
    4. The harness is what the lesson is testing; the policy is the substitutable seam.
    Show the answer

    D. The harness is what the lesson is testing; the policy is the substitutable seam. The harness contract does not care whether the policy is an LLM or a state machine. Removing the LLM removes the network and the variance and turns the lesson into a reproducible integration test.

  2. The agent's state machine has five states. In what order does it execute them on a normal pass?

    1. RUN_TESTS -> SURVEY -> INSPECT -> FIX -> VERIFY.
    2. SURVEY -> RUN_TESTS -> INSPECT -> FIX -> VERIFY.
    3. SURVEY -> FIX -> VERIFY -> RUN_TESTS -> INSPECT.
    4. SURVEY -> INSPECT -> FIX -> RUN_TESTS -> VERIFY.
    Show the answer

    B. SURVEY -> RUN_TESTS -> INSPECT -> FIX -> VERIFY. The agent surveys the project, runs the tests, inspects the failing test, fixes the bug, then verifies. RUN_TESTS comes before INSPECT because the test failure is the signal the policy reads.

  3. The bundled fixture's bug is an off-by-one in src/fizz.py: range(1, n) instead of range(1, n + 1). Where does the policy decide to apply the fix?

    1. Inside the SURVEY state, by reading the file header.
    2. Outside the state machine, in the gate chain.
    3. Inside the VERIFY state, after the rerun.
    4. Inside the INSPECT state, after the test failure has named the expected output.
    Show the answer

    D. Inside the INSPECT state, after the test failure has named the expected output. INSPECT is where the policy reads the failing test and learns the expected shape. The fix is identified there and written in the next FIX state.

  4. Every tool call goes through what chain of components in order?

    1. ObservationLedger -> GateChain -> Sandbox -> SpanBuilder.
    2. SpanBuilder -> GateChain -> Sandbox -> ObservationLedger.
    3. GateChain -> SpanBuilder -> Sandbox -> ObservationLedger.
    4. Sandbox -> GateChain -> SpanBuilder -> ObservationLedger.
    Show the answer

    C. GateChain -> SpanBuilder -> Sandbox -> ObservationLedger. Gate first (allow / refuse), span next (so the refusal is also captured), sandbox third (only if allowed), ledger last (only on success). The lesson code wires it that way.

  5. You replace the deterministic policy with a real LLM. Which harness component changes?

    1. The sandbox has to be hardened with seccomp.
    2. The Prometheus exposition format becomes incompatible.
    3. The gate chain has to be rewritten to handle LLM stochasticity.
    4. Nothing in the harness changes; the policy is the only piece that swaps.
    Show the answer

    D. Nothing in the harness changes; the policy is the only piece that swaps. The lesson's point: the harness is policy-agnostic. The seam is the next_action / observe interface. The LLM plugs in there without touching gate, sandbox, ledger, or spans.

  6. The demo asserts the agent solves the fixture in fewer than 12 steps. The deterministic policy uses 5. What is a real-LLM agent likely to do, and how does the harness handle it?

    1. The LLM cannot run inside this harness.
    2. The LLM uses 5 steps too; the harness is unchanged.
    3. The harness silently extends the budget.
    4. The LLM may use more steps; the step_budget bounds the loop and an unsolved run is reported as a failure with a halted_reason.
    Show the answer

    D. The LLM may use more steps; the step_budget bounds the loop and an unsolved run is reported as a failure with a halted_reason. The step budget is the loud, deterministic stopping criterion. An LLM that wanders gets capped and the report tells you exactly why.

Questions people ask

Why test an AI agent with a scripted policy instead of a real model?

A scripted policy gives the same steps every time, so the test is repeatable and free. It proves the harness, gates, sandbox and logging work. Once those pass, a real model can be plugged into the same contract.

What parts does a production coding agent need?

A loop that talks to the model, a tool registry with schemas, checks that block risky calls, a sandbox that runs commands, an evaluation suite, and tracing for cost and timing. The model is one part among several.

Run the code Read it on GitHub

Next lessonBPE Tokenizer From ScratchPrevious: Capstone Lesson 28: Observability with OTel GenAI Spans and Prometheus Metrics

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes