Phase 19 · Lesson 16 of 85

Capstone 16 — GitHub Issue-to-PR Autonomous Agent

Label an issue, get a PR — the 2026 product shape for autonomous coding agents: run an agent in a cloud sandbox, verify tests pass, and post a review-ready PR with rationale. AWS Remote SWE Agents, Cursor Background Agents, OpenAI Codex cloud, and Google Jules all ship it. The hard parts are reproducing the repo's build environment automatically, preventing credential leakage, enforcing per-repo budgets, and making sure the agent cannot force-push. This capstone builds the self-hosted version and compares it on cost and pass rate to the hosted alternatives.

Lesson 16 of 85Capstone30 hoursPython (agent), TypeScript (GitHub App), YAML (Actions)

Before this: Phase 11 (LLM engineering), Phase 13 (tools), Phase 14 (agents), Phase 15 (autonomous), Phase 17 (infrastructure)

On this page

Problem

The async cloud coding agent is a separate product category from interactive coding agents (capstone 01). The UX is a GitHub label. You label an issue @agent fix this, a worker spins up in a cloud sandbox, clones the repo, runs tests, edits files, verifies, and opens a PR with the agent's rationale in the body. No interactive loop, no terminal. AWS Remote SWE Agents, Cursor Background Agents, OpenAI Codex cloud, Google Jules, and Factory Droids all converge on this.

The engineering challenges are concrete: environment reproduction (the agent has to build the repo from scratch without a cached dev image), flaky tests (must be re-run or isolated), credential scoping (a GitHub App with minimal fine-grained permissions), budget enforcement per repo per day, and no-force-push policy. The capstone measures pass rate, cost, and safety vs the hosted alternatives.

Concept

The trigger is a GitHub webhook (issue label or PR comment). A dispatcher enqueues work to ECS Fargate or Lambda. The worker pulls the repo into a Daytona or E2B sandbox with a generic Dockerfile inferred from the repo (language, framework). The agent runs a mini-swe-agent or SWE-agent v2 loop against Claude Opus 4.7 or GPT-5.4-Codex. It iterates: read code, propose fix, apply patch, run tests.

Verification is the gating step. Full CI must pass in the sandbox before the PR opens. Coverage delta is computed; if negative beyond a threshold, the PR opens but gets labeled needs-review. The agent posts the rationale as the PR description plus an @agent thread the reviewer can ping for follow-ups.

Safety is scoped through two different GitHub surfaces: the App provides a short-lived installation token with workflows: read and narrow repo contents/PR scopes; branch protection (not app permissions) enforces "no direct writes to main" and "no force-push" — the app is never added to the bypass list. Path-scoped read-only access to .github/workflows is not a real GitHub App primitive, so the agent's allow-list on file edits has to enforce that at the worker. Budget ceilings per repo per day are enforced at the dispatcher (e.g., max 5 PRs per repo per day, $20 per PR).

Architecture

GitHub issue labeled `@agent fix` or PR comment
            |
            v
    GitHub App webhook -> AWS Lambda dispatcher
            |
            v
    ECS Fargate task (or GitHub Actions self-hosted runner)
       - pull repo
       - infer Dockerfile (language, package manager)
       - Daytona / E2B sandbox with target runtime
       - clone -> git worktree -> agent branch
            |
            v
    mini-swe-agent / SWE-agent v2 loop
       Claude Opus 4.7 or GPT-5.4-Codex
       tools: ripgrep, tree-sitter, read/edit, run_tests, git
            |
            v
    verify CI passes in-sandbox + coverage delta check
            |
            v (verified)
    git push + open PR via GitHub App
       PR body = rationale + diff summary + trace URL
       label: needs-review
            |
            v
    operator reviews; can @-mention agent for follow-ups

Stack

  • Trigger: GitHub App with fine-grained token; webhook receiver via Lambda or Fly.io
  • Worker: ECS Fargate task (or GitHub Actions self-hosted runner)
  • Sandbox: Daytona devcontainer or E2B sandbox per task
  • Agent loop: mini-swe-agent baseline or SWE-agent v2 over Claude Opus 4.7 / GPT-5.4-Codex
  • Retrieval: tree-sitter repo-map + ripgrep
  • Verification: full CI in-sandbox + coverage delta gate
  • Observability: Langfuse with per-PR trace archive linked from the PR body
  • Budget: per-repo daily dollar ceiling; max PRs per repo per day
cf-issue-to-pr

Build It

  1. GitHub App. Fine-grained installation token: issues read+write, pull_requests write, contents read+write, workflows read. Branch protection (the only surface that can do this) enforces "no direct push to main" and "no force-push"; the app is not in the bypass list. The worker enforces "no writes under .github/workflows" as an allow-list check on the proposed diff, since GitHub App permissions are not path-scoped.

  2. Webhook receiver. Lambda function accepts issue label / PR comment webhooks. Filters by label @agent fix this. Enqueues to SQS.

  3. Dispatcher. Pops tasks from SQS. Enforces per-repo per-day budget. Spins up an ECS Fargate task with the repo URL, issue body, and a fresh Daytona sandbox.

  4. Environment inference. Detect language (Python, Node, Go, Rust) and package manager (uv, pnpm, go mod, cargo). Generate a Dockerfile on the fly if one does not exist.

  5. Agent loop. mini-swe-agent or SWE-agent v2 with Claude Opus 4.7. Tools: ripgrep, tree-sitter repo-map, read_file, edit_file, run_tests, git. Hard limits: $20 cost, 30 min wall-clock, 30 agent turns.

  6. Verification. After the loop concludes, run the full test suite in-sandbox. Compute coverage delta via jacoco / coverage.py. If CI red: halt, do not open PR. If coverage drops more than 2%: open PR with needs-review label.

  7. PR posting. Push the agent branch. Open PR via GitHub API with: title, rationale, diff summary, trace URL, cost, turns.

  8. Credential hygiene. Worker runs with a short-lived GitHub App installation token. Logs are scrubbed for secrets before archival.

  9. Eval. 30 seeded internal issues of varying difficulty. Measure pass rate, PR quality (diff size, style, coverage), cost, latency. Compare with Cursor Background Agents and AWS Remote SWE Agents on the same issues.

Use It

# on github.com
  - user labels issue #842 with `@agent fix this`
  - PR #1903 appears 14 minutes later
  - body:
    > Fixed NPE in widget.dedupe() caused by null comparator entry.
    > Added regression test widget_test.go::TestDedupeNullComparator.
    > Coverage delta: +0.12%
    > Turns: 7  Cost: $1.80  Trace: langfuse:...
    > Label: needs-review

Ship It

outputs/skill-issue-to-pr.md is the deliverable. A GitHub App + async cloud worker that turns labeled issues into review-ready PRs with bounded cost and scoped credentials.

Weight Criterion How it is measured
25 Pass rate on 30 issues End-to-end success (CI green + coverage OK)
20 PR quality Diff size, coverage delta, style conformance
20 Cost and latency per resolved issue $ and wall-clock per PR
20 Safety Scoped token, per-repo budget, no force-push, credential hygiene
15 Operator UX Rationale comments, retry affordance, @-mention follow-up
100

Exercises

  1. Add a "fix flaky test" mode: the label @agent stabilize-flake TestX runs the test 50 times in-sandbox and proposes a minimal change that stabilizes it.

  2. Compare cost vs Cursor Background Agents on three shared issues. Report which tools win where.

  3. Implement a budget dashboard: per-repo per-day cost, per-user cost. Alert on anomaly.

  4. Build a "dry-run" mode that opens a draft PR without running CI, so reviewers can examine the plan cheap.

  5. Add a retention policy: PR branches older than 7 days without merge get deleted automatically.

Key Terms

Term What people say What it actually means
GitHub App "Scoped bot identity" App with fine-grained permissions + short-lived installation token
Async cloud agent "Background agent" Non-interactive worker that runs in a cloud sandbox, not a terminal
Environment inference "Dockerfile synthesis" Detect language + package manager, generate a Dockerfile if absent
Verification "CI-in-sandbox" Run the full test suite inside the worker before opening a PR
Coverage delta "Coverage preservation" Change in test coverage % from base to agent branch
Per-repo budget "Daily ceiling" Dollar and PR-count cap enforced at the dispatcher
Rationale "PR body explanation" Agent's summary of what changed and why; required in the PR body

Further Reading

Check yourself

Quick quiz

  1. How is the async cloud coding agent triggered in production?

    1. A GitHub webhook fired by an issue label or PR comment, dispatched to a sandbox worker
    2. A manual file upload
    3. A long-lived SSH session into a VM
    4. A scheduled cron job hitting the repo
    Show the answer

    A. A GitHub webhook fired by an issue label or PR comment, dispatched to a sandbox worker

  2. Why must the agent run with a short-lived GitHub App installation token rather than a personal access token?

    1. Personal access tokens cannot open PRs
    2. App tokens grant fine-grained, scoped, short-lived credentials suited for per-task workers
    3. Apps avoid GitHub rate limits entirely
    4. Apps require less code than PATs
    Show the answer

    B. App tokens grant fine-grained, scoped, short-lived credentials suited for per-task workers

  3. Which two protections prevent the agent from clobbering main?

    1. Disabling git on the worker
    2. Branch protection enforcing no direct push or force-push to main, with the app not on the bypass list
    3. Requiring a captcha per push
    4. Encrypting the repo before clone
    Show the answer

    B. Branch protection enforcing no direct push or force-push to main, with the app not on the bypass list

  4. Why is path-scoped protection for .github/workflows enforced at the worker rather than as a GitHub App permission?

    1. .github/workflows is a virtual path
    2. GitHub App permissions are not path-scoped, so the worker's allow-list check on the proposed diff is the right place to enforce it
    3. Path scoping is impossible in any form
    4. Workflows are stored outside the repo
    Show the answer

    B. GitHub App permissions are not path-scoped, so the worker's allow-list check on the proposed diff is the right place to enforce it

  5. What gating happens after the agent loop concludes, before opening a PR?

    1. The reviewer model rewrites the diff
    2. The PR opens immediately with no checks
    3. Full CI runs in the sandbox; coverage delta is computed; PRs with a regression beyond a threshold are labeled needs-review
    4. The repo is force-pushed to main
    Show the answer

    C. Full CI runs in the sandbox; coverage delta is computed; PRs with a regression beyond a threshold are labeled needs-review

  6. Which budget knobs does the dispatcher enforce per repo per day?

    1. Only a turn cap
    2. A dollar ceiling per PR plus a maximum number of PRs per repo per day
    3. Unlimited PRs and unlimited cost
    4. Only a wall-clock cap
    Show the answer

    B. A dollar ceiling per PR plus a maximum number of PRs per repo per day

  7. What goes into the PR body that the agent posts?

    1. Rationale, diff summary, trace URL, cost, and turn count so reviewers can audit and follow up
    2. Only the diff
    3. Nothing; the PR has an empty body
    4. A generated changelog only
    Show the answer

    A. Rationale, diff summary, trace URL, cost, and turn count so reviewers can audit and follow up

Questions people ask

Can an AI agent fix GitHub issues automatically?

For well-scoped issues, often yes. The agent reads the issue, sets up the project in a sandbox, writes a fix, runs tests and opens a pull request explaining the change. A developer still reviews and merges it. Vague or large issues usually need human help.

How do you stop a coding agent from leaking API keys?

Keep secrets out of the sandbox unless needed, give only short-lived and narrowly scoped tokens, block outbound network access by default, and scan outputs and commits for anything that looks like a key. Logs should mask secrets too.

Run the code Read it on GitHub

Next lessonCapstone 17 — Personal AI Tutor (Adaptive, Multimodal, with Memory)Previous: Capstone 15 — Constitutional Safety Harness + Red-Team Range

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes