Phase 17 · Lesson 21 of 29

A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem

Traditional A/B testing was not built for non-deterministic LLMs. The critical distinction: evals answer "can the model do the job?" A/B tests answer "do users care?" Both are required; shipping on vibe checks is over. What to test in 2026: prompt engineering (wording), model selection (GPT-4 vs GPT-3.5 vs OSS; accuracy vs cost vs latency), generation parameters (temperature, top-p). Real cases: a chatbot reward-model variant delivered +70% conversation length and +30% retention; Nextdoor AI subject-line experiments delivered +1% CTR after reward-function refinement; Khan Academy Khanmigo iterated on a latency-vs-math-accuracy axis. Platform split: Statsig (acquired by OpenAI for $1.1B in September 2025) — sequential testing, CUPED, all-in-one. GrowthBook — open-source, warehouse-native, Bayesian + Frequentist + Sequential engines, CUPED, SRM checks, Benjamini-Hochberg + Bonferroni corrections. You pick based on warehouse-SQL preference and whether "acquired by OpenAI" matters to your organization.

Lesson 21 of 29Learn~60 minutesPython (stdlib, toy sequential test simulator)

Before this: Lesson 13 of Phase 17 (Observability), lesson 20 of Phase 17 (Progressive Deployment)

On this page

Learning Objectives

  • Distinguish evals ("can the model do the job") from A/B tests ("do users care").
  • Enumerate three testable axes (prompt, model, parameters) and pick the metric for each.
  • Explain CUPED, sequential testing, and Benjamini-Hochberg multiple-comparison corrections.
  • Pick Statsig or GrowthBook based on warehouse-SQL posture and corporate acquisition stance.

The Problem

You hand-tuned a system prompt. It feels better. You ship it. Conversion changes by noise. You blame the metric. Or you shipped a new model and conversion didn't move — did the model degrade or was the change too small to detect? You don't know, because you shipped without an A/B.

Evals answer whether the model can do a task on a labeled set. They do not answer whether users prefer the output. Only a controlled online experiment answers that, and only if the experiment has enough power, controls for non-determinism, and corrects for multiple comparisons.

The Concept

Evals vs A/B tests

Evals — offline, labeled set, judge (rubric or LLM-as-judge or human). Answer: "Is the output correct / helpful / safe on this fixed distribution?"

A/B test — online, live users, randomized. Answer: "Does the new variant move the user-level metric that matters?"

Both required. Evals catch regressions before exposure; A/B confirms product impact after.

What to test

  1. Prompt engineering — wording, system-prompt structure, examples. Metric: task success, user retention, cost/request.
  2. Model selection — GPT-4 vs GPT-3.5-Turbo vs Llama-OSS. Metric: accuracy (task) + cost/request + latency P99. Multi-objective.
  3. Generation parameters — temperature, top-p, max_tokens. Metric: task-specific (output diversity vs determinism).

CUPED — variance reduction

Controlled-experiments Using Pre-Experiment Data. Regress out pre-period variance before comparing post-period. Typical variance reduction: 30-70%. Effective sample size goes up for free.

Implementation: both Statsig and GrowthBook implement.

Sequential testing

Classical A/B assumes fixed sample size. Sequential tests ("peek-and-decide") control false-positive rate under repeated looks. Always-valid sequential procedures (mSPRT, Howard's confidence sequences) let you stop early on clear winners.

Multiple-comparison corrections

Running 20 A/B tests at 95% confidence produces one false positive by chance. Bonferroni correction tightens α per-test; Benjamini-Hochberg controls false-discovery rate. GrowthBook implements both.

SRM — sample ratio mismatch

Assignment hash randomizes users to variants. If 50/50 split delivers 47/53, something is broken — SRM check flags it. Both platforms implement.

Statsig vs GrowthBook

Statsig:

  • Acquired by OpenAI for $1.1B (September 2025). Hosted, SaaS.
  • Sequential testing, CUPED, held-out populations.
  • All-in-one: feature flags + experimentation + observability.
  • Best fit: team already wants a bundled product, doesn't care about OpenAI ownership.

GrowthBook:

  • Open-source (MIT); warehouse-native (reads from Snowflake/BigQuery/Redshift directly).
  • Multiple engines: Bayesian, Frequentist, Sequential.
  • CUPED, SRM, Bonferroni, BH corrections.
  • Self-host or managed cloud.
  • Best fit: warehouse-SQL shop, data team controls the metric layer, wants OSS.

Non-determinism complicates power

Same prompt produces varying outputs. Traditional power calculations assume IID observations. With LLM non-determinism, effective sample size is lower than nominal. Multiply required sample size by ~1.3-1.5x as a safety margin.

Real case outcomes

  • Chatbot reward model variant: +70% conversation length, +30% retention.
  • Nextdoor subject lines: +1% CTR after reward-function refinement.
  • Khan Academy Khanmigo: iterative latency-vs-math-accuracy trade.

The anti-pattern: shipping on vibes

Every senior engineer can name a feature that was shipped because "it feels better" with no A/B. Most of them regressed product metrics the team didn't notice for months. A/B is the forcing function.

Numbers you should remember

  • Statsig acquired by OpenAI: $1.1B, September 2025.
  • GrowthBook: open-source MIT; Bayesian + Frequentist + Sequential.
  • CUPED variance reduction: 30-70%.
  • LLM non-determinism → +30-50% sample-size buffer.
mx-sequential-test

Use It

code/main.py simulates a sequential A/B test with fixed and sequential boundaries. Shows how sequential lets you stop early.

Ship It

This lesson produces outputs/skill-ab-plan.md. Given feature change, workload, baseline, picks platform, gates, sample size.

Exercises

  1. Run code/main.py. For an expected 5% lift with baseline 3% conversion, what sample size to 80% power?
  2. Pick Statsig or GrowthBook for a healthcare-regulated on-prem customer.
  3. Design an A/B that tests GPT-4 vs GPT-3.5 on cost-per-resolved-ticket. What's the primary metric, guardrail metric, secondary?
  4. Your canary passes but A/B shows -1.2% conversion. Do you ship? Write the escalation criteria.
  5. Apply CUPED with a pre-period covariate that explains 60% of the post-period variance (squared correlation 0.6). Compute the effective-sample-size boost.

Key Terms

Term What people say What it actually means
Eval "offline test" Labeled-set evaluation of model capability
A/B test "experiment" Live randomized comparison on users
CUPED "variance reduction" Pre-period regression to reduce variance
Sequential test "peek-ok test" Always-valid procedure allowing early stop
Multiple comparison "the family error" Running many tests inflates false positives
Bonferroni "tight correction" Divide α by number of tests
Benjamini-Hochberg "BH FDR" False-discovery-rate control, less conservative
SRM "bad split" Sample ratio mismatch; assignment bug
Statsig "OpenAI owned" Commercial all-in-one, acquired 2025
GrowthBook "the OSS one" MIT warehouse-native platform
mSPRT "sequential probability ratio test" Classical sequential procedure

Further Reading

Check yourself

Quick quiz

  1. What is the precise distinction between evals and A/B tests in the lesson?

    1. Only A/B tests are required
    2. Evals answer "can the model do the job?" on a labeled set; A/B tests answer "do users care?" with live randomized traffic
    3. Evals are user-facing; A/B tests are offline
    4. They are interchangeable
    Show the answer

    B. Evals answer "can the model do the job?" on a labeled set; A/B tests answer "do users care?" with live randomized traffic

  2. What does CUPED do for an experiment?

    1. Disables multiple-comparison correction
    2. Regresses out pre-period variance before comparing post-period, typically reducing variance 30-70% and boosting effective sample size
    3. Increases sample size by hiring more users
    4. Replaces sequential testing entirely
    Show the answer

    B. Regresses out pre-period variance before comparing post-period, typically reducing variance 30-70% and boosting effective sample size

  3. Why do you need multiple-comparison corrections (Bonferroni or Benjamini-Hochberg) when running many A/Bs?

    1. They speed up experiments
    2. They are only needed for sequential tests
    3. They are required by GDPR
    4. Running 20 tests at 95% confidence produces one false positive by chance; corrections control family-wise error or false discovery rate
    Show the answer

    D. Running 20 tests at 95% confidence produces one false positive by chance; corrections control family-wise error or false discovery rate

  4. What does SRM (sample ratio mismatch) detect?

    1. Slow tokenizer performance
    2. PII leakage
    3. An assignment-hash bug producing a delivered split that diverges from the intended (e.g. 47/53 when targeting 50/50)
    4. Memory leaks in the experiment platform
    Show the answer

    C. An assignment-hash bug producing a delivered split that diverges from the intended (e.g. 47/53 when targeting 50/50)

  5. Why does LLM non-determinism require buffering sample size?

    1. Non-determinism reduces required samples
    2. It violates IID assumptions; effective sample size drops, so multiply required size by roughly 1.3-1.5x as a safety margin
    3. It is irrelevant to power calculations
    4. It only matters offline
    Show the answer

    B. It violates IID assumptions; effective sample size drops, so multiply required size by roughly 1.3-1.5x as a safety margin

  6. How does the lesson contrast Statsig and GrowthBook?

    1. GrowthBook is closed source
    2. Identical feature sets
    3. Statsig is warehouse-native only
    4. Statsig is all-in-one SaaS (acquired by OpenAI Sept 2025, $1.1B); GrowthBook is open-source MIT, warehouse-native, with Bayesian/Frequentist/Sequential engines and CUPED/SRM/BH
    Show the answer

    D. Statsig is all-in-one SaaS (acquired by OpenAI Sept 2025, $1.1B); GrowthBook is open-source MIT, warehouse-native, with Bayesian/Frequentist/Sequential engines and CUPED/SRM/BH

Questions people ask

How do you A/B test an LLM feature?

Split users at random into groups, give each group a different prompt or model, and pick one main metric in advance, such as task completion or conversion. Run long enough to get a reliable result, since AI answers vary from one call to the next.

What is the difference between evals and A/B tests?

Evals check model output against test cases before launch, asking if the model can do the task. A/B tests run with live users after launch, asking if the change improves real behavior. Teams need both.

Run the code Read it on GitHub

Next lessonLoad Testing LLM APIs — Why k6 and Locust LiePrevious: Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs

Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License

Next step

Rather have it built for you?

Reading is optional. Tell me what's stuck and I'll build it into the business you already run.

Tell me what's stuck
  • A real conversation, not a pitch
  • You keep your accounts
  • Nothing sends without your yes