Design Success Metrics Before the Result Exists
Measurement should answer a decision, not decorate a dashboard. Start with the goal, derive questions, then choose the smallest metrics that answer them.
Lesson 52 of 54Learn + Build~70 minutesPython (stdlib)
Before this: Lessons 47 and 51 of Phase 14
On this page
Learning Objectives
- Derive questions and metrics from an outcome goal.
- Define thresholds, windows, sources, and directions before observing results.
- Pair outcome metrics with guardrails and counter-metrics.
- Match evaluation evidence to the decision the build must support.
Goal, Question, Metric
Start with a goal:
Reduce time to identify the affected service without increasing unsafe actions.
Derive questions:
- How quickly is the correct service identified?
- How often is the identified service correct?
- Does diagnosis remain read-only?
- Does the workflow increase alert dismissal or operator workload?
Then choose metrics that operationalize those questions.
A Metric Needs a Contract
Every metric needs:
| Field | Example |
|---|---|
| Name | median_identification_seconds |
| Direction | at most |
| Threshold | 120 |
| Window | ten incident replays |
| Source | replay event log |
| Population | on-call engineers in the pilot |
| Kind | outcome or guardrail |
Without source and window, a number cannot be reproduced. Without a threshold, it cannot drive a decision.
Outcome, Guardrail, and Counter-Metric
- Outcome metric: did the desired state improve?
- Guardrail: did a fixed constraint remain true?
- Counter-metric: did the local improvement shift cost or harm elsewhere?
For an incident workflow, speed is not enough. Correctness, production writes, operator workload, and missed alerts protect against a fast but unsafe result.
Offline and Online Evidence
Offline replay is useful for repeatability and edge coverage. A bounded pilot is useful for real behavior, trust, and workflow effects. Neither substitutes for the other.
Use the cheapest evidence that can answer the current decision. Do not expose real users merely because the implementation is ready.
Decide Before You Measure
Write the pass, fail, and ambiguous paths before seeing results. Otherwise the team will move the threshold to protect the build.
Example:
- pass: correct service rate at least 0.9 and median time at most 120 seconds;
- fail: any production write or correct rate below 0.75;
- ambiguous: small improvement with wide variance, requiring a larger replay set.
Build It
The lab validates a measurement plan, evaluates inclusive thresholds, records missing values, and writes outputs/measurement-report.json.
python3 code/main.py
python3 -m unittest discover code/tests -vRemove the guardrail metric and observe why the plan becomes invalid even when the outcome metrics remain.
Exercises
- Derive three questions from one outcome goal.
- Add a counter-metric that catches cost shifted to another role.
- Define the source, population, and window for every metric.
- Write pass, fail, and ambiguous decisions before generating values.
- Identify one metric that is easy to collect but cannot change the decision. Remove it.
Further Reading
- Basili, Software Modeling and Measurement: The Goal/Question/Metric Paradigm, for deriving operational measurements from explicit goals.
- Basili, Caldiera, and Rombach, The Goal Question Metric Approach, for applying the method as a feedback and improvement system.
What You Keep
Keep outputs/measurement-report.json. It defines the evidence gate for the prototype, pilot, or production stage.
Check yourself
Quick quiz
What is the correct measurement order?
- Data, model, feature
- Metric, question, goal
- Dashboard, metric, goal
- Goal, question, metric
Show the answer
D. Goal, question, metric Goals define information needs, questions refine them, and metrics operationalize answers.
Why define thresholds before looking at the results?
- To avoid questions
- To reduce storage
- To prevent moving the decision rule to protect the build
- To guarantee a pass
Show the answer
C. To prevent moving the decision rule to protect the build Precommitted thresholds reduce post-result rationalization.
What does a guardrail metric protect?
- A fixed constraint or unacceptable harm
- Only speed
- A release name
- A visual style
Show the answer
A. A fixed constraint or unacceptable harm Guardrails ensure improvement does not violate safety or policy.
What is a counter-metric?
- An unmeasured opinion
- A measure that catches cost or harm shifted elsewhere
- A second copy of the outcome metric
- A model score
Show the answer
B. A measure that catches cost or harm shifted elsewhere Local gains can create downstream burden that a counter-metric exposes.
Why must a metric name its source and window?
- For branding
- To replace a threshold
- To avoid users
- For reproducibility and interpretation
Show the answer
D. For reproducibility and interpretation A number without provenance and time or sample boundaries cannot support a reliable decision.
When is offline replay insufficient?
- When the decision depends on real user behavior or workflow effects
- When tests are fast
- Never
- Always
Show the answer
A. When the decision depends on real user behavior or workflow effects Replay is repeatable but cannot fully reproduce trust, coordination, and live behavior.
Questions people ask
How do you measure the success of an AI project?
Start from the goal, then ask what would show it is met. Choose a few metrics with a target, time window and data source set in advance, and add guardrail metrics for things that must not get worse.
What is a guardrail metric?
A guardrail metric tracks something that should not get worse while you improve the main goal. For example, faster replies should not raise error rates or complaints. It stops a win on one number from hiding damage elsewhere.
Lessons from AI Engineering from Scratch by Rohit Ghumare and contributors, MIT license. Notes for owners by Lozo. License