Trust but Measure
Turns "this should be faster now" into a before number, an after number, and a delta.
When to reach for it
Any time performance is claimed, improved, or worried about — before optimising, and before believing that optimising worked.
What changes
- The claim becomes a named metric with a unit and a threshold, both fixed before the measurement is taken.
- You get a before number, an after number, the delta, and the number of runs with their spread.
- Improvements smaller than run-to-run noise are reported as no change rather than as a win.
- The procedure is written down precisely enough — build mode, cache state, data, machine — that someone else can repeat it and get the same answer.
Pairs with
Trust but Measure
"Should be faster" is a prediction. Produce the number.
1. Name the metric
Turn the claim into something with a unit and a place to read it: p95 of a specific handler in milliseconds, bytes in the entry bundle, renders per keystroke, queries per page load, peak resident memory. "Faster" is not a metric; "faster to interactive on the list page" is.
2. Set the threshold before measuring
Write the number that would count as success now, while it is still possible to be wrong about it. A threshold chosen after seeing the result is a description of the result.
3. Measure before
No baseline, no claim. Record the machine, the build mode, the dataset, the cache state, and any throttling. Take the baseline immediately before the change, not from memory of last week.
4. Change one thing
One change, then measure again with exactly the same procedure. Two changes measured together produce a number that cannot be attributed, and one of them is often making things worse.
5. Run it enough times to see the noise
At least five runs each side. Report the median and the spread. A difference smaller than the run-to-run variation is not a difference, however good the story around it is.
6. Report so it can be repeated
Before, after, delta, unit, number of runs, spread, and the exact procedure. Someone else should be able to reproduce the number without asking a question.
Rules
- Do not report a percentage without the absolute numbers. "40% faster" hides whether that is 200ms or 2ms.
- Do not benchmark a development build and claim a production result. They are different programs.
- Do not compare a warm run against a cold one, or a populated cache against an empty one. State the cache state on both sides.
- Do not accept a complexity argument as a measurement. Big-O predicts behavior at sizes you may never reach; the profiler describes the size you have.
- Do not optimise a metric no user experiences. A faster function inside a request dominated by a network call has improved nothing.
- If the number does not move, say so plainly and revert the change. A neutral result honestly reported is worth more than a win narrated into existence.