CI-first knowledge base Reliability Engineering

Article · Flaky Test Detection & Quarantine Engineering

Writing a Flaky Test Triage Runbook

Most flaky-test investigations restart from zero. Someone opens the failure, forms a hypothesis, tries a longer timeout, and moves on — and the next person to meet the same test repeats the process. A runbook replaces that with an ordered sequence of cheap, decisive checks, so triage takes twenty minutes and produces a classification rather than a guess. This guide gives the procedure that supports the routing described in Flaky Test Triage & Ownership.

12 sections URL: /flaky-test-detection-quarantine-engineering/flaky-test-triage-and-ownership/writing-a-flaky-test-triage-runbook/
Triage as an ordered sequence of cheap checks Five checks in increasing cost: solo re-run, repetition, seed replay, environment comparison, then deep investigation. 1. re-run alone — 30 s — separates leakage from everything else 2. repeat ×100 — 5 min — measures the real rate 3. replay the seed — 5 min — confirms order dependence 4. compare environment fingerprints — 10 min — finds drift 5. read the trace and the code — hours — only if 1–4 were inconclusive the ordering matters: each step is cheaper than the next and eliminates a whole class of cause
Four cheap mechanical checks resolve most cases; the expensive step is reserved for what survives them.

Root cause #

Triage is hard not because the failures are subtle but because the search space is large. A flaky test could be leaking state, racing internally, depending on order, depending on the machine, or reporting a genuine intermittent bug — five causes with overlapping symptoms and completely different fixes. Faced with that, people reach for the cheapest available action, which is usually adding a wait, and that action has a high false-success rate: the test stops failing for a fortnight and returns.

A runbook helps because the five causes are separable by mechanical checks that require no insight. Running the test alone separates leakage from the rest in one execution. Repeating it a hundred times measures the rate, which distinguishes an every-run race from a rare environmental blip. Replaying the recorded seed confirms or eliminates order dependence. Comparing environment fingerprints between a passing and a failing run finds drift. None of these needs understanding of the test — they need a procedure and about twenty minutes.

The second thing a runbook fixes is inconsistency between people. Without one, the quality of a triage depends on who picked it up, so the same test gets a different diagnosis each time it recurs and the record accumulates contradictions. With one, the output is a classification drawn from a fixed set, which is what makes the routing in the previous guide meaningful.

Step-by-step fix #

1. Re-run the test alone #

The single most informative execution available, and the cheapest.

# Trade-off: 30 seconds, and it eliminates or confirms an entire class of cause
# before any thinking is required.
npx playwright test path/to/spec.ts -g "the failing test title" --repeat-each=1

Green alone and red in the suite means state leakage — go to Test Isolation & State Leakage and bisect. Red alone means the cause is inside the test or the product, and the next checks apply.

2. Measure the actual rate by repetition #

“Flaky” is not a rate. Measure it, because the number decides what kind of problem this is.

# Trade-off: five minutes of compute, and it converts an anecdote into a
# number you can act on and re-measure after a fix.
npx playwright test path/to/spec.ts -g "title" --repeat-each=100 --workers=4 \
  --reporter=line | tee repeat.log
grep -c "✘\|failed" repeat.log

A rate above roughly 5% is usually an internal race and reproduces locally. Below 1% it is more likely environmental, and the next two checks are the productive ones.

3. Replay the recorded seed and worker count #

If the suite shuffles, the seed is in the log. Replaying it with the same worker count reproduces the exact ordering that failed.

# Trade-off: requires the seed to have been logged, which is why the shuffle
# step prints it — an unlogged seed makes this check impossible.
npx jest --randomize --seed="$SEED_FROM_CI" --maxWorkers=2

Reproduction here confirms order dependence, and the bisection procedure in Eliminating Test Order Dependence in Jest names the polluter.

Decision table from the first three checks Solo result, repetition rate and seed replay together identify leakage, an internal race or an environmental cause. observation conclusion green alone, red in suite state leakage — bisect for the polluting test fails ≥ 5 in 100 alone internal race — in the test's waits or the product 0 in 100 locally environmental — compare fingerprints, check load
Three observations partition the search space; everything after this point is a targeted investigation rather than a hunt.

4. Compare the environment between a pass and a failure #

If it will not reproduce locally at any rate, the machine is part of the cause. Diff the recorded fingerprints.

# Trade-off: requires the fingerprint artifact to exist, which is the argument
# for emitting one on every run rather than adding it during an incident.
diff <(jq -S . passing-run/env-fingerprint.json) \
     <(jq -S . failing-run/env-fingerprint.json)

Any difference in browser build, Node version, core count, zone or image digest is a candidate cause, and the routing goes to the pipeline owner rather than the spec owner — the reasoning in CI Environment & Browser Drift.

5. Only then read the trace #

By this point you know which of five causes you are looking at, so the trace is being read for a specific question rather than scanned for anything unusual.

npx playwright show-trace test-results/*/trace.zip

Look at whether the element was in view, whether it was still moving, what the network log shows immediately before the failing action, and whether the assertion’s actual value was wrong or merely late. That distinction — wrong versus late — is the one that separates a probable product bug from a wait problem.

6. Record the outcome in a fixed shape #

The runbook’s output should be structured, so the next person meeting this test starts from your conclusion rather than from scratch.

## Triage: checkout › applies discount
- Solo re-run: **passed** (10/10)
- Repetition in suite: **6 failures / 100**
- Seed replay: **reproduced** at seed 20260802, workers=2
- Environment diff: none
- **Classification:** order dependence — polluted by `cart.spec.ts` (writes `globalThis.__CART__`)
- **Owner:** @checkout · **Action:** add teardown in cart.spec.ts · **Expires:** 2026-08-30

Pitfalls #

  • Starting with the trace. Hours spent reading before knowing which question to ask. Mitigation: run the four mechanical checks first.
  • Skipping the solo re-run. The cheapest, most decisive check is the one people skip. Mitigation: make it step one, always.
  • Calling a test flaky without a rate. No baseline to verify a fix against. Mitigation: measure by repetition before and after.
  • Adding a wait as the first fix. High false-success rate; the test returns. Mitigation: fix only after classification.
  • Unstructured triage notes. The next person repeats everything. Mitigation: a fixed template with the four check results.
  • No seed in the CI log. Order dependence becomes unreproducible. Mitigation: print the seed on every shuffled run.
  • Triaging without a time box. One test consumes a day. Mitigation: box it at twenty minutes; if unresolved, quarantine with a date and move on.
Wrong value versus late value An assertion that saw a wrong value indicates a probable product race; one that saw the value arrive late indicates a wait problem. expected 42, received 41 the app computed a different answer probable product race → feature team expected 42, received nothing the value had not arrived yet wait problem → spec owner
One line of the failure message routes the item to a different team — which is why the runbook records it explicitly.

Reliability targets #

Metric Target Notes
Median triage time < 20 minutes Four mechanical checks, time-boxed
Triages producing a classification 100% Including “unresolved, quarantined with a date”
Triage notes following the template 100% So the next person starts from the conclusion
Fixes verified by repetition 100% Same measurement before and after
Reopen rate after a declared fix < 10% Higher means symptoms are being treated
Triage scorecard Targets for triage time, classification rate, verification and reopen rate. < 20 minmedian triage 100%classified 100%fixes verified < 10%reopen rate
Verifying a fix by re-measuring the rate is what keeps the reopen rate low; declaring victory after one green run does not.

Frequently Asked Questions #

Q: How long should someone spend before giving up and quarantining? A: Twenty minutes for the mechanical checks, and up to an hour beyond that if they produced a clear direction. Past that, quarantine with an owner and a date and schedule it properly — an open-ended investigation squeezed between other work is how a test stays flaky for a quarter while consuming an hour a week.

Q: The repetition run passes a hundred times but it fails in CI weekly. What next? A: Repeat under CI-like constraints rather than on an idle laptop: the same container, the same core and memory limits, the same worker count, with a load generator if the runner is shared. Most “will not reproduce locally” cases are compute headroom, and constraining the machine reproduces them at a measurable rate.

Q: Should the runbook be automated? A: The first two checks can be, and it is worth doing: a workflow that takes a test name and reports “passed 10/10 alone, 6/100 in suite” removes the friction that stops people triaging at all. The judgement steps — reading the trace, deciding wrong versus late — stay human, and are much faster once the mechanical results are in hand.

Q: How do I verify a fix actually worked? A: Re-measure with the same repetition count that established the baseline, and require the same number twice — once locally and once in CI. A single green run after a change proves almost nothing when the original rate was 4%, since roughly 96% of runs were green before the fix. This is the step teams skip most often, and skipping it is why the reopen rate is the metric that exposes shallow fixes.

Q: Who should own the runbook itself? A: Whoever owns the testing tooling, and it should be revised whenever a triage does not fit it. A runbook that has not changed in a year is either perfect or unused, and the second is far more likely — each investigation that goes off-script is telling you about a check the sequence is missing.

Q: What if two of the checks give contradictory answers? A: Trust the cheaper one and look harder at the expensive one. A test that passes alone but also fails at a measurable rate under repetition in isolation has two contributing causes — leakage that raises the rate, plus an internal race that produces the residual — which is common in tests that both share a fixture and await several things at once. Fix the leakage first, then re-measure; the second cause is often much smaller than it looked.