CI-first knowledge base Reliability Engineering

Article · Flaky Test Detection & Quarantine Engineering

Unquarantining Tests with a Stability Gate

Getting a test into quarantine is automated in most mature pipelines. Getting it out is almost always manual, optimistic and wrong: someone changes a wait, the test passes once, and it goes straight back into the blocking suite — where it fails again a week later and consumes another triage cycle. A stability gate replaces that judgement call with evidence. This guide completes the loop opened by Building Auto-Quarantine Workflows.

13 sections URL: /flaky-test-detection-quarantine-engineering/building-auto-quarantine-workflows/unquarantining-tests-with-a-stability-gate/
The quarantine lifecycle with a stability gate A quarantined test runs in a non-blocking lane, accumulates consecutive passes, and only rejoins the blocking suite after meeting a threshold. quarantinednon-blocking lane still runningresults recorded, not gating stability gateN consecutive passes required back in the blocking suite any failurecounter resets to zero a quarantined test that stops running is a deleted test — the non-blocking lane is what makes the gate possible
Quarantine must mean "runs but does not block", not "skipped" — otherwise there is no evidence to unquarantine on.

Root cause #

The statistics of a rare failure make single-run verification worthless. A test that failed 4% of the time before a change passes on the first attempt afterwards with probability 0.96, whether or not the change fixed anything. So “I fixed it and it passed” carries almost no information, and the test returns to the blocking suite with its failure rate essentially unmeasured.

What a stability gate does is set a required number of consecutive passes such that the probability of a still-broken test sneaking through is acceptably small. For a test with an underlying 4% failure rate, twenty consecutive passes happen by chance with probability 0.96²⁰ ≈ 0.44, which is still too generous; fifty gets it to about 13%, and combining consecutive passes with a repetition run tightens it much faster than raising the count alone.

The second, more common failure is structural: quarantine implemented as test.skip. A skipped test produces no results, so there is nothing to accumulate and the only way out is someone’s judgement — which returns you to single-run verification. Quarantine has to mean the test still executes and reports, in a lane whose failures do not block anyone. That is a small implementation difference with a large consequence: it is the difference between a quarantine you can exit on evidence and one you can only exit on hope.

Step-by-step fix #

1. Run quarantined tests in a non-blocking lane #

Keep them executing. A separate project or job carries their results into the record without gating the pipeline.

// playwright.config.ts
// Trade-off: running quarantined tests costs runner minutes for tests that
// cannot fail the build; that cost is what buys the evidence to release them.
projects: [
  { name: 'blocking', grepInvert: /@quarantined/ },
  { name: 'quarantined', grep: /@quarantined/, retries: 0 },   // no retries: measure honestly
],
- name: Quarantined suite (non-blocking)
  continue-on-error: true          # results recorded, pipeline unaffected
  run: npx playwright test --project=quarantined --reporter=json > quarantined.json

Retries are deliberately off here. The point of the lane is to measure the true rate, and a retry would inflate the pass count with rescued failures.

2. Track consecutive passes per test #

The gate needs a counter that survives across runs, incremented on a pass and reset to zero on any failure.

// scripts/stability.js
// Trade-off: consecutive passes is a simple, explainable rule; a Bayesian
// estimate of the failure rate is more informative and much harder to argue about.
const state = loadState();               // { [testId]: { streak, lastFailure } }

for (const result of quarantinedResults) {
  const s = (state[result.id] ??= { streak: 0, lastFailure: null });
  if (result.status === 'passed') s.streak += 1;
  else { s.streak = 0; s.lastFailure = runId; }   // any failure resets
}

saveState(state);

3. Require both a streak and a repetition run #

Consecutive passes across ordinary runs prove the test survives real conditions; a concentrated repetition run proves the rate is low. Requiring both closes the gap that either leaves alone.

// Trade-off: two conditions is stricter and slower to satisfy, and it stops a
// low-frequency test from graduating on a lucky fortnight of few executions.
const READY = (s, id) =>
  s.streak >= 20 &&                          // survived 20 real pipeline runs
  repetitionRate(id) === 0 &&                // 0 failures in 100 repetitions
  daysSince(s.lastFailure) >= 7;             // and not recently unstable

const graduating = Object.entries(state).filter(([id, s]) => READY(s, id));
Why one green run proves almost nothing A test failing four percent of the time passes a single run 96 percent of the time and twenty consecutive runs 44 percent of the time. 1 green run 96% chance a broken test passes 20 consecutive 44% still not conclusive 20 + 100 repetitions < 1% acceptable evidence
Figures for a test with a 4% underlying failure rate: the repetition run is what makes the gate decisive.

4. Graduate automatically, with a pull request #

Automation proposes; a human merges. Opening a pull request that removes the tag keeps the change reviewable and attributes it.

# Trade-off: an automated pull request adds a small amount of repository noise
# and keeps a human decision point where the risk actually is.
for id in $GRADUATING; do
  node scripts/remove-quarantine-tag.js "$id"
done
gh pr create --title "Unquarantine the stabilised test(s)" \
  --body "Each passed 20 consecutive runs and 100/100 repetitions with no failures."

5. Re-quarantine immediately on relapse, and record it #

A test that fails within a short window of graduating goes straight back, and the relapse is recorded against the fix. A high relapse rate is the clearest evidence that fixes are treating symptoms.

// Trade-off: fast re-quarantine avoids a rescued-failure cycle and can bounce
// a test that hit a genuine infrastructure blip — check the class before acting.
if (daysSinceGraduation(id) <= 14 && classify(failure) !== 'infrastructure') {
  requarantine(id, { reason: 'relapse', previousFix: lastFixCommit(id) });
}

6. Expire the quarantine independently of the gate #

The gate handles tests that get better. Tests that do not get better need the expiry from Flaky Test Triage & Ownership, or the non-blocking lane becomes a permanent parking space consuming runner minutes for tests nobody will ever fix.

Pitfalls #

  • Quarantine implemented as skip. No results, so no evidence to graduate on. Mitigation: run in a non-blocking lane.
  • Unquarantining after one green run. Statistically meaningless. Mitigation: require a streak plus a repetition run.
  • Retries enabled in the quarantine lane. Rescued failures inflate the pass count. Mitigation: zero retries there.
  • A streak that survives a failure. The counter must reset, or it measures nothing. Mitigation: reset to zero on any failure.
  • No relapse handling. A graduated test flakes and consumes a fresh triage cycle. Mitigation: automatic re-quarantine within a watch window.
  • No expiry on the lane. Permanently quarantined tests accumulate. Mitigation: expiry dates independent of the gate.
  • Graduating during a quiet period. A test that ran three times in a fortnight can hit a streak by chance. Mitigation: require a minimum number of executions, not just consecutive passes.
Skip versus non-blocking lane A skipped test produces no data; a non-blocking test produces the results the stability gate needs. test.skip no executions, no results exit requires a guess non-blocking lane runs, reports, never gates exit is earned on evidence
The implementation choice made at quarantine time determines whether the test can ever leave on merit.

Reliability targets #

Metric Target Notes
Quarantined tests still executing 100% Non-blocking lane, not skipped
Consecutive passes required ≥ 20 Plus a clean repetition run
Relapse rate within 14 days < 10% Higher means symptoms are being fixed
Retries in the quarantine lane 0 Measure the true rate
Tests graduating per month > 0 A gate nothing passes is a parking space
Stability gate scorecard Targets for executing quarantined tests, required streak, relapse rate and graduations. 100%still executing 20 + 100runs + repetitions < 10%relapse rate > 0graduations / month
A gate that nothing ever passes is as broken as one that lets everything through — watch the graduation rate.

Frequently Asked Questions #

Q: Twenty consecutive passes seems excessive. Is it? A: It is roughly right for a test that was failing a few percent of the time and too lenient for one that was failing at ten. Scale the requirement to the recorded rate: the higher the rate before quarantine, the more evidence needed to believe it is gone. Pairing the streak with a repetition run is what keeps the numbers manageable, because a hundred concentrated repetitions gather more evidence in five minutes than a fortnight of pipeline runs.

Q: Should the quarantine lane run on every pipeline? A: On trunk builds, yes; on pull requests it is usually wasted time, since a quarantined test’s result there tells you nothing about the change. Running it on trunk gives a steady, comparable stream of executions, which is exactly what a streak counter needs.

Q: What if a quarantined test starts failing for a completely new reason? A: Reset the streak and re-triage rather than counting it as continued instability. A different error signature means a different cause, and the runbook in Writing a Flaky Test Triage Runbook applies afresh. Treating it as more of the same is how a test acquires two overlapping problems and never leaves.

Q: Does the graduating pull request need a human reviewer? A: Yes, briefly. The gate provides the evidence; a person confirms that the fix in the linked commit is plausible and that nothing else about the test changed. It takes a minute, and it is the step that prevents an automated system from returning a test to the blocking suite on a technicality.

Scaling the Gate to the Original Rate #

A single graduation threshold applied to every test is either too strict for a mildly unstable one or too lenient for a badly broken one. Scaling the requirement to the rate recorded before quarantine fixes that with no extra machinery.

A test that was failing at one percent needs relatively little evidence to believe it is fixed; one that was failing at fifteen needs a great deal, because chance alone produces long green streaks at that rate far less often — but the repetition run is what makes the difference decisive in either case. Storing the pre-quarantine rate in the quarantine entry gives the gate the input it needs, and it costs one field.

Requiring a minimum number of executions alongside the streak prevents a quiet fortnight from producing a graduation by accident — a test that ran three times and passed three times has not demonstrated anything.