1. CI-First Detection Architecture #
Reliable detection must occur at the pipeline layer, not locally. Configure parallel retry logic with deterministic seed tracking to separate true regressions from environmental noise. Integrate Automated Flaky Test Detection Tools directly into your CI runner to capture execution variance, timing drift, and resource contention at scale. Set threshold-based triggers that only activate quarantine when flakiness exceeds defined SLOs.
2. Production-Ready Quarantine Workflows #
Quarantine must be automated, auditable, and reversible. Use dynamic test exclusion lists and metadata tagging to route unstable suites to isolated execution pools. Follow Building Auto-Quarantine Workflows to implement GitOps-managed quarantine states with automated re-validation cycles and precise ownership routing.
3. Measurable Stability & Observability #
Quarantine without telemetry creates hidden technical debt. Track flake rates, mean time to quarantine (MTTQ), and pass-rate recovery curves across sprint cycles. Implement Historical Flakiness Tracking & Analytics to correlate test instability with dependency bumps, infrastructure changes, and test authorship patterns. Surface these KPIs via Reliability Dashboards for QA Teams to align engineering and QA on data-driven stability targets.
4. Shift-Left Integration & PR Gates #
Prevent flaky code from merging by embedding stability gates early in the development lifecycle. Configure progressive PR checks that differentiate between true failures and environmental noise. Enforce deterministic test patterns that ensure external services are properly mocked and async handling is standardized before code reaches main.
5. Retry Budgets & Honest Accounting #
Retries are the most effective flakiness treatment available and the most effective way to stop measuring flakiness at all. Which of those describes a given pipeline depends entirely on whether the rescues are counted.
The arithmetic explains why this matters more at scale than intuition suggests. A test failing independently 10% of the time fails a run 1% of the time with one retry and 0.1% with two — genuinely useful. The same arithmetic means a test failing half the time still passes 87% of runs at two retries, so a badly broken test looks merely unlucky. And in a thousand-test suite where each test fails 0.1% of the time, a run is red 63% of the time without retries and essentially never with one. Retries therefore feel indispensable at scale while being able to absorb an unbounded amount of real breakage.
The resolution is not to remove them but to bound them at one and to treat the retry count as the measured quantity. A suite whose retry count is flat is stable; one whose retry count is rising is degrading, however green the pipeline looks. Because the pass rate has been engineered to be uninformative, the retry count has to carry the signal instead — which means extracting rescued results from the runner’s structured output on every run and gating on a rate rather than watching a dashboard. CI Retry Strategies & Budgets covers the extraction, the budget calibration and the per-level policy that keeps deterministic failures from being retried at all.
// Rescued failures are data, not plumbing — extract them every run.
// Trade-off: parsing the report couples this to a format; a custom reporter is
// more robust and more code.
const flaky = specs.filter((s) => s.results.length > 1 && s.results.at(-1).status === 'passed');
recordFlaky(flaky); // counted against the budgetCalibration decides whether the budget survives. Set it from the measured baseline rather than from an aspiration — a threshold below the current rate fails every build on day one and is removed within a week — then ratchet it down monthly as the achieved rate falls. Scope matters as much as the number: gate pull requests on the tests the change actually touched, with zero tolerance, and gate trunk on the aggregate so a breach blocks the release rather than an individual’s merge.
6. Triage, Ownership & Exit Criteria #
Detection tells you which tests are unstable and quarantine stops them blocking the pipeline. Neither answers the question that decides whether the backlog shrinks: who is fixing this one, and by when. Teams with excellent tooling and no ownership end up with a quarantine list that only ever grows.
Three mechanisms close that gap. Classification comes first, because the same symptom has three different owners: a connection error belongs to whoever owns the pipeline, a wrong asserted value that passes on retry is a probable product race for the feature team, and a timeout is a test-quality item for the spec’s owner. Routing everything to one queue guarantees that the signal drowns.
Automatic assignment comes second, derived from CODEOWNERS rather than from a spreadsheet that goes stale within a quarter, with an explicit fallback team so unmatched paths cannot accumulate unassigned. The value here is latency: a flake assigned within minutes is fixed while the change that caused it is still small and recent, while one assigned six weeks later requires archaeology.
An expiry date comes third, and it is the mechanism that makes the other two matter. A quarantined test with no date is a deleted test that still appears in coverage reports — the worst combination, since it carries the risk of no coverage plus the cost of maintenance and periodic re-triage. At expiry there are exactly three legitimate outcomes: fixed and restored, deleted with the coverage gap recorded, or extended by the owner with a stated reason. Silence is not one, and a build that fails on an expired entry is what enforces that. Flaky Test Triage & Ownership covers the routing, the cost-based prioritisation and the review cadence.
Prioritisation deserves its own note, because sorting by failure count sends effort to the wrong place. The worst flaky test is the one that costs the most developer time, which combines how often it fails, how many pipelines it blocks and how long a rerun takes. A test failing three times per hundred runs on a twenty-minute release pipeline routinely outranks one failing twenty times in a forty-second unit suite.
Production Configuration Examples #
Jest CI Retry Configuration #
// jest.config.js
// jest-circus (the default runner since Jest 27) supports retryTimes via jest.retryTimes().
// There is no built-in flakyTestConfig key — quarantine logic lives in custom reporters.
module.exports = {
// Trade-off: retryTimes masks underlying race conditions.
// Use strictly for CI isolation; disable for local dev.
testRunner: 'jest-circus/runner',
reporters: ['default'],
};// In a test setup file (e.g., jest.setup.js), enable per-test retries:
// jest.retryTimes(3, { logErrorsBeforeRetry: true });
// Call this inside describe/beforeEach for targeted retry, not globally.Playwright GitHub Actions Quarantine Step #
# .github/workflows/quarantine.yml
- name: Run Quarantined Tests
run: npx playwright test --grep @quarantined --reporter=html
env:
# Trade-off: Running quarantined suites in parallel increases CI compute cost.
# Offset by scheduling during off-peak hours to maintain budget.
CI: trueCypress CI YAML Isolation #
# .github/workflows/cypress-quarantine.yml
- name: Cypress Quarantine Execution
run: npx cypress run --spec "cypress/e2e/quarantine/**/*"
env:
# Trade-off: Headless mode reduces browser overhead but may hide
# rendering-specific flakiness. Enable video for post-mortem analysis.
CYPRESS_VIDEO: true
CYPRESS_RETRIES: 2The lifecycle a flaky test should follow #
Everything on this page is one loop, and its integrity depends on each stage handing something concrete to the next. Where teams stall is almost always a specific missing handover rather than a missing tool.
Detected. A rate crosses a threshold over a rolling window — not a single failure, which is noise at any non-zero rate. The detection needs a denominator, which means storing every result rather than only failures.
Classified. The error signature assigns the failure to a class: wait, infrastructure, assertion, unknown. This is where most pipelines lose the plot, because an unclassified aggregate cannot be routed and cannot be acted on by anyone in particular.
Owned. A named team, derived automatically, within minutes. Latency here translates directly into cost, since context evaporates within days.
Quarantined — but still running. This is the handover that most implementations get wrong. Quarantine implemented as a skip produces no results, so there is no evidence on which the test could ever be released; it can only be restored on somebody’s optimism, which is how a fixed test returns to the blocking suite and fails again a week later. Quarantine has to mean the test executes in a lane whose failures do not block anyone.
Graduated on evidence. A stability gate — a run of consecutive passes plus a clean repetition run — replaces the judgement call. Twenty consecutive passes happen by chance 44% of the time for a test that was failing at 4%, so the streak alone is not enough; pairing it with a hundred concentrated repetitions is what makes the gate decisive. Unquarantining Tests with a Stability Gate covers the counters and the relapse handling.
Or deleted, openly. When the coverage is duplicated elsewhere and stabilising is expensive, deletion is a legitimate outcome — provided the owner makes the call, the review happens like any other change, and the residual gap is written down with a condition under which it should be revisited.
A loop missing any one of these stages degrades in a predictable way. Without classification the queue is undifferentiated; without ownership nothing is prioritised; without a running quarantine lane nothing graduates; without expiry the list only grows.
Preventing inflow, not just processing it #
Everything described so far handles flakiness that already exists. The control that decides whether the backlog shrinks is the one that stops new instability arriving, and it is the cheapest intervention available.
Stress-running changed specs before merge turns a rare event into a measurement at the moment it is introduced. Fifty repetitions with retries disabled takes a couple of minutes and reliably catches anything failing above roughly five percent — which is most newly written flakiness, since fresh tests tend to fail more often than seasoned ones that have already had their sharpest edges filed off. The author is present, holds the context, and can fix it in minutes rather than having a colleague discover it three weeks later.
Concurrency in that stress run matters as much as the count. Running a test a hundred times sequentially finds internal races; running copies of it concurrently finds shared-resource assumptions — a fixed port, a hard-coded record id, a shared temporary file — that never appear when only one copy exists. Those collisions are among the most common causes of flakiness in parallel suites and are invisible until two copies collide, which in normal CI may take weeks.
The nightly counterpart repeats every test a smaller number of times across the whole suite, gates on a rate rather than on any single failure, and feeds the results into the same history store as everything else. Stress-Running Tests to Surface Flakes covers the repetition counts that match a given detection threshold, and why choosing a round number instead produces false confidence.
Common Pitfalls #
- Over-relying on local retries instead of CI-level detection
- Quarantining tests without automated re-validation windows
- Ignoring environmental variance (CPU throttling, network latency) as root causes
- Blocking PRs without clear flakiness SLOs or exception workflows
- Failing to tag quarantined tests with ownership and remediation deadlines
- Treating quarantine as permanent deletion instead of a temporary isolation state
Splitting the metric so someone can own it #
A single blended flakiness number is the most common reason accountability stalls. When it mixes runner failures, container start-up problems and registry timeouts together with genuine test instability, no team can be held to it: the spec owners cannot fix the runner, and the platform team is judged on tests it did not write. The number becomes something everyone reports and nobody moves.
Split it at the point of collection, by error signature, and publish both halves side by side. The test-flake rate drives the budget and belongs to the teams that own the specs. The infrastructure-failure rate belongs to whoever owns the pipeline and deserves its own threshold and escalation path. Neither is harder to compute than the blended figure, and each is something a specific group can act on.
The same decomposition applies across a shared repository. One repository-wide budget means teams with clean suites are blocked by teams with dirty ones, while the dirty ones face no specific pressure because the number is diluted. Per-package budgets — strictest for shared libraries, since their tests protect the most consumers — put the pressure where the instability is, and the roll-up remains available for reporting. Quarantine Policies in Monorepos covers the sign-off and ratio controls that shared ownership additionally requires.
Reliability Metrics #
- Flake Rate (%)
- Mean Time to Quarantine (MTTQ)
- Quarantine Re-validation Pass Rate
- Pipeline Stability Index (PSI)
- Test Execution Variance (ms)
- Flake-to-Fix Ratio
Making the history queryable #
Every metric on this page is an aggregate over runs, which means none of them can be computed from a directory of per-run artifacts without a script per question. That friction is why most teams’ flakiness data is technically retained and practically unused: someone computes a number once, pastes it into a document, and the team reasons from a stale snapshot for months.
A single database file removes the friction. Two tables are enough — one row per run with its metadata, one row per test result — and a suite producing a thousand results per run across twenty runs a day generates a few million rows a year, comfortably within an embedded database on a file that lives in an artifact store. There is no service to operate, and the analytical difference is the difference between having data and being able to use it.
The metadata columns are what make the interesting questions askable. Rate per test needs only the results table; correlating a rate change with a runner image bump, a worker-count change, a branch or an hour of the day needs the run table joined to it. Recording image digest, core count, worker count, branch and start time costs nothing at write time and is impossible to reconstruct later.
Three standing queries cover most needs: rate per test over thirty days sorted worst-first, which produces the ranked worklist; rate by runner image, which separates environment drift from test decay; and rate by hour of day, which reveals contention. Storing Test History in SQLite for Flake Analysis covers the schema and the retention policy, and Correlating Flakiness with CI Runner Load works through the contention analysis in detail.
One storage rule is easy to get wrong and expensive to discover late: store passes as well as failures. Without the denominator, “twelve failures” describes a rarely-run test and a constantly-run one identically, and no rate can be computed from it at all.
FAQ #
What is the acceptable flakiness threshold for production CI? Industry practice targets <1% flake rate for main branch pipelines. Quarantine triggers should activate at 2–3 consecutive non-deterministic failures, or when execution variance exceeds 15% of the baseline duration.
How do we prevent quarantined tests from becoming permanent technical debt? Enforce automated re-validation windows (e.g., 72 hours), assign remediation owners via metadata tags, and block new feature merges if the quarantine backlog exceeds defined SLOs.
Should flaky tests block pull requests? Yes, but only when integrated with deterministic PR checks that differentiate between true regressions and environmental noise. Use progressive gating rather than hard blocks to maintain developer velocity.
Choosing a detection architecture #
The framework a team already uses shapes what detection looks like, and the differences are structural rather than cosmetic.
Playwright reports a rescued failure as a distinct flaky status in its own results, so the rate is available directly from the runner’s structured output with no extra instrumentation. Its retries are per test, its sharding is deterministic given the same test list, and its trace artifacts can be retained on a failure even when a later attempt passes — which is what keeps a rescued failure diagnosable weeks later.
Cypress reports retries differently and carries a stronger notion of per-test isolation, with the page, cookies and storage cleared between tests by default. That default is a significant reliability asset and the most commonly disabled one, since turning it off makes a spec measurably faster and converts it into a single long stateful session.
Jest and Vitest sit in a different regime entirely. Their failures are almost always deterministic — a real defect or an order dependence — so retries there hide bugs rather than absorbing environmental noise, and the useful instrumentation is a shuffled run with a recorded seed rather than a retry count. Cypress vs Playwright Detection Architecture compares the browser-level options in detail, and Automated Flaky Test Detection Tools covers the per-runner configuration.
What should be identical across all of them is the downstream pipeline: the same store, the same classification, the same budget, the same routing. A team running three runners with three separate flakiness reports has three numbers nobody compares and no aggregate anyone owns.
What good looks like #
The observable markers of a team that has this working are unglamorous and easy to check.
Every rescued failure is recorded with its artifacts, classified by error signature and attributed to a named owner within minutes. The quarantine list is short — single digits for a suite of a few thousand tests — and every entry has an expiry date in the future. The retry budget is a committed file with a visible history of ratchets rather than a number buried in pipeline settings. Nobody argues about whether a failure is a test problem or a product problem, because the classifier proposes an answer and the error signature settles it.
The most telling marker involves production rather than CI. When someone reports an intermittent bug, the first action is to search the rescued-failure history for the same area — and often the defect is already there, recorded, with a trace attached, having been retried away for weeks. That search taking two minutes instead of two days is the compound return on recording, classifying and routing every rescue rather than discarding them.
None of this needs unusual tooling. It needs every detected flake to leave the pipeline with a class, an owner and a date attached, and something to fail when one of those is missing.
Starting From Nothing #
For a team with none of this, the order of adoption matters more than the tooling choice, and the first two steps are unusually cheap relative to what they unlock.
Record every result with its commit, branch, runner image and worker count, and run one weekly query: rate per test over thirty days, worst twenty first. That is an afternoon of work and it converts an unbounded complaint about flakiness into a ranked list of named tests — the input every other mechanism here needs, and the thing most teams are missing when they say they have a flakiness problem.
Then stop the inflow, by stress-running changed specs before merge. This is the highest-leverage control available, because it stops the backlog growing while the rest of the work proceeds, and it puts the fix in front of the person best placed to make it.
Everything else — budgets, quarantine lanes, ownership routing, stability gates — is worth adding in that order, and each is much easier once the first two exist. A team that starts with a quarantine mechanism and no history has no basis for its thresholds; a team that starts with a dashboard and no ownership has numbers nobody moves.