Building Auto-Quarantine Workflows

Pipeline Architecture & Trigger Logic #

Auto-quarantine relies on a continuous feedback loop between test execution, result parsing, and state management. The workflow begins when a CI job completes and routes test artifacts to a centralized processor. By leveraging Automated Flaky Test Detection Tools, teams can parse JUnit/XML reports, extract failure signatures, and calculate flakiness scores in real time. The quarantine trigger executes via a dedicated CI stage that updates a shared configuration file or database before the next pipeline run.

Data Flow & State Management #

Test results are serialized into a structured format containing test ID, failure count, pass count, and execution environment. A lightweight script evaluates these metrics against predefined thresholds. If the flakiness ratio exceeds the limit, the test is flagged for isolation. State persistence is handled via Git commits, artifact storage, or a lightweight KV store, ensuring idempotent pipeline behavior.

Framework-Specific Implementation Patterns #

Cypress and Playwright handle test isolation differently, requiring tailored quarantine logic. For Cypress, a pre-run plugin can skip quarantined tests by reading an external manifest. Playwright leverages its built-in test.skip() and test.fixme() APIs, which can be injected programmatically via a test hook or custom reporter. For a complete Cypress implementation, refer to How to Auto-Quarantine Flaky Cypress Tests.

Dynamic Test Filtering #

Instead of hardcoding skips, use a manifest-driven approach. Generate a quarantine-manifest.json during the analysis phase. In the pre-test hook, read the manifest and apply framework-specific skip directives. This keeps the quarantine logic decoupled from test code and enables rapid rollback.

Threshold Configuration & Historical Baselines #

Effective quarantine requires statistically sound thresholds. Relying on single-run failures causes over-quarantining, while lenient thresholds allow flaky tests to persist. Integrate Historical Flakiness Tracking & Analytics to establish rolling baselines. Configure a sliding window where a test is quarantined if it fails intermittently across multiple distinct commits or environments.

Adaptive Thresholds #

Implement exponential backoff for re-evaluation. Once quarantined, a test enters a cooldown period. After the cooldown, it runs in a dedicated reliability suite. If it passes consistently, it graduates back to the main pipeline. If it fails again, it remains isolated until a code fix is merged.

Governance, Recovery & Team Alignment #

Automation must be paired with clear ownership. Auto-quarantine generates actionable tickets, assigns them to the owning squad, and tracks mean time to resolution. While automation handles volume, human review remains critical for complex race conditions or infrastructure drift.

Quarantine Lifecycle Management #

Define SLAs for quarantined tests. Use CI badges and messaging webhooks to notify stakeholders. Implement a mandatory PR check that prevents merging until quarantined tests are either fixed or explicitly approved for extended isolation.

Production-Ready Implementation Examples #

Cypress Dynamic Quarantine Hook #

// File: cypress/support/quarantine.ts
import { readFileSync, existsSync } from 'fs';

const MANIFEST_PATH = './quarantine-manifest.json';

// Cypress supports skipping via before() hooks in support files.
// Calling test.skip() from Cypress.on('test:before:run') is not supported;
// use before() at the spec level or the task API instead.
if (existsSync(MANIFEST_PATH)) {
  const manifest = JSON.parse(readFileSync(MANIFEST_PATH, 'utf-8'));
  const quarantinedTests = new Set<string>(manifest.quarantined);

  before(function () {
    if (quarantinedTests.has(this.currentTest?.title ?? '')) {
      this.skip();
    }
  });
}

CI Pipeline Impact: Executes synchronously before each spec run. Adds <50ms overhead per test suite.
Trade-offs: Requires manifest synchronization across parallel CI runners. Use a centralized artifact store or Git LFS for distributed execution.

Playwright Reporter-Based Quarantine #

// File: tests/reporters/quarantine-reporter.ts
import type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';
import { readFileSync, writeFileSync, existsSync } from 'fs';

export default class QuarantineReporter implements Reporter {
  private manifestPath = 'quarantine-manifest.json';

  onTestEnd(test: TestCase, result: TestResult) {
    // Playwright marks a test 'flaky' when it fails then passes on retry.
    // retries must be >= 1 in playwright.config.ts for this status to appear.
    if (result.status === 'flaky') {
      const manifest = existsSync(this.manifestPath)
        ? JSON.parse(readFileSync(this.manifestPath, 'utf-8'))
        : { quarantined: [] };

      if (!manifest.quarantined.includes(test.title)) {
        manifest.quarantined.push(test.title);
        writeFileSync(this.manifestPath, JSON.stringify(manifest, null, 2));
      }
    }
  }
}

CI Pipeline Impact: Runs post-execution. Zero impact on test runtime. Manifest updates trigger a follow-up commit or artifact upload.
Trade-offs: flaky status requires retries >= 1 in playwright.config.ts. Reporter only captures final state, not intermediate retry failures.

GitHub Actions Auto-Quarantine Step #

# File: .github/workflows/ci-quarantine.yml
- name: Analyze & Quarantine Flaky Tests
  id: analyze
  run: |
    python scripts/analyze_flakiness.py \
      --report junit-results.xml \
      --threshold 0.15 \
      --output quarantine-manifest.json
  continue-on-error: true

- name: Commit Quarantine Manifest
  if: steps.analyze.outputs.changed == 'true'
  run: |
    git config user.name 'ci-bot'
    git config user.email '[email protected]'
    git add quarantine-manifest.json
    git commit -m 'chore: auto-quarantine flaky tests [skip ci]'
    git push origin ${{ github.head_ref }}

CI Pipeline Impact: Adds ~15–30s to pipeline duration. [skip ci] prevents infinite commit loops.
Trade-offs: Direct branch commits bypass PR review gates. Enforce branch protection rules that allow only service accounts to push quarantine manifests.

Quarantine Must Not Mean Skip #

The single most consequential implementation choice in this whole area is what “quarantined” does mechanically, and the intuitive option is the wrong one.

Implementing quarantine as a skip produces no results for the quarantined test. It stops blocking the pipeline, which is the immediate goal, and it also stops generating the only evidence on which the test could ever be released. The exit from quarantine then depends on somebody’s judgement — usually “I changed the wait and it passed once” — which is statistically meaningless for a test that was failing a few percent of the time, and which is why a fixed test so often returns to the blocking suite and fails again within a fortnight.

Implementing quarantine as a non-blocking lane keeps the test executing and reporting while its failures gate nothing. That preserves the data, which makes an evidence-based exit possible: a streak of consecutive passes across real pipeline runs, plus a concentrated repetition run, together give a defensible answer to “is it actually fixed”. Retries should be disabled in that lane, since the point is to measure the true rate rather than to rescue it.

The cost is runner minutes spent on tests that cannot fail the build, and that cost is what buys the ability to ever release them. A team unwilling to pay it is choosing, in effect, to delete those tests slowly while continuing to list them in coverage reports.

// Quarantined tests still run — they simply cannot block.
// Trade-off: extra runner minutes for non-gating tests; without them nothing
// can graduate on evidence and the list only grows.
projects: [
  { name: 'blocking', grepInvert: /@quarantined/ },
  { name: 'quarantined', grep: /@quarantined/, retries: 0 },
],

There is a second-order benefit worth noting: a quarantined test that keeps running will occasionally catch a real regression, which a skipped one cannot. That has an awkward consequence — a failing quarantined test that nobody looks at — and the answer is the expiry date, not switching the lane off.

Thresholds, Hysteresis and Bouncing #

An automated quarantine mechanism with a single threshold oscillates. A test hovering near the line gets quarantined, its rate improves because it is no longer running in the contended blocking suite, it graduates, its rate worsens again, and it returns — consuming a triage cycle each time and teaching everyone that the automation is unreliable.

The fix is the same one used in any control system: separate the thresholds for entering and leaving. Quarantine at a rate above the budget over a rolling window with a minimum execution count; graduate only at a materially lower rate sustained over a longer period plus a clean repetition run. That gap — hysteresis — is what stops the oscillation, and it should be wide enough that a test bouncing between the two states is a genuine signal rather than noise.

Two other guards matter in practice. A relapse watch re-quarantines automatically if a graduated test fails within a couple of weeks, and records that relapse against the fix, since a high relapse rate is the clearest evidence that symptoms are being treated rather than causes. And a rate cap on how many tests may be quarantined at once — expressed as a proportion of the suite — prevents the mechanism from becoming a route to a green, meaningless pipeline.

Automation should also stop short of the last step. Proposing the graduation as a reviewed change, rather than silently removing the tag, keeps a human decision point at the place where the risk actually sits, and it takes about a minute. Unquarantining Tests with a Stability Gate covers the counters, and Quarantine Policies in Monorepos covers what changes when several teams share the repository.

Common Pitfalls & Mitigation Strategies #

1. Over-quarantining stable tests due to environment-specific failures (network timeouts, third-party API rate limits). Mitigation: Filter out infrastructure-level errors via regex classification before applying flakiness thresholds.
2. Failing to implement a graduation or re-evaluation mechanism, causing permanent test debt accumulation. Mitigation: Enforce a mandatory cooldown_runs counter that triggers re-inclusion after N successful executions.
3. Hardcoding skip logic directly in test files instead of using external manifests or dynamic hooks. Mitigation: Decouple quarantine state from source code using runtime configuration or CI environment variables.
4. Ignoring root-cause analysis and treating quarantine as a permanent fix rather than a temporary isolation state. Mitigation: Auto-link quarantined tests to Jira/Linear tickets with a 72-hour SLA for triage.
5. Running quarantined tests in parallel with main suites without resource isolation, causing CI bottlenecks. Mitigation: Route quarantined tests to a dedicated, low-priority runner pool with extended timeouts.

The Coverage Debt Quarantine Creates #

Quarantining a test is borrowing: the pipeline unblocks now, and the coverage that test provided is gone until someone repays it. Treating it as a neutral administrative action is how a quarantine list reaches forty entries that nobody can account for.

Making the debt visible costs little. Each quarantine entry should record what the test verified, so a reader can tell whether the gap is a duplicate of lower-level coverage or the only check on a critical path. Where several teams consume the code, the entry should also list who else is affected, since quarantining a shared library’s test spends other teams’ safety margin without asking them.

The expiry date is what converts the debt from indefinite to bounded, and its enforcement has to be mechanical: a build step that fails when an entry has passed its date leaves exactly three outcomes — fixed, deleted with the gap recorded, or extended with a stated reason — and removes silence as an option. Teams that rely on a recurring meeting to review the list find that the meeting is the first thing cancelled in a busy week.

Two aggregate numbers keep the whole mechanism honest. Inflow versus outflow — tests entering quarantine per week against tests leaving — predicts the state of the suite months ahead and diverges long before the list becomes unmanageable. Median quarantine age shows whether expiry dates are being acted on or serially extended. Both are cheap to compute from the same data the quarantine file already contains, and neither requires anyone to remember to look.

Reliability Metrics & KPIs #

Automating Proposal, Not Decision #

There is a line worth drawing deliberately in any automated quarantine system: automation should propose, and a person should decide.

Automatic quarantine is safe to fully automate, because the cost of a false positive is small — a test moves to a non-blocking lane for a week and someone reviews it. Delay here is expensive, since an unstable test blocks everyone until it is quarantined, so an automated action within minutes is worth far more than a correct decision within a day.

Automatic graduation is different, because the cost of a false positive is a test returning to the blocking suite and failing again — consuming a triage cycle and eroding trust in the mechanism. Proposing the graduation as a reviewed change keeps a human at the point where the risk sits, and it costs about a minute.

Automatic deletion should not exist. Removing coverage is a decision with consequences no counter can weigh, and it needs an owner who can say what the gap is and why it is acceptable.

The same gradient applies to notifications: alert automatically, assign automatically, and let people choose what to do. Systems that automate the decision as well tend to be switched off after the first time they do something surprising, taking the useful automation with them.

Frequently Asked Questions #

How do I prevent auto-quarantine from masking real bugs? Auto-quarantine should only trigger on intermittent failures with a consistent pass/fail pattern across identical commits. Combine it with deterministic failure classification and require a human review step before permanent isolation.

What is the recommended flakiness threshold for triggering quarantine? Start with a 15–20% failure rate over a 30-day rolling window. Adjust based on suite maturity and CI frequency. Use statistical confidence intervals rather than raw counts to avoid noise.

Can auto-quarantine workflows run in monorepos with multiple test runners? Yes. Implement a centralized quarantine service that aggregates results from Cypress, Playwright, and unit test frameworks. Use a unified manifest format and route framework-specific skip logic via pre-run hooks.

Where the Quarantine List Should Live #

The quarantine list is configuration with unusually high consequences, and where it lives determines whether it stays honest.

A data file in the repository — a JSON or YAML list of entries with owner, reason, dates and affected consumers — is the arrangement that works. Changes appear in review, so adding an entry is visible and adding a dozen is conspicuous. The history explains why each entry exists long after the context has faded. And an expiry check can run as an ordinary build step, reading the file and failing when a date has passed.

Annotations in the test files themselves — a tag on the test — are convenient for the runner and poor for governance: there is no single place to see the list, no natural home for the metadata, and no way to notice that it has grown. A hybrid works well: the file is the source of truth, and a build step derives the tags from it.

Configuration in the CI system is the arrangement to avoid entirely. It is invisible in review, has no history anyone reads, and can be changed during a difficult week without leaving a trace — which is exactly when a quarantine list acquires entries that should never have been added.

The same reasoning applies to the budget threshold and the exemption list. Anything that decides what may fail belongs in the repository, where changing it is a reviewable act rather than an administrative one.

Related guides #

• Automated Flaky Test Detection Tools
• Historical Flakiness Tracking & Analytics
• How to Auto-Quarantine Flaky Cypress Tests
• Auto-Quarantining Flaky Playwright Tests