How to Auto-Quarantine Flaky Cypress

The list is the pivot: detection writes it, the before hook reads it, and it lives in version control.

Define Quarantine Thresholds & Triggers #

Establish failure metrics before automating. A test should enter quarantine only after exceeding a statistical threshold — for example, 3 failures in 10 runs — never on a single failure, which produces false positives from a one-off network blip.

A statistical threshold keeps a single flaky network call from quarantining a healthy test.

Parse Cypress Results & Generate Quarantine List #

Cypress writes JSON to the path set by --reporter-options. A post-run script counts failed and passed attempts per test and appends genuinely unstable ones to a version-controlled quarantine.json, using atomic writes so parallel CI jobs do not clobber it.

Deduplicate on a stable `spec::title` id so the same flake is never added twice.

// scripts/quarantine-parser.js — count attempts and append unstable tests.
const fs = require('fs');
const results = JSON.parse(fs.readFileSync('cypress/results/output.json', 'utf8'));
const quarantine = fs.existsSync('cypress/quarantine.json')
  ? JSON.parse(fs.readFileSync('cypress/quarantine.json', 'utf8')) : [];

results.runs.forEach(run => {
  run.tests.forEach(test => {
    const failed = test.attempts.filter(a => a.state === 'failed').length;
    const passed = test.attempts.filter(a => a.state === 'passed').length;
    if (failed > 3 && passed < 7) { // statistical threshold, not a single failure
      const id = `${run.spec.relative}::${test.title.join(' > ')}`;
      if (!quarantine.includes(id)) quarantine.push(id); // dedupe on a stable id
    }
  });
});
fs.writeFileSync('cypress/quarantine.json', JSON.stringify(quarantine, null, 2));

Dynamically Skip Quarantined Tests in Cypress #

Read the list at runtime in a support file and skip matching tests in a beforeEach hook, so the pipeline stays green while the test’s logs remain available for debugging.

Logging the skip keeps quarantine visible rather than a silent coverage drop.

// cypress/support/e2e.ts — skip quarantined tests, but log every skip.
import { existsSync, readFileSync } from 'fs';
const QUARANTINE_PATH = 'cypress/quarantine.json';
const quarantineSet = existsSync(QUARANTINE_PATH)
  ? new Set<string>(JSON.parse(readFileSync(QUARANTINE_PATH, 'utf-8'))) : new Set<string>();

beforeEach(function () {
  const id = `${Cypress.spec.relative}::${this.currentTest?.fullTitle() ?? ''}`;
  if (quarantineSet.has(id)) {
    cy.log(`Quarantined: ${id}`); // visible, not a silent skip
    this.skip();
  }
});

Automate PR Checks & Notification Routing #

Wire the parser into CI, commit the updated list with [skip ci] to avoid loops, and require a reliability sign-off before re-enabling. Block merges when the quarantine rate breaches its SLO.

The `[skip ci]` marker on the commit prevents the quarantine update from looping.

# .github/workflows/cypress-ci.yml — parse and commit the quarantine list.
- name: Run Cypress Tests
  run: npx cypress run --reporter json --reporter-options "output=cypress/results/output.json"
- name: Update Quarantine List
  if: always()
  run: node scripts/quarantine-parser.js
- name: Commit Updated Quarantine
  if: always()
  run: |
    git diff --quiet cypress/quarantine.json || (
      git config user.name 'ci-bot'
      git add cypress/quarantine.json
      git commit -m 'chore: update quarantine list [skip ci]'  # [skip ci] breaks the loop
      git push origin HEAD
    )

Common Pitfalls #

Each red habit lets quarantine hide problems; the green fix keeps it honest and reversible.

Quarantining on single failures instead of statistical thresholds.
Failing to version-control the list, causing CI race conditions across runners.
Skipping without logging, leading to silent coverage degradation.
No automated unquarantine validation before re-enabling.

FAQ #

How do I safely unquarantine a Cypress test? Run it in isolation against staging for 10+ consecutive passes, then remove it from the list only after CI confirms stability.

Does auto-quarantine affect coverage metrics? Yes — skipped tests reduce execution coverage. Track the quarantine rate as a separate reliability metric with an SLA to restore coverage.

Can this integrate with Cypress Cloud? Yes. Use --record to capture flakiness metadata and cross-reference the Cloud API with your parser for centralized tracking.

Reliability Metrics #

Track hit rate and false positives so quarantine improves the pass rate without hiding bugs.

Quarantine hit rate (%).
Mean Time to Quarantine (MTTQ).
False-positive quarantine rate.
CI pass-rate delta (pre/post quarantine).
Test recovery SLA compliance.

Quarantining a Test, Not a Spec #

Cypress groups tests into specs that share a browser session, which makes the unit of quarantine an awkward decision. Excluding a whole spec because one test in it is unstable removes coverage that was working; excluding a single test from a spec whose tests depend on each other can break the ones that remain.

The dependency question is the one to answer first. A spec whose tests each set up their own state can have a single test excluded cleanly. A spec written as a sequence — one test creating something the next asserts on — cannot, because removing a step breaks the chain. That second shape is common in Cypress suites precisely because a spec feels like a session, and it is worth treating the discovery as a finding: a spec that cannot survive one test being skipped has an ordering dependency that will cause other problems later.

Where the tests are independent, excluding one is straightforward and the granularity is worth having, since it keeps the remaining coverage in the gating run. Where they are not, the pragmatic choice is to quarantine the spec, record in the entry that the reason is coupling rather than instability, and treat decoupling as the actual fix.

// Skip one test by tag when tests are independent; quarantine the spec when not.
// Trade-off: per-test granularity keeps more coverage gating and only works in
// specs whose tests do not depend on each other.
const quarantined = require('../quarantine.json').map((e) => e.test);
const maybe = (title) => (quarantined.includes(title) ? it.skip : it);

maybe('applies discount')('applies discount', () => { /* … */ });

Keeping Quarantined Specs Running #

The mechanism most teams reach for first is the runner’s skip facility, and it quietly removes the possibility of ever releasing the test. A skipped spec produces no results, so there is no evidence to graduate on and the only available exit is somebody’s impression that the fix worked — which, for a test that was failing a few percent of the time, is indistinguishable from luck.

The alternative costs a little runner time and preserves everything: run quarantined specs in a separate, non-blocking invocation. Tag them, exclude the tag from the gating run, and execute the tagged set in a job allowed to fail. Retries should be off in that job, because its purpose is to measure the true rate rather than to rescue it.

// cypress.config.js — a second, non-gating spec pattern
// Trade-off: extra minutes for specs that cannot fail the build; without them
// nothing can graduate on evidence and the quarantine list only grows.
module.exports = defineConfig({
  e2e: {
    excludeSpecPattern: process.env.QUARANTINE_RUN ? [] : ['cypress/e2e/**/*.quarantined.cy.js'],
    retries: { runMode: process.env.QUARANTINE_RUN ? 0 : 1, openMode: 0 },
  },
});

The naming convention matters more than it looks. Encoding quarantine in the filename makes the list visible in a directory listing and in every pull request that touches it, which is a considerably stronger social signal than a tag buried in a spec — and it makes the count trivial to compute for the ratio cap that keeps quarantine from becoming a package’s default state.

Exit Criteria and the Bounce Problem #

An automated quarantine system with a single threshold oscillates. A spec hovering near the line is quarantined, its rate improves because it is no longer competing in the contended gating run, it graduates, its rate worsens again, and back it goes — consuming a triage cycle each time and eroding confidence in the automation.

Separating the entry and exit thresholds fixes it. Quarantine on a rate above the budget over a rolling window with a minimum execution count; graduate only at a materially lower rate sustained over a longer period, plus a clean repetition run. The gap between the two should be wide enough that a spec bouncing between the states is a real signal rather than noise.

Two guards complete the arrangement. A relapse watch re-quarantines automatically if a graduated spec fails within a couple of weeks and records that relapse against the fix, since a high relapse rate is the clearest evidence that a symptom was treated rather than a cause. And the graduation itself should arrive as a reviewed change rather than a silent tag removal, which keeps a person at the point where the risk actually sits.

Every entry needs an expiry date regardless of the gate, because the gate only handles specs that get better. Without expiry, the non-blocking lane becomes a permanent parking space burning runner minutes on specs nobody intends to fix — which is deletion with extra steps and none of the honesty.

A quarantine entry should read like a commitment rather than a note: which spec, who owns it, why it was isolated, what the recorded failure rate was, and the date by which a decision is due. Entries written that way are still legible six weeks later, when the person picking it up was not the person who wrote it — which is the situation the list exists to serve.

Reviewing the quarantine list at a fixed cadence — fifteen minutes a week, with the owning teams present — is what keeps entries moving through it rather than accumulating. The review needs no preparation beyond the list itself, since each entry already carries its owner, its rate and its expiry date.