Auto-Quarantining Flaky Playwright Tests

Root cause #

A Playwright test becomes a quarantine candidate when retries keep classifying it as flaky run after run — the non-determinism is real and recurring, usually an auto-waiting timeout, a parallel-worker state collision, or a network response that lands outside the expected window. Letting that test stay in the blocking suite means every red run could be the flaky test rather than a genuine regression, which is exactly what erodes trust in CI.

The fix is structural, not a code patch on the test. Playwright lets you tag a test (@quarantine), grep on that tag, and assign tags to a dedicated project. By running quarantined tests in their own project you keep collecting their pass/fail data for the trend, but you exclude that project from the gate that blocks merges. The quarantine list itself can be a tag in source, a test.fixme annotation, or a generated grep file produced by your detection step.

Step-by-step fix #

1. Tag flaky tests #

Apply the @quarantine tag to tests your detection step flagged. Tags live in the title or the tag option and are greppable.

// checkout.spec.ts
import { test, expect } from '@playwright/test';

// The tag option marks this test without changing its title text.
test('applies discount code', { tag: '@quarantine' }, async ({ page }) => {
  await page.goto('/cart');
  await expect(page.getByTestId('total')).toHaveText('$90');
});
// Trade-off: tagging in source is explicit and reviewable but requires a
// commit; a generated grep file (step 3) avoids commits at higher risk
// of drift between the list and the code.

2. Add a non-blocking quarantine project #

Define two projects: the main one excludes the tag, the quarantine one includes only the tag. The quarantine project gets generous retries because its job is observation, not gating.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [['json', { outputFile: 'results.json' }]],
  projects: [
    {
      name: 'main',
      grepInvert: /@quarantine/, // blocking suite skips quarantined tests
    },
    {
      name: 'quarantine',
      grep: /@quarantine/,        // observe-only suite runs them in isolation
      retries: 3,
    },
  ],
});
// Trade-off: grepInvert keeps the gate fast and clean, but a tag typo
// silently leaves a flaky test in the blocking suite — lint the tags.

3. Generate the quarantine grep from detection output #

Instead of editing source, write the flaky set to a grep file your CI passes to --grep. This couples quarantine to your detection JSON.

// build-quarantine-grep.js
const fs = require('fs');
const flaky = require('./flaky-playwright.json'); // from the detection step
// Escape titles into an alternation for --grep.
const pattern = flaky.map((f) => f.title.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')).join('|');
fs.writeFileSync('quarantine.grep', pattern || '__none__');
// Trade-off: a generated list never drifts from detection data, but a
// huge alternation is slow to match — cap the list and quarantine in bulk.

4. Gate the merge in GitHub Actions #

Run the main project as the required check and the quarantine project with continue-on-error so its result is recorded but never blocks.

# .github/workflows/playwright.yml
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      # Required check: only the blocking project can fail the job.
      - name: Run main suite
        run: npx playwright test --project=main

      # Observe-only: failures here are recorded but do not block merge.
      - name: Run quarantine suite
        continue-on-error: true
        run: npx playwright test --project=quarantine

      - name: Annotate quarantine result
        if: always()
        run: echo "Quarantine job conclusion: ${{ job.status }}"

Pitfalls #

• A tag typo leaves a flaky test in the blocking suite. Mitigation: lint tags in CI and assert the quarantine project matched at least the expected count.
• Quarantine becomes a graveyard. Mitigation: enforce an exit policy — 10+ consecutive passes in the quarantine project before removing the tag.
• continue-on-error masks an infrastructure outage. Mitigation: still surface the quarantine conclusion in the job summary and alert on a spike.
• Coverage silently drops as tests move to quarantine. Mitigation: track quarantine count as its own metric and cap it with an SLO.
• Using test.fixme instead of a project: fixme skips entirely, so you lose the pass/fail data. Mitigation: prefer a non-blocking project so observation continues.

Reliability targets #

Frequently Asked Questions #

Q: Should I use test.fixme or a quarantine project? A: A non-blocking project. test.fixme skips the test entirely, so you stop gathering the data you need to know whether it is fixed; a separate project keeps running it without blocking the gate.

Q: How do I make only the main project a required check? A: Run the projects as separate steps (or jobs) and apply continue-on-error to the quarantine step, then mark only the main step’s job as a required status check in branch protection.

Q: How is this different from the Cypress approach? A: The concept matches but the mechanism differs — Cypress uses a runtime skip list. See how to auto-quarantine flaky Cypress tests for that pattern.

Keep Them Running, or Nothing Can Graduate #

The implementation choice made when a test is quarantined determines whether it can ever leave. Excluding it from the run entirely stops it blocking the pipeline and also stops it producing the results that would justify releasing it — so the exit depends on somebody’s impression that the fix worked, which for a test failing a few percent of the time is indistinguishable from luck.

Running quarantined tests in a separate, non-gating project preserves the evidence. The tag excludes them from the blocking project and selects them in a second one whose failures do not fail the build. Retries should be disabled there: the purpose is to measure the true rate, and a retry inflates the pass count with rescued failures.

// playwright.config.ts — two projects, one gating, one measuring.
// Trade-off: runner minutes spent on tests that cannot fail the build; without
// them the quarantine list only ever grows.
projects: [
  { name: 'blocking', grepInvert: /@quarantined/ },
  { name: 'quarantined', grep: /@quarantined/, retries: 0 },
],

The graduation criterion then has data to work with: a streak of consecutive passes across real pipeline runs, plus a clean repetition run, plus a minimum number of executions so a quiet fortnight cannot produce a streak by accident. Twenty consecutive passes alone are not enough — for a test that was failing at four percent, that happens by chance roughly half the time.

Tagging, Storage and Review #

Where the quarantine list lives decides whether it stays honest, and the convenient options are the weak ones.

A data file in the repository — entries with the test identifier, owner, reason, dates and affected consumers — makes every addition visible in review, keeps a history explaining why each entry exists, and gives an expiry check something to read. A build step that fails on a passed expiry date is what converts “we should look at that” into a decision.

Tags in the spec files are convenient for the runner and poor for governance: there is no single place to see the list, nowhere natural for the metadata, and no way to notice it growing. The hybrid works well — the file is the source of truth and a build step derives the tags from it.

Configuration in the CI system is the arrangement to avoid: invisible in review, no history, and changeable during a difficult week without leaving a trace, which is exactly when a quarantine list acquires entries that should never have been added.

The same reasoning applies to the budget threshold and any exemptions. Anything that decides what is allowed to fail belongs in the repository, where changing it is a reviewable act.

One Source of Truth for the List #

Deriving the tag from a committed data file rather than editing spec files directly keeps additions visible in review and gives the expiry check something to read. It also means unquarantining is a one-line change to data rather than an edit scattered across specs.

An automated quarantine should record the failure rate that triggered it. That number is what the graduation gate later compares against, and reconstructing it after the fact means digging through history that may already have been pruned.

An entry without an expiry date is a decision deferred indefinitely, which is why the date is the field worth enforcing in the build.

Related guides #

• Building Auto-Quarantine Workflows
• How to Auto-Quarantine Flaky Cypress Tests
• Surfacing Flakiness With Playwright Test Retries