CI-first knowledge base Reliability Engineering

Article · Flaky Test Detection & Quarantine Engineering

Surfacing Flakiness With Playwright Test Retries

Playwright already tracks retries for you and labels a test flaky when it fails then passes within the same run, so the work is not adding retries but extracting that signal from the report. This guide is part of Automated Flaky Test Detection Tools and walks through reading testInfo.retry, parsing the JSON reporter, and pulling out the exact set of tests Playwright already knows are unstable.

13 sections URL: /flaky-test-detection-quarantine-engineering/automated-flaky-test-detection-tools/surfacing-flakiness-with-playwright-retries/
Playwright retry status flow A test result moves through attempts and is assigned one of three outcomes: passed, flaky, or failed, with the JSON reporter capturing each. retry 0 first attempt fail retry 1 testInfo.retry=1 status: flaky passed on retry status: failed all attempts fail JSON reporter
Playwright assigns a `flaky` status automatically; the JSON reporter is where you read it back out.

Root cause #

When retries is greater than zero, Playwright re-runs a failed test in a fresh worker context. If a later attempt passes, the test’s outcome is recorded as flaky rather than passed or failed. That status is the product of genuine non-determinism — typically an auto-waiting timeout that occasionally fires, a network response that arrives outside the expected window, or shared backend state colliding across parallel workers.

The reason teams miss this signal is that a run with only flaky tests still exits zero, so CI is green and nobody looks closer. Playwright is doing the hard part of classification for you; the gap is purely in surfacing. Because each retry exposes testInfo.retry, you can also attach per-attempt diagnostics and then mine the JSON report for the complete flaky set after the run.

Pass-on-retry is the signal A first-attempt fail that passes on retry is non-deterministic — record it rather than swallow it. attempt 1 fails retry passes record as flakynot green
Keep the retry to stay unblocked, but emit a record so the flakiness never vanishes.

Step-by-step fix #

1. Enable retries in config #

Set retries so Playwright can classify outcomes. Keep the count low on CI and zero locally so developers see failures immediately.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Retries only on CI; local runs stay strict so flakiness is visible.
  retries: process.env.CI ? 2 : 0,
  reporter: [['list'], ['json', { outputFile: 'results.json' }]],
});
// Trade-off: 2 retries keeps CI unblocked but can mask a 50%-failing
// test; rely on the JSON parse below to catch recurrence, not the exit code.

2. Tag the attempt with testInfo.retry #

Inside tests or a fixture, use testInfo.retry to record that an attempt was a retry. This lets you collect richer artifacts only when it matters.

// fixtures.ts
import { test as base } from '@playwright/test';

export const test = base.extend({
  page: async ({ page }, use, testInfo) => {
    // testInfo.retry is 0 on the first attempt, >0 on retries.
    if (testInfo.retry > 0) {
      console.warn(`Retry ${testInfo.retry} for "${testInfo.title}"`);
    }
    await use(page);
  },
});
// Trade-off: capturing traces on every retry aids debugging but adds
// CI storage cost; gate trace capture on testInfo.retry > 0.

3. Parse the JSON report for the flaky set #

The JSON reporter writes per-spec results. A test whose status is flaky is exactly what you want; you do not need to recompute it from attempts.

// extract-flaky.js
const fs = require('fs');
const report = JSON.parse(fs.readFileSync('results.json', 'utf8'));
const flaky = [];

function walk(suite, path = []) {
  for (const s of suite.suites ?? []) walk(s, [...path, s.title]);
  for (const spec of suite.specs ?? []) {
    for (const t of spec.tests) {
      // Playwright sets status 'flaky' when a retry passed after a failure.
      if (t.status === 'flaky') {
        flaky.push({ file: spec.file, title: [...path, spec.title].join(' > ') });
      }
    }
  }
}
report.suites.forEach((s) => walk(s));
fs.writeFileSync('flaky-playwright.json', JSON.stringify(flaky, null, 2));
console.warn(`Surfaced ${flaky.length} flaky test(s).`);
// Trade-off: reading the report is free, but merge per-shard JSON files
// before counting so sharded runs do not undercount the flaky set.

4. Feed the flaky set forward #

Use the extracted list to open issues, gate a budget, or hand it to quarantine. The same JSON shape works as the input to a quarantine workflow.

// budget.js
const flaky = require('./flaky-playwright.json');
const BUDGET = 5;
if (flaky.length > BUDGET) {
  console.error(`Flaky budget exceeded: ${flaky.length} > ${BUDGET}`);
  process.exit(1); // block merge until the regression is triaged
}
// Trade-off: a strict budget catches drift early but can stall PRs;
// start with the current baseline and ratchet down weekly.
Merge shard reports before counting Each worker writes a partial report, so merge all shard JSON before computing a real flake rate. shard 1 json shard 2 json shard N json merge real flake rate
Merging per-shard reports before counting prevents a sharded run from undercounting flakes.

Pitfalls #

  • Reading the exit code instead of the report: a flaky-only run exits zero. Mitigation: always parse results.json and act on status === 'flaky'.
  • Recomputing flakiness from attempts: Playwright already labels it. Mitigation: trust the flaky status; only walk attempts for diagnostics.
  • Undercounting across shards: each shard emits a partial report. Mitigation: use blob reporter and merge-reports, or merge JSON before counting.
  • Retries hiding hard failures over time: a steadily worsening test stays “flaky”. Mitigation: track the flake rate trend, not just the per-run count.
  • Capturing traces on every attempt: storage and time balloon. Mitigation: gate artifacts on testInfo.retry > 0.
Gate on a flaky budget A post-run check compares the detected flaky count to a budget, failing the job past the threshold. flaky count vs budget under → pass over → fail job
A budget keeps retries from becoming a permanent crutch — start lenient, then ratchet down.

Reliability targets #

Metric Target
Retries on CI 1-2
Retries locally 0
Flaky tests per run ≤ 5 (budgeted)
Flaky rate per suite < 1.5%
CI pass rate (post-retry) ≥ 99%
Retry-detection scorecard Targets for retry count, flaky per run, pass-on-retry rate, and CI pass rate. 1–2retries (CI) budgetedflaky/run < 1.5%pass-on-retry ≥ 99%CI pass
Low retry counts plus a budget keep the pass-on-retry rate under control.

Frequently Asked Questions #

Q: What is the difference between failed and flaky in Playwright? A: failed means every attempt failed; flaky means at least one attempt failed but a later one passed. Only flaky indicates non-determinism worth recording.

Q: Do retries change the worker or context? A: Yes. Each retry runs in a fresh worker and a new browser context, so leaked in-memory state from a prior attempt does not carry over — which is why ordering and backend-state issues still cause flakiness.

Q: How does this relate to the Jest approach? A: The mechanism is the same idea applied to a different runner. See detecting flaky tests with Jest retryTimes for the unit-test equivalent and a custom reporter.

Turning the Flaky Status Into Data #

Reporting a rescued failure as a distinct status is only useful if something consumes it. The status lives in the run’s structured output, and a short extraction step converts it into rows that every downstream mechanism — budgets, trends, ownership routing — depends on.

Three fields make the difference between a count and a finding. The error message from the failed attempt supports classification: a timeout is a wait problem, a connection error is infrastructure, a wrong asserted value is a probable product race. The attempt count distinguishes a single blip from a test that needed both retries. And the artifact path is what makes the failure diagnosable later, which is the property most extraction scripts forget.

// Extract rescued failures with enough context to classify and diagnose them.
// Trade-off: parsing the report couples this to a format; a custom reporter is
// more robust and slightly more code.
const flaky = [];
for (const suite of report.suites ?? []) {
  for (const spec of suite.specs ?? []) {
    const attempts = spec.tests?.[0]?.results ?? [];
    if (attempts.length > 1 && attempts.at(-1).status === 'passed') {
      const failed = attempts.find((a) => a.status !== 'passed');
      flaky.push({
        testId: `${suite.file}::${spec.title}`,
        attempts: attempts.length,
        error: failed?.error?.message ?? '',
        trace: failed?.attachments?.find((a) => a.name === 'trace')?.path ?? null,
      });
    }
  }
}

Retaining Evidence for Attempts That Passed #

The default artifact settings in most setups discard traces and videos once a run finishes green, which is precisely when a rescued failure exists. The result is a suite that can report its flake rate accurately and explain none of it.

Retaining on failure rather than on run outcome is the change, and it applies to traces, screenshots and video alike. The storage cost is a few megabytes per rescued failure, retained for a month, and it buys the ability to open the exact failing attempt weeks later — including the DOM snapshot, the network log and the action timeline at the moment things went wrong.

There is a second reason to keep them that has nothing to do with test maintenance. When someone reports an intermittent bug in production, the rescued-failure archive is the cheapest place to look first, and it frequently already contains the defect — recorded, with a trace attached, having been retried away for weeks. A search that takes two minutes instead of two days is the return on one configuration line.

The one thing worth tuning is retention. Keeping artifacts for every rescued failure indefinitely grows without bound; thirty days covers the practical window for both triage and production correlation, and older entries can keep their metadata row without the attachment.

Reporting Rescues Where They Will Be Seen #

Extracted rescue data is only useful if it reaches someone. Two surfaces carry most of the value, and they serve different readers.

The pull-request summary should mention only rescues in specs the change touched, with a link to the trace. That is actionable by the author immediately, and it stays short enough to be read.

The weekly digest carries the ranked list across the suite with owners attached, which is the input the triage process consumes. Aggregates belong here rather than in every pull request, where they are noise the author cannot act on.

Both draw from the same extracted rows, so the reporting is a formatting difference rather than a second pipeline.

Merging per-shard reports before counting is essential in a sharded suite, since each shard writes its own file and counting one of them undercounts the flaky set by the shard factor — a mistake that makes a degrading suite look stable.