Cypress vs Playwright: Flaky Test Detection Architecture

The core difference is where the “flaky” judgment lives. Playwright bakes a first-class flaky outcome into its JSON reporter when a test fails then passes on retry, so detection is mostly a matter of reading status fields. Cypress has no built-in flaky verdict — you infer it from attempt counts in the mocha-style results, or you lean on Cypress Cloud’s analytics. That single distinction ripples through every layer below.

Prerequisites #

Install the reporters you will parse before wiring detection. For Cypress, pair the native JSON output with mochawesome for attempt-level detail; for Playwright, the built-in json reporter already carries status, retry, and results[].

Step-by-step implementation #

1. Enable retries so flakiness becomes observable #

Retries are what convert a hard failure into a measurable flake signal. In Cypress, configure split run/open retry counts so local debugging stays strict.

// cypress.config.ts
import { defineConfig } from 'cypress';

export default defineConfig({
  retries: {
    runMode: 2,   // CI: a pass after a fail flags a flake (cost: up to 3x on failure)
    openMode: 0,  // local: never hide nondeterminism while debugging
  },
});

Playwright exposes the same idea but emits an explicit verdict.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0, // CI-only retries; status becomes "flaky" on retry-pass
  reporter: [['json', { outputFile: 'results.json' }], ['list']],
});

Trade-off: Retries shrink red builds but also mask genuine regressions if you treat retry-passes as green. Always record the retry count as a separate metric instead of discarding it. The same retry-tooling discipline drives automated flaky test detection tools.

2. Emit a machine-readable report #

The detector consumes the report, not the console. Cypress JSON gives you attempts[] per test, which is your flake proxy.

// scripts/detect-cypress-flakes.ts
import results from '../results/mocha.json';

const flaky = results.tests.filter(
  (t: any) => t.attempts && t.attempts.length > 1 && t.state === 'passed'
); // passed-after-retry == flaky in Cypress (no native flag)
console.log(`flaky specs: ${flaky.length}`);

Playwright hands you the verdict directly.

// scripts/detect-pw-flakes.ts
import report from '../results.json';

const flaky = report.suites
  .flatMap((s: any) => s.specs)
  .filter((spec: any) => spec.tests.some((t: any) => t.status === 'flaky'));
console.log(`flaky tests: ${flaky.length}`);

Trade-off: Cypress’s inferred signal can misclassify a deterministically-fixed-by-reseeding test as flaky; Playwright’s flag is precise but only fires when retries are enabled. Choose retry counts deliberately.

3. Attach a trace artifact for root-cause triage #

Detection tells you that a test flaked; the artifact tells you why. Playwright’s trace.zip is the richer artifact — it bundles DOM snapshots, network, and console into one viewer.

// playwright.config.ts (artifact slice)
export default {
  use: {
    trace: 'on-first-retry', // capture only when a retry happens, to control storage cost
    video: 'retain-on-failure',
  },
};

Cypress captures video and screenshots automatically on failure; opt into per-test screenshots for finer signal.

// cypress/e2e/checkout.cy.ts
afterEach(function () {
  if (this.currentTest?.state === 'failed') {
    cy.screenshot(`fail-${this.currentTest.title}`); // narrows triage to the failing step
  }
});

Trace network analysis is where mocking discipline pays off — flakes that trace back to unstubbed calls are best fixed with Cypress Network Interception Patterns or Playwright Route Mocking Strategies rather than more retries.

4. Parallelize without poisoning the signal #

Parallelism is the single biggest source of false flakes from shared state. Cypress balances at the spec level through Cypress Cloud.

# CI: Cypress spec-level parallelism via Cloud
npx cypress run --record --parallel --ci-build-id "$GITHUB_RUN_ID"
# trade-off: load balancing needs Cloud; without it, manual spec splitting drifts

Playwright shards deterministically without a hosted service.

# CI matrix: shard 1 of 4, N workers per shard
npx playwright test --shard=1/4 --workers=2
# trade-off: cross-shard state collisions look like flakes; isolate fixtures per worker

Trade-off: More workers means faster builds but higher odds of race conditions in parallel runs leaking across tests. Cap worker counts until per-worker isolation is proven, then route confirmed flakes into auto-quarantine workflows.

Configuration reference #

The comparison worth making is therefore not “which runner is more reliable” but “which runner makes instability easier to see and cheaper to act on” — a question about reporting, isolation defaults and artifact retention rather than about API ergonomics.

Migration Is Not the Answer to Flakiness #

Teams comparing the two runners are often really asking whether switching would fix their flakiness, and the honest answer is usually no.

The failure modes catalogued across this site are properties of the application, the environment and the test design rather than of the runner. State leaking between tests, waits on conditions that do not imply readiness, over-subscribed workers, contended runners, unpinned browser builds, mocks that drifted from the API — none of these is fixed by a different automation library. A migration typically produces an initial improvement, because rewriting a suite means rewriting its worst waits, and then the rate drifts back toward where it was as the new suite accumulates the same habits.

Where the runner genuinely does help is at the margins that shape behaviour: a default that isolates per test rather than per file, retries reported as distinct statuses so rescues are countable, tracing that can be retained on rescued failures, and deterministic sharding that makes selective retry possible. Those are real advantages and they are enablers rather than cures — they make good practice easier and bad practice more visible.

The productive version of the question is therefore narrower: which runner makes the reliability work you already know you need cheaper to do? A team with no history store, no budget and no ownership routing will have the same problems in either, and the effort spent migrating would buy more if spent on instrumentation. A team that has those things and is fighting its runner’s isolation model or reporting has a genuine case.

Running Both Without Doubling the Work #

Plenty of teams end up with both runners — one inherited, one adopted for new work — and that is a workable steady state provided the reliability machinery is shared rather than duplicated.

What should be common is everything downstream of collection: one history store with one schema, one classification vocabulary, one budget definition, one ownership routing path, one quarantine mechanism and file format. Each runner then needs only a small adapter converting its report into the shared schema, which is a contained piece of work and the only place the differences should be visible.

What legitimately differs is the collection detail — how rescued failures are identified, how shards are addressed, where traces land — and the per-level retry policy, since the two runners are often used at different levels. Those differences belong inside the adapter, not in a second parallel process.

The arrangement to avoid is two reliability programmes running side by side, each with its own dashboard, thresholds and vocabulary. It doubles the maintenance, produces numbers nobody can compare, and reliably ends with each group believing the other has the worse problem. A single set of thresholds fed by two adapters is both less work and more informative — and it makes the eventual decision about consolidating suites an evidence-based one.

Interpreting the data #

Each runner produces a different primary metric. From Playwright, count status === 'flaky' per test across a window of runs — a test crossing ~1% flake rate over 50 runs warrants quarantine. From Cypress, compute attempts.length > 1 && state === 'passed' as the flake proxy and divide by total runs.

Read the trend, not the snapshot. A single flaky verdict is noise; a test that flakes in 3 of the last 20 builds is a stable signal. Stream both counts into historical flakiness tracking analytics so the verdict survives across builds. Escalate to quarantine when a test’s rolling flake rate exceeds your SLO budget for two consecutive windows — escalating on a single spike just churns the suite.

Isolation Models Shape Everything Downstream #

The deepest architectural difference between the two runners is not their APIs but what they consider a unit of isolation, and nearly every practical divergence follows from it.

Playwright’s default unit is a browser context per test: cookies, storage and permissions start empty, tests within a file can run concurrently, and a worker reuses a browser process while discarding the context. The consequences are that parallelism inside a file is available by default, that state leakage between tests is structurally difficult, and that anything shared deliberately — a signed-in session, a seeded account — is an explicit opt-in that a reader can see in the configuration.

Cypress’s unit is the spec file, with per-test isolation clearing the page, cookies and storage between tests when enabled. Tests in a spec run sequentially in one browser, which makes a spec feel like a session and makes ordering assumptions easy to write accidentally. The isolation setting is a single switch, and disabling it — usually for speed — converts the spec into one long stateful flow where every test inherits the previous one’s world.

For detection work this matters in two ways. First, the unit that gets quarantined differs naturally: a Playwright test is independently runnable, so quarantining one test is clean, while a Cypress spec with ordering assumptions may not survive having one of its tests removed. Second, the reproduction procedure differs: re-running a single Playwright test in isolation is a faithful reproduction, whereas re-running a single Cypress test from a stateful spec may not reproduce the conditions at all.

The practical recommendation that falls out: keep per-test isolation on in Cypress even at a speed cost, and cache the expensive part — the login — rather than the page state. That preserves the property both models need for detection to work, which is that a test can be run alone and mean the same thing.

Making Reports Comparable Across Runners #

A team running both runners ends up with two flakiness numbers that are not comparable, and comparing them anyway leads to the wrong conclusions.

Three normalisations are needed. The unit of measurement must be the same: rate per test execution, not per spec and not per pipeline, since a spec containing twenty tests and one containing three are not comparable units. Retry semantics must be aligned: one runner may report a rescued failure as a distinct status while another reports only the final outcome, so the ingestion step has to derive “rescued” consistently rather than trusting the label. The denominator must include passes from both runners, or a suite that runs more often appears more stable purely because of volume.

Beyond normalisation, the downstream pipeline should be identical: the same history store, the same classification by error signature, the same budget mechanics, the same ownership routing. What differs legitimately is the collection adapter — one per runner, converting its report format into the shared schema — and nothing else.

The failure mode to avoid is two parallel reliability programmes, each with its own dashboard, threshold and vocabulary. That arrangement produces numbers nobody compares, a budget nobody owns, and two teams each believing the other has the worse problem. One schema and one set of thresholds, fed by two adapters, is both less work and more useful.

Common pitfalls & mitigations #

• Treating Playwright retry-passes as plain green. They are not — preserve the flaky status into your report or you lose every detection signal. Mitigation: always parse status, never just exit codes.
• Inferring Cypress flakiness from exit code alone. A passed-after-retry spec exits 0, hiding the flake. Mitigation: parse attempts[] from the JSON/mochawesome report.
• Cross-shard state bleed read as flakiness. Shared DB rows or seeds across Playwright shards mimic nondeterminism. Mitigation: isolate fixtures per worker and per shard.
• Comparing the two runners on raw failure counts. The retry semantics differ, so raw counts are not comparable. Mitigation: normalize to a rolling flake-rate percentage.
• Storing every trace. trace: 'on' explodes artifact storage. Mitigation: use on-first-retry so only suspect runs persist.

Frequently Asked Questions #

Q: Does Playwright’s built-in flaky status mean I do not need a detection pipeline? A: It removes the classification step, but you still need to aggregate verdicts across runs, persist history, and trigger quarantine. The flag is the input to detection, not a replacement for it.

Q: Can I get a native flaky verdict in Cypress without Cypress Cloud? A: Not a first-class one. You derive it by parsing attempts[] from the JSON or mochawesome reporter — a passed test with more than one attempt is your flake signal. Cypress Cloud adds hosted analytics on top of that same data.

Q: Which runner makes root-cause triage faster? A: Playwright’s single trace.zip (DOM snapshots, network, console in one viewer) is generally faster to triage than Cypress’s separate video and screenshot artifacts, especially for network-driven flakes.

Which runner produces more useful failure evidence by default? Playwright’s trace, when retained on failure, is the richer artifact: DOM snapshots, network log and an action timeline in one file, viewable offline. Cypress’s strengths are the interactive runner and its command log, which are excellent while developing and less useful for diagnosing a failure that happened on a runner two days ago. For a team whose diagnosis mostly happens after the fact, that difference is worth more than most API comparisons.

Does one runner handle sharding better for selective retry? Deterministic shard assignment is the requirement, and it is easier to guarantee when the runner shards a stable test list itself. Whichever runner is used, the assignment must be persisted if partitioning is duration-based, or a retry of a failed shard may execute a different set of tests than the one that failed — which quietly invalidates the whole mechanism.

Can a team run both and still have one flakiness number? Yes, and it is the arrangement worth aiming for: one history store and one schema, fed by a small adapter per runner. Normalise the unit to a test execution, derive “rescued” consistently rather than trusting each runner’s label, and include passes from both in the denominator. Everything downstream — budget, classification, routing, quarantine — then works identically regardless of which runner produced the result.

Related guides #

• Automated flaky test detection tools
• Building auto-quarantine workflows
• Cypress network interception patterns
• Playwright route mocking strategies