Stress-Running Tests to Surface Flakes

Root cause #

Flakiness is a rate, and rates need samples. A test that fails 2% of the time appears roughly once every fifty runs, so a normal pipeline surfaces it every week or two — long after the change that introduced it, in a context where it looks like an isolated blip. Each occurrence blocks someone, gets retried, and produces no measurement.

Deliberate repetition changes the economics completely. A hundred executions concentrated into five minutes on four workers gives an estimate of the rate immediately, at the moment the test is written, before anyone else is affected. The same run, repeated after a fix, is the only honest way to verify it: a single green run after changing a 2%-failing test proves nothing, because 98% of runs were already green.

Repetition also exposes a class of problem that sequential runs hide. Running the same test concurrently with itself surfaces shared-resource assumptions — a fixed port, a hard-coded record id, a shared temporary file, a database row with a fixed key — that never appear when only one copy is running. Those are among the most common causes of flakiness in parallel suites, and they are invisible until two copies collide.

Step-by-step fix #

1. Repeat the test, with concurrency #

Both runners have first-class support, and the concurrency is as important as the count.

# Playwright: 100 executions of each matching test, 4 in parallel
# Trade-off: high concurrency reproduces resource collisions and can itself
# cause contention on a small machine — match workers to available cores.
npx playwright test cart.spec.ts --repeat-each=100 --workers=4

# Jest / Vitest: repetition through the runner's own retry-free loop
npx vitest run cart.test.ts --repeat=100 --sequence.shuffle

Adding shuffle to the repetition catches order dependence and rate instability in the same run, which is often the cheapest way to learn that a “flaky” test is actually order-dependent.

2. Run stress on changed tests automatically #

The highest-value moment to stress a test is before it is merged. Detect which spec files a pull request touches and repeat those.

# .github/workflows/stress-changed.yml
# Trade-off: a few minutes added to pull requests that touch tests, in exchange
# for catching instability before it becomes everyone's problem.
- name: Stress changed specs
  run: |
    CHANGED=$(git diff --name-only origin/main... | grep -E '\.(spec|test)\.[tj]s$' || true)
    [ -z "$CHANGED" ] && exit 0
    npx playwright test $CHANGED --repeat-each=50 --workers=4 --retries=0

Fifty repetitions catches anything above roughly a 5% rate reliably and takes a couple of minutes; the nightly job below goes deeper.

3. Stress the whole suite on a schedule #

A nightly job that repeats every test a smaller number of times finds instability across the suite without gating anyone.

on:
  schedule:
    - cron: '0 1 * * *'
jobs:
  stress:
    steps:
      # Trade-off: 10 repetitions of everything is a large job; sharding it keeps
      # the wall clock reasonable and the results are worth a nightly runner.
      - run: npx playwright test --repeat-each=10 --retries=0 --shard=${{ matrix.shard }}/8
      - run: node scripts/ingest.js         # rates land in the history store

4. Vary the conditions, not just the count #

Repetition under identical conditions finds internal races. Varying the conditions finds the environmental sensitivities that cause most CI-only failures.

# Trade-off: each variation multiplies the run time; pick the two or three that
# match your CI reality rather than sweeping every combination.
npx playwright test cart.spec.ts --repeat-each=50 --workers=8      # over-subscribed
npx playwright test cart.spec.ts --repeat-each=50 --workers=1      # serialised
TZ=Pacific/Auckland npx playwright test cart.spec.ts --repeat-each=50

A test that is stable at one worker and unstable at eight has a shared-resource or contention problem, which is a completely different fix from an internal race.

5. Fail the stress job on a rate threshold, not on any failure #

A single failure in a thousand repetitions of an end-to-end test is not necessarily actionable. Gate on the measured rate so the job produces decisions rather than noise.

// scripts/stress-gate.js
// Trade-off: a threshold above zero tolerates genuine rare environmental
// events; set it low enough that a real defect cannot hide beneath it.
const THRESHOLD_PCT = 1.0;
const rate = (failures / repetitions) * 100;

if (rate > THRESHOLD_PCT) {
  console.error(`::error::${testId} failed ${failures}/${repetitions} (${rate.toFixed(1)}%)`);
  process.exit(1);
}

6. Use the same command to verify every fix #

Whatever count established the baseline should establish the verification. Recording both numbers in the triage note is what keeps the reopen rate low, as set out in Writing a Flaky Test Triage Runbook.

Pitfalls #

• Leaving retries on during a stress run. Rescued failures hide the rate you are trying to measure. Mitigation: --retries=0 always.
• Repeating at one worker only. Resource collisions never appear. Mitigation: repeat at realistic and over-subscribed concurrency.
• Choosing a round number of repetitions. Fifty cannot detect a 1% rate. Mitigation: pick the count from the rate you need to detect.
• Stressing on an idle laptop and concluding it is fixed. CI is not idle. Mitigation: run under CI-like CPU and memory limits.
• Failing on any single failure. The job becomes noise and gets ignored. Mitigation: gate on a rate threshold.
• Stressing the whole suite on every pull request. Minutes added for little benefit. Mitigation: stress changed specs on pull requests, everything nightly.
• Not recording the results. Each stress run is thrown away instead of building history. Mitigation: ingest into the flakiness store.

Reliability targets #

Frequently Asked Questions #

Q: How many repetitions are enough? A: Enough that the rate you care about would very likely produce at least one failure. Fifty repetitions catch anything at 5% or above with high confidence; a hundred catch 3%; detecting 1% reliably needs several hundred. Below about 1%, stress running becomes expensive and the passive CI history in Calculating Flake Rate from CI History is the better instrument.

Q: Should stress runs block a pull request? A: For specs the pull request changed, yes — the author is the right person to fix a test they just wrote, and the failure is unambiguous. Do not block on stressing tests the change did not touch: that punishes the wrong person for pre-existing instability and gets the check disabled.

Q: Why does the test only fail at high worker counts? A: Because copies of it are competing for something. The usual suspects are a fixed port, a shared temporary path, a database record with a hard-coded key, or an external account being mutated by two copies at once. The fix is namespacing per worker, along the lines of Isolating Database State in Parallel Jest Workers — not lowering the worker count.

Q: Is a stress run a substitute for fixing the underlying design? A: No, it is an instrument. It tells you the rate, tells you whether concurrency matters, and tells you whether a fix worked. What it cannot do is make an inherently timing-dependent test reliable, and a test that needs a stress run to stay honest is usually asking to be rewritten at a level where its dependencies are controlled.

Budgeting the Stress Run #

Repetition costs runner time, and the budget is worth setting explicitly rather than discovering. A pre-merge run over changed specs should stay within a couple of minutes, which at typical end-to-end durations means fifty repetitions of a handful of tests — enough to catch anything above roughly five percent.

The nightly whole-suite pass is where depth belongs, since nothing is waiting on it. Sharding it keeps the wall clock reasonable, and feeding the results into the same history store means the rates it produces are comparable with everything else rather than living in a separate report.

Recording the stress result alongside the change gives the next person a baseline. Without it, a later regression in the same test starts from zero, and the question of whether it was ever stable cannot be answered.

Related guides #

• Automated Flaky Test Detection Tools
• Writing a Flaky Test Triage Runbook
• Storing Test History in SQLite for Flake Analysis