CI Retry Strategies & Budgets

Prerequisites #

A budget is only meaningful alongside detection and quarantine, which is why this sits next to Automated Flaky Test Detection Tools and Building Auto-Quarantine Workflows rather than standing alone.

Step-by-step implementation #

1. Retry at the test level, not the job level #

A job-level rerun repeats the whole suite, which multiplies cost, hides which test was unstable, and re-runs hundreds of healthy tests to rescue one. Test-level retries isolate the instability and make it attributable.

// playwright.config.ts
// Trade-off: retries rescue a genuinely flaky test and also mask a real
// intermittent product bug — which is why the count must be reported.
export default defineConfig({
  retries: process.env.CI ? 1 : 0,     // one retry in CI, none locally
  reporter: [['json', { outputFile: 'results.json' }], ['line']],
});

One retry is the right default. It converts an isolated blip into a pass while leaving a persistent failure red, and it caps the cost at double for the affected test rather than for the pipeline.

2. Report every retry as a first-class event #

A retry that is not recorded is a fact the team has decided not to know. Extract the flaky results from the structured output and send them to the same store used for Historical Flakiness Tracking & Analytics.

// scripts/extract-flaky.js
// Trade-off: parsing runner output couples this to a report format; the
// alternative is a custom reporter, which is more code and more robust.
import { readFileSync, writeFileSync } from 'node:fs';

const report = JSON.parse(readFileSync('results.json', 'utf8'));
const flaky = [];

for (const suite of report.suites ?? []) {
  for (const spec of suite.specs ?? []) {
    const attempts = spec.tests?.[0]?.results ?? [];
    if (attempts.length > 1 && attempts.at(-1).status === 'passed') {
      flaky.push({ title: spec.title, file: suite.file, attempts: attempts.length });
    }
  }
}

writeFileSync('flaky.json', JSON.stringify({ count: flaky.length, flaky }, null, 2));

3. Set a budget and fail the build when it is exceeded #

The budget turns a soft metric into a hard boundary. Express it as a share of executions rather than an absolute count, so it stays meaningful as the suite grows.

// scripts/enforce-budget.js
// Trade-off: a hard failure on budget breach blocks work that did not cause it,
// which is the pressure that makes the budget real rather than aspirational.
const BUDGET_PCT = 1.0;                       // 1% of executions may be flaky
const { count } = JSON.parse(readFileSync('flaky.json', 'utf8'));
const total = JSON.parse(readFileSync('results.json', 'utf8')).stats.expected;

const rate = (count / total) * 100;
console.log(`flaky ${count}/${total} = ${rate.toFixed(2)}% (budget ${BUDGET_PCT}%)`);

if (rate > BUDGET_PCT) {
  console.error('::error::flakiness budget exceeded — quarantine or fix before merging');
  process.exit(1);
}

4. Never retry what must not be retried #

Some failures are meaningless when retried and should be excluded by policy. A shuffled-order failure is deterministic given its seed, so a retry either reproduces it or hides a real ordering bug. A failure in a test that mutates external state may leave that state dirty, so the retry starts from a different position than the first attempt.

// Trade-off: excluding categories from retry produces more red builds and
// ensures those reds mean something.
projects: [
  { name: 'unit-shuffled', retries: 0, use: { /* … */ } },   // order bugs must not be retried
  { name: 'e2e', retries: 1 },
  { name: 'destructive', retries: 0, grep: /@mutates-state/ },
],

5. Make the retry cost visible #

Retries consume runner minutes, and unbudgeted minutes are how a pipeline drifts from eight to twenty-five minutes without any single decision. Report retry time alongside retry count, so the conversation includes both reliability and cost.

// Trade-off: tracking retry minutes adds a metric to maintain and gives the
// budget an economic argument that a percentage alone does not.
const retryMinutes = flaky.reduce((n, f) => n + (f.durationMs ?? 0), 0) / 60_000;
console.log(`retry cost this run: ${retryMinutes.toFixed(1)} minutes`);

6. Escalate on repeat offenders instead of retrying forever #

A test that consumes the budget week after week is not flaky in a way retries help with. Route it to quarantine automatically, so the retry mechanism is spent on genuine blips rather than on a known-bad test.

7. Understand what a retry does to your confidence #

Retries change the statistics of the suite in a way worth stating precisely, because the intuition is misleading. A test that fails independently with probability p passes a run with n retries with probability 1 − pⁿ⁺¹. At one retry, a test failing 10% of the time now fails a run 1% of the time; at two retries, 0.1%. That is the useful effect.

The same arithmetic is the problem. At two retries a test that fails half the time still passes 87% of runs, so a badly broken test looks merely unlucky. And because retries are per test, a suite of a thousand tests each failing 0.1% of the time still produces a red run roughly 63% of the time without retries, and about 0.1% with one retry — which is why retries feel indispensable at scale and why the same mechanism can absorb an unbounded amount of genuine breakage.

The conclusion is not to avoid retries but to bound them at one, and to treat the retry count as the measured quantity. A suite whose retry count is flat is stable; a suite whose retry count is rising is degrading, regardless of how green the pipeline looks. The pass rate has been engineered to be uninformative, so the retry count has to carry the signal instead.

8. Separate infrastructure failure from test failure #

A meaningful share of retried failures are not about tests at all: a container that failed to start, a database that was not ready, a runner that ran out of memory, a registry that timed out during install. Lumping these into the flake rate makes the number unactionable — the test owners cannot fix the runner, and the platform team never sees a signal addressed to them.

Classify at the point of collection, using the error signature, and keep two counters. The test-flake rate drives the budget and belongs to the teams that own the specs; the infrastructure-failure rate belongs to whoever owns the pipeline, and it deserves its own threshold and its own escalation.

// Trade-off: classification by error signature is imperfect and is far better
// than one aggregate that nobody can act on.
const INFRA = /ECONNREFUSED|ETIMEDOUT|no space left|OOMKilled|failed to start|registry/i;

const [infra, tests] = partition(rescuedFailures, (f) => INFRA.test(f.error ?? ''));
report({ infraRate: rate(infra), flakeRate: rate(tests) });   // two owners, two numbers

Configuration reference #

Data-driven analysis #

• Flake rate. Flaky results divided by executions, over a rolling window. This is the primary number the budget gates on, and the only one that stays comparable as the suite grows.
• Retry concentration. The share of retries consumed by the top five tests. High concentration is good news — a handful of named tests to fix. A flat distribution means the instability is systemic, usually environmental, and points back at CI Environment & Browser Drift.
• Retry success rate. How often a retry turns red into green. Near 100% suggests genuine flakiness; a substantial share of retries that fail again suggests real defects being retried pointlessly, which is pure cost.
• Retry minutes per pipeline. The economic view. Rising retry minutes with a flat flake rate means tests are getting slower, not less stable.
• Time to quarantine. How long a repeat offender keeps consuming budget before it is quarantined. Long times mean the escalation is manual and is not happening.

Choosing a retry policy by test level #

One policy across a whole repository is almost always wrong, because the levels differ in what a failure means.

Unit tests should have zero retries. A unit test runs in one process with no network and no browser, so a failure is either a genuine defect or order dependence — both deterministic, both hidden by a retry. A team that needs retries at this level has an isolation problem, and the retry is preventing them from finding it.

Component tests are nearly the same case. They add a DOM and usually a mocked network, neither of which introduces real non-determinism. A flaky component test is normally an unresolved asynchronous update or a leaked module mock, so zero retries here keeps the pressure where it belongs.

Integration tests touching a real database or a local service warrant one retry. Container start-up ordering, connection pool exhaustion and port allocation are genuinely environmental, and blocking a merge on a rare start-up race is disproportionate — provided the rescue is counted.

End-to-end tests are the case retries were designed for: a real browser, a real server, shared runners and a network in between. One retry, occasionally two for the longest flows, with every rescue recorded and classified.

Contract and shuffled runs should have zero, for the same reason as unit tests: their failures are reproducible by construction, and hiding one costs the entire value of running them.

The practical consequence is that retry configuration belongs per project rather than at the root of the config, and that a repository with a single global retries value is almost certainly retrying something it should not.

Common pitfalls & mitigation strategies #

• Retrying at the job level. Costly, and it destroys the attribution. Mitigation: retry per test and report the flaky results.
• Retries with no reporting. The flake rate becomes unobservable. Mitigation: extract flaky results into a durable store on every run.
• Unlimited or high retry counts. Three retries make a 50%-failing test look stable. Mitigation: one retry, two at most for genuinely slow end-to-end suites.
• No budget. The rate drifts upward with no forcing function. Mitigation: gate on a percentage of executions.
• Retrying order-dependent failures. A deterministic bug is hidden by chance. Mitigation: zero retries on shuffled runs.
• Retrying state-mutating tests. The second attempt starts from a different world. Mitigation: exclude by tag, and make those tests idempotent.
• A budget nobody owns. The build fails and everyone waits for someone else. Mitigation: name an owner per suite, as in Flaky Test Triage & Ownership.

Frequently Asked Questions #

Q: Are retries an admission of defeat? A: No, they are a queueing decision. Some non-determinism is genuinely outside your control — a shared runner’s scheduler, a network hiccup between containers — and blocking a merge on a one-in-five-hundred event costs more than it prevents. What makes retries defensible is that every one is counted, attributed and budgeted; what makes them corrosive is using them instead of measuring.

Q: How many retries is too many? A: Two. At three retries, a test that fails half the time passes with probability above 90%, which means the mechanism is no longer rescuing blips but manufacturing green. If a suite needs three retries to be stable, it has a systemic problem that retry configuration cannot fix.

Q: Should the budget block merges from day one? A: Set it at the current rate first, so it fails only on regression, then ratchet it down as the rate improves. A budget set below the current rate fails every build immediately and gets removed within a week — the same dynamic that kills any threshold set at an aspirational level rather than an observed one.

Q: Our pipeline is green every day. Do we need any of this? A: A permanently green pipeline with retries enabled and no reporting is exactly the state this topic warns about, because greenness has been engineered to be uninformative. The diagnostic question is whether you can answer “how many tests were rescued by a retry last week, and which ones” from data rather than memory. If not, the pipeline is green in a way that carries no information about the suite.

Q: How do retries interact with sharding? A: They compose, and the interaction is worth being deliberate about. Test-level retries handle an individual test’s blip inside a shard; a shard-level retry handles a failure that takes out the whole shard, such as a runner dying or a service failing to start. Configuring only one leaves a gap — per-test retries cannot rescue a dead process, and shard retries are a wasteful way to rescue one flaky assertion.

Q: What is the relationship between retries and quarantine? A: Retries handle the tail of an otherwise healthy test; quarantine handles a test that is reliably unreliable. The escalation path between them is the important part: a test consuming budget repeatedly should be quarantined automatically rather than retried indefinitely, which keeps the retry mechanism available for what it is good at.

Where should the retry count be configured? Per project, in a committed file, alongside the budget it feeds. A single global setting inevitably retries something it should not — a unit suite, a contract check, a shuffled run — and a value stored in CI settings can be raised during a difficult week without leaving a trace. Both properties matter: the granularity keeps deterministic failures honest, and the visibility keeps the number from drifting upward unnoticed.

Should a retry run on the same machine as the original attempt? Preferably not. Part of a retry’s value is escaping a local condition — a noisy neighbour, exhausted memory, a dependency that failed to start — and reusing the same machine keeps that condition in place. A fresh runner also makes the retry an independent trial, which is the assumption the budget arithmetic rests on.

How do retries interact with test-level artifacts? They determine whether a rescued failure is diagnosable at all. Artifacts must be retained for the failing attempt even when a later one passes; the common default discards them once the run is green, which is precisely when the evidence is needed. One configuration line separates a countable rate from an investigable finding.

Related guides #

• Setting a Retry Budget That Fails the Build
• Retrying Only Failed Shards in GitHub Actions
• Why Blanket Retries Hide Real Bugs
• Building Auto-Quarantine Workflows
• Flaky Test Triage & Ownership