Setting a Retry Budget That Fails the Build

Root cause #

Flakiness accumulates because nothing resists it. Adding a flaky test costs its author nothing at merge time — the retry rescues it — while fixing one costs an afternoon. Over a year of that gradient, every suite becomes unstable, and no amount of dashboards or awareness campaigns reverses an incentive.

A budget resists it by moving the cost forward. When the aggregate rate crosses a line, the build fails, so the next person to add a flaky test pays immediately instead of everybody paying later. The mechanism is unremarkable; the calibration is what determines whether it survives contact with a delivery deadline.

Three calibration mistakes account for nearly all abandoned budgets. Setting the threshold from an aspiration means the build is red on day one and the check is disabled by the end of the week. Expressing it as an absolute count means it silently tightens as the suite grows, so a team adding tests is penalised for growth. And gating it on the wrong scope — failing a feature branch for a rate driven by a different team’s tests — produces failures the author cannot act on, which is the fastest route to a check being routed around.

Step-by-step fix #

1. Measure the current rate before choosing a number #

Run for a week and compute the rate over that window. The budget starts just above whatever you observe.

// scripts/measure-rate.js
// Trade-off: a week-long baseline delays enforcement and is what stops the
// threshold from being a guess that fails everyone immediately.
const runs = loadRuns({ days: 7 });                  // from the flakiness store
const executions = runs.reduce((n, r) => n + r.total, 0);
const flaky = runs.reduce((n, r) => n + r.flaky, 0);

console.log(`baseline: ${((flaky / executions) * 100).toFixed(2)}% over ${runs.length} runs`);

2. Express the budget as a rate, with a minimum sample #

A percentage on a small sample is noise: one flaky test in a twenty-test smoke run is 5%. Require a minimum number of executions before the gate applies.

// scripts/enforce-budget.js
// Trade-off: a minimum sample means small runs are never gated, which is the
// correct behaviour — a rate computed from twenty executions means nothing.
const BUDGET_PCT = Number(process.env.FLAKE_BUDGET_PCT ?? 1.5);
const MIN_EXECUTIONS = 200;

const { flaky, total } = readResults();
if (total < MIN_EXECUTIONS) {
  console.log(`sample too small (${total}) — budget not enforced`);
  process.exit(0);
}

const rate = (flaky / total) * 100;
if (rate > BUDGET_PCT) {
  console.error(`::error::flake rate ${rate.toFixed(2)}% exceeds budget ${BUDGET_PCT}%`);
  console.error(topOffenders(5).map((t) => `  ${t.count}× ${t.title}`).join('\n'));
  process.exit(1);
}

Printing the top offenders in the failure message matters more than it looks: a gate that says “the budget is exceeded” produces frustration, while one that says “these five tests account for it” produces a fix.

3. Gate on the right scope #

A pull request should be gated on what its author can influence. Two scopes work well together: a per-run gate on the tests the change touches, and a trunk-level gate on the whole suite that blocks the release rather than the merge.

# .github/workflows/flake-budget.yml
# Trade-off: two gates is more configuration than one, and it prevents a
# developer being blocked by instability in code they have never opened.
jobs:
  pr-budget:
    if: github.event_name == 'pull_request'
    steps:
      - run: node scripts/enforce-budget.js --scope=changed --budget=0     # zero tolerance, small scope

  trunk-budget:
    if: github.ref == 'refs/heads/main'
    steps:
      - run: node scripts/enforce-budget.js --scope=all --budget=1.5       # aggregate, blocks release

The pull-request gate can afford zero tolerance because its scope is tiny: a test the change touched should not be flaky, and if it is, the author is the right person to know.

4. Ratchet the budget down on a schedule #

A static budget becomes a target that the rate rises to meet. Step it down as the achieved rate falls, automatically and visibly.

// scripts/ratchet.js — run monthly
// Trade-off: automatic ratcheting keeps pressure on without a meeting, and it
// can tighten during a bad month unless it only ever moves in one direction.
const achieved = rateOverLastDays(30);
const current = Number(readFileSync('.flake-budget', 'utf8'));
const proposed = Math.max(0.25, Math.min(current, Math.ceil(achieved * 1.2 * 4) / 4));

if (proposed < current) {
  writeFileSync('.flake-budget', String(proposed));
  console.log(`budget ratcheted ${current}% → ${proposed}%`);
}

5. Make exceeding the budget produce work, not just a red build #

A gate with no follow-through becomes an obstacle to route around. Wire the breach into the quarantine and triage path so the failure creates an owned task, along the lines of Building Auto-Quarantine Workflows.

# Trade-off: auto-filing issues can generate noise; dedupe on the test title so
# a persistent offender accumulates comments rather than duplicate issues.
gh issue list --label flaky --search "$TEST_TITLE" --json number \
  | jq -e '.[0].number' >/dev/null \
  || gh issue create --label flaky --title "Flaky: $TEST_TITLE" \
       --body "Consumed $COUNT retries in the last 7 days. Owner: $CODEOWNER"

6. Exempt what cannot be acted on, explicitly #

Some instability is genuinely infrastructural for a period — a known vendor problem, a runner migration in progress. Allow a temporary, dated exemption rather than letting people quietly raise the threshold.

// Trade-off: exemptions with expiry dates are honest and require someone to
// revisit them; an open-ended exemption is just a higher budget in disguise.
const exemptions = [
  { pattern: /@payments-vendor/, until: '2026-09-01', reason: 'vendor sandbox instability, ticket OPS-1421' },
];

Pitfalls #

• Setting the budget below the current rate. Every build is red and the check is removed. Mitigation: start above the measured baseline and ratchet.
• An absolute count instead of a rate. Growing the suite trips the gate. Mitigation: gate on a percentage of executions.
• No minimum sample size. Small runs produce meaningless rates. Mitigation: skip enforcement below a few hundred executions.
• A failure message with no names. People re-run rather than fix. Mitigation: print the top offenders and their owners.
• Gating a pull request on the whole suite. Authors are blocked by instability they cannot influence. Mitigation: narrow scope on pull requests, aggregate scope on trunk.
• Open-ended exemptions. The budget quietly stops applying. Mitigation: require an expiry date and a reason.
• A budget with no escalation. The build fails and nothing changes. Mitigation: create an owned task on breach.

Reliability targets #

Frequently Asked Questions #

Q: Will a budget just stop people writing tests? A: It can, if the only lever available is “delete the test”. Pair the budget with a quarantine path so an unstable test can be moved out of the blocking set with its owner recorded, and the pressure lands on fixing rather than on avoiding. A budget without quarantine is a choice between a red build and a deleted test, and teams choose deletion.

Q: Who should be blocked when the trunk budget is exceeded? A: The release, not individual merges. Blocking every merge on an aggregate the current author did not cause creates collective punishment and gets the gate disabled. Blocking the release creates a shared, visible reason to spend an afternoon on the top offenders, which is the outcome you want.

Q: How do we handle a big migration that temporarily raises the rate? A: A dated exemption for the affected tag, with a ticket reference. That keeps the mechanism intact and time-bounded, unlike raising the global threshold, which is easy to do and much harder to undo — the number stays high long after the migration ends because nobody remembers why it was raised.

Q: Where should the budget live so it is visible? A: In a file in the repository, not in a CI setting. A committed threshold shows up in review when someone changes it, has a history explaining every step, and can be ratcheted by an automated pull request. A number buried in pipeline configuration gets raised quietly during a difficult week and nobody notices it never came back down.

Q: Should the budget cover unit tests as well as end-to-end? A: Separately, with different numbers. Unit flakiness is almost always order dependence or a real defect and should have a budget of zero, since retries there mask deterministic bugs. End-to-end suites deal with genuine environmental non-determinism and warrant a small non-zero budget. One combined number lets healthy unit executions dilute an unhealthy end-to-end rate.

Related guides #

• CI Retry Strategies & Budgets
• Why Blanket Retries Hide Real Bugs
• Calculating Flake Rate from CI History