Why Blanket Retries Hide Real Bugs

Root cause #

An intermittent product defect and a flaky test produce the same observable: a failure that does not reproduce on the next attempt. A race between a state update and a render, a request whose response is occasionally processed out of order, a cache invalidation that sometimes runs late — each fails a well-written test at exactly the rate the underlying race occurs. Retry the test and it passes, because the race resolved the other way.

This is the strongest argument for treating the retry count as data rather than plumbing. The retry itself is defensible; the loss of information is not. Once a rescued failure is discarded, the two causes are permanently indistinguishable, and the only remaining detector is a user reporting the behaviour in production.

The severity is amplified by which tests tend to be flaky. Instability concentrates in tests that exercise concurrency, network interaction and asynchronous state — exactly the areas where genuine intermittent defects live. So a blanket policy is not indiscriminately hiding a random sample of failures; it is preferentially hiding the failures most likely to be real. That is the opposite of the risk profile you want, and it is why “retry everything twice” feels harmless while quietly being the most expensive default in a pipeline.

Step-by-step fix #

1. Keep the evidence from the failed attempt #

The single most valuable change is to preserve what the first attempt produced. A trace, a screenshot and the error make a rescued failure diagnosable weeks later.

// playwright.config.ts
// Trade-off: retaining artifacts on retried failures costs storage, and it is
// what makes the difference between a counted flake and a diagnosable one.
export default defineConfig({
  retries: 1,
  use: {
    trace: 'retain-on-failure',        // kept even when the retry passes
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
});

2. Classify each rescued failure instead of counting it #

A raw count tells you how much instability exists; a classification tells you what to do. Three buckets cover nearly everything, and the error signature usually decides it.

// scripts/classify-flaky.js
// Trade-off: heuristics misclassify some cases and are far better than no
// triage at all — the point is to route, not to be perfectly right.
export function classify(failure) {
  const msg = failure.error?.message ?? '';
  if (/Timeout .* exceeded|waiting for (locator|selector)/i.test(msg)) return 'wait';
  if (/ECONNREFUSED|socket hang up|502|503|Disallowed net connect/i.test(msg)) return 'infra';
  if (/expected .* received|toEqual|toBe\b/i.test(msg)) return 'assertion';   // suspicious
  return 'unknown';
}

An assertion failure that passes on retry is the alarming one. A wait timeout can plausibly be a slow runner; an assertion that got the wrong value and then the right value means the application produced two different answers for the same input, which is a race until proven otherwise.

3. Escalate assertion-class rescues immediately #

Route the suspicious class to a human rather than into an aggregate. The volume is low, which is what makes this affordable.

# Trade-off: alerting on a category rather than a threshold means occasional
# false alarms, and it is the only way a real race gets looked at the same week.
- name: Flag suspicious rescued failures
  if: steps.classify.outputs.assertion_count != '0'
  run: |
    gh issue create --label "possible-product-race" \
      --title "Assertion-class flake: ${{ steps.classify.outputs.first_title }}" \
      --body "Passed on retry after asserting a different value. Trace attached to run ${{ github.run_id }}."

4. Reproduce a suspected race by repetition, not by reasoning #

Once a test is suspect, run it many times under load. A product race usually reproduces at a measurable rate; a flaky test’s rate typically drops sharply when the environment is quiet.

# Trade-off: a hundred repetitions is minutes of compute and is decisive in a
# way that reading the code is not.
npx playwright test invoices.spec.ts --repeat-each=100 --workers=4 \
  | tee repeat.log
grep -c "failed" repeat.log     # a stable non-zero rate is evidence of a race

The technique is developed further in the stress-running approach under Automated Flaky Test Detection Tools.

5. Disable retries where a failure is always meaningful #

Retry policy should be per category, not global. Unit tests, contract checks and shuffled-order runs should never be retried, because in those contexts a failure is deterministic and hiding it has no upside.

// Trade-off: some red builds that a retry would have rescued, in exchange for
// never hiding a deterministic failure behind a second attempt.
projects: [
  { name: 'unit', retries: 0 },
  { name: 'contract', retries: 0 },
  { name: 'e2e', retries: 1 },
],

6. Treat “retry passed” as a weaker green #

A run that needed a retry is not the same as a run that passed cleanly, and the reporting should say so. Marking such a build as passed-with-flakes keeps the distinction visible to the person deciding whether to release.

Pitfalls #

• Discarding first-attempt artifacts. The rescued failure is no longer diagnosable. Mitigation: retain traces on failure even when the retry passes.
• Counting rescues without classifying them. A race and a slow wait are averaged into one number. Mitigation: classify by error signature and route separately.
• Global retry settings. Deterministic failures get hidden alongside genuine blips. Mitigation: per-project retry policy.
• Assuming reproduction requires understanding. Teams reason about a suspected race for days. Mitigation: repeat the test a hundred times and measure.
• Treating a passed-on-retry build as clean. Release decisions are made on incomplete information. Mitigation: report passed-with-flakes distinctly.
• Retrying a test that mutates state. The second attempt starts from a different world and its pass means little. Mitigation: exclude those tests from retry.

Reliability targets #

Frequently Asked Questions #

Q: How can I tell whether a rescued failure was a real bug? A: Start with the error class. A wrong asserted value that becomes the right value on retry means the application produced two answers for one input, which is a race until you disprove it. Then reproduce by repetition under load: a rate that holds steady points at the product, a rate that collapses when the machine is idle points at the test’s waits.

Q: Is it acceptable to retry a test that talks to a real external service? A: Yes, and that is one of the clearest legitimate cases — the non-determinism genuinely belongs to something you do not control. Record it as infrastructure-class rather than as test flakiness, so the number reflects what it is and does not mix vendor availability into your own quality metric.

Q: Does this apply to retries inside the application as well? A: The same logic applies with higher stakes. A client that silently retries a failing request hides intermittent server errors from your monitoring exactly as a test retry hides them from CI — and in production nobody re-runs to check. Count application-level retries and alarm on the rate, for the same reason the test-level rate is worth gating on; the idempotency considerations are covered in Retrying Idempotent Requests Without Masking Flakiness.

Q: We have thousands of rescued failures. Where do we start? A: Classify them and start with the assertion class, which is usually a small minority and contains most of the real defects. Then take the wait class by frequency, since a handful of tests normally account for most of it. The infrastructure class is a platform conversation rather than a test one.

Q: A user reported a bug that only happens sometimes. How do I check whether we already hid it? A: Search the rescued-failure history for the same area before writing a new reproduction. A team that has been recording rescues will frequently find the defect already recorded — failing at some rate for weeks, rescued each time — which turns an open-ended investigation into a trace you can open. That search is the concrete payoff for retaining artifacts, and it is impossible for teams that discard them.

Q: Does this mean retries should be removed entirely? A: No. Removing them makes the pipeline noisy enough that people stop trusting red, which is its own failure mode. Keep one retry where non-determinism is genuinely environmental, remove them where failures are deterministic, and make sure every rescue is recorded and classified — the policy in Setting a Retry Budget That Fails the Build.

Related guides #

• CI Retry Strategies & Budgets
• Setting a Retry Budget That Fails the Build
• Retrying Idempotent Requests Without Masking Flakiness