Retrying Idempotent Requests Without Masking Flakiness

Root cause #

There are two completely different things called “retry” and conflating them is how flakiness hides. A request retry lives in the application’s HTTP client: when a fetch to a flaky upstream fails, the client tries again. A test retry lives in the test runner: when the whole test fails, the runner re-runs it. The first is a legitimate resilience feature your users benefit from; the second is a last-resort flakiness suppressant. The failure mode is using either one to hide a deterministic bug.

App-level request retries become dangerous in three ways. First, retrying a non-idempotent request (a POST that creates an order) can double-charge or duplicate data — the retry is not safe even though it “succeeds.” Second, retrying on a 500 masks a real server error that your test should have caught. Third, an unobserved retry makes a degraded dependency look healthy: the request fails three times, succeeds on the fourth, the test goes green, and nobody learns the upstream is on fire. The fix is not to ban retries but to constrain them — restrict to idempotent methods and retryable status codes, cap the attempts, add backoff, and emit a metric on every retry so the suppression is measurable. Then you can tell legitimate resilience from masked flakiness, because the masking has a number attached to it.

Step-by-step fix #

1. Retry only idempotent methods on retryable failures #

Gate the retry on both the HTTP method and the failure class. A GET timing out is retryable; a POST returning 400 is not.

// src/http/retry.ts — narrow, observable request retry
const RETRYABLE_STATUS = new Set([502, 503, 504]);
const IDEMPOTENT = new Set(['GET', 'HEAD', 'PUT', 'DELETE']);

export async function fetchWithRetry(url: string, init: RequestInit = {}, max = 3) {
  const method = (init.method ?? 'GET').toUpperCase();
  for (let attempt = 0; ; attempt++) {
    const res = await fetch(url, init);
    // Only retry safe methods on transient infra errors; a 500 or a 4xx is a
    // real signal and must surface, not be swallowed by another attempt.
    const retryable = IDEMPOTENT.has(method) && RETRYABLE_STATUS.has(res.status);
    if (res.ok || !retryable || attempt >= max) return res;
    onRetry(url, attempt); // emit a metric — see step 3
    await backoff(attempt);
  }
}

2. Add capped exponential backoff with jitter #

Immediate retries hammer a struggling upstream. Back off, and add jitter so parallel clients do not synchronize their retries into a thundering herd.

// src/http/backoff.ts — exponential backoff with jitter, hard ceiling
export function backoff(attempt: number): Promise<void> {
  const base = Math.min(1000 * 2 ** attempt, 8000); // cap at 8s, not unbounded
  // Jitter prevents synchronized retries from N parallel test workers from
  // colliding into a self-inflicted load spike against the dependency.
  const delay = base / 2 + Math.random() * (base / 2);
  return new Promise(r => setTimeout(r, delay));
}

3. Emit a metric on every retry so masking is visible #

This is the line that separates resilience from suppression. Count every retry and tag it, so a climbing retry rate shows up where you track flaky test detection rather than vanishing into a green checkmark.

// src/http/observe.ts — make every retry measurable
export function onRetry(url: string, attempt: number) {
  // A spike in this counter means an upstream is degrading — surface it as a
  // reliability signal instead of letting the retry quietly hide the failure.
  metrics.increment('http.request.retry', {
    tags: [`endpoint:${new URL(url).pathname}`, `attempt:${attempt}`],
  });
}

4. Keep request retries and test retries separate #

Do not let the test runner’s retry compensate for an app that lacks retries, and do not let app retries hide a deterministic test failure. Configure runner retries narrowly and assert on the retry metric in tests that exercise resilience.

// playwright.config.ts — runner retries are a safety net, not a fix
export default defineConfig({
  // One CI retry catches true infra flakes; more than that hides real bugs.
  // The app's own request retry is tested explicitly, not via runner retries.
  retries: process.env.CI ? 1 : 0,
});

A test that mocks three 503s followed by a 200 should assert the client recovered and that exactly three retries were recorded — proving the resilience works without letting it mask a permanent failure.

Pitfalls #

• Retrying non-idempotent requests. A retried POST can duplicate side effects. Mitigation: restrict retries to idempotent methods, or require an idempotency key on writes.
• Retrying on 500. A 500 is usually a real defect, not a transient blip. Mitigation: retry only on 502/503/504 and network-level timeouts.
• Silent retries. A retry with no metric makes a degraded dependency invisible. Mitigation: emit a counter on every retry and alert on the rate.
• Stacking app retries under test retries. Both layers retrying multiplies attempts and deeply buries the real failure. Mitigation: keep runner retries at ≤ 1 and test the app retry path explicitly.
• Unbounded backoff. Backoff without a ceiling can stall a test until it times out. Mitigation: cap the delay and the attempt count.

Reliability targets #

Frequently Asked Questions #

Q: When is a request retry legitimate versus masking flakiness? A: Legitimate when the failure is transient and the request is idempotent — a GET hitting a 503 during a deploy. Masking when it converts a deterministic failure (a 500 from a real bug, or a POST that should not repeat) into a pass. The metric on every retry is what lets you tell them apart over time.

Q: Should I just turn on test runner retries instead of app retries? A: No — they solve different problems. Runner retries re-run the entire test and hide everything, so keep them at one as a thin safety net. App retries are a product feature you should test explicitly. Treat a rising runner-retry rate as a flakiness signal to investigate, not a setting to crank up.

Q: How do I prove a retry isn’t hiding a real bug? A: Make the retry observable and assert on it. In the resilience test, mock a finite number of transient failures, assert recovery, and assert the exact retry count. In production, alert when the retry-rate metric climbs so a degrading dependency surfaces instead of staying green.

Retrying in the Client Hides What Retrying in CI Hides #

The argument against unreported test retries applies with more force to application retries, because there is no second run to reveal what was hidden.

A client that silently retries a failing request keeps the user’s experience smooth and removes the failure from every downstream signal: the error rate looks healthy, the monitoring shows nothing, and a dependency degrading steadily produces no alert until it fails outright. The retry is doing its job; the absence of accounting is the problem.

Counting retries in the client and reporting them as their own metric restores the signal for the same effort it takes to implement the retry. A rising retry rate against a flat error rate is a dependency getting worse, and that is exactly the early warning the error rate cannot provide.

The testable consequences follow directly. A test can assert that a transient failure produced a bounded number of attempts with increasing delays, that a non-idempotent request was not repeated, and that the retry emitted whatever telemetry the monitoring expects. Those assertions are cheap with route-level control and impossible against a real dependency.

// Assert the retry policy, not merely that the request eventually succeeded.
// Trade-off: the test encodes the attempt count, so a deliberate policy change
// requires updating it — which is the point.
let attempts = 0;
await page.route('**/api/rates', (route) => {
  attempts += 1;
  return attempts < 3
    ? route.fulfill({ status: 503, json: { error: 'unavailable' } })
    : route.fulfill({ status: 200, json: { rate: 1.08 } });
});
// …after the flow completes:
expect(attempts).toBe(3);

Idempotency Keys Are What Make a Retry Safe #

A retry is only safe if repeating the request cannot repeat its effect, and for writes that guarantee has to be constructed rather than assumed.

An idempotency key supplied by the client — the same value on every attempt of the same logical operation — lets the server recognise a repeat and return the original result instead of performing the action twice. Without it, a retry after a response that was lost in transit creates a second order, a second charge or a second message, and the client has no way to tell the difference between a request that failed and one that succeeded invisibly.

Two properties of the key matter. It must be stable across attempts, which means generating it when the operation begins rather than inside the retry loop — a key regenerated per attempt provides no protection at all. And it must be unique per logical operation, so two genuinely different submissions are not collapsed into one.

Both are directly testable: assert that all attempts carried the same key, and that a repeated key produced one effect rather than two. Those two assertions cover the failure mode that makes duplicate charges possible, and they are among the highest-value tests a payment or ordering flow can have.

Related guides #

• Network Latency & Volatility Handling
• Flaky Test Detection & Quarantine Engineering
• Tracking Test Flakiness Trends Over Time