Detecting Flaky Tests with Jest

A pass-on-retry is the signal worth recording: it is neither a clean pass nor a hard failure.

Root cause #

A Jest test that fails on its first attempt and passes on a retry is, by definition, non-deterministic. In a single-process Jest run the most common drivers are leaked async state between tests, a timer that resolves at a different point relative to an assertion, or shared module state that the previous test left dirty. None of these are fixed by the retry — the retry only proves the outcome depends on timing or order rather than on the code under test.

The danger with jest.retryTimes is that it defaults to silence. If a test passes on attempt two, Jest reports the suite as green and you never learn the test is flaky. The whole point of using it as a detection tool is to invert that: keep the retries so CI stays unblocked, but emit a record every time a retry was needed so the flakiness shows up in your historical flakiness tracking instead of vanishing.

Keep the retry to stay unblocked, but emit a record so the flakiness never vanishes.

Step-by-step fix #

1. Enable retries with error logging #

Call jest.retryTimes in a setup file so it applies to every suite. The logErrorsBeforeRetry option prints the failing error before each retry, which is what makes the retry observable instead of invisible.

// jest.setup.js  — referenced from setupFilesAfterEach in jest.config
// retryTimes is a global; it must run before the tests in each file.
jest.retryTimes(2, { logErrorsBeforeRetry: true });
// Trade-off: 2 retries caps CI cost; higher counts mask deeper bugs
// and inflate runtime, so keep n small and treat retries as a signal.

2. Count retried-but-passed tests with a custom reporter #

A Jest reporter sees each test result, including how many invocations it took. testResult exposes invocations (total attempts) — when a test ends passed but took more than one invocation, it is flaky.

// flaky-reporter.js
const fs = require('fs');

class FlakyReporter {
  constructor() { this.flaky = []; }

  onTestResult(_test, testResult) {
    for (const r of testResult.testResults) {
      // r.invocations > 1 means a retry happened; status 'passed' means it recovered.
      if (r.status === 'passed' && r.invocations > 1) {
        this.flaky.push({
          file: testResult.testFilePath,
          title: r.fullName,
          attempts: r.invocations,
        });
      }
    }
  }

  onRunComplete() {
    // Persist as JSON for later aggregation, not stdout noise.
    fs.writeFileSync('flaky-jest.json', JSON.stringify(this.flaky, null, 2));
    if (this.flaky.length) {
      console.warn(`Detected ${this.flaky.length} flaky test(s) via retry.`);
    }
  }
}
module.exports = FlakyReporter;
// Trade-off: writing a file per shard is cheap, but you must merge
// shard outputs before computing a real flake rate (CI cost is low).

3. Register the reporter and fail the build on regressions #

Wire the reporter into Jest config alongside the default reporter so you keep normal output, then decide whether a newly flaky test should fail the build.

// jest.config.js
module.exports = {
  setupFilesAfterEach: ['<rootDir>/jest.setup.js'],
  reporters: ['default', '<rootDir>/flaky-reporter.js'],
};
// Trade-off: keeping CI green on retry keeps velocity, but pair it with
// a budget check (step below) so flakiness cannot grow unbounded.

4. Gate on a flaky budget #

Add a tiny post-run check that compares the detected flaky count against a budget. This keeps retries from becoming a permanent crutch.

// check-flaky-budget.js
const flaky = require('./flaky-jest.json');
const BUDGET = 3; // max tolerated flaky tests per run
if (flaky.length > BUDGET) {
  console.error(`Flaky budget exceeded: ${flaky.length} > ${BUDGET}`);
  process.exit(1); // fail the job so the regression is addressed
}
// Trade-off: a hard budget surfaces drift early but can block merges;
// start lenient, then ratchet down as you stabilize the suite.

Merging per-shard reports before counting prevents a sharded run from undercounting flakes.

Pitfalls #

Treating retries as a fix: passing on retry hides the bug. Mitigation: always emit a record and review it; never let a green retry close the loop.
Setting retryTimes too high: 5+ retries can turn a 90%-failing test green. Mitigation: cap at 1-2 and rely on the reporter to flag recurrence.
Losing data across shards: each worker writes its own JSON. Mitigation: merge all flaky-jest.json artifacts before computing rates.
Order-dependent flakiness: a test only flakes after another runs first. Mitigation: run with --randomize periodically so retries surface ordering bugs.
Confusing invocations with numPassingAsserts: only invocations > 1 indicates a retry. Mitigation: assert on the attempt count, not assertion count.

A budget keeps retries from becoming a permanent crutch — start lenient, then ratchet down.

Reliability targets #

Metric	Target
Retry count (`retryTimes` n)	1-2
Flaky tests detected per run	≤ 3 (budgeted)
Pass-on-retry rate per suite	< 1%
Time-to-quarantine after detection	< 1 sprint
CI pass rate (post-retry)	≥ 99%

Low retry counts plus a budget keep the pass-on-retry rate under control.

Frequently Asked Questions #

Q: Does jest.retryTimes hide real failures? A: It can, which is why detection matters. A test that fails all retries still fails the build; only pass-on-retry outcomes are softened, and the reporter records each one so they are never truly hidden.

Q: Where do I call jest.retryTimes? A: In a setupFilesAfterEach file so it executes before each test file. It is a global and has no effect if called from inside an individual test body.

Q: How do I turn detection into a trend? A: Persist the reporter’s JSON per run and aggregate it over time. See tracking test flakiness trends over time for the rolling-window approach.

Why Unit Retries Deserve Scepticism #

A Jest test runs in one process with no browser and no network, which narrows the possible sources of non-determinism considerably: order dependence, an unrestored spy, a module-level cache, a real timer, or a genuine defect. Every one of those is reproducible given the same seed and worker count.

That changes what a retry means at this level. In an end-to-end suite a retry absorbs environmental noise nobody controls; in a unit suite it hides a bug that is findable in minutes. A test needing retryTimes to stay green is telling you which of the five sources is present, and the productive response is to identify it rather than to configure around it.

The configuration that removes several sources at once is cheap and belongs in the config rather than in individual files, so a spec written next quarter inherits it:

// jest.config.js — remove the causes rather than retrying the symptom.
// Trade-off: enabling these fails tests that relied on a spy surviving between
// tests, which is exactly the coupling worth removing.
module.exports = {
  restoreMocks: true,     // spies uninstalled after each test
  resetModules: true,     // fresh registry per file, clearing memoised singletons
  randomize: true,        // shuffle so order dependence fails fast
};

Where retryTimes earns its place is at the integration boundary — a test starting a container, binding a port, waiting on a local service — and it should be scoped to that project rather than applied across the suite.

Making the Retry Visible When You Do Use It #

Jest does not surface retried tests in its default output the way a browser-focused runner does, which means a retry configured here is invisible unless something records it. That invisibility is the real risk: the suite stabilises, the rate is never measured, and the underlying instability accumulates unobserved.

A custom reporter closes the gap. It receives each test result along with its invocation count, so recording the tests that needed more than one attempt is a short addition, and it produces the same rows any other runner’s extraction would.

// A minimal reporter that records rescued failures for the flakiness store.
// Trade-off: a reporter is more code than reading the JSON output, and it is
// the only place Jest exposes the retry count reliably.
class FlakyReporter {
  onTestResult(_test, result) {
    for (const t of result.testResults) {
      if (t.invocations > 1 && t.status === 'passed') {
        appendFlaky({ testId: `${result.testFilePath}::${t.fullName}`, attempts: t.invocations });
      }
    }
  }
}
module.exports = FlakyReporter;

With that in place the same discipline applies as anywhere else: every rescue is counted, classified by its error signature, and attributed to an owner. A retry that nobody records is not a stabilisation measure — it is a decision not to know.

Scope Retries to Where They Belong #

retryTimes applies within a file, which makes it easy to enable broadly and hard to reason about later. Applying it in a setup file scoped to the integration project — and leaving the pure unit project at zero — keeps the distinction visible in configuration rather than in someone’s memory.

Recording the retry count per test, rather than only whether the run passed, is what makes the setting observable. Without that record a suite can be quietly relying on retries for months while its reported pass rate looks unchanged.

Treat any need for retries here as a question rather than a setting, and answer it with a shuffled run before reaching for configuration.