Detecting Flaky Tests with Vitest

Only a pass-on-retry is the flake signal worth recording — neither a clean pass nor a hard failure.

Root cause #

The retry does not fix the flake — it reveals that the outcome depends on timing or order.

A Vitest test that fails its first attempt and passes on a retry is non-deterministic by definition — in a jsdom or node environment the usual drivers are leaked module state, an unmocked timer, or an order-dependent global that a prior test left dirty. The retry does not fix any of that; it only proves the outcome depends on timing or order. The danger is Vitest’s default silence: once a retry passes, the run is green and you never learn the test is flaky.

Making retry a detection tool means keeping the retries so CI stays unblocked while emitting a record every time one was needed. Vitest exposes retryCount on each task’s result, so a reporter can collect the exact set of tests that only passed after a retry and feed it into your historical flakiness tracking rather than letting it vanish.

Step-by-step fix #

1. Enable retries in the config #

Set retry in vitest.config.ts, scoped to CI so local runs stay strict and surface failures immediately.

Retries are a CI safety net, not a local crutch — keep them off while developing.

// vitest.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    retry: process.env.CI ? 2 : 0, // trade-off: 2 keeps CI unblocked; higher masks deeper bugs
    reporters: ['default', './flaky-reporter.ts'],
  },
});

2. Record pass-on-retry from a custom reporter #

A Vitest reporter receives the finished task tree. Walk it and collect any test whose result state is pass but whose retryCount is greater than zero.

// flaky-reporter.ts — collect tests that only passed after a retry
import type { Reporter } from 'vitest/node';
import { writeFileSync } from 'node:fs';

export default class FlakyReporter implements Reporter {
  onFinished(files = []) {
    const flaky: { file: string; name: string; retries: number }[] = [];
    const walk = (tasks, file) => {
      for (const t of tasks) {
        if (t.type === 'suite') walk(t.tasks ?? [], file);
        // retryCount > 0 with a passing result means it recovered on retry.
        else if (t.result?.state === 'pass' && (t.result.retryCount ?? 0) > 0) {
          flaky.push({ file, name: t.name, retries: t.result.retryCount });
        }
      }
    };
    for (const f of files) walk(f.tasks ?? [], f.filepath);
    writeFileSync('flaky-vitest.json', JSON.stringify(flaky, null, 2)); // persist for aggregation
  }
}

3. Gate on a flaky budget #

Compare the detected count against a budget so retries cannot become a permanent crutch, exactly as with the Jest retryTimes approach.

// check-budget.js — fail the job if flakiness exceeds the budget
const flaky = require('./flaky-vitest.json');
const BUDGET = 3;
if (flaky.length > BUDGET) {
  console.error(`Flaky budget exceeded: ${flaky.length} > ${BUDGET}`);
  process.exit(1); // block the merge so the regression is addressed
}

Pitfalls #

Use retry for flake detection; repeats is a different tool.

Treating a retry-pass as clean. It hides the non-determinism. Mitigation: always emit and review the record.
Setting retry too high. Five retries can green a 90%-failing test. Mitigation: cap at 1–2 and rely on the reporter.
Ignoring retryCount across shards. Each worker reports separately. Mitigation: merge the JSON files before counting.
Order-dependent flakiness. A test only flakes after another runs first. Mitigation: run with a shuffled sequence periodically to surface ordering bugs.
Confusing retry with repeats. repeats runs a passing test again; retry re-runs a failed one. Mitigation: use retry for flake detection.

Reliability targets #

A budgeted flaky count keeps retries from becoming a permanent crutch.

Metric	Target
Retry count (`retry`)	1–2 on CI, 0 locally
Flaky tests detected per run	≤ 3 (budgeted)
Pass-on-retry rate per suite	`< 1%`
Time-to-quarantine after detection	`< 1 sprint`
CI pass rate (post-retry)	`≥ 99%`

Frequently Asked Questions #

Does Vitest retry hide real failures? Only pass-on-retry outcomes are softened; a test that fails every attempt still fails the build. The reporter records each recovery so nothing is truly hidden.

Where do I read the retry count? On each task’s result.retryCount in a custom reporter’s onFinished. A passing result with retryCount > 0 is your flake signal.

How is this different from the Jest approach? The mechanism is identical — enable retries, capture the recovered set, budget it. Only the reporter API differs; see the Jest retryTimes guide.

Concurrency Inside a File Changes the Rules #

Vitest can run tests within a file concurrently, which is excellent for speed and removes an assumption many suites rely on without stating it: that tests in a file run one at a time.

The assumption shows up as setup-then-consume pairs — a test that creates something and a following test that asserts on it — and as shared module state that happens to work because only one test touches it at a time. Both break the moment concurrency is enabled, and the failures look like races because that is exactly what they are.

The productive reading is that concurrency exposes coupling rather than creating it. A test that cannot run alongside its neighbours could not have been run alone either, and the fix — each test creating what it needs, with unique keys — makes the suite both parallelisable and independently debuggable.

Where a genuine multi-step sequence must be verified, expressing it as one test with several steps is clearer than splitting it into tests that secretly depend on order. A journey is one behaviour; splitting it for a more granular report buys nothing and creates an ordering dependency.

// Concurrency surfaces coupling; unique data removes it.
// Trade-off: each test does its own setup, which is slightly more code and
// makes every test runnable alone.
test.concurrent('creates an invoice', async () => {
  const ref = `INV-${crypto.randomUUID()}`;
  await createInvoice({ reference: ref });
  expect(await findInvoice(ref)).toBeDefined();
});

Retries at the Unit Level Are Usually a Warning #

Vitest supports retries, and using them routinely in a unit suite is worth thinking twice about, because unit failures are almost always deterministic.

A unit test runs in one process with no browser and no network. The sources of non-determinism available to it are narrow: order dependence, an unrestored spy, a module-level cache, a real timer, or a genuine defect. Every one of those is reproducible given the same seed and the same worker count, which means a retry does not absorb environmental noise — it hides a bug that would otherwise be findable.

The productive reading of a needed retry at this level is diagnostic rather than operational. If a unit test needs one, the question is which of the five sources is involved, and the shuffle-with-a-seed run answers it faster than any amount of retry tuning. Enabling restoreMocks, unstubEnvs and unstubGlobals in configuration removes several of them outright.

// vitest.config.ts — remove the causes rather than retrying the symptom.
// Trade-off: turning these on fails tests that depended on a spy surviving
// between tests, which is precisely the coupling worth removing.
test: {
  retry: 0,
  restoreMocks: true,
  unstubEnvs: true,
  unstubGlobals: true,
  sequence: { shuffle: true },
},

Where retries genuinely help is at the integration boundary — a test starting a container, binding a port, or waiting for a local service — and there the retry should be scoped to that project rather than applied to the whole suite.

Measuring Instead of Retrying #

If a unit test is suspected of being flaky, the fastest path to certainty is repetition rather than observation. Running it a few hundred times takes seconds at this level, because there is no browser to start, and the result is a rate rather than an impression.

Two variations make the measurement informative. Repetition with shuffling exposes order dependence: a rate that changes with the seed is a data dependency between tests rather than a race. Repetition at high concurrency exposes shared-resource assumptions — a temporary file, a fixed port, a module-level counter — which are invisible when only one copy runs.

Once a rate exists, verification of a fix becomes possible: the same command, the same count, before and after. That closes the loop that otherwise leaves a unit test being “fixed” repeatedly, since a single green run after a change proves almost nothing about a test that was failing a small percentage of the time.

Recording those measurements alongside the change — the rate before, the rate after, the seed used — is what keeps the reopen rate low and turns unit flakiness from a recurring nuisance into a closed piece of work.

Prefer Configuration to Per-File Hooks #

Reset settings expressed in configuration protect every file including the ones not yet written, which is what keeps a unit suite deterministic as it grows.

Where a retry is genuinely warranted at the integration boundary, scoping it to that project rather than the whole suite keeps the unit tests strict — and keeps the retry count meaningful as a signal about infrastructure rather than about test quality.