Root Causes of JavaScript Test Flakiness

Asynchronous Execution & State Synchronization #

Unpredictable promise resolution, unhandled microtasks, and implicit polling are the most common flake source: the event loop defers work, and the assertion fires before the state it checks has settled. The fix is explicit await chains and asserting on a real readiness signal rather than a timeout — the full patterns live in Async State Management in E2E Tests.

DOM Mutation & Rendering Races #

Virtual DOM diffing, CSS transitions, and lazy-loaded components open timing gaps that break selector resolution — a headless browser can render frames faster than local dev, masking the instability until CI. Stable selectors and explicit stability waits close the gap; see DOM Mutation & Rendering Races.

Network Volatility & API Mocking #

Real latency, CORS preflight delays, and third-party downtime make live-endpoint tests non-deterministic by definition. Deterministic interceptors replace that variance with a fixed, reproducible response, and timeout calibration keeps the wait honest — the discipline is detailed in Network Latency & Volatility Handling.

Parallel Execution & Concurrency #

Shared state across workers, global fixtures, and non-atomic database writes cause cross-test pollution, and multi-worker defaults amplify the contention. Isolating contexts and using transactional rollbacks resolves it — the full treatment is in Race Conditions in Parallel Test Runs.

Infrastructure & Dependency Consistency #

Node version mismatches, lockfile drift, and OS-level library differences produce environment-specific failures because a laptop rarely mirrors an ephemeral CI runner. Pin versions with .nvmrc and engines, audit lockfile changes, and run tests in a container so the environment is identical everywhere.

Resource Exhaustion & Context Isolation #

Unclosed browser contexts, accumulated heap, and orphaned WebSockets degrade a runner over a long suite, so failures late in the run are memory pressure, not logic. Strict teardown hooks and heap monitoring keep the runner healthy — the same discipline as Debugging Async State Leaks in React E2E Tests.

Test Isolation & State Leakage #

The failure that most reliably wastes an afternoon is the one where the test is correct, the application is correct, and the previous test left something behind. A cookie, a localStorage key, a memoised module singleton, a spy that was never restored, a database row with a fixed identifier: each survives into the next test and changes what it sees. Because runners schedule files by worker availability rather than in a fixed order, the polluting test and its victim only share a worker on some runs, which is why the symptom is intermittent and the cause is not.

The distinguishing signal is cheap to obtain. Re-run the failing test on its own: green alone and red in the suite means leakage, and red both ways means a genuine defect or a timing race. That single execution routes the investigation correctly and stops engineers from “fixing” application code that was never broken. From there, the work is bisection — halve the set of tests that run before the victim until one file is left — and then a reset at the writer rather than a workaround at the reader.

The durable version of the fix is configuration rather than discipline. restoreMocks and resetModules in the unit runner, testIsolation left on in Cypress, per-worker database schemas, and a shuffled run with a recorded seed in CI together remove most of the category and keep it removed as the suite grows. Test Isolation & State Leakage works through each layer — browser storage, module registry, global stubs, timers and external stores — and the reset that actually clears it.

// Enforce isolation in configuration, not in each spec's afterEach.
// Trade-off: resetModules re-evaluates dependency trees per file and costs time;
// apply it where module-level state exists rather than everywhere by default.
module.exports = { restoreMocks: true, resetModules: true, randomize: true };

CI Environment & Browser Drift #

Some failures belong to the machine. A runner image rebuilt overnight, a browser binary upgraded by a lockfile refresh, a container in UTC while the author is at UTC+2, two cores where the suite was tuned on eight — each changes behaviour the tests silently depended on, and none of them appears in any commit. The signature is distinctive: a wave of unrelated failures with a sharp onset and no matching merge, or a rate that rises and falls with the working day.

Diagnosis starts by making the environment part of the test record. Emitting a fingerprint per run — Node version, browser build, core count, time zone, locale, image digest — turns “it started failing on Tuesday” into a diff. Comparing the fingerprint of a passing run against a failing one usually identifies the changed variable before anyone opens a spec file, and it assigns the work correctly: an image change belongs to whoever owns the pipeline, not to the author of the test that noticed.

The preventive half is pinning. Reference container images by digest rather than by tag, pin the test runner to an exact version so the browser cannot move with a lockfile refresh, install dependencies with a lockfile-only command, and set time zone, locale, viewport and worker count explicitly rather than inheriting them. CI Environment & Browser Drift covers the pinning, the fingerprinting and the budgets — per-worker compute and timeout headroom — that decide whether a suite is comfortable or one slow runner away from red.

// playwright.config.ts — make the environment explicit rather than inherited.
// Trade-off: forcing UTC removes a class of failures and stops the suite from
// exercising zone conversion; add a second project for a non-UTC zone.
use: { timezoneId: 'UTC', locale: 'en-GB', viewport: { width: 1280, height: 720 } },
workers: process.env.CI ? 2 : undefined,

Production Configuration Examples #

The two configs below encode the cross-cutting defaults: CI-only retries with trace capture, and elevated CI timeouts for shared-runner contention.

// playwright.config.ts — shared reliability baseline
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,   // retries mask instability — restrict to CI, keep local feedback honest
  fullyParallel: true,               // max throughput; demands strict fixture isolation
  use: { trace: 'on-first-retry' },  // capture diagnostics only on failure — low storage cost
});

// cypress.config.js — CI-aware timeouts and retries
const { defineConfig } = require('cypress');

module.exports = defineConfig({
  e2e: {
    defaultCommandTimeout: process.env.CI_NODE_INDEX ? 10000 : 4000, // CI headroom for shared runners
    retries: { runMode: 2, openMode: 0 },                            // open mode: 0 forces immediate debugging
  },
});

Common Pitfalls #

• Arbitrary hard-coded sleeps instead of explicit wait conditions.
• Sharing mutable global state across parallel workers.
• Ignoring network interception for third-party analytics and ads.
• Skipping context teardown between suites in headless browsers.
• Relying on local environment state instead of a containerized runner.

A staged plan for reducing flakiness #

Teams that succeed at this rarely start by fixing tests. They start by making the problem observable, because a flakiness effort without measurement cannot tell improvement from luck.

Stage one: measure. Record every test result, including passes, with enough run metadata to correlate later — branch, commit, image digest, worker count, hour. Without the denominator no rate exists, and without the metadata no question about why a rate moved can be answered. A single file store is enough at almost any team size, and the first query — rate per test over thirty days, worst twenty first — usually reveals that a handful of tests account for most of the pain.

Stage two: stop the inflow. Stress-run changed specs before merge, so a newly written flaky test is caught by its author rather than discovered by a colleague three weeks later. Fifty repetitions with retries disabled takes a couple of minutes and detects anything above roughly a five percent rate. This is the single highest-leverage control available, because it stops the backlog growing while everything else works through it.

Stage three: bound the existing damage. Introduce a retry budget set just above the current rate and ratchet it down, quarantine repeat offenders into a non-blocking lane rather than skipping them, and give every quarantine an owner and an expiry date. The lane matters: a quarantined test that still executes can graduate on evidence, while a skipped one can only be released on somebody’s optimism.

Stage four: work the ranked list. Prioritise by cost — frequency multiplied by how many pipelines the test blocks and how long a rerun takes — rather than by failure count alone. A test failing three times in a hundred runs on a twenty-minute release pipeline usually costs more than one failing twenty times in a forty-second unit suite.

Stage five: keep it from returning. Shuffle order on every pipeline with a recorded seed, pin the environment by digest, and treat any rise in the retry count as a regression regardless of how green the pipeline looks. The pass rate has been deliberately engineered to be uninformative once retries exist, so the retry count carries the signal instead.

Reading a failure message as evidence #

The first line of a failure carries more diagnostic weight than most teams extract from it, and learning to read it turns triage from exploration into classification.

“Timeout exceeded waiting for locator” says the condition never became true within the budget. That is compatible with a slow machine, a wait on the wrong condition, an element outside a smaller viewport, or an element still animating — but it is not compatible with the application producing a wrong answer, which rules out an entire branch of the search. Check whether the element was in view and settled before reasoning about timing.

“expected 42, received 41” is the alarming one when it passes on retry, because it means the application produced two different answers for the same input. A test cannot be non-deterministic about a value it read correctly; something upstream was. Treat this class as a probable product race until disproven, and route it to the team that owns the feature rather than logging it as test flakiness.

“expected 42, received undefined” usually means late rather than wrong: the value had not arrived when the assertion ran. That is a wait problem and belongs to the spec’s owner, and the fix is a condition tied to the data rather than a longer timeout.

“ECONNREFUSED”, “socket hang up”, “OOMKilled” are infrastructure statements. No change to the test will fix them, and counting them in a test-flakiness metric makes that metric unactionable for both the test owners and the platform team.

“Cannot read properties of undefined” in a file unrelated to the change is the signature of a leaked spy or mock: something replaced a function, returned undefined where an object was expected, and was never restored. The crash is usually several tests downstream of the writer, which is why the file it appears in is misleading.

Classifying at the point of collection, rather than during a later investigation, is what makes the routing and the metrics in Flaky Test Detection & Quarantine Engineering work at all.

Where flakiness actually comes from #

The distribution matters when deciding where to spend effort, and it is consistent enough across JavaScript suites to plan against.

Timing and waiting problems dominate by count — assertions that fire before the interface has settled, waits on conditions correlated with but not caused by the state the test needs, fixed sleeps that were tuned on a fast machine. They are also the cheapest class to fix, because the remedy is usually a better condition rather than an architectural change.

Isolation problems come second by count and first by confusion, since their symptoms appear in files unrelated to their cause. They are highly fixable and their fixes generalise: one configuration change removes a whole category rather than one instance.

Environmental problems are less frequent but have the largest blast radius. A browser upgrade or an image rebuild produces a wave of unrelated failures at once, and because the signature looks like widespread test decay it is often misdiagnosed as a quality problem and answered with higher timeouts.

Genuine intermittent product defects are the smallest group and the most expensive to miss, because a blanket retry policy hides them preferentially — they concentrate in exactly the concurrency-heavy, network-heavy areas where tests are most likely to be flaky in the first place.

The practical implication is that effort should not be spread evenly. Instrumentation first, because it tells you which of these four you actually have; then isolation, because its fixes generalise; then waiting conditions, because they are the largest count; and environmental pinning throughout, because it is cheap and prevents the misdiagnoses that waste the most time.

Choosing the level a test belongs at #

A surprising share of flakiness is a placement problem rather than a defect. The same behaviour verified at three different levels has three different exposures to non-determinism, and choosing the highest level by habit is what produces suites that are simultaneously slow and unreliable.

A unit test controls everything: no browser, no network, no shared state beyond the module registry. Failures there are deterministic almost by construction, which is why retries at that level hide bugs rather than absorbing noise. A component test adds a DOM and usually a mocked network, so it gains realism about rendering while keeping timing under control. An end-to-end test adds a real browser, a real server, real latency and shared infrastructure — every one of which is a source of the failure modes catalogued above.

The practical rule is to verify each behaviour at the lowest level that can genuinely observe it, and to reserve end-to-end coverage for the wiring between parts that no lower level sees. A pricing rule belongs in a unit test; a form’s validation behaviour belongs in a component test; the fact that submitting the form reaches the API and updates the page belongs in one end-to-end test rather than in fifteen. When a flaky end-to-end test is expensive to stabilise, re-expressing its coverage at a lower level is frequently cheaper than fixing it — and produces a faster suite as a side effect.

FAQ #

What is the primary metric for tracking test flakiness in CI? Flaky Test Rate (FTR) = (flaky failures / total executions) × 100. Target < 2% for production-grade pipelines, and stream it into Historical Flakiness Tracking & Analytics.

How do I differentiate a flaky test from a genuine bug? Reproduce with identical CI environment variables. If it passes deterministically locally but fails intermittently in CI, it is flakiness — use trace artifacts and network logs to isolate the non-determinism.

Should I auto-retry flaky tests in CI? Sparingly (max 2) with trace capture. Retries mask instability, so pair them with automated quarantine and mandatory root-cause analysis via Building Auto-Quarantine Workflows.

How does parallel execution impact flakiness? It amplifies shared-state pollution and resource contention. Mitigate with strict isolation, atomic transactions, and per-worker provisioning.

A test passes alone and fails in the suite. Which failure mode is that? State leakage, essentially always. The solo re-run is the cheapest diagnostic available and it partitions the search space in one execution: green alone means something earlier in the run wrote state this test reads, and the fix belongs at the writer rather than in the failing test. Bisecting the tests that run before it names the culprit in roughly log₂(n) runs — six rounds for a sixty-four file suite — and Test Isolation & State Leakage covers the reset for each layer that can leak.

Failures cluster between nine and eleven in the morning across dozens of unrelated tests. Why? Contention rather than any code problem. Every wait in a suite is a budget against available compute, so when runners are busiest each timing-sensitive assertion has less headroom than it was written with. Two numbers confirm it: the peak-to-off-peak failure-rate ratio, and the count of distinct tests failing — contention produces few failures across many tests, while a genuine bug produces many failures in a few. The usual cause is worker over-subscription, and matching worker count to available cores costs nothing.

How many retries should CI use, and does that change by test level? One, and yes. Unit, component, contract and shuffled-order runs should have none, because their failures are deterministic and a retry hides a real bug. Integration and end-to-end suites warrant one retry, since some non-determinism there genuinely belongs to shared infrastructure. What makes retries defensible in any case is that each rescue is recorded, classified and counted against a budget; an unreported retry converts an unreliable suite into a confidently green one.

Our pipeline is green every day. Do we still have a flakiness problem? Possibly, and you cannot tell without the retry data. The diagnostic question is whether you can say how many tests were rescued by a retry last week, and which ones, from records rather than memory. If not, greenness carries no information about the suite — it only tells you the retry mechanism is working.

Which failure mode should a team tackle first? Whichever the data names, but if the data does not exist yet, start with isolation. Order dependence is deterministic once you know the seed, so it is reproducible and genuinely fixable, and the enforcement — reset settings in configuration plus a shuffled run — removes an entire class rather than one test. Timing and environmental work is more valuable per hour spent, and it is much harder to do before the isolation noise is out of the way.

Verifying that a fix actually worked #

The step teams skip most often is the one that decides whether the work counted. A test that failed four percent of the time passes on the first attempt after a change with probability 0.96, so a single green run is almost entirely uninformative — and declaring victory on that basis is why the same test reappears a fortnight later with a fresh investigation attached.

Verification uses the same instrument that established the problem: repetition. Measure the rate before the change with a hundred repetitions at realistic concurrency and retries disabled, apply the fix, then measure again with the identical command. Two numbers, same conditions, and the comparison is meaningful. Where the failure was order-dependent rather than timing-dependent, the equivalent is replaying the recorded seed with the same worker count, since that reproduces the exact ordering that failed.

Recording both numbers in the triage note has a second benefit beyond honesty about this fix. Over time it produces a reopen rate — the share of tests that return to quarantine after being declared fixed — and that single metric exposes shallow fixes better than any review process. A team whose reopen rate is above roughly one in ten is treating symptoms, and the usual symptom being treated is a wait that was lengthened rather than corrected.

Reliability Metrics #

Related guides #

• Async State Management in E2E Tests
• DOM Mutation & Rendering Races
• Network Latency & Volatility Handling
• Race Conditions in Parallel Test Runs
• Timer & Animation Flakiness in JS Tests