CI-first knowledge base Reliability Engineering

Article · Root Causes of JavaScript Test Flakiness

Testing Offline and Connection Loss States

Offline handling is the part of an application that is written once, demonstrated by unplugging a cable, and then never tested again — which is why the retry banner still says "Loading…" three releases later. The tests that do exist are usually the flakiest in the suite, because they simulate a dropped connection with a sleep and a hope. This guide extends Network Latency & Volatility Handling with deterministic ways to take the network away, put it back, and assert on what the interface did in between.

12 sections URL: /root-causes-of-javascript-test-flakiness/network-latency-volatility-handling/testing-offline-and-connection-loss-states/
Four distinct failure shapes people call "offline" Connection refused, request timeout, partial response and captive-portal style wrong response each produce different application behaviour. connection refusedfetch rejects immediatelyerror path runs fast request hangsno response, no rejectionspinner forever unless timed out partial responsestream cut mid-bodyJSON parse error path wrong responseportal or proxy returns HTMLlooks like success, is not an application that handles one of these gracefully may hang on the other three
"Offline" is four different code paths; testing only the first leaves the three that actually strand users in production.

Root cause #

navigator.onLine is not a network test. It reports whether the operating system believes an interface is up, so it stays true behind a captive portal, on a connection with no route, and on a Wi-Fi network whose upstream has failed. An application that gates its retry logic on that flag therefore behaves correctly in a demo — where you disable the adapter — and incorrectly in the situations users actually hit, where the interface is up and the requests fail anyway. Tests written against the flag inherit the same blind spot.

The second mechanism is that a lost connection is not one event. A refused connection rejects the promise immediately; a black-holed route leaves the request pending until something times it out; a connection dropped mid-response delivers a truncated body that fails to parse; and a proxy or portal returns a perfectly valid HTTP response containing the wrong content entirely. These run four different branches of the application, and only the first is exercised by the naive “go offline” test.

The third is timing. Tests that simulate connection loss with a delay and a toggle are racing the application’s own retry schedule: if the retry fires before the test restores the network, the assertion sees a second failure rather than a recovery. Deterministic offline testing therefore requires control over which request fails and when it is allowed to succeed, not just a global switch — which is exactly what request interception provides.

Step-by-step fix #

1. Fail a specific request, not the whole context #

Aborting one route models a partial outage precisely, and leaves the rest of the page working — which is the realistic case and the one that exposes bugs.

// Trade-off: aborting one route is precise but only covers requests you name;
// a context-wide offline switch is blunter and catches requests you forgot.
await page.route('**/api/invoices*', (route) => route.abort('connectionfailed'));
await page.goto('/invoices');
await expect(page.getByRole('alert')).toHaveText(/could not load invoices/i);
await expect(page.getByRole('button', { name: 'Retry' })).toBeEnabled();

route.abort() accepts an error code, so you can distinguish a refused connection from a name-resolution failure and check that the interface does not present a DNS problem as a server error.

2. Model the hang, which is the branch that strands users #

A request that never resolves is the failure mode that produces an eternal spinner. Simulate it by taking the route and never fulfilling it, then assert that the application gives up on its own.

// Trade-off: this test must not wait for the app's real timeout if that is
// 60 s — shorten the client timeout in test config, or the suite pays for it.
await page.route('**/api/invoices*', () => { /* never resolve */ });
await page.goto('/invoices');

// The application must impose its own deadline; without one this assertion
// is the test that proves users would wait forever.
await expect(page.getByRole('alert')).toBeVisible({ timeout: 15_000 });

If this test fails because nothing ever appears, that is not a flaky test — it is the bug, and it is the single most valuable result this guide produces.

Abort versus hang versus recover An aborted request runs the error path immediately, a hanging request needs a client-side deadline, and a restored route lets the retry succeed. abortrejects now error banner appears within one render — cheap to assert hangnever resolves only the client's own deadline ends the spinner restore routethen retry recovery path asserted deterministically, no sleeps
Three routes, three branches: the middle one is where an application without a client-side deadline strands its users.

3. Restore the network on your terms and assert recovery #

Recovery is half the feature and is almost never tested. Fail the first attempt, then let the retry through, and assert that the interface returns to a working state without a reload.

// Trade-off: a counter in the handler makes the scenario explicit and is
// stateful — reset it per test or it leaks into the next one.
let attempts = 0;
await page.route('**/api/invoices*', async (route) => {
  attempts += 1;
  if (attempts === 1) return route.abort('connectionfailed');
  return route.fulfill({ status: 200, json: { invoices: [{ id: 'INV-1' }] } });
});

await page.goto('/invoices');
await page.getByRole('button', { name: 'Retry' }).click();
await expect(page.getByRole('row', { name: /INV-1/ })).toBeVisible();

That handler state is exactly the kind of shared mutable value discussed in Test Isolation & State Leakage — declare it inside the test, never at module scope.

4. Use the context-wide switch for the whole-app case #

For a genuine “the device lost connectivity” scenario, Playwright can take the whole context offline, which also updates navigator.onLine so the application’s own listeners fire.

// Trade-off: this is realistic for a device going offline and blunt for
// partial outages, where only some hosts are unreachable.
await context.setOffline(true);
await expect(page.getByRole('status')).toHaveText(/you are offline/i);
await context.setOffline(false);
await expect(page.getByRole('status')).toBeHidden();

In Cypress the analogous control is an intercept that forces a network error, since the runner has no context-level offline switch:

cy.intercept('GET', '/api/invoices*', { forceNetworkError: true }).as('down');
cy.visit('/invoices');
cy.findByRole('alert').should('contain.text', 'could not load');

5. Cover the truncated and wrong-content cases #

The two remaining shapes are cheap to add once the interception is in place, and they catch parser and validation bugs that no amount of “abort” testing reaches.

// A body that is cut off mid-JSON, and a portal-style HTML response.
// Trade-off: these are unusual cases, and they are precisely the ones whose
// error handling nobody has ever run before a real incident.
await page.route('**/api/invoices*', (route) =>
  route.fulfill({ status: 200, contentType: 'application/json', body: '{"invoices":[{"id":' }));

await page.route('**/api/invoices*', (route) =>
  route.fulfill({ status: 200, contentType: 'text/html', body: '<html>Sign in to Wi-Fi</html>' }));

6. Decide what “handled” means before writing the assertion #

A test can only assert a contract that exists, and offline behaviour is usually specified loosely — “show an error” — which produces assertions that pass against a blank screen. Write the contract down first, in three parts: what the user sees, what the application does next, and what happens to unsaved work.

For a read request the contract is usually: an error region announced to assistive technology, a retry affordance, and the previously loaded data left on screen rather than cleared. For a write request it is stricter, because there is state at stake: the pending change must survive the failure, the retry must not duplicate it, and the interface must not report success. That second contract is where offline handling most often fails silently — the request never reached the server, the optimistic update stayed on screen, and the user believes the work is saved.

// Assert all three parts of the contract, not just the banner.
// Trade-off: three assertions per scenario is more verbose and is what stops
// "handled" from meaning "the spinner stopped".
await expect(page.getByRole('alert')).toContainText(/could not save/i);
await expect(page.getByRole('button', { name: 'Retry' })).toBeEnabled();
await expect(page.getByLabel('Notes')).toHaveValue('draft text kept');  // work not lost

The idempotency question that follows — whether the retry can safely repeat the request — is the subject of Retrying Idempotent Requests Without Masking Flakiness.

Pitfalls #

  • Testing navigator.onLine instead of request outcomes. The flag is true behind a captive portal and on a dead route. Mitigation: assert on what happened to the request, not on the flag.
  • Only testing the immediate-rejection case. The hang is the branch that strands users. Mitigation: add a route that never resolves and require a client-side deadline.
  • Restoring the network with a sleep. The retry schedule and the sleep race each other. Mitigation: control the network per request with a counter in the handler.
  • Handler state at module scope. The attempt counter leaks into the next test. Mitigation: declare it inside the test body.
  • Waiting out a 60-second production timeout in the suite. One test dominates the run. Mitigation: make the client timeout configurable and shorten it under test.
  • Assuming a 200 means success. A portal or proxy returns valid HTTP with the wrong body. Mitigation: validate the content type and shape, then test that path.
Coverage checklist for connection loss Five scenarios: refuse, hang, truncate, wrong content and recovery. refuseerror banner hangclient deadline truncateparse failure wrong bodycontent-type check recoverno reload needed five short tests per critical request — the recovery case is the one users notice most
Each scenario is a few lines once interception is in place, and each maps to a different branch of the application's error handling.

Reliability targets #

Metric Target Notes
Critical requests with an offline test 100% Refuse + hang at minimum
Requests with a client-side deadline 100% Proven by the hang test
Recovery paths asserted 100% of retryable requests Retry succeeds without a reload
Offline tests using fixed sleeps 0 Control the route, not the clock
Suite time spent in offline tests < 5% Short client timeouts under test
Connection-loss scorecard Targets for offline coverage, client deadlines, recovery assertions and sleep-free tests. 100%critical reqs covered 100%client deadlines 100%recovery asserted 0sleeps
The hang test is the one that pays for itself: it either proves a deadline exists or finds the spinner that never ends.

Frequently Asked Questions #

Q: Should offline tests run against a real backend? A: No. The whole point is controlling exactly which request fails and when it recovers, which a real backend cannot give you without an outage. Interception is the correct tool; keep the real-backend coverage for the happy path.

Q: How do I test an application that caches responses in a service worker? A: Decide which behaviour you are asserting. For “works offline from cache”, prime the cache with a first online visit, then take the context offline and assert the cached view renders. For “shows an error when there is nothing cached”, clear the caches first — the clearing sequence is in Clearing Browser Storage Between Tests. Mixing the two in one test is how these specs become flaky.

Q: Our client retries automatically. How do I keep that from making tests non-deterministic? A: Count the attempts in the route handler and decide the outcome per attempt, rather than toggling a global switch on a timer. That makes the retry behaviour itself the thing under test — you can assert that exactly three attempts were made with backoff, instead of hoping the timing lines up.

Q: Is it worth testing the captive-portal case? A: If your users are ever on hotel, airport or conference networks, yes. It is the case where the application receives a 200 with an HTML body and, without a content-type check, renders a parse error or silently shows empty data. One test costs a few lines and covers a support-ticket category that is otherwise impossible to reproduce on request.