CI Environment & Browser Drift

Prerequisites #

Pinning is only half of the work — the other half is measurement, which is why drift belongs in the same reporting pipeline as everything else in Flaky Test Detection & Quarantine Engineering.

Step-by-step implementation #

1. Pin the browser, not a range #

Playwright ships browser binaries tied to the runner version, so a caret range in package.json upgrades Chromium on whatever day the lockfile is refreshed. Pin the exact version and let a bot propose upgrades as reviewable pull requests.

{
  "devDependencies": {
    "@playwright/test": "1.49.1",
    "cypress": "13.17.0"
  }
}

# Trade-off: pinning avoids surprise upgrades but means security and bug fixes
# arrive only when someone bumps deliberately — pair it with a scheduled bot PR.
npx playwright install --with-deps chromium   # installs the build this version pins
npx playwright --version                      # echo into the CI log for the record

The follow-through for containers, where the browser lives in the image rather than in node_modules, is in Pinning Browser Versions in CI Containers.

2. Fix time zone and locale in configuration #

Date handling is the single largest source of environment-dependent failures, because a test written at UTC+2 and run at UTC crosses a day boundary for any timestamp within two hours of midnight.

// playwright.config.ts
// Trade-off: forcing one zone makes tests deterministic but stops them from
// catching real zone bugs — add a second project for a non-UTC zone instead.
export default defineConfig({
  use: {
    timezoneId: 'UTC',
    locale: 'en-GB',
    viewport: { width: 1280, height: 720 },
  },
  projects: [
    { name: 'utc', use: { timezoneId: 'UTC' } },
    { name: 'berlin', use: { timezoneId: 'Europe/Berlin' } },
  ],
});

Running two projects turns the drift into a covered case: the zone-sensitive bug fails on purpose in the second project rather than by accident in six months. Fixing Timezone and Locale Dependent Test Failures works through the assertion patterns that stay correct in both.

3. Set concurrency from the runner, not from the laptop #

Timeouts are budgets against available CPU. A suite tuned on eight cores and run on two gets roughly a quarter of the compute per worker, and every timing-sensitive assertion tightens accordingly.

// playwright.config.ts
// Trade-off: fewer workers means a longer wall clock but far more headroom per
// test; over-subscribing a 2-core runner is the most common cause of "CI-only" timeouts.
export default defineConfig({
  workers: process.env.CI ? 2 : undefined,   // undefined = half the local cores
  timeout: process.env.CI ? 60_000 : 30_000, // CI runners are slower, be explicit
});

If timeouts have to be raised for CI, treat that as a measurement rather than a fix — the underlying wait is the real problem, as covered in Network Latency & Volatility Handling.

4. Record the environment with every run #

Drift is only diagnosable if the environment is part of the test record. Emit a fingerprint alongside the results so a failure can be correlated with an image change.

// scripts/env-fingerprint.js
// Trade-off: one extra artifact per run, a few hundred bytes — cheap insurance
// against "it started failing on Tuesday and nobody changed the tests".
import { writeFileSync } from 'node:fs';
import { cpus, platform, release } from 'node:os';

writeFileSync('env-fingerprint.json', JSON.stringify({
  node: process.version,
  platform: `${platform()} ${release()}`,
  cpus: cpus().length,
  tz: Intl.DateTimeFormat().resolvedOptions().timeZone,
  locale: Intl.DateTimeFormat().resolvedOptions().locale,
  image: process.env.RUNNER_IMAGE ?? 'unknown',
}, null, 2));

5. Separate the two questions a CI-only failure raises #

Every failure that reproduces in CI and not locally is really two questions wearing one coat: is the environment different? and does the test depend on that difference? Answering them in order saves most of the debugging time, because the first is cheap and the second is not.

The first question is answered by the fingerprint. Compare the browser build, Node version, zone, locale, core count and image digest between the two runs; if any differ, you have a candidate cause before reading a single line of test code. The second is answered by removing the difference: run locally in a container built from the same digest, with the same worker count and the same zone. If the failure follows you, the test depends on the environment and the fix belongs in the test or in the configuration. If it does not follow, the difference is elsewhere in the machine — usually load, disk speed or a network route — and the fix belongs in the runner.

# Reproduce CI locally rather than guessing at the delta.
# Trade-off: running the suite in a container is slower than running it natively
# and is the only way to compare like with like.
docker run --rm -it \
  -v "$PWD":/app -w /app \
  -e TZ=UTC -e CI=true \
  --cpus 2 --memory 4g \
  mcr.microsoft.com/playwright@sha256:9f2c... \
  npx playwright test --workers=2

Constraining CPU and memory matters more than most teams expect. A suite tuned on an unconstrained laptop and run under a two-core, four-gigabyte limit sees longer paint times, slower module evaluation and more garbage-collection pauses — enough to turn a comfortable assertion into a marginal one.

6. Treat the runner as a dependency with an owner #

Environment drift keeps recurring in teams where the image is nobody’s responsibility. The image is a dependency of the test suite exactly as the framework is: it has a version, a changelog and an upgrade cost, and someone has to own bumping it. Naming that owner — and making image bumps their own pull requests rather than side effects of unrelated work — is what turns a recurring class of flakiness into scheduled maintenance.

The practical marker of ownership is the fingerprint history. If you can answer “what changed in the runtime between last Tuesday and today?” in under a minute, the environment is owned. If the answer requires archaeology across three pipelines, it is not, and the next drift incident will cost the same as the last one.

Configuration reference #

Data-driven analysis #

• Local-versus-CI delta. The share of tests that pass locally and fail in CI. A non-zero, stable delta means the environments differ in a way the config does not capture; a delta that jumps on one day means something in the image moved.
• Failure onset alignment. For every new failure, compare the first failing run against the last image change and the last merge. Alignment with the image is drift and belongs to the platform owner; alignment with a merge is a regression and belongs to the author.
• Timeout headroom. Median test duration divided by the configured timeout. Below 20% means comfortable headroom; above 50% means the next slow runner turns the suite red. Track this per project rather than per test.
• Environment uniqueness. Count distinct fingerprints seen in a week. More than one browser build or Node version across runs of the same branch means pinning is incomplete.

Which differences actually break tests #

Not every environmental difference is worth eliminating, and chasing all of them is its own kind of waste. Ranked by how often they cause a real failure:

Viewport and window size come first by a wide margin. They decide what is on screen, and therefore what can be clicked. A test that fails to find a button in CI is far more likely to be looking at a 1280-pixel-wide layout than to be racing anything. Pin it in configuration and the class largely disappears.

Time zone and locale come second, and they have the sharpest signature: failures concentrated in a window of the day, or on particular calendar dates. They are cheap to eliminate and expensive to diagnose from scratch, which makes them the best return on a few lines of configuration.

CPU and memory limits come third. They do not cause failures directly; they shrink the margin on every timing-sensitive assertion until an already-marginal wait tips over. The signature is a suite that fails in different places each run, with timeouts rather than assertion errors — which is why raising the timeout appears to work and then stops working.

Browser build comes fourth in frequency but first in blast radius. When it does cause failures, it causes many at once, across unrelated features, starting on a specific day. That signature — a wave of unrelated failures with a sharp onset and no matching merge — is close to diagnostic on its own.

Fonts and GPU compositing matter only to suites doing visual comparison or asserting positions. For everything else they are noise, and hardening tests against them is better spent effort than making the image pixel-identical to a laptop.

Common pitfalls & mitigation strategies #

• Using :latest or a floating tag for the runner image. The image changes without a commit. Mitigation: reference by digest and bump deliberately.
• A caret range on Playwright or Cypress. The browser upgrades with a lockfile refresh. Mitigation: pin the exact version and take upgrades as reviewed pull requests.
• Assuming the runner is in UTC. Many are, some are not, and Node reads TZ independently of the browser context. Mitigation: set TZ in the container and timezoneId in the browser config.
• Tuning worker counts on a developer laptop. Eight local cores hide the over-subscription that a 2-core runner exposes. Mitigation: set workers explicitly under process.env.CI.
• Raising timeouts until CI is green. This buys time and hides the trend until the next slow week. Mitigation: fix the wait, and treat timeout changes as data.
• Ignoring font availability. Missing fonts change layout, which changes visual comparisons and element positions. Mitigation: install the same font package in the image and assert on roles, not pixel positions.

Budgeting for the environment you actually have #

Most timing configuration is written against the machine the author used, and the mismatch shows up as a class of failure that no amount of test-level fixing resolves. Two budgets are worth setting explicitly.

The first is the per-worker CPU budget. Divide the runner’s cores by the worker count and treat the result as the compute each test gets. Below roughly one core per worker, browser-driven tests spend measurable time waiting for the scheduler rather than for the application, and every wait in the suite tightens. Two workers on a two-core runner is a reasonable ceiling; four workers on the same runner reliably produces timeouts that look like application slowness.

The second is the timeout headroom budget. Take the median duration of the assertions in a suite and compare it to the configured timeout. A median at 10–20% of the timeout is comfortable; a median above half the timeout means the suite is one slow runner away from red, and raising the timeout only moves the cliff. When headroom is thin, the fix is a better wait condition rather than a bigger number — the waiting patterns in DOM Mutation & Rendering Races are what buy headroom back without hiding regressions.

Both budgets are worth recording next to the suite rather than discovered during an incident, because they are the numbers that explain why a suite that is green on a laptop is marginal on a runner with a quarter of the compute.

Frequently Asked Questions #

Q: How do I tell environment drift apart from state leakage? A: Leakage follows the shuffle seed; drift follows the machine. Re-run the failing test alone on the same runner: if it still fails there and passes on a different image, it is drift. If it passes alone on the same runner and fails in-suite, it is leakage, and Test Isolation & State Leakage is the right starting point.

Q: Is it acceptable to force TZ=UTC everywhere? A: As a baseline, yes — but a suite that only ever runs in UTC will never catch a genuine time-zone bug in the product. Keep UTC as the default project and add one non-UTC project so zone handling stays covered on purpose.

Q: The runner is slower than a laptop. Should timeouts simply be higher in CI? A: A higher CI timeout is reasonable as an acknowledgement that the machine is slower — the mistake is using it as the fix for a marginal wait. Set the CI timeout once, from measured headroom rather than by trial and error, and treat any subsequent increase as a signal that a wait condition has degraded. A suite whose timeouts have been raised three times is telling you about its waits, not about its runners.

Q: How much of this matters for unit tests, or is it an end-to-end concern? A: Time zone, locale and CPU limits apply to unit suites too — date formatting and worker over-subscription do not care whether a browser is involved. What is specific to browser-driven suites is the browser build, the viewport and the font set. A Node-only suite therefore needs the container digest, TZ, LANG and a sane worker count, and can ignore the rest.

Q: How often should the browser version be bumped? A: On a schedule you control, not on a lockfile refresh. Monthly is a reasonable cadence: a bot opens the pull request, the suite runs against the new build, and any breakage is attributed to the upgrade rather than discovered days later mixed in with feature work.

Q: Can we just record the environment and skip the pinning? A: Recording without pinning tells you what changed after the fact, which is genuinely useful during an incident and does nothing to prevent the next one. Pinning without recording prevents drift and leaves you unable to prove it when something still differs. The two are complementary and cheap; do both, and the category stops consuming debugging time.

Related guides #

• Fixing Timezone and Locale Dependent Test Failures
• Headless vs Headed Browser Differences
• Pinning Browser Versions in CI Containers
• Test Isolation & State Leakage
• Network Latency & Volatility Handling