Pinning Browser Versions in CI

Reproducibility is the property that all four layers are fixed; any single unpinned layer is enough to lose it.

Root cause #

A container tag is a mutable pointer. mcr.microsoft.com/playwright:v1.49.1-jammy names a version, but the tag is still a label that the publisher can move to a rebuilt image containing newer OS packages, newer fonts and newer certificates. :latest is the extreme case, but even a version-looking tag is re-pushed routinely for security rebuilds. Only a digest — image@sha256:… — refers to exactly one immutable set of bytes.

The npm layer moves for a different reason. A caret range means “any compatible minor or patch”, so the range is not consulted on every install — the lockfile is — but any operation that refreshes the lockfile silently accepts a newer runner. That matters more for browser automation than for ordinary dependencies, because the Playwright package version determines which browser build gets installed: bumping the runner from 1.49 to 1.50 replaces Chromium wholesale, along with its rendering, its default timeouts and occasionally its behaviour around focus and downloads.

The third mechanism is the install step itself. npx playwright install chromium fetches a binary from a CDN at build time. If the build does not cache by exact version, two builds of the same commit can end up with different bytes — and if the CDN is unreachable, the build fails for reasons that have nothing to do with the code. Baking the browser into a digest-pinned image removes both problems at once, which is why the official image is the shortest path to a reproducible runner.

Step-by-step fix #

1. Pin the base image by digest #

Resolve the tag to a digest once, commit the digest, and let a bot propose new digests as reviewable changes.

# Trade-off: a digest is fully reproducible but stops receiving security
# rebuilds silently — you must bump it deliberately, so schedule that work.
docker buildx imagetools inspect mcr.microsoft.com/playwright:v1.49.1-jammy \
  --format '{{.Manifest.Digest}}'
# sha256:9f2c…  ← commit this

# Dockerfile — the digest, not the tag, is what makes this reproducible
FROM mcr.microsoft.com/playwright@sha256:9f2c...
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci                      # lockfile only — never npm install in CI
COPY . .

npm ci matters as much as the digest: it installs exactly what the lockfile says and fails if package.json and the lockfile disagree, whereas npm install will happily resolve a newer version and rewrite the lockfile inside the build.

2. Pin the runner version exactly, and match it to the image #

The Playwright package version and the image version must agree. A mismatch produces the “Executable doesn’t exist” class of failure, or worse, silently uses a browser the runner was not tested against.

{
  "devDependencies": {
    "@playwright/test": "1.49.1"
  }
}

# Trade-off: exact pins mean more upgrade pull requests, which is the point —
# each one is a small, attributable change instead of an invisible one.
npx playwright --version          # 1.49.1
npx playwright install --dry-run  # shows the browser builds this version expects

For Cypress the equivalent is an exact cypress version plus the matching cypress/included image, because the bundled Electron and Chrome builds travel with the Cypress release rather than being installed separately.

The runner and the image are one unit; upgrading either alone is what produces the confusing executable errors.

3. Verify the runtime at the start of every run #

A pin you do not verify is a pin you find out about during an incident. Print the versions as the first step of the test job and fail fast on a mismatch.

# Trade-off: a hard failure on drift is noisier than a warning, and that noise
# is exactly what stops a silent upgrade from being blamed on the tests.
set -euo pipefail
EXPECTED_RUNNER="1.49.1"
ACTUAL_RUNNER="$(npx playwright --version | awk '{print $2}')"
[ "$ACTUAL_RUNNER" = "$EXPECTED_RUNNER" ] || {
  echo "::error::runner drift — expected $EXPECTED_RUNNER, got $ACTUAL_RUNNER"; exit 1;
}
node -e "console.log('node', process.version, '| cpus', require('os').cpus().length)"

Attach the output to the run record so a failure weeks later can be correlated with the exact runtime, in the same way the fingerprint artefact described in CI Environment & Browser Drift is used.

4. Take upgrades as their own change #

The point of pinning is not to stay on an old browser forever — it is to make the upgrade a discrete, attributable event. Schedule a bot to open the bump, run the full suite against it, and treat any failure as information about the upgrade rather than as flakiness.

# .github/workflows/browser-bump.yml
# Trade-off: a monthly cadence keeps the delta small and reviewable; quarterly
# batches several browser releases into one pull request that is hard to bisect.
on:
  schedule:
    - cron: '0 6 1 * *'      # first of the month
jobs:
  bump:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Bump the runner and re-resolve the image digest
        run: |
          npm install --save-exact @playwright/test@latest
          VERSION="$(node -p "require('./package.json').devDependencies['@playwright/test']")"
          echo "new runner: $VERSION"

When the upgrade pull request is red, the failures are a list of behaviours the new browser changed — a genuinely useful artefact, and one you only get if the upgrade is isolated from feature work.

5. Cache by version, never by “latest” #

If you install browsers at build time rather than using a pre-built image, key the cache on the exact runner version so a bump invalidates it and an unchanged version reuses it.

- name: Cache browser binaries
  uses: actions/cache@v4
  with:
    path: ~/.cache/ms-playwright
    # Trade-off: keying on the lockfile hash is stricter than keying on the
    # version and invalidates more often; it also guarantees no stale binary.
    key: pw-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
- run: npx playwright install --with-deps chromium

Pitfalls #

Referencing an image by tag. Tags are re-pushed, so the same tag can be two different images. Mitigation: reference by digest and bump it deliberately.
Running npm install in CI. It can resolve newer versions and rewrite the lockfile mid-build. Mitigation: use npm ci, which fails rather than drifting.
Upgrading the runner without the image. The runner looks for a browser build the image does not contain. Mitigation: bump both in one change and verify the version at run start.
Caching browser binaries under a static key. A bump silently reuses the old binary. Mitigation: key the cache on the lockfile hash or the exact version.
Batching browser upgrades with feature work. A red suite has two candidate causes and neither is attributable. Mitigation: upgrade in its own pull request.
Pinning and then never upgrading. Six months of browser releases arrive at once and the upgrade becomes a project. Mitigation: a monthly bot bump keeps each delta small.

The same failures are diagnostic in one workflow and noise in the other; the difference is only whether the upgrade was isolated.

Reliability targets #

Metric	Target	Notes
Image references by digest	100%	No floating tags in any Dockerfile or workflow
Dependency install command	`npm ci` only	Fails on lockfile drift instead of resolving
Runtime version verified at run start	100% of pipelines	Hard failure on mismatch
Distinct browser builds seen per week	1	More than one means a pin is incomplete
Browser upgrade cadence	Monthly, isolated	Bot-opened, reviewed, never batched with features

One browser build per week across every pipeline is the observable proof that the pins actually hold.

Frequently Asked Questions #

Q: Is pinning to a digest worth losing automatic security updates? A: For a test runner image, yes. It executes your own code against your own application in an ephemeral container, so the threat model is mild, and reproducibility is worth far more than an unattended rebuild. Pair the pin with a monthly bump so updates still arrive — just on a schedule you chose.

Q: Why does the browser version matter when the tests only use standard DOM APIs? A: Because the failures are rarely about DOM semantics. Browser releases change default download behaviour, focus handling, dialog timing, font fallback and the pixels a screenshot produces. Those are exactly the surfaces automated tests touch, which is why a browser bump reads as a wave of unrelated flakiness if it is not isolated.

Q: The upgrade pull request is red on twenty tests. Do we hold the upgrade? A: Read the failures first — they are the most informative artefact the upgrade produces. If they concentrate in one area (downloads, focus, dialogs) the browser changed a behaviour and the tests encoded the old one; fix the tests and take the upgrade. If they are scattered with timeout signatures, the new build is slower in your image and the suite has thin headroom, which is a separate problem worth fixing before the bump lands. Holding indefinitely is the one option that guarantees a harder upgrade later.

Q: We use the official image — do we still need to pin the npm package? A: Yes. The image supplies the browsers; the lockfile supplies the runner that drives them. If the runner drifts to a version expecting different browser builds, you get a missing-executable error or an untested pairing. Pin both, and verify at run start that they agree.