Detecting Stale Recorded Fixtures in

A stale fixture converts an unreliable test into a confidently wrong one, which is strictly worse than flakiness.

Root cause #

Mocked tests verify the application against a description of the API rather than against the API. That is the whole point — it is what makes them fast and deterministic — and it means the description is now a dependency with no version, no owner and no expiry. When the provider renames a field, adds a required parameter, tightens validation or changes a status code, the fixture keeps returning the old shape and every assertion continues to hold.

The absence of a signal is what makes this dangerous. A flaky test announces itself; a stale fixture is silent by construction. The failure surfaces at integration time, in an environment where the real API is involved, which is usually staging on a release day or production shortly after. By then the drift may be months old and span several changes, so the debugging starts from “what has changed since March” rather than from a single diff.

There are two distinct kinds of drift, and they need different detection. Structural drift is a change in shape: fields added, removed or renamed, types changed, nullability tightened. It is detectable by comparing the fixture against the current schema or against a fresh capture, and it is what breaks parsing and rendering. Semantic drift is a change in meaning with the same shape — a status value that now means something else, a monetary amount that switched from cents to units, an identifier format that changed. Semantic drift survives structural comparison and is only caught by exercising the real API in a small, deliberate set of tests.

Step-by-step fix #

1. Re-record on a schedule and diff structurally #

The cheapest detector is a nightly job that captures fresh traffic and compares the shapes. Compare structure rather than bytes, or timestamps and generated ids will make every run look like a change.

// scripts/shape-of.js — reduce a payload to its structure
// Trade-off: a shape summary ignores values, so it catches renames and type
// changes and misses semantic drift; that is a deliberate split of concerns.
export function shapeOf(value) {
  if (Array.isArray(value)) return [shapeOf(value[0] ?? null)];
  if (value === null) return 'null';
  if (typeof value !== 'object') return typeof value;
  return Object.fromEntries(
    Object.keys(value).sort().map((k) => [k, shapeOf(value[k])])
  );
}

// scripts/detect-drift.js
// Compare the committed fixture's shape against a freshly captured one.
import { readFileSync } from 'node:fs';
import { shapeOf } from './shape-of.js';

const committed = shapeOf(JSON.parse(readFileSync('fixtures/invoices.json', 'utf8')));
const fresh = shapeOf(await (await fetch(`${process.env.STAGING_URL}/api/invoices`)).json());

const drifted = JSON.stringify(committed) !== JSON.stringify(fresh);
if (drifted) {
  console.log('::warning::fixture shape differs from staging');
  console.log('committed:', JSON.stringify(committed, null, 2));
  console.log('fresh    :', JSON.stringify(fresh, null, 2));
}
process.exit(0);   // warn, do not fail — see step 3

2. Validate fixtures against the published schema #

If the provider publishes an OpenAPI or GraphQL schema, the comparison does not need a live call at all: validate every fixture against the schema as part of the normal pipeline. This catches drift the moment the schema is updated, which is usually before the API deploys.

// Trade-off: schema validation is fast and offline, and it only catches what
// the schema describes — a provider with a loose schema needs the live diff too.
import Ajv from 'ajv';

const ajv = new Ajv({ strict: false });
const validate = ajv.compile(openapi.components.schemas.InvoiceList);

for (const file of fixtureFiles) {
  const data = JSON.parse(readFileSync(file, 'utf8'));
  if (!validate(data)) {
    console.error(`::error file=${file}::fixture no longer matches the schema`);
    console.error(validate.errors);
    process.exitCode = 1;      // this one SHOULD fail the build
  }
}

The distinction matters: a fixture that contradicts the published schema is definitely wrong and should block; a fixture that merely differs from today’s staging response might just mean staging has new data. The schema-driven half of this is developed further in Validating OpenAPI Contracts in E2E Pipelines.

No single detector covers both structural and semantic drift; the three together do, at very different costs.

3. Warn on drift, fail on contradiction #

Getting the severity right is what makes the signal survive. A nightly job that fails the build because staging has new seed data will be muted within a fortnight.

# .github/workflows/fixture-drift.yml
# Trade-off: warnings are ignorable, which is why the schema check stays a hard
# failure — the two severities carry different amounts of certainty.
on:
  schedule:
    - cron: '0 4 * * 1-5'
jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node scripts/validate-fixtures.js      # hard failure on schema breach
      - run: node scripts/detect-drift.js           # warning on staging difference
        env:
          STAGING_URL: ${{ secrets.STAGING_URL }}
      - name: Open an issue when drift persists
        if: failure()
        run: gh issue create --title "Fixture drift detected" --body "See the run log."

4. Keep a small live-contract suite for semantic drift #

Shapes cannot tell you that status: "pending" now means something different. Keep a handful of tests — one per critical integration — that call the real API and assert on meaning rather than structure.

// Trade-off: these tests are slow and can fail for reasons outside your
// control; keep them few, tag them, and never put them on the pull-request path.
test('billing API still returns amounts in minor units @contract', async ({ request }) => {
  const res = await request.get(`${process.env.STAGING_URL}/api/invoices/INV-1`);
  const invoice = await res.json();
  // 12.50 EUR must be 1250, not 12.5 — a unit change is invisible to shape checks
  expect(Number.isInteger(invoice.amount)).toBe(true);
  expect(invoice.amount).toBe(1250);
});

5. Track fixture age and make it visible #

Age is the cheapest proxy for risk and needs no live call at all. Stamp each fixture when it is recorded and report the oldest ones.

// Trade-off: age is a weak signal on a stable API and a strong one on a fast
// moving integration; weight the threshold per provider rather than globally.
const ages = fixtureFiles.map((f) => ({
  file: f,
  days: Math.floor((Date.parse(process.env.RUN_DATE) - Date.parse(meta(f).recordedAt)) / 86_400_000),
}));
for (const { file, days } of ages.filter((a) => a.days > 30)) {
  console.log(`::warning file=${file}::recorded ${days} days ago — consider re-recording`);
}

6. Make acceptance of a diff an explicit act #

When drift is real, someone has to decide whether the application should change or the fixture should. Re-recording automatically removes that decision and with it the only chance to notice a breaking change before users do. Require the updated fixture to arrive as a reviewed pull request, with the diff visible.

Pitfalls #

Failing the build on any staging difference. New seed data trips it and the check gets muted. Mitigation: warn on differences, fail only on schema contradictions.
Diffing raw bytes. Timestamps and generated ids make every comparison a false positive. Mitigation: compare structure after normalising volatile fields.
Auto-committing re-recorded fixtures. The suite silently adopts breaking changes. Mitigation: require review of the diff.
Relying on shape comparison alone. Unit and meaning changes pass straight through. Mitigation: keep a few live-contract assertions.
Running live-contract tests on every pull request. Provider outages block unrelated work. Mitigation: schedule them, tag them, keep them off the merge path.
No owner for the warning. An unowned nightly warning is noise within a month. Mitigation: route it to the team owning that integration, as covered in Flaky Test Triage & Ownership.

Only one of the three outcomes is "update the fixture" — automating that step throws the other two away.

Reliability targets #

Metric	Target	Notes
Fixtures validated against a schema	100% where a schema exists	Hard failure on breach
Median fixture age	< 30 days	Reported by the age check
Drift warnings open for more than a week	0	Routed to the owning team
Live-contract tests	1 per critical integration	Scheduled, never on the merge path
Contract breaks first found in production	0 per quarter	The number this whole loop exists to protect

The last number is the outcome; the first three are the leading indicators that keep it at zero.

Frequently Asked Questions #

Q: Why not just run everything against the real API and skip fixtures? A: Because you would trade a silent failure mode for a loud one: the suite becomes as available as the provider, as fast as its slowest endpoint, and dependent on data other people edit. Mocking is the right default; drift detection is the price of it, and it is much cheaper than the alternative.

Q: How often should the drift job run? A: Daily on weekdays is a good default. More often produces noise on a provider that reseeds staging; less often lets several changes accumulate into one confusing diff. Match the cadence to how fast the provider actually ships.

Q: The provider has no schema. What is the best available detector? A: The structural diff plus a small live-contract suite. Without a schema there is no authority to validate against, so the fresh capture becomes the reference — which means you must review differences rather than assert on them, and the live-contract tests carry more of the weight.

Q: Should a stale fixture block a release? A: A fixture that contradicts a published schema should, because it is provably wrong. A fixture that merely differs from today’s staging response should not, because the difference is as likely to be test data as a contract change. Encode that difference in severity, or the check gets ignored.