Keeping Fixtures in Sync with Staging Data Shapes

Root cause #

Fixtures are written from a single observation. Someone opened the network panel, copied a response, trimmed it, and committed it — capturing not just the shape but a particular instance: one status value, non-null everywhere, one item in every array. Real data is a distribution, and the fixture is a single draw from it.

The divergence then happens on two axes. The schema moves as the product grows: fields appear, optionality loosens, enums gain values, nested structures are flattened for performance. The population moves too, and this one is invisible to any schema check: the proportion of records with a null description rises, arrays that were always short start containing hundreds of items, a status that was theoretical becomes common. An application handles the fixture’s draw perfectly and falls over on a different one.

The reason this persists is that nothing in a normal test run compares the fixture to reality. Tests read the fixture, the fixture satisfies the tests, and the loop is closed. Breaking that loop requires an external reference — the schema, a fresh sample from staging, or both — and a scheduled comparison against it.

Step-by-step fix #

1. Derive fixtures from the schema, not from a copied response #

Generating a base fixture from the schema guarantees it has every required field and correct types by construction, and the generation fails loudly when the schema changes.

// Trade-off: generated fixtures are always structurally correct and read less
// naturally than a hand-picked example; override only the fields a test cares about.
import { JSONSchemaFaker } from 'json-schema-faker';

JSONSchemaFaker.option({ useDefaultValue: true, alwaysFakeOptionals: true });

export function makeInvoice(overrides = {}) {
  const base = JSONSchemaFaker.generate(openapi.components.schemas.Invoice);
  return { ...base, ...overrides };     // the test states only what matters to it
}

A factory with overrides is the pattern that scales: each test declares the two fields relevant to its assertion, and everything else stays schema-correct without anyone maintaining it.

2. Sample the real population, not one record #

Take a periodic sample from staging and summarise its shape statistically — which fields are ever null, which enum values occur, how long arrays get. That summary is the reference your fixtures should represent.

// scripts/profile-population.js
// Trade-off: a profile is an aggregate and therefore safe to commit, unlike the
// sampled records themselves, which may contain personal data.
const profile = {};
for (const record of sample) {
  for (const [key, value] of Object.entries(record)) {
    const p = (profile[key] ??= { nulls: 0, total: 0, values: new Set(), maxLen: 0 });
    p.total += 1;
    if (value === null) p.nulls += 1;
    if (typeof value === 'string' && p.values.size < 20) p.values.add(value);
    if (Array.isArray(value)) p.maxLen = Math.max(p.maxLen, value.length);
  }
}

3. Keep an edge-case fixture per risky field #

For each field the profile shows as sometimes null, and each enum with more values than your fixtures use, add a fixture that exercises the awkward case. This is where the profile turns into coverage.

// Trade-off: more fixtures to maintain, each one covering a real branch that
// production data will eventually take.
export const invoiceFixtures = {
  typical: makeInvoice(),
  nullDescription: makeInvoice({ description: null }),      // 12% of real records
  voidStatus: makeInvoice({ status: 'void' }),              // rare but real
  manyLineItems: makeInvoice({ lineItems: Array.from({ length: 250 }, makeLineItem) }),
  zeroAmount: makeInvoice({ amount: 0 }),                   // renders differently
};

4. Fail the build when the schema moves #

Structural drift should not wait for a nightly job. Validate every fixture on every run, so a schema update in the same repository breaks immediately and one in a dependency breaks on the next upgrade.

// Trade-off: a hard failure is disruptive when a provider ships a change you
// did not expect, which is precisely when you want to know.
test.each(Object.entries(invoiceFixtures))('%s fixture matches the schema', (_name, fixture) => {
  expect(validateInvoice(fixture), ajv.errorsText(validateInvoice.errors)).toBe(true);
});

5. Re-profile on a schedule and report the delta #

Population drift is slower and needs a scheduled comparison. Report what changed rather than failing, since new seed data on staging is a normal event.

# .github/workflows/data-profile.yml
# Trade-off: weekly is enough for population drift, which moves on the timescale
# of product changes rather than deploys.
on:
  schedule:
    - cron: '0 5 * * 1'
jobs:
  profile:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node scripts/profile-population.js > profile.json
        env:
          STAGING_URL: ${{ secrets.STAGING_URL }}
      - run: node scripts/compare-profile.js profile.json fixtures/profile.baseline.json

The comparison output — “field description is now null in 12% of records, was 0%” — is directly actionable: it names the fixture you are missing.

6. Never sample production data into a fixture #

Staging with seeded data is the only acceptable source. A fixture derived from production carries personal data into the repository, into every developer’s machine, and into CI logs — a compliance problem that no test-reliability benefit can offset. Where production-like variety is genuinely needed, generate it: the profile tells you the distribution, and a generator can reproduce the distribution without reproducing anyone’s data.

Pitfalls #

• Copying a response once and committing it. The fixture captures one draw and ages from that day. Mitigation: generate from the schema and override per test.
• Fixtures with every field populated. Nullability bugs are never exercised. Mitigation: keep an explicit null-heavy fixture per entity.
• Arrays of length one. Pagination, virtualisation and layout bugs stay hidden. Mitigation: a large-collection fixture sized from the observed maximum.
• Enums represented by a single value. Unhandled branches ship silently. Mitigation: one fixture per enum value that changes rendering.
• Sampling production. Personal data enters the repository. Mitigation: seeded staging only, and generate variety from a profile.
• Validating only the happy fixture. The edge-case fixtures drift unchecked. Mitigation: validate every fixture in the same test.
• Treating population drift as failure. New seed data trips the check and it gets muted. Mitigation: report the delta, fail only on schema breaches.

Reliability targets #

Frequently Asked Questions #

Q: Should fixtures be realistic or minimal? A: Structurally realistic, semantically minimal. Every required field present with correct types — which generation gives you for free — and only the values the test cares about stated explicitly. A fixture full of hand-written plausible-looking data reads well and hides which fields the assertion actually depends on.

Q: How do I keep fixture generation from producing meaningless data? A: Constrain the schema rather than the generator. Formats, patterns and enums in the schema produce sensible generated values, and they also make the schema better for everyone. Where a value must be meaningful — an identifier the test asserts on — override it explicitly in the factory call.

Q: Is a weekly population profile worth the effort for a small team? A: Only where the data is genuinely variable. For an internal tool with a stable domain, schema validation is most of the value. For anything consuming customer-shaped data, the null percentages alone justify it: “12% of descriptions are now null” is the difference between a rendering bug found in a test and one found by a user.

Q: Our fixtures are shared across dozens of specs. How do we change one safely? A: Do not change the shared fixture — change the override in the spec that needs something different. A shared fixture edited to suit one test is how a suite acquires failures in unrelated files, and it is the same coupling problem as any other shared mutable state. Keep one canonical fixture per entity, generated from the schema, and let each test state its own deviations through a factory.

Q: How does this relate to seeding deterministic data for tests? A: They are the same problem approached from opposite ends. Seeding controls what the system under test contains, as covered in Seeding Deterministic Mock Data Across Environments; fixture parity controls what the mocked dependencies return. Both need the same reference — the schema plus a profile of the real population — and are best generated from it rather than maintained by hand.

Generate, Then Override #

Structural correctness from the schema, semantic intent from the test — that division keeps fixtures both accurate and readable.

Related guides #

• Environment Parity & Mock Data Management
• Diffing OpenAPI Fixtures Against Live Schemas
• Seeding Deterministic Mock Data Across Environments