Detecting Breaking API Changes in CI

Root cause #

E2E tests assert on rendered behavior, not on the contract that produced it. When a backend drops a previously required field, the UI may simply render an empty string and the test still passes — until that field becomes load-bearing in production. The flakiness is structural: the suite cannot see a contract regression because it never inspects the contract. By the time a real environment exposes the mismatch, the change has already merged, and the failure looks intermittent because it depends on which payload variant the run happened to hit.

Catching breaking changes requires three complementary checks, because each sees a different layer. oasdiff compares the spec before and after a change and classifies modifications as breaking or not. ajv validates actual live responses against the schema, catching cases where the implementation diverges from its own spec. Contract tests encode what the consumer actually relies on, so a change that is technically non-breaking but removes something this app needs still fails. Run all three as gates in GitHub Actions and a breaking change cannot reach merge.

Step-by-step fix #

1. Gate on breaking spec changes with oasdiff #

Compare the base branch spec against the PR spec and fail on backward-incompatible changes.

# .github/workflows/contract.yml
name: contract
on: [pull_request]
jobs:
  oasdiff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: Breaking-change gate
        # --fail-on ERR blocks merge on any backward-incompatible contract change.
        run: |
          git show origin/${{ github.base_ref }}:openapi.json > base.json
          npx oasdiff breaking base.json openapi.json --fail-on ERR

2. Validate live responses against the schema with ajv #

Compile the schema once and assert each captured response conforms, catching spec-versus-implementation drift.

// tests/contract/validate.spec.js
const Ajv = require('ajv');
const addFormats = require('ajv-formats');
const userSchema = require('../../schemas/user.json');

const ajv = new Ajv({ allErrors: true });
addFormats(ajv);
const validate = ajv.compile(userSchema);

test('GET /users response matches the contract', async () => {
  const res = await fetch(`${process.env.API_URL}/api/v1/users/1`).then(r => r.json());
  // allErrors surfaces every violation at once, not just the first — faster triage.
  expect(validate(res)).toBe(true);
});

3. Encode consumer expectations as contract tests #

Assert specifically on the fields this application depends on, so non-breaking-but-relevant removals still fail.

test('user payload provides fields the UI binds to', async () => {
  const res = await fetch(`${process.env.API_URL}/api/v1/users/1`).then(r => r.json());
  // These are what the UI renders; losing any of them breaks this consumer even if the spec allows it.
  expect(res).toEqual(expect.objectContaining({
    id: expect.any(String),
    displayName: expect.any(String),
    role: expect.stringMatching(/^(admin|member)$/)
  }));
});

4. Require all gates before merge #

Make each job a required status check so a single failure blocks the PR.

  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      # Both checks must pass; either failing reports a red status that branch protection enforces.
      - run: npm run test:contract
      - run: echo "validated for ${{ github.sha }}"

For the broader pipeline that runs these alongside full E2E flows, see Validating OpenAPI Contracts in E2E Pipelines.

Pitfalls #

• Relying on E2E assertions to catch contract drift. Mitigation: add schema validation that inspects the payload directly.
• Treating all spec changes as breaking. Mitigation: use oasdiff breaking so additive changes pass.
• Validating the spec but never a live response. Mitigation: run ajv against real responses to catch implementation drift.
• Contract tests that mirror the spec instead of consumer needs. Mitigation: assert only on the fields this app actually binds to.
• Gates that report but do not block. Mitigation: mark each job as a required status check in branch protection.

Reliability targets #

Frequently Asked Questions #

Q: Why use three checks instead of just oasdiff? A: Each covers a blind spot. oasdiff sees spec changes, ajv sees implementation drift from the spec, and contract tests see what your consumer actually depends on. Together they leave no gap.

Q: What counts as a breaking change? A: Removing or renaming a field, narrowing a type or enum, adding a required request parameter, or changing a status code — anything that would break an existing consumer. oasdiff breaking classifies these for you.

Q: Should this run on every PR or only on backend PRs? A: On every PR that can change the contract, including consumer PRs, because the consumer’s expectations are part of the contract the gates enforce.

What Counts as Breaking #

Not every schema difference breaks a consumer, and treating them all as equal produces a check that fails constantly and gets bypassed. Three categories, with different responses:

Breaking for everyone. A required response field removed or renamed, a type changed, an enum value removed, a required request parameter added. These break any consumer that used the field, and they should fail a build.

Breaking for you specifically. An optional field removed that your client relies on, or a nullability loosened on a field your rendering assumes is present. Nothing in the provider’s schema was violated — optional means optional — which is exactly why this category is invisible to schema validation and needs consumer expectations to catch.

Additive and safe. New optional fields, new enum values in a response you handle exhaustively with a fallback, new endpoints. These should not fail anything, and a check that treats them as breaking will be switched off before it ever catches something real.

The middle category is the one worth building for, because it is both the most common in practice and the least likely to be flagged by generic tooling. A short list of the fields your client actually reads, checked against each new schema version, catches it for very little effort.

// Check the fields this client depends on, not the whole surface.
// Trade-off: a hand-maintained dependency list needs updating as the client
// evolves, and it catches the optional-field removal nothing else sees.
const DEPENDS_ON = ['invoice.id', 'invoice.amount', 'invoice.status', 'invoice.customer.name'];

for (const path of DEPENDS_ON) {
  if (!schemaHasPath(newSchema, path)) {
    console.error(`::error::${path} is gone from the new schema and this client reads it`);
    process.exitCode = 1;
  }
}

Catching It Before It Ships #

The value of this check depends entirely on when it runs, and there are three placements in increasing order of usefulness.

After the provider deploys, by comparing a fresh capture against committed fixtures. This is the easiest to build and the least valuable, because the change is already live and the conversation is about recovery rather than prevention.

When the provider publishes a new schema version, by validating against the document rather than against a live response. Schemas are usually published ahead of the deployment, which turns the check into a warning with lead time — often days.

In the provider’s own pipeline, by having them verify your consumer expectations before merging. This is the only placement that prevents the break rather than detecting it, and it requires an agreement rather than a tool: the provider’s build fails when a change would break a named consumer.

Which of these is available is a function of the relationship, not of the tooling. For an internal service owned by a neighbouring team, the third is achievable and worth the conversation. For a third-party vendor, the second is the ceiling, and the remaining risk is best absorbed by making the client resilient — tolerating unexpected fields, degrading gracefully on missing optional data, and validating at the boundary rather than trusting the payload.

Give the Check a Home and an Owner #

A contract check with no owner becomes noise within a quarter. Routing its output to the team that owns the integration — the same routing used for flaky tests — keeps it actionable, and reviewing the check itself when it produces a false positive keeps it trusted.

The output of this check is only as good as its message. Naming the field, the change and the consumer that depends on it turns a blocked build into a short conversation; a generic failure turns it into an investigation nobody scheduled time for.

Reviewing a false positive is cheaper than losing trust in the check, so tune the comparison rather than muting it.

Related guides #

• API Contract Validation in E2E Tests
• Validating OpenAPI Contracts in E2E Pipelines
• Diffing OpenAPI Fixtures Against Live Schemas