Detecting Breaking API Changes in CI

Root cause #

E2E tests assert on rendered behavior, not on the contract that produced it. When a backend drops a previously required field, the UI may simply render an empty string and the test still passes — until that field becomes load-bearing in production. The flakiness is structural: the suite cannot see a contract regression because it never inspects the contract. By the time a real environment exposes the mismatch, the change has already merged, and the failure looks intermittent because it depends on which payload variant the run happened to hit.

Catching breaking changes requires three complementary checks, because each sees a different layer. oasdiff compares the spec before and after a change and classifies modifications as breaking or not. ajv validates actual live responses against the schema, catching cases where the implementation diverges from its own spec. Contract tests encode what the consumer actually relies on, so a change that is technically non-breaking but removes something this app needs still fails. Run all three as gates in GitHub Actions and a breaking change cannot reach merge.

Step-by-step fix #

1. Gate on breaking spec changes with oasdiff #

Compare the base branch spec against the PR spec and fail on backward-incompatible changes.

# .github/workflows/contract.yml
name: contract
on: [pull_request]
jobs:
  oasdiff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: Breaking-change gate
        # --fail-on ERR blocks merge on any backward-incompatible contract change.
        run: |
          git show origin/${{ github.base_ref }}:openapi.json > base.json
          npx oasdiff breaking base.json openapi.json --fail-on ERR

2. Validate live responses against the schema with ajv #

Compile the schema once and assert each captured response conforms, catching spec-versus-implementation drift.

// tests/contract/validate.spec.js
const Ajv = require('ajv');
const addFormats = require('ajv-formats');
const userSchema = require('../../schemas/user.json');

const ajv = new Ajv({ allErrors: true });
addFormats(ajv);
const validate = ajv.compile(userSchema);

test('GET /users response matches the contract', async () => {
  const res = await fetch(`${process.env.API_URL}/api/v1/users/1`).then(r => r.json());
  // allErrors surfaces every violation at once, not just the first — faster triage.
  expect(validate(res)).toBe(true);
});

3. Encode consumer expectations as contract tests #

Assert specifically on the fields this application depends on, so non-breaking-but-relevant removals still fail.

test('user payload provides fields the UI binds to', async () => {
  const res = await fetch(`${process.env.API_URL}/api/v1/users/1`).then(r => r.json());
  // These are what the UI renders; losing any of them breaks this consumer even if the spec allows it.
  expect(res).toEqual(expect.objectContaining({
    id: expect.any(String),
    displayName: expect.any(String),
    role: expect.stringMatching(/^(admin|member)$/)
  }));
});

4. Require all gates before merge #

Make each job a required status check so a single failure blocks the PR.

  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      # Both checks must pass; either failing reports a red status that branch protection enforces.
      - run: npm run test:contract
      - run: echo "validated for ${{ github.sha }}"

For the broader pipeline that runs these alongside full E2E flows, see Validating OpenAPI Contracts in E2E Pipelines.

Pitfalls #

• Relying on E2E assertions to catch contract drift. Mitigation: add schema validation that inspects the payload directly.
• Treating all spec changes as breaking. Mitigation: use oasdiff breaking so additive changes pass.
• Validating the spec but never a live response. Mitigation: run ajv against real responses to catch implementation drift.
• Contract tests that mirror the spec instead of consumer needs. Mitigation: assert only on the fields this app actually binds to.
• Gates that report but do not block. Mitigation: mark each job as a required status check in branch protection.

Reliability targets #

Frequently Asked Questions #

Q: Why use three checks instead of just oasdiff? A: Each covers a blind spot. oasdiff sees spec changes, ajv sees implementation drift from the spec, and contract tests see what your consumer actually depends on. Together they leave no gap.

Q: What counts as a breaking change? A: Removing or renaming a field, narrowing a type or enum, adding a required request parameter, or changing a status code — anything that would break an existing consumer. oasdiff breaking classifies these for you.

Q: Should this run on every PR or only on backend PRs? A: On every PR that can change the contract, including consumer PRs, because the consumer’s expectations are part of the contract the gates enforce.

Related guides #

• API Contract Validation in E2E Tests
• Validating OpenAPI Contracts in E2E Pipelines
• Diffing OpenAPI Fixtures Against Live Schemas