Consumer Contract Tests vs Schema Validation

Root cause #

Both techniques exist because mocked tests verify an application against a description of an API rather than the API itself, and descriptions drift. Where they differ is in what plays the role of the authority.

Schema validation treats a published document as the truth. Your fixtures, and optionally live responses, are checked against it: does this payload satisfy the declared types, required fields and enums? It is cheap, runs offline, needs no cooperation from the provider, and catches the large class of breakages that are structural. Its blind spot is everything the schema does not say. A loosely typed schema with additionalProperties everywhere, a field typed as string that is really an enum, or a meaning change with no shape change all pass validation cleanly.

Consumer contract testing inverts the direction. The consumer records what it actually needs — these fields, these types, for these requests — and publishes that as a contract. The provider then replays those expectations against its real implementation in its own pipeline, and its build fails if it would break you. Its power is that the check happens where the change is made, before the change ships. Its requirement is organisational: the provider has to participate. For an internal service owned by a neighbouring team that is achievable; for a third-party vendor it is not.

The practical consequence is that these are complements chosen by relationship, not competitors chosen by preference. You validate schemas for APIs you consume and do not control; you exchange contracts with teams that will run your expectations in their build.

Step-by-step comparison #

1. Validate fixtures against a published schema #

The lightweight baseline, appropriate everywhere a schema exists.

// Trade-off: catches structural drift for free and is only as strong as the
// schema — a permissive schema validates almost anything.
import Ajv from 'ajv';
import addFormats from 'ajv-formats';

const ajv = addFormats(new Ajv({ strict: false, allErrors: true }));
const validate = ajv.compile(openapi.components.schemas.Invoice);

for (const fixture of invoiceFixtures) {
  if (!validate(fixture)) {
    throw new Error(`fixture drifted from schema:\n${ajv.errorsText(validate.errors)}`);
  }
}

2. Tighten the schema before trusting it #

A validation pass that never fails is usually measuring a permissive schema rather than a correct application. Check the schema’s strictness explicitly.

// Trade-off: demanding strict schemas creates friction with providers who
// prefer loose ones, and it is what makes validation meaningful at all.
const weak = Object.entries(openapi.components.schemas).filter(([, s]) =>
  s.additionalProperties !== false || !Array.isArray(s.required) || s.required.length === 0
);
if (weak.length) {
  console.log(`::warning::${weak.length} schema(s) too permissive to catch drift`);
}

3. Express what the consumer actually needs #

The contract’s value is that it records your real requirements rather than the provider’s full surface. A consumer that reads three fields should assert three fields — no more, so the provider retains freedom to change everything else.

// Trade-off: narrow expectations give the provider room to evolve and will not
// warn you about changes to fields you do not use — which is the intent.
export const invoiceExpectation = {
  request: { method: 'GET', path: '/api/invoices/INV-1' },
  response: {
    status: 200,
    body: {
      id: 'INV-1',            // required by the list view
      amount: 1250,           // integer minor units — the semantic we depend on
      status: 'open',         // one of: open | paid | void
    },
  },
};

4. Have the provider verify, and gate on it #

The mechanism only pays off if the provider’s pipeline runs the expectations and fails when they break. Without that step, a contract is a document, and documents drift like any other.

// In the provider's test suite: replay each consumer expectation against the
// real handler. Trade-off: the provider takes on a build dependency from its
// consumers, which is the cost of not breaking them.
for (const expectation of publishedExpectations) {
  test(`satisfies ${expectation.consumer}: ${expectation.request.path}`, async () => {
    const res = await request(app)[expectation.request.method.toLowerCase()](expectation.request.path);
    expect(res.status).toBe(expectation.response.status);
    expect(res.body).toMatchObject(expectation.response.body);
  });
}

5. Cover semantics with a small live suite #

Neither technique reliably catches a change in meaning with an unchanged shape — cents becoming units, a status value repurposed. A handful of assertions against the real API, scheduled rather than blocking, is the cheapest cover, along the lines of Detecting Stale Recorded Fixtures in CI.

6. Choose by relationship, not by fashion #

The decision comes down to three questions. Does a schema exist, and is it strict? Then validate — it is nearly free. Does the provider’s team run your expectations in their build? Then exchange contracts — it is the only approach that catches a break before it ships. Is the provider a third party who will do neither? Then schema validation plus a scheduled live check is the realistic ceiling, and the effort is better spent on making the application resilient to unexpected payloads than on more elaborate verification.

Pitfalls #

• Validating against a permissive schema. The check passes on payloads that would break the application. Mitigation: measure schema strictness and push for required and additionalProperties: false.
• Writing contracts nobody verifies. A contract not run in the provider’s pipeline is documentation. Mitigation: gate the provider’s build on it, or do not build contracts at all.
• Contracts that mirror the whole response. The provider cannot change anything without breaking you. Mitigation: assert only the fields you consume.
• Assuming validation covers semantics. Units and meanings change without shape changes. Mitigation: a few live assertions on the semantics you depend on.
• Choosing contract testing for a third-party API. They will not run your expectations. Mitigation: validate the schema, and harden the client.
• Running live contract checks on the merge path. A provider outage blocks unrelated work. Mitigation: schedule them.

Reliability targets #

Frequently Asked Questions #

Q: If we validate against the schema, do we need contract testing at all? A: Only for providers you can influence. Validation tells you your fixtures match the published document; it cannot tell you the provider is about to remove a field you rely on but the schema marks optional. Contract testing moves that check into the provider’s build, which is the only place it can fail before the change ships.

Q: How narrow should a consumer expectation be? A: As narrow as the code’s actual dependency. Asserting the whole response makes every provider change a breakage and trains them to ignore your contract; asserting three fields you genuinely read gives them room to evolve and makes a failure meaningful. Narrow contracts are the ones that survive politically.

Q: We consume a large third-party API. What is the realistic best practice? A: Validate against their schema if they publish one, keep a small scheduled suite asserting the semantics you depend on, and harden the client so unexpected fields and missing optional values degrade gracefully rather than throwing. You cannot make a vendor run your tests, so resilience carries the weight that verification cannot.

Q: Which should a team adopt first? A: Schema validation, without exception. It is a day of work, needs nobody’s cooperation, and removes the largest category of drift. Consumer contract testing is worth adding afterwards, and only for providers whose teams will genuinely gate their builds on your expectations — the tooling is the easy part of that decision, and the agreement is the hard part.

Q: Can GraphQL replace this, since the schema is introspectable? A: It removes the “no schema” problem and not the drift problem. A GraphQL schema is strongly typed and available at runtime, which makes structural validation excellent — but a field can still be deprecated, a nullable field can start returning null, and an enum can gain a value your client does not handle. Query-level assertions play the same role contracts do; see Stubbing GraphQL Queries in Cypress for the mocking side.

Start With What Needs No Agreement #

Schema validation requires nobody’s cooperation, which makes it the right first step regardless of which approach a team eventually settles on.

Related guides #

• API Contract Validation in E2E Tests
• Validating OpenAPI Contracts in E2E Pipelines
• Detecting Breaking API Changes in CI