CI-first knowledge base Reliability Engineering

Article · Flaky Test Detection & Quarantine Engineering

Deciding When to Delete a Flaky Test

Deleting a test feels like giving up, so teams avoid it — and end up with a quarantine list of forty tests that provide no coverage, cost review attention every sprint, and will never be fixed. Deletion is a legitimate engineering decision when it is made explicitly, by the owner, with the coverage gap written down. This guide gives the criteria, as the closing decision in Flaky Test Triage & Ownership.

12 sections URL: /flaky-test-detection-quarantine-engineering/flaky-test-triage-and-ownership/deciding-when-to-delete-a-flaky-test/
Three states, and the one that is not a decision Fixed, deleted with the gap recorded and extended with a reason are all decisions; indefinite quarantine is deletion without acknowledgement. fixedcoverage retainedcost: engineering time deleted openlygap recordedcost: known risk extendedreason and new datecost: deferred quarantined indefinitely deletion without the acknowledgement — the worst option
An indefinitely quarantined test provides the coverage of a deleted one and the maintenance cost of a live one.

Root cause #

Test suites accumulate because adding is easy and removing requires a judgement nobody wants to sign. The asymmetry is social rather than technical: nobody is blamed for a test that exists and costs an hour a week, and everybody remembers the person who deleted the test that would have caught the incident. So tests are quarantined instead, which defers the decision indefinitely while preserving the appearance of coverage.

That appearance is the real damage. A quarantined test is counted in coverage reports, listed in the suite, and mentioned in reviews as though it verifies something. It does not — it is skipped — so the team is carrying the risk of no coverage plus the cost of maintenance, review noise and periodic re-triage. Every sprint that passes makes the decision harder, because the context is further away and the list is longer.

Delete decisions become tractable once the question is reframed. It is not “is this test valuable?” — almost any test is somewhat valuable — but “is the coverage this test provides worth what it costs to stabilise, given what else already covers this behaviour?” That question has an answer, and it is frequently no.

Step-by-step fix #

1. Establish what the test uniquely covers #

The first question is whether the behaviour is verified anywhere else. Duplicated coverage at a lower level is the most common finding, and it makes deletion straightforward.

# Trade-off: a keyword search is crude and finds the obvious overlaps quickly;
# a coverage-tool comparison is more rigorous and much slower to run.
rg -l "applies discount|discountCode" --glob '*.test.*' --glob '*.spec.*'
# → 1 unit test on the pricing function
# → 1 component test on the discount field
# → 1 e2e test (the flaky one)

If the pricing rule is covered by a fast unit test and the field behaviour by a component test, the end-to-end test is verifying the wiring between them. That is real coverage, and it is much less than the test’s presence implies.

2. Cost the two options honestly #

Put a number on both sides. The estimate does not need to be precise — it needs to be written down so the decision is about magnitudes rather than feelings.

// Trade-off: any cost model is approximate, and an approximate model beats an
// unstated one, which is what "we should probably fix it" amounts to.
const annualCost = (t) => ({
  keepFlaky: t.failuresPerYear * (t.rerunMinutes + t.triageMinutes) / 60,   // engineer-hours
  fix: t.estimatedFixHours,
  risk: t.uniqueCoverage ? 'real gap if deleted' : 'covered elsewhere',
});
// { keepFlaky: 34, fix: 6, risk: 'covered elsewhere' }  → fix it, it pays back in months
// { keepFlaky: 3,  fix: 40, risk: 'covered elsewhere' } → delete it
Delete criteria as a two-axis decision Unique coverage and cost to stabilise together determine whether to fix, rewrite at a lower level, or delete. unique duplicated cheap to fix expensive to fix fix it rewrite lower downkeep the coverage, drop the flakiness fix if quick, else delete delete
The expensive-and-unique quadrant is the interesting one: the answer there is usually to re-express the coverage at a cheaper level.

3. Prefer rewriting at a lower level to deleting outright #

When coverage is genuinely unique but the end-to-end form is inherently unstable, the productive move is to move it down the pyramid rather than to choose between flaky and absent.

// Instead of an end-to-end test racing a real payment redirect, verify the
// wiring at the component level with the network controlled.
// Trade-off: less realistic than the full flow, dramatically more stable, and
// it keeps a check on the behaviour that mattered.
test('submits the discount code and renders the new total', async () => {
  server.use(http.post('/api/cart/discount', () =>
    HttpResponse.json({ total: 3800, discountApplied: true })));

  render(<CartSummary />);
  await user.type(screen.getByLabelText('Discount code'), 'SAVE20');
  await user.click(screen.getByRole('button', { name: 'Apply' }));
  expect(await screen.findByText('£38.00')).toBeVisible();
});

4. Record the gap when you do delete #

A deletion with a written record is a managed risk; one without is an accident waiting to be discovered during an incident review.

<!-- docs/coverage-gaps.md -->
## Deleted: checkout › applies discount (e2e)
- **Deleted:** 2026-08-02 by @payments
- **Why:** rate 4%, ~40h to stabilise (third-party redirect timing), unique coverage was the
  wiring between the discount field and the cart total.
- **Replacement:** component test `CartSummary.discount.test.tsx` covers field → total.
- **Residual gap:** the real redirect round trip is no longer exercised in CI.
- **Revisit if:** the payment provider offers a deterministic sandbox.

5. Delete the whole thing, not just the assertion #

A test left in place with its assertions commented out, or skipped with no expiry, is the indefinite quarantine in another form. Remove the file, remove its fixtures, remove its entry from the quarantine list.

6. Make deletion a normal, reviewable act #

The social problem needs a social fix. A pull request labelled coverage-change, referencing the gap record and reviewed by the owning team, makes deletion an ordinary decision with a paper trail — which is what stops it being either taboo or careless.

Pitfalls #

  • Indefinite quarantine. No coverage, full cost, no decision. Mitigation: mandatory expiry with three legitimate outcomes.
  • Deleting without recording the gap. The risk becomes invisible. Mitigation: a gap record with a revisit condition.
  • Assuming coverage is unique. Most end-to-end tests duplicate lower-level coverage. Mitigation: check before deciding.
  • Fixing an expensive test out of principle. Forty hours to stabilise duplicated coverage is a poor trade. Mitigation: cost both options explicitly.
  • Skipping instead of deleting. A skipped test still shows up in reports and reviews. Mitigation: remove it fully.
  • One person deciding alone. Deletion needs the owning team’s agreement to be legitimate. Mitigation: review it like any other change.
  • Never deleting anything. The suite becomes a museum with a growing maintenance bill. Mitigation: treat a healthy deletion rate as normal.
What a deletion record must contain Reason, replacement coverage, residual gap and a revisit condition. whyrate + cost to fix replacementwhat still covers it residual gapstated plainly revisit ifa named condition four lines that turn a deletion from an omission into a decision someone can audit
The revisit condition matters most: it is what makes the gap re-openable rather than forgotten.

Reliability targets #

Metric Target Notes
Tests quarantined longer than 30 days 0 Expiry forces the decision
Deletions with a recorded gap 100% Reason, replacement, residual, revisit
Deletions reviewed by the owning team 100% Same standard as any change
Unique-coverage checks before deletion 100% Usually reveals duplication
Skipped-but-present tests 0 Deleted properly or restored
Deletion discipline scorecard Targets for long-lived quarantine, recorded gaps, review and skipped tests. 0quarantined > 30 d 100%gaps recorded 100%reviewed 0skipped-but-present
Zero long-lived quarantine entries is the number that keeps the other three honest.

Frequently Asked Questions #

Q: What if we delete a test and the bug it would have caught ships? A: That is the risk the gap record exists to make visible in advance, and it is a risk the team accepted knowingly rather than one that materialised by accident. It is also worth being precise about what was lost: a test that fails 4% of the time and is retried away catches very few real regressions, so the coverage forgone is usually smaller than it feels.

Q: Is it better to delete or to skip with a comment? A: Delete. A skipped test is counted, listed and reviewed as though it verifies something, so it carries the maintenance cost of a live test and provides the coverage of a deleted one. If the intent is to restore it later, record that in the gap document with a revisit condition rather than leaving the corpse in the suite.

Q: Who has the authority to delete a test? A: The team that owns the code it covers, through a normal reviewed change. Central platform teams should not delete other people’s coverage, and individuals should not delete quietly. The review is what converts a unilateral act into an accountable decision.

Q: The test covers a compliance requirement. Does that change the calculus? A: Yes — it removes deletion as an option and turns the question into how to make the coverage reliable. Where the requirement is genuinely regulatory, the honest path is to invest in stabilising it, usually by moving the deterministic part of the check down a level and keeping a single, well-instrumented end-to-end run on a schedule rather than on the merge path. What is not acceptable is a quarantined compliance test, since that combines the absence of coverage with a claim that it exists.

Q: How do we avoid deletion becoming the default response to flakiness? A: Watch the ratio of fixes to deletions and the reason recorded on each. A team deleting more than it fixes, with “expensive to stabilise” on everything, is using deletion to avoid the underlying instability — usually environmental, and usually addressable through the budget and quarantine mechanisms in CI Retry Strategies & Budgets rather than one test at a time.

Q: How often should the coverage-gap document be reviewed? A: Quarterly is enough. The purpose is not to re-litigate each decision but to check whether any revisit condition has been met — a provider shipping a deterministic sandbox, a flow being rewritten, a component becoming testable at a lower level. Most entries will still be valid; the one or two that are not are exactly the coverage worth restoring cheaply.