Flaky Test Triage & Ownership

Prerequisites #

Ownership routing depends on the quarantine machinery in Building Auto-Quarantine Workflows and on the history described in Historical Flakiness Tracking & Analytics; this topic is what connects them to people.

Step-by-step implementation #

1. Classify before routing #

The same symptom has three owners. Sending everything to the team that owns the spec file wastes their time on infrastructure problems and hides genuine product races.

// scripts/triage.js
// Trade-off: heuristic classification is imperfect and beats sending everything
// to one queue, where the signal drowns.
export function classify(failure) {
  const msg = failure.error?.message ?? '';
  if (/ECONNREFUSED|OOMKilled|no space left|failed to start/i.test(msg)) return 'infrastructure';
  if (/expected .* received|toEqual|toBe\b/i.test(msg)) return 'product-suspect';
  if (/Timeout|waiting for (locator|selector)/i.test(msg)) return 'test';
  return 'unknown';
}

Infrastructure goes to whoever owns the pipeline. Product-suspect goes to the team that owns the feature, flagged as a possible race rather than as a flaky test. Only the test class belongs to the spec’s owner as a test-quality item.

2. Derive the owner from the repository, not from a spreadsheet #

A mapping that someone maintains by hand goes stale. CODEOWNERS is already maintained for review routing and is the natural source.

// Trade-off: CODEOWNERS granularity may be coarser than you want, and it has
// the enormous advantage of being kept current for reasons unrelated to tests.
import { match } from './codeowners.js';

const owner = match(failure.file) ?? '@qa-platform';   // explicit fallback owner

An explicit fallback matters: an unmatched path must land somewhere, or those failures accumulate unassigned, which is where most flakiness backlogs come from.

3. Prioritise by cost, not by rate alone #

The worst flaky test is not the one that fails most often — it is the one that costs the most developer time. Cost combines frequency, how many pipelines it blocks, and how long it takes to re-run.

// Trade-off: a composite score is less transparent than "sort by failure count"
// and it puts effort where the time is actually being lost.
const score = (t) =>
  t.failuresPer100Runs *          // how often
  t.blockedPipelines *            // how much it stops
  Math.log10(1 + t.rerunMinutes); // how expensive a rerun is

const worklist = tests.sort((a, b) => score(b) - score(a)).slice(0, 10);

4. Create one durable item per test, not one per failure #

A new issue per failure buries the tracker. Deduplicate on a stable identifier and accumulate evidence on the existing item.

# Trade-off: deduping needs a stable key, so a renamed test creates a second
# item — acceptable, and much better than an issue per pipeline run.
KEY="flaky:$(echo "$TEST_FILE::$TEST_TITLE" | sha1sum | cut -c1-12)"
EXISTING=$(gh issue list --label flaky --search "$KEY" --json number --jq '.[0].number')

if [ -n "$EXISTING" ]; then
  gh issue comment "$EXISTING" --body "Failed again in run $GITHUB_RUN_ID (rate now ${RATE}%)."
else
  gh issue create --label flaky --assignee "$OWNER" \
    --title "Flaky: $TEST_TITLE" \
    --body "$KEY
Rate: ${RATE}% over 30 days. Class: ${CLASS}. Owner: ${OWNER}.
Quarantine expires: ${EXPIRY}."
fi

5. Put an expiry on every quarantine #

The expiry is the mechanism that makes the rest work. A quarantined test with no date is a test that has been silently deleted, with the added cost of everyone believing it still provides coverage.

// quarantine.json — data, not code, so it is easy to audit and to expire
// Trade-off: a hard expiry can force an inconvenient decision at a bad time,
// which is preferable to an indefinite quarantine nobody revisits.
[
  { "test": "checkout › applies discount", "owner": "@payments", "since": "2026-07-15", "expires": "2026-08-15" },
  { "test": "search › paginates results", "owner": "@discovery", "since": "2026-07-28", "expires": "2026-08-28" }
]

At expiry there are exactly three legitimate outcomes: fixed and returned to the suite, deleted with the coverage gap recorded, or explicitly extended by the owner with a reason. Silence is not one of them, and a build that fails on an expired quarantine entry is what enforces that.

6. Review the list on a fixed cadence #

Automation routes; people decide. A short weekly review of the top ten by cost, with owners present, is enough — and its main value is that it makes the list visible to the people who can shrink it.

7. Give the process a budget of its own #

Triage competes with feature work, and it loses every time unless it has explicitly allocated capacity. The teams that keep a suite stable over years almost always have a standing arrangement: a fixed share of each sprint, or a rotating role, dedicated to whatever is at the top of the flakiness worklist.

A rotation works better than a volunteer arrangement for two reasons. It spreads the knowledge — after a few months everyone has debugged a race and used the runbook, which raises the floor across the team — and it removes the social cost of saying no to a feature request, because the rotation is already committed. One engineer for one week in every six is enough for most teams, and it maps neatly onto the weekly review: the person on rotation takes the top items from the cost-ranked list.

The alternative arrangement — fixing flakiness when it gets bad enough — has a predictable failure mode. “Bad enough” is reached during a release crunch, when nobody can be spared, so the response is to raise the retry count instead. The suite then stabilises on paper and degrades in fact, which is the dynamic described in Why Blanket Retries Hide Real Bugs.

8. Separate the metric the team owns from the one the platform owns #

Ownership disputes almost always trace back to a single blended number. When “flakiness” includes runner failures, container start-up problems and registry timeouts alongside genuine test instability, no team can be held to it: the spec owners cannot fix the runner, and the platform team is judged on tests it did not write.

Split the number at the point of collection and give each half a threshold and an owner. The test-flake rate belongs to the teams that own the specs and drives the budget. The infrastructure-failure rate belongs to whoever owns the pipeline and deserves its own escalation path. Publishing both, side by side, does more for accountability than any amount of process, because each team can see the number it is actually responsible for moving.

// Two counters, two owners, two thresholds — one aggregate helps nobody.
// Trade-off: classification is imperfect; a mis-attributed failure is a small
// cost next to a metric neither team believes they own.
report({
  testFlakeRate: rate(failures.filter((f) => classify(f) !== 'infrastructure')),
  infraFailureRate: rate(failures.filter((f) => classify(f) === 'infrastructure')),
});

Configuration reference #

Data-driven analysis #

• Quarantine inflow versus outflow. Tests added to quarantine per week against tests leaving it. Sustained inflow above outflow is the single clearest sign that ownership is not working, and it predicts the state of the suite six months out.
• Median quarantine age. How long a test sits before being fixed or deleted. Rising age means expiry dates are being extended rather than acted on.
• Unassigned share. The proportion of flaky tests with no owner. Anything above zero grows, because unowned work is nobody’s to prioritise.
• Cost concentration. The share of lost developer time attributable to the top ten tests. Usually high, which is good news: a small worklist recovers most of the time.
• Reopen rate. How often a test returns to quarantine after being declared fixed. A high rate means fixes are addressing symptoms — usually a longer timeout — rather than causes.

What good looks like after six months #

The difference between a team that has this working and one that does not is visible in a handful of observable behaviours rather than in any dashboard.

Flaky tests acquire an owner within minutes, automatically, and the owner is a team rather than a person. The quarantine list is short — single digits for a suite of a few thousand tests — and every entry has a date that is in the future. Nobody argues about whether a failure is a test problem or a product problem, because the classifier proposes an answer and the error signature settles it. A weekly review takes fifteen minutes because the worklist is ranked by cost and the top item is obvious.

The most telling sign is what happens when someone finds an intermittent bug in production. In a team without this discipline, the investigation starts from scratch. In a team with it, the first action is to search the rescued-failure history for the same area — and often the defect is already there, recorded, with a trace attached, having been rescued by retries for weeks. That search taking two minutes instead of two days is the compound return on recording, classifying and routing every rescue.

None of this requires unusual tooling. It requires that every detected flake leaves the pipeline with a class, an owner and a date attached, and that something fails when one of those is missing.

Common pitfalls & mitigation strategies #

• Quarantine with no expiry. The test is deleted in practice while appearing to be covered. Mitigation: mandatory expiry date, enforced in the build.
• One issue per failure. The tracker becomes unusable and the signal is lost. Mitigation: deduplicate on a stable key and comment on the existing item.
• Routing everything to one queue. Infrastructure problems and product races land with spec owners who cannot fix them. Mitigation: classify first, then route.
• No fallback owner. Unmatched paths accumulate unassigned. Mitigation: an explicit default team.
• Prioritising by failure count. Effort goes to cheap, noisy tests while an expensive one waits. Mitigation: score by cost.
• Fixing with a longer timeout. The test returns to quarantine within weeks. Mitigation: track the reopen rate and treat a reopen as an unfinished fix.
• A review with no owners present. Decisions are deferred to people who are not there. Mitigation: invite the owning teams, keep it to fifteen minutes.

The organising idea behind all of it is that a detected flake should never sit in an undifferentiated queue: it leaves the pipeline with a class, an owner and a date, or something fails.

Keeping the Backlog Bounded #

A flakiness backlog behaves like any other queue: it is stable only when the rate of items leaving matches the rate arriving. Most teams work hard on the outflow — triaging, fixing, graduating — and leave the inflow unmanaged, which is why the list grows despite genuine effort.

Three controls bound the inflow, in ascending order of strength. Stress-running changed specs before merge catches newly introduced instability while its author is present, which is both the cheapest moment to fix it and the only moment when the context is free. A zero-tolerance gate on touched tests makes that pressure concrete without penalising anyone for pre-existing problems elsewhere. And a ratio cap — no more than a small percentage of a package’s tests quarantined at once — stops a team from quarantining its way to a green, meaningless pipeline.

On the outflow side, the metric that matters is not how many items are open but how long they stay. A median quarantine age that is rising means expiry dates are being extended rather than acted on, and no amount of additional triage capacity fixes a process where the exit criteria are negotiable.

Watching the two rates together turns an unbounded quality problem into an ordinary queue management one, with the useful property that it diverges visibly long before the suite becomes unusable.

Frequently Asked Questions #

Q: Should a dedicated team own all flaky tests? A: A dedicated team should own the tooling — detection, quarantine, budgets, reporting — and the teams that own the code should own the individual tests. Central ownership of the tests themselves fails for a structural reason: the people best placed to know whether a failure is a test problem or a product race are the people who wrote the feature, and a central team has to reconstruct that context for every item.

Q: How long should a test stay quarantined? A: Long enough to schedule the work and short enough that it stays in living memory: two to four weeks. Past a month, the original context is gone, whoever wrote it has moved on, and the fix becomes an archaeology exercise. If a test cannot be fixed within a month, the honest question is whether the coverage it provides is worth the effort at all.

Q: What if the owning team disputes that it is their problem? A: That is usually a classification failure and worth taking seriously. A test failing on an assertion may well be a product race the feature team should own; one failing on a timeout inside a shared fixture is a test-infrastructure problem. Route by the error signature rather than by the file path where the two disagree, and let the dispute improve the classifier.

Q: Is deleting a flaky test ever the right answer? A: Yes, more often than teams admit. A test that is expensive to stabilise, duplicates coverage that exists elsewhere, or verifies behaviour nobody depends on is worth removing. What makes deletion legitimate rather than negligent is that the decision is explicit, the owner makes it, and the coverage gap is recorded — the same standard applied to any other scope reduction.

How should triage handle a test nobody wants to own? Treat unowned as a finding rather than a state. Every unmatched path is a gap in the ownership mapping, so the fallback team’s job is to close that gap rather than to absorb the work indefinitely — report the unmatched paths each run, and the list shrinks. A test that genuinely belongs to nobody, because the feature was retired, is a deletion candidate rather than a triage item.

Related guides #

• Routing Flaky Tests to Code Owners
• Writing a Flaky Test Triage Runbook
• Deciding When to Delete a Flaky Test
• Building Auto-Quarantine Workflows
• CI Retry Strategies & Budgets