Routing Flaky Tests to Code Owners

Root cause #

Flakiness reports are aggregate by nature — a rate, a list, a dashboard — and aggregates have no owner. When a report lands in a shared channel, the reasonable behaviour for each individual reading it is to assume it concerns somebody else, because for any given item it probably does. The result is a well-instrumented team that measures its flakiness precisely and fixes very little of it.

The second mechanism is timing. A flake detected today is cheap to fix today, while the change that introduced it is small and recent, and expensive to fix in six weeks when the author has moved on and the surrounding code has changed. Manual triage introduces exactly that delay: someone has to notice, decide, and assign, and each step waits for a human with other priorities.

The third is that ownership data goes stale unless it is maintained for another reason. A dedicated flaky-test ownership spreadsheet is accurate for a quarter and wrong thereafter. CODEOWNERS stays current because review routing depends on it every day, which makes it the right source for test routing even where its granularity is imperfect.

Step-by-step fix #

1. Resolve an owner from the file path #

Parse CODEOWNERS and match the failing spec’s path, taking the last matching rule as the file format specifies.

// scripts/codeowners.js
// Trade-off: a small parser handles the common syntax and not every edge case
// of the format; it is enough for routing, and it fails to the fallback safely.
import { readFileSync } from 'node:fs';
import { minimatch } from 'minimatch';

const rules = readFileSync('.github/CODEOWNERS', 'utf8')
  .split('\n')
  .map((l) => l.replace(/#.*$/, '').trim())
  .filter(Boolean)
  .map((l) => { const [pattern, ...owners] = l.split(/\s+/); return { pattern, owners }; });

export function ownersFor(filePath, fallback = '@qa-platform') {
  const matched = rules.filter((r) => minimatch(filePath, r.pattern.replace(/^\//, '')));
  return matched.at(-1)?.owners ?? [fallback];    // last match wins, per the format
}

2. Route by class, not only by path #

The spec’s owner is the right recipient for a test-quality problem. An infrastructure failure belongs to the pipeline owner, and a suspected product race belongs to the feature team with a different framing — a possible bug rather than a flaky test.

// Trade-off: class-based routing needs the classifier to be roughly right, and
// mis-routing is cheaper than the alternative of one undifferentiated queue.
const ROUTES = {
  infrastructure: () => ['@platform-ci'],
  'product-suspect': (file) => ownersFor(file),      // same team, different label
  test: (file) => ownersFor(file),
  unknown: () => ['@qa-platform'],
};

const owners = ROUTES[classify(failure)](failure.file);
const label = classify(failure) === 'product-suspect' ? 'possible-product-race' : 'flaky';

3. Create the item automatically, once per test #

Automation is what removes the delay. Deduplicate on a stable key so a persistent offender accumulates evidence instead of duplicates.

# .github/workflows/triage-flaky.yml
# Trade-off: auto-filing creates some noise on a bad week; deduping and a
# frequency threshold keep it proportionate.
- name: Route flaky tests
  if: always()
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: node scripts/route-flaky.js flaky.json

// scripts/route-flaky.js
// Trade-off: filing only above a threshold avoids an issue for every one-off
// blip, at the cost of a short delay before a new flake gets an owner.
for (const test of flaky.filter((t) => t.ratePct >= 1)) {
  const key = `flaky:${sha1(`${test.file}::${test.title}`).slice(0, 12)}`;
  const existing = await findIssue(key);
  if (existing) {
    await comment(existing, `Failed again (rate ${test.ratePct}%) in run ${runId}.`);
  } else {
    await createIssue({
      title: `Flaky: ${test.title}`,
      assignees: ROUTES[classify(test)](test.file),
      labels: [labelFor(test)],
      body: `${key}\nFile: ${test.file}\nRate: ${test.ratePct}%\nExpires: ${expiry(28)}`,
    });
  }
}

4. Notify the team’s channel, not a shared firehose #

Route the notification to where the owning team already works. A message in a channel the team reads is acted on; a message in a general engineering channel is scrolled past.

// Trade-off: per-team routing needs a team-to-channel mapping to maintain,
// which is small and is the difference between a message read and ignored.
const CHANNELS = { '@payments': 'C01PAY', '@discovery': 'C02DIS', '@platform-ci': 'C03OPS' };
await postToSlack(CHANNELS[owners[0]] ?? CHANNELS['@platform-ci'], summary(test));

The alerting mechanics and dedupe rules are the same ones set out in Sending Slack Alerts for Flakiness SLO Breaches.

5. Make unowned paths visible and fix them at the source #

Every fallback assignment is a gap in CODEOWNERS. Report them so the mapping improves rather than the fallback team absorbing an ever-growing share.

// Trade-off: reporting gaps creates a small ongoing task and prevents the
// fallback owner from silently becoming the owner of everything.
const unmatched = flaky.filter((t) => ownersFor(t.file, null) === null);
if (unmatched.length) {
  console.log(`::warning::${unmatched.length} spec path(s) have no CODEOWNERS entry`);
  for (const t of unmatched) console.log(`  ${t.file}`);
}

6. Close the loop when the test recovers #

An item that stays open after the test stabilises trains people to ignore the label. Close it automatically when the rate falls below the threshold for a sustained period, with a comment recording the recovery.

Pitfalls #

• No fallback owner. Unmatched paths accumulate unassigned forever. Mitigation: an explicit default team, plus a report of the gaps.
• Routing by path alone. Infrastructure failures land with spec owners. Mitigation: route by classification as well as path.
• An issue per failure. The label becomes unusable. Mitigation: deduplicate on a stable key.
• Filing on the first occurrence. One-off blips generate noise. Mitigation: file above a frequency threshold over a rolling window.
• A shared firehose channel. Everyone assumes it is someone else’s. Mitigation: notify the owning team’s own channel.
• Never closing recovered items. The backlog looks worse than it is and the label loses meaning. Mitigation: auto-close on sustained recovery.
• Maintaining a separate ownership spreadsheet. It goes stale within a quarter. Mitigation: derive from CODEOWNERS.

Reliability targets #

Frequently Asked Questions #

Q: Should the assignee be a team or an individual? A: A team. Individuals are on holiday, change roles and leave; a team handle survives all three, and the team decides internally who picks it up. Assigning to the last person who touched the file feels precise and produces items that sit unread for a fortnight.

Q: Our specs live in one directory owned by QA. How do we route by feature? A: Either give the spec directory finer-grained CODEOWNERS entries mirroring the feature structure, or map from the test’s own metadata — a tag or an annotation naming the owning team. The second is often easier to introduce and has the advantage that the ownership travels with the test if the file moves.

Q: Is auto-filing issues going to overwhelm the tracker? A: Only without a threshold and deduplication. Filing at one percent over a rolling window, one durable item per test, closing on recovery: in a healthy suite that is a handful of open items at any time. If it is more than that, the tracker is telling you something accurate about the suite.

Q: How do we route tests that span several teams’ code? A: Route to the team that owns the spec, and mention the others. A checkout flow touching payments, cart and identity has one team that wrote the test and knows what it is asserting; that team is the right first responder and can pull in the others once the failure is classified. Assigning to three teams at once reliably means nobody starts.

Q: Should routing depend on who last changed the test? A: Only as a tie-breaker. Blame-based assignment is unreliable — the last change may have been a rename or a lint fix — and it creates an incentive to avoid touching test files. Ownership from CODEOWNERS is more stable and matches how the rest of the repository already routes work.

Q: What if the classifier routes a product race to the wrong team? A: Correcting a mis-route costs one reassignment, which is much cheaper than the alternative of never routing at all. Track how often items are reassigned and use it to improve the classifier — a steady reassignment rate above roughly a fifth means the signatures need work, not that the routing was a bad idea.

Q: Does the routing need to run on every pipeline? A: Running it on trunk builds is enough, and it avoids filing items from a branch whose changes were never merged. Pull-request runs still produce useful data for the flakiness history, but the assignment step belongs where the failure represents the shared state of the codebase rather than someone’s work in progress.

Related guides #

• Flaky Test Triage & Ownership
• Writing a Flaky Test Triage Runbook
• Building Auto-Quarantine Workflows