Calculating Flake Rate From CI History

Flake rate is the share of runs in a window where a test recovered on retry rather than passing or failing cleanly.

Root cause #

Flakiness is statistical, so any metric computed from one build is noise. A test that fails 1 run in 50 looks identical to a hard failure if you only see the failing build. The data you need already exists — every CI run emits a JUnit XML or JSON report listing each test’s outcome and whether a retry recovered it — but it is scattered across thousands of ephemeral artifacts that disappear when the runner is recycled.

The job is therefore aggregation, not detection. You persist each run’s per-test outcomes into a durable store keyed by a stable test identifier (file plus full title), then compute flake rate over a rolling window of recent runs. The standard definition: flake rate is the number of runs in which a test produced a flaky outcome (failed at least once, then passed on a retry) divided by the total number of runs of that test in the window. A rolling window keeps the number responsive — a test fixed last week should not be penalized by failures from a month ago.

Persisting per-test outcomes keyed by a stable id turns ephemeral artifacts into a trend.

Step-by-step fix #

1. Normalize each run’s report into rows #

Parse the JUnit XML or runner JSON into one record per test per run, with a stable id and a status of passed, failed, or flaky.

// ingest.js — normalize one run into rows for the store
const { XMLParser } = require('fast-xml-parser');
const fs = require('fs');

function toRows(xmlPath, runId, ts) {
  const doc = new XMLParser({ ignoreAttributes: false }).parse(fs.readFileSync(xmlPath));
  const cases = [].concat(doc.testsuites?.testsuite ?? []).flatMap((s) => [].concat(s.testcase ?? []));
  return cases.map((c) => ({
    run_id: runId,
    ts,
    test_id: `${c['@_classname']}::${c['@_name']}`,
    // 'flaky' = recovered on retry; JUnit marks reruns with a 'flakyFailure' node.
    status: c.flakyFailure ? 'flaky' : c.failure ? 'failed' : 'passed',
  }));
}
module.exports = { toRows };
// Trade-off: keying on classname+name is stable across runs but breaks
// if you rename a test; treat a rename as a new id and accept a reset.

2. Persist rows to a durable store #

Append rows to a table that outlives the runner. One row per test per run keeps the window query simple.

-- schema.sql
CREATE TABLE IF NOT EXISTS test_runs (
  run_id   TEXT NOT NULL,
  ts       TIMESTAMP NOT NULL,
  test_id  TEXT NOT NULL,
  status   TEXT NOT NULL CHECK (status IN ('passed','failed','flaky'))
);
-- Index the window query path: most-recent rows per test.
CREATE INDEX IF NOT EXISTS idx_runs_test_ts ON test_runs (test_id, ts DESC);
-- Trade-off: storing every row is simple and auditable but grows fast;
-- prune rows older than the longest window you query (CI storage cost).

3. Compute flake rate over a rolling window #

Aggregate the last N runs per test. Flake rate is flaky outcomes over total runs in the window.

-- flake_rate.sql — rolling 50-run window per test
WITH windowed AS (
  SELECT test_id, status,
         ROW_NUMBER() OVER (PARTITION BY test_id ORDER BY ts DESC) AS rn
  FROM test_runs
)
SELECT test_id,
       COUNT(*) AS runs,
       SUM(CASE WHEN status = 'flaky' THEN 1 ELSE 0 END) AS flaky_runs,
       ROUND(100.0 * SUM(CASE WHEN status = 'flaky' THEN 1 ELSE 0 END) / COUNT(*), 2) AS flake_rate_pct
FROM windowed
WHERE rn <= 50            -- rolling window of the last 50 runs
GROUP BY test_id
HAVING COUNT(*) >= 10     -- ignore tests with too little history to be meaningful
ORDER BY flake_rate_pct DESC;
-- Trade-off: a 50-run window reacts fast but is jumpy for rarely-run
-- tests; widen it for low-frequency suites to avoid noisy percentages.

4. Or aggregate in Node when there is no database #

For small teams a JSON ledger and an in-memory reduce is enough. Same formula, no SQL.

// flake-rate.js — compute per-test flake rate from a JSON ledger
const rows = require('./test_runs.json'); // [{test_id, ts, status}, ...]
const WINDOW = 50;
const byTest = new Map();
for (const r of rows.sort((a, b) => b.ts - a.ts)) {
  const bucket = byTest.get(r.test_id) ?? [];
  if (bucket.length < WINDOW) bucket.push(r.status); // keep newest N
  byTest.set(r.test_id, bucket);
}
const report = [...byTest].map(([test_id, s]) => ({
  test_id,
  runs: s.length,
  flake_rate_pct: +(100 * s.filter((x) => x === 'flaky').length / s.length).toFixed(2),
})).filter((r) => r.runs >= 10);
require('fs').writeFileSync('flake-rate.json', JSON.stringify(report, null, 2));
// Trade-off: in-memory is fine for thousands of rows but will not scale
// to full history; move to the SQL store once the ledger gets large.

Only failed-then-recovered outcomes belong in the numerator — a hard failure is not flakiness.

Pitfalls #

Counting one build as the rate: a single failure is not a percentage. Mitigation: require a minimum window (e.g. 10 runs) before reporting.
Unstable test ids: renames split history. Mitigation: key on a slug you control and treat renames as deliberate resets.
Mixing failed and flaky: a hard failure is not flakiness. Mitigation: only count flaky (failed-then-recovered) outcomes in the numerator.
Infinite history: old failures haunt fixed tests. Mitigation: use a rolling window and prune rows past the longest window.
Sharded double-counting: one logical run emits several reports. Mitigation: dedupe by run_id before aggregating.

A 50-run window (≥10 minimum) reacts fast without penalizing a test fixed last week.

Reliability targets #

Metric	Target
Rolling window size	50 runs (≥ 10 minimum)
Per-test flake rate	< 1%
Suite-wide flake rate	< 0.5%
Quarantine threshold	≥ 5% flake rate
History retention	≥ longest window

A minimum window of 10 runs keeps a single failure from masquerading as a rate.

Frequently Asked Questions #

Q: What exactly counts as a flaky outcome? A: A run in which the test failed at least one attempt and then passed on a retry. A clean pass and an all-attempts failure are both excluded from the numerator; only recovery-on-retry is flakiness.

Q: Why use a rolling window instead of all history? A: A rolling window makes the rate responsive. A test fixed last week should trend toward zero quickly, which it cannot do if failures from months ago stay in the denominator forever.

Q: How do I visualize the result? A: Feed the per-test percentages to a dashboard. See tracking test flakiness trends over time for trends and the reliability dashboards for QA teams guide for charting.

Defining the Numerator and the Denominator #

Most disagreements about a flake rate are disagreements about its definition rather than its arithmetic, and writing the definition down next to the query removes an argument that otherwise recurs quarterly.

The denominator should be test executions, not runs and not specs. A run contains a variable number of tests, and a spec containing twenty tests is not comparable with one containing three, so any rate expressed per run drifts as the suite changes shape. Executions include passes — a point worth stating explicitly, because a store that records only failures cannot produce a rate at all.

The numerator needs a decision about what counts. Rescued failures — a test that failed and passed on retry within the same run — are the core case. Whether to include hard failures is a genuine choice: including them measures “tests that failed for any reason”, which conflates broken tests with unstable ones; excluding them measures instability specifically, which is usually what the budget is meant to control. Recording both separately and choosing at query time avoids committing to one.

Two exclusions are worth applying consistently. Infrastructure failures belong to a separate metric with a different owner. And quarantined tests should be reported separately, or the headline rate improves every time something is quarantined, which rewards hiding rather than fixing.

-- Rate per execution, with rescued and hard failures kept distinguishable.
-- Trade-off: two numerators is slightly more complex and prevents an argument
-- about whether a broken test counts as flaky.
SELECT test_id,
       COUNT(*)                        AS executions,
       SUM(status = 'flaky')           AS rescued,
       SUM(status = 'failed')          AS hard,
       ROUND(100.0 * SUM(status = 'flaky') / COUNT(*), 2) AS flake_rate_pct
FROM results
WHERE run_started_at > datetime('now', '-30 days')
  AND NOT quarantined
GROUP BY test_id
HAVING executions >= 20;

Confidence, Sample Size and When Not to Report #

A rate computed from a handful of executions is not a rate, and applying a threshold to one manufactures the false alarms that get monitoring switched off.

The minimum sample worth reporting depends on the threshold being applied: detecting a one percent rate reliably needs hundreds of executions, while a five percent rate is visible in dozens. A practical rule is to suppress the rate entirely below twenty executions in the window and to show the raw counts instead, which is honest about what the data supports.

The complementary habit is to show sample size alongside every rate. A dashboard row reading “4.0% (2/50)” invites a very different response from “4.0% (40/1000)”, and omitting the denominator is how a team ends up investigating a test that failed twice on a quiet week.

Where a rate must be gated on despite a small sample — a newly added test, a rarely-run suite — the productive alternative is active measurement: run it a few hundred times deliberately and derive the rate from that, rather than waiting weeks for passive history to accumulate enough executions to say anything.