Data Collection & Aggregation Strategy #
Trend tracking begins with consistent metadata. Export execution timestamps, environment variables, retry counts, and failure traces to a time-series store, normalize across parallel runners to remove infrastructure skew, and tag each execution with a commit hash and deployment id so a spike can be correlated with a code or environment change.
{
"reporter": [
["json", { "outputFile": "results/flakiness-report.json" }],
["list"]
],
"globalSetup": "./setup-track-metadata.js"
}For Jest, use the built-in JSON reporter (there is no standard flakiness-tracker package) and parse numFailingTests versus numPassingTests per run alongside jest.retryTimes().
Trend Visualization & Threshold Configuration #
Plot the rate with rolling averages — a 7-day window for immediate changes and a 30-day for systemic drift — and set alert thresholds from historical standard deviation, not a static percentage. When the rate crosses the upper control limit, trigger a diagnostic workflow automatically.
// scripts/compute-threshold.js — 3-sigma upper control limit for flake rate.
function computeUCL(rates) {
const mean = rates.reduce((a, b) => a + b, 0) / rates.length;
const variance = rates.reduce((sum, r) => sum + (r - mean) ** 2, 0) / rates.length;
const stdDev = Math.sqrt(variance);
return mean + 3 * stdDev; // flag any rate above this — trade-off: 3σ is conservative, tune per suite
}
module.exports = { computeUCL };Diagnostic Workflows for Trend Spikes #
A sustained breach triggers structured triage: verify infrastructure health (CPU throttling, latency, container limits), audit async operations and race conditions in the spec, then cross-reference the spike timeline with dependency upgrades. Record findings in a reliability log so the same cause is not re-investigated.
Choosing the Window, and Detecting Change #
A trend is a rate computed over a window, and the window changes the answer more than the data does.
A short window reacts quickly and is noisy: a test executing twenty times a week produces a rate with wide error bars, so a single bad day can push it over a threshold and it resolves itself the next day. Gating on that produces alerts that clear before anyone looks, which is the fastest way to train a team to ignore them.
A long window is stable and slow. Ninety days of history is comparable across quarters and ideal for budget setting, and it dilutes a regression that started last week to the point where no threshold notices for a fortnight.
Detecting change needs both. Compute a short-window rate and a long-window baseline for each test, and flag the ones where the recent rate is substantially above the baseline — that comparison finds regressions neither window alone would surface, and it distinguishes “this test has always been bad” from “this test got worse on Tuesday”, which are different problems with different owners.
-- Regression = recent rate well above the established baseline, with enough samples.
-- Trade-off: noisier than either window alone, and the only view that answers
-- "what changed recently" rather than "what is bad in general".
SELECT test_id, rate_over(7) AS recent, rate_over(90) AS baseline
FROM test_rates
WHERE executions_over(7) >= 20
AND rate_over(7) > rate_over(90) * 2
ORDER BY recent DESC;The minimum execution count is not optional: a rate computed from five runs is not a rate, and applying a threshold to one manufactures exactly the false alarms that get trend monitoring switched off.
Annotating the Timeline #
A trend line without context invites speculation. The same chart becomes decisive when the events that could explain a change are drawn on it.
Four annotations carry most of the value. Runner image changes, because a step change aligned with a digest bump and no merge is environment drift rather than test decay. Dependency or framework upgrades, which typically produce a wave of unrelated changes at once. Worker-count or infrastructure changes, which shift contention. And deliberate reliability work, so the effect of a fix campaign is visible rather than argued about.
Each of these is already recorded somewhere — a commit, a pipeline configuration change, a ticket — and pulling them onto the timeline turns a conversation about whether the suite is improving into an examination of what changed when. It also protects reliability work from the common fate of being invisible: a chart showing the rate halving after a two-week effort is a far better argument for the next one than a recollection that things feel better.
The habit worth building is to annotate at the time rather than reconstructing later. A one-line entry when an image is bumped costs nothing; reconstructing three months of changes during an incident costs a day.
Common Pitfalls #
- Masking root causes with excessive retries, which suppresses the trend.
- Ignoring environment drift between CI runners and local setups.
- Correlating spikes with code changes only, overlooking third-party rate limits or CDN caching.
- Using static pass/fail thresholds instead of statistical process control.
Reliability Metrics #
- Flakiness rate:
(failures / total executions) × 100. - Mean time between flakes (MTBF): per suite.
- Trend slope: rate of change over rolling windows — aim for ≤ 0.
- Quarantine duration: average days a test stays isolated.
- Retry success ratio: share of passes on first retry.
FAQ #
How do I distinguish genuine flakiness from a real regression? A regression fails consistently across environments and commits; flakiness is non-deterministic under identical conditions. Cross-reference failure traces with retry logs and environment metrics.
What is the optimal rolling window? Use both: a 7-day window catches immediate pipeline changes, a 30-day reveals systemic infrastructure or framework drift.
Should trends auto-quarantine on a spike? Only after a sustained breach — e.g. more than three consecutive days above the control limit — plus a failed manual triage. Premature quarantine hides the root cause.
Why does the trend look worse after we started fixing things? Usually because detection improved at the same time. Enabling retry reporting, ingesting previously missing shards or normalising test identifiers all increase the measured rate without changing the underlying suite. Annotate the change and treat the series before and after as two measurements rather than one trend — comparing across a definition change is the most common way a reliability programme convinces itself it is failing.
Should the trend include quarantined tests? Report them separately. Including them in the headline rate makes the number improve every time something is quarantined, which rewards exactly the wrong behaviour. A quarantine lane rate alongside the blocking-suite rate keeps both visible, and the pair — blocking rate falling while quarantine inflow rises — is the signature of a team hiding rather than fixing.
How far back is the history actually useful? Ninety days of detail answers essentially every operational question; beyond that a monthly summary preserves the shape at a fraction of the size. The value of older detail is almost entirely in answering “was this test always like this”, which a summary answers just as well.
Trends Answer Different Questions at Different Zoom Levels #
A single trend chart is asked to serve three questions, and separating them makes each one answerable.
At a weekly zoom the question is “did something regress” — a short-window rate compared against the established baseline, filtered to tests with enough executions to be meaningful. At a quarterly zoom the question is “is the programme working”, which needs the long-window series with annotations for the interventions that were made. And at a per-test zoom the question is “when did this one change”, which is a single series with its own history and the commits that touched it.
Trying to answer all three from one view produces a chart that is too noisy for the quarterly question and too coarse for the weekly one. Keeping the underlying data identical and varying only the window, the filter and the annotations is what lets one store serve all three without anyone maintaining three pipelines.
Publishing the trend alongside the interventions that were made is what turns a chart into an argument. A rate halving after a two-week reliability effort is a far stronger case for the next one than anybody’s recollection that things feel better, and it costs one annotation per intervention to keep.
A trend that nobody reviews is data rather than information, so pair the chart with a short scheduled message naming the tests behind any movement.