Correlating Flakiness with CI Runner

A time-of-day shape this strong is the signature of contention; no code change produces it.

Root cause #

Every wait in a test suite is an implicit budget against available compute. Rendering, module evaluation, garbage collection and browser start-up all take longer when a runner is sharing its cores with other jobs, so an assertion with 200 ms of headroom on a quiet machine can have none at 10 a.m. Nothing about the test or the application changed — the denominator did.

Two properties make contention distinctive. It affects many unrelated tests at once, because it shifts the whole timing distribution rather than breaking one code path. And it correlates with external variables — hour of day, day of week, queue depth, worker count — rather than with commits. That combination is diagnostic: a flat, broad distribution of failures across the suite that moves with the clock is almost never a set of individual test bugs.

The reason it goes undiagnosed is that each failure looks local. An engineer sees their own test time out, investigates their own test, adds a wait, and moves on; the next person does the same somewhere else. Without the aggregate view, a systemic capacity problem is experienced as fifty separate small ones, and the accumulated response — longer timeouts everywhere — degrades the suite’s ability to detect real regressions.

Step-by-step fix #

1. Record the variables that describe load #

Correlation is impossible without the independent variables. Capture them per run, alongside the results, in the store described in Storing Test History in SQLite for Flake Analysis.

// scripts/run-metadata.js
// Trade-off: a handful of extra columns per run, and they are the only way to
// ask why a rate moved rather than merely observing that it did.
export const runMetadata = () => ({
  started_at: process.env.RUN_STARTED_AT,          // pass in; do not read the clock in-script
  hour_utc: Number(process.env.RUN_STARTED_AT.slice(11, 13)),
  cpu_count: os.cpus().length,
  workers: Number(process.env.PW_WORKERS ?? 1),
  load_avg_1m: os.loadavg()[0],
  concurrent_pipelines: Number(process.env.QUEUE_DEPTH ?? 0),   // from the CI API
  runner_image: process.env.RUNNER_IMAGE_DIGEST,
});

2. Compare rates across the load variable #

The first query is a group-by. If the rate differs materially between buckets, contention is a live hypothesis.

-- Failure rate by hour of day, with the sample size
-- Trade-off: hour buckets are coarse and easy to read; queue depth is a better
-- predictor and needs the CI API to be recorded per run.
SELECT runs.hour_utc,
       COUNT(*)                                                     AS executions,
       ROUND(100.0 * SUM(results.status IN ('flaky','failed')) / COUNT(*), 2) AS rate_pct
FROM results JOIN runs ON runs.id = results.run_id
WHERE runs.started_at > datetime('now', '-30 days')
GROUP BY runs.hour_utc
ORDER BY runs.hour_utc;

A rate that doubles or triples between the quietest and busiest hours is a strong signal. A rate that is flat across hours means the instability is not contention, and the effort belongs on individual tests instead.

3. Check the breadth of the affected set #

Contention spreads failures thinly across many tests; a genuine test bug concentrates them. Measuring the concentration separates the two without any further instrumentation.

-- How many distinct tests failed in peak versus off-peak windows?
-- Trade-off: this is a crude breadth measure and is usually decisive on its own.
SELECT CASE WHEN runs.hour_utc BETWEEN 8 AND 16 THEN 'peak' ELSE 'off-peak' END AS window,
       COUNT(DISTINCT results.test_id) AS distinct_failing_tests,
       COUNT(*)                        AS failure_events
FROM results JOIN runs ON runs.id = results.run_id
WHERE results.status IN ('flaky','failed')
  AND runs.started_at > datetime('now', '-30 days')
GROUP BY window;

Two numbers — distinct tests and failure events — separate a capacity problem from a quality one.

4. Test the hypothesis by changing the load #

Correlation invites an experiment, and this one is cheap. Halve the worker count on a subset of runs and compare rates; if contention is the cause, the rate falls even though the tests are unchanged.

# Trade-off: fewer workers means longer wall-clock time, which is the trade the
# experiment is designed to quantify — slower runs against fewer false failures.
PW_WORKERS=2 npx playwright test    # instead of the usual 4 on a 2-core runner

A suite that is materially more stable at two workers on a two-core runner was over-subscribed, and the correct fix is capacity or concurrency, not test changes.

5. Fix the cause, in order of cost #

Three levers, cheapest first. Reduce over-subscription so worker count matches available cores — usually free and often sufficient. Move heavy suites off peak hours where the work is not merge-blocking, such as nightly full runs. Add capacity, which costs money and is the right answer once the first two are exhausted.

// playwright.config.ts
// Trade-off: matching workers to cores costs wall-clock time on big suites and
// removes the largest single source of contention-driven flakiness.
workers: process.env.CI ? Math.max(1, Math.floor(os.cpus().length / 2)) : undefined,

6. Record the finding so it is not re-diagnosed #

Contention recurs whenever the team grows or the suite does. Publish the hour-of-day rate as a standing chart, so the next occurrence is recognised in a minute rather than investigated for a week.

Pitfalls #

Investigating individual tests. Fifty separate investigations for one systemic cause. Mitigation: check breadth and time-of-day correlation first.
Not recording load variables. Correlation is unaskable after the fact. Mitigation: capture workers, cores, queue depth and hour per run.
Over-subscribing workers. Four workers on two cores multiplies contention. Mitigation: match workers to available cores in CI.
Raising timeouts in response. The suite gets slower and less sensitive to real regressions. Mitigation: fix capacity; treat timeout changes as measurements.
Comparing rates without sample sizes. A 20% rate over five executions is noise. Mitigation: report executions alongside every rate.
Assuming contention because it feels slow. Run the experiment. Mitigation: halve the workers and measure the difference.
Blaming the runner when the image changed too. Two variables moved at once. Mitigation: check the image digest before concluding it is load.

Most teams find the first box resolves it, because worker counts are usually set once and never revisited against the runner's actual size.

Reliability targets #

Metric	Target	Notes
Peak-to-off-peak rate ratio	< 1.3×	Above this, contention dominates
Workers per available core	≤ 1	Measured on the CI runner, not a laptop
Runs with load metadata recorded	100%	Cores, workers, queue depth, hour
Timeout increases per quarter	0	Raise capacity, not deadlines
Time to recognise a contention pattern	< 1 hour	Standing chart, published

The peak-to-off-peak ratio is the single number that tells you whether to look at capacity or at tests.

Frequently Asked Questions #

Q: Our runners are hosted and we cannot see their load. What can we still measure? A: Quite a lot. Hour of day, day of week, your own queue depth, the worker count you configured and the core count reported by the runtime are all available inside the job, and together they explain most contention. Test-duration percentiles are a good proxy for the rest: when the median duration of a stable test rises, the machine is slower, whatever the cause.

Q: Is adding capacity the real answer? A: Sometimes, and it should be the last lever rather than the first, because over-subscription is so common. A suite configured for four workers on a runner with two cores is contending with itself before any other pipeline is involved, and fixing that costs nothing. If matching workers to cores and moving non-blocking suites off peak do not close the gap, then capacity is the honest answer.

Q: How does this differ from environment drift? A: Drift is a change in what the environment is — a new browser build, a different image, another time zone — and it produces a step change with a sharp onset. Contention is a change in how much of the environment you get, and it produces a cyclical pattern that tracks the clock. Checking the image digest alongside the load variables distinguishes them, which is why both are recorded in the fingerprint described in CI Environment & Browser Drift.

Q: What if only some tests are load-sensitive? A: That is the normal case, and it is useful. Tests near their timeout budget fail first under contention, so the set that fails at peak is effectively a ranked list of the tests with the least headroom. Fixing the waits in that set — with the techniques in Handling API Timeouts Without Arbitrary Waits — buys back margin across the whole suite.

Record Now, Correlate Later #

Load variables cannot be reconstructed after the fact, which is the whole argument for capturing them on every run rather than when an investigation starts.

Reporting the peak-to-off-peak ratio alongside the headline rate gives the number context: a suite whose rate doubles during working hours has a capacity problem, and no amount of work on individual tests will move it.