Root cause #
Reliability data that lives only in a dashboard is data most contributors never see. A new engineer opening a pull request has no signal that the suite they are touching flakes 4% of the time, so they treat an intermittent red as a fluke and re-run instead of investigating. The badge closes that visibility gap by rendering the current flake rate on the README, but a naive badge — a static image committed once — lies the moment the rate changes.
The honest pattern is a Shields endpoint badge: Shields fetches a small JSON document your CI keeps current, and renders the label, message, and color from it. The value updates whenever CI recomputes the flake rate, so the badge always matches the number your historical flakiness tracking reports.
Step-by-step fix #
1. Publish a Shields endpoint JSON from CI #
Compute the rate, write the endpoint JSON, and publish it to a stable URL (a gist, a pages branch, or an artifact bucket).
// scripts/write-badge.js — Shields endpoint schema, colored by threshold
const { writeFileSync } = require('node:fs');
const rate = Number(process.argv[2]); // flake rate %, computed upstream
const color = rate > 5 ? 'red' : rate > 2 ? 'yellow' : 'brightgreen';
writeFileSync('badge.json', JSON.stringify({
schemaVersion: 1, label: 'flaky', message: `${rate.toFixed(1)}%`, color,
})); // Shields reads this exact shape2. Point the README badge at the endpoint #
<!-- README.md — Shields renders the live endpoint on each page view -->
3. Refresh the endpoint on every main-branch run #
# .github/workflows/ci.yml — recompute and publish on push to main
- name: Publish flakiness badge
if: always() && github.ref == 'refs/heads/main'
run: |
node scripts/write-badge.js "$(cat rate.txt)" # rate.txt from the detection step
# publish badge.json to your pages branch / gist / bucket herePitfalls #
- Committing a static image goes stale — use a Shields endpoint badge.
- Updating on every PR makes the badge branch-specific — refresh on main only.
- No color thresholds hides severity — map the rate to green/yellow/red.
- Expecting instant updates — GitHub’s camo proxy caches images, so allow a short lag.
Reliability targets #
| Metric | Target | How to hit it |
|---|---|---|
| Badge staleness | < 1 CI run behind |
Refresh endpoint on every main run |
| Threshold coloring | 3 tiers (green/yellow/red) | Map rate to color in the JSON |
| Value accuracy | Matches the dashboard exactly | Same computed rate feeds both |
| Refresh coverage | 100% of main runs |
if: github.ref == 'refs/heads/main' |
Frequently Asked Questions #
Why not commit a generated badge image? It is stale the moment the rate changes. A Shields endpoint badge reads a JSON value CI keeps current, so it never lies.
Why does the badge lag after a run? GitHub proxies and caches README images through its camo service, so a fresh value can take a few minutes to appear. Treat the table in the run summary as authoritative for the exact number.
Can one endpoint power per-suite badges? Yes — publish one JSON file per suite and point separate badges at each, or encode the suite in the endpoint query so a single script emits several.
Choosing a Metric That Cannot Be Gamed #
Every visible number creates an incentive, and a badge is the most visible number a repository has. The metric behind it should therefore be one that improves only when the suite genuinely improves.
A raw failure count fails that test badly: it falls when tests are quarantined, when specs are deleted, and when a suite runs less often. Each of those makes the badge greener while making the suite worse, which is precisely the wrong signal to put on a front page.
A flake rate per execution is far more robust, because the denominator moves with the suite. It still has one loophole — quarantining a test removes it from both numerator and denominator — and pairing the rate with a quarantine count closes it. The two together make the trade visible: a falling rate beside a rising quarantine count is not an improvement, and anyone glancing at the pair can see that.
A first-attempt pass rate is a useful third option where retries are in play, since it measures how often the pipeline is green without rescue. It has the pleasant property of degrading immediately when retries start doing more work, which a headline flake rate can mask.
Whichever is chosen, the number should come from the same store as every other reliability metric rather than from a separate calculation. A badge that disagrees with the dashboard loses, and the disagreement is usually a duplicated computation drifting from the original.
What a Badge Can and Cannot Say #
A badge is a single number in a place people glance at, which makes it excellent for one job and unsuited to most others.
It works as a standing reminder of a headline metric: a flake rate, a quarantine count, a first-attempt pass rate. Seeing that number every time someone opens the repository keeps it in the ambient awareness of the team, which is worth more than it sounds for a metric that otherwise lives in a dashboard nobody visits.
It fails as a diagnostic. A badge cannot say which tests are responsible, who owns them, or whether the number moved because the suite improved or because detection changed. Teams that treat it as the primary interface end up debating the number rather than acting on the list behind it.
The practical consequence is to pick the metric a badge can carry honestly. A flake rate over a rolling window is a good candidate: it is comparable over time and its direction is meaningful. A raw failure count is a poor one, because it rises with suite size and falls when tests are quarantined — so a team can improve the badge by hiding tests, which is precisely the wrong incentive.
A second badge showing quarantine count alongside the rate closes that loophole cheaply, since the pair makes the trade visible: a falling rate with a rising quarantine count is not an improvement.
Generating and Refreshing It Safely #
Badges are generated from data, and the generation path is where they go wrong.
The value must come from the same store as every other reliability number, not from a separate calculation, or the badge and the dashboard will disagree and the badge will lose. A scheduled job that reads the rolling rate, writes a small JSON endpoint and lets the badge service render it keeps a single source of truth and avoids embedding computation in the badge layer.
Caching is the second trap. Badge images are aggressively cached by content delivery networks and by the code-hosting platform, so a badge can show a stale number for hours after the underlying value changed. Setting an explicit short cache lifetime, and treating a badge as an approximate indicator rather than a live gauge, avoids the confusion of a green badge on a repository whose suite has been red since morning.
The third consideration is honesty about scope. A badge in a monorepo root that shows one number for a dozen packages tells each team something about somebody else’s tests. Per-package badges in per-package readme files are more work and considerably more useful, and they align with the per-package budgets that make the underlying number actionable in the first place.
One Number, Refreshed on a Schedule #
The refresh cadence deserves a deliberate choice. A badge recomputed on every pipeline reflects the last run rather than the trend, which makes it jump around for reasons nobody can act on. A value computed once a day from a rolling window is stable, meaningful and cheap, and it matches the timescale on which the underlying number actually moves.
Because badge images are cached aggressively by content delivery networks, a badge is an approximate indicator rather than a live gauge in any case. Treating it that way — daily refresh, rolling window, explicit short cache lifetime — avoids the confusion of a stale green badge on a repository whose suite has been red since morning.
Pairing the rate with a quarantine count in the same badge row keeps the trade visible, since a rate that falls because tests were removed from the gating set is not an improvement and should not look like one.