Data Ingestion & Metric Standardization #
Standardize the output format across runners before visualizing. Export Jest, Playwright, or Cypress results to structured JSON and forward to Prometheus (via a push gateway) or Loki. Tag every metric with repo, branch, test_suite, and environment so panels can filter — this aligns ingestion with the broader detection pipeline.
// scripts/push-metrics.js — push a flaky counter to the Prometheus Pushgateway.
const { Gauge, Pushgateway, Registry } = require('prom-client');
const registry = new Registry();
const flakyGauge = new Gauge({
name: 'test_flaky_total',
help: 'Number of flaky test executions',
labelNames: ['suite', 'branch'], // labels enable per-suite/branch filtering
registers: [registry],
});
async function pushMetrics({ suite, branch, flakyCount }) {
flakyGauge.set({ suite, branch }, flakyCount);
const gw = new Pushgateway('http://pushgateway:9091', [], registry);
await gw.push({ jobName: 'ci_test_results' }); // trade-off: pushgateway persists last value — clear stale jobs
}
module.exports = { pushMetrics };Core Panel Configuration #
Deploy four panels: a Stat for daily pass rate, a Time Series for the flake trend, a Table for quarantined tests, and a Gauge for the stability score. Compute a rolling 7-day average in PromQL to smooth daily CI noise, and apply thresholds that turn a panel red above 5%.
# Flakiness rate: passed-on-retry executions over total executions, last 7 days.
sum(increase(test_flaky_total{status="passed_on_retry"}[7d]))
/
sum(increase(test_executions_total[7d])) * 100{ "fieldConfig": { "defaults": { "thresholds": { "mode": "absolute", "steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 3 },
{ "color": "red", "value": 5 }
] } } } }Dynamic Filtering & Drill-Downs #
Add dashboard template variables for repo, branch, and test_status, then link panels so clicking a quarantined row filters the time series to that test. Preserve CI execution context in every drill-down so a filtered view never misattributes a failure.
/ci/artifacts?suite=${__data.fields.suite}&run=${__data.fields.run_id}
Getting the Data Model Right First #
A dashboard is a view over a data model, and most disappointing dashboards are disappointing because the model underneath them cannot answer the questions being asked.
Three modelling decisions do the work. The unit of measurement should be a test execution, since runs and specs contain variable numbers of tests and any rate expressed over them drifts as the suite changes shape. Passes must be recorded, or there is no denominator and a rarely-run test looks identical to a constantly-run one. And run metadata must be captured at write time — image digest, worker count, branch, commit, start hour — because those dimensions cannot be reconstructed afterwards and they are what turn a rate change into an attributable one.
With those in place, the panels become straightforward queries rather than bespoke pipelines, and adding a new question later is a query rather than a re-instrumentation. Without them, each new question requires changing what is collected and waiting weeks for enough history to accumulate.
The related decision is where thresholds live. Encoding them in the panel means every consumer sees a different definition; deriving them from the same committed configuration the pipeline gates on keeps the dashboard and the build agreeing about what “over budget” means.
Panels That Earn Their Space #
A dashboard degrades in proportion to how many panels nobody looks at, so the useful discipline is subtraction. Four panels cover the practical questions for a QA reliability view.
Top offenders by cost, this week. Not by failure count — by frequency multiplied by how many pipelines the test blocks and how long a rerun takes. The panel should show the test name, the owning team and the number, because a name with an owner is a task while a percentage is trivia.
Rate trend with annotations. A single line over ninety days with markers for image bumps, framework upgrades and reliability pushes. Its job is to answer “are we getting better” once a month, not to be watched daily.
Quarantine inflow versus outflow. Two series on one axis. Divergence here predicts the state of the suite months ahead and is invisible in any single-number view.
Retry minutes and infrastructure failures. The cost side, separated from test instability so the platform owner has a number of their own rather than being folded into a metric they cannot move.
Everything else — per-suite breakdowns, duration histograms, pass-rate gauges — is drill-down material that belongs behind a click or in an ad-hoc query rather than on the front page. A dashboard with four panels people read outperforms one with twenty they scroll past.
Keeping the Data Trustworthy #
Panels are only as good as the ingestion behind them, and confidence is lost in three predictable ways.
Incomplete ingestion silently deflates every rate. If cancelled runs, killed shards or one runner’s results fail to land, the denominator is wrong and the dashboard is optimistic. A panel showing expected runs against ingested runs costs one query and makes the gap visible rather than assumed.
Unstable identifiers dissolve history. A test whose name embeds a timestamp or a generated id becomes a new test every run, so its rate is computed over a single execution and no trend can exist. Normalising dynamic segments at ingestion decides whether the data accumulates or evaporates.
Silent definition changes break comparability. When “flaky” shifts from “rescued by a retry” to “failed at least once”, the series becomes a mixture of two measurements. Version the definition, store the version with each row, and annotate the chart at the change.
The unifying principle is that a dashboard should be able to show its own working: a number that cannot be traced to the runs behind it will be disputed the first time it is inconvenient, and a disputed number stops driving decisions.
Common Pitfalls & Troubleshooting #
- Aggregating unrelated suites masks localized flakiness — filter by the suite label.
- Omitting CI environment labels adds staging-versus-production noise.
- Static thresholds miss seasonal volume spikes — prefer dynamic baselines.
- Pass/fail counts alone ignore retry frequency and duration variance.
Core Reliability Metrics #
- Flakiness rate:
(tests passing on retry / total executions) × 100. - Quarantine hit rate:
auto-quarantined / total flagged. - Stability score:
100 − flakiness rate. - MTTR: average time from detection to fix merge.
Troubleshooting FAQ #
How do I prevent Grafana from showing stale metrics after a pipeline failure?
Set a staleness interval in the Prometheus scrape config (default 5 minutes). For infrequent CI, increase it or use last_over_time to forward-fill within a window.
Can I correlate flakiness spikes with dependency updates?
Tag metrics with a dependency_version label at push time and overlay package-update timestamps as Grafana Annotations from a SQL or Loki datasource.
What refresh interval suits a QA dashboard?
5m or 10m — faster intervals add backend load without value, since pipelines run in 15–30 minute batches.
Should the dashboard be the primary way people see this data? No — it should be the drill-down. The numbers that change behaviour arrive where work already happens: touched-test warnings in the pull request, a ranked worklist in the team’s weekly message, a trend in a monthly summary. A dashboard people must remember to visit competes with everything else they could do, and it loses.
How much history should the panels query? Enough to be stable and no more. Ninety days for trends, seven to thirty for the worklist, and a hard minimum execution count on any panel showing a rate. Querying a year of detail to render a weekly view is the usual reason a dashboard becomes slow enough that people stop opening it.
Deciding What Not to Build #
The most common failure in this area is over-building: a dashboard with twenty panels, three variables and a drill-down hierarchy, delivered once and opened rarely.
The economical path is to start with the single query that produces the ranked worklist, publish it as a scheduled message, and add a panel only when someone asks a question the message cannot answer. That order keeps every addition tied to a real question, and it tends to produce four or five panels rather than twenty.
Deriving panel thresholds from the same committed configuration the pipeline gates on keeps the dashboard and the build agreeing about what over-budget means, which is what stops a chart showing green while a merge is blocked.
Start with the query, add the panel only when someone asks for it.