Retrying Only Failed Shards in GitHub Actions

Root cause #

Sharding multiplies exposure. If each shard has a 1% chance of an unrelated blip, a twelve-shard suite has roughly an 11% chance that at least one shard fails, and re-running all twelve carries the same 11% again. That is why large suites develop a folklore of “just re-run it, twice if you have to”: the retry strategy is fighting a probability it keeps recreating.

The second mechanism is attribution. A full rerun replaces the whole result set, so the record of which shard was unstable is overwritten by a green run. The flakiness data that a budget depends on quietly disappears, and the pipeline looks healthier than it is — the accounting failure described throughout Historical Flakiness Tracking & Analytics.

The third is cost, which grows with the square of the team’s tolerance. Every full rerun costs the whole suite’s runner minutes, and a team that reruns twice a day on a twenty-minute twelve-shard suite is spending several hours of compute a week rescuing single-shard blips.

Step-by-step fix #

1. Shard deterministically so a shard index means something #

Selective retry requires that “shard 3” refers to the same set of tests on the retry as on the first attempt. Playwright’s built-in sharding is deterministic given the same test list, which is what makes the index a stable handle.

# .github/workflows/e2e.yml
# Trade-off: static sharding is deterministic and can be unbalanced; duration-
# based sharding is faster and must record its assignment for retry to work.
jobs:
  e2e:
    strategy:
      fail-fast: false            # one failing shard must not cancel the others
      matrix:
        shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --shard=${{ matrix.shard }}/12
      - name: Record the outcome
        if: always()
        run: echo "${{ job.status }}" > shard-${{ matrix.shard }}.status
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: shard-${{ matrix.shard }}-result
          path: |
            shard-${{ matrix.shard }}.status
            results.json

fail-fast: false is essential: with the default, the first failing shard cancels the rest and you lose the information about which others would have passed.

2. Collect the failed shard indices #

A gather job reads the artifacts and produces the list of shards to retry, as a matrix input for the next job.

  collect:
    needs: e2e
    if: always()
    outputs:
      failed: ${{ steps.list.outputs.failed }}
    steps:
      - uses: actions/download-artifact@v4
        with: { pattern: 'shard-*-result', path: results }
      - id: list
        # Trade-off: shelling out to jq keeps this readable; a small Node script
        # is easier to test if the logic grows beyond "which files say failure".
        run: |
          FAILED=$(for f in results/shard-*-result/shard-*.status; do
            idx=$(basename "$f" .status | sed 's/shard-//')
            grep -q failure "$f" && echo "$idx"
          done | jq -R . | jq -sc .)
          echo "failed=$FAILED" >> "$GITHUB_OUTPUT"
          echo "failed shards: $FAILED"

3. Re-run only those shards, once #

The retry job runs a matrix built from the collected list, and is skipped entirely when the list is empty.

  retry:
    needs: collect
    if: needs.collect.outputs.failed != '[]'
    strategy:
      fail-fast: false
      matrix:
        shard: ${{ fromJson(needs.collect.outputs.failed) }}
    steps:
      - uses: actions/checkout@v4
      # Trade-off: a single retry pass bounds the cost and means a genuinely
      # broken shard still fails, which is what you want it to do.
      - run: npx playwright test --shard=${{ matrix.shard }}/12

4. Keep the first-attempt record #

The retry must not erase the evidence. Upload both attempts’ results and feed the difference into the flakiness store, so a shard that needed a retry is counted even though the pipeline ended green.

// scripts/record-shard-retries.js
// Trade-off: keeping both attempts means more artifacts to manage and is the
// only way a selective retry does not quietly improve your statistics.
const first = loadResults('attempt-1');
const second = loadResults('attempt-2');

const rescued = first.failures.filter((f) => !second.failures.some((s) => s.title === f.title));
recordFlaky(rescued);            // counted against the budget

5. Prefer duration-based sharding, and record the assignment #

Static index sharding is deterministic but often unbalanced — one shard finishes in four minutes and another in fourteen. Duration-based partitioning fixes the imbalance but makes the shard index depend on timing data, so the assignment has to be recorded for the retry to run the same tests.

// Trade-off: balanced shards cut wall-clock time significantly and require the
// partition to be persisted, or a retry may run a different set of tests.
const partition = partitionByDuration(testList, durations, { shards: 12 });
writeFileSync(`shard-assignment.json`, JSON.stringify(partition));

6. Cap the retry and escalate the rest #

One selective retry is a blip absorber. A shard that fails twice has a real problem — either a genuine regression or a test that belongs in quarantine — and should be treated as such rather than retried a third time.

Pitfalls #

• Leaving fail-fast at its default. The first failing shard cancels the others and you cannot tell which else would have failed. Mitigation: set it to false.
• Re-running the whole workflow. Costly, and it overwrites the flakiness record. Mitigation: selective retry with preserved artifacts.
• Non-deterministic shard assignment. The retry runs different tests than the failure. Mitigation: persist the partition and reuse it.
• Not recording the first attempt. The rescued failures vanish from the statistics. Mitigation: count rescued tests against the budget.
• Unlimited selective retries. A broken shard cycles forever. Mitigation: one retry pass, then escalate.
• Retrying a shard whose failure was environmental. The same runner problem recurs. Mitigation: check the fingerprint before retrying, as in CI Environment & Browser Drift.

Reliability targets #

Frequently Asked Questions #

Q: Does Playwright’s built-in retries make this redundant? A: No, they solve adjacent problems. Test-level retries handle an individual test blip inside a shard and are the first line. Shard retry handles a failure that kills the whole shard — a runner dying, a service failing to start, an out-of-memory condition — where no per-test retry can help because the process is gone.

Q: Why not just re-run the failed job from the interface? A: You can, and it is the manual version of the same idea. Automating it matters because a manual re-run does not record what was rescued, happens only when someone is watching, and tempts people into re-running everything when the interface makes that the easier button.

Q: Should the retry run on a fresh runner or reuse the original? A: A fresh one. Part of the value of a retry is escaping whatever local condition caused the failure — a noisy neighbour, exhausted memory, a service that failed to start — and reusing the same machine keeps that condition in place. A fresh runner also makes the retry a genuinely independent trial, which is what the statistics behind the budget assume.

Q: Our shards are wildly unbalanced. Does that break this? A: It makes it less effective, not incorrect: a slow shard retried alone still costs its full duration. Balance by partitioning on recorded durations rather than by index, persist that partition, and reuse it on the retry so the same tests run again.

Q: What happens if the collector job itself fails? A: Treat that as a pipeline failure rather than letting it pass silently. The collector runs with if: always(), so it executes even when shards fail — but if it errors while parsing artifacts, the retry job’s matrix input is missing and the workflow can end green with failures unaddressed. Fail closed: if the collector cannot determine the failed set, mark the run red and let a human look.

Q: Does this work with other CI providers? A: The mechanism is generic — run shards, record per-shard status, compute the failed set, run a second matrix from it. What differs is how each provider expresses a dynamic matrix, and some require the retry job to be a separate pipeline triggered with parameters rather than a downstream job. The constraint that matters everywhere is deterministic shard assignment; without it, the retry runs a different set of tests.

Q: How does this interact with the flake budget? A: Directly. Every failure rescued by a shard retry must be recorded and counted, or selective retry becomes a cheaper way of hiding the rate — the same failure mode as an unreported test retry, and the reason Setting a Retry Budget That Fails the Build insists on reporting before enforcement.

Related guides #

• CI Retry Strategies & Budgets
• Preventing Sharded Runner Seed Collisions
• Embedding Flakiness Summaries in GitHub Actions