Skip to content

ci: retry the k3d cluster creation on a transient install failure - #14960

Open
asaf-sw wants to merge 1 commit into
mainfrom
asaf/ci-10-retry-k3d-cluster
Open

ci: retry the k3d cluster creation on a transient install failure#14960
asaf-sw wants to merge 1 commit into
mainfrom
asaf/ci-10-retry-k3d-cluster

Conversation

@asaf-sw

@asaf-sw asaf-sw commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Problem

Same class as #14952, different install site. AbsaOSS/k3d-action downloads the k3d binary from GitHub with no retry of its own, so a transient upstream 503 fails system_test_hybrid at a setup step.

Observed on run 31639041988, during the same 503 wave that motivated #14952:

Downloading k3d@v5.5.0 see: https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh
curl: (22) The requested URL returned error: 503
Failed to install k3d
##[error]Process completed with exit code 22

Notably, that run had already survived the Foundry outage thanks to #14952: the Anvil install failed four times and succeeded on the fifth attempt, and then the job died here instead.

Change

Attempt the step twice. continue-on-error is what makes a uses: step retriable, and unlike the composite action in #14952 it is available here, because this is a workflow step rather than a step inside a composite action.

Between attempts, a cleanup step discards a partial cluster. The observed failure is the binary download, which leaves no cluster behind, but a first attempt that got as far as creating one would otherwise make the retry fail on an existing cluster. It is guarded by command -v k3d, since a failed download means no k3d binary to call. It also waits 60 seconds, so the retry is not fired into the same outage.

The cluster name is passed through env: rather than interpolated into the run: block, matching the fix Semgrep asked for on #14952.

Scope

One retry, not five. Duplicating the with: block is the cost of retrying a uses: step, so this trades a second attempt for readability, and a sustained multi-minute outage will still fail the job. That is a deliberate limit rather than an oversight: the alternative is reimplementing the action's install in shell, which is a bigger change with more ways to break the 14 workflows that depend on this job.

Verification

The workflow parses, and the three k3d steps are in the intended order with continue-on-error set only on the first:

yaml ok; k3d-related steps in order:
   - Create k3d cluster (Local k8s)
   - Discard a partial k3d cluster before retrying
   - Retry creating k3d cluster (Local k8s)
continue-on-error: True

This one cannot be proven locally, since the failure it handles is an upstream outage. The real evidence would be a future 503 that costs a retry instead of a job.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@reviewable-StarkWare

Copy link
Copy Markdown

This change is Reviewable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants