ci: raise build stage budget 16->24 and retry flaky artifact downloads#3
Merged
Merged
Conversation
A clean ungoogled-chromium-based build needs more than 16 sequential
"Run Stage" checkpoints to finish ninja: the last x64 tag build ran
~39.5 wall-hours across all 16 stages and still hit the hard-coded cap
("Build did not finish after 16 stages."); arm (~63h) and x86 (~43h)
blew past it too. Raise the cap to 24 stages (each stage's own ninja
timeout + the staged build/src checkpoint carry-over are unchanged),
giving enough resumable budget for a clean build to actually complete.
Separately, the same x64 run's stage-10 job died outright (not from
the stage cap) when downloading the ~14GB inter-stage checkpoint
artifact: "Artifact download failed after 5 retries." The upload path
already retries 5x with backoff; the download path had no retry at
all, so one flaky multi-GB transfer discarded ~35 build-hours of
already-completed ninja work and failed the whole tag build. Wrap the
download in an outer retry (3 attempts, 60s backoff) mirroring the
existing upload retry loop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
reusable-build.yml) has never completed: it chains sequential "Run Stage" jobs, each ninja-timeboxed to 3.5h, capped at 16 stages, then hard-fails withBuild did not finish after 16 stages.The last x64 tag attempt ran ~39.5 wall-hours across all 16 stages and still hit the cap; arm (~63h) and x86 (~43h) blew past it further. Raised the cap to 24 stages — same per-stage ninja timeout, same stagedbuild/srccheckpoint carry-over (already resumes ninja correctly between stages, verified in.github/actions/stage/index.js) — just more of them.gh run view --log-failedon run 28727618024). The upload path inindex.jsalready retries 5x with backoff; the download path had none, so a single flaky multi-GB transfer discarded ~35 build-hours of completed ninja work and failed the whole tag build regardless of the stage cap. Added an outer retry (3 attempts, 60s backoff) around the download, mirroring the existing upload retry loop.Verified
build-x64run 28727618024 (16/16 stages, cap-exhausted at build-16, but ALSO an unretried download failure at build-10),build-armrun 28727617984 (16/16 stages all succeeded individually,completefailed purely on the stage cap — the clean stage-cap-exhaustion case),build-x86run 28727617973 (same cap pattern).reusable-build.ymlprogrammatically (24 near-identical stage blocks) and validated the result parses as well-formed YAML with the correctneedschain (python3 -c "import yaml; ..."— confirmed 25 jobs,build-24needsbuild-23,completeneedsbuild-24).node --check .github/actions/stage/index.jspasses.package.json/Cargo.tomlat repo root, so the version-increment gate is a no-op (matches prior PRs in this repo).Bump rationale
dig-browser is grandfathered (no commitlint/version-gate imposed) — CI-config-only change, no product version bump.
Closes DIG-Network/dig_ecosystem#48