feat(mirror): mirror-latest — manual post-release GitCode top-up + runbook - #372
Merged
Conversation
…led top-up Enables Plan A for the GitCode big-file gap (probe PR #371 proved GitHub runners cannot upload >8 MiB to GitCode): a CN machine with a healthy GitCode route runs this on a timer to top up whatever CI could not upload. - tools/mirror-latest.sh: resolve the latest published release (gh, or unauthenticated GitHub API fallback), load GITCODE_TOKEN from ~/.config/gitcode-tool/config.json when unset, exec mirror_res.sh. Safe for a non-interactive timer. - tools/mirror_res.sh: replace the eager 'download every asset up front' with expected sizes from ONE release-API call (no body download) + lazy ensure_local() that fetches an asset only when an upload needs it, with 3 retries (CN->GitHub is ~100 KB/s and a single give-up must not abort the whole run — that was the observed failure). Steady-state re-runs now move zero bytes. gh-side failures are collected/reported symmetrically with the gitcode side. Verified: syntax clean; live steady-state run against the fully-mirrored v0.4.65 skips all assets, downloads nothing.
Follow-up hardening after live testing from a CN host: - WANT size-map parse used an f-string with escaped quotes that raised SyntaxError (swallowed by '|| true') -> map always empty -> every GitCode asset fell through to a false 'broken attachment' verdict and the GitHub block re-downloaded all ~48 MB. Use plain concatenation (matches the working 'registered' parse). - probe_ok(): retry cross-border GETs (3x, short cap) so a flaky runner/host probe is not misread as a missing/broken asset; used for the skip check and the final verify. - Source metadata + download now use the PUBLIC GitHub API / release URL (no gh auth) so a headless timer with a locked keyring still works. - gtc PUT socket timeout already 120s; probe cap 90s->60s. Verified from a CN host against the fully-mirrored v0.4.65: 16 skips, 0 downloads, all 16 URLs OK, 40s (was 12m with the empty-WANT bug).
…top-up Per decision: no standing automation — the GitCode big-asset gap is filled by a maintainer running tools/mirror-latest.sh from any CN environment after a release (not tied to a personal machine, no timer). - tools/mirror-latest.sh: reframed header as THE manual post-release step; documents why it's manual (correctness already covered by the GitHub fallback; ~2-4 files/release) and where it must run. - tools/mirror_res.sh: stall + incomplete messages now point to running mirror-latest.sh from a CN environment. - release.yml / mirror-binaries.yml: annotate the known GitHub-runner GitCode big-asset limitation and the manual remediation. - .agents/docs/2026-07-15-gitcode-large-asset-mirror-runbook.md: single source of truth (measurements, rationale, exact procedure).
Sunrisepeak
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the tooling + documentation for finishing the GitCode binary mirror after a release.
Probe PR #371 proved a GitHub-hosted runner cannot upload large (>~8 MiB) assets to GitCode — the cross-border runner→Huawei-Cloud-OBS path sustains only ~15–30 KB/s and every method times out. The same files upload in ~10 s from a CN host, so it is a network-route fact, not a script/GitCode fault.
Decision: no standing automation. A maintainer finishes the GitCode mirror by running one command from any CN environment (a CN shell / cloud terminal / box — not tied to any personal machine). Correctness never depends on it: CN clients already fall back to the GitHub asset URLs (
build_xlings_res_fallback_urls_), so the GitCode copy is a CN-acceleration convenience for the ~2–4 big files per release.tools/mirror-latest.sh xlings # after a release, from a CN-routed environmentWhat it adds
tools/mirror-latest.sh(new): the manual post-release command. Resolves the latest published release (public GitHub API, no auth needed), loadsGITCODE_TOKENfrom~/.config/gitcode-tool/config.jsonwhen unset, thenexecsmirror_res.sh. Idempotent (~40 s no-op when nothing is missing).tools/mirror_res.shhardening (shared by release CI + this command): expected sizes from one release-API call → skip already-mirrored assets → lazy-download only the gaps via public URLs (noghauth) with retries; retrying cross-border probes so a flaky GET is not misread as a missing/broken asset; keeps the fix(release): bound every mirror network op + idempotent re-runs #369/fix(mirror): stall-aware single-retry + broken-attachment detection #370 bounded-upload + stall-no-retry behaviour.release.yml+mirror-binaries.yml: document the GitHub-runner GitCode big-asset limitation and point to the manual step + runbook..agents/docs/2026-07-15-gitcode-large-asset-mirror-runbook.md: single source of truth — the test(ci): GitCode upload network probe [THROWAWAY — do not merge] #371 measurements, why it is manual, the exact procedure, and the validation below.Verified end-to-end (from a CN host)
sha256-identical to source; a second run was a clean idempotent skip. Throwaway GitHub releases were deleted afterward.SyntaxErrorthat silently emptied the size-map (caused false "broken attachment" verdicts) and cross-border probe flakiness (added retry-before-conclude). See commit history.Scope / safety
Touches only release-mirror shell scripts + workflow comments + one design doc — no impact on the build / install / runtime paths. The two CI failures seen during review (
e2epatchelf bootstrap; Windowsbuild-and-testraw.githubusercontent connection reset) were unrelated network flakes and passed on rerun.