Repository navigation
CI: tolerate transient sccache backend failures - #3346
jpascucci-nv wants to merge 1 commit into
Conversation
closes NVIDIA#3273 Skip startup cache checks in Linux wheel builds so transient GHA cache failures do not prevent compilation. Configure cache I/O backoff after five errors within five seconds. Apply the same backoff to Linux sdist builds using local-disk sccache. Omit startup-check skipping there because the disk check performs no I/O. Validation: actionlint, YAML configuration checks, and git diff --check.
|
No actionable comments were generated in the recent review. Summary
Change: Bug fix · Severity of issue fixed: Medium Merge Risk:
 |
|
rwgk
left a comment
rwgk
left a comment
There was a problem hiding this comment.
codex GPT-6.1-Sol ultra:
No actionable findings in PR 3346. The settings make sense, and the exact pinned sccache release supports them.
SCCACHE_SKIP_CACHE_CHECK=trueskips the startup probe that caused #3273. Subsequent cache failures allow compilation to proceed. This is the key fix. Implementation- The two rate-limit settings provide reasonable backoff: five recorded errors within a rolling five-second window temporarily bypass cache I/O. Operations resume as errors age out. Implementation
- Omitting the startup setting for local-disk sdist builds is appropriate: their startup check performs no I/O. Implementation
One nonblocking caveat: this sccache version’s limiter counts read timeouts and write errors, but misses ordinary returned read errors. Its backoff coverage is therefore narrower than the PR description suggests; the startup fix still works. Source
I verified that the variables reach all three Linux wheel builds. Linux wheel and sdist CI passed; I did not simulate a cache outage.
Description
closes #3273
Skip startup cache checks in Linux wheel builds so transient GHA cache failures do not prevent compilation. Configure cache I/O backoff after five errors within five seconds.
Apply the same backoff to Linux sdist builds using local-disk sccache. Omit startup-check skipping there because the disk check performs no I/O.
I don't think there's a way to test to see if this actually fixes anything, since the condition requires network degradation.
Validation: actionlint, YAML configuration checks, and git diff --check.
Checklist