ci: shard 1 carries more, so give it more - #83
Merged
Conversation
The master run after the merge reported failure with seven shards green and shard 1 `cancelled`. Every step in shard 1 succeeded, including the tests and the cache write — it was killed by the 45-minute timeout eleven minutes after the last useful work, and the aggregate turned that into a red run. Shard 1 is not one of eight equal jobs. It runs the doctests, and it is the single writer for the shared build cache, so on a cache miss it pays an upload the other seven skip. Measured on that run, where the Cargo.lock change from the dependency bumps guaranteed a miss: 4 min of setup, 33 min of tests on a cold cache, 20 s of doctests, and 16 min saving the cache. Fifty-three against a budget of forty-five. The budget was mine, and the comment above it said a timeout should catch a hang rather than a slow week. It did the second, because I sized it from the average shard rather than from the one it also applies to. Not a blanket raise: the other seven finish inside 45 and keeping their ceiling low is what makes a genuine hang visible. Only shard 1 gets the room its extra work needs. The cache is warm now, so the next run skips the save either way — which is precisely why this would have gone unnoticed until the next dependency bump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The master run after #82 reported failure with seven shards green and shard 1
cancelled. Every step in shard 1 succeeded — tests included, and the cache was written — but the job was killed by the 45-minute timeout eleven minutes after the last useful work, and the aggregate turned that into a red run.Why shard 1 is different
It is not one of eight equal jobs. It runs the doctests, and it is the single writer for the shared build cache, so on a cache miss it pays an upload the other seven skip.
Measured on that run, where the
Cargo.lockchange from the dependency bumps guaranteed a miss:targetcacheAgainst a budget of 45.
The fix
timeout-minutes: ${{ matrix.shard == 1 && 75 || 45 }}.Not a blanket raise. The other seven finish inside 45, and keeping their ceiling low is what makes a genuine hang visible. Only the shard that does the extra work gets the extra room.
Note
The cache is warm now, so the next run skips the save either way — which is precisely why this would have gone unnoticed until the next dependency bump.