test(guestd): widen TestBrokenExecStreamReapsChild reap ceiling to 30s - #732
Merged
Merged
Conversation
|
Compass engineering docs preview: https://compass-runner-rig-2980-flak.compass-eng-docs.pages.dev Deployed from |
The reap-wait loop gave the ExecStream cancel→SIGKILL→reap path only 10s of wall-clock to remove the exec_id from the table. That races HTTP/2 RST_STREAM propagation over the loopback h2c transport plus goroutine scheduling: the reap fires only once the receive-loop observes the client cancel as a stream error. On a saturated CI runner (deploy+e2e+microvm+ pgtest+4 moon legs concurrent) that delivery exceeded 10s and the test false-failed with 'child still present after stream break' (moon (go) leg, PR #720 CI, test wall 15.01s). The reap logic is correct and always completes; only the test ceiling was too tight. Widen to 30s — a genuine never-reap hang still fails at the ceiling, while extreme load no longer trips a false positive. Local reproduction of the original: 80/80 PASS (10x plain, 30x -race, 25x GOMAXPROCS=1, 15x under full 32-core saturation). Test-only, no production delta. Flaky test = real bug, no retries. Refs RIG-2980 Co-authored-by: Matt Wilkinson <matt@rigel.build>
rigel-mintaka
force-pushed
the
compass-runner/rig-2980-flaky-reap-ceiling
branch
from
August 30, 2026 04:47
c2bca26 to
5b30327
Compare
mattwilkinsonn
approved these changes
Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reap-wait loop gave the ExecStream cancel→SIGKILL→reap path only 10s
of wall-clock to remove the exec_id from the table. That races HTTP/2
RST_STREAM propagation over the loopback h2c transport plus goroutine
scheduling: the reap fires only once the receive-loop observes the client
cancel as a stream error. On a saturated CI runner (deploy+e2e+microvm+
pgtest+4 moon legs concurrent) that delivery exceeded 10s and the test
false-failed with 'child still present after stream break' (moon (go) leg,
PR #720 CI, test wall 15.01s).
The reap logic is correct and always completes; only the test ceiling was
too tight. Widen to 30s — a genuine never-reap hang still fails at the
ceiling, while extreme load no longer trips a false positive. Local
reproduction of the original: 80/80 PASS (10x plain, 30x -race, 25x
GOMAXPROCS=1, 15x under full 32-core saturation). Test-only, no production
delta. Flaky test = real bug, no retries.
Refs RIG-2980
Co-authored-by: Matt Wilkinson matt@rigel.build