Repository navigation
GH-3968: make EnvelopeHistory thread-safe on 5.x (backport of #3069/#3070) - #4585
Merged
Merged
Conversation
…3070) EnvelopeHistory kept its EnvelopeRecords in a bare List<T> with no synchronisation while records arrive concurrently. A resize racing a read threw out of IsComplete(), surfacing as an AggregateException from TrackedSession.AssertNoExceptionsWereThrown() with no hint of the real cause. #3069 reported it through RecordCrossApplication with RabbitMQ. #3968 confirmed the same race independently on the purely LOCAL path with no external transport, reached through WolverineRuntime.MessageSucceeded -- so it races on handler-completion threads too, not only transport-listener threads. Their trigger was MultipleHandlerBehavior.Separated: one ExecuteAndWaitAsync fanning out to many envelopes completing at once. Reproduced roughly 1 run in 7 of a ~1850-test suite, ~3% over a longer history. The reporter verified by reflection that no shipped 5.x has the fix: 5.39.3 and 5.40.1 both lack the _lock field, and 5.40.1 published ~8 weeks AFTER 6.8.0 carried it. So the only route for a 5.x user was a major-version upgrade to fix test-harness flakiness. The locking topology matches 6.x member for member -- verified mechanically, not by eye: - the two writers, RecordLocally and RecordCrossApplication, hold the lock across their whole body - IsComplete, Has, MessageFor, Message and Records take it for the read - the three private helpers (lastOf, both markLastCompleted overloads) stay unlocked, because they are only ever called from inside an already-locked writer Records also now hands out a snapshot rather than the live list. On 5.x it was `public IEnumerable<EnvelopeRecord> Records => _records;`, which lets a caller enumerate the list while a recording thread is adding to it -- the same race one level removed. This is test-support code (Wolverine.Tracking is not in a production path), no public API changes. ## On the tests Three, driven hard and directly: a 3%-of-full-suite race is no use as a regression test. All three fail on the unfixed class on EVERY run -- checked three times over, with the baseline's own build asserted clean first so it could not be a stale binary. Worth recording why the first one needed strengthening. As first written it exercised IsComplete(), Has() and Records.Count(), and it PASSED unfixed: IsComplete and Has walk the list with an indexed for loop, and List<T> answers Count without enumerating, so none of the three throw under concurrent mutation -- they can read a torn value, but quietly. Only LINQ over the list raises "Collection was modified", so the test now drives Message and MessageFor, which enumerate. The indexed readers are kept for coverage but are not what makes the test fail. Full CoreTests on 5.0: 1695 green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3968. Backport of #3069 / #3070 to the 5.x line. Targets
5.0, notmain.Doing this now because we are already committed to a 5.x release for #4583 (#3645 step one).
The race
EnvelopeHistorykept itsEnvelopeRecords in a bareList<T>with no synchronisation, while records arrive concurrently. A resize racing a read threw out ofIsComplete(), surfacing as anAggregateExceptionfromTrackedSession.AssertNoExceptionsWereThrown()with no hint of the real cause.#3069 reported it via
RecordCrossApplicationwith RabbitMQ. #3968 confirmed the same race on the purely local path with no external transport, reached throughWolverineRuntime.MessageSucceeded— so it races on handler-completion threads too, not only transport-listener threads. Their trigger wasMultipleHandlerBehavior.Separated: oneExecuteAndWaitAsyncfanning out to many envelopes completing at once. Roughly 1 run in 7 of a ~1850-test suite; ≈3% over a longer history.The reporter verified by reflection that no shipped 5.x carries the fix — 5.39.3 and 5.40.1 both lack the
_lockfield, and 5.40.1 published ~8 weeks after 6.8.0 got it. The only route for a 5.x user was a major-version upgrade to fix test-harness flakiness.What changed
The locking topology matches 6.x member for member, verified mechanically rather than by eye:
RecordLocally,RecordCrossApplicationIsComplete,Has,MessageFor,Message,RecordslastOf, bothmarkLastCompletedoverloadsRecordsalso now hands out a snapshot rather than the live list. On 5.x it waspublic IEnumerable<EnvelopeRecord> Records => _records;, which lets a caller enumerate while a recording thread adds — the same race one level removed.Test-support code only (
Wolverine.Trackingis not in a production path), no public API change, no behavioural risk since contention there is irrelevant.The tests, and why the first one had to be strengthened
Three tests, driven hard and directly — a race that shows up in 3% of full-suite runs is no use as a regression test. All three fail on the unfixed class on every run, checked three times over, with the baseline's own build asserted clean first so it could not be a stale binary reporting false red.
One thing worth flagging to a reviewer, because it is the kind of test that looks fine and proves nothing. As first written, the reader test exercised
IsComplete(),Has()andRecords.Count()— and it passed on the unfixed class:IsComplete()andHas()walk the list with an indexedforloop, which does not throw on concurrent modificationList<T>answersCountwithout enumeratingSo all three readers stay quiet under concurrent mutation — they can read a torn value, but silently. Only LINQ over the list raises
Collection was modified, so the test now drivesMessageandMessageFor, which enumerate. The indexed readers are kept for coverage but are not what makes it fail.Verification
Full CoreTests on
5.0: 1695 green. Clean pinned Release build ofwolverine.slnx -c Release -f net9.0.🤖 Generated with Claude Code
https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj