Skip to content

GH-3968: make EnvelopeHistory thread-safe on 5.x (backport of #3069/#3070) - #4585

Merged
jeremydmiller merged 1 commit into
5.0from
gh-3968-envelope-history-thread-safety-5.0
Sep 23, 2026
Merged

jeremydmiller merged 1 commit into
5.0from
gh-3968-envelope-history-thread-safety-5.0

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3968. Backport of #3069 / #3070 to the 5.x line. Targets 5.0, not main.

Doing this now because we are already committed to a 5.x release for #4583 (#3645 step one).

The race

EnvelopeHistory kept its EnvelopeRecords in a bare List<T> with no synchronisation, while records arrive concurrently. A resize racing a read threw out of IsComplete(), surfacing as an AggregateException from TrackedSession.AssertNoExceptionsWereThrown() with no hint of the real cause.

#3069 reported it via RecordCrossApplication with RabbitMQ. #3968 confirmed the same race on the purely local path with no external transport, reached through WolverineRuntime.MessageSucceeded — so it races on handler-completion threads too, not only transport-listener threads. Their trigger was MultipleHandlerBehavior.Separated: one ExecuteAndWaitAsync fanning out to many envelopes completing at once. Roughly 1 run in 7 of a ~1850-test suite; ≈3% over a longer history.

The reporter verified by reflection that no shipped 5.x carries the fix — 5.39.3 and 5.40.1 both lack the _lock field, and 5.40.1 published ~8 weeks after 6.8.0 got it. The only route for a 5.x user was a major-version upgrade to fix test-harness flakiness.

What changed

The locking topology matches 6.x member for member, verified mechanically rather than by eye:

member lock
RecordLocally, RecordCrossApplication held across the whole body
IsComplete, Has, MessageFor, Message, Records taken for the read
lastOf, both markLastCompleted overloads unlocked — only ever called from inside an already-locked writer

Records also now hands out a snapshot rather than the live list. On 5.x it was public IEnumerable<EnvelopeRecord> Records => _records;, which lets a caller enumerate while a recording thread adds — the same race one level removed.

Test-support code only (Wolverine.Tracking is not in a production path), no public API change, no behavioural risk since contention there is irrelevant.

The tests, and why the first one had to be strengthened

Three tests, driven hard and directly — a race that shows up in 3% of full-suite runs is no use as a regression test. All three fail on the unfixed class on every run, checked three times over, with the baseline's own build asserted clean first so it could not be a stale binary reporting false red.

One thing worth flagging to a reviewer, because it is the kind of test that looks fine and proves nothing. As first written, the reader test exercised IsComplete(), Has() and Records.Count() — and it passed on the unfixed class:

  • IsComplete() and Has() walk the list with an indexed for loop, which does not throw on concurrent modification
  • List<T> answers Count without enumerating

So all three readers stay quiet under concurrent mutation — they can read a torn value, but silently. Only LINQ over the list raises Collection was modified, so the test now drives Message and MessageFor, which enumerate. The indexed readers are kept for coverage but are not what makes it fail.

Verification

Full CoreTests on 5.0: 1695 green. Clean pinned Release build of wolverine.slnx -c Release -f net9.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj

…3070)

EnvelopeHistory kept its EnvelopeRecords in a bare List<T> with no synchronisation while records
arrive concurrently. A resize racing a read threw out of IsComplete(), surfacing as an
AggregateException from TrackedSession.AssertNoExceptionsWereThrown() with no hint of the real cause.

#3069 reported it through RecordCrossApplication with RabbitMQ. #3968 confirmed the same race
independently on the purely LOCAL path with no external transport, reached through
WolverineRuntime.MessageSucceeded -- so it races on handler-completion threads too, not only
transport-listener threads. Their trigger was MultipleHandlerBehavior.Separated: one
ExecuteAndWaitAsync fanning out to many envelopes completing at once. Reproduced roughly 1 run in 7 of
a ~1850-test suite, ~3% over a longer history.

The reporter verified by reflection that no shipped 5.x has the fix: 5.39.3 and 5.40.1 both lack the
_lock field, and 5.40.1 published ~8 weeks AFTER 6.8.0 carried it. So the only route for a 5.x user
was a major-version upgrade to fix test-harness flakiness.

The locking topology matches 6.x member for member -- verified mechanically, not by eye:

  - the two writers, RecordLocally and RecordCrossApplication, hold the lock across their whole body
  - IsComplete, Has, MessageFor, Message and Records take it for the read
  - the three private helpers (lastOf, both markLastCompleted overloads) stay unlocked, because they
    are only ever called from inside an already-locked writer

Records also now hands out a snapshot rather than the live list. On 5.x it was
`public IEnumerable<EnvelopeRecord> Records => _records;`, which lets a caller enumerate the list while
a recording thread is adding to it -- the same race one level removed.

This is test-support code (Wolverine.Tracking is not in a production path), no public API changes.

## On the tests

Three, driven hard and directly: a 3%-of-full-suite race is no use as a regression test. All three fail
on the unfixed class on EVERY run -- checked three times over, with the baseline's own build asserted
clean first so it could not be a stale binary.

Worth recording why the first one needed strengthening. As first written it exercised IsComplete(),
Has() and Records.Count(), and it PASSED unfixed: IsComplete and Has walk the list with an indexed for
loop, and List<T> answers Count without enumerating, so none of the three throw under concurrent
mutation -- they can read a torn value, but quietly. Only LINQ over the list raises "Collection was
modified", so the test now drives Message and MessageFor, which enumerate. The indexed readers are kept
for coverage but are not what makes the test fail.

Full CoreTests on 5.0: 1695 green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VDUrBeB4tTnKj4AExCS1nj
@jeremydmiller
jeremydmiller merged commit b7d362b into 5.0 Sep 23, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant