Repository navigation
GH-5247 DurabilityAgent.StopAsync cancels and waits for its in-flight command - #5248
Merged
Merged
Conversation
… command The agent built its command block with a one-argument async lambda, which binds to Block(Action<T>): every durability command ran as async void, so StopAsync() could not wait for one already running even in principle, and no cancellation token reached it. A DeleteExpiredHandledEnvelopesCommand in flight at shutdown then outlived the host and failed against the disposed NpgsqlDataSource. - The block now uses the (item, token) overload, so it owns the running command, and passes an agent-owned cancellation token into executor.InvokeAsync(). - StopAsync() disposes the timers first (no more producers), completes the block, cancels the token and awaits the block, bounded by the caller's stop token. It stays safe to call more than once. - AgentCommandHandler treats an OperationCanceledException raised after its token was cancelled as a stop rather than a failure, like the TimeoutException case beside it, so the executor no longer logs "Invocation of ... failed!" for it. Genuine storage errors are still logged. Regression tests hold the real cleanup command at OpenConnectionAsync() and cover agent removal (cancelled, and slow to cancel) and host shutdown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5247.
Cause
DurabilityAgentbuilt its command block with a one-argument async lambda (async batch => ...). That binds toBlock(Action<T>), so every durability command ran as async void: the block considered the item done the moment the firstawaityielded.StopAsync()only calledComplete(), but even awaiting the block couldn't have helped, and no cancellation token ever reached the command. ADeleteExpiredHandledEnvelopesCommandin flight at shutdown outlived the host and failed against the disposedNpgsqlDataSource, exactly as the issue's repro shows.Change
(item, token)overload, so it owns the running command, and passes an agent-owned cancellation token intoexecutor.InvokeAsync().StopAsync()disposes the timers first (no more producers), completes the block, cancels the token, then awaits the block, bounded by the caller's stop token. The CTS isn't linked or disposed, soStopAsync()is still safe to call twice (the first draft disposed it and broke 49 Postgres agent tests that stop agents more than once).AgentCommandHandlertreats anOperationCanceledExceptionraised after its token was cancelled as a stop, not a failure, the same way it already handlesTimeoutException. That keeps the executor from loggingInvocation of ... failed!for a cancelled shutdown. Genuine storage errors are still logged.Tests
PersistenceTests/Durability/durability_agent_stops_in_flight_commands_5247.csneeds no database. The real five-second cleanup timer dispatches the real command, and a gatedDbDataSourceholds it atOpenConnectionAsync():StopAsyncreturns with the command finished and nothing logged at ErrorStopAsyncdoesn't return until the command doeshost.StopAsync()returns, and nothing opens a connection afterwardsAll 3 fail on
mainand pass with the fix.Local verification (net9.0, Release):
dotnet build wolverine.slnx -c Release -f net9.0: cleanFullyQualifiedName~Agent: 643 passed, 6 skipped~urability|~Agent|~LeaderElection: 84/84🤖 Generated with Claude Code
https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U