Skip to content

GH-5247 DurabilityAgent.StopAsync cancels and waits for its in-flight command - #5248

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-5247-durability-agent-stop-join
Oct 28, 2026
Merged

jeremydmiller merged 1 commit into
mainfrom
gh-5247-durability-agent-stop-join

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Fixes #5247.

Cause

DurabilityAgent built its command block with a one-argument async lambda (async batch => ...). That binds to Block(Action<T>), so every durability command ran as async void: the block considered the item done the moment the first await yielded. StopAsync() only called Complete(), but even awaiting the block couldn't have helped, and no cancellation token ever reached the command. A DeleteExpiredHandledEnvelopesCommand in flight at shutdown outlived the host and failed against the disposed NpgsqlDataSource, exactly as the issue's repro shows.

Change

  • The block uses the (item, token) overload, so it owns the running command, and passes an agent-owned cancellation token into executor.InvokeAsync().
  • StopAsync() disposes the timers first (no more producers), completes the block, cancels the token, then awaits the block, bounded by the caller's stop token. The CTS isn't linked or disposed, so StopAsync() is still safe to call twice (the first draft disposed it and broke 49 Postgres agent tests that stop agents more than once).
  • AgentCommandHandler treats an OperationCanceledException raised after its token was cancelled as a stop, not a failure, the same way it already handles TimeoutException. That keeps the executor from logging Invocation of ... failed! for a cancelled shutdown. Genuine storage errors are still logged.

Tests

PersistenceTests/Durability/durability_agent_stops_in_flight_commands_5247.cs needs no database. The real five-second cleanup timer dispatches the real command, and a gated DbDataSource holds it at OpenConnectionAsync():

  • agent removal, where the command honours cancellation: StopAsync returns with the command finished and nothing logged at Error
  • agent removal, where the command is slow to cancel: StopAsync doesn't return until the command does
  • host shutdown: the command is finished when host.StopAsync() returns, and nothing opens a connection afterwards

All 3 fail on main and pass with the fix.

Local verification (net9.0, Release):

  • dotnet build wolverine.slnx -c Release -f net9.0: clean
  • CoreTests FullyQualifiedName~Agent: 643 passed, 6 skipped
  • PostgresqlTests ~urability|~Agent|~LeaderElection: 84/84

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U

… command

The agent built its command block with a one-argument async lambda, which binds to
Block(Action<T>): every durability command ran as async void, so StopAsync() could not
wait for one already running even in principle, and no cancellation token reached it.
A DeleteExpiredHandledEnvelopesCommand in flight at shutdown then outlived the host
and failed against the disposed NpgsqlDataSource.

- The block now uses the (item, token) overload, so it owns the running command, and
  passes an agent-owned cancellation token into executor.InvokeAsync().
- StopAsync() disposes the timers first (no more producers), completes the block,
  cancels the token and awaits the block, bounded by the caller's stop token. It stays
  safe to call more than once.
- AgentCommandHandler treats an OperationCanceledException raised after its token was
  cancelled as a stop rather than a failure, like the TimeoutException case beside it,
  so the executor no longer logs "Invocation of ... failed!" for it. Genuine storage
  errors are still logged.

Regression tests hold the real cleanup command at OpenConnectionAsync() and cover
agent removal (cancelled, and slow to cancel) and host shutdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
@jeremydmiller
jeremydmiller merged commit 40e230a into main Oct 28, 2026
47 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RDBMS DurabilityAgent.StopAsync does not join in-flight cleanup commands before data-source disposal

1 participant