Skip to content

GH-5236 Kafka DeferAsync no longer re-enters the receiver inline - #5240

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-5236
Oct 27, 2026
Merged

jeremydmiller merged 1 commit into
mainfrom
gh-5236

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #5236

Problem

KafkaListener.DeferAsync() (and KafkaTopicGroupListener's) called _receiver.ReceivedAsync() directly. When the durable inbox write keeps failing, DurableReceiver.receiveOneAsync settles the delivery by deferring it, from inside its own receive, so the cycle in the reported stack trace never unwinds:

DurableReceiver.receiveOneAsync → KafkaListener.DeferAsync → DurableReceiver.ReceivedAsync → RetryBlock.PostAsync → DurableReceiver.receiveOneAsync → …

The process dies of a stack overflow (exit 139). A latched receiver loops the same way through its _deferBlock. The loop also kept the receiver busy, which got in the way of the drain that PauseForInboxRecoveryAsync runs.

Fix

New KafkaDeferredRedelivery, used by both Kafka listeners:

  • DeferAsync returns at once. The envelope goes back to the receiver on its own task, after a short pause (KafkaDeferredRedelivery.Delay, 1 second, internal), so the stack can't grow and a receiver that keeps deferring can't spin.
  • It only redelivers while the listener is running. Once the listener stops, including the pause for inbox recovery, the envelope is left alone. Its offset was never completed, so the KafkaOffsetCommitter watermark keeps the committed position below it and the restarted consumer reads it again. Offsets never move past a message that wasn't persisted.

Behavior note: Kafka has no native requeue, so Requeue error policies go through this same path and now redeliver after about 1 second instead of immediately.

Tests

  • deferred_redelivery_does_not_recurse_5236 (broker-free): a receiver that defers every envelope, as the failing inbox does. The stack depth stays flat across 20 redeliveries, DeferAsync doesn't wait for the redelivery, and nothing is redelivered after the listener stops.
  • Against a local broker + Postgres: duplicate_message_handling_with_postgres_inbox, durable_topic_group_batching_4026, disable_requeueing, commit_strategy_end_to_end, compliance suites: 70/70.
  • Pinned wolverine.slnx -c Release -f net9.0 build is clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U

KafkaListener and KafkaTopicGroupListener implemented DeferAsync() by calling
_receiver.ReceivedAsync() directly. A durable receiver whose inbox write keeps failing
defers from inside its own receive, so receive -> fail -> defer -> receive recursed until
the process died of a stack overflow (a latched receiver looped the same way through its
defer block).

A deferred envelope now goes back to the receiver on its own task after a short pause
(KafkaDeferredRedelivery.Delay, 1 second), and only while the listener is still running.
Once the listener stops, including a pause for inbox recovery, the envelope is left alone:
its offset was never completed, so the committed position can't move past it and the
restarted consumer reads it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DCggXgxnG3EBXoR9rrq37U
@jeremydmiller
jeremydmiller merged commit 8e70f0a into main Oct 27, 2026
0 of 46 checks passed
@jeremydmiller jeremydmiller mentioned this pull request Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stack overflow in Kafka durable inbox when SQL Server is unavailable

1 participant