Repository navigation
fix(rumqttc/v5): bound socket write/flush with a timeout - #1079
Open
florissmit10 wants to merge 1 commit into
Open
florissmit10 wants to merge 1 commit into
florissmit10 wants to merge 1 commit into
Conversation
A peer that stops reading fills the TCP send buffer, parking the v5 eventloop inside write/flush forever, so the keepalive branch never runs and no ConnectionError is produced until the kernel gives up (~15 min). Route every write+flush in the v5 select loop through a helper that bounds it by MqttOptions::connection_timeout and returns the new ConnectionError::FlushTimeout, mirroring the v3.1.1 eventloop. poll() then cleans up the connection so the next poll reconnects. Signed off: Floris Smit <florissmit10@gmail.com> Issue: bytebeamio#1078 Attribute: @florissmit10
|
Looks clean! Fails fast approach, as I am seeing, instead of hanging, it throws right away. I used something similar with Concord, but in my case it is deterministic, and if it fails, it simply retries by refetching journals from other nodes on subsequent ticks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1078
Summary
In the v5 eventloop, every write/flush inside
select()is now bounded byMqttOptions::connection_timeout. If the bound is hit, it returns a newConnectionError::FlushTimeout, mirroring the v3.1.1 eventloop.poll()then cleans up the connection, so the nextpoll()reconnects instead of sitting until the kernel's TCP retransmit timeout (~15 min).select! { next_request => - network.write(outgoing).await?; network.flush().await?; + write_and_flush(network, outgoing, network_timeout).await?; network.readb => - network.flush().await?; + write_and_flush(network, None, network_timeout).await?; keepalive => - network.write(PingReq).await?; network.flush().await?; + write_and_flush(network, PingReq, network_timeout).await?; }Differences from v3.1.1:
writeas well asflush.Framed::feedflushes once its buffer passes the backpressure boundary, sowritealone can also block on a full socket.MqttOptions::connection_timeout, the value v5 already uses for connect. v5's eventloop has no separatenetwork_optionstimeout.Evidence
New test
rumqttc/tests/v5_flush_timeout.rs: a raw TCP "broker" replies CONNACK and then never reads again. The client publishes 16 KB payloads in a loop (QoS 0 and QoS 1) with a 1 KB send buffer.poll()never returns, and the test fails on the guard:eventloop.poll() hung while the broker was not reading(30.01s)2 passed ... finished in 1.14s. The fullcargo test --all-featuressuite passes.Merge Danger
Door: two-way
Revert the commit to undo it. There is no persisted state or protocol change.
Blast Radius: v5-clients
v5::ConnectionError, which is not#[non_exhaustive]. Downstream exhaustivematches will need an arm. It is listed in the CHANGELOG under Unreleased.connection_timeout(default 5s) now errors and reconnects instead of waiting. This is the same as v3.1.1.Network::readbstill have no timeout. This is pre-existing, and they can't hangpoll()because they run insideselect!, so keepalive still fires.