Skip to content

fix(rumqttc/v5): bound socket write/flush with a timeout - #1079

Open
florissmit10 wants to merge 1 commit into
bytebeamio:mainfrom
florissmit10:fix/v5-flush-timeout
Open

florissmit10 wants to merge 1 commit into
bytebeamio:mainfrom
florissmit10:fix/v5-flush-timeout

Conversation

@florissmit10

Copy link
Copy Markdown

Fixes #1078

Summary

In the v5 eventloop, every write/flush inside select() is now bounded by MqttOptions::connection_timeout. If the bound is hit, it returns a new ConnectionError::FlushTimeout, mirroring the v3.1.1 eventloop. poll() then cleans up the connection, so the next poll() reconnects instead of sitting until the kernel's TCP retransmit timeout (~15 min).

 select! {
   next_request =>
-    network.write(outgoing).await?; network.flush().await?;
+    write_and_flush(network, outgoing, network_timeout).await?;
   network.readb =>
-    network.flush().await?;
+    write_and_flush(network, None, network_timeout).await?;
   keepalive =>
-    network.write(PingReq).await?; network.flush().await?;
+    write_and_flush(network, PingReq, network_timeout).await?;
 }

Differences from v3.1.1:

  • The timeout covers write as well as flush. Framed::feed flushes once its buffer passes the backpressure boundary, so write alone can also block on a full socket.
  • The timeout comes from MqttOptions::connection_timeout, the value v5 already uses for connect. v5's eventloop has no separate network_options timeout.

Evidence

New test rumqttc/tests/v5_flush_timeout.rs: a raw TCP "broker" replies CONNACK and then never reads again. The client publishes 16 KB payloads in a loop (QoS 0 and QoS 1) with a 1 KB send buffer.

loop eventloop.poll() until Err   (guarded by a 30s timeout)
expect Err == ConnectionError::FlushTimeout
  • Before: poll() never returns, and the test fails on the guard: eventloop.poll() hung while the broker was not reading (30.01s)
  • After: 2 passed ... finished in 1.14s. The full cargo test --all-features suite passes.

Merge Danger

Door: two-way

Revert the commit to undo it. There is no persisted state or protocol change.

Blast Radius: v5-clients

  • Adds a variant to the public v5::ConnectionError, which is not #[non_exhaustive]. Downstream exhaustive matches will need an arm. It is listed in the CHANGELOG under Unreleased.
  • A link that is alive but slow and stalls a write for longer than connection_timeout (default 5s) now errors and reconnects instead of waiting. This is the same as v3.1.1.
  • Not covered: the ack writes inside Network::readb still have no timeout. This is pre-existing, and they can't hang poll() because they run inside select!, so keepalive still fires.

A peer that stops reading fills the TCP send buffer, parking the v5
eventloop inside write/flush forever, so the keepalive branch never runs
and no ConnectionError is produced until the kernel gives up (~15 min).

Route every write+flush in the v5 select loop through a helper that bounds
it by MqttOptions::connection_timeout and returns the new
ConnectionError::FlushTimeout, mirroring the v3.1.1 eventloop. poll() then
cleans up the connection so the next poll reconnects.

Signed off: Floris Smit <florissmit10@gmail.com>

Issue: bytebeamio#1078

Attribute: @florissmit10
@neiltalap

Copy link
Copy Markdown

Looks clean!

Fails fast approach, as I am seeing, instead of hanging, it throws right away. I used something similar with Concord, but in my case it is deterministic, and if it fails, it simply retries by refetching journals from other nodes on subsequent ticks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v5: socket flush/write has no timeout, client hangs silently when broker stops reading

2 participants