You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
client: resolve failed writes at the next login; drop the connection on a failed send - #6
UBD-side half of the resync design (HomeBlocks #198, docs/craft/resync.md): failed writes are resolved at the next login instead of during the session, the in-memory reference cluster's login does that resolution, and the async transport can no longer leave a half-sent frame on a live connection.
1. Failed writes resolved during the session race in-flight acks
Problem: On a sub-quorum write the client fired a resolution round (request_resolution) and retired every unresolved slot up to the failed one off the leader's reply, including slots whose writes were still in flight. A slot could be declared Empty and then reach its quorum ack, so the application was told a voided write succeeded. Making that safe needed an explicit failed-slot list and a "verdict epoch" in every request. Also, in production the application (XFS) goes offline on a write error and the volume detaches, so keeping it alive to finish the write gains nothing.
Diagram:
sequenceDiagram
participant A as Application
participant U as UBD
participant L as Leader
participant R as Replicas
U->>R: write 100 (fails: only 1 of 3)
U->>R: write 99 (still in flight)
U-->>A: write 100: error
U->>L: RESOLVE up to 100
L->>R: nobody holds 99 yet → 99 is Empty
R-->>U: acks for 99 arrive → quorum
U-->>A: write 99: success (but it was voided)
Loading
Solution: The client sends no resolution request. It reports the error, keeps the slot pinning its frontier, and keeps serving overlapping reads as of just before the slot (reads never block). The next login decides the slot under the login fence: it is filled everywhere if a live member holds it, or, if nobody holds it, the login's starting point is below it and the dLSN is reused. The client's resolution driver and the tracker/router state only it used are removed. The request_resolution verb and wire resolve op stay for compatibility (HomeBlocks references them).
sequenceDiagram
participant A as Application
participant U as UBD
participant R as Replicas
participant L as Leader
U->>R: write 101 (only Replica C stores it)
U-->>A: error (XFS goes offline, volume detaches)
Note over U,R: no RESOLVE; slot 101 pins the commit point, reads see the old data
Note over L,R: next attach: login under the fence
L->>R: C holds 101 → fill everywhere (or nobody holds it → dLSN reused)
Loading
2. The reference cluster's login did not resolve failed writes
Problem:MemTransport::run_login only applied up to the starting point. It never filled a slot a member lacked or voided one nobody held, so "resolved at the next login" didn't hold in the model, and a lagging member stayed behind after login.
Diagram:
sequenceDiagram
participant L as Leader (holds 1)
participant B as Replica B
Note over L,B: dLSN 1 failed: only the leader holds it
L->>B: login: SyncRSCommitLSN(1)
B->>B: before: apply stalls at missing 1 (nothing fetched)
B->>B: after: login filled 1 from the leader → commit reaches 1
Loading
Solution: Login runs the same fill-or-Empty step the resolution round already had (factored into MemTransport::resolve_slots_upto), for every slot up to its starting point.
3. A half-sent frame stays on a live connection (D17)
Problem:ring_send_all can stop partway through a frame: a short send that errors, or no free submission slot. round_trip then released the send slot and returned net_error::send but kept the connection. That error maps to REPLICA_DOWN, "never delivered". The next message's bytes then followed the fragment, so the server could parse a write whose payload mixed the two messages, and data digests are off by default.
Diagram:
sequenceDiagram
participant U as UBD (craft_async_conn)
participant S as Replica
U->>S: write 5: first half of the frame, then the send fails
U->>U: before: report REPLICA_DOWN, keep the socket
U->>S: write 6 frame (follows the fragment on the same stream)
S->>S: parses write 5 with write 6's bytes as payload → corrupt slot
Loading
Solution: On any send failure, round_trip drops the socket before handing the send slot on. The pump sees the close and fails every other pending op, a send already queued behind it fails on the dead fd, and the next op reconnects. That's the same teardown the existing reconnect path uses.
Testing
Built and tested on Linux (Debug + coverage): all 15 test suites pass.
Coverage of the changed files: client.cpp 87%, dlsn_tracker.cpp 98%, mem/cluster.cpp 97%, read_route_map.hpp 97%, async_conn.cpp 96%. For async_conn.cpp gcov reports 0 hits on every line of round_trip, including the normal path that clearly runs, so per-line coverage there isn't reliable. The new send-failure branch has no dedicated test: forcing a short send deterministically needs a fault-injection hook the transport doesn't have yet.
New and updated tests:
in-memory:
a failed write is not resolved during the session (frontier pinned, old data served, no copy made);
the next login fills it when a live member holds it;
the next login drops it when nobody does, and the new session reuses the dLSN.
TCP: the failed write stays unresolved during the session.
ReLoginMustNotServeStaleFollower: keeps member 2 down during the login, since a live member is now healed by the login.
docs/client-internals.md is updated; docs/resync.md points to the HomeBlocks design.
Not in this PR: data digests on by default for peer traffic (follow-up), and removing the now-unused request_resolution verb / resolve op (needs a coordinated HomeBlocks change).
…on a failed send
- A failed (sub-quorum) write no longer fires the client-requested resolution
round. The client reports the error, pins its frontier and keeps serving
overlapping reads as of just before the slot; the next login resolves it
(HomeBlocks docs/craft/resync.md §4.4-§4.5: mid-session verdicts raced
in-flight acks). Removes the client's resolution driver and the tracker /
router state only it used. The request_resolution verb and wire resolve op
stay for compatibility.
- The in-memory reference cluster's login now resolves every unresolved slot
up to its starting point (fill from a live holder, or Empty), sharing the
step its resolution round already had, so the model matches the design.
- craft_async_conn::round_trip drops the socket on any send failure before
handing the send slot on. A partially sent frame could otherwise be followed
by the next message's bytes while the caller reports REPLICA_DOWN ("never
delivered").
- Tests: failed write unresolved during the session (in-memory and TCP); the
next login fills it when a live member holds it, and drops it when nobody
does; ReLoginMustNotServeStaleFollower keeps member 2 down during the login.
- docs: client-internals.md updated; docs/resync.md points to the HomeBlocks
design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR moves failed-write resolution from the active client session to the next login, aligning the client and in-memory reference cluster with the HomeBlocks resync design. It also closes connections after send failures to prevent subsequent messages from extending a partial frame.
Changes:
Remove client-driven resolution rounds and their tracker/router state.
Resolve missing slots during reference-cluster login, with updated in-memory and TCP tests.
Drop failed-send connections before releasing the send slot and document the revised behavior.
File
Description
test/test_client.cpp
Test session-pinned failures and next-login recovery.
test/test_client_tcp.cpp
Verify failed writes remain unresolved during TCP sessions.
src/read_route_map.hpp
Remove resolution-request tracking.
src/net/async_conn.cpp
Close the connection before releasing a failed send.
src/mem/cluster.hpp
Declare the shared slot-resolution helper.
src/mem/cluster.cpp
Share slot resolution between login and explicit resolution.
src/dlsn_tracker.hpp
Remove resolution watermarks and bulk retirement APIs.
src/dlsn_tracker.cpp
Remove resolution-watermark implementation.
src/client.cpp
Stop issuing resolution requests after failed writes.
src/client_impl.hpp
Remove the resolution-driver declaration.
docs/resync.md
Link the HomeBlocks design and summarize client responsibilities.
docs/client-internals.md
Document next-login recovery and retained compatibility verbs.
…tion-guarded drop on send failure
- MemTransport::run_login fences every live replica (switches it to the new
term) before polling and resolving, so an old-session write can no longer
reach quorum after the absence scan and then be voided by it.
- craft_async_conn tracks a connection generation; a failed send drops the
socket only if it is still the connection the frame went out on, so a stale
queued send's late failure cannot close a newer socket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The send-error teardown and generation guard lack deterministic regression tests for partial-send failures with queued operations and stale failures after reconnect.
The reason will be displayed to describe this comment to others. Learn more.
Added in 6a68754. craft_async_conn gets a deterministic test seam (set_send_fault_for_test: cap a send to N bytes, or fail it; bump_conn_gen_for_test). Two on-ring tests against a scripted replica in test_auth_handshake.cpp:
OnRingSendFailureMidFrameDropsTheConnection: the first frame sends 10 bytes, then its send fails. The op fails with send, the op queued behind it fails too, the connection is no longer ready, and the next op reconnects (2 connections accepted) and carries traffic.
OnRingStaleSendFailureLeavesTheNewConnection: the connection generation moves on before the send fails. The op fails, but the socket stays ready, the next op succeeds, and no reconnect happens (1 connection accepted).
Deterministic send-fault seam on craft_async_conn (set_send_fault_for_test:
cap a send to N bytes or fail it; bump_conn_gen_for_test). Two on-ring tests
against a scripted replica:
- a frame that fails after a partial send drops the connection; the op
queued behind it fails, and the next op reconnects and carries traffic;
- a stale send failure (the connection generation moved on) leaves the
current socket open and usable, with no reconnect.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A critical send-slot ownership race lets an old completion release a replacement frame’s slot and interleave frames, while the tests fail to verify late old-session writes and fence-before-poll ordering.
Make send-slot release generation-aware across reconnects
src/net/async_conn.cpp:248
The generation guard protects the replacement socket, but the unconditional release_send() can still release its send slot. If fail_all() resets sending_ and reconnect starts before an old send's error completion is processed, that late completion can release a slot owned by a new frame, allowing frames to interleave. Drain old sends before reconnecting, or make send-slot ownership and release generation-aware across teardown and both completion paths.
Test old-session writes between the absence scan and Empty installation
src/mem/cluster.cpp:396
The new tests do not exercise an old-session write arriving during the login poll/resolution scan; they would still pass if this fence were moved after resolution. The 10-second straggler test checks its data before that delivery runs. Please add a deterministic interleaving test that delivers an old-term write after the absence scan but before Empty installation and verifies rejection, so the fence-before-poll ordering that prevents acknowledged data loss is regression-protected.
Verify delayed old writes cannot overwrite a reused slot
test/test_client.cpp:308
The new reuse test reads the replacement value before the old write's 10-second delivery, with no assertion after that delivery (remaining callbacks run during MemTransport::shutdown). It would therefore still pass if the late old-session write overwrote the reused slot. Please make the delayed delivery observable, wait for it after writing the replacement, and then verify that the replacement remains intact.
…sion writes
- craft_async_conn: the send slot is owned explicitly. send_awaitable yields
true only to an op that now holds the slot, with the slot's epoch.
fail_all() frees the slot by force and advances the epoch, so the old
owner's late release_send(epoch) is a no-op instead of releasing a slot a
newer frame owns (a third frame could then interleave with it). Ops
fail_all wakes from the queue get no slot and return `closed` without
sending.
- Test OnRingStaleSendCannotReleaseANewerFramesSlot reproduces the race
deterministically (fails with the epoch check removed).
- Model: replica_stats.stale_term_writes counts writes rejected for another
session's term; MemTransport::set_login_poll_hook_for_test runs between the
login poll and resolution.
- N3_OldSessionWriteBetweenPollAndResolutionIsRejected: an old-term write
delivered between the poll and resolution is rejected on every member and
never lands.
- N3_NextLoginDropsAFailedWriteNobodyHolds now waits for the delayed
old-session write to be delivered (and rejected) and checks the reused slot
afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re Copilot's three "previously missed" findings on 6a68754, all addressed in c411698:
Send-slot ownership across teardown (async_conn.cpp). Confirmed and fixed: this was a real race, not only a missing test. The send slot is now owned explicitly:
send_awaitable yields true only to an op that holds the slot, and records the slot's epoch.
fail_all() frees the slot and advances the epoch, so the old owner's late release_send(epoch) does nothing.
Ops that fail_all wakes up from the queue get no slot and return closed without sending.
OnRingStaleSendCannotReleaseANewerFramesSlot reproduces the race deterministically, with frame A mid-send, a teardown plus reconnect, B taking the slot, and A failing late. It checks that C waits for B. With the epoch check removed, that test fails.
Old-session write between the poll and Empty installation (cluster.cpp). Added set_login_poll_hook_for_test (it runs after the poll, before resolution) and a stale_term_writes counter in the model replica's stats. N3_OldSessionWriteBetweenPollAndResolutionIsRejected delivers an old-term write to every member in that window: all 3 reject it with STALE_TERM and the slot never lands. The test would fail if the fence moved after resolution.
Delayed old write vs the reused slot (test_client.cpp). N3_NextLoginDropsAFailedWriteNobodyHolds now uses a 1 s delay, waits until replica 0 reports the delayed write as delivered and rejected (stale_term_writes == 1), and then re-reads the reused slot and checks the journals.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
An unsent queued frame is reported as net_error::closed rather than net_error::send, incorrectly making delivery uncertain and leaving the client slot pinned until another login when all write legs are cancelled.
An op woken from the send queue by fail_all() returns net_error::send (never
delivered), not closed (delivery uncertain), so a write whose legs all end
this way can be retired instead of pinning its slot until the next login.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UBD-side half of the resync design (HomeBlocks #198,
docs/craft/resync.md): failed writes are resolved at the next login instead of during the session, the in-memory reference cluster's login does that resolution, and the async transport can no longer leave a half-sent frame on a live connection.1. Failed writes resolved during the session race in-flight acks
Problem: On a sub-quorum write the client fired a resolution round (
request_resolution) and retired every unresolved slot up to the failed one off the leader's reply, including slots whose writes were still in flight. A slot could be declared Empty and then reach its quorum ack, so the application was told a voided write succeeded. Making that safe needed an explicit failed-slot list and a "verdict epoch" in every request. Also, in production the application (XFS) goes offline on a write error and the volume detaches, so keeping it alive to finish the write gains nothing.Diagram:
sequenceDiagram participant A as Application participant U as UBD participant L as Leader participant R as Replicas U->>R: write 100 (fails: only 1 of 3) U->>R: write 99 (still in flight) U-->>A: write 100: error U->>L: RESOLVE up to 100 L->>R: nobody holds 99 yet → 99 is Empty R-->>U: acks for 99 arrive → quorum U-->>A: write 99: success (but it was voided)Solution: The client sends no resolution request. It reports the error, keeps the slot pinning its frontier, and keeps serving overlapping reads as of just before the slot (reads never block). The next login decides the slot under the login fence: it is filled everywhere if a live member holds it, or, if nobody holds it, the login's starting point is below it and the dLSN is reused. The client's resolution driver and the tracker/router state only it used are removed. The
request_resolutionverb and wireresolveop stay for compatibility (HomeBlocks references them).sequenceDiagram participant A as Application participant U as UBD participant R as Replicas participant L as Leader U->>R: write 101 (only Replica C stores it) U-->>A: error (XFS goes offline, volume detaches) Note over U,R: no RESOLVE; slot 101 pins the commit point, reads see the old data Note over L,R: next attach: login under the fence L->>R: C holds 101 → fill everywhere (or nobody holds it → dLSN reused)2. The reference cluster's login did not resolve failed writes
Problem:
MemTransport::run_loginonly applied up to the starting point. It never filled a slot a member lacked or voided one nobody held, so "resolved at the next login" didn't hold in the model, and a lagging member stayed behind after login.Diagram:
sequenceDiagram participant L as Leader (holds 1) participant B as Replica B Note over L,B: dLSN 1 failed: only the leader holds it L->>B: login: SyncRSCommitLSN(1) B->>B: before: apply stalls at missing 1 (nothing fetched) B->>B: after: login filled 1 from the leader → commit reaches 1Solution: Login runs the same fill-or-Empty step the resolution round already had (factored into
MemTransport::resolve_slots_upto), for every slot up to its starting point.3. A half-sent frame stays on a live connection (D17)
Problem:
ring_send_allcan stop partway through a frame: a short send that errors, or no free submission slot.round_tripthen released the send slot and returnednet_error::sendbut kept the connection. That error maps toREPLICA_DOWN, "never delivered". The next message's bytes then followed the fragment, so the server could parse a write whose payload mixed the two messages, and data digests are off by default.Diagram:
sequenceDiagram participant U as UBD (craft_async_conn) participant S as Replica U->>S: write 5: first half of the frame, then the send fails U->>U: before: report REPLICA_DOWN, keep the socket U->>S: write 6 frame (follows the fragment on the same stream) S->>S: parses write 5 with write 6's bytes as payload → corrupt slotSolution: On any send failure,
round_tripdrops the socket before handing the send slot on. The pump sees the close and fails every other pending op, a send already queued behind it fails on the dead fd, and the next op reconnects. That's the same teardown the existing reconnect path uses.Testing
client.cpp87%,dlsn_tracker.cpp98%,mem/cluster.cpp97%,read_route_map.hpp97%,async_conn.cpp96%. Forasync_conn.cppgcov reports 0 hits on every line ofround_trip, including the normal path that clearly runs, so per-line coverage there isn't reliable. The new send-failure branch has no dedicated test: forcing a short send deterministically needs a fault-injection hook the transport doesn't have yet.ReLoginMustNotServeStaleFollower: keeps member 2 down during the login, since a live member is now healed by the login.docs/client-internals.mdis updated;docs/resync.mdpoints to the HomeBlocks design.Not in this PR: data digests on by default for peer traffic (follow-up), and removing the now-unused
request_resolutionverb /resolveop (needs a coordinated HomeBlocks change).🤖 Generated with Claude Code