Skip to content

client: resolve failed writes at the next login; drop the connection on a failed send - #6

Open
Ilyin-Ilya wants to merge 5 commits into
dev/v0.xfrom
resync/client-failed-writes
Open

Ilyin-Ilya wants to merge 5 commits into
dev/v0.xfrom
resync/client-failed-writes

Conversation

@Ilyin-Ilya

Copy link
Copy Markdown
Collaborator

UBD-side half of the resync design (HomeBlocks #198, docs/craft/resync.md): failed writes are resolved at the next login instead of during the session, the in-memory reference cluster's login does that resolution, and the async transport can no longer leave a half-sent frame on a live connection.


1. Failed writes resolved during the session race in-flight acks

Problem: On a sub-quorum write the client fired a resolution round (request_resolution) and retired every unresolved slot up to the failed one off the leader's reply, including slots whose writes were still in flight. A slot could be declared Empty and then reach its quorum ack, so the application was told a voided write succeeded. Making that safe needed an explicit failed-slot list and a "verdict epoch" in every request. Also, in production the application (XFS) goes offline on a write error and the volume detaches, so keeping it alive to finish the write gains nothing.

Diagram:

sequenceDiagram
    participant A as Application
    participant U as UBD
    participant L as Leader
    participant R as Replicas
    U->>R: write 100 (fails: only 1 of 3)
    U->>R: write 99 (still in flight)
    U-->>A: write 100: error
    U->>L: RESOLVE up to 100
    L->>R: nobody holds 99 yet → 99 is Empty
    R-->>U: acks for 99 arrive → quorum
    U-->>A: write 99: success (but it was voided)
Loading

Solution: The client sends no resolution request. It reports the error, keeps the slot pinning its frontier, and keeps serving overlapping reads as of just before the slot (reads never block). The next login decides the slot under the login fence: it is filled everywhere if a live member holds it, or, if nobody holds it, the login's starting point is below it and the dLSN is reused. The client's resolution driver and the tracker/router state only it used are removed. The request_resolution verb and wire resolve op stay for compatibility (HomeBlocks references them).

sequenceDiagram
    participant A as Application
    participant U as UBD
    participant R as Replicas
    participant L as Leader
    U->>R: write 101 (only Replica C stores it)
    U-->>A: error (XFS goes offline, volume detaches)
    Note over U,R: no RESOLVE; slot 101 pins the commit point, reads see the old data
    Note over L,R: next attach: login under the fence
    L->>R: C holds 101 → fill everywhere (or nobody holds it → dLSN reused)
Loading

2. The reference cluster's login did not resolve failed writes

Problem: MemTransport::run_login only applied up to the starting point. It never filled a slot a member lacked or voided one nobody held, so "resolved at the next login" didn't hold in the model, and a lagging member stayed behind after login.

Diagram:

sequenceDiagram
    participant L as Leader (holds 1)
    participant B as Replica B
    Note over L,B: dLSN 1 failed: only the leader holds it
    L->>B: login: SyncRSCommitLSN(1)
    B->>B: before: apply stalls at missing 1 (nothing fetched)
    B->>B: after: login filled 1 from the leader → commit reaches 1
Loading

Solution: Login runs the same fill-or-Empty step the resolution round already had (factored into MemTransport::resolve_slots_upto), for every slot up to its starting point.


3. A half-sent frame stays on a live connection (D17)

Problem: ring_send_all can stop partway through a frame: a short send that errors, or no free submission slot. round_trip then released the send slot and returned net_error::send but kept the connection. That error maps to REPLICA_DOWN, "never delivered". The next message's bytes then followed the fragment, so the server could parse a write whose payload mixed the two messages, and data digests are off by default.

Diagram:

sequenceDiagram
    participant U as UBD (craft_async_conn)
    participant S as Replica
    U->>S: write 5: first half of the frame, then the send fails
    U->>U: before: report REPLICA_DOWN, keep the socket
    U->>S: write 6 frame (follows the fragment on the same stream)
    S->>S: parses write 5 with write 6's bytes as payload → corrupt slot
Loading

Solution: On any send failure, round_trip drops the socket before handing the send slot on. The pump sees the close and fails every other pending op, a send already queued behind it fails on the dead fd, and the next op reconnects. That's the same teardown the existing reconnect path uses.


Testing

  • Built and tested on Linux (Debug + coverage): all 15 test suites pass.
  • Coverage of the changed files: client.cpp 87%, dlsn_tracker.cpp 98%, mem/cluster.cpp 97%, read_route_map.hpp 97%, async_conn.cpp 96%. For async_conn.cpp gcov reports 0 hits on every line of round_trip, including the normal path that clearly runs, so per-line coverage there isn't reliable. The new send-failure branch has no dedicated test: forcing a short send deterministically needs a fault-injection hook the transport doesn't have yet.
  • New and updated tests:
    • in-memory:
      • a failed write is not resolved during the session (frontier pinned, old data served, no copy made);
      • the next login fills it when a live member holds it;
      • the next login drops it when nobody does, and the new session reuses the dLSN.
    • TCP: the failed write stays unresolved during the session.
    • ReLoginMustNotServeStaleFollower: keeps member 2 down during the login, since a live member is now healed by the login.
  • docs/client-internals.md is updated; docs/resync.md points to the HomeBlocks design.

Not in this PR: data digests on by default for peer traffic (follow-up), and removing the now-unused request_resolution verb / resolve op (needs a coordinated HomeBlocks change).

🤖 Generated with Claude Code

…on a failed send

- A failed (sub-quorum) write no longer fires the client-requested resolution
  round. The client reports the error, pins its frontier and keeps serving
  overlapping reads as of just before the slot; the next login resolves it
  (HomeBlocks docs/craft/resync.md §4.4-§4.5: mid-session verdicts raced
  in-flight acks). Removes the client's resolution driver and the tracker /
  router state only it used. The request_resolution verb and wire resolve op
  stay for compatibility.
- The in-memory reference cluster's login now resolves every unresolved slot
  up to its starting point (fill from a live holder, or Empty), sharing the
  step its resolution round already had, so the model matches the design.
- craft_async_conn::round_trip drops the socket on any send failure before
  handing the send slot on. A partially sent frame could otherwise be followed
  by the next message's bytes while the caller reports REPLICA_DOWN ("never
  delivered").
- Tests: failed write unresolved during the session (in-memory and TCP); the
  next login fills it when a live member holds it, and drops it when nobody
  does; ReLoginMustNotServeStaleFollower keeps member 2 down during the login.
- docs: client-internals.md updated; docs/resync.md points to the HomeBlocks
  design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical issues remain: unfenced old-session writes can cause acknowledged data loss, and stale send failures can close replacement connections.

2 open issues
What changed in this PR

This PR moves failed-write resolution from the active client session to the next login, aligning the client and in-memory reference cluster with the HomeBlocks resync design. It also closes connections after send failures to prevent subsequent messages from extending a partial frame.

Changes:

  • Remove client-driven resolution rounds and their tracker/router state.
  • Resolve missing slots during reference-cluster login, with updated in-memory and TCP tests.
  • Drop failed-send connections before releasing the send slot and document the revised behavior.
File Description
test/​test_client.cpp Test session-pinned failures and next-login recovery.
test/​test_client_tcp.cpp Verify failed writes remain unresolved during TCP sessions.
src/​read_route_map.hpp Remove resolution-request tracking.
src/​net/​async_conn.cpp Close the connection before releasing a failed send.
src/​mem/​cluster.hpp Declare the shared slot-resolution helper.
src/​mem/​cluster.cpp Share slot resolution between login and explicit resolution.
src/​dlsn_tracker.hpp Remove resolution watermarks and bulk retirement APIs.
src/​dlsn_tracker.cpp Remove resolution-watermark implementation.
src/​client.cpp Stop issuing resolution requests after failed writes.
src/​client_impl.hpp Remove the resolution-driver declaration.
docs/​resync.md Link the HomeBlocks design and summarize client responsibilities.
docs/​client-internals.md Document next-login recovery and retained compatibility verbs.

Review effort: Balanced

Comment thread src/mem/cluster.cpp Resolved
Comment thread src/net/async_conn.cpp Resolved
…tion-guarded drop on send failure

- MemTransport::run_login fences every live replica (switches it to the new
  term) before polling and resolving, so an old-session write can no longer
  reach quorum after the absence scan and then be voided by it.
- craft_async_conn tracks a connection generation; a failed send drops the
  socket only if it is still the connection the frame went out on, so a stale
  queued send's late failure cannot close a newer socket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 18:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The send-error teardown and generation guard lack deterministic regression tests for partial-send failures with queued operations and stale failures after reconnect.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread src/net/async_conn.cpp Outdated
Comment on lines +240 to +241
if (gen == conn_gen_) drop_fd();
release_send();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 6a68754. craft_async_conn gets a deterministic test seam (set_send_fault_for_test: cap a send to N bytes, or fail it; bump_conn_gen_for_test). Two on-ring tests against a scripted replica in test_auth_handshake.cpp:

  • OnRingSendFailureMidFrameDropsTheConnection: the first frame sends 10 bytes, then its send fails. The op fails with send, the op queued behind it fails too, the connection is no longer ready, and the next op reconnects (2 connections accepted) and carries traffic.
  • OnRingStaleSendFailureLeavesTheNewConnection: the connection generation moves on before the send fails. The op fails, but the socket stays ready, the next op succeeds, and no reconnect happens (1 connection accepted).

All 15 suites pass.

Deterministic send-fault seam on craft_async_conn (set_send_fault_for_test:
cap a send to N bytes or fail it; bump_conn_gen_for_test). Two on-ring tests
against a scripted replica:
- a frame that fails after a partial send drops the connection; the op
  queued behind it fails, and the next op reconnects and carries traffic;
- a stale send failure (the connection generation moved on) leaves the
  current socket open and usable, with no reconnect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 19:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A critical send-slot ownership race lets an old completion release a replacement frame’s slot and interleave frames, while the tests fail to verify late old-session writes and fence-before-poll ordering.

1 open issue
3 previously missed issues

In code that hasn't changed since last review

High severity Make send-slot release generation-aware across reconnects

src/​net/​async_conn.cpp:248

The generation guard protects the replacement socket, but the unconditional release_send() can still release its send slot. If fail_all() resets sending_ and reconnect starts before an old send's error completion is processed, that late completion can release a slot owned by a new frame, allowing frames to interleave. Drain old sends before reconnecting, or make send-slot ownership and release generation-aware across teardown and both completion paths.

Medium severity Test old-session writes between the absence scan and Empty installation

src/​mem/​cluster.cpp:396

The new tests do not exercise an old-session write arriving during the login poll/resolution scan; they would still pass if this fence were moved after resolution. The 10-second straggler test checks its data before that delivery runs. Please add a deterministic interleaving test that delivers an old-term write after the absence scan but before Empty installation and verifies rejection, so the fence-before-poll ordering that prevents acknowledged data loss is regression-protected.

Medium severity Verify delayed old writes cannot overwrite a reused slot

test/​test_client.cpp:308

The new reuse test reads the replacement value before the old write's 10-second delivery, with no assertion after that delivery (remaining callbacks run during MemTransport::shutdown). It would therefore still pass if the late old-session write overwrote the reused slot. Please make the delayed delivery observable, wait for it after writing the replacement, and then verify that the replacement remains intact.

Review effort: Balanced

…sion writes

- craft_async_conn: the send slot is owned explicitly. send_awaitable yields
  true only to an op that now holds the slot, with the slot's epoch.
  fail_all() frees the slot by force and advances the epoch, so the old
  owner's late release_send(epoch) is a no-op instead of releasing a slot a
  newer frame owns (a third frame could then interleave with it). Ops
  fail_all wakes from the queue get no slot and return `closed` without
  sending.
- Test OnRingStaleSendCannotReleaseANewerFramesSlot reproduces the race
  deterministically (fails with the epoch check removed).
- Model: replica_stats.stale_term_writes counts writes rejected for another
  session's term; MemTransport::set_login_poll_hook_for_test runs between the
  login poll and resolution.
- N3_OldSessionWriteBetweenPollAndResolutionIsRejected: an old-term write
  delivered between the poll and resolution is rejected on every member and
  never lands.
- N3_NextLoginDropsAFailedWriteNobodyHolds now waits for the delayed
  old-session write to be delivered (and rejected) and checks the reused slot
  afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 20:03
@Ilyin-Ilya

Copy link
Copy Markdown
Collaborator Author

Re Copilot's three "previously missed" findings on 6a68754, all addressed in c411698:

  1. Send-slot ownership across teardown (async_conn.cpp). Confirmed and fixed: this was a real race, not only a missing test. The send slot is now owned explicitly:

    • send_awaitable yields true only to an op that holds the slot, and records the slot's epoch.
    • fail_all() frees the slot and advances the epoch, so the old owner's late release_send(epoch) does nothing.
    • Ops that fail_all wakes up from the queue get no slot and return closed without sending.

    OnRingStaleSendCannotReleaseANewerFramesSlot reproduces the race deterministically, with frame A mid-send, a teardown plus reconnect, B taking the slot, and A failing late. It checks that C waits for B. With the epoch check removed, that test fails.

  2. Old-session write between the poll and Empty installation (cluster.cpp). Added set_login_poll_hook_for_test (it runs after the poll, before resolution) and a stale_term_writes counter in the model replica's stats. N3_OldSessionWriteBetweenPollAndResolutionIsRejected delivers an old-term write to every member in that window: all 3 reject it with STALE_TERM and the slot never lands. The test would fail if the fence moved after resolution.

  3. Delayed old write vs the reused slot (test_client.cpp). N3_NextLoginDropsAFailedWriteNobodyHolds now uses a 1 s delay, waits until replica 0 reports the delayed write as delivered and rejected (stale_term_writes == 1), and then re-reads the reused slot and checks the journals.

All 15 suites pass.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An unsent queued frame is reported as net_error::closed rather than net_error::send, incorrectly making delivery uncertain and leaving the client slot pinned until another login when all write legs are cancelled.

2 open issues

Review effort: Balanced

Comment thread src/net/async_conn.cpp Resolved
An op woken from the send queue by fail_all() returns net_error::send (never
delivered), not closed (delivery uncertain), so a write whose legs all end
this way can be retired instead of pinning its slot until the next login.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 11, 2026 20:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/net/async_conn.cpp
Comment on lines +239 to 248
if (!co_await send_awaitable{this}) {
// Woken by fail_all() without the slot: the connection went down while we queued. Nothing was sent, so this
// is a send failure -- provably never delivered -- not `closed` (delivery uncertain).
pending_.erase(rid);
co_return std::unexpected(net_error::send);
}
uint64_t const epoch = send_epoch_; // our ownership of the send slot
uint64_t const gen = conn_gen_; // the connection this frame goes out on
int const sr = co_await ring_send_all(out);
release_send();
if (sr < 0) {
Comment thread src/net/async_conn.cpp
Comment on lines 256 to +261
pending_.erase(rid);
if (gen == conn_gen_) drop_fd();
release_send(epoch);
co_return std::unexpected(net_error::send);
}
release_send(epoch);
Comment thread docs/resync.md

- **Failed writes are resolved at the next login, not during the session.** On a sub-quorum write the client
reports the error, pins its frontier `F` below the slot, and keeps serving overlapping reads as of just before
it. It no longer calls `request_resolution`, and the `resolve` op (13/14) is retired.
Comment thread src/net/async_conn.cpp
Comment on lines 146 to +149
std::size_t off = 0;
while (off < data.size()) {
sisl::async::cqe_awaitable ev;
std::size_t len = data.size() - off;
Comment thread src/net/async_conn.cpp
Comment on lines 155 to 156
auto* sqe = acquire_sqe();
if (nullptr == sqe) co_return -EAGAIN;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants