diff --git a/docs/client-internals.md b/docs/client-internals.md index e864c6f..3f1651e 100644 --- a/docs/client-internals.md +++ b/docs/client-internals.md @@ -115,22 +115,13 @@ holds the slot, it is a permanent no-op, and the frontier may pass over it. A su degrades to `failed`. Counting a timeout here would advance `F` past a dLSN a peer later applies, which is divergence. In every other respect all non-acks are equal. -A `failed` outcome no longer waits for the next login: it fires the **client-requested resolution round** -(`craft_replica::request_resolution`, the design's client-request `SyncRSCommitLSN` trigger). The client -cannot know who the leader is mid-session (it learned one at login; leadership may have moved), so the -request is **broadcast to every member** as detached fire-and-forget legs, with at most **one outstanding per -peer** -- exactly the keep_alive collapse (`rr_inflight_` beside `ka_inflight_`), so a burst of failed -in-flight writes collapses into the pending legs rather than one request per failure. The failed dLSNs -CAS-max into a single *want* watermark (`dlsn_tracker`); whichever member is the current leader resolves -every unresolved slot <= the want -- fills it from a holder (the failed write completes late) or verdicts it -Empty on quorum-lacks evidence -- while non-leaders answer `NOT_LEADER` and leave the want untouched (only a -successful round consumes it). The client retires the covered slots off the reply -(`dlsn_tracker::retire_upto`): Empty verdicts resolve `empty`, filled slots resolve `acked` with the router -told first (`note_filled`: the slot becomes verdict-durable and reads route only to its certain holders until -the commit certificates catch up). A replica REJECTS a late arrival into an Empty-verdicted slot, so a write -the round voided fails deterministically on its own ack path -- the verdict and the client's local `Empty` -conclusion agree. If no leg can resolve (leader down), the slot simply stays pinned and the next failed write -re-fires the legs. +A `failed` outcome is **resolved at the next login, not during the session**. The application gets the error (in +production XFS then goes offline and the volume is detached), the failed slot pins the frontier `F`, and reads that +overlap it keep being served as of just before it (see *Why per-block*). The client sends no resolution request. +The next login decides the slot under the login fence: if a live member holds it, it is filled everywhere (the +write completed late); if nobody holds it, the login's starting point is below it and the dLSN is simply reused. +The `request_resolution` verb and the wire `resolve` op still exist for compatibility but are no longer used by +the client. Background: HomeBlocks `docs/craft/resync.md` §4.4-§4.5 (mid-session verdicts raced in-flight acks). ### Where a write actually goes @@ -396,8 +387,8 @@ Also outstanding, in rough order: - the server-side **watchdog** the keep_alive drive is meant to reset (a deferred seam today, `mem_craft_replica.cpp`), and server-side **resync** (`SyncRSCommitLSN`) that makes the recovery erase above - effective -- both are server/peer concerns, not client work (the client-REQUESTED resolution round and its - retire path now exist: `request_resolution` / `retire_upto`, see *A non-ack is not one thing*); + effective -- both are server/peer concerns, not client work (failed writes are resolved at the next login, + see *A non-ack is not one thing*); - re-login on `STALE_TERM`; - the client-side all-zero scan that picks the CRAFT zero-write form (the server does not re-scan a data write), plus ublk `DISCARD` / `WRITE_ZEROES` → a CRAFT zero write (empty `sg_list`); diff --git a/docs/resync.md b/docs/resync.md new file mode 100644 index 0000000..d97f932 --- /dev/null +++ b/docs/resync.md @@ -0,0 +1,14 @@ +# CRAFT resync + +The resync design lives in HomeBlocks: [`docs/craft/resync.md`](https://github.com/eBay/HomeBlocks/blob/dev/v6.x/docs/craft/resync.md). +Resync is driven by the CRAFT BM side; this package (UBD's CRAFT client) does not drive it. + +What the design asks of this repo (PR F there): + +- **Failed writes are resolved at the next login, not during the session.** On a sub-quorum write the client + reports the error, pins its frontier `F` below the slot, and keeps serving overlapping reads as of just before + it. It no longer calls `request_resolution`, and the `resolve` op (13/14) is retired. +- **The transport must keep "never delivered" true.** `craft_async_conn` drops the connection on any send failure, + so a half-sent frame can never be followed by the next message (D17 in the design's Appendix A). +- **Peer-plane codec v2** (`include/craft/peer.hpp`): range-based, byte-bounded `fetch_data` with continuation and + a `reclaimed_below` status, plus term and partition fields on peer requests (PR E). diff --git a/src/client.cpp b/src/client.cpp index 1eaba11..2977a38 100644 --- a/src/client.cpp +++ b/src/client.cpp @@ -229,54 +229,13 @@ async_result< size_t > craft_client::write(::io_uring* q, uint64_t addr, uint64_ bool const provably_empty = (refused == replicas_.size()); tracker_->resolve(dlsn, provably_empty ? slot_outcome::empty : slot_outcome::failed); maybe_fold(); - // A failed (sub-quorum, not provably-absent) slot pins the frontier until the leader fills or Empties it: - // request the resolution round NOW (the design's client-request SyncRSCommitLSN trigger) instead of - // waiting for a watchdog / periodic cadence that the client cannot see. - if (!provably_empty) request_resolution_round(q, dlsn); + // A failed (sub-quorum, not provably-absent) slot pins the frontier. It is NOT resolved during the session: + // the application gets the error (XFS then goes offline and the volume detaches), and the next login + // resolves the slot under the login fence -- filled from a holder or declared Empty (HomeBlocks + // docs/craft/resync.md §4.4). Overlapping reads keep being served as of just before the slot meanwhile. co_return std::unexpected(last_err ? last_err : make_error_condition(craft_error::NO_QUORUM)); } -// One detached resolution leg to member `idx` -- fire-and-forget, like fire_keepalive: it captures the -// tracker/route shared_ptrs and the backend by value, never the client. It drains the want watermark: a burst -// of failed writes collapses into this one outstanding request per peer, and a want that lands while the -// leader is resolving is picked up by the next loop pass. A member that is NOT the leader answers NOT_LEADER -// and the leg simply ends, LEAVING the want set -- only the leader's leg (whichever member that currently is) -// resolves and consumes it. On success the covered slots are retired off the verdicts: Empty ones as Empty, -// filled ones as acked with the router told first (note_filled) so reads route only to certain holders until -// the commit_lsn certificates catch up. A leg that errors (member down, deposed term) also just ends: the -// slot stays pinned and the next failed write re-fires the legs. -static async_status fire_resolution(::io_uring* q, std::shared_ptr< craft_replica > h, std::size_t idx, uint64_t term, - std::shared_ptr< dlsn_tracker > tracker, std::shared_ptr< read_route_map > route) { - for (;;) { - while (auto const want = tracker->resolution_want()) { - client_hdr const hdr{term, tracker->frontier(), route->all_committed()}; - auto r = co_await h->request_resolution(q, hdr, *want); - if (!r.has_value()) { // NOT_LEADER / down / stale: this peer cannot resolve; leave the want alone - route->end_resolution(idx); - co_return ok(); - } - tracker->retire_upto(r->resolved_upto, r->empty_slots, [&](int64_t d) { route->note_filled(d, idx); }); - tracker->clear_resolution_want(r->resolved_upto); - } - route->end_resolution(idx); - // Close the note-then-begin race: a want recorded between our last peek and the flag release would - // strand until the next failure. Reclaim this peer's flag only if a want is actually pending. - if (!tracker->resolution_want() || !route->try_begin_resolution(idx)) co_return ok(); - } -} - -void craft_client::request_resolution_round(::io_uring* q, int64_t upto) { - tracker_->note_resolution_want(upto); - // BROADCAST, not a leader walk: the client learned the leader at login and leadership may have moved - // since -- mid-session it cannot know who leads. Every member gets the request, at most one outstanding - // per peer (the keep_alive collapse); whichever member IS the leader runs the round, the rest answer - // NOT_LEADER (a real replica may instead forward to its leader -- either way the client need not know). - // Each leg rides the failing write's ring, so its resumptions stay on that queue's reap thread. - for (std::size_t m = 0; m < replicas_.size(); ++m) { - if (route_->try_begin_resolution(m)) fire_resolution(q, replicas_[m], m, term_, tracker_, route_).detach(); - } -} - // Issue a whole read plan to one target, filling `dest` in place; resolves to the target's piggybacked // watermarks (the highest across a split's segments -- same member, monotonic). Factored out of read() so the // router can retry it against the next eligible member on a transport failure. `dest` is copied per attempt diff --git a/src/client_impl.hpp b/src/client_impl.hpp index 861353a..a1cbce3 100644 --- a/src/client_impl.hpp +++ b/src/client_impl.hpp @@ -80,12 +80,6 @@ class craft_client { std::optional< std::error_condition > precheck(uint64_t addr, uint64_t len) const; async_result< lsn_pair > issue_plan(::io_uring* q, std::shared_ptr< craft_replica > const& target, client_hdr hdr, read_plan const& plan, uint64_t addr, uint64_t len, sisl::sg_list& dest); - // Fire the client-requested resolution round for failed slot `upto`: record the want, then BROADCAST a - // detached request to every peer without one outstanding (per-peer single-flight, the keep_alive - // collapse) -- the client cannot know who leads mid-session, so whichever member is the leader resolves. - // Each leg captures the tracker's and router's shared_ptrs, never `this`; it rides the ring of the write - // whose failure fired it. - void request_resolution_round(::io_uring* q, int64_t upto); std::vector< std::shared_ptr< craft_replica > > replicas_; uint32_t leader_{0}; diff --git a/src/dlsn_tracker.cpp b/src/dlsn_tracker.cpp index c0c4911..9631ca6 100644 --- a/src/dlsn_tracker.cpp +++ b/src/dlsn_tracker.cpp @@ -62,7 +62,6 @@ void dlsn_tracker::reset_at(int64_t login_dlsn, uint32_t lba_size) { highest_acked_.store(login_dlsn, std::memory_order_release); // A want must not leak across sessions: slot numbers are REUSED above the login watermark, so a stale // want could void the new session's in-flight slots. - resolve_want_.store(-1, std::memory_order_relaxed); // StreamTracker(name, start_idx) bases its window at start_idx + 1, i.e. the first dLSN we will issue. tracker_.emplace("craft_dlsn", login_dlsn); } @@ -131,28 +130,6 @@ void dlsn_tracker::advance_frontier() { } } -// ── the resolution-round want watermark ── - -void dlsn_tracker::note_resolution_want(int64_t upto) { - int64_t w = resolve_want_.load(std::memory_order_relaxed); - while (upto > w && - !resolve_want_.compare_exchange_weak(w, upto, std::memory_order_release, std::memory_order_relaxed)) {} -} - -std::optional< int64_t > dlsn_tracker::resolution_want() const { - int64_t const w = resolve_want_.load(std::memory_order_acquire); - if (w < 0) return std::nullopt; - return w; -} - -void dlsn_tracker::clear_resolution_want(int64_t resolved_upto) { - // Consume only what the round covered: a HIGHER want that landed mid-round survives the clear and the - // leader's leg loops for it. - int64_t w = resolve_want_.load(std::memory_order_relaxed); - while (w >= 0 && w <= resolved_upto && - !resolve_want_.compare_exchange_weak(w, -1, std::memory_order_acq_rel, std::memory_order_relaxed)) {} -} - tracker_stats dlsn_tracker::stats(std::size_t sample_limit) const { tracker_stats s; s.winner_scans = winner_scans_.load(std::memory_order_relaxed); diff --git a/src/dlsn_tracker.hpp b/src/dlsn_tracker.hpp index 2c2e9e7..6c0c2f9 100644 --- a/src/dlsn_tracker.hpp +++ b/src/dlsn_tracker.hpp @@ -131,47 +131,6 @@ class dlsn_tracker { // are still completing -- whichever lands second is dropped, not fought over. void resolve(int64_t dlsn, slot_outcome oc); - // ── the client-requested resolution round (the design's client-request SyncRSCommitLSN trigger) ── - // - // The WANT watermark: the highest failed dLSN awaiting a round, hosted here so the detached per-peer - // resolution legs outlive the client safely (each captures the tracker's shared_ptr). A burst of failed - // writes CAS-maxes into one value; only a SUCCESSFUL round consumes it (clear_resolution_want) -- a - // non-leader leg answering NOT_LEADER must leave it for the leader's leg. The per-peer one-outstanding - // gate lives in read_route_map (rr_inflight_), beside the keep_alive collapse it mirrors. - void note_resolution_want(int64_t upto); // CAS-max: record a failed slot needing a round - std::optional< int64_t > resolution_want() const; // peek (never consumes); nullopt = nothing pending - void clear_resolution_want(int64_t resolved_upto); // consume iff the want is <= what the round resolved - - // After a round: every slot <= upto is resolved server-side. Walk OUR unresolved slots <= upto and retire - // each: Empty if it is in `empties` (ascending), else acked (it was filled from a holder -- the failed - // write completed late). `on_filled(dlsn)` fires BEFORE the acked resolve, so the router can queue its - // conservative miss / verdict flag first (the same push-before-completed-bit ordering record_completion - // keeps). A slot whose create() is not yet visible is skipped: its own legs resolve it (they are rejected - // deterministically into a verdicted slot). - template < typename Fn > - void retire_upto(int64_t upto, std::vector< int64_t > const& empties, Fn&& on_filled) { - if (!tracker_) return; - int64_t const hi = std::min(upto, next_dlsn_.load(std::memory_order_acquire) - 1); - for (int64_t h = frontier() + 1; h <= hi;) { - int64_t const u = std::min(tracker_->completed_upto(h), hi) + 1; - if (u > hi) break; - bool active = false; - tracker_->foreach_all_active(u, [&](int64_t idx, dlsn_slot&) { - active = (idx == u); - return false; - }); - if (active) { - if (std::binary_search(empties.begin(), empties.end(), u)) { - resolve(u, slot_outcome::empty); - } else { - on_filled(u); - resolve(u, slot_outcome::acked); - } - } - h = u + 1; - } - } - int64_t frontier() const { return frontier_.load(std::memory_order_acquire); } int64_t read_horizon() const { return std::max(frontier(), highest_acked_.load(std::memory_order_acquire)); } @@ -214,7 +173,6 @@ class dlsn_tracker { std::atomic< int64_t > highest_acked_{-1}; // Ha: max quorum-acked dLSN std::atomic< int64_t > last_trunc_{-1}; std::atomic< uint64_t > winner_scans_{0}; - std::atomic< int64_t > resolve_want_{-1}; // highest failed dLSN awaiting a resolution round std::optional< sisl::StreamTracker< dlsn_slot > > tracker_; // constructed at login resolution_gate gate_; diff --git a/src/mem/cluster.cpp b/src/mem/cluster.cpp index bd8a3fd..4c4ac8a 100644 --- a/src/mem/cluster.cpp +++ b/src/mem/cluster.cpp @@ -335,6 +335,44 @@ std::vector< MemCraftReplica* > MemTransport::live_replicas_locked() const { return live; } +// The leader's resolution step over the live set (a quorum), shared by login and the resolution round: every +// slot <= upto that is not yet contiguously present everywhere is filled from a live holder, or -- when the whole +// live set lacks it, or a prior Empty verdict exists -- verdicted Empty everywhere. Run by login under the +// session switch, this is where a failed (sub-quorum) write is resolved: it completes late if any live member +// holds it, and is voided otherwise (HomeBlocks docs/craft/resync.md §4.4). +void MemTransport::resolve_slots_upto(std::vector< MemCraftReplica* > const& live, int64_t upto) { + // Everything <= a member's commit_lsn is contiguously present there, so the min over the live set bounds + // the scan: below it nothing can be unresolved. + int64_t floor = std::numeric_limits< int64_t >::max(); + for (auto* r : live) + floor = std::min(floor, r->peek_lsns().commit_lsn); + + for (int64_t d = floor + 1; d <= upto; ++d) { + // One pass over the live set: a prior Empty verdict is authoritative (it beats a held copy -- the + // copy was never acked); else any live holder's copy fills the others; else the whole live set + // (a quorum) positively lacks the slot and it is verdicted Empty everywhere. + std::optional< MemCraftReplica::MemJournalSlot > copy; + bool verdicted = false; + for (auto* r : live) { + if (auto s = r->peek_slot(d)) { + if (s->is_empty) { + verdicted = true; + break; + } + if (!copy) copy = std::move(s); + } + } + if (verdicted || !copy) { + for (auto* r : live) + r->cold_mark_empty(d); + } else { + for (auto* r : live) { + if (!r->peek_slot(d)) r->cold_install_slot(d, *copy); + } + } + } +} + result< LoginResult > MemTransport::run_login(MemCraftReplica* caller, uint64_t client_token) { std::vector< MemCraftReplica* > live; std::vector< replica_endpoint > members_copy; @@ -349,10 +387,21 @@ result< LoginResult > MemTransport::run_login(MemCraftReplica* caller, uint64_t members_copy = members_; } // Drive the cold path WITHOUT holding mu_ (each replica call takes only its own lock). + // Phase 0: fence. Every live replica switches to the new term BEFORE it is polled, so from here on it rejects + // the old session's writes: none can reach quorum after the absence scan below and then be voided by it. In + // this model a write is applied atomically under the replica's lock, so there is no admitted-but-unfinished + // write to drain; a delayed delivery that arrives later carries the old term and is rejected. Nobody writes + // under the new term until this login returns, so the early switch admits no one. + for (auto* r : live) + r->cold_fence(nt); // Phase 1: GetRSCommitLSN -> rs_commit_lsn = max(quorum.last_append_lsn). int64_t rs = -1; for (auto* r : live) rs = std::max(rs, r->peek_lsns().last_append_lsn); + if (login_poll_hook_) login_poll_hook_(); + // Phase 1b: resolve every unresolved slot <= rs (fill from a holder, or verdict Empty) -- failed writes of the + // previous session are decided here, not mid-session. + resolve_slots_upto(live, rs); // Phase 2: SyncRSCommitLSN(rs) applied to all live members. for (auto* r : live) r->cold_apply_sync(rs, client_token); @@ -379,36 +428,7 @@ result< resolution_result > MemTransport::run_resolution(MemCraftReplica* caller if (live.size() < quorum(members_.size())) return fail(craft_error::NO_QUORUM); } - // Everything <= a member's commit_lsn is contiguously present there, so the min over the live set bounds - // the scan: below it nothing can be unresolved. - int64_t floor = std::numeric_limits< int64_t >::max(); - for (auto* r : live) - floor = std::min(floor, r->peek_lsns().commit_lsn); - - for (int64_t d = floor + 1; d <= upto; ++d) { - // One pass over the live set: a prior Empty verdict is authoritative (it beats a held copy -- the - // copy was never acked); else any live holder's copy fills the others; else the whole live set - // (a quorum) positively lacks the slot and it is verdicted Empty everywhere. - std::optional< MemCraftReplica::MemJournalSlot > copy; - bool verdicted = false; - for (auto* r : live) { - if (auto s = r->peek_slot(d)) { - if (s->is_empty) { - verdicted = true; - break; - } - if (!copy) copy = std::move(s); - } - } - if (verdicted || !copy) { - for (auto* r : live) - r->cold_mark_empty(d); - } else { - for (auto* r : live) { - if (!r->peek_slot(d)) r->cold_install_slot(d, *copy); - } - } - } + resolve_slots_upto(live, upto); for (auto* r : live) r->cold_apply_sync(upto, 0); diff --git a/src/mem/cluster.hpp b/src/mem/cluster.hpp index 416f754..64e689c 100644 --- a/src/mem/cluster.hpp +++ b/src/mem/cluster.hpp @@ -157,8 +157,13 @@ class MemTransport { // How long the transport waits before abandoning an op. 0 (the default) means wait forever, which is // the model's original behavior: without it a delay merely slows the client down. void set_op_timeout(std::chrono::milliseconds t); + // Test seam: run by the model login right after it fenced and polled the live members, before it resolves any + // slot -- lets a test deliver an old-session write exactly between the poll and the Empty verdicts. + void set_login_poll_hook_for_test(std::function< void() > f) { login_poll_hook_ = std::move(f); } private: + // Leader resolution step over the live set, shared by run_login and run_resolution (see cluster.cpp). + static void resolve_slots_upto(std::vector< MemCraftReplica* > const& live, int64_t upto); // What a suspended op resumes on. Single-shot and completed from a service thread; the awaiting // coroutine resumes there, which is why nothing here may hold mu_ across a co_await. using sleep_event = sisl::async::shared_awaitable< std::monostate >; @@ -227,6 +232,7 @@ class MemTransport { // The client's set-wide deadline (0 == wait forever). NOT a replica fault; read lock-free on the IO path. std::atomic< int64_t > op_timeout_ms_{0}; + std::function< void() > login_poll_hook_; // test seam only // Built once in the ctor from members_ and never rehashed, so a replica_service* stays stable and the // per-pool locks need no protection from mu_. diff --git a/src/mem/replica.cpp b/src/mem/replica.cpp index 0deb3d1..0a24eae 100644 --- a/src/mem/replica.cpp +++ b/src/mem/replica.cpp @@ -237,7 +237,10 @@ result< lsn_pair > MemCraftReplica::do_write(client_hdr hdr, int64_t dlsn, uint6 return std::unexpected(std::make_error_condition(std::errc::invalid_argument)); } std::lock_guard< std::mutex > g{mu_}; - if (hdr.term != state_.term) return fail(craft_error::STALE_TERM); + if (hdr.term != state_.term) { + ++stale_term_writes_; + return fail(craft_error::STALE_TERM); + } if (auto it = journal_->find(dlsn); it != journal_->end() && it->second.is_empty) { // An Empty verdict is permanent (reconciliation: Empty beats data). A late arrival into the slot is // REJECTED -- deterministically -- so that write's own ack path concludes the slot is void, matching @@ -458,6 +461,7 @@ replica_stats MemCraftReplica::stats() const { s.term = state_.term; s.client_token = state_.client_token; s.mapped_blocks = index_->size(); + s.stale_term_writes = stale_term_writes_; s.journal_slots = journal_->size(); if (!journal_->empty()) { @@ -511,6 +515,11 @@ void MemCraftReplica::cold_apply_sync(int64_t rs_commit_lsn, uint64_t /*client_t // SyncRSCommitLSN: (seam) fetch any missing non-Empty slots <= rs from peers, then advance commit. apply_up_to(rs_commit_lsn); } +void MemCraftReplica::cold_fence(uint64_t term) { + std::lock_guard< std::mutex > g{mu_}; + state_.term = term; // old-session writes now fail STALE_TERM; the session itself is installed by cold_apply_login + on_state_changed(); +} void MemCraftReplica::cold_apply_login(uint64_t client_token, uint64_t term) { std::lock_guard< std::mutex > g{mu_}; state_.client_token = client_token; diff --git a/src/mem/replica.hpp b/src/mem/replica.hpp index 65409c3..5f78623 100644 --- a/src/mem/replica.hpp +++ b/src/mem/replica.hpp @@ -91,6 +91,8 @@ struct replica_stats { std::vector< int64_t > missing_sample; // ascending, capped at k_missing_sample std::size_t mapped_blocks{0}; // index_.size(); coverage of the applied prefix + + uint64_t stale_term_writes{0}; // writes rejected for carrying another session's term (e.g. a late old-session one) }; // One replica's fault/latency knobs, in an immutable snapshot the IO path reads with a single acquire load @@ -276,6 +278,7 @@ class MemCraftReplica : public craft_replica, protected: virtual void on_state_changed() {} // called under mu_ after every state_ mutation; override to persist + void cold_fence(uint64_t term); // login fence: reject the old session's writes before the poll void cold_apply_login(uint64_t client_token, uint64_t term); void cold_truncate_above(int64_t rs_commit_lsn); void cold_install_slot(int64_t dlsn, MemJournalSlot s); // fill a hole; never overwrites an entry @@ -299,6 +302,7 @@ class MemCraftReplica : public craft_replica, std::shared_ptr< MemTransport > net_; CraftPartitionState state_; + uint64_t stale_term_writes_{0}; // guarded by mu_; reported in stats() // shared_ptr for restart support using registry_manager. std::shared_ptr< journal_t > journal_; // dLSN -> slot (out-of-order arrival tolerated) std::shared_ptr< index_t > index_; // applied prefix (<= commit_lsn); an absent LBA is a hole diff --git a/src/net/async_conn.cpp b/src/net/async_conn.cpp index 1c45a97..ca94b5f 100644 --- a/src/net/async_conn.cpp +++ b/src/net/async_conn.cpp @@ -16,6 +16,7 @@ #include "net/async_conn.hpp" #include "net/auth_client.hpp" +#include #include #include #include @@ -66,14 +67,16 @@ void craft_async_conn::drop_fd() noexcept { ready_ = false; } -void craft_async_conn::release_send() noexcept { +void craft_async_conn::release_send(uint64_t epoch) noexcept { + if (epoch != send_epoch_) return; // fail_all() took the slot back meanwhile: it is not ours to release if (send_waiters_.empty()) { sending_ = false; return; } - auto const h = send_waiters_.front(); - send_waiters_.erase(send_waiters_.begin()); // FIFO; sending_ STAYS true -- the slot is handed to h - if (h) h.resume(); + auto const w = send_waiters_.front(); + send_waiters_.erase(send_waiters_.begin()); // FIFO; sending_ STAYS true -- the slot is handed to w + *w.granted = true; + if (w.h) w.h.resume(); } void craft_async_conn::fail_all() noexcept { @@ -85,14 +88,16 @@ void craft_async_conn::fail_all() noexcept { slot->ready = true; if (slot->waiter) slot->waiter.resume(); } - // Ops parked for the send slot must unwind too: their reply_slots are already flagged failed above, so once - // resumed they send on the (now dead) fd, fall through the failed reply, and return an error. Resume after the - // reply-waiters so none observes a half-torn-down state. + // Ops parked for the send slot must unwind too. They resume WITHOUT the slot (granted stays false) and return an + // error without sending. The slot itself is freed by force; advancing the epoch makes its current owner's late + // release_send a no-op (see send_awaitable). Resume after the reply-waiters so none observes a half-torn-down + // state. sending_ = false; + ++send_epoch_; auto sw = std::move(send_waiters_); send_waiters_.clear(); - for (auto const h : sw) - if (h) h.resume(); + for (auto const& w : sw) + if (w.h) w.h.resume(); } void craft_async_conn::shutdown() noexcept { @@ -116,6 +121,7 @@ void craft_async_conn::shutdown() noexcept { sisl::async::light_task< int > craft_async_conn::ring_connect() { fd_ = ::socket(AF_INET, SOCK_STREAM, 0); + ++conn_gen_; if (fd_ >= 0) { // see set_nodelay in conn.cpp: CRAFT is request/response, so Nagle only ever adds latency int const one = 1; ::setsockopt(fd_, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one)); @@ -140,9 +146,15 @@ sisl::async::light_task< int > craft_async_conn::ring_send_all(std::span< uint8_ std::size_t off = 0; while (off < data.size()) { sisl::async::cqe_awaitable ev; + std::size_t len = data.size() - off; + if (send_fault_) { + long const f = send_fault_(off, len); + if (f < 0) co_return static_cast< int >(f); + if (f > 0) len = std::min(len, static_cast< std::size_t >(f)); + } auto* sqe = acquire_sqe(); if (nullptr == sqe) co_return -EAGAIN; - ::io_uring_prep_send(sqe, fd_, data.data() + off, data.size() - off, MSG_NOSIGNAL); + ::io_uring_prep_send(sqe, fd_, data.data() + off, len, MSG_NOSIGNAL); ::io_uring_sqe_set_data64(sqe, sisl::async::encode_managed_user_data(&ev)); int const n = co_await ev; if (n <= 0) co_return (n == 0 ? -EPIPE : n); // 0 = peer closed; <0 = error @@ -224,13 +236,29 @@ craft_async_conn::round_trip(wire::op o, std::span< uint8_t const > op_hdr, std: // Hold the wire's send slot for the WHOLE of this message so its bytes are contiguous on the socket, then // release before waiting for the reply (the next op sends while we wait; replies demux by request_id). - co_await send_awaitable{this}; + if (!co_await send_awaitable{this}) { + // Woken by fail_all() without the slot: the connection went down while we queued. Nothing was sent, so this + // is a send failure -- provably never delivered -- not `closed` (delivery uncertain). + pending_.erase(rid); + co_return std::unexpected(net_error::send); + } + uint64_t const epoch = send_epoch_; // our ownership of the send slot + uint64_t const gen = conn_gen_; // the connection this frame goes out on int const sr = co_await ring_send_all(out); - release_send(); if (sr < 0) { + // The frame may be PARTIALLY on the socket (ring_send_all stops after a short send that errors, or when no + // SQE is free). Keeping the connection would let the next message's bytes follow the fragment, so the server + // could parse a frame whose payload mixes the two, and the caller would report it as "never delivered" + // (REPLICA_DOWN). Drop the socket BEFORE handing the send slot on: the pump sees the close and fails every + // other pending op, a send already queued behind us fails on the dead fd, and the next op reconnects. + // Only if this is still the connection we sent on: a stale send's late failure must never close a newer + // socket that a reconnect opened meanwhile. pending_.erase(rid); + if (gen == conn_gen_) drop_fd(); + release_send(epoch); co_return std::unexpected(net_error::send); } + release_send(epoch); co_await reply_awaitable{&slot}; // the pump fills slot + resumes us (or fail_all does, with failed=true) pending_.erase(rid); // defensive: the pump already erased on delivery diff --git a/src/net/async_conn.hpp b/src/net/async_conn.hpp index ba23acd..04a20ef 100644 --- a/src/net/async_conn.hpp +++ b/src/net/async_conn.hpp @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -92,6 +93,16 @@ class craft_async_conn { // The status of the HELO that failed ensure_ready with net_error::helo_rejected. wire::status helo_rejection() const noexcept { return helo_rejection_; } + // Test seam for deterministic send faults: consulted before every send SQE with the bytes of the current frame + // already sent and still to send. 0 = send normally; > 0 = send at most that many bytes this time (a real partial + // send on the socket); < 0 = fail this send with that -errno, sending nothing. Unset in production. + using send_fault_fn = std::function< long(std::size_t sent, std::size_t remaining) >; + void set_send_fault_for_test(send_fault_fn f) { send_fault_ = std::move(f); } + // Test seam: behave as if a reconnect had opened a newer socket (socket() bumps this), leaving the current one. + void bump_conn_gen_for_test() noexcept { ++conn_gen_; } + // Test seam: the teardown a recv fault runs (fail every pending op, free the send slot by force). + void fail_all_for_test() noexcept { fail_all(); } + // Stop the pump, close the fd, and fail every pending leg. The CALLER must have quiesced the ring (no SQE of // ours left un-reaped) before this destroys the pump frame -- same rule as ublkpp's queue_service dtor: the // ublk queue tears its io_uring down (kernel-cancelling our recv) before the disk that owns us is dropped. @@ -127,19 +138,31 @@ class craft_async_conn { // next op's bytes land BETWEEN the chunks -- corrupting the stream so the peer resets the connection. This // gates only sending; reply-waits still overlap freely (they demux by request_id). Acquire the slot in // await_ready when free, else park; release_send() hands it to the next waiter. + // + // Ownership is explicit: co_await yields true only to an op that now OWNS the slot (taken free, or handed to it + // by release_send), together with the slot's epoch. fail_all() frees the slot by force -- its owner may still have + // a send in flight on the dead socket -- and advances the epoch, so that owner's late release_send(epoch) is a + // no-op instead of releasing a slot a newer frame owns by then (two frames would then interleave). Ops fail_all + // wakes up from the queue resume with false: they own nothing and must not send. + struct send_waiter { + std::coroutine_handle<> h; + bool* granted; + }; struct send_awaitable { craft_async_conn* c; - bool await_ready() const noexcept { + bool granted{false}; + bool await_ready() noexcept { if (!c->sending_) { c->sending_ = true; + granted = true; return true; } return false; } - void await_suspend(std::coroutine_handle<> h) noexcept { c->send_waiters_.push_back(h); } - void await_resume() const noexcept {} + void await_suspend(std::coroutine_handle<> h) noexcept { c->send_waiters_.push_back({h, &granted}); } + bool await_resume() const noexcept { return granted; } }; - void release_send() noexcept; // hand the send slot to the next parked op, or free it + void release_send(uint64_t epoch) noexcept; // hand the slot to the next parked op, or free it (if still ours) // Frame `op_hdr` + `body` as `o` at a fresh request_id, send it on the ring, and await the reply via the // pump. Returns the raw reply message bytes (the caller decodes the typed rsp), or a net_error. @@ -174,12 +197,15 @@ class craft_async_conn { wire::status helo_rejection_{wire::status::ok}; // the last attempt's HELO refusal, ok if none uint64_t helo_rejected_term_{0}; int fd_{-1}; + uint64_t conn_gen_{0}; // bumped for every new socket: a stale op's teardown must not close a newer one + send_fault_fn send_fault_; // test seam only (set_send_fault_for_test) bool ready_{false}; bool stopping_{false}; bool connecting_{false}; std::vector< std::coroutine_handle<> > connect_waiters_; // ops parked while the first connects bool sending_{false}; - std::vector< std::coroutine_handle<> > send_waiters_; // ops parked while another holds the send slot + std::vector< send_waiter > send_waiters_; // ops parked while another holds the send slot + uint64_t send_epoch_{0}; // advanced by fail_all(): a release from before it is stale std::vector< uint8_t > rx_; // recv accumulation across message boundaries diff --git a/src/read_route_map.hpp b/src/read_route_map.hpp index a52a883..4cfa372 100644 --- a/src/read_route_map.hpp +++ b/src/read_route_map.hpp @@ -134,7 +134,6 @@ class read_route_map { for (std::size_t i = 0; i < n; ++i) { synced_[i].store(k_unsynced, std::memory_order_relaxed); // raised per-member from its real commit_lsn ka_inflight_[i].store(false, std::memory_order_relaxed); - rr_inflight_[i].store(false, std::memory_order_relaxed); } all_committed_.store(k_unsynced, std::memory_order_relaxed); // min synced_; rises as members report in { @@ -329,12 +328,6 @@ class read_route_map { bool try_begin_keepalive(std::size_t idx) { return !ka_inflight_[idx].exchange(true, std::memory_order_acq_rel); } void end_keepalive(std::size_t idx) { ka_inflight_[idx].store(false, std::memory_order_release); } - // The identical collapse for the client-requested resolution round: at most ONE outstanding - // request_resolution per member. A burst of failed in-flight writes tops a peer up again only once its - // previous request completed, never one request per failure (the want watermark in dlsn_tracker is what - // accumulates the burst). - bool try_begin_resolution(std::size_t idx) { return !rr_inflight_[idx].exchange(true, std::memory_order_acq_rel); } - void end_resolution(std::size_t idx) { rr_inflight_[idx].store(false, std::memory_order_release); } // Observability / tests. int64_t folded() const { return folded_.load(std::memory_order_acquire); } @@ -387,8 +380,6 @@ class read_route_map { synced_{}; // per member: client's model of its commit_lsn (>= L) std::array< std::atomic< bool >, k_max_members > ka_inflight_{}; // per member: a keep_alive is outstanding (the one-per-leg collapse) - std::array< std::atomic< bool >, k_max_members > - rr_inflight_{}; // per member: a request_resolution is outstanding (same collapse) std::mutex pending_mu_; // guards pending_ std::vector< pending_miss > pending_; // durable non-universal completed slots awaiting fold diff --git a/test/test_auth_handshake.cpp b/test/test_auth_handshake.cpp index 1c99fbc..aa75a63 100644 --- a/test/test_auth_handshake.cpp +++ b/test/test_auth_handshake.cpp @@ -262,6 +262,13 @@ std::expected< void, net::net_error > ready_on_ring(replica_script rs, std::opti return r; } +using ka_result = std::optional< std::expected< net::lsn_reply, net::net_error > >; +// Named wrapper, not a capturing lambda: its frame must outlive the first suspension. +async_status issue_conn_keep_alive(net::craft_async_conn* c, std::shared_ptr< ka_result > out) { + *out = co_await c->keep_alive(/*commit_lsn=*/-1, /*all_committed_lsn=*/-1); + co_return sisl::ok(); +} + async_status issue_keep_alive(CraftTcpReplica* p, ::io_uring* q, client_hdr hdr, std::shared_ptr< std::optional< result< lsn_pair > > > out) { *out = co_await p->keep_alive(q, hdr); @@ -573,3 +580,158 @@ TEST(CraftAuthHandshake, OnRingRetryAfterHandshakeFailure) { } replica.join(); } + +// ── send faults on the ring connection (review on craft_client #6) ── + +// A send that fails after part of its frame is on the socket drops the connection: the fragment can never be +// followed by the next message's bytes. The op queued behind it fails too (on the dead socket), nothing is left +// pending, and the next op reconnects and goes through. +TEST(CraftAuthHandshake, OnRingSendFailureMidFrameDropsTheConnection) { + auto rs = replica_script{}; + rs.enforce = false; + rs.connections = 2; + scripted_replica replica{rs}; + { + ring_session ring{replica.port(), std::nullopt}; + ASSERT_TRUE(ring.ready().has_value()); + + int calls = 0; + ring.conn->set_send_fault_for_test([&calls](std::size_t, std::size_t) -> long { + ++calls; + if (calls == 1) return 10; // the first frame: only 10 bytes go out... + if (calls == 2) return -EIO; // ...then its send fails + return 0; + }); + auto a = std::make_shared< ka_result >(); + auto b = std::make_shared< ka_result >(); + issue_conn_keep_alive(ring.conn.get(), a).detach(); // takes the send slot + issue_conn_keep_alive(ring.conn.get(), b).detach(); // queued behind it + ASSERT_TRUE(ring.driver.drive_until([&] { return a->has_value() && b->has_value(); })); + ASSERT_FALSE(a->value().has_value()); + EXPECT_EQ(a->value().error(), net::net_error::send); + ASSERT_FALSE(b->value().has_value()) << "the queued op must not follow the fragment"; + EXPECT_FALSE(ring.conn->ready()); + + ring.conn->set_send_fault_for_test(nullptr); + ASSERT_TRUE(ring.ready().has_value()) << "the next op reconnects"; + auto c = std::make_shared< ka_result >(); + issue_conn_keep_alive(ring.conn.get(), c).detach(); + ASSERT_TRUE(ring.driver.drive_until([&] { return c->has_value(); })); + EXPECT_TRUE(c->value().has_value()) << "the new connection carries traffic"; + ring.driver.drain_for(std::chrono::milliseconds{50}); + } + replica.join(); + EXPECT_EQ(replica.accepted(), 2); +} + +// A send failure that belongs to an older connection (a reconnect opened a newer one meanwhile) must not close the +// newer socket: the connection stays usable and no reconnect happens. +TEST(CraftAuthHandshake, OnRingStaleSendFailureLeavesTheNewConnection) { + auto rs = replica_script{}; + rs.enforce = false; + rs.connections = 2; + scripted_replica replica{rs}; + { + ring_session ring{replica.port(), std::nullopt}; + ASSERT_TRUE(ring.ready().has_value()); + + auto* conn = ring.conn.get(); + ring.conn->set_send_fault_for_test([conn](std::size_t, std::size_t) -> long { + conn->bump_conn_gen_for_test(); // as if a reconnect had replaced the socket this send was issued on + return -EIO; + }); + auto a = std::make_shared< ka_result >(); + issue_conn_keep_alive(ring.conn.get(), a).detach(); + ASSERT_TRUE(ring.driver.drive_until([&] { return a->has_value(); })); + ASSERT_FALSE(a->value().has_value()); + EXPECT_EQ(a->value().error(), net::net_error::send); + EXPECT_TRUE(ring.conn->ready()) << "the stale failure left the current socket alone"; + + ring.conn->set_send_fault_for_test(nullptr); + auto b = std::make_shared< ka_result >(); + issue_conn_keep_alive(ring.conn.get(), b).detach(); + ASSERT_TRUE(ring.driver.drive_until([&] { return b->has_value(); })); + EXPECT_TRUE(b->value().has_value()); + ring.driver.drain_for(std::chrono::milliseconds{50}); + } + replica.join(); + EXPECT_EQ(replica.accepted(), 1) << "no reconnect"; +} + +// The send slot survives a teardown it raced with: frame A is mid-send when a recv fault fails everything +// (fail_all frees the slot) and a reconnect opens a newer socket; frame B takes the free slot and starts sending. +// A's send then fails late. A's release must NOT free B's slot -- a third frame C would otherwise start sending +// while B is still on the socket, interleaving the two (review on craft_client #6). +TEST(CraftAuthHandshake, OnRingStaleSendCannotReleaseANewerFramesSlot) { + auto rs = replica_script{}; + rs.enforce = false; + rs.connections = 2; + scripted_replica replica{rs}; + { + ring_session ring{replica.port(), std::nullopt}; + ASSERT_TRUE(ring.ready().has_value()); + + auto* conn = ring.conn.get(); + auto b = std::make_shared< ka_result >(); + int calls = 0; + ring.conn->set_send_fault_for_test([&, conn, b](std::size_t, std::size_t) -> long { + if (++calls != 1) return 0; // B and C send normally + // A's first send: the connection is torn down and replaced under it, and B takes the freed slot. + conn->fail_all_for_test(); + conn->bump_conn_gen_for_test(); + issue_conn_keep_alive(conn, b).detach(); // acquires the slot, submits its send (call 2), suspends + return -EIO; // ...and A's send fails late + }); + auto a = std::make_shared< ka_result >(); + issue_conn_keep_alive(ring.conn.get(), a).detach(); + ASSERT_TRUE(a->has_value()); + ASSERT_FALSE(a->value().has_value()); + EXPECT_EQ(a->value().error(), net::net_error::send); + ASSERT_EQ(calls, 2); + + auto c = std::make_shared< ka_result >(); + issue_conn_keep_alive(ring.conn.get(), c).detach(); + EXPECT_EQ(calls, 2) << "C must wait for B's slot, not send next to it"; + + ASSERT_TRUE(ring.driver.drive_until([&] { return b->has_value() && c->has_value(); })); + EXPECT_TRUE(b->value().has_value()); + EXPECT_TRUE(c->value().has_value()); + EXPECT_EQ(calls, 3); + ring.driver.drain_for(std::chrono::milliseconds{50}); + } + replica.join(); +} + +// An op queued for the send slot when the connection is torn down never sends anything: it fails with `send` +// (provably never delivered), not `closed` (delivery uncertain) -- callers retire a never-delivered write's slot +// instead of pinning it until the next login (review on craft_client #6). +TEST(CraftAuthHandshake, OnRingQueuedOpFailsAsNeverSentOnTeardown) { + auto rs = replica_script{}; + rs.enforce = false; + rs.connections = 2; + scripted_replica replica{rs}; + { + ring_session ring{replica.port(), std::nullopt}; + ASSERT_TRUE(ring.ready().has_value()); + + auto* conn = ring.conn.get(); + auto b = std::make_shared< ka_result >(); + int calls = 0; + ring.conn->set_send_fault_for_test([&, conn, b](std::size_t, std::size_t) -> long { + if (++calls == 1) { + issue_conn_keep_alive(conn, b).detach(); // queues behind A + conn->fail_all_for_test(); // teardown while B waits for the slot + } + return 0; + }); + auto a = std::make_shared< ka_result >(); + issue_conn_keep_alive(ring.conn.get(), a).detach(); + ASSERT_TRUE(b->has_value()); + ASSERT_FALSE(b->value().has_value()); + EXPECT_EQ(b->value().error(), net::net_error::send); + EXPECT_EQ(calls, 1) << "B never reached the socket"; + ASSERT_TRUE(ring.driver.drive_until([&] { return a->has_value(); })); + ring.driver.drain_for(std::chrono::milliseconds{50}); + } + replica.join(); +} diff --git a/test/test_client.cpp b/test/test_client.cpp index 394b559..ccd137c 100644 --- a/test/test_client.cpp +++ b/test/test_client.cpp @@ -85,24 +85,21 @@ std::vector< uint8_t > rd(craft::craft_client& c, uint64_t off_blk, uint64_t nbl return dest; } -// Only the leader accepts the write, so it journals the slot while quorum(3)=2 goes unmet: the slot fails -- -// and the failure fires the client-requested resolution round, which (leader up, quorum live) FILLS the slot -// from the leader's copy and retires it: the failed write completes late, durable everywhere. Use this to -// exercise the round; use write_subquorum_unhealed for the unresolved WINDOW. +// Only the leader accepts the write, so it journals the slot while quorum(3)=2 goes unmet: the write fails and +// the slot stays unresolved for the rest of the session (failed writes are resolved at the next login). void write_subquorum(Cluster& cl, uint64_t off_blk, std::vector< uint8_t >& buf) { cl.set.net->force_subquorum({craft::mem_replica_id(cl.vid, 0)}); EXPECT_FALSE(wr(*cl.client, off_blk, buf)); cl.set.net->clear_faults(); } -// Fail a write sub-quorum WITHOUT letting the resolution round heal it: the LEADER is down for the write, so -// the round (leader-only) dies with REPLICA_DOWN and the slot stays unresolved -- the fencing window the -// horizon tests pin. Replica 1 is the surviving sub-quorum holder. +// Fail a write sub-quorum with the LEADER down for it, so replica 1 is the only holder -- the unresolved +// window the horizon tests pin. void write_subquorum_unhealed(Cluster& cl, uint64_t off_blk, std::vector< uint8_t >& buf) { cl.set.replicas[0]->set_up(false); cl.set.net->force_subquorum({craft::mem_replica_id(cl.vid, 1)}); EXPECT_FALSE(wr(*cl.client, off_blk, buf)); - cl.set.net->clear_faults(); // everyone healthy again -- but nothing re-fires the dead round + cl.set.net->clear_faults(); // everyone healthy again -- the slot stays unresolved until the next login } } // namespace @@ -245,44 +242,109 @@ TEST(CraftClient, N3_SplitReadServesEachBlockAtItsOwnHorizon) { EXPECT_EQ(cl.client->commit_lsn(), 0) << "the failed write still pins commit"; } -// --- the client-requested resolution round: a failed write is resolved NOW, not at the next login --- +// --- failed writes: resolved at the next login, never during the session (HomeBlocks docs/craft/resync.md §4.4) --- -// The round FILLS the slot: the leader holds the sub-quorum copy, so the round fetches it everywhere and the -// failed write completes late (the design's benign false-include). The frontier releases immediately -- the -// round runs synchronously in the in-process model -- and the value becomes the durable version of the block. -TEST(CraftClient, N3_ResolutionRoundFillsAFailedWriteAndReleasesTheFrontier) { +namespace { +// A fresh client logging into the same replicas (the "next attach"). +std::shared_ptr< craft::craft_client > relogin(Cluster& cl, uint64_t token) { + std::vector< std::shared_ptr< craft::craft_replica > > backends(cl.set.replicas.begin(), cl.set.replicas.end()); + auto fresh = std::make_shared< craft::craft_client >(std::move(backends)); + EXPECT_TRUE(rg(fresh->login(token)).has_value()); + return fresh; +} +} // namespace + +// A failed write is NOT resolved during the session: no resolution request is sent, the slot pins the +// frontier, the other members never receive it, and overlapping reads are still served (old data). +TEST(CraftClient, N3_FailedWriteIsNotResolvedDuringTheSession) { auto cl = make_cluster(3); auto v0 = page_of(0x40); auto v1 = page_of(0x41); ASSERT_TRUE(wr(*cl.client, 4, v0)); // dLSN 0 - EXPECT_EQ(cl.client->commit_lsn(), 0); - write_subquorum(cl, 4, v1); // dLSN 1 fails; the round fires, fills from the leader, retires the slot + write_subquorum(cl, 4, v1); // dLSN 1 fails: only the leader journals it - EXPECT_EQ(cl.client->commit_lsn(), 1) << "the round resolved the slot; nothing pins the frontier"; - EXPECT_EQ(rd(*cl.client, 4), v1) << "the failed write completed late: it IS the durable version now"; + EXPECT_EQ(cl.client->commit_lsn(), 0) << "the failed write pins the frontier for the rest of the session"; + EXPECT_EQ(rd(*cl.client, 4), v0) << "an overlapping read is served as of just before the failed write"; + EXPECT_EQ(cl.set.replicas[1]->stats().journal_slots, 1u) << "no mid-session round copied the slot"; + EXPECT_EQ(cl.set.replicas[2]->stats().journal_slots, 1u); +} - // Every live member ends up holding the filled slot (the model's round pushes synchronously). +// The next login resolves it: a live member holds the slot, so it is filled everywhere -- the failed write +// completed late (the design's benign false-include) and the frontier passes it. +TEST(CraftClient, N3_NextLoginFillsAFailedWriteHeldByALiveMember) { + auto cl = make_cluster(3); + auto v0 = page_of(0x40); + auto v1 = page_of(0x41); + ASSERT_TRUE(wr(*cl.client, 4, v0)); + write_subquorum(cl, 4, v1); + + auto fresh = relogin(cl, TOKEN + 1); + EXPECT_EQ(fresh->commit_lsn(), 1) << "login resolved slot 1"; + EXPECT_EQ(rd(*fresh, 4), v1) << "the failed write completed late: it is the durable version now"; for (auto& r : cl.set.replicas) - EXPECT_EQ(r->stats().journal_slots, 2u); + EXPECT_EQ(r->stats().journal_slots, 2u) << "every live member holds the filled slot"; } -// The round declares EMPTY: nobody holds the slot (the one copy is stuck in a 10s-delayed transport), so the -// leader verdicts it Empty on quorum-lacks evidence and tombstones it -- the late arrival is then REJECTED -// (reconciliation: Empty beats data), so no replica can resurrect a voided write. -TEST(CraftClient, N3_ResolutionRoundVerdictsEmptyWhenNobodyHoldsTheSlot) { +// When no live member holds the failed write (its only copy is stuck in a 1 s-delayed transport), no replica's +// last_append covers it, so the next login's starting point is below it: the write simply never happened, and +// the new session reuses the dLSN. The late arrival carries the old term and is fenced out -- the test waits for +// it to land and checks the reused slot afterwards. +TEST(CraftClient, N3_NextLoginDropsAFailedWriteNobodyHolds) { auto cl = make_cluster(3); cl.set.net->set_op_timeout(std::chrono::milliseconds{20}); - cl.set.net->set_delay(craft::mem_replica_id(cl.vid, 0), std::chrono::seconds{10}); // stuck, lands late - cl.set.net->force_subquorum({craft::mem_replica_id(cl.vid, 0)}); // 1 and 2 refuse outright + cl.set.net->set_delay(craft::mem_replica_id(cl.vid, 0), std::chrono::seconds{1}); // stuck, lands late + cl.set.net->force_subquorum({craft::mem_replica_id(cl.vid, 0)}); // 1 and 2 refuse outright auto v = page_of(0x50); EXPECT_FALSE(wr(*cl.client, 7, v)); // leg 0 times out, legs 1-2 refuse -> failed (not provably absent) cl.set.net->clear_faults(); + EXPECT_EQ(cl.client->commit_lsn(), -1) << "unresolved during the session"; + + auto fresh = relogin(cl, TOKEN + 1); + EXPECT_EQ(fresh->commit_lsn(), -1) << "nobody holds dLSN 0, so the login starting point is below it"; + EXPECT_EQ(rd(*fresh, 7), page_of(0)) << "the failed write reads as never-written"; + auto w = page_of(0x51); + ASSERT_TRUE(wr(*fresh, 7, w)) << "the new session reuses dLSN 0"; + EXPECT_EQ(rd(*fresh, 7), w); + + // Wait for the old write's delayed delivery to replica 0, then check it changed nothing (review on #6). + auto const deadline = std::chrono::steady_clock::now() + std::chrono::seconds{5}; + while (cl.set.replicas[0]->stats().stale_term_writes == 0 && std::chrono::steady_clock::now() < deadline) + std::this_thread::sleep_for(std::chrono::milliseconds{10}); + ASSERT_EQ(cl.set.replicas[0]->stats().stale_term_writes, 1u) << "the late old-session write was delivered"; + EXPECT_EQ(rd(*fresh, 7), w) << "and rejected: the reused slot keeps the new session's data"; + for (auto& r : cl.set.replicas) + EXPECT_EQ(r->stats().journal_slots, 1u); +} - EXPECT_EQ(cl.client->commit_lsn(), 0) << "the round verdicted dLSN 0 Empty and the frontier passed it"; - EXPECT_EQ(rd(*cl.client, 7), page_of(0)) << "a voided write reads as never-written"; - EXPECT_GE(cl.set.replicas[1]->stats().empty_slots, 1u) << "the verdict tombstone is journaled"; +// An old-session write that reaches the members after the login polled them but before it resolves any slot is +// rejected: the fence comes BEFORE the poll. Were it moved after resolution, this write could land in a slot the +// scan saw absent, be acked, and then be voided by an Empty verdict (review on #6). +TEST(CraftClient, N3_OldSessionWriteBetweenPollAndResolutionIsRejected) { + auto cl = make_cluster(3); + auto v = page_of(0x60); + ASSERT_TRUE(wr(*cl.client, 1, v)); // dLSN 0 + uint64_t const old_term = cl.client->term(); + + auto late = page_of(0x61); + int rejected = 0; + cl.set.net->set_login_poll_hook_for_test([&] { + for (auto& r : cl.set.replicas) { + auto res = rg(r->write(nullptr, client_hdr{.term = old_term, .commit_lsn = 0, .all_committed_lsn = 0}, + /*dlsn=*/1, blk(2), PAGE, one_iov(late))); + if (!res && res.error() == make_error_condition(craft_error::STALE_TERM)) ++rejected; + } + }); + auto fresh = relogin(cl, TOKEN + 1); + cl.set.net->set_login_poll_hook_for_test(nullptr); + + EXPECT_EQ(rejected, 3); + for (auto& r : cl.set.replicas) + EXPECT_EQ(r->stats().stale_term_writes, 1u); + EXPECT_EQ(fresh->commit_lsn(), 0); + EXPECT_EQ(rd(*fresh, 2), page_of(0)) << "the late write never landed"; + EXPECT_EQ(rd(*fresh, 1), v); } // --- session establishment --- @@ -737,8 +799,12 @@ TEST(CraftClient, ReLoginMustNotServeStaleFollower) { // Re-login drops the Missing map; member 2 is now gated only by Synced, which the client seeds from member // 2's own commit_lsn (< L, since it never applied dLSN 1) -- so it stays ineligible despite still holding 0xAA. + // Member 2 is DOWN during the login: a live member would be healed by the login's resolution step (filled + // from a holder), and this test is about a member that is still behind afterwards. ASSERT_TRUE(rg(cl.client->logout()).has_value()); + cl.set.replicas[2]->set_up(false); ASSERT_TRUE(rg(cl.client->login(TOKEN)).has_value()); + cl.set.replicas[2]->set_up(true); EXPECT_FALSE(cl.client->route_caught_up(2)) << "member 2 still behind after re-login -- must not be trusted"; // Every read must return the committed 0xBB. Read enough times that the round-robin rotor would visit all diff --git a/test/test_client_tcp.cpp b/test/test_client_tcp.cpp index 9be72c7..4ba84c7 100644 --- a/test/test_client_tcp.cpp +++ b/test/test_client_tcp.cpp @@ -146,25 +146,21 @@ TEST(CraftClientTcp, N3_BelowQuorumFails) { EXPECT_FALSE(wr(*tc.client, 6, buf)); } -// A failed write fires the client-requested resolution round over the wire (RESOLVE to the leader): the -// leader fills the slot from its own copy set-wide, the client retires it off the reply, the frontier -// releases, and the failed write completes late as the durable version. -TEST(CraftClientTcp, N3_FailedWriteResolutionRoundHealsOverTcp) { +// A failed write is NOT resolved during the session over the wire either: the client sends no RESOLVE, the slot +// pins the frontier, the other replicas never receive it, and an overlapping read is served as of just before it. +// The next login resolves it (covered by the in-memory tests in test_client.cpp). +TEST(CraftClientTcp, N3_FailedWriteStaysUnresolvedDuringSessionOverTcp) { auto tc = make_tcp_cluster(3); auto v0 = page_of(0x60); auto v1 = page_of(0x61); ASSERT_TRUE(wr(*tc.client, 4, v0)); // dLSN 0: durable everywhere tc.set.server->force_subquorum({0}); - EXPECT_FALSE(wr(*tc.client, 4, v1)); // dLSN 1 fails; the detached RESOLVE round fires at the leader + EXPECT_FALSE(wr(*tc.client, 4, v1)); // dLSN 1 fails: only the leader journals it tc.set.server->clear_faults(); - // The round is asynchronous over TCP (it rides the leader proxy's worker): wait for the retire. - auto const deadline = std::chrono::steady_clock::now() + std::chrono::seconds{5}; - while (tc.client->commit_lsn() < 1 && std::chrono::steady_clock::now() < deadline) - std::this_thread::sleep_for(std::chrono::milliseconds{2}); - EXPECT_EQ(tc.client->commit_lsn(), 1) << "the round filled and retired the failed slot"; - EXPECT_EQ(rd(*tc.client, 4), v1) << "the failed write completed late: it is the durable version"; - EXPECT_EQ(tc.set.server->journal_slots(1), 2u) << "the fill landed set-wide"; + EXPECT_EQ(tc.client->commit_lsn(), 0) << "the failed write pins the frontier for the session"; + EXPECT_EQ(rd(*tc.client, 4), v0) << "an overlapping read is served as of just before the failed write"; + EXPECT_EQ(tc.set.server->journal_slots(1), 1u) << "no mid-session round copied the slot"; } // Two replicas taken fully down fail their reads with REPLICA_DOWN over the wire; the client routes the read