Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 9 additions & 18 deletions docs/client-internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,22 +115,13 @@ holds the slot, it is a permanent no-op, and the frontier may pass over it. A su
degrades to `failed`. Counting a timeout here would advance `F` past a dLSN a peer later applies, which is
divergence. In every other respect all non-acks are equal.

A `failed` outcome no longer waits for the next login: it fires the **client-requested resolution round**
(`craft_replica::request_resolution`, the design's client-request `SyncRSCommitLSN` trigger). The client
cannot know who the leader is mid-session (it learned one at login; leadership may have moved), so the
request is **broadcast to every member** as detached fire-and-forget legs, with at most **one outstanding per
peer** -- exactly the keep_alive collapse (`rr_inflight_` beside `ka_inflight_`), so a burst of failed
in-flight writes collapses into the pending legs rather than one request per failure. The failed dLSNs
CAS-max into a single *want* watermark (`dlsn_tracker`); whichever member is the current leader resolves
every unresolved slot <= the want -- fills it from a holder (the failed write completes late) or verdicts it
Empty on quorum-lacks evidence -- while non-leaders answer `NOT_LEADER` and leave the want untouched (only a
successful round consumes it). The client retires the covered slots off the reply
(`dlsn_tracker::retire_upto`): Empty verdicts resolve `empty`, filled slots resolve `acked` with the router
told first (`note_filled`: the slot becomes verdict-durable and reads route only to its certain holders until
the commit certificates catch up). A replica REJECTS a late arrival into an Empty-verdicted slot, so a write
the round voided fails deterministically on its own ack path -- the verdict and the client's local `Empty`
conclusion agree. If no leg can resolve (leader down), the slot simply stays pinned and the next failed write
re-fires the legs.
A `failed` outcome is **resolved at the next login, not during the session**. The application gets the error (in
production XFS then goes offline and the volume is detached), the failed slot pins the frontier `F`, and reads that
overlap it keep being served as of just before it (see *Why per-block*). The client sends no resolution request.
The next login decides the slot under the login fence: if a live member holds it, it is filled everywhere (the
write completed late); if nobody holds it, the login's starting point is below it and the dLSN is simply reused.
The `request_resolution` verb and the wire `resolve` op still exist for compatibility but are no longer used by
the client. Background: HomeBlocks `docs/craft/resync.md` §4.4-§4.5 (mid-session verdicts raced in-flight acks).

### Where a write actually goes

Expand Down Expand Up @@ -396,8 +387,8 @@ Also outstanding, in rough order:

- the server-side **watchdog** the keep_alive drive is meant to reset (a deferred seam today,
`mem_craft_replica.cpp`), and server-side **resync** (`SyncRSCommitLSN`) that makes the recovery erase above
effective -- both are server/peer concerns, not client work (the client-REQUESTED resolution round and its
retire path now exist: `request_resolution` / `retire_upto`, see *A non-ack is not one thing*);
effective -- both are server/peer concerns, not client work (failed writes are resolved at the next login,
see *A non-ack is not one thing*);
- re-login on `STALE_TERM`;
- the client-side all-zero scan that picks the CRAFT zero-write form (the server does not re-scan a data
write), plus ublk `DISCARD` / `WRITE_ZEROES` → a CRAFT zero write (empty `sg_list`);
Expand Down
14 changes: 14 additions & 0 deletions docs/resync.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# CRAFT resync

The resync design lives in HomeBlocks: [`docs/craft/resync.md`](https://github.com/eBay/HomeBlocks/blob/dev/v6.x/docs/craft/resync.md).
Resync is driven by the CRAFT BM side; this package (UBD's CRAFT client) does not drive it.

What the design asks of this repo (PR F there):

- **Failed writes are resolved at the next login, not during the session.** On a sub-quorum write the client
reports the error, pins its frontier `F` below the slot, and keeps serving overlapping reads as of just before
it. It no longer calls `request_resolution`, and the `resolve` op (13/14) is retired.
- **The transport must keep "never delivered" true.** `craft_async_conn` drops the connection on any send failure,
so a half-sent frame can never be followed by the next message (D17 in the design's Appendix A).
- **Peer-plane codec v2** (`include/craft/peer.hpp`): range-based, byte-bounded `fetch_data` with continuation and
a `reclaimed_below` status, plus term and partition fields on peer requests (PR E).
49 changes: 4 additions & 45 deletions src/client.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -229,54 +229,13 @@ async_result< size_t > craft_client::write(::io_uring* q, uint64_t addr, uint64_
bool const provably_empty = (refused == replicas_.size());
tracker_->resolve(dlsn, provably_empty ? slot_outcome::empty : slot_outcome::failed);
maybe_fold();
// A failed (sub-quorum, not provably-absent) slot pins the frontier until the leader fills or Empties it:
// request the resolution round NOW (the design's client-request SyncRSCommitLSN trigger) instead of
// waiting for a watchdog / periodic cadence that the client cannot see.
if (!provably_empty) request_resolution_round(q, dlsn);
// A failed (sub-quorum, not provably-absent) slot pins the frontier. It is NOT resolved during the session:
// the application gets the error (XFS then goes offline and the volume detaches), and the next login
// resolves the slot under the login fence -- filled from a holder or declared Empty (HomeBlocks
// docs/craft/resync.md §4.4). Overlapping reads keep being served as of just before the slot meanwhile.
co_return std::unexpected(last_err ? last_err : make_error_condition(craft_error::NO_QUORUM));
}

// One detached resolution leg to member `idx` -- fire-and-forget, like fire_keepalive: it captures the
// tracker/route shared_ptrs and the backend by value, never the client. It drains the want watermark: a burst
// of failed writes collapses into this one outstanding request per peer, and a want that lands while the
// leader is resolving is picked up by the next loop pass. A member that is NOT the leader answers NOT_LEADER
// and the leg simply ends, LEAVING the want set -- only the leader's leg (whichever member that currently is)
// resolves and consumes it. On success the covered slots are retired off the verdicts: Empty ones as Empty,
// filled ones as acked with the router told first (note_filled) so reads route only to certain holders until
// the commit_lsn certificates catch up. A leg that errors (member down, deposed term) also just ends: the
// slot stays pinned and the next failed write re-fires the legs.
static async_status fire_resolution(::io_uring* q, std::shared_ptr< craft_replica > h, std::size_t idx, uint64_t term,
std::shared_ptr< dlsn_tracker > tracker, std::shared_ptr< read_route_map > route) {
for (;;) {
while (auto const want = tracker->resolution_want()) {
client_hdr const hdr{term, tracker->frontier(), route->all_committed()};
auto r = co_await h->request_resolution(q, hdr, *want);
if (!r.has_value()) { // NOT_LEADER / down / stale: this peer cannot resolve; leave the want alone
route->end_resolution(idx);
co_return ok();
}
tracker->retire_upto(r->resolved_upto, r->empty_slots, [&](int64_t d) { route->note_filled(d, idx); });
tracker->clear_resolution_want(r->resolved_upto);
}
route->end_resolution(idx);
// Close the note-then-begin race: a want recorded between our last peek and the flag release would
// strand until the next failure. Reclaim this peer's flag only if a want is actually pending.
if (!tracker->resolution_want() || !route->try_begin_resolution(idx)) co_return ok();
}
}

void craft_client::request_resolution_round(::io_uring* q, int64_t upto) {
tracker_->note_resolution_want(upto);
// BROADCAST, not a leader walk: the client learned the leader at login and leadership may have moved
// since -- mid-session it cannot know who leads. Every member gets the request, at most one outstanding
// per peer (the keep_alive collapse); whichever member IS the leader runs the round, the rest answer
// NOT_LEADER (a real replica may instead forward to its leader -- either way the client need not know).
// Each leg rides the failing write's ring, so its resumptions stay on that queue's reap thread.
for (std::size_t m = 0; m < replicas_.size(); ++m) {
if (route_->try_begin_resolution(m)) fire_resolution(q, replicas_[m], m, term_, tracker_, route_).detach();
}
}

// Issue a whole read plan to one target, filling `dest` in place; resolves to the target's piggybacked
// watermarks (the highest across a split's segments -- same member, monotonic). Factored out of read() so the
// router can retry it against the next eligible member on a transport failure. `dest` is copied per attempt
Expand Down
6 changes: 0 additions & 6 deletions src/client_impl.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -80,12 +80,6 @@ class craft_client {
std::optional< std::error_condition > precheck(uint64_t addr, uint64_t len) const;
async_result< lsn_pair > issue_plan(::io_uring* q, std::shared_ptr< craft_replica > const& target, client_hdr hdr,
read_plan const& plan, uint64_t addr, uint64_t len, sisl::sg_list& dest);
// Fire the client-requested resolution round for failed slot `upto`: record the want, then BROADCAST a
// detached request to every peer without one outstanding (per-peer single-flight, the keep_alive
// collapse) -- the client cannot know who leads mid-session, so whichever member is the leader resolves.
// Each leg captures the tracker's and router's shared_ptrs, never `this`; it rides the ring of the write
// whose failure fired it.
void request_resolution_round(::io_uring* q, int64_t upto);

std::vector< std::shared_ptr< craft_replica > > replicas_;
uint32_t leader_{0};
Expand Down
23 changes: 0 additions & 23 deletions src/dlsn_tracker.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ void dlsn_tracker::reset_at(int64_t login_dlsn, uint32_t lba_size) {
highest_acked_.store(login_dlsn, std::memory_order_release);
// A want must not leak across sessions: slot numbers are REUSED above the login watermark, so a stale
// want could void the new session's in-flight slots.
resolve_want_.store(-1, std::memory_order_relaxed);
// StreamTracker(name, start_idx) bases its window at start_idx + 1, i.e. the first dLSN we will issue.
tracker_.emplace("craft_dlsn", login_dlsn);
}
Expand Down Expand Up @@ -131,28 +130,6 @@ void dlsn_tracker::advance_frontier() {
}
}

// ── the resolution-round want watermark ──

void dlsn_tracker::note_resolution_want(int64_t upto) {
int64_t w = resolve_want_.load(std::memory_order_relaxed);
while (upto > w &&
!resolve_want_.compare_exchange_weak(w, upto, std::memory_order_release, std::memory_order_relaxed)) {}
}

std::optional< int64_t > dlsn_tracker::resolution_want() const {
int64_t const w = resolve_want_.load(std::memory_order_acquire);
if (w < 0) return std::nullopt;
return w;
}

void dlsn_tracker::clear_resolution_want(int64_t resolved_upto) {
// Consume only what the round covered: a HIGHER want that landed mid-round survives the clear and the
// leader's leg loops for it.
int64_t w = resolve_want_.load(std::memory_order_relaxed);
while (w >= 0 && w <= resolved_upto &&
!resolve_want_.compare_exchange_weak(w, -1, std::memory_order_acq_rel, std::memory_order_relaxed)) {}
}

tracker_stats dlsn_tracker::stats(std::size_t sample_limit) const {
tracker_stats s;
s.winner_scans = winner_scans_.load(std::memory_order_relaxed);
Expand Down
42 changes: 0 additions & 42 deletions src/dlsn_tracker.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -131,47 +131,6 @@ class dlsn_tracker {
// are still completing -- whichever lands second is dropped, not fought over.
void resolve(int64_t dlsn, slot_outcome oc);

// ── the client-requested resolution round (the design's client-request SyncRSCommitLSN trigger) ──
//
// The WANT watermark: the highest failed dLSN awaiting a round, hosted here so the detached per-peer
// resolution legs outlive the client safely (each captures the tracker's shared_ptr). A burst of failed
// writes CAS-maxes into one value; only a SUCCESSFUL round consumes it (clear_resolution_want) -- a
// non-leader leg answering NOT_LEADER must leave it for the leader's leg. The per-peer one-outstanding
// gate lives in read_route_map (rr_inflight_), beside the keep_alive collapse it mirrors.
void note_resolution_want(int64_t upto); // CAS-max: record a failed slot needing a round
std::optional< int64_t > resolution_want() const; // peek (never consumes); nullopt = nothing pending
void clear_resolution_want(int64_t resolved_upto); // consume iff the want is <= what the round resolved

// After a round: every slot <= upto is resolved server-side. Walk OUR unresolved slots <= upto and retire
// each: Empty if it is in `empties` (ascending), else acked (it was filled from a holder -- the failed
// write completed late). `on_filled(dlsn)` fires BEFORE the acked resolve, so the router can queue its
// conservative miss / verdict flag first (the same push-before-completed-bit ordering record_completion
// keeps). A slot whose create() is not yet visible is skipped: its own legs resolve it (they are rejected
// deterministically into a verdicted slot).
template < typename Fn >
void retire_upto(int64_t upto, std::vector< int64_t > const& empties, Fn&& on_filled) {
if (!tracker_) return;
int64_t const hi = std::min(upto, next_dlsn_.load(std::memory_order_acquire) - 1);
for (int64_t h = frontier() + 1; h <= hi;) {
int64_t const u = std::min(tracker_->completed_upto(h), hi) + 1;
if (u > hi) break;
bool active = false;
tracker_->foreach_all_active(u, [&](int64_t idx, dlsn_slot&) {
active = (idx == u);
return false;
});
if (active) {
if (std::binary_search(empties.begin(), empties.end(), u)) {
resolve(u, slot_outcome::empty);
} else {
on_filled(u);
resolve(u, slot_outcome::acked);
}
}
h = u + 1;
}
}

int64_t frontier() const { return frontier_.load(std::memory_order_acquire); }
int64_t read_horizon() const { return std::max(frontier(), highest_acked_.load(std::memory_order_acquire)); }

Expand Down Expand Up @@ -214,7 +173,6 @@ class dlsn_tracker {
std::atomic< int64_t > highest_acked_{-1}; // Ha: max quorum-acked dLSN
std::atomic< int64_t > last_trunc_{-1};
std::atomic< uint64_t > winner_scans_{0};
std::atomic< int64_t > resolve_want_{-1}; // highest failed dLSN awaiting a resolution round

std::optional< sisl::StreamTracker< dlsn_slot > > tracker_; // constructed at login
resolution_gate gate_;
Expand Down
80 changes: 50 additions & 30 deletions src/mem/cluster.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,44 @@ std::vector< MemCraftReplica* > MemTransport::live_replicas_locked() const {
return live;
}

// The leader's resolution step over the live set (a quorum), shared by login and the resolution round: every
// slot <= upto that is not yet contiguously present everywhere is filled from a live holder, or -- when the whole
// live set lacks it, or a prior Empty verdict exists -- verdicted Empty everywhere. Run by login under the
// session switch, this is where a failed (sub-quorum) write is resolved: it completes late if any live member
// holds it, and is voided otherwise (HomeBlocks docs/craft/resync.md §4.4).
void MemTransport::resolve_slots_upto(std::vector< MemCraftReplica* > const& live, int64_t upto) {
// Everything <= a member's commit_lsn is contiguously present there, so the min over the live set bounds
// the scan: below it nothing can be unresolved.
int64_t floor = std::numeric_limits< int64_t >::max();
for (auto* r : live)
floor = std::min(floor, r->peek_lsns().commit_lsn);

for (int64_t d = floor + 1; d <= upto; ++d) {
// One pass over the live set: a prior Empty verdict is authoritative (it beats a held copy -- the
// copy was never acked); else any live holder's copy fills the others; else the whole live set
// (a quorum) positively lacks the slot and it is verdicted Empty everywhere.
std::optional< MemCraftReplica::MemJournalSlot > copy;
bool verdicted = false;
for (auto* r : live) {
if (auto s = r->peek_slot(d)) {
if (s->is_empty) {
verdicted = true;
break;
}
if (!copy) copy = std::move(s);
}
}
if (verdicted || !copy) {
for (auto* r : live)
r->cold_mark_empty(d);
} else {
for (auto* r : live) {
if (!r->peek_slot(d)) r->cold_install_slot(d, *copy);
}
}
}
}

result< LoginResult > MemTransport::run_login(MemCraftReplica* caller, uint64_t client_token) {
std::vector< MemCraftReplica* > live;
std::vector< replica_endpoint > members_copy;
Expand All @@ -349,10 +387,21 @@ result< LoginResult > MemTransport::run_login(MemCraftReplica* caller, uint64_t
members_copy = members_;
}
// Drive the cold path WITHOUT holding mu_ (each replica call takes only its own lock).
// Phase 0: fence. Every live replica switches to the new term BEFORE it is polled, so from here on it rejects
// the old session's writes: none can reach quorum after the absence scan below and then be voided by it. In
// this model a write is applied atomically under the replica's lock, so there is no admitted-but-unfinished
// write to drain; a delayed delivery that arrives later carries the old term and is rejected. Nobody writes
// under the new term until this login returns, so the early switch admits no one.
for (auto* r : live)
r->cold_fence(nt);
// Phase 1: GetRSCommitLSN -> rs_commit_lsn = max(quorum.last_append_lsn).
int64_t rs = -1;
for (auto* r : live)
rs = std::max(rs, r->peek_lsns().last_append_lsn);
if (login_poll_hook_) login_poll_hook_();
// Phase 1b: resolve every unresolved slot <= rs (fill from a holder, or verdict Empty) -- failed writes of the
// previous session are decided here, not mid-session.
resolve_slots_upto(live, rs);
Comment thread
Copilot marked this conversation as resolved.
Resolved
// Phase 2: SyncRSCommitLSN(rs) applied to all live members.
for (auto* r : live)
r->cold_apply_sync(rs, client_token);
Expand All @@ -379,36 +428,7 @@ result< resolution_result > MemTransport::run_resolution(MemCraftReplica* caller
if (live.size() < quorum(members_.size())) return fail(craft_error::NO_QUORUM);
}

// Everything <= a member's commit_lsn is contiguously present there, so the min over the live set bounds
// the scan: below it nothing can be unresolved.
int64_t floor = std::numeric_limits< int64_t >::max();
for (auto* r : live)
floor = std::min(floor, r->peek_lsns().commit_lsn);

for (int64_t d = floor + 1; d <= upto; ++d) {
// One pass over the live set: a prior Empty verdict is authoritative (it beats a held copy -- the
// copy was never acked); else any live holder's copy fills the others; else the whole live set
// (a quorum) positively lacks the slot and it is verdicted Empty everywhere.
std::optional< MemCraftReplica::MemJournalSlot > copy;
bool verdicted = false;
for (auto* r : live) {
if (auto s = r->peek_slot(d)) {
if (s->is_empty) {
verdicted = true;
break;
}
if (!copy) copy = std::move(s);
}
}
if (verdicted || !copy) {
for (auto* r : live)
r->cold_mark_empty(d);
} else {
for (auto* r : live) {
if (!r->peek_slot(d)) r->cold_install_slot(d, *copy);
}
}
}
resolve_slots_upto(live, upto);
for (auto* r : live)
r->cold_apply_sync(upto, 0);

Expand Down
Loading