Repository navigation
docs/craft: resync design - #198
Ilyin-Ilya wants to merge 14 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved gaps in login fencing, Sync handling, and snapshot recovery can truncate committed data or incorrectly classify writes as Empty.
9 open issues
Login can truncate below a committed resolved watermark · New Do not treat reclaimed journal gaps as quorum-lacks evidence · New Filter Empty verdicts after failed fetches adjust N · New Aborted login term can be allocated again · New Drain in-flight writes before reporting probe absences · New Retention cap must preserve durability and snapshot-pin bounds · New Reconcile compacted login entries before admitting replica I/O · New Reject sources whose checkpoint is below the advertised commit point · New Renew source lease during long tail transfers · New
What changed in this PR
Adds the resync design reference for CraftReplDev, guiding implementation PRs #196, #197, and follow-up work.
Changes:
- Documents recovery cases, safety invariants, implementation gaps, and a staged test and delivery plan.
- Defines BM-driven healing, login fencing, and peer-to-peer snapshot recovery.
- Links the design from the CRAFT index and bumps the package version for the repository’s version check.
| File | Description |
|---|---|
docs/craft/resync.md |
Adds the resync design, protocol contracts, diagrams, risks, and implementation plan. |
docs/craft/README.md |
Adds the design to the documentation index. |
conanfile.py |
Bumps the package version from 6.0.17 to 6.0.18. |
Review effort: Balanced
A fenced replica waits for every write it admitted before the fence to finish before reporting its LSNs. Such writes have already advanced last_append_lsn and marked their slot missing; reporting earlier would let the leader's final round count them as lacking and verdict them Empty while they are still being journalled (review on #198). Test: LoginPollWaitsForInFlightWritesBeforeReporting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved recovery, fencing, and snapshot-installation issues can stall catch-up, re-admit fenced sessions, and expose reads to data newer than their requested horizon.
2 open issues
9 resolved since last review
Renew source lease during long tail transfers Reject sources whose checkpoint is below the advertised commit point Reconcile compacted login entries before admitting replica I/O Retention cap must preserve durability and snapshot-pin bounds Drain in-flight writes before reporting probe absences Aborted login term can be allocated again Filter Empty verdicts after failed fetches adjust N Do not treat reclaimed journal gaps as quorum-lacks evidence Login can truncate below a committed resolved watermark
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Unresolved critical protocol gaps can truncate acknowledged writes or expose inconsistent reads through incomplete login recovery, snapshot fence regression, unsafe membership cutovers, and unfenced snapshot replacement.
No open issues 🎉
2 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because findings omitted from the published review raise interdependent concerns about session fencing, snapshot read consistency, and recovery completeness.
No open issues 🎉
Review effort: Balanced
Case-by-case resync design for CraftReplDev with sequence diagrams: replica misses a write, healing a lagging replica, failed writes resolved at login, login with pre-sync and a fence lifted only by a committed outcome, login abort, snapshot past retention (data streamed BM-to-BM, nuraft carries control state only), new member, snapshot source failure, UBD disappearing. Includes the invariants, the as-built state of CraftReplDev on dev/v6.x with defects H1-H8, a design-to-code map, failure walk-throughs, alternatives, the test plan, the HomeBlocks PR sequence (#196, #197, ...), gaps and risks, and open questions. Linked from docs/craft/README.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- I6: login's rs_commit_lsn is bounded below by the last committed Sync watermark; Sync apply keeps raising last_append_lsn to N. - Leader round: a reclaimed gap is never lacks-evidence; Empty verdicts are filtered after every fetch failure has lowered N. - Login term is strictly above session, fence and aborted terms; aborted_upto is persisted, probed and snapshotted. - A fenced poll drains already-admitted writes before reporting. - The retention cap relaxes only all_committed_lsn; durability and active snapshot pins stay hard bounds. - A control snapshot reconciles the journal for skipped logins. - A snapshot source must cover the receiver's advertised commit_lsn; tail fetches and snap_keepalive renew the session lease. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und; persist the catch-up target Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Align the resync design with docs/craft/subtasks.md (S8, S10) and the HomeObject ReplicationStateMachine: - §4.9: baseline resync rides nuraft's snapshot mechanism on the volume's RAFT group; HomeBlocks supplies the content through the listener callbacks. Object stream: type|region|batch ids, partition metadata with the data watermark W and Sync watermark S, checksum-verified LBA region batches that each replace their whole range, LAST_OBJ_ID. Fuzzy read plus re-fetch of (W, S] converges; resumable receiver cursor persisted after a CP flush; IO gate while installing. - §4.8: RAFT-log compaction follows the journal cleanup floor; no snapshot pin (all_committed_lsn holds the floor at W). - C2: a member current on RAFT but behind on data is re-added as a learner (replace_member) to get a snapshot. - Cases 6-8, walk-throughs, tests, PR plan (SN1-SN4) and open questions updated; the CRAFT peer-plane stream (old option b') moves to §6 as rejected. Status notes on what has merged on dev/v6.x since. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6ca79c7 to
a69f24d
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved fencing and resynchronization flaws can block valid owners, allow login without the required committed prefix, and cause data rollback or stale reads.
7 open issues
Count absence replies only after the fence-and-drain barrier · New Reject fencing polls at or below the persisted abort term · New Require login catch-up to commit the full advertised prefix · New Pin snapshot watermark retention before streaming to learners · New Require snapshot watermark to cover receiver commit point · New Keep IO gated until copied versions are covered by commit LSN · New Invalidate stale journal records through the installed snapshot watermark · New
Review effort: Balanced
UBD no longer re-logs in when a replica moves. Its health check polls OM, sees the partition's replica list change, re-exposes once for the endpoint and swaps the member in place (craft::replace_member via ublkpp's craft_disk; see craft_client#7, ublkpp#17, ublk_nublox#163). Writes above the swap point J go to the new replica, reads wait until its commit_lsn reaches J, and the session and term are unchanged. Figure 7 now shows OM and the swap. The "UBD never writes to a learner" guarantee is replaced: UBD may already write to the new replica while it is a learner, which is safe because reads are gated on J and everything at or below J comes by resync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical flaws in login fencing and snapshot recovery can expose stale reads, discard acknowledged data, and allow required replay history to be reclaimed before recovery completes.
12 open issues
Bind fetches and replies to the applicable CRAFT session · New Bootstrap volumes from snapshot metadata before forwarding Use dLSN and Missing-map state for the read gate Recover acknowledged tail data before reopening member reads Force a fresh RAFT state when replacing a member Invalidate stale journal records through the installed snapshot watermark Keep IO gated until copied versions are covered by commit LSN Require snapshot watermark to cover receiver commit point Pin snapshot watermark retention before streaming to learners Require login catch-up to commit the full advertised prefix Reject fencing polls at or below the persisted abort term Count absence replies only after the fence-and-drain barrier
Review effort: Balanced
Login (§4.2, §4.6): - A responder counts as lacking a slot only if it has fenced and drained at the login term; the login probe fences like the poll. - A poll at or below aborted_upto leaves the fence alone, so a delayed poll cannot re-raise a fence LoginAborted already lowered. - The final round must reach rs_commit_lsn and commit its Sync before InternalLogin; otherwise retry, then abort. Peer plane (§4.7): requests and replies carry session_term; data from a responder that has not applied the current InternalLogin is discarded above that session's truncation point. Retention (§4.8, §9 C6): the leader pins W through RAFT (RetainFrom) before sending region data, cleared once the receiver reaches W or by the retention cap; a learner's progress is not in all_committed_lsn. Snapshot (§4.9): - The receiver refuses W below its own commit_lsn. - Completion drops the whole journal and sets journal_floor = W. - Reads stay gated until commit_lsn reaches G = max(W_hi, pre-install last_append); W_hi comes from a new trailer object. The gate is a dLSN, not the snapshot's RAFT index. - An empty joiner creates its volume from object 0 (bootstrap) before region objects are forwarded. - C2 needs remove + fresh re-add, not replace_member(C, C); open question 14 for the HomeStore mechanism. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved snapshot retention and installation gaps can lose acknowledged writes or expose stale data, while fencing and bootstrap flaws can block current owners or prevent learners from joining.
14 open issues
Bind fetches and replies to the applicable CRAFT session Bootstrap volumes from snapshot metadata before forwarding Use dLSN and Missing-map state for the read gate Recover acknowledged tail data before reopening member reads Force a fresh RAFT state when replacing a member Persist control state consistent with the advertised snapshot index · New Drain in-flight operations before replacing snapshot ranges · New Invalidate stale journal records through the installed snapshot watermark Keep IO gated until copied versions are covered by commit LSN Require snapshot watermark to cover receiver commit point Pin snapshot watermark retention before streaming to learners Require login catch-up to commit the full advertised prefix Reject fencing polls at or below the persisted abort term Count absence replies only after the fence-and-drain barrier
Review effort: Balanced
…arrier - create_snapshot captures the control state, W, S and the (W, S] verdicts at index L and persists them; object 0 serves that tuple on every retry and resume, so the receiver never replays an entry already reflected in it. - The receiver raises an exclusive barrier on object 0: no new IO, catch-up stopped, peer fetches refused, and every admitted write and commit drained before any range is replaced; P is read after the drain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
004dc3b to
4c34000
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved snapshot consistency, installation-race, and recovery-gating issues can miss writes, expose incomplete data, or fault partitions, while the proposed rejection path lacks API support.
13 open issues
Define a supported snapshot-refusal and watermark-refresh handshake · New Drain and fence index mutators before replacing snapshot ranges · New Bind resume to immutable transfer identity and baseline metadata · New Persist and restore the read gate across recovery · New Drain in-flight operations before replacing snapshot ranges Persist control state consistent with the advertised snapshot index Invalidate stale journal records through the installed snapshot watermark Keep IO gated until copied versions are covered by commit LSN Require snapshot watermark to cover receiver commit point Pin snapshot watermark retention before streaming to learners Require login catch-up to commit the full advertised prefix Reject fencing polls at or below the persisted abort term Count absence replies only after the fence-and-drain barrier
5 resolved since last review
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved snapshot-recovery gaps, particularly non-durable read gates and missing snapshot-generation identity, can expose versions above the read horizon or lose required writes after restart or donor changes.
11 open issues
Retain replay history for the full lifetime of a served context · New Reconcile local login fences before releasing them after installation · New Persist and restore the read gate across recovery Define a supported snapshot-refusal and watermark-refresh handshake Invalidate stale journal records through the installed snapshot watermark Keep IO gated until copied versions are covered by commit LSN Require snapshot watermark to cover receiver commit point Pin snapshot watermark retention before streaming to learners Require login catch-up to commit the full advertised prefix Reject fencing polls at or below the persisted abort term Count absence replies only after the fence-and-drain barrier
4 resolved since last review
Review effort: Balanced
…te, monotonic control merge - write_snapshot_obj returns void, so the receiver cannot refuse: the donor checks that W covers the receiver's commit_lsn (probe) and that the tail above W is still retained before serving object 0, and otherwise fails read_snapshot_obj and regenerates the snapshot. - Resume binds to a transfer identity, L plus the control tuple's digest; anything else restarts every region. - The install barrier also drains a suspended commit() and the catch-up runner's in-flight work, and an install epoch discards any pre-barrier completion that lands later. - P is persisted at the barrier and G at completion; a restart restores the read gate. - Installing merges session_term, fence_term and aborted_upto monotonically instead of adopting the tuple, so a fence raised after L is not lowered without a committed outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The resync design has an unresolved retention race that can make replay impossible, lacks receiver-aware transfer prerequisites, and can leave sessions permanently fenced after merging committed abort state.
3 open issues
11 resolved since last review
Reconcile local login fences before releasing them after installation Retain replay history for the full lifetime of a served context Persist and restore the read gate across recovery Define a supported snapshot-refusal and watermark-refresh handshake Invalidate stale journal records through the installed snapshot watermark Keep IO gated until copied versions are covered by commit LSN Require snapshot watermark to cover receiver commit point Pin snapshot watermark retention before streaming to learners Require login catch-up to commit the full advertised prefix Reject fencing polls at or below the persisted abort term Count absence replies only after the fence-and-drain barrier
Review effort: Balanced
…ol-only install; fence reconciliation
- The leader pins RetainFrom{W, L} and waits for it before validating the retained tail, closing the window in
which reclaim could pass W between check and pin.
- HomeStore's snapshot callbacks do not name the receiver, so the pin is keyed by the snapshot context and cleared
when no transfer of it is active and every member has reached W.
- The "never roll back" check moves to the receiver: if W is below its own commit_lsn it does a control-only
install (merge control state and verdicts, keep its data, ask for LAST_OBJ_ID), so no HomeStore change is needed.
- The control-state merge reconciles fence_term with committed outcomes: a fence at or below
max(session_term, aborted_upto) is lowered, so an abort already at or below L cannot leave a fence up forever.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The resync design can deadlock snapshot recovery by waiting on recipients that cannot acknowledge the pin, and its control-only branch can expose stale journal payloads across sessions.
2 open issues
3 resolved since last review
Review effort: Balanced
…stall drops the uncertified tail - The pin and the retained-tail check wait only for journal sources, not snapshot recipients, which cannot apply RAFT entries after L before they install; every durable slot still has a source holder. - A control-only install drops the journal above the receiver's own commit_lsn (truncations by compacted logins will not replay), marks that range missing and keeps reads gated until the acknowledged tail is fetched back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because critical quorum-safety and liveness concerns could cause acknowledged-write loss and indefinitely block snapshot recovery or reads.
No open issues 🎉
2 resolved since last review
Review effort: Balanced



Adds
docs/craft/resync.md, the resync design forCraftReplDev, linked fromdocs/craft/README.md. It is the reference for the resync implementation PRs, including the snapshot work (SN1–SN4 in §8).Problem
CRAFT acknowledges a write once a quorum stores it, so replicas fall behind by design. The wiki states what resync must guarantee, but there was no design for how
CraftReplDevdelivers it:dev/v6.xnow has the RAFT group per volume (#195),append()and the Sync triggers (#189), log-order apply (#211) and distinct-voter quorum evidence (#206). The snapshot callbacks are still stubs, and several gaps remain (§3.2, §9).Diagram
How the cases in the doc fit together:
flowchart TD W[UBD writes to all replicas] -->|2 of 3 store it| OK[Success to the application] W -->|fewer than 2 store it| ERR[Error to the application<br/>XFS goes offline, volume detaches] OK --> MISS[One replica missed it<br/>UBD routes reads around it] MISS --> HEAL[Leader BM heals it<br/>periodic / watchdog fill round] HEAL -->|data still in a journal| FETCH[Replica fetches the writes BM-to-BM] HEAL -->|data past retention| SNAP[nuraft snapshot from the leader<br/>then fetch the writes above W] ERR --> LOGIN[Next login under the fence<br/>slot filled or declared Empty] LOGIN -->|leader crashes after fencing| ABORT[New leader commits LoginAborted]The snapshot (§4.9), modeled on HomeObject's
ReplicationStateMachine:sequenceDiagram participant L as Leader (donor) participant R as Behind replica participant P as Any member L->>L: RAFT log compacted up to the cleanup floor L->>R: nuraft: install a snapshot R->>L: object 0|0 L-->>R: control state, watermarks W and S loop every LBA region, in batches R->>L: next object (region, batch) L-->>R: LBA range with data and holes (checksummed) R->>R: replace that range in its own B-tree end R->>R: commit point = W, (W, S] marked missing R->>P: fetch the writes above WAnd the central login safety rule:
sequenceDiagram participant Old as Old UBD participant L as Leader participant R as Replicas L->>R: fence the old session, how far are you? R-->>L: last write held → starting point Old->>R: write R-->>Old: rejected (fenced): nothing acked after the poll can be lost L->>R: RAFT: InternalLogin (truncate above starting point, admit new session) Note over R: the fence lifts only on InternalLogin or LoginAborted, never on a timerSolution
The doc covers:
CraftReplDevwith defects H1–H9, and a design-to-code map. Status notes record what has merged ondev/v6.xsince.docs/craft/subtasks.mdS8, S10):type | region | batch, plus a trailer carryingW_hi;Wbelow its owncommit_lsn;journal_floor = W);commit_lsnreachesG = max(W_hi, its pre-install last append);Wfor the length of the transfer;J(craft_client#7, ublkpp#17, ublk_nublox#164).rs_commit_lsnbeforeInternalLogin;session_term.Decisions recorded with the team:
Diagrams are Mermaid source in the doc (rendered by GitHub), with PNG copies in
docs/craft/resync-diagrams/for viewers that don't render Mermaid. The conan version bump (6.0.25) exists only because the repo's version check counts the PNGs as non-doc files.🤖 Generated with Claude Code