Skip to content

docs/craft: resync design - #198

Open
Ilyin-Ilya wants to merge 14 commits into
dev/v6.xfrom
resync/design-doc
Open

Ilyin-Ilya wants to merge 14 commits into
dev/v6.xfrom
resync/design-doc

Conversation

@Ilyin-Ilya

@Ilyin-Ilya Ilyin-Ilya commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Adds docs/craft/resync.md, the resync design for CraftReplDev, linked from docs/craft/README.md. It is the reference for the resync implementation PRs, including the snapshot work (SN1–SN4 in §8).

Problem

CRAFT acknowledges a write once a quorum stores it, so replicas fall behind by design. The wiki states what resync must guarantee, but there was no design for how CraftReplDev delivers it:

  • how a replica catches up through the journal;
  • how failed writes are resolved;
  • how a login fences the old session;
  • how a replica past journal retention gets a full copy.

dev/v6.x now has the RAFT group per volume (#195), append() and the Sync triggers (#189), log-order apply (#211) and distinct-voter quorum evidence (#206). The snapshot callbacks are still stubs, and several gaps remain (§3.2, §9).

Diagram

How the cases in the doc fit together:

flowchart TD
    W[UBD writes to all replicas] -->|2 of 3 store it| OK[Success to the application]
    W -->|fewer than 2 store it| ERR[Error to the application<br/>XFS goes offline, volume detaches]
    OK --> MISS[One replica missed it<br/>UBD routes reads around it]
    MISS --> HEAL[Leader BM heals it<br/>periodic / watchdog fill round]
    HEAL -->|data still in a journal| FETCH[Replica fetches the writes BM-to-BM]
    HEAL -->|data past retention| SNAP[nuraft snapshot from the leader<br/>then fetch the writes above W]
    ERR --> LOGIN[Next login under the fence<br/>slot filled or declared Empty]
    LOGIN -->|leader crashes after fencing| ABORT[New leader commits LoginAborted]
Loading

The snapshot (§4.9), modeled on HomeObject's ReplicationStateMachine:

sequenceDiagram
    participant L as Leader (donor)
    participant R as Behind replica
    participant P as Any member
    L->>L: RAFT log compacted up to the cleanup floor
    L->>R: nuraft: install a snapshot
    R->>L: object 0|0
    L-->>R: control state, watermarks W and S
    loop every LBA region, in batches
        R->>L: next object (region, batch)
        L-->>R: LBA range with data and holes (checksummed)
        R->>R: replace that range in its own B-tree
    end
    R->>R: commit point = W, (W, S] marked missing
    R->>P: fetch the writes above W
Loading

And the central login safety rule:

sequenceDiagram
    participant Old as Old UBD
    participant L as Leader
    participant R as Replicas
    L->>R: fence the old session, how far are you?
    R-->>L: last write held → starting point
    Old->>R: write
    R-->>Old: rejected (fenced): nothing acked after the poll can be lost
    L->>R: RAFT: InternalLogin (truncate above starting point, admit new session)
    Note over R: the fence lifts only on InternalLogin or LoginAborted, never on a timer
Loading

Solution

The doc covers:

  • Overview by case: nine cases, each with a flow, a sequence diagram and its guarantees.
  • Invariants I1–I8, the as-built state of CraftReplDev with defects H1–H9, and a design-to-code map. Status notes record what has merged on dev/v6.x since.
  • Design:
    • triggers and their safe targets;
    • the leader round, where Empty is decided only at login on quorum evidence;
    • follower catch-up off the apply path;
    • failed writes resolved at login only;
    • login with pre-sync and a fence lifted only by a committed outcome;
    • peer wire v2;
    • journal reclaim;
    • persistence;
    • lease / logout / stand-in.
  • Snapshots (§4.9): through nuraft's snapshot callbacks on the volume's RAFT group, modeled on HomeObject (docs/craft/subtasks.md S8, S10):
    • an object stream laid out as type | region | batch, plus a trailer carrying W_hi;
    • data, not the B-tree: the leader walks its B-tree and sends each mapped LBA's data; the receiver writes it to its own blocks and builds its own index;
    • checksum-verified LBA region batches that replace their whole range;
    • a resumable receiver cursor saved after a checkpoint flush;
    • safety on install:
      • the receiver refuses a W below its own commit_lsn;
      • it drops its whole journal (journal_floor = W);
      • it serves no read until its commit_lsn reaches G = max(W_hi, its pre-install last append);
    • a RAFT-replicated retention pin at W for the length of the transfer;
    • bootstrap of a joiner that has no volume yet.
  • Snapshot triggers: RAFT-log compaction follows the journal cleanup floor. A member current on RAFT but behind on data is removed and re-added as a fresh learner (C2; the HomeStore mechanism is open question 14).
  • Member change (Case 7): UBD follows a replica move without re-login. It polls OM and swaps the member in place at swap point J (craft_client#7, ublkpp#17, ublk_nublox#164).
  • Login hardening:
    • Empty is decided only from responders that fenced and drained;
    • a delayed poll cannot re-raise an aborted fence;
    • the final round must reach rs_commit_lsn before InternalLogin;
    • peer traffic carries session_term.
  • Also: failure walk-throughs, alternatives considered, the test plan, the PR sequence, gaps and risks, and open questions.

Decisions recorded with the team:

  • resync is driven by the CRAFT BM side, never by UBD;
  • failed writes are resolved at the next login (Brian's review);
  • the login fence lifts only on a committed outcome, with pre-sync first;
  • snapshots use nuraft's snapshot mechanism, not a separate CRAFT stream (the earlier BM-to-BM option is in §6);
  • receiving a snapshot merges onto the existing B-tree, each batch replacing its range;
  • region-level delta snapshots are deferred.

Diagrams are Mermaid source in the doc (rendered by GitHub), with PNG copies in docs/craft/resync-diagrams/ for viewers that don't render Mermaid. The conan version bump (6.0.25) exists only because the repo's version check counts the PNGs as non-doc files.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved gaps in login fencing, Sync handling, and snapshot recovery can truncate committed data or incorrectly classify writes as Empty.

9 open issues
What changed in this PR

Adds the resync design reference for CraftReplDev, guiding implementation PRs #196, #197, and follow-up work.

Changes:

  • Documents recovery cases, safety invariants, implementation gaps, and a staged test and delivery plan.
  • Defines BM-driven healing, login fencing, and peer-to-peer snapshot recovery.
  • Links the design from the CRAFT index and bumps the package version for the repository’s version check.
File Description
docs/​craft/​resync.md Adds the resync design, protocol contracts, diagrams, risks, and implementation plan.
docs/​craft/​README.md Adds the design to the documentation index.
conanfile.py Bumps the package version from 6.0.17 to 6.0.18.

Review effort: Balanced

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Copilot AI balanced review requested due to automatic review settings November 11, 2026 18:00
Ilyin-Ilya added a commit that referenced this pull request Nov 11, 2026
A fenced replica waits for every write it admitted before the fence to finish
before reporting its LSNs. Such writes have already advanced last_append_lsn
and marked their slot missing; reporting earlier would let the leader's final
round count them as lacking and verdict them Empty while they are still being
journalled (review on #198). Test: LoginPollWaitsForInFlightWritesBeforeReporting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved critical protocol gaps can truncate acknowledged writes or expose inconsistent reads through incomplete login recovery, snapshot fence regression, unsafe membership cutovers, and unfenced snapshot replacement.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because findings omitted from the published review raise interdependent concerns about session fencing, snapshot read consistency, and recovery completeness.

No open issues 🎉

Review effort: Balanced

Ilyin-Ilya and others added 5 commits November 18, 2026 14:18
Case-by-case resync design for CraftReplDev with sequence diagrams: replica
misses a write, healing a lagging replica, failed writes resolved at login,
login with pre-sync and a fence lifted only by a committed outcome, login
abort, snapshot past retention (data streamed BM-to-BM, nuraft carries control
state only), new member, snapshot source failure, UBD disappearing.

Includes the invariants, the as-built state of CraftReplDev on dev/v6.x with
defects H1-H8, a design-to-code map, failure walk-throughs, alternatives, the
test plan, the HomeBlocks PR sequence (#196, #197, ...), gaps and risks, and
open questions. Linked from docs/craft/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- I6: login's rs_commit_lsn is bounded below by the last committed Sync
  watermark; Sync apply keeps raising last_append_lsn to N.
- Leader round: a reclaimed gap is never lacks-evidence; Empty verdicts are
  filtered after every fetch failure has lowered N.
- Login term is strictly above session, fence and aborted terms;
  aborted_upto is persisted, probed and snapshotted.
- A fenced poll drains already-admitted writes before reporting.
- The retention cap relaxes only all_committed_lsn; durability and active
  snapshot pins stay hard bounds.
- A control snapshot reconciles the journal for skipped logins.
- A snapshot source must cover the receiver's advertised commit_lsn; tail
  fetches and snap_keepalive renew the session lease.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und; persist the catch-up target

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Align the resync design with docs/craft/subtasks.md (S8, S10) and the
HomeObject ReplicationStateMachine:
- §4.9: baseline resync rides nuraft's snapshot mechanism on the volume's
  RAFT group; HomeBlocks supplies the content through the listener
  callbacks. Object stream: type|region|batch ids, partition metadata with
  the data watermark W and Sync watermark S, checksum-verified LBA region
  batches that each replace their whole range, LAST_OBJ_ID. Fuzzy read plus
  re-fetch of (W, S] converges; resumable receiver cursor persisted after a
  CP flush; IO gate while installing.
- §4.8: RAFT-log compaction follows the journal cleanup floor; no snapshot
  pin (all_committed_lsn holds the floor at W).
- C2: a member current on RAFT but behind on data is re-added as a learner
  (replace_member) to get a snapshot.
- Cases 6-8, walk-throughs, tests, PR plan (SN1-SN4) and open questions
  updated; the CRAFT peer-plane stream (old option b') moves to §6 as
  rejected. Status notes on what has merged on dev/v6.x since.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Copilot AI balanced review requested due to automatic review settings November 18, 2026 14:27
Copilot AI balanced review requested due to automatic review settings November 19, 2026 16:30
UBD no longer re-logs in when a replica moves. Its health check polls OM, sees the partition's replica list change,
re-exposes once for the endpoint and swaps the member in place (craft::replace_member via ublkpp's craft_disk; see
craft_client#7, ublkpp#17, ublk_nublox#163). Writes above the swap point J go to the new replica, reads wait until
its commit_lsn reaches J, and the session and term are unchanged. Figure 7 now shows OM and the swap. The
"UBD never writes to a learner" guarantee is replaced: UBD may already write to the new replica while it is a
learner, which is safe because reads are gated on J and everything at or below J comes by resync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/craft/resync.md Resolved
Copilot AI balanced review requested due to automatic review settings November 19, 2026 16:38
Login (§4.2, §4.6):
- A responder counts as lacking a slot only if it has fenced and drained at the login term; the login probe fences
  like the poll.
- A poll at or below aborted_upto leaves the fence alone, so a delayed poll cannot re-raise a fence LoginAborted
  already lowered.
- The final round must reach rs_commit_lsn and commit its Sync before InternalLogin; otherwise retry, then abort.

Peer plane (§4.7): requests and replies carry session_term; data from a responder that has not applied the
current InternalLogin is discarded above that session's truncation point.

Retention (§4.8, §9 C6): the leader pins W through RAFT (RetainFrom) before sending region data, cleared once the
receiver reaches W or by the retention cap; a learner's progress is not in all_committed_lsn.

Snapshot (§4.9):
- The receiver refuses W below its own commit_lsn.
- Completion drops the whole journal and sets journal_floor = W.
- Reads stay gated until commit_lsn reaches G = max(W_hi, pre-install last_append); W_hi comes from a new trailer
  object. The gate is a dLSN, not the snapshot's RAFT index.
- An empty joiner creates its volume from object 0 (bootstrap) before region objects are forwarded.
- C2 needs remove + fresh re-add, not replace_member(C, C); open question 14 for the HomeStore mechanism.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Copilot AI balanced review requested due to automatic review settings November 19, 2026 16:48
Ilyin-Ilya and others added 2 commits November 19, 2026 16:50
…arrier

- create_snapshot captures the control state, W, S and the (W, S] verdicts at index L and persists them; object 0
  serves that tuple on every retry and resume, so the receiver never replays an entry already reflected in it.
- The receiver raises an exclusive barrier on object 0: no new IO, catch-up stopped, peer fetches refused, and
  every admitted write and commit drained before any range is replaced; P is read after the drain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Copilot AI balanced review requested due to automatic review settings November 19, 2026 16:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
…te, monotonic control merge

- write_snapshot_obj returns void, so the receiver cannot refuse: the donor checks that W covers the receiver's
  commit_lsn (probe) and that the tail above W is still retained before serving object 0, and otherwise fails
  read_snapshot_obj and regenerates the snapshot.
- Resume binds to a transfer identity, L plus the control tuple's digest; anything else restarts every region.
- The install barrier also drains a suspended commit() and the catch-up runner's in-flight work, and an install
  epoch discards any pre-barrier completion that lands later.
- P is persisted at the barrier and G at completion; a restart restores the read gate.
- Installing merges session_term, fence_term and aborted_upto monotonically instead of adopting the tuple, so a
  fence raised after L is not lowered without a committed outcome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 17:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
…ol-only install; fence reconciliation

- The leader pins RetainFrom{W, L} and waits for it before validating the retained tail, closing the window in
  which reclaim could pass W between check and pin.
- HomeStore's snapshot callbacks do not name the receiver, so the pin is keyed by the snapshot context and cleared
  when no transfer of it is active and every member has reached W.
- The "never roll back" check moves to the receiver: if W is below its own commit_lsn it does a control-only
  install (merge control state and verdicts, keep its data, ask for LAST_OBJ_ID), so no HomeStore change is needed.
- The control-state merge reconciles fence_term with committed outcomes: a fence at or below
  max(session_term, aborted_upto) is lowered, so an abort already at or below L cannot leave a fence up forever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 17:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The resync design can deadlock snapshot recovery by waiting on recipients that cannot acknowledge the pin, and its control-only branch can expose stale journal payloads across sessions.

2 open issues
3 resolved since last review

Review effort: Balanced

Comment thread docs/craft/resync.md Resolved
Comment thread docs/craft/resync.md Resolved
…stall drops the uncertified tail

- The pin and the retained-tail check wait only for journal sources, not snapshot recipients, which cannot apply
  RAFT entries after L before they install; every durable slot still has a source holder.
- A control-only install drops the journal above the receiver's own commit_lsn (truncations by compacted logins
  will not replay), marks that range missing and keeps reads gated until the acknowledged tail is fetched back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 14:47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because critical quorum-safety and liveness concerns could cause acknowledged-write loss and indefinitely block snapshot recovery or reads.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants