Skip to content

SDSTOR-22745 craft: CRAFT partitions survive restarts, reclaim their journal, and propose through their group (S8) - #213

Draft
shosseinimotlagh wants to merge 55 commits into
eBay:dev/v6.xfrom
shosseinimotlagh:S8_volume_create_delete
Draft

shosseinimotlagh wants to merge 55 commits into
eBay:dev/v6.xfrom
shosseinimotlagh:S8_volume_create_delete

Conversation

@shosseinimotlagh

@shosseinimotlagh shosseinimotlagh commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

S8 (SDSTOR-22745): the CRAFT volume lifecycle after create. A CRAFT partition's state now survives a restart (SDSTOR-22895, SDSTOR-22911), its journal is reclaimed, RAFT snapshot transfers are refused until CRAFT can serve them (SDSTOR-22897), member changes are stubbed (SDSTOR-22912), and each partition's device proposes through its volume's RAFT group (the rest of SDSTOR-22894). It builds on #211's apply queue; the persistence and the reclaim are ported from the closed S8a/S8b PRs (#201, #202), with their review fixes. Deleting a CRAFT volume is not part of it (SDSTOR-26581).


1. What a partition keeps on disk

Problem: A CRAFT partition kept its session, its Empty verdicts and its commit frontier in memory only. A restart came back with no session and commit_lsn at -1, and forgot the verdicts: a replayed Sync would then free a voided slot's blocks a second time.

Solution: CraftDurableState, written ahead of every change it protects, in a meta blk record of its own per partition (CraftState):

  • the session (term, client token) and the last RAFT entry applied;
  • the commit frontier a completed checkpoint covers, and the Sync watermark;
  • every Empty verdict: a late write to a voided slot must still be refused, and a client that lost a resolution reply
    asks again;
  • the entries committed but not applied yet, in log order, with the slots a queued Sync frees.
flowchart LR
    R["RAFT log (HomeStore)<br/>replayed above checkpoint_lsn"] -- "committed entries" --> D["CraftReplDev"]
    D -- "client writes" --> J["CRAFT journal<br/>data slots"]
    D -- "written ahead of each change" --> S["CraftState meta blk<br/>session, last applied RAFT index,<br/>durable frontier, Sync watermark,<br/>Empty verdicts, pending entries"]
    D -- "slots it commits" --> I["Index table"]
Loading

Write-ahead points:

  • An InternalLogin persists the session before admitting it. If that fails, the session isn't admitted, and a restart applies the login again.
  • A verdict is persisted before any block it voids is freed: by the Sync's apply, by a write it voided, or by a catch-up or pre-resolution fill. If persisting fails, the blocks leak rather than be freed twice.
  • The durable frontier moves only when a checkpoint started for it completes. A block an index write supersedes is freed only once that frontier covers the replacing slot.

One writer persists; concurrent callers share a round. A state record or journal tail that can't be read faults the partition. A partition that never persisted a record (a new one: a write at term 0 needs no login) restarts from the default image and reads its journal like any other.

Data blocks. HomeStore's allocator persists an allocation only once its consumer commits it (commit_blk), and after a restart the consumer commits again the blocks its journal still references. A write, a catch-up fill and a pre-resolution fill commit their blocks once the journal slot referencing them is durable, and if they can't, the partition faults rather than publish a slot whose blocks the allocator may hand out again; the restart recovery commits again those of every live slot above the durable frontier that no checkpoint persisted. The allocator holds the others since its load, and once HomeStore has started it refuses to persist an allocation again.

Faults are durable. A partition that faults (a failed recovery, or an apply it can't complete) marks its state record faulted and persists nothing more: a restart then faults it at once, rather than bringing it back from an image that lacks what it applied since.

Fixed on the way (H9 from S8a): a Sync's apply moved commit_lsn over the slots it resolved without applying them to the index, so a read could return the previous data. The apply now advances through commit().


2. Restart: recover before anything applies

Problem: During its start, HomeStore's RAFT log replay and the group rejoin deliver committed entries, which it never delivers again. The device can recover only once its journal is replayed, and HomeStore reports a journal's store open before replaying it, with the log device locked.

Solution: From its load, each CRAFT volume holds the entries committed for it, and its reattach is counted, so every checkpoint waits for it. When its journal's replay is done, a worker makes the device, recovers it, hands it the held entries in log order, and publishes it. init_homeblocks returns once every partition is recovered, or faulted (counted in craft_recoveries_failed).

sequenceDiagram
    participant HS as HomeStore start
    participant HB as HomeBlocks
    participant V as Volume
    participant W as Worker
    participant D as CraftReplDev
    HS->>HB: meta blks: every CraftState record, then the volumes
    HB->>V: load: take its record, reopen its journal
    Note over V: from here on it holds what commits for it
    HS->>V: RAFT log replay and rejoin: committed entries
    V->>V: hold them, in log order
    HS->>HB: journal replay done
    HB->>W: reattach, off HomeStore's thread
    W->>D: make it: journal, state store, proposer
    D->>D: recover(): restore the state, rebuild the overlay from the journal, commit the live slots' blocks again, queue the pending entries
    W->>D: hand over the held entries, in log order
    W->>V: publish the device
    HB->>HB: init_homeblocks returns once every reattach is done
Loading

Every way an entry can arrive after a restart (the saved pending entries, the RAFT replay, the hold, a live commit) goes through the same check: an entry at or below the last RAFT index the state counts as applied is skipped, as its effect is in the restored state. Pending entries apply again without their commit-time change, which the state already holds. A Sync still queued at the restart frees the slots its verdicts voided when it applies, their blocks kept allocated until then; the one being applied comes without them, as its frees may have happened: a leak is better than a double free.

A group without a replica here, or whose replica is being destroyed, has its entries dropped, logged once per group. HomeStore never reports the replay of a journal whose log store is missing: when init ends, such a partition is faulted for good, its record marked faulted before its count lets a checkpoint cover the entries held for it, which are dropped. A later boot whose journal does replay faults it again, rather than bring it back from a record that lacks them.


3. A checkpoint never covers an entry whose effect isn't durable

Problem: After a restart HomeStore replays a RAFT group's log only above its last checkpoint, and a checkpoint covers every entry committed when it starts. CRAFT applies entries asynchronously, so a checkpoint could cover an entry not applied yet, and a crash then lost it for good (a login, for one, left the old session in place).

Solution: An HS_CLIENT checkpoint consumer, which HomeStore flushes before the replication service records the checkpoint. It first waits for pending reattaches, so held entries are in their devices, then saves every partition's state with its pending entries. A failed save stops the process: HomeStore records the checkpoint whatever the flush returns. A partition faulted before or during its save (a journal write's timeout, say) saves nothing and fails nothing, once its record is durably marked faulted. The consumer is registered while HomeStore starts, so this holds for the checkpoint that ends the start too.

sequenceDiagram
    participant CP as HomeStore checkpoint
    participant C as HomeBlocks consumer (HS_CLIENT)
    participant D as CraftReplDev, each partition
    participant RS as Replication service
    CP->>C: flush
    C->>C: wait for pending reattaches
    loop every CRAFT volume, except one being destroyed or without its device yet
        C->>D: save_state(): the state, with the entries committed but not applied yet
    end
    C-->>CP: flushed (a failed save stops the process, a faulted partition saves nothing)
    CP->>RS: flushed last: records checkpoint_lsn
    Note over CP,RS: after a crash, HomeStore replays above checkpoint_lsn, and the saved pending entries are queued again
Loading

4. Journal reclaim

Problem: A partition's journal only grew.

Solution: It is reclaimed below min(persisted durable frontier, all_committed_lsn), and below every slot a committed Sync has yet to free:

  • below the persisted frontier the index is durable, so a restart no longer applies those slots again;
  • below all_committed_lsn every member has committed, so no catch-up needs them;
  • a Sync's verdicts let the commit step over the slots they void long before its apply frees their blocks, which it reads from the slots' journal records. A Sync queued again at a restart holds the floor the same way, and a write voided while it was journalled frees the blocks it holds rather than read them back.

all_committed_lsn is the client's word. The min with this member's own frontier keeps a wrong value from reclaiming anything a restart here still needs. A peer's fetch of a reclaimed slot fails the whole batch instead of reporting the slot as not held, so it never counts toward an Empty verdict for a committed slot. The reclaim runs when a persist moves the frontier and on each keep_alive, one runner at a time; without a state store nothing is reclaimed. The HomeStore backend truncates its log store, never past the last entry. The in-memory Empty verdicts below the floor are kept: one proposed again would otherwise fault the member.


5. Stopping a device: shutdown and destroy

Problem: A shutdown stopped none of a device's own work (its apply queue, journal reclaim, watchdog), so an apply in progress could touch HomeStore after it shut down. A destroy left a running reclaim truncating the journal it was removing.

Solution: CraftReplDev::stop() stops all of it, waiting out the apply, the reclaim and the checkpoint rounds in progress, and ends the proposer. A shutdown ends every device's proposals first, right after it refuses new ops and before it waits for the requests in flight: with craft_raft_propose_timeout_ms at 0 (no deadline), a leader without a quorum would otherwise hold its CRAFT op, and the shutdown, forever. A shutdown keeps the queued entries, and any that commit later, for HomeStore's last checkpoint to save; a destroy drops them. A journal write HomeStore never completes (its device write failed, SDSTOR-24993) times out after craft_journal_write_timeout_ms and faults the partition, so a stop never waits on it forever.

flowchart TD
    subgraph SD["Shutdown"]
        S1["requests drained"] --> S3["each device stops, keeping its queue"] --> S2["HomeBlocks' timer canceled"] --> S4["HomeStore shuts down: the last checkpoint saves each state and queue"] --> S5["iomgr stops"]
    end
    subgraph DS["Destroy of a replica"]
        D1["held entries dropped"] --> D2["the device stops, dropping its queue"] --> D3["CraftState record removed, first on disk"] --> D4["index table, journal, superblock"]
    end
Loading

The record goes first: a crash in between leaves a destroying volume without a record, never a record without a volume. A destroy resumed at boot removes the record the volume took. A record no volume takes at boot is logged and left in place.


6. Proposals through the volume's RAFT group

Problem: Nothing wired a RAFT proposer to a volume, so every leader-side round failed not_supported.

Solution: Each device owns a HomeStore RAFT proposer over its volume's repl_dev, made with the device at create and at the reattach. A resolution on the leader (request_resolution(), append()) commits its SyncRSCommitLSN through the group; a follower answers NOT_LEADER.

sequenceDiagram
    participant Cl as Client
    participant D as Leader's CraftReplDev
    participant P as Its proposer
    participant G as RAFT group
    participant T as HomeBlocks timer, every second
    Cl->>D: request_resolution(upto)
    D->>P: propose SyncRSCommitLSN
    P->>G: key-only entry
    G-->>D: committed, applied in log order
    D-->>Cl: resolved_upto, Empty slots
    Note over P,T: without a quorum, the timer completes it with timed_out after craft_raft_propose_timeout_ms (it may still commit)
Loading

The deadlines, of proposals and of journal writes, are expired by one HomeBlocks timer on its own reactors, armed at init and canceled at shutdown once the devices have stopped, not by a timer per proposer. iomgr arms and cancels a recurring timer on every worker with a blocking round on all of them; two reattaches doing that at once on workers hung init and the checkpoint waiting for them. Stopping a proposer completes its pending proposals with timed_out and refuses new ones with operation_canceled.


7. Members without a replica, snapshots, member changes

A member whose local create fails stays in the group (it votes and acks) without a replica: the failure is reported, not retried, and a partition move replaces the member. The node gauge craft_groups_without_replica counts such groups at each scrape, leaving out creates in flight here and groups being destroyed. RAFT can still elect such a member, as it has the whole log, and as leader it would fail every CRAFT op of the volume. When elected, it hands leadership off (craft_leaderships_handed_off), except while a create of the volume runs here or when it is the group's only member. A member whose replica can't serve the volume hands it off too: its device faulted, at a restart's recovery or later, or never came back (its journal didn't replay), or fault containment took the volume, or the whole node, offline (the CRAFT API then refuses every op). It checks when elected, and HomeBlocks' timer checks every second while the replica doesn't serve, which covers a device that faults while its member leads or fails its recovery after the election; one whose device is still being reattached keeps leadership. nuraft nominates the responsive member of the highest priority, normally the group's creator, which holds a replica; with the creator unresponsive, two members without one would nominate each other forever. So a member hands a group's leadership off at most once per craft_leadership_handoff_backoff_ms (10 s): elected again within it, it keeps leadership, and HomeBlocks' timer hands it off again once the backoff is over. A hand-off is counted against the shutdown, like a create, and a create of the volume that fails here checks the leadership again.

sequenceDiagram
    participant L as Leader
    participant B as Member B
    participant M as Metrics scrape
    L->>B: CreateVolume commits
    B->>B: local create fails: reported, B stays in the group
    M->>B: craft_groups_without_replica = 1
    Note over B: later elected leader, as it has the whole log
    B->>B: no replica here: yields leadership, nuraft picks the next leader
Loading

RAFT snapshots. HomeStore keeps compacting each group's log, and the CRAFT listener inherited no-op snapshot callbacks. Every member would segfault at its first snapshot (nuraft dereferences last_snapshot(), which was null), and a member behind the compaction got an endless stream of empty objects. The listener now keeps the snapshot's context, and a leader refuses transfers: read_snapshot_obj returns -1 and nuraft asks again later, counted in craft_snapshot_transfers_refused and logged once per group. Such a member stays behind until a partition move replaces it. HomeStore records a transfer's snapshot index on that member as soon as its first object arrives, and from then on skips its own entries at or below it, at a restart too: refused, the member only falls further behind. A snapshot that arrives anyway is not applied: apply_snapshot fails, and nuraft stops the process.

sequenceDiagram
    participant L as Leader
    participant B as Member behind the compaction
    L->>L: log compacted past B's last entry
    B->>L: needs a snapshot
    L-->>B: refused (read_snapshot_obj = -1), asked again later
    Note over B: stays behind until a partition move replaces it
Loading

Member changes. home_blocks::replace_member(volume_id, out, in) answers not_supported until reconfiguration exists. The listener logs HomeStore's member-change callbacks instead of ignoring them.


Changes, by commit

Commit Ticket Change
3e31590 SDSTOR-22912 replace_member stub; member-change callbacks log
832a86f SDSTOR-22912 the volume lifecycle tests run in CRAFT mode (VolumeCraftTest)
70a8f36 SDSTOR-22897 the proposer's deadline sweep armed before it returns (superseded by 4f068fd)
6795f67 SDSTOR-22897 snapshot context kept; snapshot transfers refused
a1352b3 SDSTOR-22895 gauge craft_groups_without_replica
ed5d856 SDSTOR-22895 a member without its replica hands leadership off
93f875f, 3b5a36b SDSTOR-22895 tests for both, across a restart
4b73fd3 SDSTOR-22895 the Sync apply commits its slots to the index (H9)
c6395ea version 6.0.24
f68ec58 SDSTOR-22911 durable partition state, written ahead, restored at restart
c6853ac SDSTOR-22895 pending entries in the state; recover() applies them again
c45ba53 SDSTOR-22911 journal reclaim
b8b1e2f SDSTOR-22911 a CraftState meta blk record per partition; recovery once HomeStore has started
3904081 SDSTOR-22895 the checkpoint consumer saves every partition's state first
f5301e8 SDSTOR-22895 entries held until the device is recovered; checkpoints wait for reattaches
6a7a766 SDSTOR-22895 CraftReplDev::stop() at shutdown and destroy
aff2ecf SDSTOR-22895 restart tests for journal writes and committed slots
4f068fd SDSTOR-22894 each device proposes through its volume's RAFT group
ed3c863 SDSTOR-22895 data blocks committed to the allocator, and again at a restart (review)
0ad7819 SDSTOR-22911 every Empty verdict kept in the image (review)
fd98fa0 SDSTOR-22895 a fault is durable; a faulted partition persists nothing more (review)
def13a0 SDSTOR-22895 stop() waits out the checkpoint rounds in flight (review)
a5d5355 SDSTOR-22895 a volume's device read under its lock where a destroy may reset it (review)
c69e9eb SDSTOR-22895 a Sync still queued at a restart frees the slots its verdicts voided (review)
cc597d7 SDSTOR-24993 journal writes time out instead of hanging (interim)
ac13c88 SDSTOR-22895 leadership hand-off at most once per backoff (review)
cfc0a9c SDSTOR-22895 the 3-member suite's leaders keep leadership without a quorum (test)
3c71fd3 SDSTOR-22745 the CRAFT volume tests build without _PRERELEASE (review)
1c9f73d merge dev/v6.x (#212, login poll)
c517853 version 6.0.25
d012e18 SDSTOR-24993 a journal write past its deadline keeps its entry until HomeStore writes it (review)
dfe970a SDSTOR-22895 a partition that faults during a checkpoint fails no checkpoint (review)
8d21543 SDSTOR-22895 a journal that never replays faults its partition for good (review)
291e998 SDSTOR-22895 a hand-off in progress holds up the shutdown (review)
20317a8 SDSTOR-22895 a stale hand-off job keeps the backoff going (review)
fed1de4 SDSTOR-22895 a failed create checks again whether to hand leadership off (review)
1aab376 SDSTOR-22895 the checkpoint test reads what the checkpoint saved (review)
706cdb4 merge dev/v6.x (#214, login pre-resolution)
fdad6f4 version 6.0.26
eb6ab65 SDSTOR-22895 a recovery after HomeStore's start commits again only the blocks the restart lost (found by the crash test)
40010a1 SDSTOR-22895 the reattach hold waits on a thread, not an iomgr delay flip (test hook)
24adca8 SDSTOR-22895 an abrupt crash replays what committed past the checkpoint (review)
d5a36d3 ublkpp 0.41 and craft_client 0.6, which CI builds now
9eab4f2 SDSTOR-22895 a slot whose blocks can't be committed faults the partition (review)
cbc8c98 SDSTOR-22911 the journal reclaim waits for the frees of a committed Sync (review)
698e4e1 SDSTOR-22895 a member whose replica can't serve hands leadership off (review)
adb81d1 SDSTOR-22895 the crash comes with the last Sync committed but not applied (review)
47ab7d5 SDSTOR-22895 a replica that stops serving after the election is checked every second (review)
ed21db2 merge dev/v6.x (#215, login's SyncRSCommitLSN)
0f058d4 version 6.0.27
e9e3f24 SDSTOR-22895 the leadership check reads a group's stage before its RAFT server (a use-after-free 47ab7d5 exposed)
9170c32 SDSTOR-22895 a member whose volume is offline, or whose node is restricted, hands leadership off (review)
cdec671 SDSTOR-22895 a partition without a state record restarts from its journal (review)
408a815 SDSTOR-22895 a shutdown ends the CRAFT proposals before it waits for the requests in flight (review)

Tests

73 new cases, and existing ones extended:

  • Unit (CraftRaftEntriesTest, CraftDurableStateCodec, and others): each write-ahead point and its failure, the shared writer, the durable frontier, deferred and replayed frees, a restart from a state image and its faults, or without one, pending entries, the reclaim floor and the frees it waits for, stop().
  • HomeStore backend (CraftHomeStoreBackendTest): the state record in the real meta service, the journal reclaim and tail, a write past its deadline whose entry the log writes later, whole.
  • One node (CraftVolumeTest):
    • a session that survives restarts;
    • the blocks of index-applied and journal-only slots, still allocated after a restart and readable after new writes;
    • unapplied journal writes, and committed slots, that read back after a restart;
    • a checkpoint keeping an entry not applied yet, read back from the record it saved;
    • a journal that never replays, faulting its partition for good;
    • init waiting for a held reattach;
    • an abrupt crash (a child process killed) with a voiding Sync, a login and a write committed past the last checkpoint, and a Sync committed but not applied: after the restart the RAFT log replay hands them to the volume's hold while its reattach is held, and the device comes back with the session, the verdict and the commit, every slot's blocks allocated and none freed twice;
    • a resolution through the group, before and after a restart.
  • Three processes (CraftVolumeMultiTest):
    • a member behind the compaction refused a snapshot;
    • a member without its replica reporting it and handing leadership off, at most once per backoff, a stale hand-off job included;
    • a member whose replica faults while it leads handing leadership off, and one whose volume goes offline;
    • a shutdown of a leader without a quorum ending a proposal that has no deadline;
    • entries committed while a member was down, held through its delayed reattach;
    • only the leader resolving;
    • a leader cut off from its group timing out.
  • VolumeCraftTest: the volume lifecycle suite in CRAFT mode.

Linux, on 408a815 (after merging #215; craft_client 0.6.0): Debug 18/18 suites pass, #215's 22 new tests included, and every HomeBlocks translation unit compiles without _PRERELEASE, as in a Release build. AddressSanitizer, rebuilt clean: 17/18, CraftVolumeMultiTest failing only on the known nuraft_mesg shutdown leak below (its 11 cases pass on all 3 members).

Mutation checks: breaking each guard named in the commit messages fails a test, except three that only a crash or unschedulable timing shows: a checkpoint that doesn't wait for reattaches, a shutdown that doesn't stop the devices, and the proposal timer's cancel at shutdown (iomgr removes it when it stops anyway).

Notes

  • Crash coverage: one partition on one node (ACrashReplaysWhatCommittedPastTheCheckpoint); a member of a multi-member group crashing is not covered yet.
  • iomgr delay flips in a restarting process: each file's first use of iomgr_flip::instance() (static in a header) replaces the timer of every delay flip, so one still pending fires on a destroyed object and hangs its reactor. The reattach hold uses a return-value flip and a thread instead; the iomgr fix is outside this PR.
  • No journal retention cap yet: a member that stops committing pins every member's journal, as all_committed_lsn stops moving. Planned with S10.
  • Not yet: deleting a CRAFT volume through remove_volume (SDSTOR-26581; a group destroy through HomeStore removes each replica), real snapshots and reconfiguration (SDSTOR-22915), login (SDSTOR-22744). Watchdog rounds and pre-resolution of missing slots need the peer fetcher (S9); the periodic Sync needs the client's commit LSN.
  • Known hazard, not reachable yet: the client watchdog arms its timer with the same blocking round on every worker, from the thread of the first write or keep_alive after a login, and stop() cancels it the same way. Nothing arms it until login exists; it should move to a shared timer with login.
  • Leaks, never double frees: blocks a crash catches between a verdict's persist and its frees, or between an index write and the checkpoint that lets the blocks it superseded go, stay allocated. So do the blocks that slots applied again after any restart supersede: such a slot may be applied over an index that already holds a newer block, so its apply frees nothing it replaces. These are bounded by the slots journalled above the durable commit frontier. Nothing reclaims them yet.
  • Performance: a partition's state record is written synchronously on the thread that persists it, as HomeStore writes its own superblocks; persists happen at logins, verdict frees and checkpoints, not per write.
  • Not ported from craft: apply RAFT entries in log order; InternalLogin v2 and LoginAborted #196/craft: catch-up retries with backoff and fetches by bounded ranges (PR E1) #200: fence and session terms, InternalLogin v2, LoginAborted, the catch-up runner, range fetches, journal pins.
  • Known flake: under AddressSanitizer, CraftVolumeMultiTest can fail on a leak in nuraft_mesg's shutdown (SDSTOR-26738), about one run in three; every test case passes.
  • Version: 6.0.27 (dev/v6.x is at 6.0.26 after SDSTOR-22891 craft: login's SyncRSCommitLSN; one watermark holder at a time #215).
  • Dependencies: ublkpp ^0.41 and craft_client ^0.6, as dev/v6.x requires since SDSTOR-22891 craft: login's SyncRSCommitLSN; one watermark holder at a time #215: ublkpp 0.41.0 requires craft_client 0.6. Neither release changes what HomeBlocks uses (both add member replacement to the client side).
  • api.md isn't updated yet: the CRAFT lifecycle and these limitations follow in this PR.

Generated with Claude Code

shosseinimotlagh and others added 19 commits November 17, 2026 17:40
A partition move needs a call to replace one member of a volume's RAFT
group. Reconfiguration is not ready, so home_blocks::replace_member answers
std::errc::not_supported for now; the real implementation lands behind the
same signature.

The CRAFT group listener now logs the member-change callbacks HomeStore
delivers (replace started, completed, task cleaned; member removed) instead
of ignoring them silently, so an unexpected membership change shows up.

Co-Authored-By: Claude <noreply@anthropic.com>
test_volume takes --craft: HomeBlocks starts in CRAFT mode and every volume
it creates is a CRAFT partition, a one-member RAFT group on this node. A new
CTest entry, VolumeCraftTest, runs the suite that way, on its own device
files and RAFT port so it can run alongside the legacy one.

CRAFT volumes can't be removed yet, so the cases that remove a volume skip in
CRAFT mode, and CreateVolumeThenRecover stops before its removal part. The
outstanding-IO shutdown case also skips: its IO is the legacy byte API, which
CRAFT volumes reject.

Co-Authored-By: Claude <noreply@anthropic.com>
…oser returns

HomeStoreCraftRaftProposer armed its recurring deadline sweep on every worker
without waiting. iomgr marks such a timer's device ready only once every
reactor has added it, and ignores the removal of one that is not ready yet
("Device N is not added to IOManager"). A proposer destroyed before then left
its sweep running on freed memory, and the next tick segfaulted. Wait for the
timer to be armed, as the CRAFT watchdog already does.

Found by the snapshot-refusal test of the next commit, which destroys a
proposer right after its last proposal.

Co-Authored-By: Claude <noreply@anthropic.com>
…ots exist

HomeStore keeps compacting each group's RAFT log: nuraft takes a snapshot
every snapshot_freq_distance entries, and the log is truncated behind it. The
CRAFT group listener inherited the no-op snapshot callbacks, which broke two
ways:

- nuraft reads the snapshot it just took back through last_snapshot() and
  dereferences it, and the inherited one returned null: every member of a
  CRAFT group would segfault at its first snapshot, and again on replay. The
  listener now keeps the context create_snapshot() gets (its position only;
  in memory, which nuraft tolerates after a restart).
- A member behind the compaction needs a snapshot, and the leader sent it an
  endless stream of empty objects. CRAFT can't serve one yet, so the leader
  now refuses (read_snapshot_obj returns -1, nuraft asks again later): that
  member stays behind until a partition move replaces it. Each refusal counts
  in a new node-wide metrics group (craft_snapshot_transfers_refused), and the
  first one per group is logged. A snapshot that arrives anyway is not applied:
  apply_snapshot fails, and nuraft stops the process.

Tests: a 3-member test stops one member, commits entries it misses, compacts
the leader's log past them, and restarts it: the leader refuses it a snapshot
and it stays behind. The test helper gains stop()/start(), and the 3-process
test runs on small index chunks, like the single-node one, as its volumes add
up.

Co-Authored-By: Claude <noreply@anthropic.com>
…replica

A member whose local create of a volume failed stays in the volume's RAFT
group without a replica: the failure is reported, not retried, and a
partition move replaces the member. Besides the error log, the node now
exposes the gauge craft_groups_without_replica: the RAFT groups it is a
member of with no replica here, other than those of a create in flight here
and those being destroyed.

The gauge is computed at each metrics scrape, from HomeStore's groups and
this node's volumes as they are then, so it needs no bookkeeping and holds
across restarts. The scrape callback is attached once recovery is done and
detached when shutdown starts, before HomeStore goes down.

Co-Authored-By: Claude <noreply@anthropic.com>
…p off

A member whose local create failed stays in the volume's RAFT group without a
replica until a partition move replaces it. RAFT may still elect it, as it has
the whole log, and as leader it would fail every CRAFT op of the volume. When
elected, it now hands leadership to another member, which nuraft picks, and
counts it (craft_leaderships_handed_off). It keeps leadership while a create
of the volume runs here, as that create needs it to lead, and when it is the
group's only member.

The hand-off runs on a worker, off the nuraft callback, and checks again there
that this member still leads and still has no replica.

Co-Authored-By: Claude <noreply@anthropic.com>
… leadership off

The 3-member test of a follower whose local create fails now also checks that
this member reports the group (craft_groups_without_replica), also after it
restarts, and that, made leader, it hands leadership to another member.

It restarts after a checkpoint: a CreateVolume committed after the last one is
committed again after the restart, rather than replayed, and that would create
the replica.

The tests read the node metrics through the shared test helper.

Co-Authored-By: Claude <noreply@anthropic.com>
…s leadership

The 3-member test asked member 1 to become leader right after restarting
it. Under AddressSanitizer, member 1 may not know the group's leader yet at
that point: nuraft refuses the request ("cannot find leader"),
become_leader() times out, and no hand-off happens, so the test failed after
its 120 s wait. It now asks again, once member 1 knows the leader, until the
hand-off shows.

Co-Authored-By: Claude <noreply@anthropic.com>
…e index

A SyncRSCommitLSN's apply moved commit_lsn forward over the slots it
resolved (skipping Empty ones, stopping at the first missing one) without
applying them to the index. commit() never revisits a slot below commit_lsn,
and a read serves overlay entries only above it, so a read of an LBA last
written by such a slot returned the older index value: the previous data
instead of the new write, or instead of an unmap's zeros. commit_lsn was also
no use as the durable restart frontier the next changes persist.

With an index, the apply now advances through commit(), which applies each
slot it passes to the index (same rule: skip Empty, stop at missing) and
retires its overlay entry. A commit() already running is left to finish, as
for its other callers; until the next write, keep_alive or Sync carries the
advance further, reads still find the slots in the overlay. Without an index
(the write-path-only unit tests) the frontier still moves alone.

Tests: a Sync commits a write and an unmap over LBAs whose older values are in
a fake index; a read must return the new data and the zeros. A test seam
(set_index_fns_for_test) lets commit() itself use the fake index.

Port of the H9 fix from the closed S8a PR (eBay#201).

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ored at restart

A CRAFT partition's session, its Empty verdicts and its commit frontier lived
in memory only: a restart came back with no session, forgot the verdicts (a
replay of their Sync would then free the voided slots' blocks a second time)
and restarted commit at -1.

CraftReplDev now keeps an image of what neither the journal nor the RAFT log
gives back (CraftDurableState): the session (term, client token), the last
RAFT entry applied, the commit frontier a completed checkpoint covers, the
Sync watermark and the Empty verdicts above that frontier. A CraftStateStore
given at construction persists it; without one nothing is persisted (the
light unit tests).

The image is written ahead of the change it protects. InternalLogin persists
the session before admitting it; if that fails the session isn't admitted,
and the image stops recording entries as applied, so a restart applies the
login again. A verdict, taken into the image when its Sync commits, is
persisted before any block it voids is freed: by the Sync's apply, by a write
it voided, or by a catch-up or pre-resolution fill it voided. If persisting
fails, those blocks leak rather than be freed twice. One writer persists the
image; concurrent callers share a round.

The checkpoint trigger now reports whether a checkpoint started by the call
completed; only then does the durable commit frontier move. A block an index
write supersedes is freed once that frontier covers the replacing slot: until
then a restart applies older slots again, which may still need it.

Restart recovery restores the session, commit_lsn (the durable frontier),
last_append_lsn (the journal tail, raised to the Sync watermark) and the
verdicts, then rebuilds the overlay; a slot the journal doesn't hold becomes
missing. The log replay skips the RAFT entries the image counts as applied,
and a slot journalled before the restart frees nothing it replaces when
applied again. An image or journal tail that can't be read faults the
partition. The HomeStore journal backend reports its tail, -1 when empty.

Tests: the codec; each write-ahead point and its failure; the shared writer;
the durable frontier on flushed, coalesced and failed checkpoints; deferred
and replayed frees; restart from an image, its faults, and persisting then
restarting end to end. Breaking any of these guards fails a test.

Port of S8a's persistence from the closed PR eBay#201 onto the apply queue of
eBay#211, with its review fixes: the store is in place before recovery, the codec
is bounded, and an empty journal's tail is defined.

Co-Authored-By: Claude <noreply@anthropic.com>
…ed, and recover() applies them again

HomeStore's log replay after a restart starts above its last checkpoint. A
CRAFT entry that checkpoint covers but the apply queue hadn't applied yet was
lost: its effect never made durable, and the entry never delivered again.

Every persisted image now also holds the entries committed but not yet
applied, in log order: the one being applied, then the queue. An entry leaves
them when the image records it as applied, in the same step, so an image holds
it as one or the other. HomeBlocks' checkpoint step can persist each
partition's image (save_state()) before HomeStore records the checkpoint.

recover() restores the image, rebuilds the overlay, then queues those entries
again, above the last one applied. They apply without their commit-time state
change, which the image already holds (a Sync's verdicts above the frontier and
its watermark; the journal walk rebuilds the rest): making it again could find
a verdict the restored frontier has passed, which faults the partition. A
Sync's frees are left out too: they may have happened before the restart, and
freeing twice is worse than leaking. A login raises the login fence again, as
at commit. The log replay skips the entries queued again. recover() can be
awaited, so the volume can publish the device once it is recovered;
on_restart() still starts it detached.

An apply that fails is now recorded as applied too: what it wrote ahead is in
the image either way (a login's session, admitted after a restart), and
applying it again would change nothing.

Tests: the codec of pending entries; images taken with entries queued and one
being applied; a restart applying them again, skipping those applied before it
and their replays; a pending Sync whose verdicts the frontier has passed, with
blocks it could free. Breaking any of these guards fails a test.

Co-Authored-By: Claude <noreply@anthropic.com>
…ontier and the set-wide commit

A CRAFT partition's journal only grew: nothing dropped the entries no member
needs anymore.

It is now reclaimed up to min(the durable commit frontier as persisted,
all_committed_lsn). Below the persisted frontier the index is durable, so a
restart no longer applies those slots again; below all_committed_lsn every
member has committed, so no member's catch-up needs them. all_committed_lsn
is the client's word, and the min with this member's own frontier keeps a
wrong value from reclaiming anything a restart here still needs. One that is
too large can leave a lagging member unable to catch up, but a peer's fetch of
a reclaimed slot fails the whole batch instead of reporting the slot as not
held, so no leader can count it toward verdicting a committed slot Empty.

The reclaim runs when a round persists a higher durable frontier and on each
keep_alive, one runner at a time. Without a state store nothing is persisted,
so nothing is reclaimed. A restart reads the reclaim point back from the
journal. The HomeStore backend truncates its log store, never past the last
entry (that would move the tail); HomeStore's periodic device truncation
releases the space.

Tests: the floor and each of its inputs, a frontier persisted only later, a
backend that can't reclaim or fails once, fetching a reclaimed slot, a
restart; the backend against a real log store. Breaking any of these guards
fails a test.

Port of S8b's journal reclaim from the closed PR eBay#202, without its journal
pins and range fetches, and without dropping the in-memory Empty verdicts
below the floor: a verdict for such a slot proposed again would then fault
the member.

Co-Authored-By: Claude <noreply@anthropic.com>
…k record, recovered at every restart

In production a CRAFT partition persisted nothing: the volume made its
CraftReplDev without a state store, and a restart attached a new device that
recovered nothing (term, Empty verdicts, commit frontier all gone).

Each partition now keeps its durable state in a meta blk record of its own
("CraftState": a header with the volume id and the image length, then the
encoded state). The device's first persist creates it and every later persist
rewrites it. At boot HomeBlocks reads the records ahead of the volumes, and
each CRAFT volume takes its own for its device's store.

The device is no longer attached when HomeStore opens the journal's store:
HomeStore reports that before it replays the journal, with the log device
locked. The end of the journal's replay only hands the store over. Once
HomeStore has started, HomeBlocks makes each CRAFT device, recovers it
(restores the state, rebuilds the overlay from the journal, queues the pending
entries again) and only then publishes it. init_homeblocks waits for this, so
it returns with every partition recovered, or faulted if its recovery failed.
The recovery counts itself in flight on the volume, so a destroy waits for it.

A destroy removes the record first, before anything else on disk: a crash in
between leaves a destroying volume without a record, never a record without a
volume. A destroy that resumes at boot removes the record the volume took. A
record that no volume takes at boot is logged and left in place.

Entries that HomeStore's log replay delivers before the device is published are
still dropped, as they were before it was attached.

Tests: the store against the real meta service (images of every size, the
record a boot reads back, another volume's record refused, destroy); a session
that survives two restarts; a destroy, direct or resumed at boot, that removes
the record. Breaking each guard (the record write, the destroy check, the
volume id check, the recovery, the store, the record handed to the volume, its
removal on both destroy paths) fails a test.

Co-Authored-By: Claude <noreply@anthropic.com>
…n's durable state first

After a restart HomeStore replays a RAFT group's log only above its last
checkpoint, and a checkpoint covers every entry committed when it starts. A
CRAFT partition applies committed entries asynchronously and persisted its
state only as each one applied, so a checkpoint could cover an entry not yet
applied, and a crash then lost it for good (a login, for one, left the old
session in place).

HomeBlocks now registers a checkpoint consumer (HS_CLIENT), which HomeStore
flushes before the replication service records the checkpoint. Its flush
persists every CRAFT partition's durable state with the entries committed but
not yet applied, which a restart queues again. A save that fails stops the
process: HomeStore records the checkpoint whatever the flush returns, and it
would cover entries whose state is not durable. A volume being destroyed, or
without its device yet, is skipped. A volume being saved counts itself in
flight, so its destroy waits.

The consumer is registered while HomeStore starts, ahead of its log replay: a
CreateVolume that the replay applies makes a CRAFT device during the start.

Test: an entry committed and held from applying, a forced checkpoint, a
restart; the entry applies from the saved state. Without the consumer, or with
a flush that saves nothing or skips every partition, it fails.

Co-Authored-By: Claude <noreply@anthropic.com>
…recovered, and let checkpoints wait for it

After a restart, a committed RAFT entry that reached a CRAFT volume before its
device was published was dropped, though HomeStore never delivers it again,
and HomeStore's own checkpoint at the end of its start could cover it. The
RAFT log replay and the rejoin deliver entries during HomeStore's start, and
the device was made only after it.

Each journal now reattaches its volume as soon as its log replay is done, on a
worker: the device is made, recovered, handed the entries held meanwhile in log
order, and published. Until then the volume holds every entry committed for
it, as long as it expects a device (its journal was reopened at load and it is
not being destroyed). A group without a replica here, or whose replica is being
destroyed, has its entries dropped, logged once per group instead of once per
entry. init_homeblocks still returns only once every reattach is done; a volume
whose journal never replays drops its hold then, logged.

A checkpoint's CRAFT step now first waits for the pending reattaches, so held
entries are in their devices, and saved with them, before the checkpoint
covers them. The consumer is registered while HomeStore starts, so this also
holds for the checkpoint that ends the start; its comment gave a wrong reason
for that (a CreateVolume replayed during the start creates nothing). A reattach
depends on no checkpoint, so the wait always ends.

A recovery that fails is counted in a new node counter,
craft_recoveries_failed; the partition stays faulted.

Tests: a member that restarts with its reattach held for 10 s holds the login
its group committed while it was down, then applies it; init waits for a held
reattach. Dropping instead of holding, never handing the held entries over, or
returning from init before the reattaches, fails a test. The checkpoint's wait
only matters across a crash, which these tests can't make.

Co-Authored-By: Claude <noreply@anthropic.com>
…wn, and fully at a destroy

A CRAFT device runs work of its own: its apply queue, its journal reclaim and
its client watchdog. A HomeBlocks shutdown stopped none of them, so an apply in
progress could still touch the index, the journal or the meta service after
HomeStore had shut down. A volume's destroy stopped the applies but left a
running reclaim truncating the journal it was removing.

CraftReplDev::stop() stops all three, waiting out the apply and the reclaim in
progress. A destroy drops the queued entries, as before. A shutdown keeps them
queued, and any that commit later too: HomeStore's last checkpoint saves them
with the partition's state, and the restart applies them. HomeBlocks stops
every CRAFT device once its last request has drained, before HomeStore shuts
down.

Tests: a shutdown's stop waits out the apply in progress and keeps the queued
entries, also those committed after it, for the checkpoint; a stop waits out
the reclaim in progress, and no reclaim runs after it. Breaking any of those
fails a test. Not stopping the devices at shutdown fails none: only an apply or
reclaim that outlives HomeStore's shutdown shows it, which the tests can't
time.

Co-Authored-By: Claude <noreply@anthropic.com>
…slots read back after a restart

Restart recovery had unit tests on mocks, and volume-level tests for the
session, the state record and the hold, but none for the data. Two volume-level
tests now restart HomeBlocks over a real HomeStore:
- writes acknowledged but never committed to the index: the restart rebuilds
  them from the journal, and they read back;
- slots a SyncRSCommitLSN committed to the index, and one it left in the
  journal: all read back.

Skipping the overlay rebuild at recovery, or restoring last_append_lsn without
the journal tail, fails both.

The suite runs on small index chunks and keeps most volumes it creates: these
two ran it out of chunks. Every restart test of the suite now removes its
volume when done.

Not covered: entries HomeStore replays from the RAFT log after a crash, and the
order in which it replays the RAFT log and the journal. The tests restart
gracefully, and HomeStore's checkpoint at shutdown covers every committed entry.

Co-Authored-By: Claude <noreply@anthropic.com>
…s volume's RAFT group

A CRAFT volume's device now owns a HomeStore RAFT proposer over the volume's repl_dev, made with the device at create
and at the reattach after a restart. append()'s resolution rounds and request_resolution() commit their
SyncRSCommitLSN through the group instead of refusing with not_supported. The device's stop() ends the proposer
last: its pending proposals complete with timed_out (they may still commit), later ones fail with
operation_canceled.

The proposer no longer arms its own deadline sweep. iomgr arms and cancels a recurring timer on every worker with a
blocking round on all of them; a reattach makes its device on a worker, and two reattaches doing it at once hung init
and the checkpoint waiting for them. HomeBlocks instead expires every device's overdue proposals from one timer on its
own reactors, armed at init and canceled at shutdown.

Tests: a resolution commits through the group, and again after a restart; only the leader runs one; a leader cut off
from its group gets timed_out; closed deadlines hand back what is pending and track no more.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 20:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues can lose Empty verdicts, overwrite recovery state, expose acknowledged data to reallocation, and race checkpoint teardown or device destruction.

5 open issues
What changed in this PR

This PR extends CRAFT’s volume lifecycle beyond creation, building on #211’s ordered apply queue to add restart recovery, journal reclamation, and proposals through each volume’s RAFT group. CRAFT volume deletion remains out of scope.

Changes:

  • Adds durable partition state, recovery sequencing, and checkpoint integration.
  • Adds journal reclamation, device shutdown coordination, and group-backed proposals with shared timeout handling.
  • Refuses unsupported snapshot transfers, adds replica-health metrics and leadership handoff, and expands lifecycle tests.
File Description
src/​lib/​volume/​volume.hpp Declares recovery, durable-state ownership, and committed-entry holding.
src/​lib/​volume/​volume.cpp Wires recovery, state storage, group proposals, and teardown.
src/​lib/​volume/​tests/​test_volume.cpp Runs applicable lifecycle cases in CRAFT mode.
src/​lib/​volume/​tests/​test_craft_volume.cpp Tests persistence, recovery, checkpoints, proposals, and state cleanup.
src/​lib/​volume/​tests/​test_craft_volume_multi.cpp Tests snapshot refusal, leadership handoff, recovery holds, and timeouts.
src/​lib/​volume/​tests/​test_common.hpp Adds separate stop/start helpers and metric access.
src/​lib/​volume/​tests/​CMakeLists.txt Registers CRAFT lifecycle tests and adjusts test chunk sizes.
src/​lib/​volume_mgr.cpp Loads state records, stubs member replacement, and handles missing replicas.
src/​lib/​listener.hpp Declares snapshot, leadership, and membership callbacks.
src/​lib/​listener.cpp Routes committed entries and implements snapshot refusal and callback logging.
src/​lib/​homeblks_impl.hpp Declares node metrics and CRAFT lifecycle coordination.
src/​lib/​homeblks_impl.cpp Integrates checkpoint saves, recovery waits, device stops, and proposal sweeps.
src/​lib/​craft/​tests/​test_craft_raft_entries.cpp Tests durable state, replay, reclamation, and stopping.
src/​lib/​craft/​tests/​test_craft_pre_resolution.cpp Tests persistence before freeing voided writes and fills.
src/​lib/​craft/​tests/​test_craft_homestore_backend.cpp Tests real state storage, journal tails, reclamation, and checkpoints.
src/​lib/​craft/​tests/​test_craft_commit.cpp Tests Sync-driven index updates and adapts checkpoint mocks.
src/​lib/​craft/​tests/​mock_journal_backend.hpp Adds a durable-state mock with failure and suspension controls.
src/​lib/​craft/​craft_repl_dev.hpp Defines persistence, recovery, reclamation, and proposer lifecycle interfaces.
src/​lib/​craft/​craft_repl_dev.cpp Implements durable recovery, safe-free sequencing, reclamation, and proposal handling.
src/​include/​homeblks/​home_blocks.hpp Exposes the unsupported member-replacement API.
conanfile.py Bumps the package version to 6.0.24.

Review effort: Balanced

Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
Comment thread src/lib/homeblks_impl.cpp Resolved
@codecov-commenter

codecov-commenter commented Nov 18, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 56.92884% with 230 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev/v6.x@ae830dd). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/lib/craft/craft_repl_dev.cpp 68.06% 5 Missing and 71 partials ⚠️
src/lib/homeblks_impl.cpp 44.18% 4 Missing and 44 partials ⚠️
src/lib/volume_mgr.cpp 39.43% 2 Missing and 41 partials ⚠️
src/lib/listener.cpp 18.51% 17 Missing and 5 partials ⚠️
src/lib/craft/craft_repl_dev.hpp 71.42% 5 Missing and 11 partials ⚠️
src/lib/volume/volume.cpp 57.14% 2 Missing and 13 partials ⚠️
src/lib/homeblks_impl.hpp 36.36% 0 Missing and 7 partials ⚠️
src/lib/volume/volume.hpp 66.66% 0 Missing and 3 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##             dev/v6.x     #213   +/-   ##
===========================================
  Coverage            ?   54.94%           
===========================================
  Files               ?       20           
  Lines               ?     2486           
  Branches            ?     1165           
===========================================
  Hits                ?     1366           
  Misses              ?      304           
  Partials            ?      816           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

shosseinimotlagh and others added 2 commits November 18, 2026 14:30
…ross restarts

HomeStore's allocator persists an allocation only once the consumer commits it (commit_blk), and after a restart the
consumer must commit again the blocks its journal still references. CRAFT allocated with async_alloc_write and never
committed: every restart, graceful or not, came back with all of a partition's blocks free, those the index references
included, for the next writes to reuse.

A write, a catch-up fill and a pre-resolution fill now commit their blocks once the journal slot that references them
is durable. Restart recovery commits again the blocks of every live slot it rebuilds the overlay from (above the
durable commit frontier, not Empty), before anything can allocate; a slot whose blocks can't be committed faults the
partition. The slots at or below the frontier need no new commit: the checkpoint that set the frontier persisted it.

Tests: after a restart the allocator holds the blocks of index-applied and journal-only slots, which read back after
new writes; the recovery commits exactly the live slots above the frontier again, and faults if it can't.

Co-Authored-By: Claude <noreply@anthropic.com>
…low the durable frontier too

The durable image dropped every Empty verdict at or below the durable commit frontier: once the index there is
durable, no apply needs them. But a client does. After a restart, a late write to a voided slot at or below the
frontier passed the Empty check, found the slot below last_append_lsn and not missing, and was acked as already
written though nothing was stored; a resolution retry by a client that lost the reply no longer listed the slot Empty.

The image now keeps every verdict, as the in-memory set always did, and a restart restores them all. The codec
accepts verdicts at or below the frontier.

Tests: verdicts kept when a checkpoint moves the frontier past them; after a restart a write to a voided slot below
the frontier is refused (EMPTY_SLOT); the codec takes them.

Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
Comment thread src/lib/volume/volume.cpp
Comment on lines +247 to +249
(void)logstores.open_log_store(
sb_->journal_logdev_id, sb_->journal_store_id, false /* append_mode */, nullptr /* log_found_cb */,
[replayed = std::move(replayed)](shared< homestore::home_log_store > journal, homestore::logstore_seq_num_t) {
shosseinimotlagh and others added 2 commits November 18, 2026 14:40
…, and persists nothing more

A recovery that failed published the device faulted, but its next checkpoint still saved the device's state over the
partition's record: default values if the image couldn't be loaded, the restored state without the entries a failed
overlay rebuild never re-queued. The restart after that came back healthy, its session and Empty verdicts gone. A
fault at runtime had the same end: checkpoints moved past entries the partition never applied, and a restart brought
it back from an image that lacked them.

Every fault now goes through fault(), which also marks the state record faulted (a flag in its header; the image, if
any, stays for whoever looks at it). A record so marked doesn't load, so a restart faults the partition at once; a
faulted partition persists nothing more, and its checkpoint save succeeds without writing. A partition faulted before
it ever persisted gets a record that says so.

Tests: a failed recovery and a runtime fault mark the record; nothing persists after a fault while a checkpoint save
still succeeds; a marked record faults the next restart; the HomeStore store keeps the mark in the record a boot
reads, with or without an image.

Co-Authored-By: Claude <noreply@anthropic.com>
A commit that crosses the checkpoint interval detaches a round that waits for HomeStore's flush, then persists the
durable frontier and frees the blocks it released. stop() didn't wait for it: after a destroy's stop the round could
free blocks of chunks the volume was releasing, and after a shutdown's it could write the meta service while HomeStore
was going down.

stop() now waits for those rounds to end, and none starts after it.

Test: stop() returns only once a held checkpoint round has completed and persisted its frontier; a commit after stop()
starts none.

Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ublkpp 0.41.0, which CI now builds, requires craft_client ^0.6, and craft_client's dev/v0.x branch is at 0.6.0:
with ublkpp ^0.40 and craft_client ^0.5 the dependency step failed on every branch. Neither release changes what
HomeBlocks uses. craft_client 0.6 adds member replacement to its client (client.hpp, tcp.hpp); HomeBlocks includes
only types.hpp, status.hpp and peer.hpp. ublkpp 0.41 adds craft_disk::replace_member; the legacy ublk driver uses
only ublk_disk.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 19, 2026 17:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shosseinimotlagh and others added 4 commits November 19, 2026 10:07
… partition

A write, a pre-resolution fill and a catch-up fill commit their blocks to the allocator once the journal slot
referencing them is durable, but ignored a failure to: the slot was published and later indexed. Once a checkpoint
moved the durable frontier past it, a restart no longer committed its blocks again, and the index pointed at blocks
the allocator could hand out. A failure now faults the partition, so nothing more is persisted, and fails the
operation, keeping the blocks the journal references, as a journal write's timeout does.

Co-Authored-By: Claude <noreply@anthropic.com>
…itted Sync

A Sync's verdicts install at commit, but its apply frees the voided slots'
blocks later, reading them from the slots' journal records. Meanwhile the
commit steps over those slots, a checkpoint can persist that frontier and
keep_alive can report it committed set-wide, so the reclaim could drop the
records first and the blocks leaked. The floor now stays below every slot a
committed (or, after a restart, re-queued) Sync still has to free, until its
apply frees or leaks them.

A write voided while it was journalled frees the blocks it holds instead of
reading them back from its slot's record, which the reclaim may have passed.

Co-Authored-By: Claude <noreply@anthropic.com>
…ip off

The hand-off kept leadership on any member with the volume in its map, also
one whose device faulted at recovery or later, or never came back at a
restart (its journal didn't replay). Elected, such a member refused every
resolution while the healthy members refused them as not the leader, and
the group couldn't resolve. Now only a replica that serves, or whose device
is still being reattached, keeps leadership (a create in progress still
does too), and the check runs again when a device is published faulted, is
given up, or faults while its member leads.

A device's fault only notes its volume, for HomeBlocks' one-second timer to
check: iomgr runs a message for the current reactor inline, and a fault may
hold the device's locks.

is_faulted() is production API now; fault_for_test() faults a device from
the volume tests.

Co-Authored-By: Claude <noreply@anthropic.com>
…not applied

The first child killed itself once the last Sync's proposal completed, but
the apply records the entry as applied and persists that just after it
completes the proposal: a persist that landed before the kill made the
restart rightly skip the replayed Sync, and the second child waited for a
commit nothing would make. The apply is now paused before that Sync is
proposed, from a thread of its own, and the child is killed once the Sync
has committed, so only the replay of the RAFT log can bring it back.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 19, 2026 19:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/lib/homeblks_impl.cpp
Comment on lines +43 to 45
inst->finish_craft_reattach();
inst->start_reaper_thread();
HomeBlocksImpl::s_instance_ = inst;
shosseinimotlagh and others added 3 commits November 19, 2026 23:43
…s checked every second

A device's fault reached HomeBlocks through its static instance, which init
publishes only after every reattach. So a device that faulted while the
entries held for it, or queued again at the restart, applied had its fault
dropped, and a member elected while its device was being reattached kept
leadership. The hook also read the instance on device threads while init
wrote it.

HomeBlocks' one-second timer now checks every CRAFT volume whose replica
doesn't serve, and hands leadership off if this member leads its group,
however and whenever the replica stopped serving. The fault hook and the
reattach paths' own checks are gone.

Co-Authored-By: Claude <noreply@anthropic.com>
…S8_volume_create_delete

Conflict in craft_repl_dev.hpp: the device keeps owning its proposer (this
branch), and append()'s single-flight flag becomes eBay#215's watermark_mu_.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 20, 2026 16:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/lib/homeblks_impl.cpp Resolved
Comment thread src/lib/volume/volume.cpp
state_change(volume_state::ONLINE);

attach_craft_dev(std::move(journal));
craft_dev_ = make_craft_dev(std::move(journal), my_id); // a new partition: nothing to recover
Comment thread src/lib/volume/volume.hpp Resolved
shosseinimotlagh and others added 4 commits November 20, 2026 09:49
… its RAFT server

47ab7d5's one-second scan queues every volume whose replica doesn't serve,
also one being destroyed, whose device is already gone. The check then
asked the group whether this member leads it before asking whether the
group was going away, and once HomeStore's reaper had run, is_leader() read
the freed RAFT server: a heap-use-after-free under AddressSanitizer. The
check now reads the group's stage first, and the scan leaves volumes being
destroyed alone: their group is going away too.

Co-Authored-By: Claude <noreply@anthropic.com>
…s restricted, hands leadership off

Fault containment takes a volume offline without faulting its device, and
the CRAFT API then refuses the volume's every op, as it does every op of a
restricted node (global fault containment). Both still counted as serving,
so an elected member kept leadership it couldn't use while the healthy
members refused as not the leader. An offline volume no longer serves, a
restricted node serves none, and HomeBlocks' timer checks them like any
other replica that doesn't serve; a volume back online serves again.

Co-Authored-By: Claude <noreply@anthropic.com>
…its journal

A new partition persists its state first at its first login or checkpoint,
but a write at term 0 needs no login, so it can be acked before then. A
crash in between left no record, and the restart returned on the missing
record without reading the journal: the write was lost and its blocks,
whose allocation no checkpoint had persisted, free for reuse. A partition
without a record now starts from the default image, and its journal is
read, served and its blocks committed again as any other's.

Co-Authored-By: Claude <noreply@anthropic.com>
…ts for the requests in flight

With craft_raft_propose_timeout_ms = 0 a proposal has no deadline, and a
leader without a quorum keeps its proposal, and the CRAFT op waiting on it,
in flight for good. The shutdown waited for every request in flight before
it stopped the devices, whose stop ends their proposals: it never got
there. It now ends every device's proposals right after it refuses new
ops; they complete timed_out, as at a deadline, and later ones fail.

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI max review requested due to automatic review settings November 20, 2026 18:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/lib/homeblks_impl.cpp
Comment on lines +622 to +623
iomanager.run_on_forget(iomgr::reactor_regex::random_worker, [this, vol = std::move(vol), journal]() mutable {
detail::detach(reattach_craft_volume(std::move(vol), std::move(journal)));
Comment thread src/lib/listener.cpp

homestore::async_status CraftGroupListener::create_snapshot(std::shared_ptr< homestore::snapshot_context > context) {
std::lock_guard lk{snapshot_mu_};
if (!last_snapshot_ || context->get_lsn() > last_snapshot_->get_lsn()) { last_snapshot_ = std::move(context); }
Comment thread src/lib/volume_mgr.cpp
auto const rd = homestore::hs()->repl_service().get_repl_dev(id);
// Its stage before its RAFT server: once a destroyed group's reaper has run, the server is freed, and is_leader()
// would read freed memory.
if (!rd || is_going_away(**rd) || !(*rd)->is_leader() || (*rd)->get_replication_quorum().size() < 2) {
int64_t last_applied_raft_idx{-1}; // RAFT entries at or below were applied: a replay of one is skipped
int64_t durable_commit_lsn{-1}; // commit frontier covered by a completed checkpoint
int64_t sync_watermark{-1}; // highest SyncRSCommitLSN rs_commit_lsn applied; last_append_lsn >= this
std::set< int64_t > empty_lsns{}; // every Empty verdict, as empty_lsns_ (a late write or a resolution retry asks)
Comment thread src/lib/homeblks_impl.cpp
auto f = shutdown_start();
// After the flag, which refuses new CRAFT ops: a request in flight may be waiting on a proposal that only a stop
// would end (no deadline, no quorum), and the wait below waits for it.
stop_craft_proposals();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants