Repository navigation
SDSTOR-22745 craft: CRAFT partitions survive restarts, reclaim their journal, and propose through their group (S8) - #213
Conversation
A partition move needs a call to replace one member of a volume's RAFT group. Reconfiguration is not ready, so home_blocks::replace_member answers std::errc::not_supported for now; the real implementation lands behind the same signature. The CRAFT group listener now logs the member-change callbacks HomeStore delivers (replace started, completed, task cleaned; member removed) instead of ignoring them silently, so an unexpected membership change shows up. Co-Authored-By: Claude <noreply@anthropic.com>
test_volume takes --craft: HomeBlocks starts in CRAFT mode and every volume it creates is a CRAFT partition, a one-member RAFT group on this node. A new CTest entry, VolumeCraftTest, runs the suite that way, on its own device files and RAFT port so it can run alongside the legacy one. CRAFT volumes can't be removed yet, so the cases that remove a volume skip in CRAFT mode, and CreateVolumeThenRecover stops before its removal part. The outstanding-IO shutdown case also skips: its IO is the legacy byte API, which CRAFT volumes reject. Co-Authored-By: Claude <noreply@anthropic.com>
…oser returns
HomeStoreCraftRaftProposer armed its recurring deadline sweep on every worker
without waiting. iomgr marks such a timer's device ready only once every
reactor has added it, and ignores the removal of one that is not ready yet
("Device N is not added to IOManager"). A proposer destroyed before then left
its sweep running on freed memory, and the next tick segfaulted. Wait for the
timer to be armed, as the CRAFT watchdog already does.
Found by the snapshot-refusal test of the next commit, which destroys a
proposer right after its last proposal.
Co-Authored-By: Claude <noreply@anthropic.com>
…ots exist HomeStore keeps compacting each group's RAFT log: nuraft takes a snapshot every snapshot_freq_distance entries, and the log is truncated behind it. The CRAFT group listener inherited the no-op snapshot callbacks, which broke two ways: - nuraft reads the snapshot it just took back through last_snapshot() and dereferences it, and the inherited one returned null: every member of a CRAFT group would segfault at its first snapshot, and again on replay. The listener now keeps the context create_snapshot() gets (its position only; in memory, which nuraft tolerates after a restart). - A member behind the compaction needs a snapshot, and the leader sent it an endless stream of empty objects. CRAFT can't serve one yet, so the leader now refuses (read_snapshot_obj returns -1, nuraft asks again later): that member stays behind until a partition move replaces it. Each refusal counts in a new node-wide metrics group (craft_snapshot_transfers_refused), and the first one per group is logged. A snapshot that arrives anyway is not applied: apply_snapshot fails, and nuraft stops the process. Tests: a 3-member test stops one member, commits entries it misses, compacts the leader's log past them, and restarts it: the leader refuses it a snapshot and it stays behind. The test helper gains stop()/start(), and the 3-process test runs on small index chunks, like the single-node one, as its volumes add up. Co-Authored-By: Claude <noreply@anthropic.com>
…replica A member whose local create of a volume failed stays in the volume's RAFT group without a replica: the failure is reported, not retried, and a partition move replaces the member. Besides the error log, the node now exposes the gauge craft_groups_without_replica: the RAFT groups it is a member of with no replica here, other than those of a create in flight here and those being destroyed. The gauge is computed at each metrics scrape, from HomeStore's groups and this node's volumes as they are then, so it needs no bookkeeping and holds across restarts. The scrape callback is attached once recovery is done and detached when shutdown starts, before HomeStore goes down. Co-Authored-By: Claude <noreply@anthropic.com>
…p off A member whose local create failed stays in the volume's RAFT group without a replica until a partition move replaces it. RAFT may still elect it, as it has the whole log, and as leader it would fail every CRAFT op of the volume. When elected, it now hands leadership to another member, which nuraft picks, and counts it (craft_leaderships_handed_off). It keeps leadership while a create of the volume runs here, as that create needs it to lead, and when it is the group's only member. The hand-off runs on a worker, off the nuraft callback, and checks again there that this member still leads and still has no replica. Co-Authored-By: Claude <noreply@anthropic.com>
… leadership off The 3-member test of a follower whose local create fails now also checks that this member reports the group (craft_groups_without_replica), also after it restarts, and that, made leader, it hands leadership to another member. It restarts after a checkpoint: a CreateVolume committed after the last one is committed again after the restart, rather than replayed, and that would create the replica. The tests read the node metrics through the shared test helper. Co-Authored-By: Claude <noreply@anthropic.com>
…s leadership
The 3-member test asked member 1 to become leader right after restarting
it. Under AddressSanitizer, member 1 may not know the group's leader yet at
that point: nuraft refuses the request ("cannot find leader"),
become_leader() times out, and no hand-off happens, so the test failed after
its 120 s wait. It now asks again, once member 1 knows the leader, until the
hand-off shows.
Co-Authored-By: Claude <noreply@anthropic.com>
…e index A SyncRSCommitLSN's apply moved commit_lsn forward over the slots it resolved (skipping Empty ones, stopping at the first missing one) without applying them to the index. commit() never revisits a slot below commit_lsn, and a read serves overlay entries only above it, so a read of an LBA last written by such a slot returned the older index value: the previous data instead of the new write, or instead of an unmap's zeros. commit_lsn was also no use as the durable restart frontier the next changes persist. With an index, the apply now advances through commit(), which applies each slot it passes to the index (same rule: skip Empty, stop at missing) and retires its overlay entry. A commit() already running is left to finish, as for its other callers; until the next write, keep_alive or Sync carries the advance further, reads still find the slots in the overlay. Without an index (the write-path-only unit tests) the frontier still moves alone. Tests: a Sync commits a write and an unmap over LBAs whose older values are in a fake index; a read must return the new data and the zeros. A test seam (set_index_fns_for_test) lets commit() itself use the fake index. Port of the H9 fix from the closed S8a PR (eBay#201). Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…ored at restart A CRAFT partition's session, its Empty verdicts and its commit frontier lived in memory only: a restart came back with no session, forgot the verdicts (a replay of their Sync would then free the voided slots' blocks a second time) and restarted commit at -1. CraftReplDev now keeps an image of what neither the journal nor the RAFT log gives back (CraftDurableState): the session (term, client token), the last RAFT entry applied, the commit frontier a completed checkpoint covers, the Sync watermark and the Empty verdicts above that frontier. A CraftStateStore given at construction persists it; without one nothing is persisted (the light unit tests). The image is written ahead of the change it protects. InternalLogin persists the session before admitting it; if that fails the session isn't admitted, and the image stops recording entries as applied, so a restart applies the login again. A verdict, taken into the image when its Sync commits, is persisted before any block it voids is freed: by the Sync's apply, by a write it voided, or by a catch-up or pre-resolution fill it voided. If persisting fails, those blocks leak rather than be freed twice. One writer persists the image; concurrent callers share a round. The checkpoint trigger now reports whether a checkpoint started by the call completed; only then does the durable commit frontier move. A block an index write supersedes is freed once that frontier covers the replacing slot: until then a restart applies older slots again, which may still need it. Restart recovery restores the session, commit_lsn (the durable frontier), last_append_lsn (the journal tail, raised to the Sync watermark) and the verdicts, then rebuilds the overlay; a slot the journal doesn't hold becomes missing. The log replay skips the RAFT entries the image counts as applied, and a slot journalled before the restart frees nothing it replaces when applied again. An image or journal tail that can't be read faults the partition. The HomeStore journal backend reports its tail, -1 when empty. Tests: the codec; each write-ahead point and its failure; the shared writer; the durable frontier on flushed, coalesced and failed checkpoints; deferred and replayed frees; restart from an image, its faults, and persisting then restarting end to end. Breaking any of these guards fails a test. Port of S8a's persistence from the closed PR eBay#201 onto the apply queue of eBay#211, with its review fixes: the store is in place before recovery, the codec is bounded, and an empty journal's tail is defined. Co-Authored-By: Claude <noreply@anthropic.com>
…ed, and recover() applies them again HomeStore's log replay after a restart starts above its last checkpoint. A CRAFT entry that checkpoint covers but the apply queue hadn't applied yet was lost: its effect never made durable, and the entry never delivered again. Every persisted image now also holds the entries committed but not yet applied, in log order: the one being applied, then the queue. An entry leaves them when the image records it as applied, in the same step, so an image holds it as one or the other. HomeBlocks' checkpoint step can persist each partition's image (save_state()) before HomeStore records the checkpoint. recover() restores the image, rebuilds the overlay, then queues those entries again, above the last one applied. They apply without their commit-time state change, which the image already holds (a Sync's verdicts above the frontier and its watermark; the journal walk rebuilds the rest): making it again could find a verdict the restored frontier has passed, which faults the partition. A Sync's frees are left out too: they may have happened before the restart, and freeing twice is worse than leaking. A login raises the login fence again, as at commit. The log replay skips the entries queued again. recover() can be awaited, so the volume can publish the device once it is recovered; on_restart() still starts it detached. An apply that fails is now recorded as applied too: what it wrote ahead is in the image either way (a login's session, admitted after a restart), and applying it again would change nothing. Tests: the codec of pending entries; images taken with entries queued and one being applied; a restart applying them again, skipping those applied before it and their replays; a pending Sync whose verdicts the frontier has passed, with blocks it could free. Breaking any of these guards fails a test. Co-Authored-By: Claude <noreply@anthropic.com>
…ontier and the set-wide commit A CRAFT partition's journal only grew: nothing dropped the entries no member needs anymore. It is now reclaimed up to min(the durable commit frontier as persisted, all_committed_lsn). Below the persisted frontier the index is durable, so a restart no longer applies those slots again; below all_committed_lsn every member has committed, so no member's catch-up needs them. all_committed_lsn is the client's word, and the min with this member's own frontier keeps a wrong value from reclaiming anything a restart here still needs. One that is too large can leave a lagging member unable to catch up, but a peer's fetch of a reclaimed slot fails the whole batch instead of reporting the slot as not held, so no leader can count it toward verdicting a committed slot Empty. The reclaim runs when a round persists a higher durable frontier and on each keep_alive, one runner at a time. Without a state store nothing is persisted, so nothing is reclaimed. A restart reads the reclaim point back from the journal. The HomeStore backend truncates its log store, never past the last entry (that would move the tail); HomeStore's periodic device truncation releases the space. Tests: the floor and each of its inputs, a frontier persisted only later, a backend that can't reclaim or fails once, fetching a reclaimed slot, a restart; the backend against a real log store. Breaking any of these guards fails a test. Port of S8b's journal reclaim from the closed PR eBay#202, without its journal pins and range fetches, and without dropping the in-memory Empty verdicts below the floor: a verdict for such a slot proposed again would then fault the member. Co-Authored-By: Claude <noreply@anthropic.com>
…k record, recovered at every restart
In production a CRAFT partition persisted nothing: the volume made its
CraftReplDev without a state store, and a restart attached a new device that
recovered nothing (term, Empty verdicts, commit frontier all gone).
Each partition now keeps its durable state in a meta blk record of its own
("CraftState": a header with the volume id and the image length, then the
encoded state). The device's first persist creates it and every later persist
rewrites it. At boot HomeBlocks reads the records ahead of the volumes, and
each CRAFT volume takes its own for its device's store.
The device is no longer attached when HomeStore opens the journal's store:
HomeStore reports that before it replays the journal, with the log device
locked. The end of the journal's replay only hands the store over. Once
HomeStore has started, HomeBlocks makes each CRAFT device, recovers it
(restores the state, rebuilds the overlay from the journal, queues the pending
entries again) and only then publishes it. init_homeblocks waits for this, so
it returns with every partition recovered, or faulted if its recovery failed.
The recovery counts itself in flight on the volume, so a destroy waits for it.
A destroy removes the record first, before anything else on disk: a crash in
between leaves a destroying volume without a record, never a record without a
volume. A destroy that resumes at boot removes the record the volume took. A
record that no volume takes at boot is logged and left in place.
Entries that HomeStore's log replay delivers before the device is published are
still dropped, as they were before it was attached.
Tests: the store against the real meta service (images of every size, the
record a boot reads back, another volume's record refused, destroy); a session
that survives two restarts; a destroy, direct or resumed at boot, that removes
the record. Breaking each guard (the record write, the destroy check, the
volume id check, the recovery, the store, the record handed to the volume, its
removal on both destroy paths) fails a test.
Co-Authored-By: Claude <noreply@anthropic.com>
…n's durable state first After a restart HomeStore replays a RAFT group's log only above its last checkpoint, and a checkpoint covers every entry committed when it starts. A CRAFT partition applies committed entries asynchronously and persisted its state only as each one applied, so a checkpoint could cover an entry not yet applied, and a crash then lost it for good (a login, for one, left the old session in place). HomeBlocks now registers a checkpoint consumer (HS_CLIENT), which HomeStore flushes before the replication service records the checkpoint. Its flush persists every CRAFT partition's durable state with the entries committed but not yet applied, which a restart queues again. A save that fails stops the process: HomeStore records the checkpoint whatever the flush returns, and it would cover entries whose state is not durable. A volume being destroyed, or without its device yet, is skipped. A volume being saved counts itself in flight, so its destroy waits. The consumer is registered while HomeStore starts, ahead of its log replay: a CreateVolume that the replay applies makes a CRAFT device during the start. Test: an entry committed and held from applying, a forced checkpoint, a restart; the entry applies from the saved state. Without the consumer, or with a flush that saves nothing or skips every partition, it fails. Co-Authored-By: Claude <noreply@anthropic.com>
…recovered, and let checkpoints wait for it After a restart, a committed RAFT entry that reached a CRAFT volume before its device was published was dropped, though HomeStore never delivers it again, and HomeStore's own checkpoint at the end of its start could cover it. The RAFT log replay and the rejoin deliver entries during HomeStore's start, and the device was made only after it. Each journal now reattaches its volume as soon as its log replay is done, on a worker: the device is made, recovered, handed the entries held meanwhile in log order, and published. Until then the volume holds every entry committed for it, as long as it expects a device (its journal was reopened at load and it is not being destroyed). A group without a replica here, or whose replica is being destroyed, has its entries dropped, logged once per group instead of once per entry. init_homeblocks still returns only once every reattach is done; a volume whose journal never replays drops its hold then, logged. A checkpoint's CRAFT step now first waits for the pending reattaches, so held entries are in their devices, and saved with them, before the checkpoint covers them. The consumer is registered while HomeStore starts, so this also holds for the checkpoint that ends the start; its comment gave a wrong reason for that (a CreateVolume replayed during the start creates nothing). A reattach depends on no checkpoint, so the wait always ends. A recovery that fails is counted in a new node counter, craft_recoveries_failed; the partition stays faulted. Tests: a member that restarts with its reattach held for 10 s holds the login its group committed while it was down, then applies it; init waits for a held reattach. Dropping instead of holding, never handing the held entries over, or returning from init before the reattaches, fails a test. The checkpoint's wait only matters across a crash, which these tests can't make. Co-Authored-By: Claude <noreply@anthropic.com>
…wn, and fully at a destroy A CRAFT device runs work of its own: its apply queue, its journal reclaim and its client watchdog. A HomeBlocks shutdown stopped none of them, so an apply in progress could still touch the index, the journal or the meta service after HomeStore had shut down. A volume's destroy stopped the applies but left a running reclaim truncating the journal it was removing. CraftReplDev::stop() stops all three, waiting out the apply and the reclaim in progress. A destroy drops the queued entries, as before. A shutdown keeps them queued, and any that commit later too: HomeStore's last checkpoint saves them with the partition's state, and the restart applies them. HomeBlocks stops every CRAFT device once its last request has drained, before HomeStore shuts down. Tests: a shutdown's stop waits out the apply in progress and keeps the queued entries, also those committed after it, for the checkpoint; a stop waits out the reclaim in progress, and no reclaim runs after it. Breaking any of those fails a test. Not stopping the devices at shutdown fails none: only an apply or reclaim that outlives HomeStore's shutdown shows it, which the tests can't time. Co-Authored-By: Claude <noreply@anthropic.com>
…slots read back after a restart Restart recovery had unit tests on mocks, and volume-level tests for the session, the state record and the hold, but none for the data. Two volume-level tests now restart HomeBlocks over a real HomeStore: - writes acknowledged but never committed to the index: the restart rebuilds them from the journal, and they read back; - slots a SyncRSCommitLSN committed to the index, and one it left in the journal: all read back. Skipping the overlay rebuild at recovery, or restoring last_append_lsn without the journal tail, fails both. The suite runs on small index chunks and keeps most volumes it creates: these two ran it out of chunks. Every restart test of the suite now removes its volume when done. Not covered: entries HomeStore replays from the RAFT log after a crash, and the order in which it replays the RAFT log and the journal. The tests restart gracefully, and HomeStore's checkpoint at shutdown covers every committed entry. Co-Authored-By: Claude <noreply@anthropic.com>
…s volume's RAFT group A CRAFT volume's device now owns a HomeStore RAFT proposer over the volume's repl_dev, made with the device at create and at the reattach after a restart. append()'s resolution rounds and request_resolution() commit their SyncRSCommitLSN through the group instead of refusing with not_supported. The device's stop() ends the proposer last: its pending proposals complete with timed_out (they may still commit), later ones fail with operation_canceled. The proposer no longer arms its own deadline sweep. iomgr arms and cancels a recurring timer on every worker with a blocking round on all of them; a reattach makes its device on a worker, and two reattaches doing it at once hung init and the checkpoint waiting for them. HomeBlocks instead expires every device's overdue proposals from one timer on its own reactors, armed at init and canceled at shutdown. Tests: a resolution commits through the group, and again after a restart; only the leader runs one; a leader cut off from its group gets timed_out; closed deadlines hand back what is pending and track no more. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can lose Empty verdicts, overwrite recovery state, expose acknowledged data to reallocation, and race checkpoint teardown or device destruction.
5 open issues
What changed in this PR
This PR extends CRAFT’s volume lifecycle beyond creation, building on #211’s ordered apply queue to add restart recovery, journal reclamation, and proposals through each volume’s RAFT group. CRAFT volume deletion remains out of scope.
Changes:
- Adds durable partition state, recovery sequencing, and checkpoint integration.
- Adds journal reclamation, device shutdown coordination, and group-backed proposals with shared timeout handling.
- Refuses unsupported snapshot transfers, adds replica-health metrics and leadership handoff, and expands lifecycle tests.
| File | Description |
|---|---|
src/lib/volume/volume.hpp |
Declares recovery, durable-state ownership, and committed-entry holding. |
src/lib/volume/volume.cpp |
Wires recovery, state storage, group proposals, and teardown. |
src/lib/volume/tests/test_volume.cpp |
Runs applicable lifecycle cases in CRAFT mode. |
src/lib/volume/tests/test_craft_volume.cpp |
Tests persistence, recovery, checkpoints, proposals, and state cleanup. |
src/lib/volume/tests/test_craft_volume_multi.cpp |
Tests snapshot refusal, leadership handoff, recovery holds, and timeouts. |
src/lib/volume/tests/test_common.hpp |
Adds separate stop/start helpers and metric access. |
src/lib/volume/tests/CMakeLists.txt |
Registers CRAFT lifecycle tests and adjusts test chunk sizes. |
src/lib/volume_mgr.cpp |
Loads state records, stubs member replacement, and handles missing replicas. |
src/lib/listener.hpp |
Declares snapshot, leadership, and membership callbacks. |
src/lib/listener.cpp |
Routes committed entries and implements snapshot refusal and callback logging. |
src/lib/homeblks_impl.hpp |
Declares node metrics and CRAFT lifecycle coordination. |
src/lib/homeblks_impl.cpp |
Integrates checkpoint saves, recovery waits, device stops, and proposal sweeps. |
src/lib/craft/tests/test_craft_raft_entries.cpp |
Tests durable state, replay, reclamation, and stopping. |
src/lib/craft/tests/test_craft_pre_resolution.cpp |
Tests persistence before freeing voided writes and fills. |
src/lib/craft/tests/test_craft_homestore_backend.cpp |
Tests real state storage, journal tails, reclamation, and checkpoints. |
src/lib/craft/tests/test_craft_commit.cpp |
Tests Sync-driven index updates and adapts checkpoint mocks. |
src/lib/craft/tests/mock_journal_backend.hpp |
Adds a durable-state mock with failure and suspension controls. |
src/lib/craft/craft_repl_dev.hpp |
Defines persistence, recovery, reclamation, and proposer lifecycle interfaces. |
src/lib/craft/craft_repl_dev.cpp |
Implements durable recovery, safe-free sequencing, reclamation, and proposal handling. |
src/include/homeblks/home_blocks.hpp |
Exposes the unsupported member-replacement API. |
conanfile.py |
Bumps the package version to 6.0.24. |
Review effort: Balanced
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev/v6.x #213 +/- ##
===========================================
Coverage ? 54.94%
===========================================
Files ? 20
Lines ? 2486
Branches ? 1165
===========================================
Hits ? 1366
Misses ? 304
Partials ? 816 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ross restarts HomeStore's allocator persists an allocation only once the consumer commits it (commit_blk), and after a restart the consumer must commit again the blocks its journal still references. CRAFT allocated with async_alloc_write and never committed: every restart, graceful or not, came back with all of a partition's blocks free, those the index references included, for the next writes to reuse. A write, a catch-up fill and a pre-resolution fill now commit their blocks once the journal slot that references them is durable. Restart recovery commits again the blocks of every live slot it rebuilds the overlay from (above the durable commit frontier, not Empty), before anything can allocate; a slot whose blocks can't be committed faults the partition. The slots at or below the frontier need no new commit: the checkpoint that set the frontier persisted it. Tests: after a restart the allocator holds the blocks of index-applied and journal-only slots, which read back after new writes; the recovery commits exactly the live slots above the frontier again, and faults if it can't. Co-Authored-By: Claude <noreply@anthropic.com>
…low the durable frontier too The durable image dropped every Empty verdict at or below the durable commit frontier: once the index there is durable, no apply needs them. But a client does. After a restart, a late write to a voided slot at or below the frontier passed the Empty check, found the slot below last_append_lsn and not missing, and was acked as already written though nothing was stored; a resolution retry by a client that lost the reply no longer listed the slot Empty. The image now keeps every verdict, as the in-memory set always did, and a restart restores them all. The codec accepts verdicts at or below the frontier. Tests: verdicts kept when a checkpoint moves the frontier past them; after a restart a write to a voided slot below the frontier is refused (EMPTY_SLOT); the codec takes them. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical recovery and lifecycle defects can erase durable state, expose freed blocks, and allow device access or background work to race teardown.
8 open issues
Prevent persistence after failed recovery or overlay rebuild · New Guard prerelease-only test helpers and callers · New Commit and restore live block allocations across restart · New Check destroy state before accessing craft_dev Prevent persistence after failed recovery Re-reserve live journal blocks during recovery Retain Empty verdicts until client acknowledgment Drain checkpoint tasks before teardown
Review effort: Max
| (void)logstores.open_log_store( | ||
| sb_->journal_logdev_id, sb_->journal_store_id, false /* append_mode */, nullptr /* log_found_cb */, | ||
| [replayed = std::move(replayed)](shared< homestore::home_log_store > journal, homestore::logstore_seq_num_t) { |
…, and persists nothing more A recovery that failed published the device faulted, but its next checkpoint still saved the device's state over the partition's record: default values if the image couldn't be loaded, the restored state without the entries a failed overlay rebuild never re-queued. The restart after that came back healthy, its session and Empty verdicts gone. A fault at runtime had the same end: checkpoints moved past entries the partition never applied, and a restart brought it back from an image that lacked them. Every fault now goes through fault(), which also marks the state record faulted (a flag in its header; the image, if any, stays for whoever looks at it). A record so marked doesn't load, so a restart faults the partition at once; a faulted partition persists nothing more, and its checkpoint save succeeds without writing. A partition faulted before it ever persisted gets a record that says so. Tests: a failed recovery and a runtime fault mark the record; nothing persists after a fault while a checkpoint save still succeeds; a marked record faults the next restart; the HomeStore store keeps the mark in the record a boot reads, with or without an image. Co-Authored-By: Claude <noreply@anthropic.com>
A commit that crosses the checkpoint interval detaches a round that waits for HomeStore's flush, then persists the durable frontier and frees the blocks it released. stop() didn't wait for it: after a destroy's stop the round could free blocks of chunks the volume was releasing, and after a shutdown's it could write the meta service while HomeStore was going down. stop() now waits for those rounds to end, and none starts after it. Test: stop() returns only once a held checkpoint round has completed and persisted its frontier; a commit after stop() starts none. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unreplayed or faulted replicas can retain RAFT leadership, causing every resolution of the affected volume to fail until a partition move replaces the member.
9 open issues
Protect journal records needed by queued frees from reclamation Propagate commit_data failures before publishing or indexing slots Handle faulted partitions during checkpoint persistence Retain serialized buffer until I/O completes Commit and restore live block allocations across restart Check destroy state before accessing craft_dev Make crash recovery test stop before the final applied-index persist · New Exclude unhealthy replicas from leadership eligibility Test abrupt-crash replay of committed entries beyond the checkpoint
Review effort: Max
ublkpp 0.41.0, which CI now builds, requires craft_client ^0.6, and craft_client's dev/v0.x branch is at 0.6.0: with ublkpp ^0.40 and craft_client ^0.5 the dependency step failed on every branch. Neither release changes what HomeBlocks uses. craft_client 0.6 adds member replacement to its client (client.hpp, tcp.hpp); HomeBlocks includes only types.hpp, status.hpp and peer.hpp. ublkpp 0.41 adds craft_disk::replace_member; the legacy ublk driver uses only ublk_disk. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The crash test races with the final Sync’s applied-index persistence, which can cause recovery to skip replay and time out; a synchronization barrier is needed to exercise the intended replay path deterministically.
9 open issues
Protect journal records needed by queued frees from reclamation Propagate commit_data failures before publishing or indexing slots Handle faulted partitions during checkpoint persistence Retain serialized buffer until I/O completes Commit and restore live block allocations across restart Check destroy state before accessing craft_dev Make crash recovery test stop before the final applied-index persist Exclude unhealthy replicas from leadership eligibility Test abrupt-crash replay of committed entries beyond the checkpoint
Review effort: Max
… partition A write, a pre-resolution fill and a catch-up fill commit their blocks to the allocator once the journal slot referencing them is durable, but ignored a failure to: the slot was published and later indexed. Once a checkpoint moved the durable frontier past it, a restart no longer committed its blocks again, and the index pointed at blocks the allocator could hand out. A failure now faults the partition, so nothing more is persisted, and fails the operation, keeping the blocks the journal references, as a journal write's timeout does. Co-Authored-By: Claude <noreply@anthropic.com>
…itted Sync A Sync's verdicts install at commit, but its apply frees the voided slots' blocks later, reading them from the slots' journal records. Meanwhile the commit steps over those slots, a checkpoint can persist that frontier and keep_alive can report it committed set-wide, so the reclaim could drop the records first and the blocks leaked. The floor now stays below every slot a committed (or, after a restart, re-queued) Sync still has to free, until its apply frees or leaks them. A write voided while it was journalled frees the blocks it holds instead of reading them back from its slot's record, which the reclaim may have passed. Co-Authored-By: Claude <noreply@anthropic.com>
…ip off The hand-off kept leadership on any member with the volume in its map, also one whose device faulted at recovery or later, or never came back at a restart (its journal didn't replay). Elected, such a member refused every resolution while the healthy members refused them as not the leader, and the group couldn't resolve. Now only a replica that serves, or whose device is still being reattached, keeps leadership (a create in progress still does too), and the check runs again when a device is published faulted, is given up, or faults while its member leads. A device's fault only notes its volume, for HomeBlocks' one-second timer to check: iomgr runs a message for the current reactor inline, and a fault may hold the device's locks. is_faulted() is production API now; fault_for_test() faults a device from the volume tests. Co-Authored-By: Claude <noreply@anthropic.com>
…not applied The first child killed itself once the last Sync's proposal completed, but the apply records the entry as applied and persists that just after it completes the proposal: a persist that landed before the kill made the restart rightly skip the replayed Sync, and the second child waited for a commit nothing would make. The apply is now paused before that Sync is proposed, from a thread of its own, and the child is killed once the Sync has committed, so only the replay of the RAFT log can bring it back. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Late publication of s_instance_ can drop device-fault callbacks during reattachment, bypassing fallback hand-off and leaving an elected leader unable to serve requests.
7 open issues
Recheck device faults that occur during instance reattachment · New Protect journal records needed by queued frees from reclamation Propagate commit_data failures before publishing or indexing slots Handle faulted partitions during checkpoint persistence Retain serialized buffer until I/O completes Commit and restore live block allocations across restart Test abrupt-crash replay of committed entries beyond the checkpoint
3 resolved since last review
Review effort: Max
| inst->finish_craft_reattach(); | ||
| inst->start_reaper_thread(); | ||
| HomeBlocksImpl::s_instance_ = inst; |
…s checked every second A device's fault reached HomeBlocks through its static instance, which init publishes only after every reattach. So a device that faulted while the entries held for it, or queued again at the restart, applied had its fault dropped, and a member elected while its device was being reattached kept leadership. The hook also read the instance on device threads while init wrote it. HomeBlocks' one-second timer now checks every CRAFT volume whose replica doesn't serve, and hands leadership off if this member leads its group, however and whenever the replica stopped serving. The fault hook and the reattach paths' own checks are gone. Co-Authored-By: Claude <noreply@anthropic.com>
…S8_volume_create_delete Conflict in craft_repl_dev.hpp: the device keeps owning its proposer (this branch), and append()'s single-flight flag becomes eBay#215's watermark_mu_. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects can hang shutdown indefinitely, keep leadership on offline volumes, and lose acknowledged writes after a crash.
10 open issues
Stop proposers before draining requests during shutdown · New Recover journal entries when no state image exists · New Exclude offline volumes from the serving predicate · New Recheck device faults that occur during instance reattachment Protect journal records needed by queued frees from reclamation Propagate commit_data failures before publishing or indexing slots Handle faulted partitions during checkpoint persistence Retain serialized buffer until I/O completes Commit and restore live block allocations across restart Test abrupt-crash replay of committed entries beyond the checkpoint
Review effort: Max
| state_change(volume_state::ONLINE); | ||
|
|
||
| attach_craft_dev(std::move(journal)); | ||
| craft_dev_ = make_craft_dev(std::move(journal), my_id); // a new partition: nothing to recover |
… its RAFT server 47ab7d5's one-second scan queues every volume whose replica doesn't serve, also one being destroyed, whose device is already gone. The check then asked the group whether this member leads it before asking whether the group was going away, and once HomeStore's reaper had run, is_leader() read the freed RAFT server: a heap-use-after-free under AddressSanitizer. The check now reads the group's stage first, and the scan leaves volumes being destroyed alone: their group is going away too. Co-Authored-By: Claude <noreply@anthropic.com>
…s restricted, hands leadership off Fault containment takes a volume offline without faulting its device, and the CRAFT API then refuses the volume's every op, as it does every op of a restricted node (global fault containment). Both still counted as serving, so an elected member kept leadership it couldn't use while the healthy members refused as not the leader. An offline volume no longer serves, a restricted node serves none, and HomeBlocks' timer checks them like any other replica that doesn't serve; a volume back online serves again. Co-Authored-By: Claude <noreply@anthropic.com>
…its journal A new partition persists its state first at its first login or checkpoint, but a write at term 0 needs no login, so it can be acked before then. A crash in between left no record, and the restart returned on the missing record without reading the journal: the write was lost and its blocks, whose allocation no checkpoint had persisted, free for reuse. A partition without a record now starts from the default image, and its journal is read, served and its blocks committed again as any other's. Co-Authored-By: Claude <noreply@anthropic.com>
…ts for the requests in flight With craft_raft_propose_timeout_ms = 0 a proposal has no deadline, and a leader without a quorum keeps its proposal, and the CRAFT op waiting on it, in flight for good. The shutdown waited for every request in flight before it stopped the devices, whose stop ends their proposals: it never got there. It now ends every device's proposals right after it refuses new ops; they complete timed_out, as at a deadline, and later ones fail. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Critical races involving RAFT teardown and journal reclamation, plus a possible indefinite shutdown wait, require human assessment of lifecycle safety.
8 open issues
Recover journal entries when no state image exists Recheck device faults that occur during instance reattachment Protect journal records needed by queued frees from reclamation Propagate commit_data failures before publishing or indexing slots Handle faulted partitions during checkpoint persistence Retain serialized buffer until I/O completes Commit and restore live block allocations across restart Test abrupt-crash replay of committed entries beyond the checkpoint
2 resolved since last review
Review effort: Max
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical recovery and lifecycle bugs can cause live-block reuse after a second crash, prevent snapshot synchronization after restart, and crash nodes during concurrent leadership hand-off and group teardown.
13 open issues
Checkpoint recovery before allocator registration can lose block allocations · New Compacted-group restart loses snapshot context needed by lagging peers · New Concurrent group destruction can free the RAFT server during handoff · New Recover journal entries when no state image exists Recheck device faults that occur during instance reattachment Protect journal records needed by queued frees from reclamation Propagate commit_data failures before publishing or indexing slots Handle faulted partitions during checkpoint persistence Retain serialized buffer until I/O completes Commit and restore live block allocations across restart Empty verdict retention grows memory and checkpoint costs without bound · New Zero journal timeout can leave shutdown waiting forever on lost completions · New Test abrupt-crash replay of committed entries beyond the checkpoint
Review effort: Max
| iomanager.run_on_forget(iomgr::reactor_regex::random_worker, [this, vol = std::move(vol), journal]() mutable { | ||
| detail::detach(reattach_craft_volume(std::move(vol), std::move(journal))); |
|
|
||
| homestore::async_status CraftGroupListener::create_snapshot(std::shared_ptr< homestore::snapshot_context > context) { | ||
| std::lock_guard lk{snapshot_mu_}; | ||
| if (!last_snapshot_ || context->get_lsn() > last_snapshot_->get_lsn()) { last_snapshot_ = std::move(context); } |
| auto const rd = homestore::hs()->repl_service().get_repl_dev(id); | ||
| // Its stage before its RAFT server: once a destroyed group's reaper has run, the server is freed, and is_leader() | ||
| // would read freed memory. | ||
| if (!rd || is_going_away(**rd) || !(*rd)->is_leader() || (*rd)->get_replication_quorum().size() < 2) { |
| int64_t last_applied_raft_idx{-1}; // RAFT entries at or below were applied: a replay of one is skipped | ||
| int64_t durable_commit_lsn{-1}; // commit frontier covered by a completed checkpoint | ||
| int64_t sync_watermark{-1}; // highest SyncRSCommitLSN rs_commit_lsn applied; last_append_lsn >= this | ||
| std::set< int64_t > empty_lsns{}; // every Empty verdict, as empty_lsns_ (a late write or a resolution retry asks) |
| auto f = shutdown_start(); | ||
| // After the flag, which refuses new CRAFT ops: a request in flight may be waiting on a proposal that only a stop | ||
| // would end (no deadline, no quorum), and the wait below waits for it. | ||
| stop_craft_proposals(); |


S8 (SDSTOR-22745): the CRAFT volume lifecycle after create. A CRAFT partition's state now survives a restart (SDSTOR-22895, SDSTOR-22911), its journal is reclaimed, RAFT snapshot transfers are refused until CRAFT can serve them (SDSTOR-22897), member changes are stubbed (SDSTOR-22912), and each partition's device proposes through its volume's RAFT group (the rest of SDSTOR-22894). It builds on #211's apply queue; the persistence and the reclaim are ported from the closed S8a/S8b PRs (#201, #202), with their review fixes. Deleting a CRAFT volume is not part of it (SDSTOR-26581).
1. What a partition keeps on disk
Problem: A CRAFT partition kept its session, its Empty verdicts and its commit frontier in memory only. A restart came back with no session and
commit_lsnat -1, and forgot the verdicts: a replayed Sync would then free a voided slot's blocks a second time.Solution:
CraftDurableState, written ahead of every change it protects, in a meta blk record of its own per partition (CraftState):asks again;
flowchart LR R["RAFT log (HomeStore)<br/>replayed above checkpoint_lsn"] -- "committed entries" --> D["CraftReplDev"] D -- "client writes" --> J["CRAFT journal<br/>data slots"] D -- "written ahead of each change" --> S["CraftState meta blk<br/>session, last applied RAFT index,<br/>durable frontier, Sync watermark,<br/>Empty verdicts, pending entries"] D -- "slots it commits" --> I["Index table"]Write-ahead points:
InternalLoginpersists the session before admitting it. If that fails, the session isn't admitted, and a restart applies the login again.One writer persists; concurrent callers share a round. A state record or journal tail that can't be read faults the partition. A partition that never persisted a record (a new one: a write at term 0 needs no login) restarts from the default image and reads its journal like any other.
Data blocks. HomeStore's allocator persists an allocation only once its consumer commits it (
commit_blk), and after a restart the consumer commits again the blocks its journal still references. A write, a catch-up fill and a pre-resolution fill commit their blocks once the journal slot referencing them is durable, and if they can't, the partition faults rather than publish a slot whose blocks the allocator may hand out again; the restart recovery commits again those of every live slot above the durable frontier that no checkpoint persisted. The allocator holds the others since its load, and once HomeStore has started it refuses to persist an allocation again.Faults are durable. A partition that faults (a failed recovery, or an apply it can't complete) marks its state record faulted and persists nothing more: a restart then faults it at once, rather than bringing it back from an image that lacks what it applied since.
Fixed on the way (H9 from S8a): a Sync's apply moved
commit_lsnover the slots it resolved without applying them to the index, so a read could return the previous data. The apply now advances throughcommit().2. Restart: recover before anything applies
Problem: During its start, HomeStore's RAFT log replay and the group rejoin deliver committed entries, which it never delivers again. The device can recover only once its journal is replayed, and HomeStore reports a journal's store open before replaying it, with the log device locked.
Solution: From its load, each CRAFT volume holds the entries committed for it, and its reattach is counted, so every checkpoint waits for it. When its journal's replay is done, a worker makes the device, recovers it, hands it the held entries in log order, and publishes it.
init_homeblocksreturns once every partition is recovered, or faulted (counted incraft_recoveries_failed).sequenceDiagram participant HS as HomeStore start participant HB as HomeBlocks participant V as Volume participant W as Worker participant D as CraftReplDev HS->>HB: meta blks: every CraftState record, then the volumes HB->>V: load: take its record, reopen its journal Note over V: from here on it holds what commits for it HS->>V: RAFT log replay and rejoin: committed entries V->>V: hold them, in log order HS->>HB: journal replay done HB->>W: reattach, off HomeStore's thread W->>D: make it: journal, state store, proposer D->>D: recover(): restore the state, rebuild the overlay from the journal, commit the live slots' blocks again, queue the pending entries W->>D: hand over the held entries, in log order W->>V: publish the device HB->>HB: init_homeblocks returns once every reattach is doneEvery way an entry can arrive after a restart (the saved pending entries, the RAFT replay, the hold, a live commit) goes through the same check: an entry at or below the last RAFT index the state counts as applied is skipped, as its effect is in the restored state. Pending entries apply again without their commit-time change, which the state already holds. A Sync still queued at the restart frees the slots its verdicts voided when it applies, their blocks kept allocated until then; the one being applied comes without them, as its frees may have happened: a leak is better than a double free.
A group without a replica here, or whose replica is being destroyed, has its entries dropped, logged once per group. HomeStore never reports the replay of a journal whose log store is missing: when init ends, such a partition is faulted for good, its record marked faulted before its count lets a checkpoint cover the entries held for it, which are dropped. A later boot whose journal does replay faults it again, rather than bring it back from a record that lacks them.
3. A checkpoint never covers an entry whose effect isn't durable
Problem: After a restart HomeStore replays a RAFT group's log only above its last checkpoint, and a checkpoint covers every entry committed when it starts. CRAFT applies entries asynchronously, so a checkpoint could cover an entry not applied yet, and a crash then lost it for good (a login, for one, left the old session in place).
Solution: An
HS_CLIENTcheckpoint consumer, which HomeStore flushes before the replication service records the checkpoint. It first waits for pending reattaches, so held entries are in their devices, then saves every partition's state with its pending entries. A failed save stops the process: HomeStore records the checkpoint whatever the flush returns. A partition faulted before or during its save (a journal write's timeout, say) saves nothing and fails nothing, once its record is durably marked faulted. The consumer is registered while HomeStore starts, so this holds for the checkpoint that ends the start too.sequenceDiagram participant CP as HomeStore checkpoint participant C as HomeBlocks consumer (HS_CLIENT) participant D as CraftReplDev, each partition participant RS as Replication service CP->>C: flush C->>C: wait for pending reattaches loop every CRAFT volume, except one being destroyed or without its device yet C->>D: save_state(): the state, with the entries committed but not applied yet end C-->>CP: flushed (a failed save stops the process, a faulted partition saves nothing) CP->>RS: flushed last: records checkpoint_lsn Note over CP,RS: after a crash, HomeStore replays above checkpoint_lsn, and the saved pending entries are queued again4. Journal reclaim
Problem: A partition's journal only grew.
Solution: It is reclaimed below
min(persisted durable frontier, all_committed_lsn), and below every slot a committed Sync has yet to free:all_committed_lsnevery member has committed, so no catch-up needs them;all_committed_lsnis the client's word. The min with this member's own frontier keeps a wrong value from reclaiming anything a restart here still needs. A peer's fetch of a reclaimed slot fails the whole batch instead of reporting the slot as not held, so it never counts toward an Empty verdict for a committed slot. The reclaim runs when a persist moves the frontier and on eachkeep_alive, one runner at a time; without a state store nothing is reclaimed. The HomeStore backend truncates its log store, never past the last entry. The in-memory Empty verdicts below the floor are kept: one proposed again would otherwise fault the member.5. Stopping a device: shutdown and destroy
Problem: A shutdown stopped none of a device's own work (its apply queue, journal reclaim, watchdog), so an apply in progress could touch HomeStore after it shut down. A destroy left a running reclaim truncating the journal it was removing.
Solution:
CraftReplDev::stop()stops all of it, waiting out the apply, the reclaim and the checkpoint rounds in progress, and ends the proposer. A shutdown ends every device's proposals first, right after it refuses new ops and before it waits for the requests in flight: withcraft_raft_propose_timeout_msat 0 (no deadline), a leader without a quorum would otherwise hold its CRAFT op, and the shutdown, forever. A shutdown keeps the queued entries, and any that commit later, for HomeStore's last checkpoint to save; a destroy drops them. A journal write HomeStore never completes (its device write failed, SDSTOR-24993) times out aftercraft_journal_write_timeout_msand faults the partition, so a stop never waits on it forever.flowchart TD subgraph SD["Shutdown"] S1["requests drained"] --> S3["each device stops, keeping its queue"] --> S2["HomeBlocks' timer canceled"] --> S4["HomeStore shuts down: the last checkpoint saves each state and queue"] --> S5["iomgr stops"] end subgraph DS["Destroy of a replica"] D1["held entries dropped"] --> D2["the device stops, dropping its queue"] --> D3["CraftState record removed, first on disk"] --> D4["index table, journal, superblock"] endThe record goes first: a crash in between leaves a destroying volume without a record, never a record without a volume. A destroy resumed at boot removes the record the volume took. A record no volume takes at boot is logged and left in place.
6. Proposals through the volume's RAFT group
Problem: Nothing wired a RAFT proposer to a volume, so every leader-side round failed
not_supported.Solution: Each device owns a HomeStore RAFT proposer over its volume's repl_dev, made with the device at create and at the reattach. A resolution on the leader (
request_resolution(),append()) commits itsSyncRSCommitLSNthrough the group; a follower answersNOT_LEADER.sequenceDiagram participant Cl as Client participant D as Leader's CraftReplDev participant P as Its proposer participant G as RAFT group participant T as HomeBlocks timer, every second Cl->>D: request_resolution(upto) D->>P: propose SyncRSCommitLSN P->>G: key-only entry G-->>D: committed, applied in log order D-->>Cl: resolved_upto, Empty slots Note over P,T: without a quorum, the timer completes it with timed_out after craft_raft_propose_timeout_ms (it may still commit)The deadlines, of proposals and of journal writes, are expired by one HomeBlocks timer on its own reactors, armed at init and canceled at shutdown once the devices have stopped, not by a timer per proposer. iomgr arms and cancels a recurring timer on every worker with a blocking round on all of them; two reattaches doing that at once on workers hung init and the checkpoint waiting for them. Stopping a proposer completes its pending proposals with
timed_outand refuses new ones withoperation_canceled.7. Members without a replica, snapshots, member changes
A member whose local create fails stays in the group (it votes and acks) without a replica: the failure is reported, not retried, and a partition move replaces the member. The node gauge
craft_groups_without_replicacounts such groups at each scrape, leaving out creates in flight here and groups being destroyed. RAFT can still elect such a member, as it has the whole log, and as leader it would fail every CRAFT op of the volume. When elected, it hands leadership off (craft_leaderships_handed_off), except while a create of the volume runs here or when it is the group's only member. A member whose replica can't serve the volume hands it off too: its device faulted, at a restart's recovery or later, or never came back (its journal didn't replay), or fault containment took the volume, or the whole node, offline (the CRAFT API then refuses every op). It checks when elected, and HomeBlocks' timer checks every second while the replica doesn't serve, which covers a device that faults while its member leads or fails its recovery after the election; one whose device is still being reattached keeps leadership. nuraft nominates the responsive member of the highest priority, normally the group's creator, which holds a replica; with the creator unresponsive, two members without one would nominate each other forever. So a member hands a group's leadership off at most once percraft_leadership_handoff_backoff_ms(10 s): elected again within it, it keeps leadership, and HomeBlocks' timer hands it off again once the backoff is over. A hand-off is counted against the shutdown, like a create, and a create of the volume that fails here checks the leadership again.sequenceDiagram participant L as Leader participant B as Member B participant M as Metrics scrape L->>B: CreateVolume commits B->>B: local create fails: reported, B stays in the group M->>B: craft_groups_without_replica = 1 Note over B: later elected leader, as it has the whole log B->>B: no replica here: yields leadership, nuraft picks the next leaderRAFT snapshots. HomeStore keeps compacting each group's log, and the CRAFT listener inherited no-op snapshot callbacks. Every member would segfault at its first snapshot (nuraft dereferences
last_snapshot(), which was null), and a member behind the compaction got an endless stream of empty objects. The listener now keeps the snapshot's context, and a leader refuses transfers:read_snapshot_objreturns -1 and nuraft asks again later, counted incraft_snapshot_transfers_refusedand logged once per group. Such a member stays behind until a partition move replaces it. HomeStore records a transfer's snapshot index on that member as soon as its first object arrives, and from then on skips its own entries at or below it, at a restart too: refused, the member only falls further behind. A snapshot that arrives anyway is not applied:apply_snapshotfails, and nuraft stops the process.sequenceDiagram participant L as Leader participant B as Member behind the compaction L->>L: log compacted past B's last entry B->>L: needs a snapshot L-->>B: refused (read_snapshot_obj = -1), asked again later Note over B: stays behind until a partition move replaces itMember changes.
home_blocks::replace_member(volume_id, out, in)answersnot_supporteduntil reconfiguration exists. The listener logs HomeStore's member-change callbacks instead of ignoring them.Changes, by commit
replace_memberstub; member-change callbacks logVolumeCraftTest)craft_groups_without_replicarecover()applies them againCraftStatemeta blk record per partition; recovery once HomeStore has startedCraftReplDev::stop()at shutdown and destroystop()waits out the checkpoint rounds in flight (review)_PRERELEASE(review)Tests
73 new cases, and existing ones extended:
CraftRaftEntriesTest,CraftDurableStateCodec, and others): each write-ahead point and its failure, the shared writer, the durable frontier, deferred and replayed frees, a restart from a state image and its faults, or without one, pending entries, the reclaim floor and the frees it waits for,stop().CraftHomeStoreBackendTest): the state record in the real meta service, the journal reclaim and tail, a write past its deadline whose entry the log writes later, whole.CraftVolumeTest):CraftVolumeMultiTest):VolumeCraftTest: the volume lifecycle suite in CRAFT mode.Linux, on 408a815 (after merging #215; craft_client 0.6.0): Debug 18/18 suites pass, #215's 22 new tests included, and every HomeBlocks translation unit compiles without
_PRERELEASE, as in a Release build. AddressSanitizer, rebuilt clean: 17/18,CraftVolumeMultiTestfailing only on the known nuraft_mesg shutdown leak below (its 11 cases pass on all 3 members).Mutation checks: breaking each guard named in the commit messages fails a test, except three that only a crash or unschedulable timing shows: a checkpoint that doesn't wait for reattaches, a shutdown that doesn't stop the devices, and the proposal timer's cancel at shutdown (iomgr removes it when it stops anyway).
Notes
ACrashReplaysWhatCommittedPastTheCheckpoint); a member of a multi-member group crashing is not covered yet.iomgr_flip::instance()(static in a header) replaces the timer of every delay flip, so one still pending fires on a destroyed object and hangs its reactor. The reattach hold uses a return-value flip and a thread instead; the iomgr fix is outside this PR.all_committed_lsnstops moving. Planned with S10.remove_volume(SDSTOR-26581; a group destroy through HomeStore removes each replica), real snapshots and reconfiguration (SDSTOR-22915), login (SDSTOR-22744). Watchdog rounds and pre-resolution of missing slots need the peer fetcher (S9); the periodic Sync needs the client's commit LSN.keep_aliveafter a login, andstop()cancels it the same way. Nothing arms it until login exists; it should move to a shared timer with login.CraftVolumeMultiTestcan fail on a leak in nuraft_mesg's shutdown (SDSTOR-26738), about one run in three; every test case passes.^0.41and craft_client^0.6, as dev/v6.x requires since SDSTOR-22891 craft: login's SyncRSCommitLSN; one watermark holder at a time #215: ublkpp 0.41.0 requires craft_client 0.6. Neither release changes what HomeBlocks uses (both add member replacement to the client side).api.mdisn't updated yet: the CRAFT lifecycle and these limitations follow in this PR.Generated with Claude Code