Repository navigation
craft: reclaim the journal below the durable, replica-wide floor (S8b) - #202
Closed
Ilyin-Ilya wants to merge 1 commit into
Closed
Ilyin-Ilya wants to merge 1 commit into
Ilyin-Ilya wants to merge 1 commit into
Conversation
Ilyin-Ilya
force-pushed
the
resync/s8-reclaim
branch
from
November 11, 2026 19:58
5193b7a to
0e8d81c
Compare
Ilyin-Ilya
force-pushed
the
resync/s8-persistence
branch
from
November 11, 2026 19:58
57bcad0 to
fa8ecce
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent reclamation can delete retained journal slots during a fetch and cause the entire fetch to fail, while a stale durable-commit cache can prevent reclamation after persistence succeeds.
2 open issues
What changed in this PR
Adds CRAFT journal reclamation using S8a’s persisted durable commit frontier, replica-wide commit progress, and snapshot pins to bound journal retention while preserving restart safety.
Changes:
- Adds a single reclaim runner, journal pins, and reclaim-boundary reporting for peer fetches.
- Implements HomeStore journal truncation and restores the reclaim point on restart.
- Adds reclamation tests and bumps the package version to 6.0.21.
| File | Description |
|---|---|
src/lib/craft/tests/test_craft_raft_entries.cpp |
Tests reclaim floors, pins, failures, fetch boundaries, and restart behavior. |
src/lib/craft/tests/test_craft_homestore_backend.cpp |
Tests prefix reclamation against the real log store. |
src/lib/craft/craft_repl_dev.hpp |
Defines backend reclaim operations, pin APIs, and reclaim state. |
src/lib/craft/craft_repl_dev.cpp |
Implements reclamation, triggers, fetch filtering, and restart restoration. |
conanfile.py |
Bumps the package version to 6.0.21. |
Review effort: Balanced
Ilyin-Ilya
force-pushed
the
resync/s8-reclaim
branch
from
November 11, 2026 20:13
0e8d81c to
7a4898e
Compare
Ilyin-Ilya
force-pushed
the
resync/s8-persistence
branch
from
November 11, 2026 20:13
fa8ecce to
94d9ffa
Compare
- Floor = min(persisted durable commit, all_committed_lsn, lowest pin - 1). Taking the min with our own durable frontier keeps a bogus client all_committed_lsn from reclaiming anything not yet durable here. No state store: no reclaim (the journal is then the only copy). - The persisted durable commit follows every successful persist round (not only the checkpoint's own), so a frontier whose first persist failed still becomes a floor. - One reclaim runner, triggered after a durable checkpoint, on keep_alive, on a persisted frontier and on pin release. A failure is retried on the next trigger; a backend that reports not_supported is not asked again. - pin_journal(from_lsn): a handle that keeps entries >= from_lsn (for the b' snapshot source). Refused for a range already reclaimed or being reclaimed. - fetch_range serves what is above the reclaim point, under a pin for the duration of the read, and reports the rest as reclaimed_below. - Verdicts at or below the floor are dropped from memory. - CraftJournalBackend::reclaim_upto / reclaimed_upto (HomeStore: home_log_store::truncate / truncated_upto); restart reads the reclaim point back from the journal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ilyin-Ilya
force-pushed
the
resync/s8-reclaim
branch
from
November 11, 2026 20:35
7a4898e to
5d430b0
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Three critical retention races can expose freed payload data or permanently leak blocks because journal reclamation does not protect pinned payloads, pending frees, or in-flight catch-up installs.
3 open issues
2 resolved since last review
Review effort: Balanced
Comment on lines
+2127
to
+2128
| int64_t floor = std::min(persisted_durable_commit_, state_.all_committed_lsn); | ||
| if (!reclaim_pins_.empty()) floor = std::min(floor, *reclaim_pins_.begin() - 1); |
| { | ||
| std::lock_guard lk{state_mu_}; | ||
| if (from_lsn <= std::max(reclaimed_upto_, reclaim_inflight_upto_)) return nullptr; | ||
| reclaim_pins_.insert(from_lsn); |
| } | ||
| reclaim_inflight_upto_ = floor; | ||
| } | ||
| auto r = co_await journal_->reclaim_upto(floor); |
shosseinimotlagh
added a commit
to shosseinimotlagh/HomeBlocks
that referenced
this pull request
Nov 18, 2026
…ontier and the set-wide commit A CRAFT partition's journal only grew: nothing dropped the entries no member needs anymore. It is now reclaimed up to min(the durable commit frontier as persisted, all_committed_lsn). Below the persisted frontier the index is durable, so a restart no longer applies those slots again; below all_committed_lsn every member has committed, so no member's catch-up needs them. all_committed_lsn is the client's word, and the min with this member's own frontier keeps a wrong value from reclaiming anything a restart here still needs. One that is too large can leave a lagging member unable to catch up, but a peer's fetch of a reclaimed slot fails the whole batch instead of reporting the slot as not held, so no leader can count it toward verdicting a committed slot Empty. The reclaim runs when a round persists a higher durable frontier and on each keep_alive, one runner at a time. Without a state store nothing is persisted, so nothing is reclaimed. A restart reads the reclaim point back from the journal. The HomeStore backend truncates its log store, never past the last entry (that would move the tail); HomeStore's periodic device truncation releases the space. Tests: the floor and each of its inputs, a frontier persisted only later, a backend that can't reclaim or fails once, fetching a reclaimed slot, a restart; the backend against a real log store. Breaking any of these guards fails a test. Port of S8b's journal reclaim from the closed PR eBay#202, without its journal pins and range fetches, and without dropping the in-memory Empty verdicts below the floor: a verdict for such a slot proposed again would then fault the member. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Part two of S8 (SDSTOR-22745): the CRAFT journal is reclaimed below a floor that neither local restart nor any peer's catch-up still needs. Design:
docs/craft/resync.md§4.8 (#198).Stacked on #201 (S8a), because the floor uses S8a's persisted durable commit frontier.
Problem
The journal grows without bound.
all_committed_lsnwas captured from every client header but never used (TODO(S8)). Two constraints make the floor non-trivial:commit_lsnat the last frontier a completed checkpoint made durable and re-applies everything above it from the journal. An entry above that frontier must not be reclaimed.all_committed_lsnis client input. On its own it could reclaim entries we have not made durable.sequenceDiagram participant C as UBD participant R as Replica (BM) participant J as Journal Note over R: durable commit (checkpoint) = 4, commit_lsn = 10 C->>R: keep_alive all_committed_lsn = 10 R->>J: reclaim ≤ 10 (if the floor were all_committed_lsn alone) R--xR: crash / restart Note over R: commit restarts at 4, slots 5..10 are gone → index missing themDiagram
sequenceDiagram participant C as UBD participant R as Replica (BM) participant S as Snapshot source (b′) participant J as Journal Note over R: flushed checkpoint → persisted durable commit = 10 C->>R: keep_alive all_committed_lsn = 50 S->>R: pin_journal(4) Note over R: floor = min(10, 50, 4 − 1) = 3 R->>J: reclaim_upto(3) S-->>R: pin released Note over R: floor = min(10, 50) = 10 R->>J: reclaim_upto(10) Note over R: peer asks fetch_range [5, 12] R-->>R: serve 11, 12, reclaimed_below = 11 → peer needs a snapshotSolution
min(persisted durable commit, all_committed_lsn, lowest pin − 1).keep_alive, and when a pin is released. It loops while the floor rises. A failure is retried on the next trigger; a backend that reportsnot_supportedis not asked again.pin_journal(from_lsn): a handle that keeps every entry ≥from_lsnalive, for the b′ snapshot source. It is refused for a range already reclaimed or being reclaimed (the in-flight target is tracked), and is safe to release after the device is gone.fetch_rangeserves what is above the reclaim point (including one in flight) and reports the rest asreclaimed_below. E1's caller already treats that as "needs a baseline", not as something to retry.empty_lsns_.CraftJournalBackend::reclaim_upto/reclaimed_upto: the HomeStore backend useshome_log_store::truncate/truncated_upto. Restart reads the reclaim point back from the journal.RAFT-log compaction stays independent of this floor under b′ (§4.8). The retention cap that relaxes
all_committed_lsnfor a long-dead member is not in this PR.Tests
test_craft_raft_entries.cpp:fetch_rangerespect a reclaim in flight;fetch_rangereportsreclaimed_below;test_craft_homestore_backend.cppagainst the real log store.src/lib/craft/line coverage is 90% (craft_repl_dev.cpp96%). Every uncovered line is pre-existing, apart from the backend interface's trivial defaultreclaimed_upto().🤖 Generated with Claude Code