Skip to content

craft: reclaim the journal below the durable, replica-wide floor (S8b) - #202

Closed
Ilyin-Ilya wants to merge 1 commit into
resync/s8-persistencefrom
resync/s8-reclaim
Closed

Ilyin-Ilya wants to merge 1 commit into
resync/s8-persistencefrom
resync/s8-reclaim

Conversation

@Ilyin-Ilya

Copy link
Copy Markdown
Collaborator

Part two of S8 (SDSTOR-22745): the CRAFT journal is reclaimed below a floor that neither local restart nor any peer's catch-up still needs. Design: docs/craft/resync.md §4.8 (#198).

Stacked on #201 (S8a), because the floor uses S8a's persisted durable commit frontier.

Problem

The journal grows without bound. all_committed_lsn was captured from every client header but never used (TODO(S8)). Two constraints make the floor non-trivial:

  • Local restart. S8a restarts commit_lsn at the last frontier a completed checkpoint made durable and re-applies everything above it from the journal. An entry above that frontier must not be reclaimed.
  • Peers. A lagging peer catches up by fetching slots from us. An entry some member has not committed must stay. A snapshot source (b′) must also keep everything it still has to stream.

all_committed_lsn is client input. On its own it could reclaim entries we have not made durable.

sequenceDiagram
    participant C as UBD
    participant R as Replica (BM)
    participant J as Journal
    Note over R: durable commit (checkpoint) = 4, commit_lsn = 10
    C->>R: keep_alive all_committed_lsn = 10
    R->>J: reclaim ≤ 10 (if the floor were all_committed_lsn alone)
    R--xR: crash / restart
    Note over R: commit restarts at 4, slots 5..10 are gone → index missing them
Loading

Diagram

sequenceDiagram
    participant C as UBD
    participant R as Replica (BM)
    participant S as Snapshot source (b′)
    participant J as Journal
    Note over R: flushed checkpoint → persisted durable commit = 10
    C->>R: keep_alive all_committed_lsn = 50
    S->>R: pin_journal(4)
    Note over R: floor = min(10, 50, 4 − 1) = 3
    R->>J: reclaim_upto(3)
    S-->>R: pin released
    Note over R: floor = min(10, 50) = 10
    R->>J: reclaim_upto(10)
    Note over R: peer asks fetch_range [5, 12]
    R-->>R: serve 11, 12, reclaimed_below = 11 → peer needs a snapshot
Loading

Solution

  • Floor = min(persisted durable commit, all_committed_lsn, lowest pin − 1).
    • The durable term uses the frontier as last successfully persisted, not the in-memory image, which runs ahead when a persist fails.
    • Taking the min with our own durable frontier means a too-large client value can never reclaim anything not yet durable here. At worst it sends a lagging peer to the snapshot path: a liveness cost, not a safety one.
    • Without a state store a restart recovers nothing from disk, so the journal is the only copy and nothing is reclaimed.
  • One reclaim runner, triggered after a durable checkpoint, on keep_alive, and when a pin is released. It loops while the floor rises. A failure is retried on the next trigger; a backend that reports not_supported is not asked again.
  • pin_journal(from_lsn): a handle that keeps every entry ≥ from_lsn alive, for the b′ snapshot source. It is refused for a range already reclaimed or being reclaimed (the in-flight target is tracked), and is safe to release after the device is gone.
  • fetch_range serves what is above the reclaim point (including one in flight) and reports the rest as reclaimed_below. E1's caller already treats that as "needs a baseline", not as something to retry.
  • Empty verdicts at or below the floor are dropped from memory, which bounds empty_lsns_.
  • CraftJournalBackend::reclaim_upto / reclaimed_upto: the HomeStore backend uses home_log_store::truncate / truncated_upto. Restart reads the reclaim point back from the journal.

RAFT-log compaction stays independent of this floor under b′ (§4.8). The retention cap that relaxes all_committed_lsn for a long-dead member is not in this PR.

Tests

  • 11 new tests in test_craft_raft_entries.cpp:
    • the floor follows the lower of its two inputs;
    • no reclaim without a store, or on a frontier that is not persisted;
    • pins hold the floor and releasing one resumes the reclaim;
    • pins and fetch_range respect a reclaim in flight;
    • fetch_range reports reclaimed_below;
    • retry after a failure; no retry when unsupported;
    • verdicts are pruned;
    • the reclaim point is recovered on restart;
    • a pin can outlive its device.
  • 1 new test in test_craft_homestore_backend.cpp against the real log store.
  • All 14 suites pass on the Linux build server (Debug + coverage).
  • src/lib/craft/ line coverage is 90% (craft_repl_dev.cpp 96%). Every uncovered line is pre-existing, apart from the backend interface's trivial default reclaimed_upto().
  • Version 6.0.21.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent reclamation can delete retained journal slots during a fetch and cause the entire fetch to fail, while a stale durable-commit cache can prevent reclamation after persistence succeeds.

2 open issues
What changed in this PR

Adds CRAFT journal reclamation using S8a’s persisted durable commit frontier, replica-wide commit progress, and snapshot pins to bound journal retention while preserving restart safety.

Changes:

  • Adds a single reclaim runner, journal pins, and reclaim-boundary reporting for peer fetches.
  • Implements HomeStore journal truncation and restores the reclaim point on restart.
  • Adds reclamation tests and bumps the package version to 6.0.21.
File Description
src/​lib/​craft/​tests/​test_craft_raft_entries.cpp Tests reclaim floors, pins, failures, fetch boundaries, and restart behavior.
src/​lib/​craft/​tests/​test_craft_homestore_backend.cpp Tests prefix reclamation against the real log store.
src/​lib/​craft/​craft_repl_dev.hpp Defines backend reclaim operations, pin APIs, and reclaim state.
src/​lib/​craft/​craft_repl_dev.cpp Implements reclamation, triggers, fetch filtering, and restart restoration.
conanfile.py Bumps the package version to 6.0.21.

Review effort: Balanced

Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
- Floor = min(persisted durable commit, all_committed_lsn, lowest pin - 1).
  Taking the min with our own durable frontier keeps a bogus client
  all_committed_lsn from reclaiming anything not yet durable here. No state
  store: no reclaim (the journal is then the only copy).
- The persisted durable commit follows every successful persist round (not
  only the checkpoint's own), so a frontier whose first persist failed still
  becomes a floor.
- One reclaim runner, triggered after a durable checkpoint, on keep_alive, on
  a persisted frontier and on pin release. A failure is retried on the next
  trigger; a backend that reports not_supported is not asked again.
- pin_journal(from_lsn): a handle that keeps entries >= from_lsn (for the b'
  snapshot source). Refused for a range already reclaimed or being reclaimed.
- fetch_range serves what is above the reclaim point, under a pin for the
  duration of the read, and reports the rest as reclaimed_below.
- Verdicts at or below the floor are dropped from memory.
- CraftJournalBackend::reclaim_upto / reclaimed_upto (HomeStore:
  home_log_store::truncate / truncated_upto); restart reads the reclaim point
  back from the journal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Three critical retention races can expose freed payload data or permanently leak blocks because journal reclamation does not protect pinned payloads, pending frees, or in-flight catch-up installs.

3 open issues
2 resolved since last review

Review effort: Balanced

Comment on lines +2127 to +2128
int64_t floor = std::min(persisted_durable_commit_, state_.all_committed_lsn);
if (!reclaim_pins_.empty()) floor = std::min(floor, *reclaim_pins_.begin() - 1);
{
std::lock_guard lk{state_mu_};
if (from_lsn <= std::max(reclaimed_upto_, reclaim_inflight_upto_)) return nullptr;
reclaim_pins_.insert(from_lsn);
}
reclaim_inflight_upto_ = floor;
}
auto r = co_await journal_->reclaim_upto(floor);
@Ilyin-Ilya Ilyin-Ilya closed this Nov 12, 2026
shosseinimotlagh added a commit to shosseinimotlagh/HomeBlocks that referenced this pull request Nov 18, 2026
…ontier and the set-wide commit

A CRAFT partition's journal only grew: nothing dropped the entries no member
needs anymore.

It is now reclaimed up to min(the durable commit frontier as persisted,
all_committed_lsn). Below the persisted frontier the index is durable, so a
restart no longer applies those slots again; below all_committed_lsn every
member has committed, so no member's catch-up needs them. all_committed_lsn
is the client's word, and the min with this member's own frontier keeps a
wrong value from reclaiming anything a restart here still needs. One that is
too large can leave a lagging member unable to catch up, but a peer's fetch of
a reclaimed slot fails the whole batch instead of reporting the slot as not
held, so no leader can count it toward verdicting a committed slot Empty.

The reclaim runs when a round persists a higher durable frontier and on each
keep_alive, one runner at a time. Without a state store nothing is persisted,
so nothing is reclaimed. A restart reads the reclaim point back from the
journal. The HomeStore backend truncates its log store, never past the last
entry (that would move the tail); HomeStore's periodic device truncation
releases the space.

Tests: the floor and each of its inputs, a frontier persisted only later, a
backend that can't reclaim or fails once, fetching a reclaimed slot, a
restart; the backend against a real log store. Breaking any of these guards
fails a test.

Port of S8b's journal reclaim from the closed PR eBay#202, without its journal
pins and range fetches, and without dropping the in-memory Empty verdicts
below the floor: a verdict for such a slot proposed again would then fault
the member.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants