Repository navigation
craft: login poll raises the fence before reporting LSNs (quiesce) - #199
Ilyin-Ilya wants to merge 4 commits into
Conversation
get_rs_commit_lsn(term, is_login=true) now raises fence_term to `term`
(monotonic, and only for a term newer than the current session) in the same
critical section that snapshots {commit_lsn, last_append_lsn}. Every write
acked before the poll is covered by the reported last_append_lsn, and no
write of the old session can be acked after it: write()'s pre- and
post-flight checks both require term >= fence_term. Non-login polls change
nothing. No timer lifts the fence; only an applied InternalLogin or
LoginAborted does (#196).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fenced replica waits for every write it admitted before the fence to finish before reporting its LSNs. Such writes have already advanced last_append_lsn and marked their slot missing; reporting earlier would let the leader's final round count them as lacking and verdict them Empty while they are still being journalled (review on #198). Test: LoginPollWaitsForInFlightWritesBeforeReporting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Two critical issues remain: a delayed poll can re-fence an aborted login and block old-session I/O indefinitely, while a write journalled after fencing can remain marked missing and leave payload blocks unreclaimed.
2 open issues
What changed in this PR
Builds on CRAFT’s existing login fencing to quiesce prior-session writes before replicas report their log positions to the leader.
Changes:
- Raises the fence for newer login terms while leaving non-login polls unchanged.
- Waits for in-flight writes before returning a fenced replica’s log positions.
- Adds six fence and drain tests, updates API comments, and bumps the package version.
| File | Description |
|---|---|
src/lib/craft/tests/test_craft_peer_exchange.cpp |
Adds fence and drain tests with a write-capable mock journal. |
src/lib/craft/craft_repl_dev.hpp |
Updates comments describing login-poll fencing. |
src/lib/craft/craft_repl_dev.cpp |
Raises the login fence and drains in-flight writes before reporting. |
conanfile.py |
Bumps the package version to 6.0.19. |
Review effort: Balanced
…s published as held - get_rs_commit_lsn does not raise the fence for a term at or below aborted_upto: a delayed poll arriving after its LoginAborted would fence again, and the InternalLogin that could lift it is a no-op for an aborted term. - A write that finishes journalling after the fence still returns STALE_TERM, but its slot is now cleared from missing_lsns_ and put in the overlay: the login's final round sees it as held (fetch_data reports it) and an Empty verdict frees its blocks. The login's truncation cannot have run yet (it waits for in-flight writes), so a slot above rs_commit_lsn is still removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
A critical missed-poll race allows below-watermark writes to publish after an Empty verdict and new-term admission, leaking payload blocks and potentially exposing voided data through the overlay.
1 open issue
2 resolved since last review
Review effort: Balanced
| // (fetch_data reports it, an Empty verdict frees its blocks), and the login's truncation removes it if it | ||
| // lies above rs_commit_lsn. That truncation cannot have run yet: it waits for this write (in-flight guard, | ||
| // released only when this call returns). | ||
| missing_lsns_.erase(dlsn); |
There was a problem hiding this comment.
Agreed, this is real. Fixed in a26635b. In post-flight, a stale write whose slot is now Empty is treated as void: it is not published (no overlay entry, not taken out of the missing set), its data blocks are freed, and it returns STALE_TERM. The Sync's verdict found the slot missing at the time, so this is the only free those blocks get. Test: OldWriteVoidedWhileInFlightOnAnUnfencedReplicaIsDiscarded (no poll reaches the replica; the write is held in write_slot; Sync rs=0 verdicts slot 0 Empty and InternalLogin(4, rs=0) admits the new term; the write is then released and fails; the overlay is empty, the slot stays Empty, and free_data is called once).
…d replica is discarded A replica that missed the login poll keeps running an old-session write at or below the login watermark; the login's Sync can verdict that slot Empty and its InternalLogin admit the new term before the write finishes (the drain covers only writes above the watermark). Post-flight now detects the verdict: the write fails STALE_TERM, stays out of the overlay and the missing set, and its data blocks are freed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Raises the login fence on the login poll, so no write of the old session can be acknowledged after the leader has taken its starting point. Stacked on #196 (which adds
fence_termand the fence-aware admission check); base isresync/raft-entries. Design:docs/craft/resync.md§3.2 (H5), §4.6, invariant I8.1. An old client can get a write acked after the login poll, then lose it
Problem: At login the leader asks each replica how far it is and takes
rs_commit_lsn = max(quorum.last_append).InternalLoginthen truncates everything above that point.get_rs_commit_lsn(term, is_login)ignored both arguments, so replicas kept acknowledging the old session after reporting their position. A write acked in that window sits abovers_commit_lsnand is truncated: an acknowledged write is lost, with no failure at all.Diagram:
sequenceDiagram participant Old as Old UBD participant L as Leader participant B as Replica B participant C as Replica C L->>B: login poll: how far are you? L->>C: login poll: how far are you? B-->>L: last_append = 100 C-->>L: last_append = 100 Note over L: rs_commit_lsn = 100 Old->>B: write 101 Old->>C: write 101 B-->>Old: ok C-->>Old: ok (quorum: application told "durable") L->>B: RAFT: InternalLogin(rs = 100) B->>B: truncate above 100 → acked write 101 lostSolution: On a login poll, the replica raises
fence_termto the new term in the same critical section that snapshots{commit_lsn, last_append_lsn}. From that momentwrite()'s pre- and post-flight checks (and, with #196, its post-commit recheck) reject the old term. Every write acked before the poll is covered by the reportedlast_append_lsn, and none can be acked after it.sequenceDiagram participant Old as Old UBD participant L as Leader participant B as Replica B L->>B: login poll(term 4, is_login) B->>B: fence_term = 4, then report last_append = 100 Old->>B: write 101 (term 3) B-->>Old: STALE_TERM (fenced) L->>B: RAFT: InternalLogin(term 4, rs = 100) → lifts the fence, admits term 42. The fence must not be lowered, lifted by a stale poll, or released by time
Problem: A delayed or duplicated poll from an older login could roll the fence back, and a poll for a term at or below the current session would fence a healthy session for nothing. Releasing the fence on a timer reopens scenario 1 whenever a login is slower than the timer.
Diagram:
sequenceDiagram participant L6 as Login (term 6) participant L5 as Delayed poll (term 5) participant R as Replica L6->>R: login poll(term 6) R->>R: fence_term = 6 L5->>R: login poll(term 5) arrives late R->>R: before: fence would drop to 5 → term-5 writes admitted again R->>R: after: max(6, 5) = 6 → unchangedSolution:
fence_term = max(...)) and is raised only for a term newer than the current session. Non-login polls (watchdog/periodic) change nothing.InternalLoginorLoginAborteddoes (craft: apply RAFT entries in log order; InternalLogin v2 and LoginAborted #196).session_term/fence_termin the reply, so a new leader can find an unfinished login, needs the peer wire v2 (craft::lsn_pairhas no room for it). That is follow-up work.3. A write still in flight at the fence could be voided while it is being journalled
Problem: A write admitted just before the fence has already advanced
last_append_lsnand marked its slot missing, but is still writing to the journal. If the replica reports right away, the leader's final round sees the slot as missing (lacking) on that replica, may verdict it Empty, and the write then finishes journalling data under a voided slot. (Review on #198.)Diagram:
sequenceDiagram participant W as Write 0 (old term) participant R as Replica participant L as Leader W->>R: admitted, journalling… L->>R: login poll(term 4) R->>R: fence_term = 4 R-->>L: before: report now → slot 0 counted as lacking W-->>R: write finishes journalling (never acked) Note over R: after: the poll waits for write 0, then reports (slot 0 now held)Solution: After raising the fence, the poll waits until no write admitted before it is still in flight (the same drain #196 uses before truncating), then reports. Those writes can no longer be acknowledged, so waiting only makes the report exact.
Testing
VolumeTest.CreateVolumeThenRecoverfailed once on a fixed 2 s sleep after volume removal, unrelated to this change, and passed on two reruns); line coverage ofsrc/lib/craft/90% (craft_repl_dev.cpp97%).test_craft_peer_exchange.cpp:InternalLogin;fetch_datatests are unaffected).🤖 Generated with Claude Code