Skip to content

craft: login poll raises the fence before reporting LSNs (quiesce) - #199

Open
Ilyin-Ilya wants to merge 4 commits into
resync/raft-entriesfrom
resync/quiesce
Open

Ilyin-Ilya wants to merge 4 commits into
resync/raft-entriesfrom
resync/quiesce

Conversation

@Ilyin-Ilya

@Ilyin-Ilya Ilyin-Ilya commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Raises the login fence on the login poll, so no write of the old session can be acknowledged after the leader has taken its starting point. Stacked on #196 (which adds fence_term and the fence-aware admission check); base is resync/raft-entries. Design: docs/craft/resync.md §3.2 (H5), §4.6, invariant I8.


1. An old client can get a write acked after the login poll, then lose it

Problem: At login the leader asks each replica how far it is and takes rs_commit_lsn = max(quorum.last_append). InternalLogin then truncates everything above that point. get_rs_commit_lsn(term, is_login) ignored both arguments, so replicas kept acknowledging the old session after reporting their position. A write acked in that window sits above rs_commit_lsn and is truncated: an acknowledged write is lost, with no failure at all.

Diagram:

sequenceDiagram
    participant Old as Old UBD
    participant L as Leader
    participant B as Replica B
    participant C as Replica C
    L->>B: login poll: how far are you?
    L->>C: login poll: how far are you?
    B-->>L: last_append = 100
    C-->>L: last_append = 100
    Note over L: rs_commit_lsn = 100
    Old->>B: write 101
    Old->>C: write 101
    B-->>Old: ok
    C-->>Old: ok (quorum: application told "durable")
    L->>B: RAFT: InternalLogin(rs = 100)
    B->>B: truncate above 100 → acked write 101 lost
Loading

Solution: On a login poll, the replica raises fence_term to the new term in the same critical section that snapshots {commit_lsn, last_append_lsn}. From that moment write()'s pre- and post-flight checks (and, with #196, its post-commit recheck) reject the old term. Every write acked before the poll is covered by the reported last_append_lsn, and none can be acked after it.

sequenceDiagram
    participant Old as Old UBD
    participant L as Leader
    participant B as Replica B
    L->>B: login poll(term 4, is_login)
    B->>B: fence_term = 4, then report last_append = 100
    Old->>B: write 101 (term 3)
    B-->>Old: STALE_TERM (fenced)
    L->>B: RAFT: InternalLogin(term 4, rs = 100) → lifts the fence, admits term 4
Loading

2. The fence must not be lowered, lifted by a stale poll, or released by time

Problem: A delayed or duplicated poll from an older login could roll the fence back, and a poll for a term at or below the current session would fence a healthy session for nothing. Releasing the fence on a timer reopens scenario 1 whenever a login is slower than the timer.

Diagram:

sequenceDiagram
    participant L6 as Login (term 6)
    participant L5 as Delayed poll (term 5)
    participant R as Replica
    L6->>R: login poll(term 6)
    R->>R: fence_term = 6
    L5->>R: login poll(term 5) arrives late
    R->>R: before: fence would drop to 5 → term-5 writes admitted again
    R->>R: after: max(6, 5) = 6 → unchanged
Loading

Solution:

  • The fence is monotonic (fence_term = max(...)) and is raised only for a term newer than the current session. Non-login polls (watchdog/periodic) change nothing.
  • No timer releases it. Only an applied InternalLogin or LoginAborted does (craft: apply RAFT entries in log order; InternalLogin v2 and LoginAborted #196).
  • Reporting session_term / fence_term in the reply, so a new leader can find an unfinished login, needs the peer wire v2 (craft::lsn_pair has no room for it). That is follow-up work.

3. A write still in flight at the fence could be voided while it is being journalled

Problem: A write admitted just before the fence has already advanced last_append_lsn and marked its slot missing, but is still writing to the journal. If the replica reports right away, the leader's final round sees the slot as missing (lacking) on that replica, may verdict it Empty, and the write then finishes journalling data under a voided slot. (Review on #198.)

Diagram:

sequenceDiagram
    participant W as Write 0 (old term)
    participant R as Replica
    participant L as Leader
    W->>R: admitted, journalling…
    L->>R: login poll(term 4)
    R->>R: fence_term = 4
    R-->>L: before: report now → slot 0 counted as lacking
    W-->>R: write finishes journalling (never acked)
    Note over R: after: the poll waits for write 0, then reports (slot 0 now held)
Loading

Solution: After raising the fence, the poll waits until no write admitted before it is still in flight (the same drain #196 uses before truncating), then reports. Those writes can no longer be acknowledged, so waiting only makes the report exact.


Testing

  • Built and tested on Linux (Debug + coverage): all 14 test suites pass (VolumeTest.CreateVolumeThenRecover failed once on a fixed 2 s sleep after volume removal, unrelated to this change, and passed on two reruns); line coverage of src/lib/craft/ 90% (craft_repl_dev.cpp 97%).
  • 6 new cases in test_craft_peer_exchange.cpp:
    • a login poll fences and rejects the old session, while the write acked before the poll is covered;
    • a non-login poll never fences;
    • the fence is monotonic;
    • a stale poll does not fence;
    • a fenced replica rejects the new session until its InternalLogin;
    • a login poll waits for an in-flight write before reporting.
  • The file's mock journal now accepts writes (it used to reject all of them; the fetch_data tests are unaffected).

🤖 Generated with Claude Code

get_rs_commit_lsn(term, is_login=true) now raises fence_term to `term`
(monotonic, and only for a term newer than the current session) in the same
critical section that snapshots {commit_lsn, last_append_lsn}. Every write
acked before the poll is covered by the reported last_append_lsn, and no
write of the old session can be acked after it: write()'s pre- and
post-flight checks both require term >= fence_term. Non-login polls change
nothing. No timer lifts the fence; only an applied InternalLogin or
LoginAborted does (#196).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fenced replica waits for every write it admitted before the fence to finish
before reporting its LSNs. Such writes have already advanced last_append_lsn
and marked their slot missing; reporting earlier would let the leader's final
round count them as lacking and verdict them Empty while they are still being
journalled (review on #198). Test: LoginPollWaitsForInFlightWritesBeforeReporting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two critical issues remain: a delayed poll can re-fence an aborted login and block old-session I/O indefinitely, while a write journalled after fencing can remain marked missing and leave payload blocks unreclaimed.

2 open issues
What changed in this PR

Builds on CRAFT’s existing login fencing to quiesce prior-session writes before replicas report their log positions to the leader.

Changes:

  • Raises the fence for newer login terms while leaving non-login polls unchanged.
  • Waits for in-flight writes before returning a fenced replica’s log positions.
  • Adds six fence and drain tests, updates API comments, and bumps the package version.
File Description
src/​lib/​craft/​tests/​test_craft_peer_exchange.cpp Adds fence and drain tests with a write-capable mock journal.
src/​lib/​craft/​craft_repl_dev.hpp Updates comments describing login-poll fencing.
src/​lib/​craft/​craft_repl_dev.cpp Raises the login fence and drains in-flight writes before reporting.
conanfile.py Bumps the package version to 6.0.19.

Review effort: Balanced

Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
Comment thread src/lib/craft/craft_repl_dev.cpp Resolved
…s published as held

- get_rs_commit_lsn does not raise the fence for a term at or below
  aborted_upto: a delayed poll arriving after its LoginAborted would fence
  again, and the InternalLogin that could lift it is a no-op for an aborted
  term.
- A write that finishes journalling after the fence still returns STALE_TERM,
  but its slot is now cleared from missing_lsns_ and put in the overlay: the
  login's final round sees it as held (fetch_data reports it) and an Empty
  verdict frees its blocks. The login's truncation cannot have run yet (it
  waits for in-flight writes), so a slot above rs_commit_lsn is still removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical missed-poll race allows below-watermark writes to publish after an Empty verdict and new-term admission, leaking payload blocks and potentially exposing voided data through the overlay.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread src/lib/craft/craft_repl_dev.cpp Outdated
// (fetch_data reports it, an Empty verdict frees its blocks), and the login's truncation removes it if it
// lies above rs_commit_lsn. That truncation cannot have run yet: it waits for this write (in-flight guard,
// released only when this call returns).
missing_lsns_.erase(dlsn);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is real. Fixed in a26635b. In post-flight, a stale write whose slot is now Empty is treated as void: it is not published (no overlay entry, not taken out of the missing set), its data blocks are freed, and it returns STALE_TERM. The Sync's verdict found the slot missing at the time, so this is the only free those blocks get. Test: OldWriteVoidedWhileInFlightOnAnUnfencedReplicaIsDiscarded (no poll reaches the replica; the write is held in write_slot; Sync rs=0 verdicts slot 0 Empty and InternalLogin(4, rs=0) admits the new term; the write is then released and fails; the overlay is empty, the slot stays Empty, and free_data is called once).

…d replica is discarded

A replica that missed the login poll keeps running an old-session write at or
below the login watermark; the login's Sync can verdict that slot Empty and its
InternalLogin admit the new term before the write finishes (the drain covers
only writes above the watermark). Post-flight now detects the verdict: the
write fails STALE_TERM, stays out of the overlay and the missing set, and its
data blocks are freed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants