Tracking: feat/disk-offload follow-up roadmap (P0-6 → P2-5)

[TinDang97]

Umbrella tracking issue for the 11 follow-ups filed after the senior-Rust review of #43. Groups them into execution waves so contributors can pick work without blocking each other.

Source plan: `.claude/plans/dreamy-whistling-haven.md` (local). The original P0 batch already shipped in #43 commit `a50eead`.

Dependency map

graph LR
  subgraph W1["Wave 1 — isolated, parallelizable"]
    I44["#44 P0-6 ext: bounded LZ4 (kv_page ×2, warm_search)"]
    I49["#49 P1-6: DiskANN io error propagation"]
    I46["#46 P1-3: streaming cold-tier decode"]
    I53["#53 P2-4: format_version headers"]
    I47["#47 P1-4: ShardManifest borrow audit"]
  end

  subgraph W2["Wave 2 — structural"]
    I51["#51 P2-2: break manifest cycle"]
    I45["#45 P1-2: precompute shard_dir"]
    I48["#48 P1-5: manifest MAX_INLINE_ENTRIES cap"]
  end

  subgraph W3["Wave 3 — recovery hardening"]
    I52["#52 P2-3: index manifest in recovery"]
    I54["#54 P2-5: recovery test coverage gaps"]
  end

  subgraph W4["Wave 4 — global cross-cut"]
    I50["#50 P2-1: Cargo feature gate disk-offload"]
  end

  I47 --> I51
  I51 --> I48
  I51 --> I52
  I52 --> I54
  I45 --> I50
  I51 --> I50
  I44 --> I50
  I46 --> I50
  I49 --> I50
  I53 --> I50
  I48 --> I50
  I54 --> I50

Loading

Wave 1 — isolated fixes (start here in parallel)

These are surgical, single-module, no dependencies on each other. Five different contributors can take them simultaneously.

#	Title	Files	Effort
#44	Extend bounded LZ4 helper to non-FPI sites	`persistence/kv_page.rs` (×2), `vector/persistence/warm_search.rs`	S
#49	DiskANN `uring_search` error propagation	`vector/diskann/{uring_search,segment}.rs`	M
#46	Streaming cold-tier vector decode	`storage/tiered/cold_tier.rs`	M
#53	`format_version` headers (KvLeafPage + DiskANN)	`persistence/kv_page.rs`, `vector/diskann/{segment,pq,vamana}.rs`	S
#47	ShardManifest borrow audit	read-only investigation (no edits)	S

Why first: zero coupling, zero risk to other in-flight work, individually mergeable. #47 is investigation-only and feeds Wave 2.

Wave 2 — structural (after Wave 1, depends on #47)

Module-shape changes that touch many files. Need #47's borrow audit to land first so we know whether `ShardManifest` needs to become `RefCell`-wrapped before the structural cleanup.

#	Title	Depends on	Effort
#51	Break manifest ↔ cold_index ↔ cold_tier circular dep	#47	M
#45	Precompute per-shard disk-offload paths	— (parallel with #51)	M
#48	ShardManifest `MAX_INLINE_ENTRIES` warning + metric	#51	S (warning only; real overflow-pages fix is its own phase)

Why second: #51 unblocks clean module boundaries. #45 is independent but should land before #50 (feature gate) to avoid reworking the cfg decoration. #48 needs the new module shape from #51.

Wave 3 — recovery hardening (after Wave 2)

#	Title	Depends on	Effort
#52	Index manifest entries once in recovery (O(files × phases) → O(files))	#51	M
#54	Recovery test coverage gaps	#52	M

Why third: #52 changes recovery internals; #54's new tests target that code so they should be authored after the refactor.

Wave 4 — global cross-cut (last)

#	Title	Depends on	Effort
#50	Cargo feature gate `disk-offload`	every wave	L

Why last: `#[cfg(feature = "disk-offload")]` will need to decorate every file the previous waves touched. Doing this last means the gating PR is purely additive (`cfg` annotations + `Cargo.toml`) and does not conflict with any in-flight work. Once it lands, the tokio CI matrix will verify the gates compile cleanly.

Suggested branch / PR layout

• `fix/disk-offload-wave-1` — bundle P0-6 follow-up: extend bounded LZ4 helper to non-FPI sites #44 + P2-4: add format_version headers to KvLeafPage and DiskANN files #53 (both `S`, both touch on-disk format hardening) into one PR; P1-3: streaming cold-tier vector decode (avoid bulk Vec allocation) #46 + P1-6: DiskANN uring_search swallows I/O errors → silent recall corruption #49 each as their own PRs because they touch different subsystems; P1-4: audit ShardManifest borrow paths under concurrent spill + warm/cold transitions #47 is comment-only.
• `fix/disk-offload-wave-2` — three PRs, one per issue.
• `fix/disk-offload-wave-3` — one PR (P2-3: recovery time is O(files × phases) — index manifest entries once #52 + P2-5: recovery test coverage gaps (truncated WAL, dual-corrupt manifest, etc.) #54 together since P2-5: recovery test coverage gaps (truncated WAL, dual-corrupt manifest, etc.) #54 is its tests).
• `feat/disk-offload-cargo-feature` — single big mechanical PR for P2-1: Cargo feature gate for disk-offload (~50 KLOC always compiles in) #50.

Out of scope for this roadmap

• Page cache O_DIRECT migration — separate phase, infrastructure-heavy.
• Full manifest overflow pages (the real fix for P1-5: ShardManifest MAX_INLINE_ENTRIES caps disk-offload at ~80 files (overflow pages) #48) — separate phase, design-heavy.
• DiskANN format versioning beyond a single `u16` — schema migration is its own milestone.

[pilotspacex-byte]

Status update on this tracking issue (2026-07-15, post-v0.7.1).

Of the 10 sub-items (#44,45,46,47,48,49,50,51,53,54):

Closed (5):

• P1-3: streaming cold-tier vector decode (avoid bulk Vec allocation) #46, P1-6: DiskANN uring_search swallows I/O errors → silent recall corruption #49, P2-2: break manifest ↔ cold_index ↔ cold_tier circular dependency #51 — superseded/obsolete: the DiskANN COLD tier
  (src/vector/diskann/, src/storage/tiered/cold_tier.rs) these targeted
  was deleted whole-cloth in commit 0db0990d (PR refactor(vector): delete experimental DiskANN COLD tier + make WARM survive restart #272, "delete experimental
  DiskANN COLD tier + make WARM survive restart") per the vector two-stack
  decision — Stack B (WARM, RAM-resident) became the sole durability path.
• P1-5: ShardManifest MAX_INLINE_ENTRIES caps disk-offload at ~80 files (overflow pages) #48 — fixed by commits db08ae6b + f5e17acf (PR fix(shard): stable multishard + disk-offload on both runtimes; kill the multishard RAM zombie #136): chained overflow
  pages remove the ~70-entry manifest cap, crash-safe.
• P2-4: add format_version headers to KvLeafPage and DiskANN files #53 — fixed for KvLeafPage (shared MoonPageHeader.format_version,
  v1/v2 migration in manifest.rs); DiskANN half moot per P1-6: DiskANN uring_search swallows I/O errors → silent recall corruption #49.

Still open (5): #44 (bounded LZ4 not yet extended to the 3 remaining
non-FPI call sites), #45 (per-shard paths still rebuilt via format!() at
40+ call sites), #47 (borrow-safety audit not yet documented/tested), #50 (no
Cargo feature gate exists — disk-offload/persistence/storage::tiered
still compile in unconditionally), #54 (2 of 5 recovery-crash scenarios have
test coverage; 3 gaps remain).

Leaving this tracking issue open — roadmap is materially advanced (half the
sub-items closed, mostly via the DiskANN deletion) but not complete.

Author: Tin Dang tindang.ht97@gmail.com