Skip to content

P1-5: ShardManifest MAX_INLINE_ENTRIES caps disk-offload at ~80 files (overflow pages) #48

Description

@TinDang97

`src/persistence/manifest.rs:339-348` rejects `commit()` once the entry vector exceeds `MAX_INLINE_ENTRIES` (~80 entries that fit in a single 4 KB inline root page). With heavy spill workloads (one heap file per evicted key) this is hit in seconds, after which every eviction returns `Frame::Error`, the shard effectively dies, and the operator sees an "OOM" error that is actually a manifest-overflow error.

Short-term mitigation (this issue):

  • Add a startup warning in `main.rs` / `config.rs` when `disk_offload_enabled && maxmemory_policy != "noeviction"` reminding the operator of the cap.
  • Surface a metric `manifest_inline_entries_used / MAX_INLINE_ENTRIES` in `INFO`.
  • Return a clearer error message from `commit()` ("manifest inline root full; overflow pages not yet implemented") so the failure mode is greppable.

Real fix (separate phase):

Implement chained overflow root pages as described in MoonStore v2 design §4.2:
`RootHeader { epoch, redo_lsn, wal_flush_lsn, file_count, root_kind: u8, payload: union { inline_entries[80] | overflow_ptr { first_page_id, page_count, total_crc } } }`. Entries beyond the inline cap serialize into chained pages appended at `2 * PAGE_4K + N * PAGE_4K`. `commit()` writes the chain first, then flips the dual-root with the new pointer.

This is non-trivial; treat as its own GSD phase.

Plan reference: `.claude/plans/dreamy-whistling-haven.md` §P1-5.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions