Skip to content

feat(fts): impact skip data for posting lists#7602

Draft
BubbleCal wants to merge 2 commits into
yang/oss-1344-make-fts-index-block-size-configurablefrom
yang/lan2-88-impact-skip-data
Draft

feat(fts): impact skip data for posting lists#7602
BubbleCal wants to merge 2 commits into
yang/oss-1344-make-fts-index-block-size-configurablefrom
yang/lan2-88-impact-skip-data

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Stacked on #7466 (configurable posting block size).

Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting
blocks, plus one level1 entry per 32 blocks, and drive block-max WAND
pruning from them instead of build-time scores that go stale as index
stats drift:

  • Entries are varint-encoded for 256-doc blocks (2-3 bytes per pair vs 8);
    128-doc-block indexes keep fixed-width u32 entries.
  • Bounds bake once per cached list into an Arc-shared slab (max doc weight
    per entry plus the list-wide max); per-query clones reuse the slab, so
    query time pays one multiply per bound instead of frontier rescans.
  • Entry doc_up_tos decode once at construction.
  • Lagging iterators park in the WAND tail under the data-driven global
    bound (query_weight x baked list max) instead of INFINITY.

Impacts shrink from 3.8 to ~0.6 bits/posting with the varint encoding.

Measured vs #7466 (its base)

Per-branch-tip wheels, 200M-doc v3-256 index (24 partitions), 1000 queries
× 8 concurrent, warm. Without impacts a 256-block index prunes on the
(valid but loose) list-level max that #7466 falls back to; the impact skip
data restores block-granular bounds on the same classic query loops:

query #7466 (list-max fallback) this PR
OR 3w k10 0.363s / 22 qps 0.103s / 76 qps (3.5×)
OR 3w k100 0.392s / 20 qps 0.198s / 40 qps (2.0×)
AND 3w k10 0.547s / 15 qps 0.115s / 69 qps (4.8×)
AND 3w k100 0.558s / 14 qps 0.245s / 33 qps (2.3×)

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting
blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32
blocks, and drive block-max WAND pruning from them instead of build-time
scores that go stale as index stats drift:

- Bounds bake once per cached list into an Arc-shared slab (max doc weight
  per entry plus the list-wide max); per-query clones reuse it, so query
  time pays one multiply per bound instead of frontier rescans.
- Entry doc_up_tos decode once at construction.
- Lagging iterators park in the WAND tail under the data-driven global
  bound (query_weight x baked list max) instead of INFINITY.

128-doc-block indexes keep fixed-width u32 impact entries.
@BubbleCal

Copy link
Copy Markdown
Contributor Author

Rebased on the updated #7466 (which now owns all V3 breaking changes, including quantized doc-length scoring). This PR's impact bounds are baked against the same quantized lengths for the Varint (256-doc) format — quantization is monotone, so the stored (freq, doc_len) pareto frontier still dominates and the bounds stay valid upper bounds of quantized scores. No breaking change here.

…nto yang/lan2-88-impact-skip-data

The base's shift/mask fix for the posting-iterator hot loops conflicts
textually with this branch's posting_block_idx/offset/start helpers,
which already specialize 128/256 to shifts and masks. Keep the helper
form everywhere in wand.rs and drop the base's now-unused
block_shift/block_mask methods; the power-of-two debug_assert stays.
@BubbleCal BubbleCal marked this pull request as draft July 5, 2026 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant