perf(index): norm-addend cache and slim top-k heap for FTS scoring#7627
Closed
BubbleCal wants to merge 7 commits into
Closed
perf(index): norm-addend cache and slim top-k heap for FTS scoring#7627BubbleCal wants to merge 7 commits into
BubbleCal wants to merge 7 commits into
Conversation
… for 256-doc blocks Folds the v3 breaking changes into this PR so the format/scoring break lands in one place; 128-block indexes are untouched on disk and keep exact scoring bit-for-bit. - BM25 doc lengths for 256-doc-block partitions are quantized to a Lucene SmallFloat-style byte code (4 mantissa bits, exact below 8, <= 6.25% relative error, decode = bucket floor). The byte-norm slab bakes lazily per loaded DocSet, quartering the doc-length bytes the scoring path pulls through the cache (200M docs: 800MB -> 200MB). - 256-doc posting blocks drop the leading block-max-score f32. The impact skip data introduced by the stacked follow-up supplies a tighter per-block bound; until it lands, block-level pruning for 256 falls back to the list-level max score, which is a valid (looser) bound. Block layout: [first_doc u32][doc num_bits u8][docs][pfor freqs]; posting_block_score_prefix_len(block_size) keys every reader/writer. BREAKING: 256-doc-block (v3) indexes must be rebuilt; v3 is unreleased so no migration is provided. BM25 scores on v3 differ from exact-length BM25 by the norm quantization, matching Lucene's norm semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that go stale as index stats drift: - Bounds bake once per cached list into an Arc-shared slab (max doc weight per entry plus the list-wide max); per-query clones reuse it, so query time pays one multiply per bound instead of frontier rescans. - Entry doc_up_tos decode once at construction. - Lagging iterators park in the WAND tail under the data-driven global bound (query_weight x baked list max) instead of INFINITY. 128-doc-block indexes keep fixed-width u32 impact entries.
Port of Lucene's MaxScoreBulkScorer, opt-in via LANCE_FTS_MAXSCORE=1: per outer window (bounded by the essential clauses' blocks with adaptive growth), clauses split into a non-essential prefix and essential rest by window max score vs the running threshold. Essential clauses bulk-stream decompressed blocks (single-essential windows stream with no accumulator); non-essential clauses are only probed for candidates that can still beat the threshold. Dead ranges with one live clause skip by scanning the baked per-block bound slab. Candidate emission matches the classic path (must beat the running threshold), so results are score-identical. Measured on a 200M-doc warm benchmark at 24 partitions: 3-word OR match k10 0.137s -> 0.035s, k100 0.250s -> 0.064s; hot single-term 250ms -> 3ms.
With right-sized partitions the bulk path reaches Lucene-parity latency (k10 0.034-0.037s/213-235qps vs Lucene 10.4's 0.037s/216qps on the same 200M-doc corpus, queries, and warm protocol) and its results are score-identical to the classic WAND loop. LANCE_FTS_MAXSCORE=0 opts back into the classic loop.
AND and phrase queries previously leapfrogged doc-at-a-time through boxed PostingIterator::next calls (~61% of the AND profile) and phrase checks decoded a whole 256-doc position block per candidate (~39% of the phrase profile). - and_bulk_search: block-max window pruning plus a k-pointer merge over decompressed block slices; per-candidate advance cost drops to a few loads. Results are identical to the classic loop (LANCE_FTS_BULK_AND=0 opts out). Phrase queries ride the same path. - seek_packed_doc_positions: PackedDelta full groups are self-describing ([num_bits u8][16*num_bits bytes]), so group offsets are recovered by hopping headers; decode only the 1-2 groups overlapping the candidate doc's delta range, with a lazily-built group index, memoized unpacked group, and a decoded-tail cache per block. - check_exact_positions_bulk: allocation-free slop=0 alignment check on the decoded scratch slices for parked lead clauses. Warm mmlb benchmarks, 8 concurrent queries: AND\@200M k10 0.114->0.060s, k100 0.240->0.118s; phrase\@50m 3-word k10 0.335->0.210s, 2-word k10 0.098->0.042s. All steps verified score-identical to the classic path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r bulk fts conjunctions Three changes to the bulk AND/phrase candidate pipeline, all score-identical to the classic loop: - Clause-count-specialized merge kernels (2 and 3 clauses) keep cursors and bounds in registers, with an AVX2 find_next_geq catch-up scan (the analogue of Lucene's VectorUtil.findNextGEQ) replacing the mispredict- heavy scalar walk. Generic kernel covers other widths. - Two-pass batched scoring: kernels only record (doc, offsets) matches; doc lengths are then gathered back-to-back so their cache misses overlap, and the prune/score/insert pass runs in doc order with the classic semantics. - A per-clause frequency-bucketed score-bound LUT lets kernels skip lead docs before any follower advance when even the doc-length-free bound cannot beat the threshold (score-first ordering, Lucene-style); the bound is monotone so the skipped set is a subset of the exact prune. Warm mmlb AND @200M, 8 concurrent: k10 0.060->0.049s (161qps), k100 0.118->0.098s (81qps); phrase\@50m 3w k10 0.210->0.204s, 2w 0.042->0.040s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two hot-loop changes, both result-identical (scores are bit-equal; only tie ordering among equal scores can shift with the slimmer heap tuple): - Scorer::doc_norm exposes the BM25 doc-length denominator addend, and quantized (v3) searches bake a per-norm-code addend cache once per partition search (Lucene's norm cache). The OR streaming/drain loops and the bulk conjunction pass score with one byte-norm load plus the cached addend instead of recomputing the denominator per doc — the factored expressions are evaluated identically, so scores don't move. - Top-k heap entries shrink to a Copy tuple; the per-candidate (term, freq) pairs go to an append-only side log and only the final top-k materialize a Vec, removing the per-insert allocation. Warm mmlb, 8 concurrent: OR@200M k10 0.0343->0.0249s (318qps), k100 0.0628->0.0467s (170qps) — both now ahead of Lucene 10.4's sliced searcher; AND@200M k10 0.0456->0.0443s, k100 0.0916->0.0883s; phrase@50M 2w 0.0392->0.0370s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BubbleCal
force-pushed
the
yang/lan2-88-fts-norm-cache
branch
from
347ca59 to
a1aea58
Compare
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7626; last of the series.
What
Two hot-loop changes, both score-identical (only tie ordering among equal scores can shift with the slimmer heap tuple):
Scorer::doc_normexposes the BM25 doc-length denominator addend, and quantized (V3) searches bake the 256 possible addends once per partition-search. The OR streaming loop (collect_window_scores), the OR drain/single-essential completion paths, and the bulk conjunction scoring pass then score with one byte-norm load plus a cached addend instead of recomputingk1*(1-b+b*dl/avgdl)per doc. The factored expressions are evaluated identically, so scores are bit-equal to the uncached path.FreqLog) and only the final top-k materialize aVec— no more per-insert allocation, and heap sifts move 24 bytes instead of a Vec-carrying tuple.Results (mmlb-200m warm, 8 concurrent, vs Lucene 10.4 sliced searcher on the same corpus/queries)
The OR win comes from the streaming loop: recomputing the BM25 denominator per (clause x doc) was its largest single cost. With this PR both OR operating points are ahead of Lucene 10.4 on this benchmark.
Verification
cargo test -p lance-index,clippy -D warnings,fmt --checkclean.🤖 Generated with Claude Code