test(fts): benchmark new FTS algo#7605
Closed
BubbleCal wants to merge 27 commits into
Closed
Conversation
…s-index-block-size-configurable
merge_all_tail_partitions folded every leftover worker builder into a single partition unconditionally; on a build whose workers never hit the flush threshold (large LANCE_FTS_PARTITION_SIZE), the entire index collapsed into one partition and queries lost all intra-partition parallelism. Fold with the same memory/doc-count checks as the partition-merge path instead, so the final partition count converges to total builder memory / budget.
ensure_num_tokens_loaded rebuilt the DocSet from the cached arrow column on every call, copying the whole num_tokens array (tens of MB per partition) once per query per partition — profiled at ~50% of single-term query time and a large slice of every warm query. Materialize it once in a OnceCell like the full DocSet, and account for it in deep_size_of.
Tail partitions hold most of the index when workers rarely hit the flush threshold; writing them sequentially serialized the posting-list compression of nearly the whole index behind one producer thread at the end of the build. Write them through buffer_unordered instead.
256-doc blocks pack frequencies with Lucene PForUtil-style patched FOR: the body uses the bit width that minimizes total bytes and up to 31 outliers are appended as (index u8, high-bits varint) exceptions. Plain FOR let a single large tf widen the whole block; measured on a 200M-doc corpus (avg tf 4) this was 4.7 bits/posting for frequencies, patched FOR brings it to ~2.5. 128-doc blocks are unchanged.
…gList Block boundary lookups re-read block headers and re-decoded the tail block on every call; bake first doc ids once per cached list and share the slab across per-query clones.
Store per-block (freq, doc_len) impact frontiers alongside 256-doc posting blocks (varint-encoded: 2-3 bytes per pair) plus one level1 entry per 32 blocks, and drive block-max WAND pruning from them instead of build-time scores that go stale as index stats drift: - Bounds bake once per cached list into an Arc-shared slab (max doc weight per entry plus the list-wide max); per-query clones reuse it, so query time pays one multiply per bound instead of frontier rescans. - Entry doc_up_tos decode once at construction. - Lagging iterators park in the WAND tail under the data-driven global bound (query_weight x baked list max) instead of INFINITY. 128-doc-block indexes keep fixed-width u32 impact entries.
Port of Lucene's MaxScoreBulkScorer, opt-in via LANCE_FTS_MAXSCORE=1: per outer window (bounded by the essential clauses' blocks with adaptive growth), clauses split into a non-essential prefix and essential rest by window max score vs the running threshold. Essential clauses bulk-stream decompressed blocks (single-essential windows stream with no accumulator); non-essential clauses are only probed for candidates that can still beat the threshold. Dead ranges with one live clause skip by scanning the baked per-block bound slab. Candidate emission matches the classic path (must beat the running threshold), so results are score-identical. Measured on a 200M-doc warm benchmark at 24 partitions: 3-word OR match k10 0.137s -> 0.035s, k100 0.250s -> 0.064s; hot single-term 250ms -> 3ms.
With right-sized partitions the bulk path reaches Lucene-parity latency (k10 0.034-0.037s/213-235qps vs Lucene 10.4's 0.037s/216qps on the same 200M-doc corpus, queries, and warm protocol) and its results are score-identical to the classic WAND loop. LANCE_FTS_MAXSCORE=0 opts back into the classic loop.
…-parity-integration
…-parity-integration
…ene-parity-integration
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
github-actions
Bot
added
A-python
github-actions
Bot
added
A-java
BubbleCal
changed the title
Contributor
Author
|
Verification results (mmlb-200m, 2.35TB text, 1000 3-word OR match queries,
No regression from the split: latency identical to the development tree. |
Contributor
Author
|
Integration branch refreshed with the second optimization round (now = #7600-#7604 + #7623-#7627): bulk conjunctions + SIMD kernels, V3 quantized norms + slim blocks (breaking, see #7606), norm cache + slim heap, and the position-prewarm deadlock fix. Warm mmlb-200m vs Lucene 10.4 sliced: OR k10 0.0249s/318qps and OR k100 0.0467s/170qps (both ahead), AND k10 0.0443s (1.29x), AND k100 0.0883s (1.94x); phrase@50M 2w 0.037s (~5x ahead), 3w 0.200s (1.05x behind). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration branch: latest main + the full FTS performance stack, for
end-to-end verification. Merges:
first-doc slab)
Do not merge; verification vehicle for the stack. Benchmark results to be
posted below.
🤖 Generated with Claude Code