fix(tests): assert exact ANN index name and ensure index creation in test_the_kill_switch_flips_real_retrieval_depth (#3619) - #3745
Open
Sanderhoff-alt wants to merge 1 commit into
Conversation
Strix Security ReviewNo security issues found. Updated for Reviewed by Strix |
…test_the_kill_switch_flips_real_retrieval_depth (vectorize-io#3619) When HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS > 0 (lazy vector index mode), bank creation does not eagerly build per-bank vector indexes. This caused PostgreSQL to have no ANN index candidate and fall back to scanning any of the ~19 B-tree indexes + a disabled Sort node. The test's coarse proxy assertion ("Index Scan" in plan and "Sort" not in plan) then flaked with confusing errors depending on which B-tree index was picked. - Explicitly create the per-bank vector index in test setup to cover lazy indexing mode. - Skip with a clear reason if the backend does not use per-bank vector indexes. - Update stale comments to document why explicit index creation is needed. - Assert the exact ANN index name (idx_mu_emb_worl_<uid>) and the absence of Sort nodes.
Sanderhoff-alt
force-pushed
the
fix/issue-3619-flaky-ann-test
branch
from
August 24, 2026 05:10
4afc7d7 to
e295f12
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixes #3619.
test_ann_iterative_scan.py::test_the_kill_switch_flips_real_retrieval_depthpreviously relied on a brittle proxy assertion ("Index Scan" in plan and "Sort" not in plan) to check if the query was served by an ANN index. In environments where the per-bank vector index was not created eagerly (e.g.HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS > 0), PostgreSQL had no viable ANN candidate and fell back to B-tree indexes + a disabledSortnode, failing the test.This PR makes the test robust by ensuring the per-bank vector index is explicitly created regardless of threshold configuration and asserting the execution plan against the exact deterministic index name (
idx_mu_emb_worl_<internal_id>).Execution Flow & Lifecycle
flowchart TD A[Start Test] --> B[Create Bank Profile] B --> C[Fetch internal_id from banks table] C --> D[Explicitly create per-bank vector index idx_mu_emb_worl_uid] D --> E[Insert 600 Fact Rows & Run ANALYZE] E --> F[Run EXPLAIN Query with disabled seqscan & sort] F --> G{Plan Verification} G -->|Contains idx_mu_emb_worl_uid & No Sort| H[Execute retrieve_semantic_bm25_combined_sql] G -->|Missing index or contains Sort| I[Fail with explicit assertion message] H --> J[Assert with_resume == 400 and without_resume <= 250]Comparative Analysis (Before vs After)
VECTOR_INDEX_MIN_ROWS=0). When lazy index thresholds were set, the bank had no vector index.ops.create_bank_vector_indexeson test setup, guaranteeingidx_mu_emb_worl_<internal_id>exists across all configurations."Index Scan" in plan and "Sort" not in plan. Any of ~19 B-tree indexes could match"Index Scan".assert index_name in planandassert "Sort" not in plan.idx_memory_units_fact_type+Sort).HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS > 0.Changes Summary
hindsight-api-slim/tests/test_ann_iterative_scan.py_BANK_INDEX_FACT_TYPES,_bank_index_name,_vector_index_clause.2. Query
internal_idand callmemory._backend.ops.create_bank_vector_indexesduring setup.3. Assert
index_name in planand"Sort" not in plan.Verification Matrix
test_the_kill_switch_flips_real_retrieval_depthVECTOR_INDEX_MIN_ROWS=0)test_the_kill_switch_flips_real_retrieval_depthHINDSIGHT_API_VECTOR_INDEX_MIN_ROWS=10000, previously 100% failing)tests/test_ann_iterative_scan.py(all tests)ruff check,ruff format --check,ty check,lint.shFixes #3619