Skip to content

fix(tests): assert exact ANN index name and ensure index creation in test_the_kill_switch_flips_real_retrieval_depth (#3619) - #3745

Open
Sanderhoff-alt wants to merge 1 commit into
vectorize-io:mainfrom
Sanderhoff-alt:fix/issue-3619-flaky-ann-test
Open

fix(tests): assert exact ANN index name and ensure index creation in test_the_kill_switch_flips_real_retrieval_depth (#3619)#3745
Sanderhoff-alt wants to merge 1 commit into
vectorize-io:mainfrom
Sanderhoff-alt:fix/issue-3619-flaky-ann-test

Conversation

@Sanderhoff-alt

Copy link
Copy Markdown
Contributor

Overview

Fixes #3619.

test_ann_iterative_scan.py::test_the_kill_switch_flips_real_retrieval_depth previously relied on a brittle proxy assertion ("Index Scan" in plan and "Sort" not in plan) to check if the query was served by an ANN index. In environments where the per-bank vector index was not created eagerly (e.g. HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS > 0), PostgreSQL had no viable ANN candidate and fell back to B-tree indexes + a disabled Sort node, failing the test.

This PR makes the test robust by ensuring the per-bank vector index is explicitly created regardless of threshold configuration and asserting the execution plan against the exact deterministic index name (idx_mu_emb_worl_<internal_id>).


Execution Flow & Lifecycle

flowchart TD
    A[Start Test] --> B[Create Bank Profile]
    B --> C[Fetch internal_id from banks table]
    C --> D[Explicitly create per-bank vector index idx_mu_emb_worl_uid]
    D --> E[Insert 600 Fact Rows & Run ANALYZE]
    E --> F[Run EXPLAIN Query with disabled seqscan & sort]
    F --> G{Plan Verification}
    G -->|Contains idx_mu_emb_worl_uid & No Sort| H[Execute retrieve_semantic_bm25_combined_sql]
    G -->|Missing index or contains Sort| I[Fail with explicit assertion message]
    H --> J[Assert with_resume == 400 and without_resume <= 250]
Loading

Comparative Analysis (Before vs After)

Dimension Before (Issue #3619) After (This PR)
Index Guarantee Implicitly relied on eager bank index creation (VECTOR_INDEX_MIN_ROWS=0). When lazy index thresholds were set, the bank had no vector index. Explicitly invokes ops.create_bank_vector_indexes on test setup, guaranteeing idx_mu_emb_worl_<internal_id> exists across all configurations.
Plan Assertion Weak proxy: "Index Scan" in plan and "Sort" not in plan. Any of ~19 B-tree indexes could match "Index Scan". Precise assertion: assert index_name in plan and assert "Sort" not in plan.
Failure Diagnostics Obscure failures reporting irrelevant B-tree indexes (e.g., idx_memory_units_fact_type + Sort). Clear assertion failures identifying missing ANN index or unexpected fallback nodes.
Environment Decoupling Fails when HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS > 0. Decoupled; passes in both default and threshold-configured test runs.

Changes Summary

File Change Description
hindsight-api-slim/tests/test_ann_iterative_scan.py Modify 1. Import _BANK_INDEX_FACT_TYPES, _bank_index_name, _vector_index_clause.
2. Query internal_id and call memory._backend.ops.create_bank_vector_indexes during setup.
3. Assert index_name in plan and "Sort" not in plan.

Verification Matrix

Test Case Environment / Configuration Result
test_the_kill_switch_flips_real_retrieval_depth Default configuration (VECTOR_INDEX_MIN_ROWS=0) PASSED (1 passed)
test_the_kill_switch_flips_real_retrieval_depth Threshold mode (HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS=10000, previously 100% failing) PASSED (1 passed)
tests/test_ann_iterative_scan.py (all tests) Default configuration PASSED (11/11 passed)
Code Quality & Static Typing ruff check, ruff format --check, ty check, lint.sh PASSED (clean)

Fixes #3619

@strix-security

strix-security Bot commented Aug 24, 2026

Copy link
Copy Markdown

Strix Security Review

No security issues found.

Updated for 4afc7d7.


Reviewed by Strix
Re-run review · Configure security review settings

…test_the_kill_switch_flips_real_retrieval_depth (vectorize-io#3619)

When HINDSIGHT_API_VECTOR_INDEX_MIN_ROWS > 0 (lazy vector index mode),
bank creation does not eagerly build per-bank vector indexes. This caused
PostgreSQL to have no ANN index candidate and fall back to scanning any of
the ~19 B-tree indexes + a disabled Sort node. The test's coarse proxy
assertion ("Index Scan" in plan and "Sort" not in plan) then flaked with
confusing errors depending on which B-tree index was picked.

- Explicitly create the per-bank vector index in test setup to cover lazy indexing mode.
- Skip with a clear reason if the backend does not use per-bank vector indexes.
- Update stale comments to document why explicit index creation is needed.
- Assert the exact ANN index name (idx_mu_emb_worl_<uid>) and the absence of Sort nodes.
@Sanderhoff-alt
Sanderhoff-alt force-pushed the fix/issue-3619-flaky-ann-test branch from 4afc7d7 to e295f12 Compare August 24, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_the_kill_switch_flips_real_retrieval_depth asserts a plan shape that any of ~19 btrees can satisfy

1 participant