Skip to content

Memory: HNSW post-filter can under-fill similar-incident recall #67

Description

@JumpTechCode

Summary

similar_resolved_incidents applies a status IN ('resolved','closed') filter after the ANN traversal with a hard LIMIT k. pgvector's HNSW filters post-traversal, so with the default ef_search=40 and a corpus dominated by open incidents, the query can silently return fewer than k results or miss closer resolved matches.

Evidence

  • sentinel/persistence/repositories.py:882-931ORDER BY i.embedding <=> CAST(:qvec AS vector) LIMIT :k (:908-909) with WHERE ... i.status IN ('resolved','closed') (:903-907).
  • No over-fetch multiplier (LIMIT :k is bound directly), no ef_search tuning, no partial index on resolved rows (the only HNSW index idx_incidents_embedding is unfiltered), no hnsw.iterative_scan.
  • The EXPLAIN test only validates the unfiltered ORDER BY.

Fix (options)

  • Over-fetch then filter: LIMIT k * N in a subquery, filter to resolved/closed, take top-k.
  • Or raise ef_search for this query (SET LOCAL hnsw.ef_search).
  • Or add a partial HNSW index on resolved/closed rows.
  • Or enable hnsw.iterative_scan (pgvector ≥ 0.8).
  • Add a recall test that seeds a resolved-sparse corpus and asserts k results.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions