Skip to content

Index hierarchy - #728

Open
MarkWolters wants to merge 119 commits into
mainfrom
index_hierarchy
Open

MarkWolters wants to merge 119 commits into
mainfrom
index_hierarchy

Conversation

@MarkWolters

@MarkWolters MarkWolters commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a generic Index API and HnswIndexBuilder, a fluent builder that makes building, compressing, persisting and searching a graph index a line or two each, instead of hand-assembling GraphIndexBuilder constructors, PQ training, writers and score providers. The builder wraps the existing GraphIndexBuilder code, so performance and recall are unchanged; GraphIndexBuilder remains fully supported.

Changes

  • Indexes.hnswBuilder(...): type-first entry point with defaults for every setting; buildAndPopulate() builds a complete graph in one call.
  • Compression as a setting: withCompressionType(PQ | BQ) trains and encodes inside the builder, with PQ defaults matching Cassandra's and overridable (withPqSubspaces, withPqGlobalCentering, withPqAnisotropicThreshold); getCompressedVectors() returns the codes for search or fused-PQ writing.
  • Incremental construction on the same builder (addGraphNode, markNodeDeleted, cleanup, rescore, withExistingGraph), with early rejection of invalid or mixed-scoring configurations.
  • Generic handles: api.Index / api.IndexSearcher; ImmutableGraphIndex is renamed GraphIndex and implements Index.
  • Simpler persistence and search: PersistableGraphIndex writer accessors return each writer's own builder, writeTo(path, vectors) writes a basic on-disk index, and GraphSearcher.search(query, topK, rerankK, vsf, filter) searches with the graph's stored vectors.
  • Fixes: inserting after cleanup(), rescore dropping pending deletes, and writer builders leaking a file handle when not built.
  • Experimental placeholders: recipes (HnswRecipe.DEFAULT only) and IVF types.

Breaking changes

ImmutableGraphIndex is renamed GraphIndex (no alias), and GraphIndexBuilder.build(ravv) / getGraph() now return PersistableGraphIndex. Callers must recompile; see UPGRADING.md.

Testing

  • New and updated unit tests for the builder, PQ options, writer accessors, stored-vector search and concurrency; the full suite passes.
  • Measured against main (100k × 128-dim): build time, search latency and recall (exact, inline, fused PQ) are the same within noise.
  • IndexApiExample exercises every pattern end to end: mvn compile exec:exec@index-api-example.

Docs

UPGRADING.md (migration guide) and release notes (docs/release notes/4.0.2/728.enhancement.md).

pr728.tar.gz

tlwillke and others added 30 commits December 22, 2025 21:33
…) ASH backend. Validated ASH error and performance correctness.
…hout loading all vectors first). Updated DistancesASH and DistancesPQ to take advantage of this. The original SiftLoader remains intact.
…pooled version for performance purposes though.
… scoring. Also made a slight change to limit copying query info from query pool (only at landmark run boundaries) during block scoring.
… scoring. Also made a slight change to limit copying query info from query pool (only at landmark run boundaries) during block scoring.
…ding payload alignment exception for recall benchmarking purposes.
@MarkWolters
MarkWolters marked this pull request as ready for review October 9, 2026 14:46
@tlwillke tlwillke assigned tlwillke and MarkWolters and unassigned tlwillke Oct 12, 2026
@tlwillke tlwillke added the 4.0.2 Release label label Oct 12, 2026
@MarkWolters MarkWolters added 4.1.0 Release Label and removed 4.0.2 Release label labels Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.1.0 Release Label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants