Index hierarchy

.gitignore

@@ -11,6 +11,7 @@ dataset_
 **results**.jsonl
 
 ### Bench caches
+compressor_cache/
 pq_cache/
 index_cache/
 dataset_cache/
@@ -64,3 +65,5 @@ dependency-reduced-pom.xml
 results.csv
 **/datasets/custom/**
 **/dataset_cache/**
+
+design-notes/*

UPGRADING.md

@@ -1,3 +1,186 @@
+# Upgrading from 4.0.2 to 4.0.3
+
+## Critical API changes
+
+If you only read one thing, read this!
+
+- **`ImmutableGraphIndex` has been renamed to `GraphIndex`**, which now extends the new generic `Index` interface.
+  There is no deprecated alias: code that names `ImmutableGraphIndex` no longer compiles. Replace the name:
+
+  | Before | Now |
+  |---|---|
+  | `ImmutableGraphIndex` | `GraphIndex` |
+  | `ImmutableGraphIndex.View`, `.ScoringView`, `.NodeAtLevel`, `.IntMarker`, `.NeighborProcessor` | `GraphIndex.View`, `.ScoringView`, `.NodeAtLevel`, `.IntMarker`, `.NeighborProcessor` |
+  | `ImmutableGraphIndex.ENTRY_NODE_ABSENT` | `GraphIndex.ENTRY_NODE_ABSENT` |
+  | `ImmutableGraphIndex.prettyPrint(graph)` | `GraphIndex.prettyPrint(graph)` |
+
+  The rename is mechanical: the methods and nested types are unchanged.
+- **`GraphIndexBuilder.build(RandomAccessVectorValues)` and `getGraph()` now return `PersistableGraphIndex`**, a
+  `GraphIndex` that can also be written to disk (see below), and `buildAndMergeNewNodes(...)` returns `GraphIndex`.
+  Code that assigns these results to `GraphIndex` compiles unchanged.
+- **Recompile against 4.0.3.** Besides the rename, the nested types are now members of `GraphIndex`, and many
+  signatures that took or returned `ImmutableGraphIndex` (for example `new GraphSearcher(graph)`,
+  `GraphSearcher.getView()`, `OnDiskGraphIndex.write`, and the graph writer builders) now use `GraphIndex`.
+  Classes compiled against 4.0.2 will fail to link.
+
+## New features
+
+- **A generic index API.** `io.github.jbellis.jvector.api.Index` is a backing-agnostic handle (`searcher()`, `ramBytesUsed()`, `close()`), and
+  `IndexSearcher` is the matching searcher type. `GraphIndex` implements `Index`, and `GraphIndex.searcher()`
+  returns a `GraphSearcher` with no cast. Code that holds only an `Index` recovers the concrete type with
+  `instanceof GraphIndex`. `IndexSearcher` is `Closeable` and `Index.close()` throws only `IOException`, so both
+  work with try-with-resources:
+  ```java
+  try (IndexSearcher searcher = index.searcher()) { ... }
+  ```
+- **`Indexes.hnswBuilder(...)`**, a fluent builder for graph indexes (`HnswIndexBuilder`). You choose how vectors
+  are scored when you create it, and every other setting has a default (the same defaults as `GraphIndexBuilder`:
+  max degree 32, beam width 100, neighbor overflow 1.2, alpha 1.2, hierarchy on, refinement on). The common case is
+  one call:
+  ```java
+  PersistableGraphIndex graph = Indexes.hnswBuilder(ravv, VectorSimilarityFunction.COSINE).buildAndPopulate();
+  ```
+  - `Indexes.hnswBuilder(RandomAccessVectorValues, VectorSimilarityFunction)` scores with the vectors themselves.
+    `withCompressionType(CompressionType.PQ)` or `BQ` makes the builder train the quantizer, encode the vectors
+    and build with the compressed scores, with no codebook or encoding step for you to run. Once the graph is
+    built, `getCompressedVectors()` returns what it trained (`PQVectors`, whose `getCompressor()` is the
+    `ProductQuantization`, or `BQVectors`), to search with or to write to disk, e.g. as a `FusedPQ` feature.
+    The PQ training defaults match Cassandra's, and each can be overridden: `withPqSubspaces(n)` sets the code size
+    in bytes (one subspace per byte; by default a dimension-dependent rule, e.g. 64 for 128 dimensions and 192 for
+    1536), `withPqGlobalCentering(boolean)` whether to center the vectors first (default `false`), and
+    `withPqAnisotropicThreshold(float)` the anisotropic weighting (default `-1.0`, unweighted). The cluster count
+    (256) still comes from `GraphIndexBuilderConfig`.
+  - `Indexes.hnswBuilder(BuildScoreProvider, dimension)` scores with your own provider, for callers that stream
+    vectors in or already have a provider. It has no vectors of its own, so populate it with
+    `populateGraph(ravv)` or `addGraphNode`.
+  - `buildAndPopulate()` builds and inserts the builder's own vectors in one call. `build()` returns the graph
+    empty, for incremental construction with `addGraphNode` (thread-safe, and the graph can be searched while it
+    grows), `markNodeDeleted`, `removeDeletedNodes` and `cleanup()`. `populateGraph(ravv)` populates an empty
+    graph from a whole `RandomAccessVectorValues` and calls `cleanup()`; it throws if the graph already has nodes.
+    `build()` is idempotent: every call returns the same graph. Settings changed after the graph is built are
+    ignored, with a logged warning.
+  - `withExistingGraph(OnHeapGraphIndex)` continues building on a graph reloaded with `OnHeapGraphIndex.load`.
+    Add the new nodes with `addGraphNode`, from the graph's `getIdUpperBound()` up, then call `cleanup()`.
+    The graph's dimension must match the builder's.
+  - `HnswIndexBuilder.rescore(builder, newProvider)` copies the graph with every edge re-scored by a new
+    provider, for example after refining a PQ codebook, and returns a builder that continues with the copy.
+  - `HnswIndexBuilder` is `Closeable`: closing it releases the per-thread scratch space used while inserting. The
+    graph it built stays usable.
+- **`PersistableGraphIndex`**, implemented by `OnHeapGraphIndex` and `OnDiskGraphIndex`, adds accessors for the
+  three graph writers, so a built graph can be written without naming the writer classes. Each accessor returns
+  that writer's own builder type, so its specific options (such as `withStartOffset`, `withParallelWorkerThreads`
+  or `withExecutor`) are available, and building gives the concrete writer (with `getOutput()` and `checksum()`
+  on the random-access writers):
+
+  | Accessor | Writer | `GraphIndexWriterTypes` |
+  |---|---|---|
+  | `getWriterBuilder(Path)` | `OnDiskGraphIndexWriter` (random access, single-threaded) | `RANDOM_ACCESS` |
+  | `getParallelWriterBuilder(Path)` | `OnDiskParallelGraphIndexWriter` | `RANDOM_ACCESS_PARALLEL` |
+  | `getWriterBuilder(IndexWriter)` | `OnDiskSequentialGraphIndexWriter` | `ON_DISK_SEQUENTIAL` |
+
+  ```java
+  try (var writer = graph.getParallelWriterBuilder(path).with(new InlineVectors(dim)).build()) {
+      writer.write(Feature.singleStateFactory(FeatureId.INLINE_VECTORS,
+              node -> new InlineVectors.State(ravv.getVector(node))));
+  }
+  ```
+  Constructing the writer builders directly and `GraphIndexWriter.getBuilderFor(...)` still work.
+- **Shortcuts for the simplest on-disk index.** `PersistableGraphIndex.writeTo(path, vectors)` writes a graph with
+  its vectors stored inline. `GraphSearcher.search(query, topK, rerankK, similarityFunction, acceptOrds)` searches a
+  graph that stores its vectors (an `OnDiskGraphIndex` with inline or NVQ vectors) without building a score
+  provider: with fused PQ it traverses with the PQ codes and reranks the best `rerankK` with the stored vectors,
+  otherwise it scores with the stored vectors. For the same `rerankK` it returns the same results as building that
+  score provider by hand. On an in-memory graph it throws `IllegalStateException`. `GraphIndex.ScoringView.hasApproximateScores()`
+  (default `false`) reports whether a view supports `approximateScoreFunctionFor`.
+  ```java
+  graph.writeTo(path, ravv);
+  try (var rs = ReaderSupplierFactory.open(path);
+       var onDisk = OnDiskGraphIndex.load(rs);
+       var searcher = onDisk.searcher()) {
+      SearchResult result = searcher.search(query, 10, 30, VectorSimilarityFunction.COSINE, Bits.ALL);
+  }
+  ```
+- **Experimental:** recipes and IVF are marked `@Experimental`. `HnswRecipe.DEFAULT` is defined (it restates the
+  builder's defaults) and can be applied with `HnswIndexBuilder.applyRecipe`; the other recipes, and
+  `IvfIndexBuilder.build()`, throw `UnsupportedOperationException` until their values and implementation exist.
+  The IVF types are `Indexes.ivfBuilder()`, `IvfIndexBuilder`, `IvfIndex` and `IvfSearcher`.
+- `jvector-examples/.../IndexApiExample.java` walks through the new API end to end, organized around Cassandra's and
+  OpenSearch's usage: the one-call build and tuning; building with PQ or BQ via `withCompressionType` and searching
+  with the trained codes; every writer type, including a graph embedded at an offset; fused PQ, NVQ with fused PQ,
+  and separately stored PQ codes on disk, all from the builder's own PQ; incremental construction with deletes;
+  streaming with caller-maintained PQ codes and a mid-build rescore; continuing a saved graph; and search options.
+  Run it from the project root with `mvn compile exec:exec@index-api-example`.
+
+## Moving from GraphIndexBuilder to HnswIndexBuilder (optional)
+
+`GraphIndexBuilder` remains supported; moving is optional in this release.
+
+| `GraphIndexBuilder` | `HnswIndexBuilder` |
+|---|---|
+| `new GraphIndexBuilder(ravv, vsf, M, beamWidth, overflow, alpha, addHierarchy).build(ravv)` | `Indexes.hnswBuilder(ravv, vsf).withMaxDegree(M).withBeamWidth(beamWidth).withNeighborOverflow(overflow).withAlpha(alpha).withAddHierarchy(addHierarchy).buildAndPopulate()`, omitting any setting that matches the default |
+| constructors taking a `BuildScoreProvider` and dimension | `Indexes.hnswBuilder(bsp, dimension)` |
+| computing PQ or BQ vectors yourself to build with compressed scores | `Indexes.hnswBuilder(ravv, vsf).withCompressionType(CompressionType.PQ)` (or `BQ`) |
+| `List<Integer>` max degrees | `withMaxDegrees(list)` |
+| `refineFinalGraph`, SIMD and parallel executor arguments | `withRefineFinalGraph`, `withBuildExecutor` (the SIMD executor), `withMaintenanceExecutor` (the parallel executor) |
+| `addGraphNode`, `markNodeDeleted`, `removeDeletedNodes`, `cleanup`, `insertsInProgress` | the same methods on `HnswIndexBuilder`, after `build()` |
+| `GraphIndexBuilder.rescore(builder, newBsp)` | `HnswIndexBuilder.rescore(builder, newBsp)` |
+| existing-graph constructor, or `buildAndMergeNewNodes` | `OnHeapGraphIndex.load(...)`, then `withExistingGraph(graph)` |
+| `builder.getGraph()` | `builder.getGraph()` or `builder.build()` (a `PersistableGraphIndex`) |
+
+Differences to be aware of:
+- `addHierarchy`, `refineFinalGraph` and the build compression are always explicit (defaulting to `true`, `true`
+  and `CompressionType.NONE`). The `GraphIndexBuilder.builder(...)` fluent builder from 4.0.x reads them from the
+  JMX `GraphIndexBuilderConfig` instead. So are the PQ subspace count, centering and anisotropic threshold
+  (`withPqSubspaces`, `withPqGlobalCentering`, `withPqAnisotropicThreshold`); only the PQ cluster count still comes
+  from `GraphIndexBuilderConfig`.
+- Settings are validated when the graph is built, by `GraphIndexBuilder`, which throws an
+  `IllegalArgumentException` for the first invalid value. Null inputs and a non-positive dimension are rejected
+  immediately. Settings changed after the graph is built are ignored, with a logged warning.
+- With `withExistingGraph`, calling `withMaxDegree(s)` or `withAddHierarchy` as well is rejected with an
+  `IllegalStateException`, because the existing graph fixes both. So is scoring of a different kind than the graph
+  was built with: a graph built with exact scores can't be continued with compressed ones, or the reverse, and
+  `withCompressionType` can't be combined with an existing graph, since it would train a new quantizer. To continue
+  a graph built with compressed vectors, use `Indexes.hnswBuilder(scoreProvider, dimension)` with the score provider
+  the graph was built with. The existing graph keeps the diversity provider
+  it was created with, which must be able to score the ordinals you append.
+- Concurrency is unchanged from `GraphIndexBuilder`: `addGraphNode` and `markNodeDeleted` are thread-safe, but
+  `cleanup()`, `removeDeletedNodes()`, `rescore` and `close()` must not run while inserts are in progress.
+- As with `GraphIndexBuilder.addGraphNode`, every node must be scoreable by ordinal through the build score
+  provider before (or when) it is added: add the vector to your vector values, or encode its PQ code, first.
+  Otherwise a later insert or `cleanup()` fails with an `IndexOutOfBoundsException`.
+
+## Behavior changes
+
+- Adding nodes to an `OnHeapGraphIndex` after `cleanup()` now works correctly. Previously the graph stayed
+  "frozen" after cleanup, so a later insert could link a node to itself (tripping an assertion) and concurrent
+  searches could see half-inserted nodes. Adding a node now unfreezes the graph; call `cleanup()` again before
+  writing it, and don't reuse views or searchers obtained while it was frozen. This also affects continuing to
+  build on a graph that has already been cleaned up, including one returned by `GraphIndexBuilder.build`.
+- `GraphIndexBuilder.rescore` now keeps nodes that were marked deleted (with `markNodeDeleted`) but not yet
+  removed. Previously the rescored graph dropped those marks, so the nodes became visible to searches again and
+  survived `cleanup()`. Callers that don't delete before rescoring, such as a compaction refining its PQ
+  codebook, see no change.
+- The writer builders that take a `Path` (`OnDiskGraphIndexWriter.Builder`, `OnDiskParallelGraphIndexWriter.Builder`,
+  `RandomAccessOnDiskGraphIndexWriter.Builder`, and the `PersistableGraphIndex` accessors) now open the file in
+  `build()` rather than in the constructor. A builder that is never built, or whose `build()` fails, no longer leaks
+  an open file. An error opening the file (for example, a missing directory) is now thrown by `build()`.
+- When `GraphIndexBuilderConfig` (JMX) selects PQ or BQ build compression for `GraphIndexBuilder`, the vectors are
+  now encoded on `PhysicalCoreExecutor.pool()` rather than `ForkJoinPool.commonPool()`. Results are unchanged.
+
+## Other changes to public classes
+
+- `GraphSearcher` implements `IndexSearcher`.
+- `AbstractGraphIndexWriter.Builder` implements `PersistableGraphIndex.GraphIndexWriterBuilder`, the options
+  every writer supports (`with`, `withMapper`, `withMap`, `withVersion`, `build`). `OnDiskGraphIndexWriter.Builder`,
+  `OnDiskParallelGraphIndexWriter.Builder` and `OnDiskSequentialGraphIndexWriter.Builder` override those to return
+  their own type, so writer-specific options can follow them in a chain.
+- `@Experimental` is now `@Documented`, so it appears in the generated Javadoc.
+- `GraphIndex` extends `Accountable`, as `ImmutableGraphIndex` did.
+- Writer documentation corrected, with no behavior change: `withParallelWorkerThreads` and
+  `withParallelDirectBuffers` apply only to the parallel writer, and `RandomAccessOnDiskGraphIndexWriter.Builder`
+  (which picks a writer from the JMX configuration) ignores them when it picks the single-threaded one. A worker
+  count of 0 or less means "use all available processors".
+
 # Upgrading from 3.0.x to 4.0.x
 
 ## New features

benchmarks-jmh/src/main/java/io/github/jbellis/jvector/bench/CompactorBenchmark.java

@@ -919,7 +919,7 @@ private SearchStats runRecall(Path indexPath) throws Exception {
         try (var rs = ReaderSupplierFactory.open(indexPath)) {
             var graph = OnDiskGraphIndex.load(rs);
             GraphSearcher searcher = new GraphSearcher(graph);
-            var view = (ImmutableGraphIndex.ScoringView) searcher.getView();
+            var view = (GraphIndex.ScoringView) searcher.getView();
             searcher.usePruning(false);
             List<SearchResult> retrieved = new ArrayList<>(queryVectors.size());
             long[] searchLatenciesNs = new long[queryVectors.size()];

benchmarks-jmh/src/main/java/io/github/jbellis/jvector/bench/ParallelWriteBenchmark.java

@@ -16,11 +16,9 @@
 package io.github.jbellis.jvector.bench;
 
 import io.github.jbellis.jvector.disk.ReaderSupplierFactory;
-import io.github.jbellis.jvector.graph.GraphIndexBuilder;
-import io.github.jbellis.jvector.graph.ImmutableGraphIndex;
-import io.github.jbellis.jvector.graph.ListRandomAccessVectorValues;
-import io.github.jbellis.jvector.graph.NodesIterator;
-import io.github.jbellis.jvector.graph.RandomAccessVectorValues;
+import io.github.jbellis.jvector.graph.*;
+import io.github.jbellis.jvector.graph.disk.*;
+import io.github.jbellis.jvector.graph.*;
 import io.github.jbellis.jvector.graph.disk.*;
 import io.github.jbellis.jvector.graph.disk.feature.Feature;
 import io.github.jbellis.jvector.graph.disk.feature.FeatureId;
@@ -82,7 +80,7 @@ public class ParallelWriteBenchmark {
     // Dataset and index state
     private RandomAccessVectorValues floatVectors;
     private PQVectors pqVectors;
-    private ImmutableGraphIndex graph;
+    private GraphIndex graph;
 
     // Feature state reused between iterations
     private NVQ nvqFeature;
@@ -184,7 +182,7 @@ public void writeSequentialThenParallelAndVerify(Blackhole blackhole) throws IOE
         Files.deleteIfExists(parallelPath);
     }
 
-    private void writeGraph(ImmutableGraphIndex graph,
+    private void writeGraph(GraphIndex graph,
                             Path path,
                             boolean parallel) throws IOException {
         try (RandomAccessOnDiskGraphIndexWriter writer = parallel ?

benchmarks-jmh/src/main/java/io/github/jbellis/jvector/bench/RecallWithRandomVectorsBenchmark.java

@@ -15,6 +15,7 @@
  */
 package io.github.jbellis.jvector.bench;
 
+import io.github.jbellis.jvector.graph.*;
 import io.github.jbellis.jvector.graph.*;
 import io.github.jbellis.jvector.graph.similarity.BuildScoreProvider;
 import io.github.jbellis.jvector.graph.similarity.DefaultSearchScoreProvider;
@@ -52,7 +53,7 @@ public class RecallWithRandomVectorsBenchmark {
     private ArrayList<VectorFloat<?>> baseVectors;
     private ArrayList<VectorFloat<?>> queryVectors;
     private GraphIndexBuilder graphIndexBuilder;
-    private ImmutableGraphIndex graphIndex;
+    private GraphIndex graphIndex;
     private PQVectors pqVectors;
 
     // Add ground truth storage

benchmarks-jmh/src/main/java/io/github/jbellis/jvector/bench/StaticSetVectorsBenchmark.java

@@ -45,7 +45,7 @@ public class StaticSetVectorsBenchmark {
     private List<VectorFloat<?>> queryVectors;
     private List<List<Integer>> groundTruth;
     private GraphIndexBuilder graphIndexBuilder;
-    private ImmutableGraphIndex graphIndex;
+    private GraphIndex graphIndex;
     int originalDimension;
 
     @Setup

docs/ash-symmetric-study/final-asymmetric-controls.csv

@@ -0,0 +1,13 @@
+dataset,mode,rep,build_s,qps,recall,visited,qps_cv,provider_s
+ada002-100k,asymmetric-cached,1,19.062567521,150638.30920807895,0.86243,547.9675,1.3794590964550122,0.720251
+cohere-english-v3-100k,asymmetric-cached,1,14.910241575,231653.94557050234,0.82729,657.4614,3.453374124251149,0.324744
+e5-large-v2-100k,asymmetric-cached,1,15.278566978,219437.87965171272,0.81166,668.5197,2.012703348778286,0.308676
+ada002-1M,asymmetric-cached,1,222.707469286,131708.20389569228,0.82501,751.7288,5.299918739807189,5.510627
+cohere-english-v3-1M,asymmetric-cached,1,308.143331078,221832.42654368113,0.69536,883.0375,4.083988556683095,1.484941
+cap-1M,asymmetric-cached,1,207.220140035,133480.11597603708,0.7895,757.8936,6.853695150484436,5.783209
+cap-1M,asymmetric-cached,2,207.996352629,147046.0693958429,0.78781,750.4877,1.6034492683831063,5.626615
+cohere-english-v3-1M,asymmetric-cached,2,155.049321232,240774.58995543895,0.69493,890.907,1.8581733690493571,1.429855
+ada002-1M,asymmetric-cached,2,229.365798497,84947.42036576896,0.82422,752.1287,3.5711789391588313,5.794614
+e5-large-v2-100k,asymmetric-cached,2,15.478295209,222987.4528446198,0.81012,669.5041,2.8123015639604487,0.334623
+cohere-english-v3-100k,asymmetric-cached,2,15.296824291,227276.05582969685,0.82779,653.8773,2.4048198767111466,0.29074
+ada002-100k,asymmetric-cached,2,18.888044342,137221.1547473062,0.86242,545.7134,1.0185284705650302,0.720043