Repository navigation
Reject duplicate neighbor IDs while preserving score order - #738
Conversation
|
Before you submit for review:
If you did not complete any of these, then please explain below. |
PQ benchmark results — performance not clearedCompleted 12 fresh PQ builds (two per arm per dataset), 12 retained-index query retests, and four separate CAP JFR construction profiles. Baseline Configuration: JDK 23, Panama 512-bit SIMD, verified Means below; QPS is from the separate retained-index retest with sequential warming, verified initial residency, three real-query warmup passes, and three samples of at least 10 seconds. Initial residency does not guarantee residency throughout execution.
CorrectnessEvery fixed graph has zero duplicate neighbor IDs at every level. Baseline extra duplicate entries per build: Ada002 180,869 / 180,227; CAP 199,123 / 199,720; Cohere 770,411 / 769,181. The 27 focused regression tests passed. Recall differences are small, but two builds do not establish statistical significance. Timing limitations and profilingDo not interpret construction means as an isolated deduplication cost. Raw build times (seconds): Ada main 249.24/916.22, fix 201.76/886.65; CAP main 316.40/241.58, fix 645.70/886.56; Cohere main 187.54/172.90, fix 784.38/673.22. All outliers are retained. Query timing also varies across JVMs despite warming. Separate CAP JFR runs locate most slow-run time in insertion. Insertion times main/fix/fix/main were 1337.00/1277.67/227.96/869.51 seconds (diagnostic timings, not pooled above). Slow runs on both branches heavily sample Recommendation: keep this PR draft. Correctness is demonstrated; performance is not cleared. Investigate the shared PQ compilation instability separately before requesting readiness. No ASH, reader implementation, query scorer, or format changes are included in this PR. |
CAP-100k construction diagnosis: shared PQ/JIT instabilityFollow-up on the construction-time variability above. Tested exactly the first 100,000 CAP vectors, with the same shared PQ codebook, graph parameters, JDK23, 48 construction workers, Panama512, and verified MemorySegmentReader. No recall measurement for this prefix. Production artifacts remain unchanged. Uninstrumented insertion times in main/fix/fix/main order: 15.84 / 55.53 / 21.09 / 83.05 seconds. Both branches exhibit the problem. Four separate JFR/compiler-log runs gave insertion 57.82 / 21.48 / 19.71 / 21.86 seconds. All reached C2 early. The slow kernel lacks Separate timing instrumentation measured 1.251B vs 1.258B insertion diversity comparisons (+0.59%), but sampled cost rose from 670 ns to 2,869 ns per comparison. Search scoring did not slow down. Wrappers can change JIT decisions; these are diagnostic measurements, not isolated branch-overhead estimates. Allocation is a major part of this: GC logs show approximately 5.21 TB cumulatively reclaimed in the slow-main profiled JVM versus 12.9 GB in fast main (rounded before/after heap sizes, whole runs, not resident memory). JFR allocation stacks identify temporary vector objects and backing arrays in PQ conversions/gather/add operations. Short GC pauses alone substantially understate the allocation cost. Controls, again main/fix/fix/main insertion seconds:
During the observed 164.79-second insertion, system I/O pressure was only 0.007 seconds over the 162.36-second monitored window, system iowait 0.006%, and no Java D-state threads were sampled. This argues against disk stalls as the main cause of that run. Separate writeInline contention and one 24.59-second flush outlier remain documented; they are not being discarded. Keep this PR draft. The shared PQ compilation/allocation problem is established, but the exact trigger for the default-mode store-intrinsic failure and a production remedy remain unresolved. Stabilize that scorer before interpreting the large 1M build differences as deduplication overhead. Neither diagnostic JVM flag is a recommended production setting. |
|
This is good, regression test shows across the board improvement in recall 
full report attached |
The existing graph-level uniqueness test builds with exact DOT_PRODUCT, which is symmetric, so it passes on main. A PQ build scores the same edge differently from each endpoint; this test fails on main (4/4 seeds) with a repeated level-0 edge and passes with the fix.
Both IntHashSets started at Agrona's default capacity and rehashed repeatedly on every insertDiverse merge. Size them to 2x their element count so they stay under the 0.65 load factor for the whole merge.
|
@jshook thanks for your related PR #765 with fixes and suggestions. Incorporated #765’s asymmetric-PQ regression test and hash-set pre-sizing, preserving both commits. Added documentation clarifications and formatting cleanup. I did not adopt:
|
Summary
Asymmetric scoring (
A · quant(B) ≠ B · quant(A)) can assign different scores to the same neighbor ID. Duplicate checks restricted to equal-score entries then admit repeated IDs, consuming adjacency capacity and reducing effective degree. The problem is particularly apparent at higher levels in hierarchical indexes.For example, in one 999,000-vector OpenAI1536 graph built from main, redundant neighbor entries consumed 0.16% of adjacency entries at level 0, 8.2% at level 1, 25.1% at level 2, and 45.7% at level 3.
NodeArraychecks IDs independently of score and deduplicates entire candidate merges, including tails. Existing scores, score ordering, and equal-score interleaving are preserved.ConcurrentNeighborMapdeduplicates initial candidate batches, preserves merge inputs across concurrent update retries, and rejects duplicate offers before copying, eviction, or changes to pruning state.Supersedes @marianotepper's draft #522 while retaining score-sorted adjacency and the existing diversity-prefix optimization. API and index format are unchanged; rebuilding regenerates existing adjacency.
Validation
TestNodeArray,TestNeighbors, andGraphIndexBuilderTest: 27 test executions each in scalar and SIMD configurations on JDK 22, with zero failures, errors, or skips.The completed comparative benchmark covered MPNet, Cohere English v3, CAP, and OpenAI3072 under JDK 22/25: 72 fresh builds and all-layer audits, plus 256 query JVMs. All 36 fixed graphs contained zero duplicate neighbor IDs.
Benchmarking
Distinct edges broaden graph coverage and can improve recall, but increase traversal work and reduce QPS at fixed search settings.
The tables cover five datasets, both JDKs, and 40 matched pairs (80 graphs). Mean recall improved in all ten dataset/JDK groups at 1× and nine of ten at 2×. OpenAI3072/JDK25 mean recall decreased by 0.225 pp at 2×. QPS decreased; construction time varied. All 40 fixed graphs had zero duplicate neighbor IDs in all-layer audits. Four of 40 matched pairs had lower recall at either depth; the individual comparisons are retained below.
Construction and throughput
Values: main → PR #738 (relative change).
Recall@10
Values: main → PR #738, in percent; changes in percentage points. Recall columns are the means of four matched pairs per row. “Lower pairs” counts individual pairs (out of a possible of 4 pairs) with lower PR recall, shown as 1× / 2×; it does not label the row's mean as a loss.
1x OQ / 2x OQ
(out of 4)
Individual pairs with a loss at either depth are shown below. PersonaHub/JDK25 pair 1 improves at 1× and loses slightly at 2×.
Method
Approximately 1M vectors per dataset; all catalog queries and four matched seed combinations per dataset/JDK. The OpenAI3072 rows summarize the completed targeted seed-confirmation study; the other four datasets use the same two-by-two PQ/graph-seed design. Results are computed separately for each dataset/JDK, with no pooling between studies. One additional OpenAI3072/JDK25 observation with nonidentical PQ codebook bytes is retained separately and excluded from the matched-pair averages. M=32, efConstruction=100, FusedPQ/NVQ, pruning disabled, one query worker on CPU0. Actual PQ codebook bytes and hierarchy assignments match within each pair.
Construction is the median per-build total for training, encoding, NVQ setup, and graph/write. QPS is the mean of per-graph median measured QPS; recall is the mean across matched pairs. Every completed pair is included. These timings are descriptive results from the recall-confirmation study.
Exhaustive scoring checks
OpenAI3072 exhaustive PQ and NVQ scans produced identical encoded bytes, per-query score-bit hashes, and top-10 results between arms on both JDKs. Recall was 0.8531/0.8537 for the two PQ codebooks and 0.9981 for NVQ. These scans do not cover fused graph scoring.
Saved-graph diagnostics and global validation are complete.