Repository navigation
Graph build is about 25% slower with the native (Highway) provider than with Panama at 128 dimensions on an AVX2 CPU (4.0.2) #771
Description
Since 4.0.2 prefers the native provider by default (#751), a plain upgrade changes the provider on Java 22+ Linux x86-64 and aarch64. On a 128-d graph build with jvector's own GraphIndexBuilder we see native on slower than native off, and we would like to know whether that is expected. I searched issues and PRs for native, Highway, FFM and Panama performance and found no report of this. The closest statements are in #761 ("Graph/write remains slower in several cases ... its cause is not established", Cohere/JDK 22 graph/write +7.6%) and #751 (total build time lower in seven of eight cases, Cohere/JDK 22 +8%, "does not isolate the default switch"); both measured 768 to 3072 dimensions with PQ on AVX-512.
What we measured
jvector alone, GraphIndexBuilder on the first 100,000 SIFT1M vectors (128-d float, EUCLIDEAN, FLAT, no PQ): M 32, beamWidth 100, neighborOverflow 1.2, alpha 1.2, hierarchy on, vectors made by vts.createFloatVector, scored through BuildScoreProvider.randomAccessScoreProvider over a ListRandomAccessVectorValues. One build per fresh JVM. Native on is the 4.0.2 default (the program prints NativeVectorizationProvider and MemorySegmentVectorFloat); native off is -Djvector.enable_native_vectorization=false (PanamaVectorizationProvider, ArrayVectorFloat). Both arms use --add-modules=jdk.incubator.vector --enable-native-access=ALL-UNNAMED.
Host: i9-12900HK (AVX2, no AVX-512 flags, so the Highway tier is avx2), Temurin 25.0.4+7 in a container with 2 pinned P-cores, -Xmx2g, -XX:+UseCompactObjectHeaders. Ratio is the second arm over the first, so above 1 means the second arm is slower. Median of per-pair ratios with a bootstrap 95% CI.
| workload | comparison | clean pairs | median ratio [95% CI] |
|---|---|---|---|
| jvector alone, 100k x 128-d | native on vs native off | 9 | 1.245 [1.141, 1.301] (native on slower by 14 to 30%) |
| jvector alone, 100k x 128-d | native off vs 4.0.1 | 12 | 1.009 [0.979, 1.063] (no detectable difference) |
For context, the same swap inside a database that uses jvector (ArcadeDB LSM_VECTOR, same M and beamWidth):
| workload | comparison | clean pairs | median ratio [95% CI] |
|---|---|---|---|
| ArcadeDB build, FLAT, 100k x 128-d | native on vs native off | 14 | 1.256 [1.221, 1.286] |
| ArcadeDB build, INT8, 100k x 128-d | native on vs native off | 19 | 1.257 [1.249, 1.276] |
| ArcadeDB build, 90k x 768-d | native on vs native off | 7 | 0.986 [0.948, 1.020] (no difference) |
| ArcadeDB build, 50k x 1536-d | native on vs native off | 6 | too few clean pairs |
Method: arms alternate in ABBA rounds (round 0 dropped) and each run is a fresh JVM. A pair is dropped when its two arms start more than 5 C apart or differ by more than 1 point in the share of their own run time spent thermally throttled, which is why the high-dimension cells keep few pairs: the native arm runs hotter on this mobile CPU and throttles a larger share, so the pair rule discards most of its pairs.
What is open
- High dimensions on a CPU that throttles. At 768-d we see no difference (7 pairs) and at 1536-d we cannot judge (6 pairs). We will rerun them on a setup that does not throttle.
- AVX-512 and aarch64 were not measured here. The 4.0.2 notes report up to 29% faster construction on Graviton, so this may be specific to AVX2 at short vectors.
- PQ builds were not measured, so the batched work from Batch native Highway operations to reduce Java/native overhead #761 (training, encoding) is not in these numbers.
- Only 2 CPUs were used; we have not looked at how it scales with more build threads.
Where the time goes (untimed, a hypothesis)
An async-profiler run of the ArcadeDB build (not of the repro below), 50k x 128-d, on a laptop with the same avx2 tier: stacks under VectorUtil.squareL2Distance were 62% of samples with native on and 21% with Panama. With native on, the Highway kernel euclidean_f32 was about 12% of samples; the Java side of the call was about half: MemorySegmentVectorFloat.length() 17.5%, the downcall method-handle chain 15%, the VarHandle read of the storage field 12%. The first load of a candidate's vector object can take a cache miss that the profiler attributes to those frames, so this does not separate instruction cost from stalls. Reading the 4.0.2 code, each distance call runs length() on both vectors in VectorUtil and again inside NativeVectorUtilSupport.squareDistance, plus two get() calls, and each vector owns an extra MemorySegment view object after its first native call. At 128-d the distance is four loop iterations, so a fixed per-call cost could be a large share of it; that would also fit the 768-d result. We have not tested a change.
Main is not different here: #761 is already in 4.0.2, and main (4f13ff4, 4.0.3-SNAPSHOT) differs from the 4.0.2 tag only by release notes and the version bump. So the numbers above are the current state of main for this path, and we did not rebuild main separately.
Question
Is the native path expected to be slower than Panama for graph builds at 128-d on AVX2? If it is known, a note in the 4.0.2 upgrade text would help, since the default switch applies it to builds that do not use PQ. If it is not expected, would a cheaper per-call path (for example fewer length() checks per distance, or a batched call over several ordinals) be something you want to look at, or is the native path meant to be chosen per dimension or ISA?
Minimal repro
Plain jvector, no other code. Run it twice, once as is and once with -Djvector.enable_native_vectorization=false, alternating, on pinned cores.
JvRepro.java
import io.github.jbellis.jvector.graph.GraphIndexBuilder;
import io.github.jbellis.jvector.graph.ListRandomAccessVectorValues;
import io.github.jbellis.jvector.graph.similarity.BuildScoreProvider;
import io.github.jbellis.jvector.vector.VectorSimilarityFunction;
import io.github.jbellis.jvector.vector.VectorizationProvider;
import io.github.jbellis.jvector.vector.types.VectorFloat;
import io.github.jbellis.jvector.vector.types.VectorTypeSupport;
import java.io.BufferedInputStream;
import java.io.DataInputStream;
import java.io.FileInputStream;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
public class JvRepro {
static final int DIM = 128;
public static void main(final String[] a) throws Exception {
final int n = Integer.parseInt(a[0]);
final VectorizationProvider vp = VectorizationProvider.getInstance();
final VectorTypeSupport vts = vp.getVectorTypeSupport();
System.out.println("provider " + vp.getClass().getSimpleName() + ", float vector " + vts.createFloatVector(DIM).getClass().getSimpleName()
+ ", cpus " + Runtime.getRuntime().availableProcessors() + ", java " + System.getProperty("java.version"));
final List<VectorFloat<?>> vecs = new ArrayList<>(n);
if (a[1].equals("random")) {
final Random r = new Random(42);
for (int i = 0; i < n; i++) {
final float[] v = new float[DIM];
for (int j = 0; j < DIM; j++)
v[j] = (float) r.nextGaussian();
vecs.add(vts.createFloatVector(v));
}
} else {
// .fvecs: per vector a little-endian int dimension (128) followed by that many little-endian floats
try (DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(a[1]), 1 << 20))) {
final byte[] rec = new byte[4 + DIM * 4];
for (int i = 0; i < n; i++) {
in.readFully(rec);
final ByteBuffer b = ByteBuffer.wrap(rec).order(ByteOrder.LITTLE_ENDIAN);
if (b.getInt() != DIM)
throw new IllegalStateException("expected 128-d vectors");
final float[] v = new float[DIM];
for (int j = 0; j < DIM; j++)
v[j] = b.getFloat();
vecs.add(vts.createFloatVector(v));
}
}
}
final var ravv = new ListRandomAccessVectorValues(vecs, DIM);
final BuildScoreProvider bsp = BuildScoreProvider.randomAccessScoreProvider(ravv, VectorSimilarityFunction.EUCLIDEAN);
final long t0 = System.nanoTime();
try (GraphIndexBuilder builder = new GraphIndexBuilder(bsp, DIM, 32, 100, 1.2f, 1.2f, true)) {
final var graph = builder.build(ravv);
System.out.printf("build %.3f s, %d nodes%n", (System.nanoTime() - t0) / 1e9, graph.size());
}
System.exit(0);
}
}java --add-modules=jdk.incubator.vector --enable-native-access=ALL-UNNAMED [-Djvector.enable_native_vectorization=false] \
-cp jvector-4.0.2.jar:agrona-1.20.0.jar:slf4j-api-2.0.20.jar:commons-math3-3.6.1.jar JvRepro.java 100000 sift_base.fvecs
sift_base.fvecs is the SIFT1M base file (the first 100,000 vectors are read); random in place of the path uses seeded Gaussian vectors.
The reproducer runs as given: one untimed run per arm on 2 laptop cores (Temurin 25, jvector 4.0.2, the same 100k SIFT vectors) built in 89.2 s with native vectorization on and 64.7 s with it off. That is a direction check only; the figures in the table come from the paired runs above.
Activity
r-devulap commentedon Oct 22, 2026
Thanks for reporting. It is not expected that AVX2 native is slower than Panama. Will need to debug.
tae898 commentedon Oct 22, 2026
Thanks. We have a candidate fix on the Java side of the native single-pair path, so you may not need to start from zero: (1) read each operand's length once from its segment instead of calling
length()per call in theNativeVectorUtilSupporttwo-vector wrappers, and (2) backMemorySegmentVectorFloatwith a finalfloat[]plus a lazily createdMemorySegmentview instead of the single volatile slot. Kernels, ABI and the Panama/default providers are untouched; jvector-native tests pass (47, including two new ones).In an untimed async-profiler run of the same 128-d build, the wrapper frames under
VectorUtil.squareL2Distancego from about 62% of samples to about 6%, and the Highway kernel frame becomes the largest one. That is a profile share, not a time: we are running a proper A/B (base / patched / Panama, 128, 768 and 1536 dimensions) and will open a PR with the timings once they are in, unless you would rather look at the patch first.r-devulap commentedon Oct 22, 2026
@tae898 quick question: what OS and compiler are being used to build the native module?
tae898 commentedon Oct 22, 2026
We did not build it: we used the
libjvector-x86_64.soshipped in the jvector 4.0.2 jar from Maven Central (META-INF/versions/22/). Its.commentsection saysGCC: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0, so your release build (Ubuntu 24.04, GCC 13.3), NEEDED: libstdc++.so.6, libm.so.6, libgcc_s.so.1, libc.so.6. Highway selects the avx2 tier on our CPU (i9-12900HK, no AVX-512).Runtime: Temurin 25.0.4+7 in a container on Ubuntu 26.04 (kernel 7.0) x86-64,
--add-modules=jdk.incubator.vector --enable-native-access=ALL-UNNAMED. The candidate fix in my previous comment changes only Java code in jvector-native; the.sois the same release binary in both arms.tae898 commentedon Oct 22, 2026
Timed results are in, and they correct what I said about our patch. JMH per-call time, ABBA-paired runs scored with bootstrap intervals (10-12 clean pairs per figure), jvector 4.0.2 jars, JDK 25, two E-cores of an Intel Core Ultra X9 388H (Highway picks the avx2 tier). Ratio = B time / A time, so above 1 means B is slower:
So:
tae898 commentedon Oct 22, 2026
Correction to the table above: those runs shared their two cores with another benchmark job of ours (a scheduling error on my side). Here is a clean rerun with the same jars and protocol and nothing else on those cores, 11 clean pairs per figure:
The conclusions stand, with smaller numbers. Native is 43-68% slower than Panama at 128 dimensions and close to even at 768 and 1536 (dotProduct 768 is still 8% slower). Our patch gains at most about 2%, and the 2% loss at squareL2 1536 I reported was an artefact of the shared cores.
r-devulap commentedon Oct 27, 2026
Hi @tae898, I'm currently measuring the performance degradation when using native vectorization. I don't think this is AVX2-specific, since we're seeing the same behavior on AVX-512 as well. At the moment, it appears that the FFI overhead is the main contributor.
While we're working on a fix, it's worth noting that although native vectorization is now the default, it can be disabled with:
tae898 commentedon Oct 27, 2026
Thanks for confirming, and for the AVX-512 data point. For the same reason, ArcadeDB main took 4.0.2 with native vectorization off unless configured (ArcadeData/arcadedb#10540), so ArcadeDB stays on Panama until a fix lands. When you have a branch that batches the scoring (or otherwise cuts the per-call cost), point me at it and I will run the same JMH protocol at 128, 768 and 1536 dimensions on our avx2 machines.