Skip to content

Optimize recall calculation and enhance CAGRA search features - #2502

Draft
sherylll wants to merge 5 commits into
NVIDIA:mainfrom
sherylll:improve-cagra-heuristic
Draft

Optimize recall calculation and enhance CAGRA search features#2502
sherylll wants to merge 5 commits into
NVIDIA:mainfrom
sherylll:improve-cagra-heuristic

Conversation

@sherylll

@sherylll sherylll commented Aug 25, 2026

Copy link
Copy Markdown
  • add new ANN_BENCH knobs: build_quality/search_quality/hashmap_min_bitlen
  • enable auto search_width selection
  • increase default hashmap_min_bitlen
  • fix slow recall calculation issue at low batch

@sherylll
Fix: reduce recall calculation time at small batch
e738eec 
Decoupes work assignment in recall calculatoin loop from batch size (n_queries) used for search.
This prevents each worker from having very little to do when batch size is small.
@sherylll
Increase default hashmap_min_bitlen from 8 to 11
0da38ef 
Increase default small hashmap bit length to give interval a chance to grow.
User can still set this to 8,9,10 manually.
@sherylll
Add automatic CAGRA search width
9f99e5c 
Allow search_width=0 to scale exploration with itopk and graph degree, avoiding overly narrow SINGLE_CTA searches at large itopk while preserving MULTI_CTA's existing CTA floor.
@sherylll
Parse CAGRA hashmap minimum in benchmarks
399a4f2 
Expose hashmap_min_bitlen through ANN benchmark configurations so hash sizing heuristics can be evaluated directly.
@sherylll
Adapt MULTI_CTA iterations to search breadth
3ef607a 
Add CAGRA search quality control

At low batch sizes, increasing itopk adds parallel CTAs cheaply, so reduce local iterations as itopk grows. Larger candidate pools tolerate shallower traversal; reserve high iteration counts for explicit extreme-recall tuning.

For multi-CTA algo, search_quality controls the iteration budget, while
for single-CTA algo, it controls the iteration * search_width budget.
@sherylll
sherylll requested review from a team as code owners August 25, 2026 11:34
@copy-pr-bot

copy-pr-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@sherylll

sherylll commented Aug 25, 2026

Copy link
Copy Markdown
Author

search_quality indicates the upper bound of recall, with a given search_quality, itopk becomes the main knob, and search_width and max_iterations should be left auto.

E.g. search_quality = 4, varying itopk:

dim: 768
distance: euclidean
gpu_driver_version: 13.3
gpu_gpuDirectRDMASupported: 1
gpu_hostNativeAtomicSupported: 0
gpu_mem_bus_width: 7680
gpu_mem_freq: 3996000000.000000
gpu_mem_global_size: 287416123392
gpu_mem_shared_size: 233472
gpu_name: NVIDIA B300 SXM6 AC
gpu_pageableMemoryAccess: 1
gpu_pageableMemoryAccessUsesHostPageTables: 0
gpu_runtime_version: 13.3
gpu_sm_count: 148
gpu_sm_freq: 2032000000.000000
host_cores_used: 14
host_cpu_freq_max: 4000000000
host_cpu_freq_min: 800000000
host_pagesize: 4096
host_processors_sysconf: 224
host_processors_used: 28
host_total_ram_size: 2164172902400
host_total_swap_size: 0
max_k: 100
max_n_queries: 10000
***WARNING*** ASLR is enabled, the results may have unreproducible noise in them.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                      Time             CPU   Iterations        GPU    Latency     Recall end_to_end items_per_second      itopk          k max_iterations  n_queries search_quality search_width total_queries
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
cuvs_cagra_wiki_all_1m_g64/0/0/process_time/real_time      0.349 ms        0.349 ms         1000   339.159u   348.608u     0.9781   0.348608       2.86856k/s         64         10              0          1              4            0            1k
cuvs_cagra_wiki_all_1m_g64/1/0/process_time/real_time      0.552 ms        0.553 ms         1000    542.22u   552.331u    0.97829   0.552331       181.052k/s         64         10              0        100              4            0          100k
cuvs_cagra_wiki_all_1m_g64/2/0/process_time/real_time       2.60 ms         2.60 ms         1000   2.59399m   2.60389m    0.97548    2.60389       384.041k/s         64         10              0         1k              4            0            1M
cuvs_cagra_wiki_all_1m_g64/0/1/process_time/real_time      0.342 ms        0.341 ms         1000   331.807u   341.852u     0.9909   0.341852       2.92526k/s        128         10              0          1              4            0            1k
cuvs_cagra_wiki_all_1m_g64/1/1/process_time/real_time      0.793 ms        0.793 ms         1000   782.297u   792.774u    0.99228   0.792774        126.14k/s        128         10              0        100              4            0          100k
cuvs_cagra_wiki_all_1m_g64/2/1/process_time/real_time       5.05 ms         5.05 ms         1000    5.0405m   5.05039m    0.99214    5.05039       198.005k/s        128         10              0         1k              4            0            1M
cuvs_cagra_wiki_all_1m_g64/0/2/process_time/real_time      0.344 ms        0.344 ms         1000   334.638u   344.499u      0.998   0.344499       2.90278k/s        256         10              0          1              4            0            1k
cuvs_cagra_wiki_all_1m_g64/1/2/process_time/real_time       1.31 ms         1.31 ms         1000   1.29806m   1.30797m    0.99761    1.30797       76.4547k/s        256         10              0        100              4            0          100k
cuvs_cagra_wiki_all_1m_g64/2/2/process_time/real_time       10.2 ms         10.2 ms         1000  0.0101769  0.0101869    0.99727    10.1869       98.1656k/s        256         10              0         1k              4            0            1M

search_quality=3:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                      Time             CPU   Iterations        GPU    Latency     Recall end_to_end items_per_second      itopk          k max_iterations  n_queries search_quality search_width total_queries
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
cuvs_cagra_wiki_all_1m_g64/0/0/process_time/real_time      0.309 ms        0.309 ms         1000   299.526u   309.052u     0.9733   0.309052       3.23572k/s         64         10              0          1              3            0            1k
cuvs_cagra_wiki_all_1m_g64/1/0/process_time/real_time      0.489 ms        0.490 ms         1000   478.973u   489.381u    0.97263   0.489381       204.341k/s         64         10              0        100              3            0          100k
cuvs_cagra_wiki_all_1m_g64/2/0/process_time/real_time       2.35 ms         2.35 ms         1000   2.34319m   2.35307m    0.97192    2.35307       424.977k/s         64         10              0         1k              3            0            1M
cuvs_cagra_wiki_all_1m_g64/0/1/process_time/real_time      0.281 ms        0.281 ms         1000   271.073u   280.577u     0.9884   0.280577       3.56411k/s        128         10              0          1              3            0            1k
cuvs_cagra_wiki_all_1m_g64/1/1/process_time/real_time      0.644 ms        0.644 ms         1000   634.251u   644.058u    0.98864   0.644058       155.266k/s        128         10              0        100              3            0          100k
cuvs_cagra_wiki_all_1m_g64/2/1/process_time/real_time       4.58 ms         4.58 ms         1000   4.56504m   4.57504m    0.99105    4.57504       218.577k/s        128         10              0         1k              3            0            1M
cuvs_cagra_wiki_all_1m_g64/0/2/process_time/real_time      0.270 ms        0.270 ms         1000   261.011u    270.45u     0.9965    0.27045       3.69756k/s        256         10              0          1              3            0            1k
cuvs_cagra_wiki_all_1m_g64/1/2/process_time/real_time       1.01 ms         1.01 ms         1000   999.602u   1.00961m     0.9961    1.00961       99.0482k/s        256         10              0        100              3            0          100k
cuvs_cagra_wiki_all_1m_g64/2/2/process_time/real_time       9.20 ms         9.20 ms         1000   9.18693m   9.19691m    0.99653    9.19691       108.732k/s        256         10              0         1k              3            0            1M

@cjnolet

cjnolet commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Hi @sherylll, please also make sure any relevant configured parameters/options are also being proposed through the Python scripts where necessary (users always go through the Python scripts).

I haven't looked at this PR, just pointing this out, since I know you're onboarding to cuVS.

@sherylll
sherylll marked this pull request as draft August 26, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sherylll @cjnolet