You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Prepare cuda-bindings and cuda-python 13.5.0 for CUDA Toolkit 13.5, including refreshed bindings and API reference documentation.
Move the build Toolkit, current-minor CI rows, and cuda-bindings/cuda-core Pixi configurations to 13.5, retaining older-runtime coverage.
Keep cuda-core’s published bindings floor unchanged. During the Toolkit minor transition, regular core CI uses bindings built against matching headers instead of the older published floor.
Relax the Toolkit wheel minor pin for local tests of the build’s own minor, allowing testing before matching Toolkit wheels are published.
Add PTX 9.5 compatibility mapping and recognize Windows CUPTI’s cupti64_13.dll.
Add regression coverage for graph-node layouts, embedded batch parameters, aliasing, read-only views, and wrapper lifetimes.
Update the 13.5 release notes, documentation version navigation, Cython cache ignores, and file-check exclusions.
Follow-up nightly CI and test-dependency refinements are tracked in #3375.
It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.
Updated CUDA bindings for CUDA Toolkit 13.5, including batch memory-copy graph nodes, fabric-copy operations, tensor-map allocation classes, and new NVML capabilities.
Added Python 3.15 support and PTX ISA 9.5 version mapping. Python 3.10 is no longer supported.
Improved Windows detection of CUDA 13 CUPTI libraries.
Bug Fixes
Improved compatibility when testing CUDA Toolkit versions and selecting bindings.
Improved callback registration safety under free-threaded Python and excluded unpopulated NVML field queries from valid results.
Documentation
Updated CUDA 13.5 API documentation and release notes, including compatibility details and known issues.
This pull request updates build and test configuration to CUDA 13.5. It adds driver, runtime, and NVML bindings, tests, documentation, and release metadata for the updated toolkit.
Changes: CUDA 13.5 upgrade
Layer / File(s)
Summary
Build and CI configuration .github/workflows/*, ci/*, cuda_bindings_13/pixi.toml, cuda_core/pixi.toml, cuda_core/AGENTS.md, .gitignore, .pre-commit-config.yaml, .spdx-ignore
Build settings and CI matrices move to CUDA 13.5. CI exports build-version components and adjusts floor-binding wheel selection and toolkit constraints. Linux and Windows workflows download artifacts for core tests using floor bindings.
Driver declarations and wrappers cuda_bindings_13/cuda/bindings/_internal/driver*, cuda_bindings_13/cuda/bindings/_v2/driver.*, cuda_bindings_13/cuda/bindings/cy*driver.*, cuda_bindings_13/cuda/bindings/driver.pxd, cuda_bindings_13/docs/source/module/driver.rst
The bindings add batch memcpy and fabric-copy graph-node support, tensor-map allocation-class APIs, and CUDA 13.5 driver enums and attributes. Driver symbol lookup, wrappers, and graph parameter handling are updated.
Runtime API and batch memcpy graph parameters cuda_bindings_13/cuda/bindings/cyruntime.*, cuda_bindings_13/cuda/bindings/runtime.*, cuda_bindings_13/docs/source/module/runtime.rst, cuda_bindings_13/tests/test_cuda.py, cuda_bindings_13/tests/test_cudart.py
The runtime adds batch memcpy graph-node parameters, locality-domain resource options, and related enums and attributes. Tests cover graph-parameter views, assignment, read-only behavior, and the locality-domain sentinel.
NVML declarations and wrappers cuda_bindings_13/cuda/bindings/_internal/nvml*, cuda_bindings_13/cuda/bindings/cynvml.*, cuda_bindings_13/cuda/bindings/nvml.*
The bindings add GPU uptime, confidential-compute settings v2, NVLink information v3, remap-table, and NVLE structures and APIs. The NVML documentation describes MPS locality-domain behavior.
Version metadata and release support cuda_bindings_13/cuda/bindings/*, cuda_bindings_13/docs/*, cuda_python/docs/source/release/*, cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py
Generated-file metadata advances to 13.5 across binding components. The changes add PTX 9.5 mapping, cufile PanFS support, CUDA 13.5 release documentation, and a Windows CUPTI DLL candidate.
Possibly related issues
NVIDIA/cuda-python#2794 — The PR updates cufile bindings and CI behavior during a Toolkit minor transition.
NVIDIA/cuda-python#2449 — CI uses bindings built from the same commit in floor-version test cases.
NVIDIA/cuda-python#3221 — The PR updates CUDA minor versions across CI and package configuration, but does not add synchronization enforcement.
important: Add targeted tests for the new NVML APIs.
The changed tests cover CUDA driver graph wrappers, not these NVML functions or struct wrappers. Add round-trip coverage through from_buffer, from_data, and from_ptr. Exercise the shared remapTabSize behavior of fla_remap_tab_addr and gpa_remap_tab_addr, and validate the 32-byte nvle_key requirement. Hardware-dependent calls can skip on NotSupportedError.
Files not fully reviewed because their changes were unavailable: 1
cuda_bindings_13/cuda/bindings/driver.pyx
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Stability & Availability · Quick win
important: Reject None before calling _get_ptr().
GraphNodeParams().memcpy_batch = None passes the typed assignment, then calls a Cython method on None. The fabric_cpy_batch setter has the same failure. This can crash the interpreter instead of raising TypeError; the build does not enable nonecheck. (docs.cython.org)
Add explicit non-null validation in the generator, regenerate both setters, and add regression tests.
The reason will be displayed to describe this comment to others. Learn more.
Major · Stability & Availability · Quick win
important: Bound attrs reads by its allocation. The public numAttrs setter can set a value larger than the array allocated by attrs. In that state, this getter copies past the array boundary. For example, setting one attribute and then setting numAttrs = 2 makes params.attrs read a second, unallocated entry. Keep the count and allocation consistent, or reject an out-of-range count before reading. Regenerate the binding after correcting its source. (raw.githubusercontent.com)
As per path instructions, “Focus on typed-memoryview safety, GIL and nogil correctness, exception specifications, fused-type behavior, Python reference ownership, and consistency with related declarations and generated type stubs.”
The reason will be displayed to describe this comment to others. Learn more.
This is a known issue in the legacy (cython_gen) bindings and fixing it would require pervasively breaking existing API (removing the count attributes on structs, in this case numAttrs but there are many examples of this). cybind-generated modules do not have this issue, so will disappear naturally when the _v2 modules ship.
The reason will be displayed to describe this comment to others. Learn more.
@mdboom suggestion: I retract the must-fix recommendation for this PR. The bounds risk remains, but your explanation establishes that this is a known cython_gen limitation with an existing migration path through cybind and _v2. No change requested here.
Learnings used
Learnt from: CR
Repo: NVIDIA/cuda-python PR: 0
File: cuda_bindings_13/AGENTS.md:0-0
Timestamp: 2026-10-21T18:37:43.407Z
Learning: Applies to cuda_bindings_13/** : Source excerpt:
## Scope and principles
- **Style**: prioritize correctness and API compatibility over convenience
wrappers. High-level ergonomics belong in `cuda_core`, not here.
Learnt from: CR
Repo: NVIDIA/cuda-python PR: 0
File: cuda_bindings_13/CLAUDE.md:0-0
Timestamp: 2026-10-31T04:59:49.354Z
Learning: Applies to cuda_bindings_13/** : Source excerpt:
## Scope and principles
- **Style**: prioritize correctness and API compatibility over convenience
wrappers. High-level ergonomics belong in `cuda_core`, not here.
Learnt from: CR
Repo: NVIDIA/cuda-python PR: 0
File: cuda_bindings_13/CLAUDE.md:0-0
Timestamp: 2026-10-31T04:59:49.354Z
Learning: Source excerpt:
## Generated-source workflow
- **Do not hand-edit generated binding files**: many files under
`cuda/bindings/` (including `*.pyx` and `*.pxd`) are generated artifacts.
Learnt from: CR
Repo: NVIDIA/cuda-python PR: 0
File: cuda_bindings_13/AGENTS.md:0-0
Timestamp: 2026-10-31T05:00:01.371Z
Learning: Source excerpt:
## Generated-source workflow
- **Do not hand-edit generated binding files**: many files under
`cuda/bindings/` (including `*.pyx` and `*.pxd`) are generated artifacts.
The reason will be displayed to describe this comment to others. Learn more.
Major · Stability & Availability · Quick win
Supported by static analysis
Script executed:
sed -n '17205,17260p' cuda_bindings_13/cuda/bindings/runtime.pyx
sed -n '17960,17985p' cuda_bindings_13/cuda/bindings/runtime.pyx
rg -n '__dealloc__|_dsts|_srcs' cuda_bindings_13/cuda/bindings/runtime.pyx | sed -n '1,40p'
Repository: NVIDIA/cuda-python
Length of output: 4026
Script executed:
set -u
f=cuda_bindings_13/cuda/bindings/runtime.pyx
start=$(rg -n -F 'cdef class cudaMemcpyBatchNodeParams'"$f"| head -1 | cut -d: -f1)echo"== batch class from line $start =="
sed -n "${start},$((start+190))p""$f"echo"== retention pattern in cudaGraphNodeParams setters =="
rg -n -F -e 'self._kernel = ' -e 'self._memcpy = ' -e 'self._memcpyBatch = ' -e 'self._conditional = ' -e 'self._free = ' -e 'self._memset = ' -- "$f"||test"$?" -eq 1
echo"== cudaGraphNodeParams __init__ region =="
gstart=$(rg -n -F 'cdef class cudaGraphNodeParams'"$f"| head -1 | cut -d: -f1)
sed -n "${gstart},$((gstart+60))p""$f"
Repository: NVIDIA/cuda-python
Length of output: 8335
important: Retain the source wrapper in memcpyBatch.
The memcpyBatch setter copies the native struct but keeps no reference to the source cudaMemcpyBatchNodeParams. The source wrapper frees its attrs allocation in __dealloc__. After the caller drops the source, cudaGraphNodeParams.memcpyBatch.attrs can point to freed memory. A later graph call that reads these parameters can then read freed memory. The source wrapper also owns the dsts, srcs, and sizes helper objects, which can release their backing storage in the same way.
Keep the source wrapper alive for as long as the node parameters hold it. Add a retained attribute in the setter:
Declare _cymemcpyBatch with the other cudaGraphNodeParams attributes in runtime.pxd. Apply the change in the binding generator source and regenerate runtime.pyx. Do not edit the generated file directly. Do not assign self._memcpyBatch. The getter depends on it as a view into the node-params struct.
@mdboom suggestion: Understood. I will treat this as the same known legacy cython_gen limitation and will not request an ownership change in this PR. The lifetime risk remains. Would you like me to open a follow-up issue?
Learnings used
Learnt from: CR
Repo: NVIDIA/cuda-python PR: 0
File: cuda_bindings_13/CLAUDE.md:0-0
Timestamp: 2026-10-31T04:59:49.354Z
Learning: Source excerpt:
## Generated-source workflow
- **Do not hand-edit generated binding files**: many files under
`cuda/bindings/` (including `*.pyx` and `*.pxd`) are generated artifacts.
Learnt from: CR
Repo: NVIDIA/cuda-python PR: 0
File: cuda_bindings_13/AGENTS.md:0-0
Timestamp: 2026-10-31T05:00:01.371Z
Learning: Source excerpt:
## Generated-source workflow
- **Do not hand-edit generated binding files**: many files under
`cuda/bindings/` (including `*.pyx` and `*.pxd`) are generated artifacts.
Treats constraints as a hint, ignoring them if necessary to reach the requested smCount. Lets smCount be a non-multiple of coscheduledSmCount, filling the difference between SM count and already assigned co-scheduled groupings with other SMs. When used with cudaDevSmResourceGroupLocalityDomainId, backfill fills up to the requested smCount using the target locality domain first, then SMs not attributed to any locality domain, then SMs from other locality domains. If no SMs can be found in the requested locality domain, cudaErrorInvalidResourceConfiguration is returned.
Treats constraints as a hint, ignoring them if necessary to reach the requested smCount. Lets smCount be a non-multiple of coscheduledSmCount, filling the difference between SM count and already assigned co-scheduled groupings with other SMs. When used with cudaDevSmResourceGroupLocalityDomainId, backfill fills up to the requested smCount using the target locality domain first, then SMs not attributed to any locality domain, then SMs from other locality domains. If no SMs can be found in the requested locality domain, cudaErrorInvalidResourceConfiguration is returned. When used with ``cudaDevSmResourceGroupLocalityDomainNonPortableId``, backfill follows the same behavior after first consuming all SMs from the target locality domain.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Functional Correctness · Quick win
important: Use the exported flag name in the backfill description.
This text names cudaDevSmResourceGroupLocalityDomainNonPortableId, but the binding exports cudaDevSmResourceGroupLocalityDomainSupplementalSms. The current text directs users to a nonexistent API identifier. Fix the upstream generator input and regenerate this file instead of editing the generated RST directly. (raw.githubusercontent.com)
The reason will be displayed to describe this comment to others. Learn more.
This is a documentation bug that comes from the upstream headers. We can patch our headers to fix (and report upstream), but I don't think this should hold up the release.
The reason will be displayed to describe this comment to others. Learn more.
@mdboom suggestion: This upstream documentation bug should not block the release. My “important” label overstated its impact. The header patch, regeneration, and upstream report can be follow-up work.
Would you like me to open a GitHub issue for that follow-up?
important: Keep memcpyBatch from aliasing arrays owned by batch.
params.memcpyBatch = batch copies the struct bytes with string.memcpy. The copy keeps the dsts and srcs pointers that batch owns. CUDA_MEMCPY_BATCH_NODE_PARAMS_V1_st.__dealloc__ frees both arrays. When batch is deleted, or when batch.set_dsts_srcs(...) replaces the arrays, params.memcpyBatch.dsts and srcs can refer to freed memory. Keeping a reference to batch on params does not cover the set_dsts_srcs case.
Make CUgraphNodeParams own copies of the dsts and srcs arrays. Free those copies on reassignment and when CUgraphNodeParams is deallocated. Update test_memcpy_batch_assignment_is_a_shallow_copy to assert safe reads after batch is deleted. Do not document the aliasing as accepted behavior.
The reason will be displayed to describe this comment to others. Learn more.
Major · Maintainability & Code Quality · Quick win
important: Regenerate cuda_core/pixi.lock. The changed cu13 environment requires CUDA 13.5, but its committed lockfile still contains CUDA 13.4 packages. Locked pixi builds cannot use this manifest and lockfile together. (raw.githubusercontent.com)
As per coding guidelines: “If a PR changes a pixi.toml, update the corresponding pixi.lock in the same PR.”
Reviewing files that changed from the base of the PR and between 6a2d754 and e04118b.
Files selected for processing: 3
cuda_bindings_13/cuda/bindings/cydriver.pxd
cuda_bindings_13/cuda/bindings/cyruntime.pxd
cuda_bindings_13/tests/test_cudart.py
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
Sources consulted:
- cython/Cython/Utility/ImportExport.c at 3.2.0 · cython/cython · GitHub — https://github.com/cython/cython/blob/3.2.0/Cython/Utility/ImportExport.c
- cython/Cython/Compiler/ModuleNode.py at 3.2.0 · cython/cython · GitHub — https://github.com/cython/cython/blob/3.2.0/Cython/Compiler/ModuleNode.py
- https://raw.githubusercontent.com/cython/cython/3.2.0/Cython/Compiler/ModuleNode.py
- …and 25 more
Declare the new constant as an enum, not a C variable.
Line 4014 sits at module scope, so Cython exports it as a pxd-defined C variable. An extension built against 13.5 bindings can fail at import when the installed cuda-bindings is 13.4.x, which the cuda-core floor permits. The file already uses anonymous cdef enum constants for values like CUDA_VERSION and CUDA_EGL_INFINITE_TIMEOUT, which do not create an exported variable.
The reason will be displayed to describe this comment to others. Learn more.
Duplicate comments: 1
cuda_bindings_13/cuda/bindings/cydriver.pxd: 1
1847-1847: Major · Potential issue
critical: Emit CUDA_FABRIC_COPY_BATCH_OP_st before this alias. The target struct is declared only at Lines 3244-3250, so Cython resolves this alias before the target type exists and the generated .pxd can fail to compile. Fix the generator ordering for the struct, aliases, and consumers. Do not hand-edit this generated file.
Reviewing files that changed from the base of the PR and between e04118b and f2547df.
Files selected for processing: 2
cuda_bindings_13/cuda/bindings/cydriver.pxd
cuda_bindings_13/cuda/bindings/cyruntime.pxd
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Duplicate comments: 1
cuda_bindings_13/cuda/bindings/cydriver.pxd: 1
1847-1847: Major · Functional Correctness
important: Emit both CUDA_FABRIC_COPY_BATCH_OP_v1 and CUDA_FABRIC_COPY_BATCH_OP after CUDA_FABRIC_COPY_BATCH_OP_st.
CUDA_FABRIC_COPY_BATCH_OP_st is declared at Line 3245, after these aliases. Cython can reject the forward aliases and stop the generated bindings from compiling. Update the generator and regenerate cydriver.pxd; do not hand-edit this generated file. This repeats the unresolved declaration-order issue from the previous review.
Reviewing files that changed from the base of the PR and between f2547df and 91aa3df.
Files selected for processing: 1
cuda_bindings_13/cuda/bindings/cydriver.pxd
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Re-confirming:
codex:
No new release-blocking red flags in MR 615 and PR 3368 at 68e7dcf. Fable’s independent review agrees.
The main fix checks out:
The actual CI wheel exports both locality sentinels and the EGL timeout as 4294967295.
The unwanted CAPI variables are gone; all six import-cycle checks pass.
Completed Linux, ARM, and Windows jobs pass the sentinel and cycle tests.
The added NVML documentation names match the exports.
One nonblocking hardening note: the EGL constant declaration can collide with the SDK macro. However, that header combination already failed before these changes, so we haven’t established a newly broken use case.
I’m comfortable proceeding with the agreed plan once the remaining required CI finishes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI/CDCI/CD infrastructurecuda.bindingsEverything related to the cuda.bindings modulecuda.coreEverything related to the cuda.core modulecuda.pathfinderEverything related to the cuda.pathfinder module
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
cuda.bindings
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
rwgk
left a comment
Prepare
cuda-bindingsandcuda-python13.5.0 for CUDA Toolkit 13.5, including refreshed bindings and API reference documentation.cupti64_13.dll.Follow-up nightly CI and test-dependency refinements are tracked in #3375.