You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: install cuda-gdb in the default fetch_ctk toolkit - #3371
Adds cuda_gdb to the default cuda-components list of the fetch_ctk action, so Linux jobs that install the mini CTK also get cuda-gdb and cuda-gdbserver on PATH (about 60–100 MB per archive depending on the CUDA version).
cuda_gdb is published for linux-x86_64 and linux-sbsa in every CUDA version in the CI matrix (12.6.3 through 13.4.2). There is no Windows build; fetch_ctk already skips components a platform does not publish (as it does for libcudla on x86_64), so Windows jobs are unchanged.
Tests enabled
cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines currently always skips in mini-CTK jobs ("cuda-gdb is not on PATH"). With this change it runs: it debugs a kernel compiled with debug=True and checks that cuda-gdb shows the NVRTC source lines. It reports xfail rather than failing on runners where ptrace is not permitted.
cuda_pathfinder/tests/test_find_nvidia_binaries.py::test_find_binary_utilities[cuda-gdb] and [cuda-gdbserver] now locate real binaries instead of returning None. These pass either way, so this only adds coverage of the found path.
Only matrix entries that use the mini CTK (LOCAL_CTK == '1') are affected; jobs that take CUDA from wheels do not get cuda-gdb.
Side effects
Build, docs, coverage, and sdist jobs also use the default list, so they download cuda_gdb on a cache miss even though they do not use it.
The component list is part of the fetch_ctk cache key, so every Linux job misses the CTK cache once after this lands.
cuda_core's test_cuda_gdb_shows_nvrtc_debug_source_lines skips when
cuda-gdb is not on PATH, which is always the case with the mini CTK.
Adding cuda_gdb (about 100 MB, Linux only) to the default components lets
the test run in every Linux job that uses fetch_ctk's default list.
fetch_ctk already skips components a platform does not publish, so
Windows jobs are unchanged.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
suggestion: Keep cuda_gdb out of the global default and add it only to the Linux mini-CTK invocation that needs the debugger. Callers that omit cuda-components inherit this list, so unrelated jobs download and cache the debugger archive on cold misses. This increases CI transfer and cache use for jobs that do not run the debugger test. (raw.githubusercontent.com)
Reviewing files that changed from the base of the PR and between deb3d04 and 724bc86.
Files selected for processing: 1
.github/actions/fetch_ctk/action.yml
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
In numba-cuda-mlir, we overlay fetch_ctk call to additionally fetch cuda-gdb in the one pipeline that needs it. Let's do the same here (if it is needed at all).
Platform filtering and cache invalidation handle the added component correctly.
All 13 toolkit-helper tests pass locally; full CI is green.
The debugger source-line test passes in both x86_64 and aarch64 CI.
The remaining tradeoff is extra download and cache usage for Linux jobs that do not need the debugger, already acknowledged in the PR description.
CUDA-GDB history and download sizes
CUDA-GDB is a long-established tool. NVIDIA’s CUDA 2.1 release in January 2009 already included GDB-based CUDA debugging. It extends GDB to debug CPU and GPU code together, including breakpoints, stepping, variables, and GPU state. cuda_gdb is the redistribution component’s name; cuda-gdb is the executable. Release history, tool overview.
For the CUDA versions relevant to this PR, the compressed debugger downloads are:
These are decimal MB for this component alone. There is no native Windows package in those manifests, so this PR adds no Windows download. CUDA-GDB does support WSL2 using Linux tooling. Supported platforms.
Compiles a CUDA source string through cuda.core.Program and NVRTC with debug=True.
Loads and launches the resulting kernel, with the debugger configured to stop at kernel launch.
Asks the debugger to display source and checks for the generated .cu filename and a known source line.
The practical benefit is verifying that a user can see their runtime-compiled CUDA source in a real debugger. That requires the generated source file, compiler debug information, source paths, and GPU debugger to work together. Existing file-management tests cover parts of that chain; this exercises the complete interaction. Compilation and launch helper.
There is also modest additional coverage of finding installed cuda-gdb and cuda-gdbserver binaries. Those lookup tests still accept None, and nothing here exercises remote debugging through cuda-gdbserver. Lookup test.
In numba-cuda-mlir, we overlay fetch_ctk call to additionally fetch cuda-gdb in the one pipeline that needs it. Let's do the same here (if it is needed at all).
Q: Who/where is using this? Our "mini" CTK is not mini anymore because cuda-gdb pulls in its own Python...
The test cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines is currently never executed. Are you suggesting that we include cuda_gdb in the fetch for test workflows only? Or live with the gap?
Q: Who/where is using this? Our "mini" CTK is not mini anymore because cuda-gdb pulls in its own Python...
The test cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines is currently never executed. Are you suggesting that we include cuda_gdb in the fetch for test workflows only? Or live with the gap?
I checked CI log of this job, The test is executed there and passes:
What is the "mini CTK" used for? My agent says it tests the scenario of downloading and installing the CUDA Toolkit. If so, wouldn't "local CTK" be a more fitting name? "mini CTK" sounds like a subset, as if it runs only some of the tests rather than all of them.
What is the "mini CTK" used for? My agent says it tests the scenario of downloading and installing the CUDA Toolkit. If so, wouldn't "local CTK" be a more fitting name? "mini CTK" sounds like a subset, as if it runs only some of the tests rather than all of them.
Both names are already in use in the workflow code, and they're basically equivalent AFAIK.
"mini CTK" sounds like a subset
Correct. The way I think about it, roughly: it's a subset of what external users usually install into /usr/local/cuda
(In the CI, I think the directory is actually elsewhere.)
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI/CDCI/CD infrastructuretestImprovements or additions to tests
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
leofang
left a comment
rwgk
left a comment
Summary
Adds
cuda_gdbto the defaultcuda-componentslist of thefetch_ctkaction, so Linux jobs that install the mini CTK also getcuda-gdbandcuda-gdbserveronPATH(about 60–100 MB per archive depending on the CUDA version).cuda_gdbis published for linux-x86_64 and linux-sbsa in every CUDA version in the CI matrix (12.6.3 through 13.4.2). There is no Windows build;fetch_ctkalready skips components a platform does not publish (as it does forlibcudlaon x86_64), so Windows jobs are unchanged.Tests enabled
cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_linescurrently always skips in mini-CTK jobs ("cuda-gdb is not on PATH"). With this change it runs: it debugs a kernel compiled withdebug=Trueand checks thatcuda-gdbshows the NVRTC source lines. It reports xfail rather than failing on runners where ptrace is not permitted.cuda_pathfinder/tests/test_find_nvidia_binaries.py::test_find_binary_utilities[cuda-gdb]and[cuda-gdbserver]now locate real binaries instead of returningNone. These pass either way, so this only adds coverage of the found path.Only matrix entries that use the mini CTK (
LOCAL_CTK == '1') are affected; jobs that take CUDA from wheels do not getcuda-gdb.Side effects
cuda_gdbon a cache miss even though they do not use it.fetch_ctkcache key, so every Linux job misses the CTK cache once after this lands.