feat(csrc): AMD ROCm logical-slicing shim, and a verification suite that can fail

[thxCode]

Delivers the AMD ROCm logical-slicing shim — the library and its verification, not the
device-manager wiring, which the spec names as a Non-Goal and a later change.

specs/2026-08-06-amd-gpu-slicing-shim.md carries the design, the measurements behind it, and the
Risks it does not close.

What lands

csrc/amd/rocm-slicing-shim/ — libvrocm.so, preloaded into a sliced container through
/etc/ld.so.preload, plus rocm-monitor (reads a slice's usage without touching the ledger's
writer) and rocm-cumask-check (fails a card whose CU mask the platform silently discarded).

• Enforces a per-card memory quota across every process in the container, through a MAP_SHARED
  ledger under an fcntl record lock. Check, allocate and charge happen under ONE lock hold, which
  is what closes the check-then-allocate race between processes.
• Reports the quota — not the card — through all three entry points a framework can ask with,
  hipGetDevicePropertiesR0600 among them. ROCm 6+ headers rewrite the call to that name, so a
  wrapper on the plain one interposes a symbol nothing calls; measured, only the R0600 variant is
  ever reached.
• One build serves every ROCm version: the library links no ROCm object and resolves at run time,
  its only DT_NEEDED is libc.so.6, and its glibc floor is GLIBC_2.4 so it loads into an
  Ubuntu 20.04 or RHEL 8 workload image.

The interception table was completed by subtraction, not by review. Every allocating name
libamdhip64 exports, minus the interposed set, measured. That found five open doors review had
missed — hipMalloc3D, hipMemCreate, hipMemAllocPitch, hipArrayCreate, hipArray3DCreate —
each of which took 512 MiB out of a 64 MiB quota before it was wrapped. The width of the
denominator is part of the claim, and the first manifest's was a guessed regex that four of the
five fell outside.

.claude/skills/gpustack-operator-xbuild-and-verify/ — the xbuild-amd-rocm arm, seven
amd-case-*.sh scripts, the AMD preflight rows and the reference pages.

Verification

Every case ran on the vendor's own hardware. 21 of 21 pass.

Backend	Host	Result
AMD	2 × RDNA3, ROCm 7.2	7 cases, 144 rows
THead	16-card PPU host	7 cases, 270 rows
NVIDIA	2-GPU host	3 cases
Ascend	8 × 910B, aarch64	4 cases

The AMD suite also ran against a CDNA host for cases 4 and 5 — the two whose arithmetic differs by
NUM_XCC — which is a partial result and says so in the spec.

The harness had to be fixed before any of that meant anything

AMD-CASE 4 printed the literal FAILS=0 into every passing row's detail column, and the verdict
grepped the output for that token. The case built to catch a silently discarded CU mask could not
itself fail. Measured: one deliberately broken assertion printed a red row, printed FAILS=1, and
still exited 0 saying PASS.

All 21 cases now decide through one helper in lib.sh that reads the count as a whole line, and
preflight.sh fails if a case decides for itself or if that helper's own arithmetic is wrong. Both
guards were proven able to fail before they were trusted; so were the three unit tests added for
the ledger defects below.

Known and recorded, not closed

• hipGraphAddMemAllocNode is measured open (512 MiB under a 64 MiB quota). A graph node allocates
  at launch rather than at capture, so the accounting is no longer one call in and one call out.
• A CU mask carries no memory-bandwidth isolation — half the CUs already reach 97.7 % of the card's
  bandwidth. .sliced is a compute ceiling, never a QoS.
• Direct hsa_* or /dev/kfd callers never pass through this library. That is a boundary of the
  approach, not a defect, and this is not a security boundary either way.
• The card's in-process lock is a bare spin held across the driver call; a pid in the region names a
  process only within the namespace that wrote it; the region file's 0600 mode is decided by
  whichever container creates it.

[gemini-code-assist]

Code Review

This pull request adds support for AMD ROCm logical-slicing verification, introducing the libvrocm.so shim, monitor and mask check tools, along with seven comprehensive test cases (amd-case-1.sh through amd-case-7.sh). It also refactors existing Ascend and THead test cases to use a centralized verdict helper (xb_verdict) to ensure robust failure detection. The review feedback highlights several improvement opportunities, including handling potential unbound variables under set -u in amd-case-5.sh, skipping leading whitespace when parsing negative quotas in vrocm_quota.c, adding defensive NULL pointer checks in hip_mem.c to prevent segmentation faults, optimizing the bare spin lock in vrocm_ledger.c with a CPU pause hint, and setting LC_ALL=C for deterministic sorting in amd-case-1.sh.

[Copilot]

Pull request overview

This PR introduces the AMD ROCm logical-slicing shim (csrc/amd/rocm-slicing-shim) to enforce per-card VRAM quotas via HIP symbol interposition, along with companion tools/tests and extensions to the gpustack-operator-xbuild-and-verify skill to build and validate the shim across environments (including PTY-only SSH targets) and to harden the case verdict logic.

Changes:

• Add libvrocm.so HIP interposers (memory allocation families + reported-capacity queries) backed by a shared, lock-coordinated usage ledger format.
• Add shim-adjacent tooling and test artifacts (rocm-monitor, hip_props_probe, ledger_lifecycle) to validate ledger behavior and query/binding surfaces.
• Extend the verification harness/scripts with robust verdict parsing (xb_fails/xb_verdict), AMD ROCm preflight checks, and AMD xbuild support (including a PTY-only target mode).

Reviewed changes

Copilot reviewed 53 out of 54 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
csrc/amd/rocm-slicing-shim/tools/rocm_monitor.c	Adds a lock-free reader for the shared usage ledger, intended for operators/scrapers.
csrc/amd/rocm-slicing-shim/testing/ledger_lifecycle.c	Adds a two-process lifecycle test driver for stranded-charge reclaim behavior.
csrc/amd/rocm-slicing-shim/testing/hip_props_probe.c	Adds a probe to report which HIP symbols bind and what capacity is reported via each path.
csrc/amd/rocm-slicing-shim/README.md	Documents design goals, boundaries, build modes, and operational usage for the shim + tools.
csrc/amd/rocm-slicing-shim/hip/hip_vmm.c	Interposes VMM allocation/free entry points (hipMemCreate/hipMemRelease) for quota enforcement.
csrc/amd/rocm-slicing-shim/hip/hip_table.h	Declares shared per-entry counters/registration macros for verification and diagnostics.
csrc/amd/rocm-slicing-shim/hip/hip_table.c	Implements entry tracking and constructor/destructor hooks for quota validation + counter dumps.
csrc/amd/rocm-slicing-shim/hip/hip_resolve.h	Defines symbol resolution interface/macros used by all HIP wrappers.
csrc/amd/rocm-slicing-shim/hip/hip_resolve.c	Implements resolver fallback strategy (RTLD_NEXT then RTLD_NOLOAD by SONAME) and caller attribution.
csrc/amd/rocm-slicing-shim/hip/hip_query.c	Interposes reported-capacity entry points to report quota instead of physical VRAM.
csrc/amd/rocm-slicing-shim/hip/hip_pool.c	Interposes stream-ordered/pool allocation paths to close non-hipMalloc* quota bypasses.
csrc/amd/rocm-slicing-shim/device/vrocm_hwid.h	Provides shared device-side HW-ID decoding for occupancy-based CU/WGP mask verification.
csrc/amd/rocm-slicing-shim/common/vrocm.h	Introduces common vocabulary and fixed bounds used across the shim.
csrc/amd/rocm-slicing-shim/common/vrocm_quota.h	Declares quota/env parsing and validation APIs for “fail closed” enforcement.
csrc/amd/rocm-slicing-shim/common/vrocm_quota.c	Implements quota/env parsing + validation and latches usability at load time.
csrc/amd/rocm-slicing-shim/common/vrocm_log.h	Defines the shim’s single log channel and level contract used by cases.
csrc/amd/rocm-slicing-shim/common/vrocm_log.c	Implements latched log-level parsing with safe fallback behavior.
csrc/amd/rocm-slicing-shim/common/vrocm_ledger.h	Defines the shared MAP_SHARED ledger/lock layout and admission/release contract.
.claude/skills/gpustack-operator-xbuild-and-verify/scripts/preflight.sh	Adds AMD ROCm target preflight checks and enforces consistent verdict parsing across cases.
.claude/skills/gpustack-operator-xbuild-and-verify/scripts/lib.sh	Adds PTY-only target mode and centralizes verdict parsing helpers used by all cases.
.claude/skills/gpustack-operator-xbuild-and-verify/scripts/build.sh	Adds xbuild-amd-rocm support with containerized and in-place ROCm build paths.
.claude/skills/gpustack-operator-xbuild-and-verify/references/troubleshooting.md	Documents AMD ROCm failure modes and debugging workflow for the shim/tools.
.claude/skills/gpustack-operator-xbuild-and-verify/references/amd-cumask-conformance.md	Adds measured CU-mask conformance tables and regeneration instructions.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/thead-case-1.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/thead-case-2.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/thead-case-3.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/thead-case-4.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/thead-case-5.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/thead-case-6.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/thead-case-7.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/nvidia-case-1.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/nvidia-case-2.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/nvidia-case-3.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/ascend-case-1.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/ascend-case-2.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/ascend-case-3.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/ascend-case-4.sh	Switches case verdict to centralized xb_verdict/xb_fails.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/amd-case-2.sh	Adds AMD injection + reported-capacity surface verification with control interposers.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/amd-case-3.sh	Adds AMD allocation-path completeness case across allocation families/entry points.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/amd-case-4.sh	Adds AMD CU-mask conformance validation (positive + fail-open negative rows).
.claude/skills/gpustack-operator-xbuild-and-verify/cases/amd-case-5.sh	Adds compute-quota semantics validation using occupancy + throughput under barriers.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/amd-case-6.sh	Adds common unit suite relay + multi-process quota + multi-card per-card keying checks.
.claude/skills/gpustack-operator-xbuild-and-verify/cases/amd-case-7.sh	Adds stranded-charge reclaim case + cross-version reach verification (same shim, two ROCm versions).

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

csrc/amd/rocm-slicing-shim/hip/hip_resolve.c:7

• hip_resolve.c calls fprintf(stderr, ...) but does not include <stdio.h>, which relies on an implicit declaration and can break builds under stricter compiler settings (or produce incorrect prototypes).

[Copilot]

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

csrc/amd/rocm-slicing-shim/hip/hip_table.c:38

• vrocm_entry_hit() updates entry->traced with a plain increment while wrappers can be called concurrently from many threads. This introduces a C data race (undefined behavior). Use an atomic fetch-add for traced and make the trace-limit decision based on the returned prior value.

[Copilot]

Pull request overview

Copilot reviewed 53 out of 54 changed files in this pull request and generated no new comments.

Suppressed comments (2)

csrc/amd/rocm-slicing-shim/tools/rocm_monitor.c:162

• mmap(sizeof(*region)) assumes the ledger file is at least sizeof(struct vrocm_region). If the file is truncated but still large enough to pass the header read, later reads from region->devices[...] can SIGBUS. Add an st_size check for the full region size before mapping.
  csrc/amd/rocm-slicing-shim/hip/hip_table.c:38
• entry->traced is incremented and read concurrently from multiple threads without synchronization. In C this is a data race (undefined behavior) and can also cause the TRACE_LIMIT gate to behave unpredictably. Use an atomic fetch/add (relaxed is sufficient) for the limit check and increment.

[Copilot]

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

.claude/skills/gpustack-operator-xbuild-and-verify/references/troubleshooting.md:275

• This troubleshooting note says VROCM_LEDGER_PATH “has no default” and that omitting it makes the shim refuse everything. In the current implementation, vrocm_quota_ledger_path() falls back to VROCM_LEDGER_DEFAULT_PATH ("/dev/shm/vrocm-ledger"), so the statement is misleading (the real risk is that the default can be unsafe when /dev/shm is shared).

  `LIBVROCM_LOG_LEVEL=2`, because the load marker and the counter dump sit above the default of 1,
  which carries denials only; and `VROCM_LEDGER_PATH`, which has **no default** — without it the
  constructor reports `VROCM_LEDGER_PATH is unset; nothing can be accounted` once and then refuses
  everything, which looks nothing like "quietly did nothing" but is easy to miss on a busy stream.

[thxCode]

Copilot's suppressed comments — outcomes

These arrive collapsed in the review body with no thread to reply on, so the disposition is
recorded here. Five across the last three reviews; two were already handled, one was right and
uncovered a defect I had introduced, two restate threads answered above.

#	Where	Verdict
1	references/troubleshooting.md:275 — says VROCM_LEDGER_PATH has no default	Right, and mine to fix — fixed in 9859125d
2	hip_resolve.c:7 — fprintf without <stdio.h>	Claim wrong, hygiene right — see the thread above
3	hip_table.c:38 — traced data race (×2)	Already fixed in d8bbee13
4	rocm_monitor.c:162 — mmap SIGBUS	Not reachable; the guard was always full-region — see the thread above

On #1, and why it matters more than the line it names. Two commits earlier I gave
VROCM_LEDGER_PATH a default and did not sweep for prose that assumed it had none. Copilot found
one instance. Sweeping the same root cause found two more it missed:

• csrc/amd/rocm-slicing-shim/README.md — the rocm-monitor section still said "There is no
  default path", one section away from the new table that documents the default.
• csrc/amd/rocm-slicing-shim/common/vrocm_ledger.c — a now-unreachable branch handling a NULL
  path, still carrying the message the troubleshooting page quoted. Removed, and
  vrocm_quota_ledger_path's header contract now states it never returns NULL, so the next reader
  knows what they would be breaking.

All three are folded into 9859125d, the commit that made them stale — a doc goes wrong at the
commit that changes the behaviour, not at the commit that wrote the doc.

Verified after: seven AMD cases 149 PASS / 0 FAIL, 44 unit cases, build.sh unit mutants silent,
build.sh check clean (the linkage assertions that pin DT_NEEDED to libc.so.6 and the
GLIBC_2.4 floor).

[Copilot]

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

.claude/skills/gpustack-operator-xbuild-and-verify/scripts/preflight.sh:242

• The preflight guard for “every case decides through xb_verdict” is currently grep -q 'xb_verdict ', which can be satisfied by a comment mentioning xb_verdict (or other non-invocation text). That would let a case script omit the real verdict call while still passing this safety check.

for c in "${HERE}/../cases"/*.sh; do
  [ -f "${c}" ] || continue
  seen=$((seen + 1))
  grep -q 'xb_verdict ' "${c}" || rogue="${rogue} $(basename "${c}")"
done

[thxCode]

Suppressed comment — outcome

One this round, and it is the best finding on the PR so far, so it gets its own note rather than a
row in a table.

scripts/preflight.sh:242 — the xb_verdict guard could be satisfied by a comment. Fixed in
0d9da296.

The guard exists because AMD-CASE 4 printed the literal FAILS=0 into a detail column and the
verdict grepped the whole output for that token, so the case could not fail. The guard I wrote to
prevent a repeat was grep -q 'xb_verdict ' — which a case could satisfy with

# this case decides through xb_verdict at the end

while hand-rolling its verdict underneath. A guard against unfailable assertions that was itself
satisfiable without the thing it asserts. Exactly the shape it was written to catch, one level up.

Now anchored to a call rather than a mention:

grep -qE '^[[:space:]]*xb_verdict ' "${c}"

Checked both directions rather than only the one that matters to me: all 21 cases still match, and
a file containing only that comment is now rejected where the old pattern accepted it.

The lesson generalises past this line, and it is the third time on this PR: a check whose subject is
"some code exists" has to match the code, not the words. Both preflight rows were proven able to
fail when they were added; neither probe covered this way of passing.

[Copilot]

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.

Suppressed comments (5)

csrc/amd/rocm-slicing-shim/hip/hip_query.c:126

• hipDeviceTotalMem unconditionally replaces the runtime-reported device total with the configured quota. If the quota is misconfigured above physical VRAM, this will report an impossible capacity to callers. Clamp the reported value to min(runtime_total, quota).
  csrc/amd/rocm-slicing-shim/hip/hip_query.c:166
• hipGetDevicePropertiesR0000 sets totalGlobalMem to the configured quota even when the runtime reports a smaller physical total. Clamp totalGlobalMem to min(runtime_total, quota) to avoid reporting impossible capacity under misconfiguration.
  csrc/amd/rocm-slicing-shim/hip/hip_query.c:191
• hipGetDeviceProperties (plain) sets totalGlobalMem to the configured quota even when the runtime reports a smaller physical total. Clamp totalGlobalMem to min(runtime_total, quota) to avoid advertising capacity the device cannot provide.
  csrc/amd/rocm-slicing-shim/hip/hip_query.c:146
• hipGetDevicePropertiesR0600 sets totalGlobalMem to the configured quota even when the runtime reports a smaller physical total. This can mislead frameworks into sizing allocations larger than the device can support. Clamp totalGlobalMem to min(runtime_total, quota).
  csrc/amd/rocm-slicing-shim/hip/hip_query.c:107
• hipMemGetInfo overrides total_bytes with the configured quota even when that quota exceeds the runtime-reported physical total. That can cause callers to believe more VRAM is available than the device can supply. Clamp the effective quota to the physical total before rewriting total/free.

This issue also appears in the following locations of the same file:

• line 123
• line 143
• line 163
• line 188