feat(worker): expose current CPU/RAM/DISK/GPU utilization via Instance metrics subresource

[thxCode]

What type of PR is this?

/kind enhancement
/kind api-change
/area worker
/area devicemanager
/area testing

What this PR does / why we need it:

Adds a read-only instances/<name>/metrics subresource to the aggregated API (worker.gpustack.ai/v1) so an Instance's current CPU/memory/disk/GPU utilization is visible without entering the instance and without Prometheus or metrics-server:

• CPU / memory / disk — read in real time from the node kubelet's stats summary through the API-server node proxy; metrics.k8s.io is a CPU/memory-only fallback when the kubelet read fails (entries measured before the pod existed are rejected).
• Accelerator metrics — the device manager samples every 15s into an atomic single latest snapshot (datax.Snapshot) served at GET /monitor/snapshot; the subresource merges only the cards recorded in the pod's allocation annotation (an Instance never sees an accelerator it did not allocate).
• Scoping/RBAC — strictly instance-scoped (name + app.kubernetes.io/part-of UID label + kubelet pod UID match, so a recreated instance never reads the previous one); callers need get on instances/metrics specifically.
• No history — product decision: only current gauges; nothing is retained anywhere (no ring buffers, no CR status).

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

• The device-manager snapshot endpoint is unauthenticated and the worker dials it with TLS verification skipped (self-signed certs): explicitly accepted for v1; mTLS + NetworkPolicy hardening is recorded as backlog in the spec.
• Spec: specs/2026-08-07-instance-utilization-metrics.md (Shipped, includes the design record and the dropped history-based first iteration). Reference doc: docs/reference/instance-metrics.md.
• Tests: unit (-race clean), plus new e2e case-37 — all checks PASS on a k3s test cluster (current sample fields, figures track the backing pod under load, unprivileged caller denied).

Does this PR introduce a user-facing change?

Add an Instance `metrics` subresource (`worker.gpustack.ai/v1`) serving the current CPU/memory/disk/GPU utilization per Instance with no Prometheus or metrics-server required: `kubectl get --raw /apis/worker.gpustack.ai/v1/namespaces/<ns>/instances/<name>/metrics`.

[gemini-code-assist]

Code Review

This pull request implements the instances/<name>/metrics subresource to serve real-time CPU, memory, disk, and GPU utilization metrics for Instances. The metrics are retrieved from the node kubelet stats summary via the API-server node proxy, with a fallback to metrics.k8s.io for CPU/memory, and merged with accelerator metrics from the DeviceManager's latest snapshot. The DeviceManager has been refactored to store only the latest monitor snapshot using a new atomic Snapshot utility, removing the previous ring-buffer history. Feedback on the E2E test script suggests a more robust way to terminate background load processes by using kill $(jobs -p) instead of hardcoded job numbers.

[Copilot]

Pull request overview

Adds an Instance metrics subresource to the worker aggregated API (worker.gpustack.ai/v1) to expose a single “current gauges” utilization sample (CPU/memory/disk from kubelet stats summary via apiserver node proxy, plus best-effort GPU metrics merged from the DeviceManager’s latest snapshot), along with supporting DeviceManager snapshot plumbing, API types/codegen, docs, and e2e coverage.

Changes:

• Implement GET instances/<name>/metrics subresource handler + unit tests in the worker aggregated API.
• Replace DeviceManager monitor history with a “latest snapshot” readout endpoint (/monitor/snapshot) backed by an atomic snapshot holder.
• Add API types + regenerated OpenAPI/proto artifacts, and document the contract + add an e2e case.

Reviewed changes

Copilot reviewed 18 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
specs/2026-08-07-instance-utilization-metrics.md	Design/spec for the “current gauges” Instance utilization endpoint and data sources.
pkg/worker/extensionapis/worker/instance.metrics.go	Implements the Instance metrics subresource (kubelet summary + metrics.k8s.io fallback + DM snapshot merge).
pkg/worker/extensionapis/worker/instance.metrics_test.go	Unit tests for subresource scoping, kubelet/fallback behavior, and snapshot merge/filtering.
pkg/worker/extensionapis/worker/instance.go	Registers the new metrics subresource handler.
pkg/utils/datax/snapshot.go	Adds a generic atomic “latest snapshot” holder utility.
pkg/utils/datax/snapshot_test.go	Tests the snapshot holder’s basic and concurrent behavior.
pkg/devicemanager/snapshot.go	Adds the /monitor/snapshot HTTP handler and snapshot envelope type alias.
pkg/devicemanager/snapshot_test.go	Tests snapshot handler method restrictions and JSON envelope shape.
pkg/devicemanager/manager.go	Registers the snapshot endpoint on the device-manager webserver.
pkg/devicemanager/detector/snapshot_test.go	Ensures monitor ticks replace the stored snapshot over time.
pkg/devicemanager/detector/option.go	Drops history options; sets monitor period default to 15s and updates flag help/validation.
pkg/devicemanager/detector/detector.go	Stores latest monitor output into a snapshot envelope instead of a ring buffer.
pkg/devicemanager/detector/config.go	Removes monitor-history configuration from detector config.
docs/reference/instance-metrics.md	User-facing reference for fields, sources, scoping/RBAC, degradation, and limits.
docs/README.md	Adds the new reference doc to the docs index.
api/worker/zz_generated.openapi.go	Regenerates OpenAPI for new InstanceMetrics types.
api/worker/v1/zz_generated.register.go	Registers InstanceMetrics in the scheme.
api/worker/v1/zz_generated.model_name.go	Adds OpenAPI model names for new v1 types.
api/worker/v1/zz_generated.deepcopy.go	Adds deepcopy methods for new API types.
api/worker/v1/instance.metrics.go	Defines the public v1 API types for Instance metrics and accelerator metrics.
api/worker/v1/generated.protomessage.pb.go	Regenerates protomessage stubs for new types.
api/worker/v1/generated.proto	Regenerates protobuf schema with new message types.
api/worker/v1/generated.pb.go	Regenerates protobuf implementation for new messages.
.claude/skills/gpustack-operator-e2e/SKILL.md	Documents the new e2e case coverage mapping.
.claude/skills/gpustack-operator-e2e/cases/case-37.sh	Adds e2e case validating the new metrics subresource behavior and RBAC.

Files not reviewed (7)

• api/worker/v1/generated.pb.go: Generated file
• api/worker/v1/generated.proto: Generated file
• api/worker/v1/generated.protomessage.pb.go: Generated file
• api/worker/v1/zz_generated.deepcopy.go: Generated file
• api/worker/v1/zz_generated.model_name.go: Generated file
• api/worker/v1/zz_generated.register.go: Generated file
• api/worker/zz_generated.openapi.go: Generated file

Suppressed comments (5)

.claude/skills/gpustack-operator-e2e/cases/case-37.sh:58

• The test Instance manifest hard-codes namespace: default, ignoring the <NS> argument documented at the top of the script. This makes the case run against a different namespace than requested.

cat <<EOF | kubectl apply -f -
apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata: { name: ${INST}, namespace: default }
spec:

.claude/skills/gpustack-operator-e2e/cases/case-37.sh:69

• Namespace is hard-coded to default when waiting for readiness. If <NS> is not default, the script will wait on the wrong object and may report a false failure.

for _ in $(seq 1 40); do
  phase=$(kubectl -n default get instance "$INST" -o jsonpath='{.status.phase}' 2>/dev/null)
  [ "$phase" = "Ready" ] && break
  sleep 3

.claude/skills/gpustack-operator-e2e/cases/case-37.sh:73

• The raw metrics path is hard-coded to namespaces/default, so running the case with a non-default <NS> will query the wrong endpoint.

RAW="/apis/worker.gpustack.ai/v1/namespaces/default/instances/${INST}/metrics"

.claude/skills/gpustack-operator-e2e/cases/case-37.sh:108

• CPU burn uses kubectl -n default exec ..., ignoring <NS>. On non-default namespaces the load is never applied to the test Instance, which can cause the “figures track the backing pod” check to fail spuriously.

kubectl -n default exec "$INST" -c main -- sh -c 'for i in 1 2 3 4 5 6 7 8; do (while :; do :; done) & done; sleep 25; kill %1 %2 %3 %4 %5 %6 %7 %8 2>/dev/null' >/dev/null 2>&1

.claude/skills/gpustack-operator-e2e/cases/case-37.sh:123

• Authorization check uses -n default, ignoring <NS>. Subresource RBAC is namespace-scoped, so this can yield the wrong result when the test runs in a different namespace.

deny=$(kubectl auth can-i get instances.worker.gpustack.ai/metrics -n default \
  --as system:serviceaccount:default:default 2>/dev/null)

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[thxCode]

Pushed a revision round: a unit-contract change, ten review findings, and a triage of the bot feedback. History was rewritten in place (fixups folded into the originating commits), so the branch is still ten commits.

Unit contract: every memory and storage figure is now MiB, rounded up

memoryWorkingSetBytes / rootfsUsedBytes / ephemeralStorageUsedBytes became memoryWorkingSetMiB / rootfsUsedMiB / ephemeralStorageUsedMiB (protobuf field numbers unchanged). The sources disagree — the kubelet measures in bytes, the vendor device libraries in MiB — and one sample should not make a consumer mix units. Scaling MiB up to bytes was rejected: it fabricates precision the vendor libraries never had.

Byte figures round up, decided after measuring an idle instance on a Kubernetes cluster:

kubelet	truncated	rounded up
workingSetBytes 585,728	0	1
ephemeral-storage.usedBytes 20,480	0	1
rootfs.usedBytes 12,288	0	1

Truncation reported all three as 0, which reads as "broken" rather than "small". 0 now means the source measured no usage.

Review findings addressed

• One device manager snapshot per allocated manufacturer. The chart rolls a DaemonSet per manufacturer and passes --manufacturer, so each snapshot only carries its own cards — substituting another manufacturer's pod guaranteed an empty result after two HTTPS round trips. A multi-manufacturer allocation also used to drop every card but the first group's.
• Dropped the device manager retry. Both attempts shared the operation deadline and repeated a pod resolution that cannot change within it. Added logging under instance-metrics so an absent accelerator section is no longer indistinguishable from "no cards allocated".
• Named the actual failure in the 503. A cluster without metrics-server produced ... and the metrics API (<nil>), hiding the real reason.
• ServiceUnavailable instead of Conflict for a backing pod of a previous incarnation — transient backing state, like the other two cases.
• Clamped negative metrics.k8s.io quantities (-250m wrapped to 1.8e19) and treat an empty container list as unserved rather than a genuine zero.
• Fall back to metrics.k8s.io when the kubelet answers without knowing the pod, instead of an empty sample stamped with a measurement that never happened.
• Bounded the unverified snapshot readout with io.LimitReader; dropped an unused Client field.
• Tests added for the gaps the plan listed but did not cover: AC3.2 at the OnGet level, malformed allocation annotation, retry/timeout behavior, negative and empty parsePodMetricsUsage input, and "never substitute another manufacturer".

Bot feedback triage

Three findings were real and are fixed (replied and resolved inline): the PeriodSeconds truncation, the NewConflict argument, and the e2e kill loop.

Two are false positives and their threads are left open for a human call:

• datax/snapshot_test.go loop variable — correct since Go 1.22 (per-iteration 3-clause loop variables); test passes under -race.
• case-37 hardcoding default — applied first, then reverted with evidence. <NS> is the operator's own namespace throughout this suite; the Instance webhook rejects a reserved namespace, so the suggested change made the case fail with cannot create instance in reserved namespace. The five suppressed comments on the same file share this root cause and are covered by the same reply. The case header now states the convention.

Verification

Image built on a build host and deployed to a Kubernetes cluster with a real accelerator:

• assert-core.sh: all PASS, running binary revision == HEAD
• CASE 1 (scheduling chain): all PASS
• CASE 37: all PASS — fields present, cpuUsageNanoCores 0 -> 998319366 under load, unprivileged caller denied
• /monitor/snapshot: periodSeconds: 15, GET serves the sample, POST returns 405
• GPU merge on a real card: only the allocated device is returned; the second card on the same node is filtered out
• Round-up confirmed end to end against live kubelet figures (table above)

go build ./..., go test (incl. -race) and make lint are clean; every commit on the branch builds independently.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 27 changed files in this pull request and generated no new comments.

Files not reviewed (7)

• api/worker/v1/generated.pb.go: Generated file
• api/worker/v1/generated.proto: Generated file
• api/worker/v1/generated.protomessage.pb.go: Generated file
• api/worker/v1/zz_generated.deepcopy.go: Generated file
• api/worker/v1/zz_generated.model_name.go: Generated file
• api/worker/v1/zz_generated.register.go: Generated file
• api/worker/zz_generated.openapi.go: Generated file

Suppressed comments (3)

pkg/utils/mathx/division.go:12

• CeilDiv uses (a + b - 1) / b, which can overflow for large unsigned values and produce an incorrect (often much smaller) quotient. Since this helper is used for converting byte counters, it’s safer to implement ceil division via quotient+remainder (no overflow) and constrain the generic to unsigned integers (the current contract doesn’t define behavior for negatives anyway).

func CeilDiv[I typex.Integer](a, b I) I {
	if b == 0 {
		return 0
	}
	return (a + b - 1) / b

pkg/worker/extensionapis/worker/instance.metrics.go:508

• If the metrics.k8s.io response has a zero/absent timestamp, currentPodUsage will accept the CPU/memory values and stamp them with meta.Now(), which removes the ability to reject previous pod incarnations and can leak stale data across pod recreation. Treat a missing timestamp as an error (or as “unserved”) so the caller never serves metrics that can’t be scoped to the current pod lifetime.

	ts := &podMetrics.Timestamp
	if ts.IsZero() {
		ts = nil
	}
	return &cpu, &memory, ts, nil

pkg/worker/extensionapis/worker/instance.metrics.go:372

• When a device-manager snapshot is dropped as stale, the worker currently returns no accelerator section without logging why. This contradicts the documented contract that “the reason is logged” for missing accelerator data, and makes GPU absence hard to diagnose in production.

		maxAge = time.Duration(snapshot.PeriodSeconds) * time.Second * 3
	}
	if time.Since(snapshot.Timestamp) > maxAge {
		return nil
	}