GH-4512: fix(metrics): pilot_queue_depth frozen on headless daemons — refresh only wired to the TUI loop

[alekspetrov]

Summary

Automated PR created by Pilot for task GH-4512.

Closes #4512

Changes

GitHub Issue GH-4512: fix(metrics): pilot_queue_depth frozen on headless daemons — refresh only wired to the TUI loop

Problem

pilot_queue_depth and pilot_failed_queue_depth are only refreshed by the interactive dashboard's 2-second loop — autopilot.RefreshQueueDepth(store, metrics) has exactly one call site, cmd/pilot/main.go:3094, inside the TUI refresh path (its own doc comment notes it was added there because the gauge previously "read a constant 0").

Consequence: any daemon running headless (pilot start --telegram --github without --dashboard) exports a frozen queue-depth gauge — whatever value was last set, forever. Alerting or dashboards built on it silently lie.

This matters now: hosted S2 tenant instances (TASK-405 / saas-roadmap) run headless by design, and fleet observability (S4 C15 "ops Prometheus w/ tenant labels + alarms") will build on exactly these gauges.

Fix

Move queue-depth refresh into the daemon core, independent of the dashboard:

• Add a ticker (30s is fine; it's one cheap COUNT query per tick via store.CountQueuedTasks) in the main daemon lifecycle that calls autopilot.RefreshQueueDepth(store, metrics) whenever metrics are being served, dashboard or not.
• Keep the TUI's 2s refresh as-is (faster UX) — the two writers are idempotent sets, no conflict.
• Wrap the goroutine in safeGo per the panic-recovery convention.

Acceptance criteria

• AC1: with a daemon started WITHOUT --dashboard, pilot_queue_depth on /metrics tracks queued-task count changes (integration-style test or a unit test on the ticker wiring proving RefreshQueueDepth is called from the non-dashboard path).
• AC2: dashboard mode behavior unchanged (both refreshers coexist).
• AC3: ticker stops cleanly on daemon shutdown (no goroutine leak; respects context cancellation).

Refs

• cmd/pilot/main.go:3094 (sole current call site), internal/autopilot/metrics_queue_depth.go:9-24
• Context: gauge divergence found during 2026-07-23 Grom dashboard verification; TUI "queue depth 9" vs gauge "1" is a separate, expected definitional split (in-memory Monitor vs store count) — NOT in scope here.

Planned Steps (execute all in sequence)

• Step 1: fix(daemon): refresh pilot_queue_depth from headless daemon lifecycle — Add a queue-depth refresh ticker to the main daemon lifecycle in cmd/pilot/main.go so pilot_queue_depth and pilot_failed_queue_depth update on headless (--telegram --github without --dashboard) runs. Launch a safeGo goroutine in the daemon startup path (where metrics are wired) that runs a time.NewTicker(30 * time.Second) loop calling autopilot.RefreshQueueDepth(store, metrics) on each tick, plus one initial call before entering the loop so the gauge is correct immediately at boot. Gate the goroutine on the same condition that starts the /metrics server so it only runs when metrics are actually being exported. Wire context cancellation: select on ctx.Done() vs ticker.C, stop the ticker in a defer, and return cleanly so daemon shutdown does not leak the goroutine (AC3). Leave the existing TUI refresh at cmd/pilot/main.go:3094 untouched — both writers are idempotent sets, so dashboard mode remains unchanged (AC2). Add a unit test in cmd/pilot/ that constructs the daemon path (or a factored-out helper for the ticker wiring) and asserts RefreshQueueDepth is invoked without the dashboard being started — e.g., seed the store with N queued tasks, run the ticker once via injected clock or a short real-tick, and assert the pilot_queue_depth gauge reads N (AC1); also assert the goroutine exits when the context is cancelled (AC3). Follow the existing safeGo panic-recovery convention already used elsewhere in main.go. All work is confined to cmd/pilot/main.go (and its test file) — do not split into multiple subtasks per the Avoid Single-Package Splits rule, as this would only create merge conflicts on the same file.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 64.70588% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
cmd/pilot/main.go	64.70%	4 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!