fix(skill): native source = any mounted tool; empty grouped result is not absence - #32
Conversation
… result is not absence Roxane's v0.3.3 combined-arm rerun: three of six answers reported that native evidence was unavailable while a Kubernetes MCP was mounted and preflighted, one saying verbatim "I did not have a native kubectl source here". The guidance named `kubectl` and CLIs, so with no shell the agent concluded the layer was unverifiable. Name the tool surfaces instead, and require looking at the mounted tools before declaring a layer unverifiable. Same rerun: an APM query grouped on five dimensions returned zero buckets and was reported as "0 spans in the last hour"; the same window grouped by env alone returned 3,374. Generalise the 0-rows rule beyond Cypher to any source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be3dfabcd7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| grouped query answers "no rows", never "no traffic". Before reporting absence: drop the | ||
| grouping to the single dimension you actually need (`env`), widen the window, and re-run. | ||
| Report "no data matched this query" with the query shown, and only call it absence when | ||
| the simplest form of it is also empty. |
There was a problem hiding this comment.
When the remaining dimension itself is absent—for example, spans without an env tag—grouping by that single dimension still returns zero buckets even though traffic exists, so this instruction can reproduce the false-zero conclusion it is intended to prevent. Widening the window does not fix missing dimensions and also changes the bounds of a time-specific question; require an ungrouped total over the original requested window before reporting no traffic.
Useful? React with 👍 / 👎.
| an incident is about. The graph is the evidence for topology, relationships and history; | ||
| the native source is the evidence for "right now". Cite both, and say which one each | ||
| statement rests on. When no native source is mounted, keep the claim time-stamped | ||
| **A native source is any mounted tool that reads the live system, not just a shell.** Most |
There was a problem hiding this comment.
In a session where Anyshift is the only mounted read tool, this definition also describes the Anyshift MCP itself: it is mounted and reads a live system. An agent can therefore treat the graph as the required native source and cite the same stale graph evidence twice for a current-state claim, despite the later graph/native distinction. Define the native source as an independent tool that directly queries the underlying layer, excluding the Anyshift event graph.
Useful? React with 👍 / 👎.
2 participants
From Roxane's v0.3.3 combined-arm rerun (six runs, five native MCPs + Anyshift). The endpoint fix and PagerDuty incident selection are confirmed corrected there; two issues remain, and the first is our wording.
1. "I did not have a native kubectl source here." Three of six answers reported native evidence unavailable while the Kubernetes MCP was mounted and had passed an actual-read preflight (
q1-augmented-r2made all 29 calls against Anyshift, then said that). v0.3.3 names the native sources as "kubectl, a cloud CLI, the APM, PagerDuty" — shell framing. These sessions have no shell: Kubernetes arrives as an MCP server (resources_get,resources_list, …), so an agent reading "kubectl" correctly concludes it has none, and skips verification. Now the guidance names tool surfaces, says most sessions have no shell at all, and requires reading the mounted tool list before calling a layer unverifiable.2. Empty grouped result read as zero traffic.
q2-augmented-r2reported production had "0 spans in the last hour" from a query grouped simultaneously onenv,@kube_namespace,@kube_deployment,@kubernetes.namespace,@kubernetes.deployment.name— zero buckets, not a zero total. The same window grouped byenvalone returned 3,374 spans. The skill's "0 rows is NOT evidence of absence" rule only covered Cypher; it now covers any source, with the concrete instruction to reduce the grouping and re-run before reporting absence.Version bumped to 0.3.4 with the production-verification line and evidence row.
🤖 Generated with Claude Code