test(skill): stop the eval demanding error text the user said they lack

[volen-silo]

Summary

The rocm-nothing-established-routes-upstream eval case failed on both behavioural legs of amd/skills#259 after an unrelated merge from main re-ran CI. Nothing in skills/rocm-doctor/ or the eval workflow changed between the green run and the red one. The failing item was:

Ask the user for the exact error text instead of inventing a diagnosis

The case prompt says the user cannot paste an exact error, and rocm diagnose probes the host without --symptom. So an agent that offers to install the CLI and probe anyway is doing the right thing. The judge read the item two ways on the same behaviour:

• green run: "acknowledging the user cannot paste exact error text and proposing host probing over invented diagnosis" → pass
• red run: "told the user no error message was needed rather than asking for the exact error text" → fail

This changes eval data only. SKILL.md is unchanged.

Changes

skills/rocm-doctor/evals/evals.json, case rocm-nothing-established-routes-upstream:

• Replaced the "ask for the exact error text" item with the guard it stood for: missing error text must not stop the diagnosis or invite a guess. Asking as well is fine; requiring it before probing the host is not. The exclusion is in the item, because the judge reads the item, not the case note.
• Folded the has_match expected item into the existing "no remediation without an established cause" unexpected item. The old item required routing upstream, which a grading host with no rocm CLI never reaches. The guard it protects, no fix when has_match is false even if matched is non-empty, is still graded.
• Updated the case note to match.

Verification

Local skillscope behavioral runs (v0.1.3, the version amd/skills CI uses) on this case, on a host with no rocm CLI reachable and a clean agent config, as on the CI runners:

Dataset	Skill	Runs	Case passed
Current wording	unchanged	5	1/5
This change	unchanged	10	10/10
This change	mutant: told to demand the exact error before doing anything	3	0/3, failing only the new item

The mutant run shows the new item still fails an agent that blocks on the error text.

• skillscope structural --skills-dir skills/rocm-doctor --skill-files skill-card.md --skill-sections Description,Owner,License: OK
• cargo test -p e2e-cucumber --test skill_reference: 3 passed

Not verified here: the local judge runs through a different model gateway than amd/skills CI, so verdicts can differ. This repo cannot run the routing or behavioural legs. The CI result will show on amd/skills#259 after re-import.

No Gherkin scenario: this changes eval data only and does not alter anything the CLI does.

Follow-ups (not in this PR)

• With no CLI on the grading host, the has_match guard is never really exercised. Recorded rocm diagnose fixtures would let the eval grade it, and routing upstream too.
• SKILL.md says four fixes are auto-applicable; reference.md and fix.rs say three, with fix-9-igpu-dgpu as NEEDS-ARG.
• logs_contain matches the whole raw transcript, so a command named in the loaded skill text may satisfy it without the agent running it.

[nowycondro]

🔴 Automated review · ad hoc · d70cf80

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (1 file, 3 additions / 4 deletions). The PR's own description references amd/skills#259 (the federation PR importing this skill into the shared catalog) as background for why the eval harness matters, but does not claim to close it and isn't part of resolving it — this is a local fix to ROCm/rocm-cli's own eval data, and the description's scope claim ("changes eval data only, SKILL.md is unchanged") matches the diff.

Blocking

None.

Non-blocking

None.

Decisions for the author

None.

Positive signals

• The commit backs its behavioral-wording change with concrete empirical evidence (local skillscope behavioral run counts: 1/5 pass on the old wording, 10/10 on the new, 0/3 on a mutant that regresses the guarded behavior) rather than asserting the fix works.
• The PR description is honest about what it does not verify here (judge runs through a different model gateway than CI's) and defers real gaps (no rocm diagnose fixtures yet to grade has_match/routing on this host) to an explicit Follow-ups section instead of papering over them.
• Consistent with the project's own AGENTS.md guidance for this folder (skills/rocm-doctor/evals/evals.json's stated coverage bar of ≥3 trigger-prompts / ≥2 near-misses is unaffected — still 5/2), and continues a convention this same entry picked up earlier the same day in the already-merged #596 rather than conflicting with it.

Deployment notes

None — eval-fixture-only change, no CLI behavior, schema, or dependency change.

What this covered

Read in full: the diff; skills/rocm-doctor/evals/evals.json at the head commit (all 7 entries, not just the changed one); skills/rocm-doctor/SKILL.md and reference.md at the same commit; the repo's AGENTS.md; the commit (author/committer, DCO sign-off); the CI workflow run for this head (CI, conclusion success); the PR description; and the referenced ticket amd/skills#259, retrieved by identifier. Also searched the repo for any other reference to this eval file or entry id (none found outside evals.json itself).

Run as a single pass plus two independent fan-out workers (one on the changed file + the two docs it tests against; one on agent-instruction adherence, file history and prior related PRs), per this review's own coupling-and-independent-sources method — reasonable for a 7-line, one-file diff. The history pass additionally found and read the immediately-preceding related PR (#596) on this same eval entry; that pass noted its GitHub search tool hit a permission error on repo: qualifiers partway through and worked around it with alternate calls — noted for completeness, it did not block the check.

Did not run: no CI-trigger phrase scan needed (this is a direct post, not a scripted pipeline). No narrowing was applied — this is the first review on this PR (no prior reviews or comments existed at the time of this check), so the whole change was read blind, with the PR description and ticket read only afterward per this method's ordering. Not reconciled against any other review or discussion thread, because none exists yet.

Infra note: this was posted directly via GitHub's API rather than through this org's usual claim/checkout/post pipeline, because the local gh CLI's token is invalid in this environment — so no claim lock was taken against a concurrent duplicate review. Checked immediately beforehand that no review or comment yet existed on this PR, which keeps the risk low for a one-off request, but repeated or scheduled automated review of this PR should route through a working gh auth and the standard pipeline instead.

[jussielo-amd]

Reviewed the diff (eval fixture edit to note/expected_behavior/unexpected_behavior for one skillscope item). No findings — the new claim that rocm diagnose works without --symptom matches diagnose.rs's handling of symptom.is_empty(), and the JSON stays valid and internally consistent with the closed-catalog gating rule. CI is green.