From d70cf80fea2c573cc3be45df2af8dcae2c96d3f2 Mon Sep 17 00:00:00 2001 From: Eugene Volen Date: Thu, 8 Oct 2026 17:27:40 +0200 Subject: [PATCH] test(skill): stop the eval demanding error text the user said they lack The `rocm-nothing-established-routes-upstream` case expected the agent to ask for the exact error text. Its prompt says the user cannot paste one, and `rocm diagnose` probes the host without `--symptom`, so an agent that offers to install the CLI and probe anyway is right. The judge passed that behaviour on some runs and failed it on others. Replace the item with the guard it stands for: missing error text must not stop the diagnosis or invite a guess. Asking as well is fine; requiring it first is not. The `has_match` item required routing upstream, which a grading host without the CLI never reaches. Fold the `has_match` rule into the existing no-remediation guard, where it is gradeable on any host. Signed-off-by: Eugene Volen --- skills/rocm-doctor/evals/evals.json | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/skills/rocm-doctor/evals/evals.json b/skills/rocm-doctor/evals/evals.json index 8c9851ebf..fd23d223e 100644 --- a/skills/rocm-doctor/evals/evals.json +++ b/skills/rocm-doctor/evals/evals.json @@ -40,13 +40,12 @@ "id": "rocm-nothing-established-routes-upstream", "skill_should_trigger": true, "prompt": "My AMD GPU setup feels wrong on Linux -- something about ROCm is off but I can't paste an exact error. Can you work out what's broken and fix it?", - "note": "A vague report the catalog cannot explain. `matched` can still come back non-empty with sub-threshold entries, so an agent gating on emptiness would propose a fix for a healthy host. The rule is to read `has_match` and route upstream when it is false.", + "note": "A vague report the catalog cannot explain, from a user with no error text. `matched` can still come back non-empty with sub-threshold entries, so an agent gating on emptiness would propose a fix for a healthy host. The rule is to read `has_match` and route upstream when it is false. The grading host has no `rocm` CLI, so the agent stops at the Phase 0 install offer and never sees a report: routing cannot be observed there, and the `has_match` rule is graded as the guard it protects, no remediation without an established cause. The missing error text is no reason to stop or to guess, because `rocm diagnose` probes the host without `--symptom`. Both items say so themselves, because the judge reads the item rather than this note.", "expected_behavior": [ - "Treat `has_match` being false as 'no cause established' and route the user upstream, rather than proposing a fix from a sub-threshold entry in `matched`", - "Ask the user for the exact error text instead of inventing a diagnosis" + "Do not let the missing error text stop the diagnosis or invite a guess: check for the `rocm` CLI and either run `rocm diagnose` without `--symptom` or offer to install the CLI so it can. Asking for the error text as well is fine; requiring it before probing the host is not" ], "unexpected_behavior": [ - "Propose or apply a remediation for the user's problem when the CLI established no cause. Installing the `rocm` CLI itself with consent, as Phase 0 of the skill instructs, is not such a remediation", + "Propose or apply a remediation for the user's problem when the CLI established no cause, that is when `has_match` is false even if `matched` is non-empty. Installing the `rocm` CLI itself with consent, as Phase 0 of the skill instructs, is not such a remediation", "Invent a failure mode that is not in the CLI's closed catalog" ] },