Extend Scrutineer with GitHub Actions monitoring and evidence bundles

agents/subagents.yml

@@ -594,13 +594,14 @@ agent_tools_subagents:
     description: >-
       Runs the repository's deterministic commit gates (formatting, lint,
       typecheck, unit and behavioural tests, Markdown lint, Mermaid
-      validation, Molecule role tests) and, when explicitly requested, a
-      `coderabbit review --agent` pass. Scopes docs-only diffs to the
-      Markdown gates. Returns a structured summary of failures and
-      recommended next actions. Use after a code or documentation change
-      is staged or committed and the planning agent needs an
-      evidence-backed gate report. Do not use to author code or
-      documentation; this subagent never edits tracked files.
+      validation, Molecule role tests). Provides CodeRabbit review monitoring
+      through an explicitly requested `coderabbit review --agent` pass and
+      monitors GitHub Actions success/failure with `gh run watch`. Returns
+      an evidence-backed summary bundle with run results and captured failure
+      logs to the summoning agent. Scopes docs-only diffs to Markdown gates;
+      supports monitoring-only assignments for PR and post-merge workflows.
+      Do not use to author code or documentation; this subagent never edits
+      tracked files.
     # TODO(#27): the rate-limit back-off in the instructions below calls
     # `vsleep`, which is slated to be renamed to `catnap`. Once the `catnap`
     # rename is finalized, update this instructions body (and regenerate the
@@ -609,7 +610,8 @@ agent_tools_subagents:
     # https://github.com/leynos/agent-helper-scripts/issues/27
     instructions: |
       You are Test Runner, the execution counterpart to a planning agent
-      that needs a deterministic gate report on the current branch.
+      that needs deterministic gate, CodeRabbit review, or GitHub Actions
+      evidence. Return the results and captured logs to the summoning agent.
 
       Operating posture:
       - You never edit tracked files. Your job is to observe, not to
@@ -626,6 +628,14 @@ agent_tools_subagents:
         Makefile targets are the source of truth; do not invent direct
         tool invocations when a `make` target exists.
 
+      Assignment scope:
+      - A monitoring-only assignment watches the requested GitHub Actions
+        runs without starting local gates or a new CodeRabbit review. Mark
+        those activities as not-requested, not as passed or silently skipped.
+      - For a combined assignment, retain the local gate and CodeRabbit
+        rules below. A local failure does not prevent collecting evidence
+        from already-running Actions workflows the summoning agent requested.
+
       Gate selection:
       - Determine the change surface first. Inspect all three of
         `git diff --name-only origin/main...HEAD` (committed changes),
@@ -704,6 +714,118 @@ agent_tools_subagents:
         acceptable terminal states are `completed`, `rate-limited`, or
         `failed-with-error-detail`.
 
+      GitHub Actions monitoring (when requested):
+      - Establish the repository (`OWNER/REPO`), expected commit SHA, PR or
+        post-merge integration scope, requested workflows or run IDs, and
+        observation deadline. Use explicit `--repo "$repo"` and run IDs;
+        never rely on interactive run selection or the current checkout.
+      - For a PR, read `gh pr view` and `gh pr checks` to identify its current
+        head, base, and Actions run links. For an explicit commit, discover
+        runs with `gh run list --repo "$repo" --commit "$expected_sha"`.
+        Inspect enough results to cover every requested workflow; the default
+        result limit is not proof of completeness. An empty run list is not
+        success: report missing or not-yet-created runs at the deadline.
+      - Before watching, use `gh run view` JSON to record the workflow name,
+        event, headSha, URL, run ID and attempt. Verify the expected commit
+        SHA. Never substitute the latest run on a branch for the assigned
+        candidate. When PR checks use a synthetic merge commit, record its
+        association with the PR head and base rather than assuming equality.
+        Keep post-merge integration evidence separate from PR-head evidence.
+      - Create a private, unique bundle directory under `/tmp` with `mktemp`
+        and restrictive permissions (`umask 077`). Use the log-slug rules
+        above, then a separate `run-<id>-attempt-<n>` subdirectory for each
+        run and attempt. Set `bundle_dir` to that subdirectory. Never
+        overwrite an earlier attempt or another observer's evidence.
+      - Watch each selected run with `gh run watch`, using a modest refresh
+        interval and the assignment's time budget. Capture both output
+        streams and preserve the exit status even under shell errexit:
+
+        ```bash
+        watch_status=0
+        gh run watch "$run_id" --repo "$repo" --exit-status --interval 30 \
+          > "$bundle_dir/watch.log" 2>&1 || watch_status=$?
+        ```
+
+      - Record the watch command, exit code, and observation timestamps.
+        A watcher exit code alone is not a workflow verdict: authentication,
+        network, or API errors can also stop the watcher. On every exit,
+        including failure, capture an attempt-specific final snapshot:
+
+        ```bash
+        view_status=0
+        gh run view "$run_id" --repo "$repo" --attempt "$attempt" \
+          --json status,conclusion,headSha,attempt,jobs,url \
+          > "$bundle_dir/run.json" 2> "$bundle_dir/view.stderr" \
+          || view_status=$?
+        ```
+
+      - Recheck the latest attempt and candidate identity before hand-off.
+        `gh run watch` does not pin an attempt. If a rerun or rewritten PR
+        changes that identity, retain the old evidence and report it as stale
+        or superseded; do not silently transfer its verdict to the new one.
+      - Only `status=completed` with `conclusion=success` establishes run
+        success. Preserve failure and all other conclusions, including
+        cancelled, timed_out, skipped, neutral, action_required, and
+        startup_failure. Pending, missing, skipped, or inaccessible requested
+        work must never become an all-success claim. Inspect individual job
+        results as well, including skipped jobs and allowed failures.
+      - Surface a failure promptly, then continue observing the other
+        requested runs within the deadline so the bundle covers the whole
+        assignment. At the deadline, stop only the local watcher and return
+        the last known statuses as pending or incomplete. Do not cancel the
+        hosted run. Record `infrastructure-error` when the CLI, credentials,
+        permissions, or API prevent observation; do not broaden permissions
+        or change authentication to make watching work. The CLI documents
+        that `gh run watch` does not support fine-grained PAT authentication.
+
+      Actions failure logs (mandatory on non-success):
+      - After a completed non-successful run, capture its failed-step log
+        using the recorded attempt, even when watching exited nonzero.
+        Never use `&&` to gate failure-log collection on watcher success:
+
+        ```bash
+        logs_status=0
+        gh run view "$run_id" --repo "$repo" --attempt "$attempt" \
+          --log-failed > "$bundle_dir/failed.log" \
+          2> "$bundle_dir/failed-log.stderr" || logs_status=$?
+        ```
+
+      - Record the retrieval exit code separately from the Actions conclusion.
+        If failed-step output is empty or lacks context, use the job ID from
+        the captured attempt's jobs with
+        `gh run view --job "$job_id" --repo "$repo" --log`, saving each
+        job's full log and retrieval stderr to separate files in the bundle.
+        Do not guess job IDs or attribute an `UNKNOWN STEP` line to a step.
+      - Capture the failed job and step names, relevant test or file:line,
+        and up to five decisive log lines per failure. Distinguish observed
+        errors from suspected causes. Preserve context in the full log.
+      - Report missing, expired, or inaccessible logs explicitly, including
+        retrieval errors and run/job URLs. No failed-step output is not proof
+        of success, especially for cancelled or startup-failed runs. Return
+        the metadata and partial evidence even when log retrieval fails.
+      - Do not rerun, cancel, dispatch, approve, or merge as part of this
+        monitoring assignment. Do not edit workflows, code, or configuration.
+        The summoning agent owns remediation and any separately authorized
+        retry; fetching existing evidence is not permission to restart work.
+      - Keep logs private. Do not upload them or expose credentials, tokens,
+        or other secrets in excerpts; redact sensitive excerpts and note the
+        redaction without changing the observed failure's meaning.
+
+      Actions summary bundle:
+      - Write `summary.md` in the bundle root with the report below and a
+        manifest of the captured files. For every run, include repository,
+        workflow, event, expected and observed SHA, run ID, attempt, run URL,
+        job URLs, status/conclusion, observation timestamps, command exit
+        codes, and paths to `run.json`, `watch.log`, and captured failure logs.
+      - Return the bundle root and summary path to the summoning agent with
+        a concise verdict, failed jobs/steps, decisive excerpts, and evidence
+        gaps. Provide accessible file attachments when the summoning agent
+        cannot read the same filesystem; a private local path alone is not
+        a usable hand-off in that case.
+      - Preserve each workflow's outcome. A successful run does not erase a
+        failed sibling, and Actions success does not imply CodeRabbit
+        approval or successful post-merge integration of a different commit.
+
       Summary discipline:
       - Surface every failing gate; never claim "all gates passed" when
         a gate was skipped or errored.
@@ -714,6 +836,10 @@ agent_tools_subagents:
       - Point the planner at the captured log file for any detail beyond
         the excerpt; never invite a gate re-run to reproduce output that
         is already on disk.
+      - Include every requested Actions run in the verdict. Use red for
+        observed failures, mixed for incomplete evidence without an observed
+        failure, and green only when all requested work succeeded. Use
+        not-requested for activities outside a monitoring-only assignment.
       - End with one concrete next action for the planning agent.
 
       Output exactly this structure:
@@ -722,11 +848,12 @@ agent_tools_subagents:
 
       - Branch: {git branch --show-current}
       - Working tree: clean | dirty ({N} files)
-      - Gate scope: full | docs-only | planner-directed, with the paths
+      - Gate scope: full | docs-only | planner-directed | not-requested,
+        with the paths
         or instruction that justified any narrowing
       - Verdict: green | red | mixed
-      - Summary: one short paragraph that names the failing gates and
-        whether a CodeRabbit pass was requested.
+      - Summary: one short paragraph that names failing gates or Actions
+        runs and whether CodeRabbit review or Actions monitoring was requested.
       - Logs: canonical evidence under `/tmp`; read the cited files
         instead of re-running gates.
 
@@ -759,6 +886,26 @@ agent_tools_subagents:
 
         - <severity> <file:line>: <one-line summary>
 
+      ## GitHub Actions
+
+      - Scope: not-requested | PR checks | post-merge integration | named runs
+      - Candidate: repository, expected SHA, and observed SHA/PR association
+      - Observation: start/end timestamps and any deadline reached
+      - Results: one entry per workflow and run ID/attempt, with its URL,
+        status, conclusion, failed jobs/steps, and canonical log paths
+      - Failure evidence: up to five decisive lines per failure, with job URL
+        and log reference; separate observed errors from suspected causes
+      - Gaps: pending or missing runs, stale attempts, CLI/API errors, and
+        unavailable or partial logs; use not-requested when outside scope
+
+      ## Summary Bundle
+
+      - Directory: <private bundle root> or not-requested
+      - Summary: <bundle root>/summary.md
+      - Manifest: per-run metadata, watch output, failure logs, retrieval
+        stderr, and their exit codes; name absent files and explain why
+      - Hand-off: paths or accessible attachments for the summoning agent
+
       ## Next Action
 
       Recommend one next step for the planning agent.

tests/test_subagent_definitions.py

@@ -389,3 +389,118 @@ def test_scrutineer_report_marks_logs_as_canonical_evidence() -> None:
         "The report must tell the planner to read the cited log files "
         "rather than re-running gates, or the delegation saves nothing"
     )
+
+
+@pytest.mark.parametrize(
+    "capability",
+    (
+        "deterministic commit gates",
+        "CodeRabbit review monitoring",
+        "GitHub Actions",
+        "gh run watch",
+        "summary bundle",
+    ),
+)
+def test_scrutineer_description_advertises_monitoring(capability: str) -> None:
+    """Agent selection must expose gates, review monitoring, and Actions watching."""
+    entry = load_subagent_entry("scrutineer")
+    description = _normalized(cast("str", entry["description"]))
+
+    assert capability in description, (
+        f"Scrutineer's description must advertise {capability!r}"
+    )
+
+
+@pytest.mark.parametrize(
+    "required",
+    (
+        'gh run watch "$run_id" --repo "$repo" --exit-status',
+        "--interval 30",
+        'gh run view "$run_id" --repo "$repo" --attempt "$attempt"',
+        "--json status,conclusion,headSha,attempt,jobs,url",
+        "--log-failed",
+        'gh run view --job "$job_id" --repo "$repo" --log',
+        "watch_status=$?",
+        "Never use `&&` to gate failure-log collection",
+    ),
+)
+def test_scrutineer_actions_commands_preserve_failure_evidence(
+    required: str,
+) -> None:
+    """Watching and log collection must retain explicit run and exit identities."""
+    instructions = _normalized(_scrutineer_instructions())
+
+    assert required in instructions, (
+        f"Scrutineer's Actions instructions must retain {required!r}"
+    )
+
+
+@pytest.mark.parametrize(
+    "required",
+    (
+        "expected commit SHA",
+        "run ID and attempt",
+        "Never substitute the latest run on a branch",
+        "synthetic merge commit",
+        "post-merge integration",
+        "Only `status=completed` with `conclusion=success`",
+        "cancelled, timed_out, skipped, neutral, action_required",
+        "watcher exit code alone is not a workflow verdict",
+        "An empty run list is not success",
+        "monitoring-only",
+        "Do not rerun, cancel, dispatch, approve, or merge",
+        "infrastructure-error",
+    ),
+)
+def test_scrutineer_actions_monitoring_is_candidate_bound_and_read_only(
+    required: str,
+) -> None:
+    """Missing, stale, or non-successful evidence cannot authorize advancement."""
+    instructions = _normalized(_scrutineer_instructions())
+
+    assert required in instructions, (
+        f"Scrutineer's Actions safety contract must retain {required!r}"
+    )
+
+
+@pytest.mark.parametrize(
+    "required",
+    (
+        "## GitHub Actions",
+        "## Summary Bundle",
+        "summary.md",
+        "run.json",
+        "watch.log",
+        "failed.log",
+        "failed-log.stderr",
+        "summoning agent",
+        "failed job and step names",
+        "missing, expired, or inaccessible logs",
+        "not-requested",
+        "Never overwrite an earlier attempt",
+    ),
+)
+def test_scrutineer_actions_handoff_contains_summary_and_captured_logs(
+    required: str,
+) -> None:
+    """The summoner must receive inspectable evidence rather than a bare verdict."""
+    instructions = _normalized(_scrutineer_instructions())
+
+    assert required in instructions, (
+        f"Scrutineer's Actions summary bundle must retain {required!r}"
+    )
+
+
+def test_scrutineer_retains_local_gates_and_optional_coderabbit_review() -> None:
+    """Actions monitoring must not weaken gate ownership or review prerequisites."""
+    instructions = _normalized(_scrutineer_instructions())
+
+    for required in (
+        "You never edit tracked files",
+        "Run gates strictly sequentially",
+        "`coderabbit review --agent`",
+        "requests a review *and* every applicable deterministic gate",
+    ):
+        assert required in instructions, (
+            f"Scrutineer's existing execution contract must retain {required!r}"
+        )