Skip to content

Make the MCP dispatch check's tool discovery deterministic - #222

Merged
handstandsam merged 1 commit into
mainfrom
fix-mcp-dispatch-check-default-target
Aug 2, 2026
Merged

Make the MCP dispatch check's tool discovery deterministic#222
handstandsam merged 1 commit into
mainfrom
fix-mcp-dispatch-check-default-target

Conversation

@handstandsam

Copy link
Copy Markdown
Collaborator

Follow-up to #221. The android-tests-mcp-dispatch job it added is failing on mainrun 30730194349. Everything else in that run is green.

The job never reached either assertion. It failed at tool discovery: it required assertVisible, which lives in the verification toolset and is only advertised when the resolved target declares it. TrailblazeMcpServer.resolveTargetScopedToolClasses scopes the advertised surface per MCP session, and a bare CI checkout resolves a narrower set than a developer machine carrying a persisted target selection — so the check passed locally and failed in CI.

Chasing that down surfaced a second, independent problem: a freshly-created MCP session does not reliably inherit the device the CLI pinned to the terminal. Locally I hit a run whose tools/list held only the session-management tools and no TrailblazeTools at all, which would have failed the job the same way for a completely different reason.

Changes

  • Bind the device to the MCP session itself with connectToDevice instead of relying on the CLI's terminal pin carrying over. That is the daemon's own per-session bind and it fires the tools/list_changed that registers the driver-scoped surface.

  • Assert on tap instead of assertVisible. tap and tapOnPoint both come from the baseline catalog every Android driver resolves with no target-declared toolsets, so they are advertised regardless of workspace. tap against an absent ref fails inside the on-device runner exactly the way assertVisible did, so the regression under test is unchanged:

    [FAILED] Failed to execute tap
    On-device execution of TapTrailblazeTool failed: … Element ref 'zzz999' not found on current screen …
    
  • Comments now record why a verification/memory tool must not be reintroduced here, and the discovery error message no longer claims "the device never bound" when the real cause is a narrower surface.

  • -t default on the connect makes the target explicit. It does not by itself pin the advertised surface — that is per-MCP-session — so the comment says so rather than implying the pin is what makes this deterministic.

Testing

Four consecutive runs of the script against an Android 35 emulator, each with a fresh daemon: exit 0 every time, identical 49-tool surface, both assertions passing. The run-to-run variance that produced the empty-surface session is gone. shellcheck clean.

The one thing local runs still cannot reproduce is CI's narrower target surface — even with -t default this machine advertises assertVisible. That tap is present in CI's surface comes from the failing job's own tool dump, not from a local run.

The job failed on main at tool discovery: it required `assertVisible`, which
lives in the `verification` toolset and is only advertised when the resolved
target declares it. A bare CI checkout resolves a narrower surface than a
developer machine with a persisted target selection, so the check passed locally
and failed in CI.

Two changes:

- Bind the device to the MCP session itself via `connectToDevice` rather than
  relying on the CLI's terminal pin carrying over. That pin does not reliably
  reach a freshly-created MCP session — observed locally producing a session
  whose tools/list held only session-management tools and no TrailblazeTools.
- Assert on `tap` instead of `assertVisible`. Both `tap` and `tapOnPoint` come
  from the baseline catalog every Android driver resolves with no
  target-declared toolsets, so they are present regardless of workspace. `tap`
  against an absent ref fails on device the same way `assertVisible` did.

Verified with four consecutive local runs against an Android 35 emulator: exit 0
each time, identical 49-tool surface, both assertions passing.
@handstandsam
handstandsam merged commit 29aefb1 into main Aug 2, 2026
4 of 5 checks passed
@handstandsam
handstandsam deleted the fix-mcp-dispatch-check-default-target branch August 2, 2026 04:09
mannydelgado pushed a commit to mannydelgado/trailblaze that referenced this pull request Aug 3, 2026
The job failed on main at tool discovery: it required `assertVisible`, which
lives in the `verification` toolset and is only advertised when the resolved
target declares it. A bare CI checkout resolves a narrower surface than a
developer machine with a persisted target selection, so the check passed locally
and failed in CI.

Two changes:

- Bind the device to the MCP session itself via `connectToDevice` rather than
  relying on the CLI's terminal pin carrying over. That pin does not reliably
  reach a freshly-created MCP session — observed locally producing a session
  whose tools/list held only session-management tools and no TrailblazeTools.
- Assert on `tap` instead of `assertVisible`. Both `tap` and `tapOnPoint` come
  from the baseline catalog every Android driver resolves with no
  target-declared toolsets, so they are present regardless of workspace. `tap`
  against an absent ref fails on device the same way `assertVisible` did.

Verified with four consecutive local runs against an Android 35 emulator: exit 0
each time, identical 49-tool surface, both assertions passing.

Signed-off-by: Manny Delgado <manny@duolingo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant