Make the MCP dispatch check's tool discovery deterministic - #222
Merged
Conversation
The job failed on main at tool discovery: it required `assertVisible`, which lives in the `verification` toolset and is only advertised when the resolved target declares it. A bare CI checkout resolves a narrower surface than a developer machine with a persisted target selection, so the check passed locally and failed in CI. Two changes: - Bind the device to the MCP session itself via `connectToDevice` rather than relying on the CLI's terminal pin carrying over. That pin does not reliably reach a freshly-created MCP session — observed locally producing a session whose tools/list held only session-management tools and no TrailblazeTools. - Assert on `tap` instead of `assertVisible`. Both `tap` and `tapOnPoint` come from the baseline catalog every Android driver resolves with no target-declared toolsets, so they are present regardless of workspace. `tap` against an absent ref fails on device the same way `assertVisible` did. Verified with four consecutive local runs against an Android 35 emulator: exit 0 each time, identical 49-tool surface, both assertions passing.
mannydelgado
pushed a commit
to mannydelgado/trailblaze
that referenced
this pull request
Aug 3, 2026
The job failed on main at tool discovery: it required `assertVisible`, which lives in the `verification` toolset and is only advertised when the resolved target declares it. A bare CI checkout resolves a narrower surface than a developer machine with a persisted target selection, so the check passed locally and failed in CI. Two changes: - Bind the device to the MCP session itself via `connectToDevice` rather than relying on the CLI's terminal pin carrying over. That pin does not reliably reach a freshly-created MCP session — observed locally producing a session whose tools/list held only session-management tools and no TrailblazeTools. - Assert on `tap` instead of `assertVisible`. Both `tap` and `tapOnPoint` come from the baseline catalog every Android driver resolves with no target-declared toolsets, so they are present regardless of workspace. `tap` against an absent ref fails on device the same way `assertVisible` did. Verified with four consecutive local runs against an Android 35 emulator: exit 0 each time, identical 49-tool surface, both assertions passing. Signed-off-by: Manny Delgado <manny@duolingo.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #221. The
android-tests-mcp-dispatchjob it added is failing onmain— run 30730194349. Everything else in that run is green.The job never reached either assertion. It failed at tool discovery: it required
assertVisible, which lives in theverificationtoolset and is only advertised when the resolved target declares it.TrailblazeMcpServer.resolveTargetScopedToolClassesscopes the advertised surface per MCP session, and a bare CI checkout resolves a narrower set than a developer machine carrying a persisted target selection — so the check passed locally and failed in CI.Chasing that down surfaced a second, independent problem: a freshly-created MCP session does not reliably inherit the device the CLI pinned to the terminal. Locally I hit a run whose
tools/listheld only the session-management tools and noTrailblazeTools at all, which would have failed the job the same way for a completely different reason.Changes
Bind the device to the MCP session itself with
connectToDeviceinstead of relying on the CLI's terminal pin carrying over. That is the daemon's own per-session bind and it fires thetools/list_changedthat registers the driver-scoped surface.Assert on
tapinstead ofassertVisible.tapandtapOnPointboth come from the baseline catalog every Android driver resolves with no target-declared toolsets, so they are advertised regardless of workspace.tapagainst an absent ref fails inside the on-device runner exactly the wayassertVisibledid, so the regression under test is unchanged:Comments now record why a
verification/memorytool must not be reintroduced here, and the discovery error message no longer claims "the device never bound" when the real cause is a narrower surface.-t defaulton the connect makes the target explicit. It does not by itself pin the advertised surface — that is per-MCP-session — so the comment says so rather than implying the pin is what makes this deterministic.Testing
Four consecutive runs of the script against an Android 35 emulator, each with a fresh daemon: exit 0 every time, identical 49-tool surface, both assertions passing. The run-to-run variance that produced the empty-surface session is gone. shellcheck clean.
The one thing local runs still cannot reproduce is CI's narrower target surface — even with
-t defaultthis machine advertisesassertVisible. Thattapis present in CI's surface comes from the failing job's own tool dump, not from a local run.