Skip to content

fix: survive transient MCP transport faults on preflight and exhausted retries#26

Merged
demianbrecht merged 3 commits into
mainfrom
fix/mcp-transient-fault-handling
Jun 19, 2026
Merged

fix: survive transient MCP transport faults on preflight and exhausted retries#26
demianbrecht merged 3 commits into
mainfrom
fix/mcp-transient-fault-handling

Conversation

@demianbrecht

Copy link
Copy Markdown
Contributor

Summary

Closes two transient-fault-is-fatal failure modes in the MCP HTTP path, where a momentary transport hiccup would abort an entire agent task.

Preflight retry (mcp.py)

The startup connectivity probe now retries transient transport faults (ConnectError / RemoteProtocolError) with capped exponential backoff before declaring a server unreachable. This mirrors what RetryTransport already does for in-session tool calls, but on the startup path the RetryTransport isn't in play yet. Persistent errors (DNS / SSL / auth) still fail on every attempt and surface, preserving diagnostic value. A GET timeout remains a benign "reachable" signal (MCP endpoints are POST-oriented).

In-band error on exhausted retries (oauth.py)

The MCP streamable-HTTP SDK runs each id-bearing tools/call as a background task in a session-scoped anyio task group, outside its own try/except. Re-raising a transport fault after retries are exhausted crashes the whole group, cancels the foreground request, and detonates at session teardown — taking the entire agent task down (a transient codesearch timeout should never be fatal).

Instead, RetryTransport now synthesizes an in-band JSON-RPC error response echoing the request id (using the SDK's own REQUEST_TIMEOUT code for consistency). The background task completes normally; the SDK demuxes the response to the foreground waiter, which raises a catchable McpError that the tool loop already handles. Falls back to re-raising when the request has no usable JSON-RPC id (notification / unparseable body) — there's no foreground waiter to route a response to.

Testing

  • make test — 741 passed, 7 skipped
  • New unit tests: TestPreflightRetry (succeeds first try, timeout-is-benign, retry-then-raise, retry-then-succeed, non-transient-no-retry) and RetryTransport exhaustion tests (synthesizes in-band error with id, re-raises without id, connect-error path).

…d retries

Two transient-fault-is-fatal failure modes in the MCP HTTP path are now absorbed:

- Preflight: retry transient transport faults (ConnectError/RemoteProtocolError)
  with capped exponential backoff before declaring a server unreachable, mirroring
  what RetryTransport does for in-session calls. Persistent errors (DNS/SSL/auth)
  still surface on every attempt.

- Exhausted retries: rather than re-raising into the MCP SDK's session-scoped task
  group (which crashes the group and takes the whole agent task down at teardown),
  synthesize an in-band JSON-RPC error response echoing the request id. The SDK
  demuxes it to the foreground waiter as a catchable McpError. Falls back to
  re-raising when there's no id to route a response to.
@github-actions github-actions Bot added bug Something isn't working tests labels Jun 19, 2026
Comment thread src/switchplane/mcp.py
_PREFLIGHT_BACKOFF_SECONDS = 1.0


async def _run_preflight(client, url: str, server_name: str) -> None:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Quality] Minor: _run_preflight(client, ...) leaves client untyped while the rest of the module annotates public-ish signatures (CLAUDE.md calls for type hints on APIs). Since httpx is imported lazily inside the function body, a string annotation works without reordering: client: "httpx.AsyncClient". Not blocking — just keeps the signature self-documenting given the docstring already commits to it being an httpx.AsyncClient.

@demianbrecht demianbrecht left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Security] No findings. Reviewed the MCP HTTP fault-handling changes (mcp.py preflight retry, oauth.py synthesized exhausted-retry response) plus test additions.

Critical: 0
High: 0
Medium: 0
Low: 0

Scope note: this PR is transport-resilience only — no auth/authz, injection sink, secret, or network-egress changes. Checked: (1) the synthesized JSON-RPC error parses the client's own outgoing request body (json.loads(request.content)), not untrusted external input — no injection vector; (2) the error message embeds the httpx transport exception string, which carries the config-supplied URL/host (already operator-known) but no credentials/tokens; (3) the preflight retry is bounded (3 attempts, capped backoff) with a narrow retryable set — no DoS amplification. The documented double-execution-of-non-idempotent-tools tradeoff on transport-fault retry is a correctness call the author accepted, consistent with the project's single-user local-only design intent — not a security defect. Tests use fictional hosts, no hard-coded real credentials.

@demianbrecht demianbrecht left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [Quality] Focused, well-executed fix for two transient-fault-is-fatal paths in the MCP HTTP layer. CI is green across the matrix (test 3.12/3.13/3.14, e2e, lint, SAST all passing).

Strengths:

  • The why is documented exceptionally well — both new functions carry comments explaining the SDK task-group teardown mechanism, and the RetryTransport idempotency caveat (retrying a transport fault can double-execute a non-idempotent tools/call) is called out honestly rather than papered over.
  • Test coverage is meaningful and exercises the real branches: preflight succeeds/timeout-benign/retry-then-raise/retry-then-succeed/non-transient, plus exhaustion-with-id (synthesizes), exhaustion-without-id (re-raises), and the connect-error path. Assertions check the actual JSON-RPC id echo, error code, and attempt counts — not framework noise.
  • The in-band-error synthesis correctly degrades to re-raise when there's no JSON-RPC id to route a response to (notification / unparseable body), which is the right boundary.
  • Patch version bump (0.10.1 -> 0.10.2) is appropriate for a bugfix.

One non-blocking nit posted inline: _run_preflight's client parameter is untyped while the surrounding code annotates signatures.

Observations (no action required):

  • Preflight backoff adds up to ~3s (1s + 2s) of startup latency on a server that flaps transiently before failing, and sessions start sequentially — acceptable for the failure mode this targets, but worth being aware of if many HTTP servers are configured.
  • The error-code-consistency claim (httpx.codes.REQUEST_TIMEOUT matching the SDK's BaseSession read-timeout path) depends on the pinned MCP SDK internals; if the SDK version moves, worth a re-check, but the chosen code is harmless either way since JSON-RPC permits any integer.

@demianbrecht demianbrecht merged commit 813fd59 into main Jun 19, 2026
9 checks passed
@demianbrecht demianbrecht deleted the fix/mcp-transient-fault-handling branch June 19, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant