Investigated and drafted with AI assistance (Claude), based on hands-on reproduction testing against a live deployment and a separately built copy of current main — see Reproduction below for the scripts used.
Environment
- Switchyard commit:
819e462c (current main at time of testing; also reproduces on 053a61e2, an earlier commit on the same branch)
switchyard-server --version reports: 0.2.0 (Cargo.toml version string; not yet bumped past the last tag)- OS: Linux, x86_64 (Debian 13)
- Rust: 1.96.1
- Backend: an OpenAI-compatible local inference server (LM Studio), reached via
format = "openai_chat"
- Backend model: a local MoE model, ~35B params
- Inbound protocol tested: Anthropic Messages (
/v1/messages) and OpenAI Chat Completions (/v1/chat/completions) — both show the same behavior
- Streaming: enabled (
"stream": true)
- Route type:
passthrough
Description
Under concurrent request load, Switchyard intermittently sends complete HTTP response headers (200 OK, Transfer-Encoding: chunked, Content-Type: text/event-stream) for a streamed request, then writes zero response body bytes, leaving the connection open indefinitely. The client never receives any SSE events, not even message_start.
Critically, Switchyard's own access log records the request as fully and successfully handled (status=200, a small handling_duration_ms) for every affected request — including the ones whose clients never receive a single body byte. This appears in both switchyard_server::request: LLM request handled log lines.
This is not specific to Anthropic-format translation. It reproduces identically when both the client and the backend speak plain OpenAI Chat Completions (no cross-format translation involved at all), which localizes the bug to Switchyard's general concurrent streaming-response delivery path rather than anything protocol-translation-specific.
Concurrency is the trigger, not request size. A large (~78KB) request sent sequentially — one at a time, never overlapping with another in-flight request — never hung in any of our tests (32 sequential attempts across two commits, generous 100s timeout, content varied per attempt to defeat backend prompt caching). The same request fired as 8 concurrent, overlapping requests hung on 6 of 8 attempts, consistently, across three separate test configurations (Anthropic-in/OpenAI-backend on two commits, and OpenAI-in/OpenAI-backend on current main).
What works
Verified directly, isolated from this bug:
- Direct backend (LM Studio) text generation — correct
- Direct backend streaming — correct
- Direct backend tool-calling (OpenAI-style
tools/tool_calls) — correct
- Switchyard non-streaming Anthropic translation — correct, full body received
- Switchyard streaming Anthropic translation, single sequential request (any size tested, up to ~78KB) — correct, full body received
- Switchyard streaming tool-call translation (Anthropic
tool_use / input_json_delta) — correct
- Switchyard streaming OpenAI-to-OpenAI passthrough, single sequential request — correct
The failure requires concurrent in-flight streamed requests to the same Switchyard instance.
Reproduction
Two equivalent reproductions are attached: a curl-only shell script (repro.sh) for a quick check, and a Python harness (repro_concurrent.py) that reports precise per-request timing (time to headers, time to first body byte, total bytes, hang/success classification). A repro_sequential.py control script is included to show the same payload does not hang when requests don't overlap.
Both use a synthetic fixture (generate_fixture.py) — repeated filler text sized to ~78KB, no proprietary prompt content. Payload shape is intentionally generic (a system block + a short user message); no tools are required to reproduce it, though we also confirmed the bug with tool definitions present.
# routes.toml — minimal config used for testing
schema_version = 1
[llm_clients.local]
format = "openai_chat"
base_url = "http://127.0.0.1:PORT/v1" # your OpenAI-compatible backend
[targets.local_model]
id = "YOUR-LOCAL-MODEL-ID"
llm_client = "local"
[routes.local]
id = "local"
type = "passthrough"
target = "local_model"
switchyard-server --config routes.toml --host 0.0.0.0 --port 4000
# in another terminal:
./repro.sh localhost:4000 local
Expect roughly 60-80% of the 8 concurrent requests to receive 200 OK headers and then nothing — curl reports Operation timed out ... with 0 bytes received after the 100s --max-time. The rest complete normally, usually within the time the backend actually takes to generate a response for that request size.
repro_concurrent.py (Python, precise timing)
python3 generate_fixture.py repro.json
# edit repro.json's "model" field to match your route id
python3 repro_concurrent.py repro.json <switchyard-host-ip> <port> 8 /v1/messages
Observed result (one hung worker, from an actual run)
> POST /v1/messages HTTP/1.1
> Content-Length: 78248
...
< HTTP/1.1 200 OK
< content-type: text/event-stream
< cache-control: no-cache
< transfer-encoding: chunked
< date: ...
[connection stays open — zero body bytes — for the full 100s client timeout]
Switchyard's log for the same request, timestamped within ~200ms of the request arriving:
INFO switchyard_server::request: LLM request handled wire_format=anthropic_messages
status=200 requested_model="local" selected_model="<backend-model-id>"
streaming=true handling_duration_ms=152.8 error=""
Packet capture (on an earlier, single-request reproduction that first surfaced this) confirms the same shape at the wire level: the response's HTTP headers are sent as a single ~200-byte packet ending in the header-terminating blank line, and no further bytes follow on that connection.
Expected result
A valid SSE stream (message_start → content deltas → message_stop, or the OpenAI-format equivalent) should begin promptly and terminate normally, for every concurrently in-flight request — matching what happens when the same requests are sent sequentially.
Reproduction frequency
Measured across three independent 8-concurrent-request batches (two different Switchyard commits, two different inbound wire formats):
Anthropic-in / OpenAI-backend, commit 053a61e2: 6 hangs in 8 concurrent attempts
Anthropic-in / OpenAI-backend, commit 819e462c (main): 6 hangs in 8 concurrent attempts
OpenAI-in / OpenAI-backend, commit 819e462c (main): 6 hangs in 8 concurrent attempts
Sequential (non-overlapping) requests, same payload, same commits: 0 hangs in 32 attempts (20 against main, 12 against 053a61e2).
This is a standalone, high-probability reproduction under concurrency — we did not push it to full determinism, and the exact concurrency level, timing, and backend response speed needed likely affect the hit rate.
Request size behavior
Larger (Claude-Code-scale, ~60-85KB) requests made the issue easy to notice in real-world use, but size alone does not appear to be the trigger — sequential requests of the same size never hung in our testing. We did not fully map the size sensitivity under concurrency (e.g., whether smaller concurrent requests also hang, just less often); we'd guess size matters insofar as it affects how long a request stays in flight (and therefore how likely it is to overlap with others), rather than being a hard threshold in its own right, but we're flagging this as a guess, not a measured result.
Additional isolation (protocol matrix)
Only rows we actually tested:
| Inbound format |
Backend format |
Streaming |
Concurrency |
Result |
| Anthropic |
OpenAI Chat |
no |
1 |
passes |
| Anthropic |
OpenAI Chat |
yes |
1 (sequential) |
passes, every size tested up to ~78KB |
| Anthropic |
OpenAI Chat |
yes |
8 concurrent |
6/8 hang |
| OpenAI Chat |
OpenAI Chat |
yes |
8 concurrent |
6/8 hang |
Possible area to investigate
We have not modified or debugged the source, so this is tentative, offered only to narrow where a maintainer might look first. The candidates we noted while reading (read-only) were:
crates/switchyard-server/src/sse.rs — frame_stream, which wraps the translated event stream in Axum's Sse response type. This is a thin wrapper over a well-tested Axum primitive, so we consider it a less likely culprit on its own, but it's the last stage before bytes reach the client.crates/libsy-llm-client/src/client.rs — around call_rewrite_model_raw / call_rewrite_model, specifically the http_response.bytes_stream() → decode_stream(...) chain (~lines 428-450 in the commit we tested) that adapts the backend's request byte stream into the neutral IR chunk stream Switchyard re-encodes from. Since the bug reproduces identically for OpenAI-to-OpenAI (no format translation), we'd look at whatever is shared across all streaming responses regardless of format, rather than anything format-specific in switchyard-translation's codecs.
We did not find a way to distinguish, from outside the process, whether the backend response is fully consumed and then dropped, or never fully consumed in the first place — a maintainer with source-level tracing/instrumentation could resolve that quickly where we couldn't from outside.
We looked for related recent changes and found none: the 7 commits on main since 053a61e2 don't touch switchyard-server, switchyard-translation, or libsy-llm-client's streaming paths (the closest, an extraction of config/routing into a new switchyard-runner crate, only touches decision-endpoint types and token-counting).
*Reproduction files:
generate_fixture.py
repro_concurrent.py
repro_sequential.py
repro.sh
Investigated and drafted with AI assistance (Claude), based on hands-on reproduction testing against a live deployment and a separately built copy of current
main— see Reproduction below for the scripts used.Environment
819e462c(currentmainat time of testing; also reproduces on053a61e2, an earlier commit on the same branch)switchyard-server --versionreports:0.2.0(Cargo.toml version string; not yet bumped past the last tag)format = "openai_chat"/v1/messages) and OpenAI Chat Completions (/v1/chat/completions) — both show the same behavior"stream": true)passthroughDescription
Under concurrent request load, Switchyard intermittently sends complete HTTP response headers (
200 OK,Transfer-Encoding: chunked,Content-Type: text/event-stream) for a streamed request, then writes zero response body bytes, leaving the connection open indefinitely. The client never receives any SSE events, not evenmessage_start.Critically, Switchyard's own access log records the request as fully and successfully handled (
status=200, a smallhandling_duration_ms) for every affected request — including the ones whose clients never receive a single body byte. This appears in bothswitchyard_server::request: LLM request handledlog lines.This is not specific to Anthropic-format translation. It reproduces identically when both the client and the backend speak plain OpenAI Chat Completions (no cross-format translation involved at all), which localizes the bug to Switchyard's general concurrent streaming-response delivery path rather than anything protocol-translation-specific.
Concurrency is the trigger, not request size. A large (~78KB) request sent sequentially — one at a time, never overlapping with another in-flight request — never hung in any of our tests (32 sequential attempts across two commits, generous 100s timeout, content varied per attempt to defeat backend prompt caching). The same request fired as 8 concurrent, overlapping requests hung on 6 of 8 attempts, consistently, across three separate test configurations (Anthropic-in/OpenAI-backend on two commits, and OpenAI-in/OpenAI-backend on current
main).What works
Verified directly, isolated from this bug:
tools/tool_calls) — correcttool_use/input_json_delta) — correctThe failure requires concurrent in-flight streamed requests to the same Switchyard instance.
Reproduction
Two equivalent reproductions are attached: a curl-only shell script (
repro.sh) for a quick check, and a Python harness (repro_concurrent.py) that reports precise per-request timing (time to headers, time to first body byte, total bytes, hang/success classification). Arepro_sequential.pycontrol script is included to show the same payload does not hang when requests don't overlap.Both use a synthetic fixture (
generate_fixture.py) — repeated filler text sized to ~78KB, no proprietary prompt content. Payload shape is intentionally generic (asystemblock + a short user message); no tools are required to reproduce it, though we also confirmed the bug with tool definitions present.Expect roughly 60-80% of the 8 concurrent requests to receive
200 OKheaders and then nothing — curl reportsOperation timed out ... with 0 bytes receivedafter the 100s--max-time. The rest complete normally, usually within the time the backend actually takes to generate a response for that request size.repro_concurrent.py(Python, precise timing)Observed result (one hung worker, from an actual run)
Switchyard's log for the same request, timestamped within ~200ms of the request arriving:
Packet capture (on an earlier, single-request reproduction that first surfaced this) confirms the same shape at the wire level: the response's HTTP headers are sent as a single ~200-byte packet ending in the header-terminating blank line, and no further bytes follow on that connection.
Expected result
A valid SSE stream (
message_start→ content deltas →message_stop, or the OpenAI-format equivalent) should begin promptly and terminate normally, for every concurrently in-flight request — matching what happens when the same requests are sent sequentially.Reproduction frequency
Measured across three independent 8-concurrent-request batches (two different Switchyard commits, two different inbound wire formats):
Sequential (non-overlapping) requests, same payload, same commits: 0 hangs in 32 attempts (20 against
main, 12 against053a61e2).This is a standalone, high-probability reproduction under concurrency — we did not push it to full determinism, and the exact concurrency level, timing, and backend response speed needed likely affect the hit rate.
Request size behavior
Larger (Claude-Code-scale, ~60-85KB) requests made the issue easy to notice in real-world use, but size alone does not appear to be the trigger — sequential requests of the same size never hung in our testing. We did not fully map the size sensitivity under concurrency (e.g., whether smaller concurrent requests also hang, just less often); we'd guess size matters insofar as it affects how long a request stays in flight (and therefore how likely it is to overlap with others), rather than being a hard threshold in its own right, but we're flagging this as a guess, not a measured result.
Additional isolation (protocol matrix)
Only rows we actually tested:
Possible area to investigate
We have not modified or debugged the source, so this is tentative, offered only to narrow where a maintainer might look first. The candidates we noted while reading (read-only) were:
crates/switchyard-server/src/sse.rs—frame_stream, which wraps the translated event stream in Axum'sSseresponse type. This is a thin wrapper over a well-tested Axum primitive, so we consider it a less likely culprit on its own, but it's the last stage before bytes reach the client.crates/libsy-llm-client/src/client.rs— aroundcall_rewrite_model_raw/call_rewrite_model, specifically thehttp_response.bytes_stream()→decode_stream(...)chain (~lines 428-450 in the commit we tested) that adapts the backend's request byte stream into the neutral IR chunk stream Switchyard re-encodes from. Since the bug reproduces identically for OpenAI-to-OpenAI (no format translation), we'd look at whatever is shared across all streaming responses regardless of format, rather than anything format-specific inswitchyard-translation's codecs.We did not find a way to distinguish, from outside the process, whether the backend response is fully consumed and then dropped, or never fully consumed in the first place — a maintainer with source-level tracing/instrumentation could resolve that quickly where we couldn't from outside.
We looked for related recent changes and found none: the 7 commits on
mainsince053a61e2don't touchswitchyard-server,switchyard-translation, orlibsy-llm-client's streaming paths (the closest, an extraction of config/routing into a newswitchyard-runnercrate, only touches decision-endpoint types and token-counting).*Reproduction files:
generate_fixture.py
repro_concurrent.py
repro_sequential.py
repro.sh