[KimiK3][AgentX]: H200 KimiK3 Day 0 support - #2353
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
cquil11
changed the title
|
Claude finished @cquil11's task in 8m 0s —— View job Review of PR #2353
Found 1 blocking issue and 1 warning (see inline comments):
Checks that passed: Note: I could not verify the day-0 image's support for |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🔴
benchmarks/multi_node/agentic/kimik3_fp4_vllm.sh:96-113— The DEP16 throughput branch sets--data-parallel-hybrid-lb(kimik3_fp4_vllm.sh:102-113) without--headless, but this repo's own docs (benchmarks/multi_node/llm-d/README.md) explain hybrid-lb makes each node run an independent api-server that only load-balances its own local DP ranks — cross-node fan-out needs an external per-node router, which the Slurm/Pyxis connector doesn't provide. Since the AgentX client (agentic_srt.sh/benchmark_lib.sh:1760) only ever sends traffic tohttp://localhost:$PORTon the head node, the second node's 8 DP ranks (start-rank 8-15) never receive real requests, so the headline 'throughput-oriented' DEP16 sweep silently measures an 8-GPU deployment instead of 16.Extended reasoning...
The bug: In the DEP/throughput branch of
kimik3_fp4_vllm.sh(lines 96-113), the server is launched with--data-parallel-size 16,--data-parallel-size-local 8,--data-parallel-hybrid-lb, and critically no--headlesson either node. This is meant to stand up a single cross-node DP16+EP engine spanning the two allocated H200 nodes, one api-server-facing endpoint that fans requests out to all 16 DP ranks.Why this doesn't do what it looks like it does: This repo's own documentation at
benchmarks/multi_node/llm-d/README.md("Why one entry per node" section) explicitly documents the semantics of--data-parallel-hybrid-lb: 'With hybrid-lb, every node runs its own api-server ... and internally load-balances its own local DP ranks', and cross-node fan-out is handled by an external router (llm-d's EPP+Envoy), which is whyserver.sh'sadd_role()deliberately emits one endpoint per node rather than one shared endpoint. In other words, hybrid-lb does not create one logical multi-node frontend — it creates N independent frontends (one per node), each of which only sees its own local ranks, and something outside vLLM must route requests across those N frontends.The code path that triggers it: The Slurm/Pyxis connector added in this PR (
runners/connectors/slurm_pyxis.sh) provides no such external router. The client-sidesrunstep runs with--nodelist="$head_node"only (line ~282), i.e. the AgentX client container is placed exclusively on the head node. The client entrypoint,benchmarks/multi_node/agentic_srt.sh, delegates request construction tobenchmark_lib.sh'sbuild_replay_cmd, which hardcodesREPLAY_CMD+=" --url http://localhost:$PORT"(benchmark_lib.sh:1760) — i.e. every request in the sweep is sent to the head node's local api-server. The connector's readiness poll (curl ... "http://$head_node:$PORT/health") similarly only ever checks the head node. There is no second router, no per-node endpoint list, and no code that ever contacts the second node's api-server.Concrete walkthrough: With
WORLD_SIZE=16,GPUS_PER_NODE=8, node 0 (rank 0) starts an api-server onlocalhost:$PORTthat hybrid-lb load-balances across its local ranks 0-7. Node 1 (rank 1) independently starts its own api-server on the same port on its own host, hybrid-lb-balancing local ranks 8-15 — but nothing ever connects a client to it. The AgentX client, running only on the head node, sends its entire concurrency sweep (16, 32, 64, 128) tohttp://localhost:$PORT, which is node 0's api-server. Node 0 spreads that load across its 8 local ranks; node 1's 8 ranks sit idle for the duration of the run (aside from any lockstep EP all-to-all participation with dummy/empty batches). The result: a benchmark advertised and labeled as "DEP16" / 16-GPU throughput-oriented actually measures an 8-GPU deployment, with no error, warning, or indication in the output that half the allocated GPUs never processed real traffic.Why nothing else catches this: The TP/latency and TEP/balanced branches in the same file correctly avoid this trap — they use a single cross-node engine with
--headlessset onNODE_RANK > 0, so only one real api-server (on the head node) exists and it legitimately spans both nodes via TP. The DEP branch is the odd one out: it's structurally similar to the llm-d pattern (which requires an external router) but is deployed through a connector that was only ever built for the single-api-server case.Suggested fix: Either (a) drop
--data-parallel-hybrid-lband instead set--headlesson the non-zero node rank (mirroring the TP/TEP branches), so vLLM's default internal-LB rank-0 api-server fans requests out across all 16 DP ranks through one endpoint, or (b) if hybrid-lb is intentionally required for this topology, extend the connector to run/route through both nodes' endpoints (analogous to llm-d's EPP+Envoy) rather than sending all client traffic to a singlelocalhostURL.
|
Claude finished @cquil11's task in 6m 59s —— View job Re-review: PR #2353 (restructured — connector removed, lifecycle folded into the H200 runner)
Re-review result: 1 new blocking issue, 1 carried-over blocking issue, 1 carried-over warning (see inline comments).
Still-applicable minor notes from the previous review (not re-posted inline): the between-point drain check in Checks that passed in the new structure: |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30316341044 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30316777432 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30318538587 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30319463051 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30342468122 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30609059800 |
Align H200 Kimi-K3 AgentX with the AIPerf revision shared by the B200 and B300 Qwen AgentX recipes. 中文：升级 AIPerf 基准测试框架，使 H200 Kimi-K3 AgentX 与 B200、B300 Qwen AgentX 配方使用同一版本。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30610938350 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30666713202 |
Align H200 Kimi-K3 AgentX with the barrier-aware AIPerf revision validated by the latest B300 Qwen AgentX work. 中文：升级 AIPerf 基准测试框架，使 H200 Kimi-K3 AgentX 与最新 B300 Qwen AgentX 验证的屏障感知版本保持一致。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30758322161 |
|
/stage-results 30758322161 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30758322161 |
|
/stage-results 30758322161 |
|
@cquil11 staged run 30758322161: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-02~r30758322161 This run remains available across future @cquil11 已将运行 30758322161 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-02~r30758322161 后续的 |
Merge current workflow, matrix, and AIPerf updates from origin/main. Resolve the AIPerf gitlink to 818c3a5a, which contains the H200 timing and barrier-watchdog fixes. 中文：将 origin/main 的最新工作流、矩阵逻辑和 AIPerf 更新合并到 H200 Kimi K3 分支。AIPerf 子模块统一到 818c3a5a，该版本包含 H200 所需的时序修复和屏障看门狗修复。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30758322161 |
Rely on the shared origin/main default and command construction so AgentX replay receives the trace idle-gap cap exactly once. 中文：复用 origin/main 的统一默认值和命令拼装逻辑，确保 AgentX 重放只传入一次轨迹空闲间隔上限参数。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30781144357 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30781313910 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30781313910 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30781313910 |
|
/stage-results 30781313910 |
|
@cquil11 staged run 30781313910: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-03~r30781313910 This run remains available across future @cquil11 已将运行 30781313910 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-03~r30781313910 后续的 |
cquil11
changed the title
|
/stage-results 30781313910 |
|
@cquil11 staged run 30781313910: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-03~r30781313910 This run remains available across future @cquil11 已将运行 30781313910 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-03~r30781313910 后续的 |
|
/stage-results 30781313910 |
|
@cquil11 staged run 30781313910: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-03~r30781313910 This run remains available across future @cquil11 已将运行 30781313910 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-03~r30781313910 后续的 |
|
/reuse-sweep-run 30781313910 |
Keep current main, reapply the Kimi K3 H200 configuration, and append the PR changelog entry at the latest tail.\n\n中文：保留当前 main，重新应用 Kimi K3 H200 配置，并将本 PR 的性能变更日志条目追加到最新文件末尾。
| max-num-seqs: 5 | ||
| max-num-batched-tokens: 4096 | ||
| speculative-config: '{"method":"dspark","model":"Inferact/Kimi-K3-DSpark","num_speculative_tokens":2,"draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":2.51}' | ||
| enforce-eager: true |
3 participants
Summary
--max-model-lenoverride or vLLM source patch.multinode_server_logsartifact on success, failure, or cancellation.Validation