Add DeepSeek-V4 GB300 Dynamo-vLLM AgentX MTP recipes / 新增 DeepSeek-V4 GB300 Dynamo-vLLM AgentX MTP 配置 by ivanium · Pull Request #2341 · SemiAnalysisAI/InferenceX

ivanium · 2026-07-26T20:39:42Z

Summary

Add dedicated GB300 Dynamo-vLLM AgentX MTP3 recipes for aggregate TP8 at concurrency 1, aggregate TP4 at concurrency 4, and two disaggregated P/D topologies at concurrency 128 and 384.
Use vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-426e59f for both aggregate and P/D recipes.
Pin GB300 vLLM agentic launches to NVIDIA/srt-slurm v1.0.36 (cb651682117df129162cd276cf567ec70dc57b36), the upstream version validated in [WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置 #2302.
Run the P/D recipes with random routing and 900-second session affinity, per-node data parallelism, and NIXL plus MooncakeStore KV transfer.
Keep asynchronous Mooncake decode loads enabled while disabling decode lookup, and update AIPerf to emit X-Dynamo-Session-ID and parent-session headers from stable correlation IDs.
Emit aggregate jobs through the multinode result schema so TP8 and TP4 are reported as 8 and 4 GPUs instead of being mislabeled as TP16/TP8.
This is the dedicated GB300 extraction of the work previously developed in [WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置 #2302.

Why

The GB300 AgentX MTP work needs an independently reviewable and runnable PR. The prior aggregate result metadata duplicated the co-located worker across prefill and decode during ingestion. The P/D recipes also need the validated random/session routing and upstream vLLM image. Pinning srt-slurm keeps all GB300 vLLM agentic sweeps reproducible while retaining the validated per-node DP, Dynamo health-count, multi-node port, Mooncake, and --no-preflight behavior.

Validation

Exact perf-changelog.yaml PR validator passed against current main.
Matrix logic: 224 tests passed.
Agentic aggregation: 31 tests passed.
Changelog processing: 13 tests passed.
Dynamo session-header tests: 2 passed.
srt-slurm dry-run passed for the aggregate TP4 recipe and representative P/D recipes.
runners/launch_gb300-nv.sh passes bash -n.
Confirmed NVIDIA/srt-slurm tag v1.0.36 resolves to cb651682117df129162cd276cf567ec70dc57b36.

中文说明

新增独立的 GB300 Dynamo-vLLM AgentX MTP3 配置：聚合式 TP8 并发 1、聚合式 TP4 并发 4，以及并发 128、384 的两组预填充/解码分离式拓扑。
聚合式和 P/D 配置统一使用 vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-426e59f 镜像。
GB300 vLLM agentic 任务固定使用 NVIDIA/srt-slurm v1.0.36（cb651682117df129162cd276cf567ec70dc57b36），即 [WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置 #2302 已验证的上游版本。
P/D 配置采用 900 秒会话亲和的随机路由、单节点数据并行，并通过 NIXL 与 MooncakeStore 传输 KV Cache。
解码侧保持 Mooncake 异步加载，同时关闭 lookup；AIPerf 根据稳定的 correlation ID 发送 X-Dynamo-Session-ID 和父会话请求头。
聚合式任务改用多节点结果结构，使 TP8 和 TP4 正确标记为 8 张和 4 张 GPU，避免被误标为 TP16/TP8。
这是此前在 [WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置 #2302 中开发的 GB300 工作拆分出的独立 PR，便于单独审阅和运行。

背景

GB300 AgentX MTP 工作需要一个可独立审阅和运行的 PR。此前聚合式结果在入库时会将同一个 worker 同时计入预填充和解码，导致 GPU 拓扑重复计算。P/D 配置还需要采用已验证的随机会话路由和上游 vLLM 镜像。固定 srt-slurm 版本可以保证所有 GB300 vLLM agentic 扫描可复现，同时保留已验证的单节点数据并行、Dynamo 健康检查计数、多节点端口、Mooncake 和 --no-preflight 行为。

验证

基于当前 main 运行的 perf-changelog.yaml PR 校验通过。
矩阵逻辑测试：224 项通过。
Agentic 聚合测试：31 项通过。
changelog 处理测试：13 项通过。
Dynamo 会话请求头测试：2 项通过。
聚合式 TP4 和代表性 P/D 配置均通过 srt-slurm dry-run。
runners/launch_gb300-nv.sh 通过 bash -n 语法检查。
已确认 NVIDIA/srt-slurm 标签 v1.0.36 指向 cb651682117df129162cd276cf567ec70dc57b36。

github-actions · 2026-07-26T20:39:50Z

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

Update the dedicated GB300 AgentX MTP changelog entry to reference its new draft pull request. 中文：将独立的 GB300 AgentX MTP changelog 条目更新为新建的草稿 PR #2341。

claude · 2026-07-26T22:40:14Z

Claude finished @ivanium's task in 8m 12s —— View job

Review of PR #2341 — GB300 Dynamo-vLLM AgentX MTP recipes

Gather context (AGENTS.md, diff vs origin/main)
Review new recipe YAMLs (agg TP8/TP4, 3× disagg P/D)
Review configs/nvidia-master.yaml + perf-changelog.yaml (pairing, ordering, image accessibility, model prefixes)
Review benchmarks/benchmark_lib.sh and runners/launch_gb300-nv.sh (enroot import pattern)
Check utils/aiperf submodule bump
Post inline comments + summary

Summary

1 warning, no blocking issues. The single inline comment: runners/launch_gb300-nv.sh switches the GB300 agentic catch-all path from a commit-pinned srt-slurm checkout to unpinned NVIDIA/srt-slurm:main, which affects all GB300 agentic jobs (including the pre-existing non-MTP recipes) and makes sweeps non-reproducible against upstream schema changes — please pin the commit validated in #2302.

Everything else checks out:

perf-changelog.yaml entry is correctly appended at the end, paired with the master-config change, and references this PR.
The image vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-426e59f is a public Docker Hub reference, and each recipe's model.container matches the master image: (required for the launcher's container-alias key).
Master-config topologies match the recipes exactly: agg TP8 = 1×8 GPUs, agg TP4 = 1×4, c128 = 1×DEP4 P + 1×DEP8 D, c384 = DEP8/DEP8, c256 = 2×DEP8 P + 1×DEP16 D.
The decode: num-worker: 0 aggregate-through-multinode-schema trick is supported: WorkerConfig.num_worker has no positive-integer constraint, and both process_result.py and utils/agentic/aggregation/process_agentic_result.py explicitly handle num_decode_gpu == 0 (zeroing decode TP/EP and using total GPUs as the throughput denominator), with an existing precedent in dsv4-fp4-gb300-dynamo-sglang-agentic-agg.
The AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING opt-out in benchmark_lib.sh is backward-compatible (defaults to on), and all five new recipes correctly opt out since dynamo 1.3.0.dev rejects nvext.session_control; the disagg recipes replace it with header-based session affinity (AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID + router-session-affinity-ttl-secs: 900).
The launcher retains the enroot import docker:// reproducibility pattern (import_squash).
synthetic_acceptance_length: 2.49 for MTP is the documented repo policy (golden_al_distribution/README.md), not a red flag.

Notes: I could not run the sweep generator or pytest in this environment (commands require approval), so I'm relying on the PR's stated test results plus manual schema tracing. I also could not verify that the utils/aiperf submodule commit ed2e040d is reachable on the remote — CI checkout will confirm that.

中文总结

1 个警告，无阻塞性问题。 唯一的行内评论：runners/launch_gb300-nv.sh 将 GB300 agentic 通用分支从固定 commit 改为跟踪未固定的 NVIDIA/srt-slurm:main，影响所有 GB300 agentic 任务并损害可复现性，建议固定到 #2302 中验证过的 commit。

其余检查全部通过：changelog 条目正确追加到文件末尾并与主配置变更配对；镜像为公开的 Docker Hub 引用且与配方的 model.container 一致；主配置拓扑与各配方完全匹配；decode: num-worker: 0 的聚合式多节点结果方案在 schema 和结果处理代码中均有支持且有既有先例；AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING 退出机制向后兼容；启动器保留了 enroot import 可复现性模式；MTP 的合成接受长度 2.49 符合仓库既定政策。因环境权限限制，未能实际运行 sweep 生成器和 pytest，也未能验证 aiperf 子模块 commit 在远端可达（CI checkout 会验证）。

claude

This PR is large and touches CODEOWNER-protected infra (configs/nvidia-master.yaml), switches the agentic srt-slurm checkout from a pinned fork/commit to the unpinned NVIDIA/srt-slurm:main branch, bumps the utils/aiperf submodule, and changes a default routing behavior in benchmark_lib.sh. No bugs were found by the bug hunting system, but given the scope and that this affects shared multi-node benchmark infra, it's worth a human look before merging.

Extended reasoning...

Overview

The PR adds five new GB300 Dynamo-vLLM AgentX MTP benchmark recipes (aggregate TP4/TP8 and three disaggregated P/D topologies), registers them in the CODEOWNER-protected configs/nvidia-master.yaml, adds a changelog entry, bumps the utils/aiperf submodule pin, and modifies runners/launch_gb300-nv.sh and benchmarks/benchmark_lib.sh.

Security risks

No injection/auth/secrets concerns — this is benchmark configuration and SLURM launcher shell script for an internal cluster. The launcher script does use sed/string interpolation on config paths and job names, but these come from trusted CI-controlled inputs, not external user input.

Level of scrutiny

Two things push this above a mechanical config PR that would normally be approvable: (1) configs/nvidia-master.yaml is explicitly CODEOWNER-protected, and (2) runners/launch_gb300-nv.sh changes which upstream repo/ref agentic jobs check out — moving from a pinned fork commit to NVIDIA/srt-slurm main (unpinned, will drift over time) — which affects the shared launcher path used by other agentic recipes, not just this PR's new ones. The benchmark_lib.sh change to conv-aware-routing defaults also affects the shared dynamo replay path used by other frameworks/recipes, guarded by an opt-out env var that these new recipes set to "0", but existing dynamo recipes that don't set it get the flag turned on by default now.

Other factors

The finder/verifier agents raised several kv_role/kv_connector questions across the disagg recipes and all were ruled out as intentional prefill-vs-decode role differences. The PR description cites passing unit test suites and dry-run validation for representative recipes, and this is described as a review-ready extraction of previously-developed work (#2302), which reduces risk, but the combination of an unpinned upstream branch switch and a CODEOWNER file touch is enough that I'm deferring rather than shadow-approving.

Update the dedicated GB300 AgentX MTP changelog entry to reference its new draft pull request. 中文：将独立的 GB300 AgentX MTP changelog 条目更新为新建的草稿 PR #2341。

github-actions · 2026-07-27T00:59:44Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30224312527
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30224312527

github-actions · 2026-07-27T01:27:10Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30228680732
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30228680732

Add three GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, and 1P/1D DEP8/DEP8. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Align both disaggregated recipes on Dynamo 1.3.0.dev20260720, preserve prefill-only connector prefetch and breakable CUDA graph settings, and configure decode-side MooncakeStore as a synchronous KV consumer with lookup disabled. 中文：新增三个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8 和 1P/1D DEP8/DEP8。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。两个分离式配置统一使用 Dynamo 1.3.0.dev20260720，连接器预取和可中断 CUDA Graph 设置仅保留在预填充侧；解码侧 MooncakeStore 设为同步 KV 消费端并关闭查找功能。

Set load_async=true for decode-side MooncakeStoreConnector while keeping asynchronous lookup and lookup disabled. This satisfies the current vLLM get_finished requirement in both disaggregated GB300 AgentX recipes. 中文：为解码侧 MooncakeStoreConnector 启用异步加载，同时保持异步查找和查找功能关闭。该修改使两个 GB300 AgentX 分离式推理配置满足当前 vLLM get_finished 的要求。

Raise the TP4 c4 scheduler and warmup limits to 16 sequences, extend MTP3 CUDA graph captures through 64 tokens, and lower GPU memory utilization to 0.93. 中文：扩大 TP4 c4 MTP 调度余量。将调度器和预热上限提高到 16 个序列，将 MTP3 CUDA Graph 捕获范围扩展到 64 个 token，并将 GPU 显存利用率下调至 0.93。

github-actions · 2026-07-27T01:40:37Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30229918070
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30229918070

Use random Dynamo session affinity and the upstream vLLM 426e59f image for aggregate and P/D recipes. Replace aggregate TP8 c4 with a B300-aligned TP4 c4 recipe and emit correct aggregate GPU metadata. Advance AIPerf for Dynamo session headers. 中文：优化 GB300 AgentX MTP 扫描拓扑。聚合式和 P/D 配置统一改用上游 vLLM 426e59f 镜像，P/D 配置启用 Dynamo 随机会话亲和路由；将聚合式 TP8 c4 替换为对齐 B300 参数的 TP4 c4，并修正聚合式结果的 GPU 拓扑标记；同时更新 AIPerf 以发送 Dynamo 会话请求头。

Update the dedicated GB300 AgentX MTP changelog entry to reference its new draft pull request. 中文：将独立的 GB300 AgentX MTP changelog 条目更新为新建的草稿 PR #2341。

github-actions · 2026-07-27T07:38:38Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30231719317
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30231719317

ivanium · 2026-07-27T08:15:27Z

vllm recipe: vllm-project/recipes#645

Oseltamivir · 2026-07-27T08:21:38Z

/reuse-sweep-run

xinli-sw · 2026-07-27T13:09:27Z

@cquil11 please review, thanks!

cquil11 · 2026-07-27T14:26:34Z

/stage-results

github-actions · 2026-07-27T14:26:45Z

@cquil11 staged run 30231719317: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-27~r30231719317

This shared staging slot remains available until the next /stage-results request. Staging workflow

cquil11 · 2026-07-27T14:42:04Z

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Additional detail section:

Signed head: afd24d5a3d6d3d1b49897bd772f465732fc0451e
PR validation and AgentX sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30231719317 — all four GB300 AgentX benchmark jobs passed.
Current-head changelog validation and reuse-sweep gate: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30275355057
Authorized reuse command: Add DeepSeek-V4 GB300 Dynamo-vLLM AgentX MTP recipes / 新增 DeepSeek-V4 GB300 Dynamo-vLLM AgentX MTP 配置 #2341 (comment)
Evals: N/A for this AgentX-only multi-node submission; no separate eval jobs ran. The four applicable AgentX benchmark jobs passed in run 30231719317.
Golden AL: https://github.com/SemiAnalysisAI/InferenceX/blob/main/golden_al_distribution/dsv4_mtp.yaml (thinking_on, 3 speculative tokens, AL 2.49)
Upstream vLLM recipe: Cover all three KV offload backends in DeepSeek-V4-Pro guide vllm-project/recipes#645 (merged)
No patch waiver is required; the upstream vLLM image is used as shipped.

Signed: cquil11

Klaud-Cold · 2026-07-27T14:47:54Z

✅✅✅ Verdict: PASS ✅✅✅

Verified independently at pinned head afd24d5 (PR tip unchanged).

✅ Check 0 (CODEOWNER): PASS — configs/nvidia-master.yaml is owned by @ankur-singh @kedarpotdar-nv @InferenceX/core; signer cquil11 is an org MEMBER signing via @InferenceX/core (team membership unreadable by bot token — not a failure); all other paths fall to the @InferenceX/core catch-all. No owner mismatch.
✅ Check 1 (passing sweep on in-PR commit): PASS — commit e1680ab8 (in this PR) ran all four multi-node agentic / benchmark jobs to success in run 30231719317; reuse-sweep-gate succeeded on the pinned head (run 30275355057).
➖ Check 2 (evals): N/A — multi-node agentic eval is unsupported by design (utils/matrix_logic/generate_sweep_configs.py, "Multi-node agentic eval is unsupported"), so no eval rows exist for these configs; the executed jobs used this PR's exact image (vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-426e59f), and accuracy control comes from the golden-AL simulation (Check 10). Sign-off documents this N/A.
➖ Check 3 (recipe link): N/A — exclusively multi-node/disagg submission (all recipes under benchmarks/multi_node/srt-slurm-recipes/**, both master entries multinode: true); recipe-link requirement covers single-node only. Sign-off nonetheless links vllm-project/recipes#645, which is MERGED (2026-07-16).
✅ Check 4 (reuse command): PASS — authorized /reuse-sweep-run by Oseltamivir (MEMBER).
✅ Check 5 (latest template): PASS — every item in the current docs/PR_REVIEW_CHECKLIST.md template is present and checked in the sign-off.
✅ Check 6 (upstream image / engine-first): PASS — both new entries use upstream vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-426e59f; they are vLLM-engine (dynamo-vllm) submissions, and vLLM-engine entries for dsv4 on GB300 already exist on main (dsv4-fp4-gb300-dynamo-vllm, dsv4-fp4-gb300-dynamo-vllm-agentic).
✅ Check 7 (no architecture hacks): PASS — no --hf-overrides/model-config edits; attention-config selects the FLASHINFER_MLA_SPARSE_DSV4 kernel backend for DSV4's native sparse attention; fp4/fp8 are precision choices.
✅ Check 8 (chat templates): PASS — agentic replay drives /v1/chat/completions with --endpoint-type chat (benchmarks/benchmark_lib.sh:1761).
✅ Check 9 (no engine patches): PASS — no patch/heredoc/site-packages modification; the runner change moves FROM a fork to upstream NVIDIA/srt-slurm v1.0.36 (orchestration, not engine); dynamo wheels are the declared framework component, same as existing dynamo-vllm entries.
✅ Check 10 (golden AL): PASS — all four configs pin "rejection_sample_method":"synthetic","synthetic_acceptance_length":2.49 with MTP num_speculative_tokens: 3, matching golden_al_distribution/dsv4_mtp.yaml (deepseek-v4-pro, thinking_on, 3 → 2.49); no synthetic knobs on non-agentic configs.

…oke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n't work with Pipeline yet, offloading & TP16 and DEP8PP2 to be done in followup PR) (#2391) * feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic recipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方（2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: link PR #2355 in changelog entry and MODELS rows 中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin agentic srt-slurm to NVIDIA v1.0.36, dynamo 1.2.1, conc-8 smoke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop OpenAI-frontend tool-choice flags from dynamo-vllm worker args The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len，由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: try dynamo wheel 1.2.0.dev20260426 for Kimi-K3 frontend tokenizer Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持，上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin dynamo to day-zero Kimi-K3 commit, restore kimi_k3 parser flags Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the rust frontend (dynamo <=1.2.1 404s every request because the model never registers) and accepts the kimi_k3 tool-call/reasoning parser worker args, so restore --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3. 中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use dynamo namespaced --dyn-* kimi_k3 parser args on the worker Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable aiperf conv-aware routing (session_control 400-rejected) Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: patch kimi-k3 image mamba_hybrid index_fill_ dtype via setup_script Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换，幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: agentic experiment D — direct vllm serve via srt-slurm PR #278 Serve Kimi-K3 directly with vllm serve (srt-slurm PR #278 frontend.type: vllm, branch kylliang/direct-aggregate-vllm): no dynamo frontend/worker/ router, which removes the dynamo tokenizer/schema gaps entirely, and the OpenAI-frontend flags --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 become legitimate. PR #278 validates single-node only, so ship patches/srt-slurm-pr278-direct-vllm-multinode.patch extending it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology. Keeps the mamba_hybrid index-dtype container patch (engine bug is frontend-agnostic). 中文：智能体实验变体 D——通过 srt-slurm PR #278（frontend.type: vllm）直接以 vllm serve 提供服务：去除 dynamo 前端/worker/router，从根本上规避 dynamo 的分词器与 schema 兼容问题，OpenAI 前端参数 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 因此可用。PR #278 仅支持单节点，故新增补丁将其扩展为 vLLM 原生多节点 serve（--master-addr/ --nnodes/--node-rank，非主节点 headless），以运行 2 节点 TP8xPP2 拓扑。保留 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2359 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2359。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop gpu-memory-utilization to 0.90 (flashinfer MoE workspace OOM) Same engine-level OOM as the dynamo-frontend variants: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool on the first forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 中文：与 dynamo 前端变体相同的引擎级 OOM：flashinfer trtllm MXFP4 MoE 内核在首个前向时于 vLLM 显存池外分配约 1.6 GiB 工作区，0.95 下仅剩约 1.35 GiB。改为 0.90。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: expandable_segments allocator, drop NCCL_CUMEM_ENABLE (prefill OOM) Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "fix: expandable_segments allocator, drop NCCL_CUMEM_ENABLE (prefill OOM)" This reverts commit 4370988. The superseded direct-vllm run served the agentic benchmark for 24 minutes on the original env (NCCL_CUMEM_ENABLE=1, no expandable_segments) without any OOM — the allocator change was precautionary carryover from the closed dynamo-frontend variants and was never justified by evidence from this serving path. Restore the env that was demonstrably running. 中文：回滚 4370988。被中断的 direct-vllm 运行在原始环境（NCCL_CUMEM_ENABLE=1、未设 expandable_segments）下已稳定运行智能体基准测试 24 分钟且无 OOM——该分配器改动只是从已关闭的 dynamo 前端变体沿袭的预防性措施，并无本服务路径上的证据支持。恢复已被验证可运行的环境。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768 Keep prefix-cache blocks alive across agentic turn gaps, matching the GB200/GB300 AgentX recipes. 中文：新增 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768，使前缀缓存块在智能体回合间隔内保持留存，与 GB200/GB300 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: widen agentic conc list to 1/8/16/32 中文：将智能体并发列表从单点 8 扩展为 1/8/16/32。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: remove VLLM_PREFIX_CACHE_RETENTION_INTERVAL (K3 scheduler block) Engine init hard-fails on Kimi-K3 with the GB200/GB300 AgentX value: "VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative and a multiple of scheduler_block_size (3145728)" — the KDA hybrid architecture gives K3 a 3.1M-token scheduler block. Default retention served fine in the earlier runs, so drop the override. 中文：移除 VLLM_PREFIX_CACHE_RETENTION_INTERVAL——Kimi-K3 的 KDA 混合架构使 scheduler_block_size 达 3145728，GB200/GB300 AgentX 的 32768 取值导致引擎初始化直接失败（必须为其整数倍）。此前运行证明默认留存策略可正常服务，故不再覆盖。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: agentic experiment G — variant D + prefix-cache retention 0 Identical GPU-resident direct-vllm config to variant D (#2359) plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0. Any positive value must be a multiple of Kimi-K3's KDA-hybrid scheduler_block_size (3145728), so 0 is the only setting below one 3.1M-token scheduler block. 中文：智能体实验变体 G——与变体 D（#2359）完全相同的 GPU 常驻直接 vllm serve 配置，另加 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0（任何正值都必须是 Kimi-K3 KDA 混合架构 scheduler_block_size 3145728 的整数倍，0 是唯一低于一个 3.1M token 调度块的取值）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2374 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2374。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: variant G conc curve 1/2/4/8/16/32 (add 2 and 4) 中文：变体 G 并发曲线扩展为 1/2/4/8/16/32（新增 2 与 4）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * probe: variant K — drop the kimi-k3 in-container patch script Variant G (#2374, fully green) minus the kimi-k3-container-deps.sh in-container patch (setup_script reference, script file, and launcher copy), to verify whether the mamba_hybrid index_fill_ dtype patch is still required by the current vllm/vllm-openai:kimi-k3 image. Expected to fail at the first decode step if it is; results will be commented on the PR. 中文：探针变体 K——在全绿的变体 G（#2374）基础上移除 kimi-k3 容器内补丁脚本（setup_script 引用、脚本文件与启动器复制），验证当前 vllm/vllm-openai:kimi-k3 镜像是否仍需 mamba_hybrid index_fill_ 类型补丁。如仍需要，预计在首个解码步失败；结果将评论在 PR 中。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2391 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2391。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: clone fork branch with multinode support, drop git-apply patch The srt-slurm PR #278 multi-node extension now lives as commits on functionstackx/srt-slurm-nv branch klaud/direct-vllm-multinode (head df5baa93), so the launcher clones that branch directly instead of applying srt-slurm-pr278-direct-vllm-multinode.patch onto the upstream kylliang/direct-aggregate-vllm branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

github-project-automation Bot added this to InferenceMAX Board Jul 26, 2026

ivanium force-pushed the agentx/gb300-pd-mtp branch from 227fe33 to 4efa36d Compare July 26, 2026 22:35

ivanium force-pushed the agentx/gb300-pd-mtp branch from 4efa36d to 4ba90d9 Compare July 26, 2026 22:37

ivanium marked this pull request as ready for review July 26, 2026 22:39

ivanium requested a review from a team July 26, 2026 22:39

ivanium requested review from Ankur-singh and kedarpotdar-nv as code owners July 26, 2026 22:39

claude Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread runners/launch_gb300-nv.sh Outdated

claude Bot reviewed Jul 26, 2026

View reviewed changes

ivanium force-pushed the agentx/gb300-pd-mtp branch from 4ba90d9 to 31895ae Compare July 26, 2026 22:54

ivanium added NVIDIA full-sweep-enabled agentx AgentX benchmarks, recipes, and infrastructure and removed full-sweep-enabled labels Jul 26, 2026

ivanium removed the full-sweep-enabled label Jul 27, 2026

ivanium added full-sweep-fail-fast full-sweep-enabled and removed full-sweep-enabled full-sweep-fail-fast labels Jul 27, 2026

ivanium added 2 commits July 27, 2026 01:39

ivanium added 2 commits July 27, 2026 01:57

chore(agentx): link GB300 MTP changelog to PR 2341

e1680ab

Update the dedicated GB300 AgentX MTP changelog entry to reference its new draft pull request. 中文：将独立的 GB300 AgentX MTP changelog 条目更新为新建的草稿 PR #2341。

ivanium force-pushed the agentx/gb300-pd-mtp branch from aeb8c8e to e1680ab Compare July 27, 2026 02:14

ivanium added the full-sweep-enabled label Jul 27, 2026

Merge branch 'main' into agentx/gb300-pd-mtp

0044b9d

cquil11 approved these changes Jul 27, 2026

View reviewed changes

Update perf-changelog.yaml

afd24d5

cquil11 approved these changes Jul 27, 2026

View reviewed changes

cquil11 merged commit e9597c7 into main Jul 27, 2026
27 checks passed

cquil11 deleted the agentx/gb300-pd-mtp branch July 27, 2026 14:45

github-project-automation Bot moved this to Done in InferenceMAX Board Jul 27, 2026

functionstackx mentioned this pull request Jul 27, 2026

[Klaud Cold][agentic experiment][Variant A] feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic bring-up / 新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试（bring-up） #2355

Closed

claude Bot mentioned this pull request Jul 28, 2026

[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: DEP tuning + LMCache refresh #2378

Closed

3 tasks

This was referenced Aug 7, 2026

perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness #2518

Closed

perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-disagg harness #2519

Closed

Uh oh!

Conversation

ivanium commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Validation

中文说明

背景

验证

Uh oh!

github-actions Bot commented Jul 26, 2026

Uh oh!

claude Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review of PR #2341 — GB300 Dynamo-vLLM AgentX MTP recipes

Summary

中文总结

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Overview

Security risks

Level of scrutiny

Other factors

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

ivanium commented Jul 27, 2026

Uh oh!

Oseltamivir commented Jul 27, 2026

Uh oh!

xinli-sw commented Jul 27, 2026

Uh oh!

cquil11 commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cquil11 commented Jul 27, 2026

Additional detail section:

Uh oh!

Uh oh!

Klaud-Cold commented Jul 27, 2026

✅✅✅ Verdict: PASS ✅✅✅

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ivanium commented Jul 26, 2026 •

edited

Loading

claude Bot commented Jul 26, 2026 •

edited

Loading

github-actions Bot commented Jul 27, 2026 •

edited

Loading