[Klaud Cold][agentic experiment][Variant C] Kimi-K3 B200 agg TP8xPP2 agentic — no parser flags / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——不带解析器参数 - #2358
Conversation
…ecipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方 （2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用 上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器 不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len， 由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持， 上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added
support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's
hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the
rust frontend (dynamo <=1.2.1 404s every request because the model never
registers) and accepts the kimi_k3 tool-call/reasoning parser worker args,
so restore --enable-auto-tool-choice --tool-call-parser kimi_k3
--reasoning-parser kimi_k3.
中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support
for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码
安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型
无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复
--enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Variant of the Kimi-K3 B200 agg TP8xPP2 agentic bring-up (#2355) with no tool-call/reasoning parser flags on the worker; chat parsing happens at the dynamo frontend, matching the parser-less DSv4 GB300 agentic recipes. Sibling experiments try dynamo's --dyn-* namespaced args and the plain vLLM spellings; whichever goes green first merges. 中文：Kimi-K3 B200 聚合式 TP8xPP2 智能体 bring-up（#2355）的解析器参数实验 变体 C：worker 不带任何解析器参数，聊天解析由 dynamo 前端处理，与不带解析器 的 DSv4 GB300 智能体配方一致。兄弟实验分别尝试 dynamo --dyn-* 命名空间参数 与 vLLM 原生写法；先通过者合并。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2358。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
functionstackx
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30299680128 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # phases, so no P/D KV transfer. Dedicated kimi-k3 vLLM bring-up image with | ||
| # VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 and the kimi_k3 tool-call/reasoning | ||
| # parsers. | ||
| # Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml |
There was a problem hiding this comment.
🟡 Documentation-only: the new header comment for kimik3-fp4-b200-dynamo-vllm-agentic in configs/nvidia-master.yaml (lines 8336-8339) states the setup uses 'the kimi_k3 tool-call/reasoning parsers,' but this is variant C, which deliberately passes NO parser flags to the worker (chat parsing happens at the dynamo frontend instead). The comment appears to be copied from the base variant-A PR (#2355) and contradicts both the recipe yaml's own comment and this PR's own perf-changelog entry, which correctly describe the no-parser-flags behavior.
Extended reasoning...
This PR adds a new header comment block to configs/nvidia-master.yaml (lines 8325-8339) documenting the new kimik3-fp4-b200-dynamo-vllm-agentic config entry. The final sentence of that comment reads: "Dedicated kimi-k3 vLLM bring-up image with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 and the kimi_k3 tool-call/reasoning parsers." The "with X and Y" phrasing pairs two things the recipe supposedly configures — and VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 genuinely is set in the recipe's aggregated_environment, so a reader naturally takes "the kimi_k3 tool-call/reasoning parsers" to also be something this recipe configures on the worker.
That is incorrect for this PR. This PR is explicitly variant C of a three-way parser-flag experiment (per its own title, "no parser flags", and its description: "no tool-call/reasoning parser flags are passed to the worker"). The actual recipe yaml this config points to, benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml (lines 84-87), spells this out unambiguously: "No parser flags on the worker (parser-flag experiment variant C — sibling PRs try dynamo's --dyn-* namespaced args and the plain vLLM spellings). Chat parsing happens at the dynamo frontend." The vllm_config.aggregated block in that same file contains no --tool-call-parser, --reasoning-parser, --dyn-tool-call-parser, or --dyn-reasoning-parser keys at all.
The stale comment almost certainly originates from the base bring-up PR, #2355 (variant A), which does pass --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3 to the worker — for that PR the "and the kimi_k3 tool-call/reasoning parsers" clause would have been accurate. When the comment block was copied over to build out this variant-C sibling PR, the parser-flag sentence was not updated to match variant C's defining behavior (passing none).
Tellingly, two other artifacts touched by this exact same PR were correctly updated to describe the no-parser-flags behavior: the recipe yaml comment quoted above, and the new perf-changelog.yaml entry for this same config key, which says "...--trust-remote-code, and NO parser flags on the worker (parser-flag experiment variant C; chat parsing happens at the dynamo frontend...)". So within this one PR, two of three descriptive artifacts say "no parser flags" and the third (the nvidia-master.yaml header) says the opposite.
Step-by-step proof:
- Read
configs/nvidia-master.yaml:8336-8338(added by this PR): "...Dedicated kimi-k3 vLLM bring-up image withVLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1and the kimi_k3 tool-call/reasoning parsers." → implies the worker runs the kimi_k3 parsers. - Read the recipe file it points to,
agg-b200-tp8pp2-agentic.yaml:84-87: "No parser flags on the worker (parser-flag experiment variant C ...)." → the worker runs no parsers. - Grep
vllm_config.aggregatedin that recipe file — there is notool-call-parserorreasoning-parserkey present, confirming point 2 is what actually gets passed to the worker at runtime. - Read the
perf-changelog.yamlentry added by this same PR for the identical config key — it explicitly says "NO parser flags on the worker (parser-flag experiment variant C...)", matching point 2 and contradicting point 1. - Conclusion: the nvidia-master.yaml comment is the one stale artifact in this PR; it was carried over from variant-A PR [Klaud Cold][agentic experiment][Variant A] feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic bring-up / 新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试（bring-up） #2355 without being updated for variant C.
Impact: this is comment-only — it has no effect on what actually gets executed (the recipe yaml, not the master-config comment, drives the worker's CLI args), so it does not change behavior or benchmark results. But it is misleading to anyone reading nvidia-master.yaml to understand what this config does, and it directly contradicts the recipe file and changelog entry sitting right next to it in the same PR.
Suggested fix: drop the "and the kimi_k3 tool-call/reasoning parsers" clause from the nvidia-master.yaml comment, or replace it with something like "and no parser flags on the worker (parser-flag experiment variant C; chat parsing happens at the dynamo frontend)" to match the recipe yaml and changelog wording.
Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常 服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方 一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30299727652 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30300716489 |
Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像 首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。 通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换， 幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30304451261 |
Seventh sweep attempt (mamba patch confirmed applied, model forward now executes): the first warmup forward OOMs in the flashinfer trtllm MXFP4 MoE kernel, which allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool — at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 0.90 matches the GB200/GB300 agentic recipes. 中文：第七次扫描（mamba 补丁已确认生效，模型前向已可执行）：首个预热前向在 flashinfer trtllm MXFP4 MoE 内核中 OOM——该内核在 vLLM 显存池之外分配约 1.6 GiB 运行时工作区，0.95 下 178 GiB B200 仅剩约 1.35 GiB 空闲。改为 0.90， 与 GB200/GB300 智能体配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30306484550 |
Eighth sweep attempt at gpu-mem-util 0.90 still OOM'd: a 2.92 GiB MLA long-context prefill transient (kv_b_proj in _compute_prefill_context) failed while 3.39 GiB sat reserved-but-unallocated — allocator fragmentation. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (what the torch OOM message recommends; the DSv4 recipes set it) and drop NCCL_CUMEM_ENABLE to trim NCCL's share of non-PyTorch device memory. 中文：第八次扫描在 0.90 显存利用率下仍 OOM：MLA 长上下文预填充的 2.92 GiB 瞬时分配（_compute_prefill_context 中的 kv_b_proj）失败，而 3.39 GiB 处于 已保留未分配状态——分配器碎片化。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True（torch OOM 报错所建议、 DSv4 配方亦采用），并移除 NCCL_CUMEM_ENABLE 以减少 NCCL 占用的非 PyTorch 显存。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Closing in favor of the direct-vLLM experiment (Variant D, #2359) — same rationale as #2355: the direct 中文：关闭本 PR，转向直接 vLLM 实验（变体 D，#2359）——理由同 #2355：直接 vllm serve 路径从根本上消除 Dynamo 兼容性问题，且变体 D 已继承此处验证的全部修复。进行中的扫描已取消。 |
Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个 长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配 状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并 移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30308358146 |
…n't work with Pipeline yet, offloading & TP16 and DEP8PP2 to be done in followup PR) (#2391) * feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic recipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方 （2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: link PR #2355 in changelog entry and MODELS rows 中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin agentic srt-slurm to NVIDIA v1.0.36, dynamo 1.2.1, conc-8 smoke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用 上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop OpenAI-frontend tool-choice flags from dynamo-vllm worker args The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器 不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len， 由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: try dynamo wheel 1.2.0.dev20260426 for Kimi-K3 frontend tokenizer Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持， 上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin dynamo to day-zero Kimi-K3 commit, restore kimi_k3 parser flags Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the rust frontend (dynamo <=1.2.1 404s every request because the model never registers) and accepts the kimi_k3 tool-call/reasoning parser worker args, so restore --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3. 中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码 安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型 无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use dynamo namespaced --dyn-* kimi_k3 parser args on the worker Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: disable aiperf conv-aware routing (session_control 400-rejected) Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常 服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方 一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: patch kimi-k3 image mamba_hybrid index_fill_ dtype via setup_script Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像 首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。 通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换， 幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: agentic experiment D — direct vllm serve via srt-slurm PR #278 Serve Kimi-K3 directly with vllm serve (srt-slurm PR #278 frontend.type: vllm, branch kylliang/direct-aggregate-vllm): no dynamo frontend/worker/ router, which removes the dynamo tokenizer/schema gaps entirely, and the OpenAI-frontend flags --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 become legitimate. PR #278 validates single-node only, so ship patches/srt-slurm-pr278-direct-vllm-multinode.patch extending it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology. Keeps the mamba_hybrid index-dtype container patch (engine bug is frontend-agnostic). 中文：智能体实验变体 D——通过 srt-slurm PR #278（frontend.type: vllm）直接以 vllm serve 提供服务：去除 dynamo 前端/worker/router，从根本上规避 dynamo 的 分词器与 schema 兼容问题，OpenAI 前端参数 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 因此可用。PR #278 仅 支持单节点，故新增补丁将其扩展为 vLLM 原生多节点 serve（--master-addr/ --nnodes/--node-rank，非主节点 headless），以运行 2 节点 TP8xPP2 拓扑。保留 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2359 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2359。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: drop gpu-memory-utilization to 0.90 (flashinfer MoE workspace OOM) Same engine-level OOM as the dynamo-frontend variants: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool on the first forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 中文：与 dynamo 前端变体相同的引擎级 OOM：flashinfer trtllm MXFP4 MoE 内核在 首个前向时于 vLLM 显存池外分配约 1.6 GiB 工作区，0.95 下仅剩约 1.35 GiB。 改为 0.90。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: expandable_segments allocator, drop NCCL_CUMEM_ENABLE (prefill OOM) Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个 长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配 状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并 移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "fix: expandable_segments allocator, drop NCCL_CUMEM_ENABLE (prefill OOM)" This reverts commit 4370988. The superseded direct-vllm run served the agentic benchmark for 24 minutes on the original env (NCCL_CUMEM_ENABLE=1, no expandable_segments) without any OOM — the allocator change was precautionary carryover from the closed dynamo-frontend variants and was never justified by evidence from this serving path. Restore the env that was demonstrably running. 中文：回滚 4370988。被中断的 direct-vllm 运行在原始环境 （NCCL_CUMEM_ENABLE=1、未设 expandable_segments）下已稳定运行智能体基准测试 24 分钟且无 OOM——该分配器改动只是从已关闭的 dynamo 前端变体沿袭的预防性 措施，并无本服务路径上的证据支持。恢复已被验证可运行的环境。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768 Keep prefix-cache blocks alive across agentic turn gaps, matching the GB200/GB300 AgentX recipes. 中文：新增 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768，使前缀缓存块在智能体 回合间隔内保持留存，与 GB200/GB300 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: widen agentic conc list to 1/8/16/32 中文：将智能体并发列表从单点 8 扩展为 1/8/16/32。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: remove VLLM_PREFIX_CACHE_RETENTION_INTERVAL (K3 scheduler block) Engine init hard-fails on Kimi-K3 with the GB200/GB300 AgentX value: "VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative and a multiple of scheduler_block_size (3145728)" — the KDA hybrid architecture gives K3 a 3.1M-token scheduler block. Default retention served fine in the earlier runs, so drop the override. 中文：移除 VLLM_PREFIX_CACHE_RETENTION_INTERVAL——Kimi-K3 的 KDA 混合架构使 scheduler_block_size 达 3145728，GB200/GB300 AgentX 的 32768 取值导致引擎 初始化直接失败（必须为其整数倍）。此前运行证明默认留存策略可正常服务， 故不再覆盖。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: agentic experiment G — variant D + prefix-cache retention 0 Identical GPU-resident direct-vllm config to variant D (#2359) plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0. Any positive value must be a multiple of Kimi-K3's KDA-hybrid scheduler_block_size (3145728), so 0 is the only setting below one 3.1M-token scheduler block. 中文：智能体实验变体 G——与变体 D（#2359）完全相同的 GPU 常驻直接 vllm serve 配置，另加 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0（任何正值都必须是 Kimi-K3 KDA 混合架构 scheduler_block_size 3145728 的整数倍，0 是唯一低于 一个 3.1M token 调度块的取值）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2374 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2374。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: variant G conc curve 1/2/4/8/16/32 (add 2 and 4) 中文：变体 G 并发曲线扩展为 1/2/4/8/16/32（新增 2 与 4）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * probe: variant K — drop the kimi-k3 in-container patch script Variant G (#2374, fully green) minus the kimi-k3-container-deps.sh in-container patch (setup_script reference, script file, and launcher copy), to verify whether the mamba_hybrid index_fill_ dtype patch is still required by the current vllm/vllm-openai:kimi-k3 image. Expected to fail at the first decode step if it is; results will be commented on the PR. 中文：探针变体 K——在全绿的变体 G（#2374）基础上移除 kimi-k3 容器内补丁 脚本（setup_script 引用、脚本文件与启动器复制），验证当前 vllm/vllm-openai:kimi-k3 镜像是否仍需 mamba_hybrid index_fill_ 类型补丁。 如仍需要，预计在首个解码步失败；结果将评论在 PR 中。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: point changelog and MODELS rows at experiment PR #2391 中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2391。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: clone fork branch with multinode support, drop git-apply patch The srt-slurm PR #278 multi-node extension now lives as commits on functionstackx/srt-slurm-nv branch klaud/direct-vllm-multinode (head df5baa93), so the launcher clones that branch directly instead of applying srt-slurm-pr278-direct-vllm-multinode.patch onto the upstream kylliang/direct-aggregate-vllm branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Parser-flag experiment (variant C) for the Kimi-K3 MXFP4 B200 aggregated TP8×PP2 agentic bring-up — no parser flags on the worker.
Identical to the base bring-up PR in every way (srt-slurm v1.0.36, dynamo hash-pinned to the day-zero Kimi-K3 commit
ba83080e/ tagv1.4.0-kimi-k3-dev.1,vllm/vllm-openai:kimi-k3, conc-8 smoke test) except no tool-call/reasoning parser flags are passed to the worker — chat parsing happens at the dynamo frontend, matching the parser-less DSv4 GB300 agentic recipes.Related experiments
Three sibling PRs race the same recipe with different worker parser flags; whichever goes green first merges and the others close:
--dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3— [Klaud Cold][agentic experiment][Variant A] feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic bring-up / 新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试（bring-up） #2355--tool-call-parser kimi_k3 --reasoning-parser kimi_k3— [Klaud Cold][agentic experiment][Variant B] Kimi-K3 B200 agg TP8xPP2 agentic — vLLM tool-call-parser flags / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——vLLM tool-call-parser 参数 #2357vllm serve(srt-slurm PR 278 + multinode patch), no Dynamo — [Klaud Cold][agentic experiment][Variant D] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve (srt-slurm PR 278) / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve（srt-slurm PR 278） #2359中文说明
Kimi-K3 MXFP4 B200 聚合式 TP8×PP2 智能体 bring-up 的解析器参数实验（变体 C）——worker 不带解析器参数。
除 worker 不传任何工具调用/推理解析器参数外与基础 PR 完全一致（srt-slurm v1.0.36、dynamo 固定到 day-zero Kimi-K3 提交
ba83080e/ 标签v1.4.0-kimi-k3-dev.1、vllm/vllm-openai:kimi-k3镜像、并发 8 冒烟测试）；聊天解析由 dynamo 前端处理，与不带解析器的 DSv4 GB300 智能体配方一致。相关实验
三个兄弟 PR 以不同的 worker 解析器参数并行竞跑同一配方，先通过者合并、其余关闭：变体 A（基础 PR，dynamo --dyn-* 命名空间参数）#2355；变体 B（vLLM 原生写法）#2357；变体 C（本 PR，不带解析器参数）；变体 D（直接 vllm serve，srt-slurm PR 278 + 多节点补丁）#2359。
🤖 Generated with Claude Code