[Klaud Cold][agentic experiment][Variant J] Kimi-K3 B200 agg TP8xPP2 agentic — DSpark @ golden AL + prefix-cache retention 0 / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——DSpark 黄金 AL + 前缀缓存留存 0 - #2379
Conversation
…ecipe Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (no expert parallelism) for the agentic-coding trace replay. Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1, fastsafetensors load format, kimi_k3 tool-call/reasoning parsers. Model pre-staged at /lustre/fsw/models/Kimi-K3; launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, the agentic recipe overlay, and the agentic cache default_mounts used by the GB200/GB300 agentic paths. 中文：新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试配方 （2 节点 / 16 GPU，纯 TP，不启用专家并行(EP)）。使用专用 bring-up 镜像 vllm/vllm-openai:kimi-k3（VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1、 fastsafetensors 加载格式、kimi_k3 工具调用/推理解析器）。模型已预置于 /lustre/fsw/models/Kimi-K3；启动器 launch_b200-dgxc.sh 增加 kimik3/fp4 模型路径映射、智能体配方覆盖及智能体缓存挂载。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：在更新日志条目与 MODELS 表格行中补充 PR #2355 链接。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oke test The cquil11/srt-slurm-nv cam/sa-submission-q2-2026 fork rejected the recipe (benchmark.aiperf_server_metrics: Unknown field). Switch the b200-dgxc agentic clone to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341), drop the aiperf_server_metrics field, pin dynamo wheel/router to 1.2.1 (the combination validated with v1.0.36), and reduce the bring-up to a single conc-8 smoke test. 中文：cquil11/srt-slurm-nv 分支的 srtctl 校验拒绝了配方字段 benchmark.aiperf_server_metrics（Unknown field）。将 b200-dgxc 智能体路径改用 上游 NVIDIA/srt-slurm v1.0.36（已在 #2302/#2341 验证），移除该字段，dynamo wheel/router 固定为 1.2.1，并将 bring-up 缩减为单并发（conc 8）冒烟测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dynamo-vllm worker entrypoint rejected --enable-auto-tool-choice --tool-call-parser kimi_k3 (unrecognized arguments; different arg parser than vllm serve). Chat parsing happens at the dynamo frontend — same convention as the DSv4 GB300 agentic recipes. Keep --reasoning-parser kimi_k3 (accepted by the worker). Also drop the explicit max-model-len and let vLLM derive the native 1M window from the model config, mirroring the agentic recipe convention. 中文：dynamo-vllm worker 入口不接受 --enable-auto-tool-choice 与 --tool-call-parser kimi_k3（unrecognized arguments，与 vllm serve 的参数解析器 不同），聊天解析由 dynamo 前端处理，与 DSv4 GB300 智能体配方约定一致；保留 worker 可接受的 --reasoning-parser kimi_k3。同时移除显式 max-model-len， 由 vLLM 从模型配置推导原生 1M 上下文窗口。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third sweep attempt: the engine loaded and served (TP8xPP2 healthy in ~14 min), but dynamo 1.2.1's rust frontend tokenizer rejects Kimi-K3's tiktoken model_type 'kimi_k3' (supported: kimi, kimi_k2, kimi_k25, deepseek_v3), so the model never registered and all chat completions returned 404, aborting the AgentX warmup. Switch to the 1.2.0.dev20260426 wheel used by the DSv4 GB300/B200 Dynamo-vLLM recipes. Upstream published v1.4.0-kimi-k3-dev.1 (2026-07-27) as the day-zero K3 build if this wheel also lacks support. 中文：第三次扫描中引擎已成功加载并提供服务（TP8xPP2 约 14 分钟就绪），但 dynamo 1.2.1 的 rust 前端分词器不支持 Kimi-K3 的 tiktoken model_type 'kimi_k3'，模型未能注册，所有请求返回 404，AgentX 预热中止。改用 DSv4 GB300/B200 Dynamo-vLLM 配方所用的 1.2.0.dev20260426 wheel；如仍不支持， 上游已于 2026-07-27 发布 day-zero 构建 v1.4.0-kimi-k3-dev.1。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin dynamo to ba83080ecd31c1ce918559e576d3c5bc9e092ff1 ("feat: Added
support for Kimi-K3", tag v1.4.0-kimi-k3-dev.1) via srt-slurm's
hash-cached source install: it adds the kimi_k3 tiktoken tokenizer to the
rust frontend (dynamo <=1.2.1 404s every request because the model never
registers) and accepts the kimi_k3 tool-call/reasoning parser worker args,
so restore --enable-auto-tool-choice --tool-call-parser kimi_k3
--reasoning-parser kimi_k3.
中文：将 dynamo 固定到 day-zero Kimi-K3 提交 ba83080（"feat: Added support
for Kimi-K3"，标签 v1.4.0-kimi-k3-dev.1），通过 srt-slurm 的哈希缓存源码
安装：该提交为 rust 前端新增 kimi_k3 tiktoken 分词器（dynamo <=1.2.1 因模型
无法注册而全部返回 404），worker 亦支持 kimi_k3 解析器参数，故恢复
--enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the vLLM OpenAI-frontend spellings (--enable-auto-tool-choice / --tool-call-parser) with dynamo's namespaced worker args: --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3. 中文：将 vLLM OpenAI 前端风格参数（--enable-auto-tool-choice / --tool-call-parser）替换为 dynamo 命名空间的 worker 参数： --dyn-tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --dyn-reasoning-parser kimi_k3。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fifth sweep attempt: the day-zero dynamo registered the kimi_k3 tiktoken tokenizer and the engine served, but all warmup requests got 400 — aiperf's conv-aware routing emits nvext.session_control, a removed POC field this dynamo build rejects (schema moved to router/routing_constraints/ agent_hints). Opt out via AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, matching the GB300 aggregate AgentX recipes; a single aggregate worker has no P/D routing to bind anyway. 中文：第五次扫描中 day-zero dynamo 已成功注册 kimi_k3 tiktoken 分词器并正常 服务，但全部预热请求返回 400——aiperf 的会话感知路由会发送 nvext.session_control（已被移除的 POC 字段，schema 已迁移至 router/routing_constraints/agent_hints）。通过 AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0 关闭，与 GB300 聚合式 AgentX 配方 一致；单聚合 worker 本无需 P/D 路由绑定。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sixth sweep attempt (both A and C variants): warmup requests 500 then the model 503s — the image's first decode step crashes in the KDA hybrid-state postprocess (mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index; torch requires an int64 index but the runner passes the int32 idx_mapping). Ship an in-container patch through srt-slurm's setup_script hook (same pattern as configs/patches/ vllm_numa_bind_hash_fix.py): coerce the index with .long(), idempotent, refuses to run if the image layout changed. 中文：第六次扫描（A、C 两个变体一致）：预热请求先 500、随后模型 503——镜像 首个解码步在 KDA 混合状态后处理中崩溃（mamba_hybrid.py postprocess_state， IndexError: index_fill_() 需要 int64 索引，但 runner 传入 int32 idx_mapping）。 通过 srt-slurm 的 setup_script 钩子在容器内打补丁：将索引用 .long() 转换， 幂等，且镜像布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Serve Kimi-K3 directly with vllm serve (srt-slurm PR #278 frontend.type: vllm, branch kylliang/direct-aggregate-vllm): no dynamo frontend/worker/ router, which removes the dynamo tokenizer/schema gaps entirely, and the OpenAI-frontend flags --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 become legitimate. PR #278 validates single-node only, so ship patches/srt-slurm-pr278-direct-vllm-multinode.patch extending it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology. Keeps the mamba_hybrid index-dtype container patch (engine bug is frontend-agnostic). 中文：智能体实验变体 D——通过 srt-slurm PR #278（frontend.type: vllm）直接以 vllm serve 提供服务：去除 dynamo 前端/worker/router，从根本上规避 dynamo 的 分词器与 schema 兼容问题，OpenAI 前端参数 --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 因此可用。PR #278 仅 支持单节点，故新增补丁将其扩展为 vLLM 原生多节点 serve（--master-addr/ --nnodes/--node-rank，非主节点 headless），以运行 2 节点 TP8xPP2 拓扑。保留 mamba_hybrid 索引类型容器补丁（引擎缺陷与前端无关）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2359。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same engine-level OOM as the dynamo-frontend variants: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's memory pool on the first forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free. 中文：与 dynamo 前端变体相同的引擎级 OOM：flashinfer trtllm MXFP4 MoE 内核在 首个前向时于 vLLM 显存池外分配约 1.6 GiB 工作区，0.95 下仅剩约 1.35 GiB。 改为 0.90。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Inherited from the closed dynamo-frontend variants (#2355/#2358): at gpu-mem-util 0.90 the first long-context MLA prefill OOM'd on a 2.92 GiB transient while 3.39 GiB sat reserved-but-unallocated (fragmentation). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and drop NCCL_CUMEM_ENABLE. 中文：继承自已关闭的 dynamo 前端变体（#2355/#2358）：0.90 显存利用率下首个 长上下文 MLA 预填充因 2.92 GiB 瞬时分配 OOM，而 3.39 GiB 处于已保留未分配 状态（碎片化）。设置 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 并 移除 NCCL_CUMEM_ENABLE。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…refill OOM)" This reverts commit 4370988. The superseded direct-vllm run served the agentic benchmark for 24 minutes on the original env (NCCL_CUMEM_ENABLE=1, no expandable_segments) without any OOM — the allocator change was precautionary carryover from the closed dynamo-frontend variants and was never justified by evidence from this serving path. Restore the env that was demonstrably running. 中文：回滚 4370988。被中断的 direct-vllm 运行在原始环境 （NCCL_CUMEM_ENABLE=1、未设 expandable_segments）下已稳定运行智能体基准测试 24 分钟且无 OOM——该分配器改动只是从已关闭的 dynamo 前端变体沿袭的预防性 措施，并无本服务路径上的证据支持。恢复已被验证可运行的环境。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep prefix-cache blocks alive across agentic turn gaps, matching the GB200/GB300 AgentX recipes. 中文：新增 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768，使前缀缓存块在智能体 回合间隔内保持留存，与 GB200/GB300 AgentX 配方一致。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
中文：将智能体并发列表从单点 8 扩展为 1/8/16/32。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Engine init hard-fails on Kimi-K3 with the GB200/GB300 AgentX value: "VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative and a multiple of scheduler_block_size (3145728)" — the KDA hybrid architecture gives K3 a 3.1M-token scheduler block. Default retention served fine in the earlier runs, so drop the override. 中文：移除 VLLM_PREFIX_CACHE_RETENTION_INTERVAL——Kimi-K3 的 KDA 混合架构使 scheduler_block_size 达 3145728，GB200/GB300 AgentX 的 32768 取值导致引擎 初始化直接失败（必须为其整数倍）。此前运行证明默认留存策略可正常服务， 故不再覆盖。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Variant D (#2359) plus DSpark speculative decoding with the Inferact/Kimi-K3-DSpark draft head: num_speculative_tokens 7, FLASHINFER_MLA attention backend, probabilistic draft sampling + block rejection sampling (the #2366 acceptance-rate optimization), and synthetic_acceptance_length 3.84 — the committed golden AL for exactly this sampling config at k=7 (#2368, measured in run 30316471205). Conc curve widened to 1/2/4/8/16/32; spec-decoding labeled draft_model. 中文：智能体实验变体 I——变体 D（#2359）加 DSpark 投机解码 （Inferact/Kimi-K3-DSpark 草稿头）：num_speculative_tokens 7、 FLASHINFER_MLA 注意力后端、probabilistic 草稿采样 + block 拒绝采样 （#2366 的接受率优化），synthetic_acceptance_length 3.84 即该采样配置在 k=7 下的黄金 AL（#2368，run 30316471205 测得）。并发曲线扩展为 1/2/4/8/16/32；spec-decoding 标注为 draft_model。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ntic-direct-vllm-dspark # Conflicts: # perf-changelog.yaml
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2376。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the spec-decoding schema Literal with "dspark" (alongside mtp/draft_model/none) and label the variant I entry dspark instead of draft_model. Downstream consumers (workflow env, result filenames, process_result) treat the value as an opaque string, so results carry spec-dspark. 中文：为 spec-decoding 的 schema Literal 新增 "dspark"（与 mtp/draft_model/none 并列），并将变体 I 的条目由 draft_model 改标为 dspark。下游（工作流环境变量、结果文件名、process_result）将该值视为 不透明字符串，结果将带有 spec-dspark 标注。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename agg-b200-tp8pp2-agentic.yaml to agg-b200-tp8pp2-dspark-agentic.yaml on this branch, matching the <topology>-<spec-method>-agentic naming of the DSv4 MTP recipes, and update the CONFIG_FILE routing and changelog. Also add method context: DSpark is DeepSeek's confidence-scheduled semi-autoregressive speculative decoding (arXiv 2607.05147), reported ~27-31% higher accepted length than EAGLE3 and 60-85% faster per-user generation than MTP-1 at matched throughput. 中文：将变体 I 的配方文件更名为 agg-b200-tp8pp2-dspark-agentic.yaml，与 DSv4 MTP 配方的「拓扑-投机方法-agentic」命名一致，并同步更新 CONFIG_FILE 路由与更新日志。补充方法背景：DSpark 为 DeepSeek 的置信度调度半自回归投机 解码框架（arXiv 2607.05147），报告显示接受长度较 EAGLE3 高约 27-31%，同等 吞吐下单用户生成速度较 MTP-1 快 60-85%。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ntic-direct-vllm-dspark # Conflicts: # perf-changelog.yaml
Identical DSpark-at-golden-AL config to variant I (#2376) plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0 (the only setting below one 3.1M-token KDA-hybrid scheduler block; positive values must be multiples of scheduler_block_size 3145728). Recipe named agg-b200-tp8pp2-dspark-retention0-agentic.yaml per the topology-spec-method naming convention. 中文：智能体实验变体 J——与变体 I（#2376）完全相同的 DSpark 黄金 AL 配置， 另加 VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0（唯一低于一个 3.1M token KDA 混合调度块的取值；正值必须为 scheduler_block_size 3145728 的整数倍）。 配方按「拓扑-投机方法」命名规范命名。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将更新日志条目与 MODELS 表格行链接指向实验 PR #2379。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same change as variant I (#2376): rejection_sample_method=synthetic with synthetic_acceptance_length 3.84, replacing the probabilistic+block serving combo the golden AL was measured under. 中文：与变体 I（#2376）相同的修改：rejection_sample_method=synthetic 并注入 synthetic_acceptance_length 3.84，替换黄金 AL 测量时所用的 probabilistic+block 服务组合。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30333909251 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Agentic experiment Variant I (of the #2359 direct-vllm Variant D): DSpark speculative decoding with the Inferact/Kimi-K3-DSpark draft head — num_speculative_tokens 7, FLASHINFER_MLA attention backend, probabilistic draft sampling + block rejection sampling (the #2366 acceptance-rate optimization), synthetic_acceptance_length 3.84 = the committed golden AL for this exact sampling config at k=7 (#2368's golden_al_distribution curve, measured in run 30316471205). Conc curve widened to 1/2/4/8/16/32; spec-decoding labeled dspark (new schema value alongside mtp/draft_model)" | ||
| - "Agentic experiment Variant J (of the #2376 DSpark Variant I): identical DSpark-at-golden-AL config plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0 (the only setting below one 3.1M-token KDA-hybrid scheduler block; positive values must be multiples of scheduler_block_size 3145728)" | ||
| - "Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-dspark-retention0-agentic.yaml on the cluster:b200-dgxc pool" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2379 |
There was a problem hiding this comment.
🟡 The changelog entry for this PR's model (kimik3-fp4-b200-dynamo-vllm-agentic) ends with a literal unfilled placeholder pr-link: .../pull/XXX instead of this PR's actual number (2379).
Extended reasoning...
What's wrong
perf-changelog.yaml line 5156, the final line of the changelog entry this PR adds, reads:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXXXX is a literal, unfilled placeholder — not PR #2379's actual number. This entry accumulates the description bullets for Variant D through Variant J (all under the single kimik3-fp4-b200-dynamo-vllm-agentic config key), and it appears this PR reused/appended to Variant I's (#2376) entry but replaced the real pull/2376 link with pull/XXX rather than updating it to pull/2379.
Why validation doesn't catch it
ChangelogEntry.pr_link in utils/matrix_logic/validation.py is declared as a bare str = Field(alias="pr-link") with no URL pattern, PR-number, or even non-placeholder-string validation. Any string, including "pull/XXX", passes schema validation silently.
Impact
Every other pr-link entry among the 642 entries in this file carries a real PR number pointing at a resolvable GitHub PR. This one entry is the sole exception, and following the link 404s. This breaks changelog → PR traceability for anyone (human or tooling) trying to trace this benchmark config change back to its originating PR — for a file whose entire purpose is that traceability. It has no runtime/CI effect (the field is not consumed for control flow, just for lookup/documentation), so it does not block the sweep or the merge itself.
Proof
- Read
perf-changelog.yaml:5156directly in the working tree:pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX. git blame -L 5156,5156 perf-changelog.yamlattributes this exact line to commite1af792("feat: agentic experiment J — DSpark variant I + prefix-cache retention 0"), which is this PR's own head commit — so the placeholder is what actually lands onmainif merged as-is, not an artifact of a later commit.git show e1af792 -- perf-changelog.yamlshows the diff hunk removingpr-link: .../pull/2376(Variant I's real link) and addingpr-link: .../pull/XXX— confirming the author appended the Variant J bullet but botched the link update.- Compare to any other entry in the file, e.g. line 5130 (
pull/2310) — every other entry has a concrete, resolvable PR number.
Fix
Replace pull/XXX with pull/2379 on line 5156.
|
|
||
| # Kimi-K3 MXFP4 B200 aggregated vLLM via Dynamo (TP8 x PP2, 2 nodes / 16 | ||
| # GPUs), agentic bring-up. The native MXFP4 checkpoint (2.8T total params, | ||
| # ~1.4TB weights) does not fit one 8xB200 node, so TP8 shards attention/dense | ||
| # and PP2 splits layers. Plain TP (NOT TEP): ep 1, no expert parallelism — | ||
| # the 896 routed experts are TP-sharded within each pipeline stage. Node | ||
| # count = tp*pp/gpus_per_node = 8*2/8 = 2. Aggregated (prefill num-worker 1 + | ||
| # decode num-worker 0, RECIPES.md section 5) — the single worker serves both | ||
| # phases, so no P/D KV transfer. Dedicated kimi-k3 vLLM bring-up image with | ||
| # VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 and the kimi_k3 tool-call/reasoning | ||
| # parsers. | ||
| # Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml | ||
| kimik3-fp4-b200-dynamo-vllm-agentic: |
There was a problem hiding this comment.
🟡 The new comment block above kimik3-fp4-b200-dynamo-vllm-agentic (configs/nvidia-master.yaml:8360-8372) is stale: it describes a plain aggregated bring-up and its trailing # Recipe: line points to agg-b200-tp8pp2-agentic.yaml, a file that doesn't exist anywhere in the tree. The entry's actual CONFIG_FILE correctly points to agg-b200-tp8pp2-dspark-retention0-agentic.yaml, and the comment prose should mention the DSpark speculative decoding and retention-0 settings that actually define this variant.
Extended reasoning...
This is a comment-only documentation bug in configs/nvidia-master.yaml. The prose block added directly above the kimik3-fp4-b200-dynamo-vllm-agentic entry (lines 8360-8370) describes a plain aggregated TP8xPP2 bring-up with no mention of speculative decoding, and its final line (8371) reads:
# Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml
That file, agg-b200-tp8pp2-agentic.yaml, does not exist anywhere in the repository. The only recipe file this PR (and its predecessors D/F/G/I) ever added under benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/ is agg-b200-tp8pp2-dspark-retention0-agentic.yaml — which is exactly what the entry's own CONFIG_FILE additional-setting (line 8396) correctly references: recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-dspark-retention0-agentic.yaml.
Proof:
git ls-files benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/shows a single file:agg-b200-tp8pp2-dspark-retention0-agentic.yaml.- The comment's
# Recipe:line namesagg-b200-tp8pp2-agentic.yaml— no such file exists, confirmed by the same listing. - The entry's functional
additional-settings: CONFIG_FILE=...(line 8396) names the correct, existing file. - The entry itself sets
spec-decoding: dspark(line 8387), yet the prose comment above it never mentions DSpark speculative decoding or theVLLM_PREFIX_CACHE_RETENTION_INTERVAL: 0setting that is the defining delta of this variant (per the PR description, Variant J = Variant I + retention 0).
This text is copy-pasted from the original Variant D bring-up comment (the plain, non-speculative aggregated config) and was never updated as the entry evolved through Variants I and J. A developer or reviewer who trusts the # Recipe: comment and goes looking for agg-b200-tp8pp2-agentic.yaml will not find it, and the summary gives no indication that this config uses DSpark speculative decoding or retention-0 — the two things that actually differentiate it.
This is purely a comment/documentation issue: the functional CONFIG_FILE setting that the launcher actually consumes is correct, so nothing breaks at runtime, no wrong recipe gets loaded, and no benchmark job is affected. The fix is simple — update the # Recipe: line to point at agg-b200-tp8pp2-dspark-retention0-agentic.yaml and expand the prose to mention DSpark speculative decoding (Inferact/Kimi-K3-DSpark draft head, k=7) and the retention-0 setting, matching the level of detail already given in the recipe YAML's own comments and in perf-changelog.yaml.
| VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" | ||
| # Prefix-cache retention (variant J): 0, on the otherwise-unchanged | ||
| # DSpark variant I config. Any positive value must be a multiple of | ||
| # Kimi-K3's KDA-hybrid scheduler_block_size (3145728; the GB recipes' | ||
| # 32768 is hard-rejected at engine init — verified on this PR family), so | ||
| # 0 is the only value below one 3.1M-token scheduler block. | ||
| VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" | ||
| # No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value | ||
| # (32768) hard-fails engine init on Kimi-K3 — the KDA hybrid gives it a | ||
| # scheduler_block_size of 3145728 and the interval must be a multiple of | ||
| # it ("VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative | ||
| # and a multiple of scheduler_block_size (3145728)"). Default retention | ||
| # served fine in earlier runs. |
There was a problem hiding this comment.
🟡 Line 76 sets VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0", but the comment block immediately below (lines 77-82) begins "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ..." and explains why the var is deliberately omitted — this is leftover copy-paste from Variant I's recipe (which didn't set the var at all) and directly contradicts the setting one line above. It's comment-only (no runtime effect), but the stale block should be removed so a future editor doesn't mistakenly delete line 76 thinking the var should stay unset.
Extended reasoning...
What the bug is: In agg-b200-tp8pp2-dspark-retention0-agentic.yaml, the backend.aggregated_environment block sets VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" on line 76. That line is correctly preceded (lines 71-75) by a comment explaining the Variant J rationale: 0 is the only value below one 3.1M-token KDA-hybrid scheduler block, since any positive value must be a multiple of scheduler_block_size (3,145,728). Immediately after that setting, lines 77-82 contain a second comment block that opens with "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init..." and closes with "Default retention served fine in earlier runs."
How it manifests / code path: This second comment is leftover text carried over verbatim from Variant I's recipe (the predecessor PR #2376), which did not set this env var at all — hence the original "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL" phrasing made sense there. When Variant J (this PR) added the explicit "0" setting on line 76, the old comment explaining the absence of the setting was never removed or updated, so it now sits directly beneath the line it contradicts.
Why existing code doesn't prevent it: This is a pure YAML-comment issue — comments have no runtime effect, are not parsed or validated by utils/matrix_logic/validation.py or any CI check, and the engine still receives the correct explicit "0" value regardless of what the comment says. Nothing in the pipeline flags contradictory adjacent comments.
Impact: No functional impact on this sweep — the YAML value is what's used at runtime, not the comment. The risk is purely to future maintainers: someone skimming lines 77-82 could reasonably conclude the variable is intentionally left unset (matching Variant I's behavior) and delete or "fix" line 76 accordingly, which would change runtime behavior (0 vs. vLLM's default retention interval). It's a maintenance/readability hazard, not a merge-blocking defect.
Proof (step-by-step):
- Read lines 71-75: "Prefix-cache retention (variant J): 0, on the otherwise-unchanged DSpark variant I config... so 0 is the only value below one 3.1M-token scheduler block." → asserts the var IS set to 0.
- Read line 76:
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0"→ confirms the var IS set to 0. - Read lines 77-82: "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init... Default retention served fine in earlier runs." → asserts the var is NOT set (uses vLLM's default retention).
- Steps 1/2 and step 3 cannot both be true for the same recipe — one says explicit 0, the other says omitted/default. Diffing against Variant I's recipe ([Klaud Cold][agentic experiment][Variant I] Kimi-K3 B200 agg TP8xPP2 agentic — DSpark spec decoding @ golden AL / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——DSpark 投机解码（黄金 AL） #2376) confirms lines 77-82 are copy-pasted verbatim from a context where the var truly was omitted.
How to fix: Delete the stale lines 77-82 comment block entirely (its information is now redundant with the correct explanation on lines 71-75), or fold the "32768 hard-fails engine init" rationale into a single unified comment above line 76 explaining both why 0 was chosen and why the GB-style 32768 value can't be used.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30334286231 |
Engine init died with "NotImplementedError: Pipeline parallelism is not supported for this model" — SpeculativeConfig verifies the draft model against a parallel config that unconditionally inherits the target's pipeline_parallel_size (2), and the Inferact/Kimi-K3-DSpark draft head does not implement SupportsPP. At runtime V1 drafters load ONLY on the final pipeline stage (vllm-project/vllm#16568), effectively draft_pipeline_parallel_size=1, and upstream has no config knob for it — so extend the in-container patch script to verify the draft against a pp=1 view of the parallel config. Idempotent; refuses to patch if the call-site shape changed. 中文：引擎初始化报 "Pipeline parallelism is not supported for this model"—— SpeculativeConfig 用无条件继承目标 pipeline_parallel_size(2) 的并行配置校验 草稿模型，而 Inferact/Kimi-K3-DSpark 草稿头未实现 SupportsPP。V1 运行时草稿 模型仅加载在最后一个流水线阶段（vllm#16568），等效 draft_pipeline_parallel_size=1，且上游无相应配置项——故扩展容器内补丁脚本， 以 pp=1 视图校验草稿模型。补丁幂等，调用点形态变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The K3 DSpark head consumes aux hidden states from five target layers (target_layer_ids [2, 23, 47, 71, 89]); the target model captures them per PP stage, so under TP8xPP2 the layers-2/23 captures live on stage 0 and never reach the last-stage drafter, whose combine_hidden_states projection expects hidden_size x 5. This build's V2 runner already broadcasts sampled counts across PP ranks (PPHandler), so the remaining gap is the aux transport — the same gap vllm-ascend PR #12507 closes for EAGLE3. Extend the container patch script with three coordinated edits to kimi_k3/nvidia/model.py: preallocate aux_hidden_<layer> receive buffers for upstream-owned aux layers, attach/pass-through captured aux states on non-last ranks via IntermediateTensors, and merge received + local aux in global layer order on the last rank. Anchors verified against the public vllm kimi-k3 branch source; idempotent; refuses to patch on layout drift. 中文：K3 DSpark 草稿头消费五个目标层（target_layer_ids [2, 23, 47, 71, 89]） 的辅助隐状态；目标模型按流水线阶段各自捕获，TP8xPP2 下第 2/23 层的捕获位于 阶段 0，无法到达最后阶段的草稿模型（其 combine_hidden_states 期望 hidden_size x 5）。该镜像的 V2 runner 已通过 PPHandler 跨阶段广播采样计数， 唯一缺口即辅助隐状态传输（与 vllm-ascend PR #12507 为 EAGLE3 修复的相同）。 容器补丁脚本新增三处协同修改：接收侧预分配上游辅助层缓冲、非最后阶段经 IntermediateTensors 附带/透传捕获的辅助隐状态、最后阶段按全局层序合并。 锚点已对照公开 kimi-k3 分支源码验证；幂等；源码布局变化时拒绝执行。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30336667613 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30338554035 |
The runner hard-raises "dspark with pipeline parallel is not supported" for aux-hidden-state spec methods — the guard exists precisely because aux captures were stage-local, which the aux-transport patch now fixes (and PPHandler already broadcasts sampled counts across ranks in this build). Downgrade the ValueError to a warning via the container patch script. Anchor verified against the public kimi-k3 branch source. 中文：V2 runner 对依赖辅助隐状态的投机方法在流水线并行下硬性报错 （"dspark with pipeline parallel is not supported"）——该防护正是因辅助隐 状态仅限本阶段捕获而设，前一补丁已实现跨阶段传输（且该构建的 PPHandler 已 跨阶段广播采样计数）。通过容器补丁脚本将 ValueError 降级为警告。锚点已对照 公开 kimi-k3 分支源码验证。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Agentic experiment (Variant J) — Variant I (#2376) plus
VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0.Identical to Variant I in every other way (direct
vllm serve, GPU-resident KV, DSpark speculative decoding with theInferact/Kimi-K3-DSparkdraft head atnum_speculative_tokens=7, synthetic rejection sampling injectingsynthetic_acceptance_length: 3.84= the committed golden AL (measured under probabilistic draft + block rejection, #2366/#2368) at k=7, conc 1/2/4/8/16/32). The single delta:VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0— the only setting below one 3.1M-token scheduler block (Kimi-K3's KDA hybrid requires positive values to be multiples ofscheduler_block_size3,145,728; the GB recipes' 32768 is hard-rejected at engine init, verified on this PR family).Recipe:
agg-b200-tp8pp2-dspark-retention0-agentic.yamlper the topology–spec-method naming convention.Related experiments
中文说明
智能体实验变体 J——变体 I（#2376）加
VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0。除此单一变更外与变体 I 完全一致（直接 vllm serve、GPU 常驻 KV、DSpark 投机解码：Inferact/Kimi-K3-DSpark 草稿头、k=7、合成拒绝采样注入 synthetic_acceptance_length 3.84 黄金 AL（在 probabilistic+block 下测得，#2366/#2368）、并发 1/2/4/8/16/32）。0 是唯一低于一个 3.1M token 调度块的取值（Kimi-K3 KDA 混合架构要求正值为 scheduler_block_size 3,145,728 的整数倍；GB 配方的 32768 会被引擎初始化硬性拒绝，已在本 PR 系列验证）。配方按「拓扑-投机方法」规范命名为
agg-b200-tp8pp2-dspark-retention0-agentic.yaml。相关实验
变体 D（基础，全绿）#2359；变体 F（SimpleCPU 卸载 + 留存 0）#2372；变体 G（留存 0）#2374；变体 I（DSpark 黄金 AL）#2376；变体 J（本 PR，I + 留存 0）。I 与 J 对比在 DSpark 配置上隔离留存参数，与 D/G 对比在普通配置上的作用互为镜像。已关闭：A #2355、B #2357、C #2358、E #2370。
🤖 Generated with Claude Code