Kimi K3 B200 Day 0 AgentX Pipeline Parallelism (non-DSpark as it doesn't work with Pipeline yet, offloading & TP16 and DEP8PP2 to be done in followup PR)

[functionstackx]

Summary

Agentic experiment (Variant K) — a probe: Variant G (#2374) without the in-container patch script.

Removes benchmarks/multi_node/srt-slurm-recipes/configs/kimi-k3-container-deps.sh entirely (the recipe's setup_script reference, the script file, and the launcher copy step). Everything else is identical to the fully-green Variant G (direct multinode vllm serve, retention 0, conc 1/2/4/8/16/32).

srt-slurm clone source: the launcher now clones functionstackx/srt-slurm-nv klaud/direct-vllm-multinode, which carries srt-slurm PR NVIDIA/srt-slurm#278 (frontend.type: vllm) plus the multi-node extension as commits — replacing the previous clone of kylliang/direct-aggregate-vllm + git apply patches/srt-slurm-pr278-direct-vllm-multinode.patch (patch file deleted).

Purpose: verify empirically whether the mamba_hybrid index_fill_ dtype patch is still required by the current vllm/vllm-openai:kimi-k3 image. When the patch was introduced, the unpatched image crashed on the first decode step with IndexError: index_fill_(): Expected dtype int64 for index (KDA hybrid-state postprocess). If the image has since been fixed, this run goes green and the patch can be retired; if not, the failure will be documented in a comment below with the error logs.

Not intended to merge as-is — this PR exists to produce a definitive answer; the outcome will be commented here.

中文说明

智能体实验变体 K——探针：去除容器内补丁脚本的变体 G（#2374）。

完整移除 kimi-k3-container-deps.sh（配方的 setup_script 引用、脚本文件与启动器复制步骤），其余与全绿的变体 G 完全一致（直接多节点 vllm serve、留存 0、并发 1/2/4/8/16/32）。

srt-slurm 克隆来源：启动器现直接克隆 functionstackx/srt-slurm-nv klaud/direct-vllm-multinode 分支（已包含 srt-slurm PR NVIDIA/srt-slurm#278 及多节点扩展提交），取代此前"克隆 kylliang/direct-aggregate-vllm 后 git apply 补丁"的方式（补丁文件已删除）。

目的：实证验证当前 vllm/vllm-openai:kimi-k3 镜像是否仍需 mamba_hybrid index_fill_ 类型补丁。引入该补丁时，未打补丁的镜像会在首个解码步崩溃（IndexError: index_fill_(): Expected dtype int64 for index）。若镜像已修复，本次运行将通过、补丁可退役；若仍需要，失败详情与错误日志将评论在下方。

本 PR 不以合并为目的——仅为得出确定性结论，结果将在评论中给出。

🤖 Generated with Claude Code

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🟡 perf-changelog.yaml:5139 — The changelog bullet for kimik3-fp4-b200-dynamo-vllm-agentic (perf-changelog.yaml:5139) still says "conc 1/8/16/32", but the config it documents (configs/nvidia-master.yaml, same PR) actually sweeps conc-list: [1, 2, 4, 8, 16, 32]. Variant G added concurrency levels 2 and 4 to the config but the changelog prose was never updated to match, leaving a stale description of what was actually benchmarked.

  Extended reasoning...

  The bug: The newly-added changelog entry for kimik3-fp4-b200-dynamo-vllm-agentic in perf-changelog.yaml (line 5139, under the bullet describing max-num-batched-tokens 8192 / prefix-caching / max-model-len behavior) ends with the phrase conc 1/8/16/32. This is meant to summarize the concurrency sweep performed for this config. However, the actual shipped config for this exact config-key — configs/nvidia-master.yaml's kimik3-fp4-b200-dynamo-vllm-agentic.scenarios.agentic-coding.search-space[0].conc-list — is [1, 2, 4, 8, 16, 32], added in this same PR family.

  Root cause / how it got stale: This changelog entry has accreted bullets across several "agentic experiment variant" probes for the same config-key (Variant D → Variant G → Variant K, per the bullet history visible in the diff). Variant G (commit f14afdc, per the PR description: "conc 1/2/4/8/16/32") added concurrency levels 2 and 4 to nvidia-master.yaml's conc-list, and the PR description for this very probe (Variant K) explicitly states the sweep uses "conc 1/2/4/8/16/32". But the specific changelog bullet that recites the sweep parameters ("... max-num-batched-tokens 8192 ...; conc 1/8/16/32") was carried over verbatim from the earlier Variant D text and never updated when Variant G's config change landed.

  Why nothing catches this: The changelog is free-text YAML documentation with no schema or CI check tying its prose back to the actual conc-list values in configs/nvidia-master.yaml. Nothing enforces that the two stay in sync, so a partial edit (updating the config but not the accompanying changelog bullet) silently ships.

  Step-by-step proof:

  1. Open configs/nvidia-master.yaml in this PR's diff — the new kimik3-fp4-b200-dynamo-vllm-agentic entry has search-space[0].conc-list: [1, 2, 4, 8, 16, 32].
  2. Open perf-changelog.yaml in this PR's diff — the bullet ending "...max-num-batched-tokens 8192 so a single long prefill cannot OOM a pipeline stage; conc 1/8/16/32" (line 5139) omits 2 and 4.
  3. Cross-check against the PR's own title/description, which states the sweep is "conc 1/2/4/8/16/32" — confirming the config value is the intended/correct one and the changelog text is what's stale.
  4. A reader relying solely on the changelog (without cross-referencing the YAML config) would incorrectly conclude that concurrency levels 2 and 4 were never benchmarked for this config.

  Impact: Purely a documentation/prose inaccuracy — the changelog is descriptive text and has no effect on what the sweep actually runs (the real conc-list in nvidia-master.yaml drives execution). It does, however, misinform anyone reading the changelog about what was actually swept.

  Fix: Update the bullet at perf-changelog.yaml:5139 to read "conc 1/2/4/8/16/32" to match the shipped conc-list and the PR's own stated sweep range.

[claude]

🟡 The recipe sets health_check.max_attempts: 1440 to give the ~1.4TB MXFP4 checkpoint load extra headroom, but the pre-existing sed at runners/launch_b200-dgxc.sh:296 unconditionally rewrites any max_attempts: N line to 720, silently halving the recipe's intentional margin back to 7200s. This is a pre-existing launcher behavior (not introduced by this PR) that applies to every recipe, and since this config is otherwise identical to the fully-green Variant G (#2374) which ran under the same 720 override, the 1440 value appears to be dead/misleading rather than functionally necessary — worth reconciling but not blocking.

Extended reasoning...

The bug: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml sets:

health_check:
  interval_seconds: 10
  max_attempts: 1440

at 2-space indentation, intended to give the ~1.4TB MXFP4 Kimi-K3 checkpoint load (off shared Lustre) double the usual health-check window (14400s vs the launcher's usual 7200s).

The code path: runners/launch_b200-dgxc.sh (line ~296, inside the IS_MULTINODE branch that this recipe hits since multinode: true) runs, right before srtctl apply:

sed -i 's/^  max_attempts: [0-9]*/  max_attempts: 720/' "${CONFIG_FILE%%:*}"

The regex anchors on exactly two leading spaces before max_attempts:, which is precisely how the recipe indents it under health_check:. This is an unconditional overwrite, not a floor/max clamp — the accompanying comment ('Bump recipe health-check timeout from 360x10s=3600s to 720x10s=7200s ... so large-model loads finish in time') describes it as always raising from a smaller default, and doesn't anticipate a recipe deliberately setting a larger value. So the sed always forces the file to 720 regardless of what was there, silently overwriting the recipe's 1440 down to 720 before srtctl apply ever reads the config.

Why nothing in the codebase prevents this: there's no validation or diffing between the recipe's declared value and what the launcher writes, and the sed runs on every CONFIG_FILE unconditionally for every recipe launched through this script — this recipe has no special-casing to opt out.

Step-by-step proof:

1. Recipe file on disk contains max_attempts: 1440 (2-space indent, matches PR diff lines 32-35).
2. Launcher does sed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "${CONFIG_FILE%%:*}" on that same file.
3. The line max_attempts: 1440 matches ^ max_attempts: [0-9]* (2 leading spaces + literal text + digits), so sed rewrites it in place to max_attempts: 720.
4. srtctl apply -f "$CONFIG_FILE" is invoked immediately after, reading the now-mutated 720 value.
5. Result: the effective health-check window for this job is 720 * 10s = 7200s, not the 1440 * 10s = 14400s the recipe author intended.

Impact: if the 1.4TB MXFP4 weight load off shared Lustre ever exceeds 7200s, the health check would exhaust and kill the job before it ever reaches the decode step the probe is meant to test. However, the impact here appears to be nil in practice: (1) this sed is pre-existing launcher behavior, unmodified by this PR, and applies identically to every recipe that goes through this launcher; (2) per the PR description, this config is otherwise identical to the already-merged, fully-green Variant G (#2374), which ran successfully under the exact same 1440-in-recipe/720-effective override — proving the model load completes comfortably within 7200s and the extra 1440 margin was never actually exercised. So the recipe's 1440 is effectively a dead/misleading value the launcher always clobbers, rather than a value that changes observed behavior.

Fix: either make the launcher's sed only raise (not lower) the value, e.g. compare against the existing number and skip the rewrite if it's already >= 720, or drop the recipe's max_attempts override entirely (since it currently has no effect) and rely on the launcher's fixed 720, with a comment noting that. Given this is an explicitly not-to-merge probe PR whose entire purpose is a one-off empirical check, this is not blocking, but the author should reconcile the recipe's declared value with the launcher's clobbering behavior before this pattern is copied into a real (mergeable) recipe.

[claude]

🟡 The comment block right after VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" (lines 72-77) still opens with 'No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ...', asserting the key is absent — directly contradicting the line right above it that sets it. This is leftover from before Variant G re-added the key set to 0; it's purely a stale comment with no runtime effect, but worth deleting so it doesn't mislead future readers.

Extended reasoning...

In benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml, the aggregated_environment block contains two comment blocks about the same environment variable that now contradict each other. Lines 66-70 correctly explain the current behavior: VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" is set (Variant G), and the comment explains why 0 was chosen (any positive value must be a multiple of the KDA-hybrid scheduler_block_size of 3145728, so 0 is the only value below one full scheduler block). Line 71 sets the key to "0".

Immediately after that, lines 72-77 open with "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init..." — i.e., a comment asserting the key is absent from the config and that default retention is used instead. That is flatly false for this file: the key is set, right above, to "0".

How this happened: per the PR history (perf-changelog.yaml), this recipe evolved through several variants. In an earlier revision (Variant D era), VLLM_PREFIX_CACHE_RETENTION_INTERVAL was removed entirely, and the "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ..." comment was written at that time to explain the omission. Variant G later re-added the key set to "0" along with its own new comment (lines 66-70), but the old "No ..." comment block was never deleted — it's a leftover duplicate that now describes a state that no longer exists in the file.

Step-by-step proof:

1. Read line 71: VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" — the key IS present, with value 0.
2. Read the very next comment, lines 72-73: # No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init... — this states the key is NOT set.
3. These two statements about the same key, three lines apart, cannot both be true. The comment is stale.
4. Confirmed against perf-changelog.yaml: the "Variant G" changelog entry explicitly says it "adds VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0" to the prior Variant D config, which is the config that originally didn't set the key (matching the stale comment's framing).

Impact: none on runtime behavior — YAML comments are not parsed, so the engine only sees VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0 regardless of what the comments say. The only cost is to future readers/maintainers of this recipe, who could be confused about whether the key is set or not, or duplicate/re-derive the rationale that's already given correctly in lines 66-70.

Fix: delete the stale block (lines 72-77) since the correct, up-to-date rationale is already captured in lines 66-70. No functional change needed, just documentation cleanup.

This is an explicitly non-merge probe PR (Variant K, meant only to produce an empirical answer about whether an image patch is still needed), so there's no urgency to block on this — flagging as a nit for whenever the recipe is next touched.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30385435921
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30385435921

[functionstackx]

/stage-results 30385435921

[github-actions]

@functionstackx staged run 30385435921: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-28~r30385435921

This shared staging slot remains available until the next /stage-results request. Staging workflow

[functionstackx]

/reuse-sweep-run 30385435921

[functionstackx]

/reuse-sweep-run