-
Notifications
You must be signed in to change notification settings - Fork 253
[Klaud Cold][agentic experiment][Variant B] Kimi-K3 B200 agg TP8xPP2 agentic — vLLM tool-call-parser flags / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——vLLM tool-call-parser 参数 #2357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
b4fd077
feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic r…
functionstackx 4dbbdc8
docs: link PR #2355 in changelog entry and MODELS rows
functionstackx e8d42a7
fix: pin agentic srt-slurm to NVIDIA v1.0.36, dynamo 1.2.1, conc-8 sm…
functionstackx c1e2a56
fix: drop OpenAI-frontend tool-choice flags from dynamo-vllm worker args
functionstackx ef35fd1
fix: try dynamo wheel 1.2.0.dev20260426 for Kimi-K3 frontend tokenizer
functionstackx be6c56e
fix: pin dynamo to day-zero Kimi-K3 commit, restore kimi_k3 parser flags
functionstackx c6917e6
fix: use dynamo namespaced --dyn-* kimi_k3 parser args on the worker
functionstackx 288349f
feat: parser-flag experiment B — plain vLLM tool-call-parser spellings
functionstackx 2fe84a8
docs: point changelog and MODELS rows at experiment PR #2357
functionstackx File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
117 changes: 117 additions & 0 deletions
117
benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| name: "kimik3-vllm-agg-b200-tp8pp2-agentic" | ||
|
|
||
| # Kimi-K3 MXFP4 B200 AGGREGATED TP8 x PP2 agentic recipe (2 nodes / 16 GPUs). | ||
| # The native MXFP4 checkpoint (2.8T total params, ~1.4TB of weights) does not | ||
| # fit one 8xB200 node, so TP8 shards attention/dense (/8) and PP2 splits the | ||
| # 93 layers (/2) across 16 GPUs. Plain TP (NOT TEP): expert parallelism is | ||
| # deliberately off, so the 896 routed experts are TP-sharded inside each | ||
| # pipeline stage. Node allocation = tp*pp/gpus_per_node = 8*2/8 = 2 nodes. | ||
| # Aggregated (single worker, decode num-worker 0) — no P/D split, no NIXL. | ||
| # VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION fuses the K3 LatentMoE tail path in | ||
| # the kimi-k3 bring-up image. | ||
| model: | ||
| path: "kimik3" | ||
| container: "vllm/vllm-openai:kimi-k3" | ||
| precision: "fp4" | ||
|
|
||
| identity: | ||
| model: | ||
| repo: "moonshotai/Kimi-K3" | ||
| container: | ||
| image: "vllm/vllm-openai:kimi-k3" | ||
| frameworks: | ||
| dynamo: "1.4.0-kimi-k3-dev.1" | ||
|
|
||
| dynamo: | ||
| install: true | ||
| # Day-zero Kimi-K3 dynamo ("feat: Added support for Kimi-K3", tag | ||
| # v1.4.0-kimi-k3-dev.1 == this commit): adds the kimi_k3 tiktoken tokenizer | ||
| # to the rust frontend (dynamo <=1.2.1 only knows kimi/kimi_k2/kimi_k25/ | ||
| # deepseek_v3, so the model never registers and every request 404s) and the | ||
| # kimi_k3 tool-call/reasoning parser worker args. | ||
| hash: "ba83080ecd31c1ce918559e576d3c5bc9e092ff1" | ||
|
|
||
| slurm: | ||
| time_limit: "8:00:00" | ||
|
|
||
| health_check: | ||
| interval_seconds: 10 | ||
| max_attempts: 1440 | ||
|
|
||
| resources: | ||
| gpu_type: "b200" | ||
| gpus_per_node: 8 | ||
| agg_nodes: 2 | ||
| agg_workers: 1 | ||
| gpus_per_agg: 16 | ||
|
|
||
| infra: | ||
| etcd_nats_dedicated_node: false | ||
| nats_max_payload_mb: 32 | ||
|
|
||
| frontend: | ||
| type: dynamo | ||
| enable_multiple_frontends: false | ||
|
|
||
| backend: | ||
| type: vllm | ||
| connector: null | ||
| aggregated_environment: | ||
| VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION: "1" | ||
| VLLM_SERVER_DEV_MODE: "1" | ||
| # ~1.4TB of MXFP4 weights off shared Lustre: keep the engine-ready window | ||
| # generous, and let one long AgentX request hold a PP stage beyond vLLM's | ||
| # 300-second model-execution default. | ||
| VLLM_ENGINE_READY_TIMEOUT_S: "3600" | ||
| VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| TILELANG_CLEANUP_TEMP_FILES: "1" | ||
| UCX_MEMTYPE_CACHE: "n" | ||
| UCX_MEMTYPE_REG_WHOLE: "n" | ||
| UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_10:1,mlx5_11:1" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" | ||
| vllm_config: | ||
| aggregated: | ||
| served-model-name: "moonshotai/Kimi-K3" | ||
| tensor-parallel-size: 8 | ||
| pipeline-parallel-size: 2 | ||
| trust-remote-code: true | ||
| load-format: fastsafetensors | ||
| moe-backend: auto | ||
| gpu-memory-utilization: 0.95 | ||
| no-enable-flashinfer-autotune: true | ||
| # kimi_k3 parsers via the plain vLLM spellings (parser-flag experiment | ||
| # variant B — sibling PRs try the --dyn-* namespaced args and no parser | ||
| # flags). Note dynamo <=1.2.1's worker rejected --tool-call-parser as | ||
| # unrecognized; the day-zero K3 dynamo pinned above may accept it. | ||
| tool-call-parser: kimi_k3 | ||
| reasoning-parser: kimi_k3 | ||
| # No explicit max-model-len: let vLLM derive the native 1M window from | ||
| # the model config (agentic trajectories blow past any small cap, and | ||
| # K3's KDA layers keep per-token KV small — only the 24 gated-MLA | ||
| # layers hold cache). Prefix caching stays on (default) for trajectory | ||
| # reuse. Cap prefill chunks so a single long request cannot OOM a | ||
| # pipeline stage; let vLLM pick max-num-seqs. | ||
| max-num-batched-tokens: 8192 | ||
|
|
||
| sbatch_directives: | ||
| segment: "1" | ||
|
|
||
| srun_options: | ||
| container-remap-root: "" | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| # Keep the aggregate worker in the multinode result schema so ingestion | ||
| # uses the zero decode-worker count instead of duplicating TP into P and D. | ||
| IS_MULTINODE: "true" | ||
| AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS: "14400" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 The recipe deliberately sets
health_check.max_attempts: 1440(4hr window) for the ~1.4TB MXFP4 checkpoint load off shared Lustre, butrunners/launch_b200-dgxc.sh's pre-existing multinode-path sed (s/^ max_attempts: [0-9]*/ max_attempts: 720/) unconditionally matches the recipe's 2-space-indented line and silently clamps it back to 720 (2hr). This is the first multinode+agentic recipe oncluster:b200-dgxcto carrymax_attempts > 720, so this PR is the first to collide with that override — the author's explicit 4hr intent is currently a dead value; worth scoping the sed (e.g. only bump if the value is below 720) or dropping/correcting the recipe's comment so it doesn't mislead future readers.Extended reasoning...
The bug:
benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yamlsets:with an explicit design rationale nearby ("~1.4TB of MXFP4 weights off shared Lustre: keep the engine-ready window generous"), i.e. a deliberate 4-hour health-check window (1440 × 10s).
However,
runners/launch_b200-dgxc.sh(unmodified by this PR, but exercised for the first time by this recipe on this runner) contains, in theIS_MULTINODEbranch:This runs unconditionally on every multinode job, after the recipe file has been copied into the srt-slurm checkout and referenced via
CONFIG_FILE. The pattern^ max_attempts: [0-9]*(anchored, exactly two leading spaces) matches the recipe'smax_attempts: 1440line exactly, so the sed rewrites it in place tomax_attempts: 720beforesrtctl applyever reads the file. The recipe's intended 4-hour window is silently reduced to 2 hours — the1440value the author wrote (and commented on) never actually reaches the health-check logic.Why nothing else in the pipeline prevents this: the sed has no guard for "only bump if lower" — it's a blind overwrite regardless of the file's existing value, so it clamps down just as readily as it bumps up. Nothing in the recipe YAML or
configs/nvidia-master.yamlrouting opts out of this launcher step, and there's no comment in the recipe warning thatmax_attemptsis moot on this runner — so a reader of the recipe today would reasonably (and incorrectly) believe the 4-hour window is honored.Why this specific PR is the one that surfaces it: all other recipes with
max_attempts: 1440in the repo are wired to the gb200/gb300 launchers (launch_gb200-nv.sh/launch_gb300-nv.sh), which don't contain this sed. This Kimi-K3 config is the first multinode+agentic recipe oncluster:b200-dgxc(launch_b200-dgxc.sh) to specify a health-check window above 720, so it's the first config to actually collide with the override — it isn't a hypothetical, it directly affects this PR's own recipe.Step-by-step proof:
configs/nvidia-master.yamlrouteskimik3-fp4-b200-dynamo-vllm-agentictorunner: cluster:b200-dgxc, i.e.launch_b200-dgxc.sh, withadditional-settings: CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml.IS_AGENTIC=1triggers the overlay step, copyingbenchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/into the srt-slurm clone'srecipes/vllm/kimi-k3/agentic/— this is the actual fileCONFIG_FILEpoints at.IS_MULTINODE == "true"for this recipe, the script reaches the linesed -i 's/^ max_attempts: [0-9]*/ max_attempts: 720/' "${CONFIG_FILE%%:*}".max_attempts: 1440at 2-space indent — matches the anchor exactly — sed rewrites it tomax_attempts: 720.srtctl apply -f "$CONFIG_FILE" ...runs immediately after, reading the now-mutated file: the health check will poll for 720 × 10s = 7200s = 2hr, not the author's intended 14400s = 4hr.Impact and why it's a nit, not a blocker: the mechanism is real and the author's stated intent is genuinely defeated, but a concrete failure at merge is unlikely for two independent reasons already present in this same PR: (1) the recipe sets
VLLM_ENGINE_READY_TIMEOUT_S: "3600"(1hr) inaggregated_environment, so vLLM's own engine-init timeout fires at 1hr regardless of the health-check window — the clamped 2hr window is never actually the binding constraint versus the intended 4hr, since both exceed the 1hr engine timeout; and (2) the PR's ownperf-changelog.yamlentry records the observed engine-ready time as "~14 min via fastsafetensors," roughly 8x under even the clamped 720-attempt/2hr window. So the failure mode the recipe's comment is defending against (Lustre contention pushing load past 2hr) is not observed in practice here.Suggested fix: either scope the launcher's sed to only raise
max_attemptswhen it's below the floor (e.g. compare before overwriting), or update the recipe'smax_attempts/comment to reflect that 720 is the effective ceiling oncluster:b200-dgxcuntil the launcher is changed — as-is, the1440value and its accompanying comment are misleading to future readers of this recipe.