[WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置 - #2302
[WIP] Test DeepSeek-V4 GB300 Dynamo AgentX recipes / 测试 DeepSeek-V4 GB300 Dynamo AgentX 配置#2302ivanium wants to merge 6 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Claude finished @ivanium's task in 6m 24s —— View job PR Review
SummaryFound 1 blocking issue and 2 warnings (details in inline comments):
The rest looks good: new recipe YAMLs are well-documented, the |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
ivanium
changed the title
There was a problem hiding this comment.
This PR adds new multi-node GB200 AgentX recipes and modifies shared launcher/routing logic, including a change to configs/nvidia-master.yaml (a CODEOWNER-protected file), so it's worth a human look before merging.
What was reviewed: the two new srt-slurm recipe YAMLs (aggregate TP8 and disaggregated 3P1D DEP8/DEP16), the launch_gb200-nv.sh model-path routing and srt-slurm source switch for agentic runs, the benchmark_lib.sh aiperf conv-aware-routing opt-out, and the new nvidia-master.yaml/perf-changelog.yaml entries.
Extended reasoning...
Overview
The PR adds two new DeepSeek-V4 GB200 Dynamo AgentX recipes (aggregate TP8 and disaggregated 3P1D DEP8/DEP16), registers them in configs/nvidia-master.yaml and perf-changelog.yaml, and modifies two shared scripts: benchmarks/benchmark_lib.sh (adds an opt-out env var for aiperf's dynamo conv-aware-routing flag) and runners/launch_gb200-nv.sh (switches the agentic srt-slurm clone from a pinned fork to NVIDIA/srt-slurm:main, and branches MODEL_PATH resolution for DSV4 FP4 based on IS_AGENTIC).
Security risks
No security-sensitive surface — this is internal benchmark/CI infrastructure (Slurm job launching, recipe YAML, shell env plumbing) with no auth, crypto, or externally-reachable inputs. The git clone URLs and container images are pinned to specific orgs/tags rather than user-controlled input.
Level of scrutiny
Higher than typical for this repo: the diff touches shared launcher logic (launch_gb200-nv.sh) and the aiperf routing default in benchmark_lib.sh, both of which affect existing non-agentic GB200 runs in addition to the new recipes. It also modifies configs/nvidia-master.yaml, which is CODEOWNER-protected (@ankur-singh @kedarpotdar-nv @InferenceX/core). Combined with the size and multi-file nature of the change (new multi-node recipes with many tuned env vars, a repo-source switch for the agentic srt-slurm path, and conditional model-path logic), this warrants a human pass rather than auto-approval.
Other factors
The bug-hunting system found no bugs, and several candidate issues (leftover NCCL symm-mem env var, an odd tokenizer string, --no-preflight usage, combining kv-cache-memory with gpu-memory-utilization) were raised and ruled out as not real bugs. The PR description/comments show no CODEOWNER sign-off yet per the standard recipe-PR checklist, which is itself a signal that human review is expected before merge.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29960434846 |
baca859 to
99a28ab
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29976615750 |
ivanium
changed the title
Add four GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, 1P/1D DEP8/DEP8, and 2P/1D DEP8/DEP16. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Configure each disaggregated decode-side MooncakeStore connector as a synchronous KV consumer with lookup disabled while preserving the prefill-side configuration. 中文：新增四个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8、1P/1D DEP8/DEP8 和 2P/1D DEP8/DEP16。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。仅将各分离式配置解码侧的 MooncakeStore 连接器设为同步 KV 消费端并关闭查找功能，同时保留预填充侧原有配置。
99a28ab to
985c29e
Compare
Add four GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, 1P/1D DEP8/DEP8, and 2P/1D DEP8/DEP16. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Align all disaggregated recipes on Dynamo 1.3.0.dev20260720, preserve prefill-only connector prefetch and breakable CUDA graph settings, and configure decode-side MooncakeStore as a synchronous KV consumer with lookup disabled. 中文：新增四个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8、1P/1D DEP8/DEP8 和 2P/1D DEP8/DEP16。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。所有分离式配置统一使用 Dynamo 1.3.0.dev20260720，连接器预取和可中断 CUDA Graph 设置仅保留在预填充侧；解码侧 MooncakeStore 设为同步 KV 消费端并关闭查找功能。
985c29e to
dfc774f
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30154534054 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30170531787 |
Add four GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, 1P/1D DEP8/DEP8, and 2P/1D DEP8/DEP16. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Align all disaggregated recipes on Dynamo 1.3.0.dev20260720, preserve prefill-only connector prefetch and breakable CUDA graph settings, and configure decode-side MooncakeStore as a synchronous KV consumer with lookup disabled. 中文：新增四个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8、1P/1D DEP8/DEP8 和 2P/1D DEP8/DEP16。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。所有分离式配置统一使用 Dynamo 1.3.0.dev20260720，连接器预取和可中断 CUDA Graph 设置仅保留在预填充侧；解码侧 MooncakeStore 设为同步 KV 消费端并关闭查找功能。
8a83d53 to
64f87af
Compare
Add four GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, 1P/1D DEP8/DEP8, and 2P/1D DEP8/DEP16. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Align all disaggregated recipes on Dynamo 1.3.0.dev20260720, preserve prefill-only connector prefetch and breakable CUDA graph settings, and configure decode-side MooncakeStore as a synchronous KV consumer with lookup disabled. 中文：新增四个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8、1P/1D DEP8/DEP8 和 2P/1D DEP8/DEP16。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。所有分离式配置统一使用 Dynamo 1.3.0.dev20260720，连接器预取和可中断 CUDA Graph 设置仅保留在预填充侧；解码侧 MooncakeStore 设为同步 KV 消费端并关闭查找功能。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30217145037 |
Add three GB300 MTP3 AgentX topologies using the bedf79e image: aggregate TP8, 1P/1D DEP4/DEP8, and 1P/1D DEP8/DEP8. Follow PR #2157's aggregate/disaggregated organization, PR #2269's CI rack networking, and PR #2302's upstream srt-slurm per-node launch path with automatic hybrid load balancing. Align both disaggregated recipes on Dynamo 1.3.0.dev20260720, preserve prefill-only connector prefetch and breakable CUDA graph settings, and configure decode-side MooncakeStore as a synchronous KV consumer with lookup disabled. 中文：新增三个使用 bedf79e 镜像的 GB300 MTP3 AgentX 拓扑：聚合式 TP8、1P/1D DEP4/DEP8 和 1P/1D DEP8/DEP8。配置组织遵循 PR #2157 的聚合式/分离式结构，网络设置对齐 PR #2269 的 CI 机架环境，并依据 PR #2302 使用上游 srt-slurm 的单节点单进程启动模式及其自动启用的混合负载均衡。两个分离式配置统一使用 Dynamo 1.3.0.dev20260720，连接器预取和可中断 CUDA Graph 设置仅保留在预填充侧；解码侧 MooncakeStore 设为同步 KV 消费端并关闭查找功能。
| import_squash() { | ||
| local squash="$1" image="$2" | ||
| local lock="${squash}.lock" | ||
| srun --partition=$SLURM_PARTITION --exclusive --time=180 bash -c " | ||
| local exclude_args=() | ||
| if [[ -n "$GB300_SLURM_EXCLUDED_NODELIST" ]]; then | ||
| exclude_args+=(--exclude="$GB300_SLURM_EXCLUDED_NODELIST") | ||
| fi | ||
| srun --partition=$SLURM_PARTITION "${exclude_args[@]}" --exclusive --time=180 bash -c " | ||
| exec 9>\"$lock\" | ||
| flock -w 600 9 || { echo 'Failed to acquire lock for $squash' >&2; exit 1; } | ||
| if unsquashfs -l \"$squash\" > /dev/null 2>&1; then |
There was a problem hiding this comment.
🟡 The DCGM exporter squash validation srun right after import_squash (runners/launch_gb300-nv.sh:117-127, specifically the standalone srun ... unsquashfs -l "$DCGM_EXPORTER_SQSH" call) builds its own srun directly and never gets the new --exclude=$GB300_SLURM_EXCLUDED_NODELIST flag that import_squash now threads in. It reads the same squash from the same unhealthy-mount node this PR is otherwise avoiding, so on a dcgm-power (fp8) lane it can still be scheduled onto im-gb300-r01-c002 and fail. This is a pre-existing-style gap only reachable on fp8 dcgm-power lanes (not this PR'''s fp4 recipes), and it fails loudly/retriably rather than corrupting results, so it'''s a nit worth a one-line fix (reuse the same exclude_args).
Extended reasoning...
The bug: This PR introduces GB300_SLURM_EXCLUDED_NODELIST (defaulted to im-gb300-r01-c002, an unhealthy-shared-mount node) and threads it into every srun wrapped by import_squash() via a new exclude_args array built at runners/launch_gb300-nv.sh:120-123. That correctly covers the IMAGE, NGINX_IMAGE, and DCGM_EXPORTER_IMAGE squash imports. But immediately after the DCGM_EXPORTER_IMAGE import (line ~177), there's a second, standalone validation step that builds its own srun invocation from scratch instead of reusing import_squash or its exclude_args:
srun --partition=$SLURM_PARTITION --exclusive --time=30 bash -c "unsquashfs -l \"$DCGM_EXPORTER_SQSH\" > /dev/null" || { echo "Error: DCGM exporter squash invalid: $DCGM_EXPORTER_SQSH" >&2; exit 1; }This line has no --exclude flag at all, so SLURM can still schedule it on im-gb300-r01-c002 even though the import right above it was explicitly steered away from that node.
Code path that triggers it: This validation only runs when USES_DCGM_POWER == 1, which the script's own guard restricts to PRECISION == fp8 (it hard-exits otherwise). None of this PR's own recipes are dcgm-power/fp8 — they're all fp4 DeepSeek-V4 agentic — so this PR's own sweep never exercises the buggy line. However, the launcher is shared infrastructure: other in-repo configs (e.g. sglang/qwen3.5/gb300-fp8/8k1k/1p1d-tp4-tp4.yaml, confirmed to have telemetry.enabled: true / provider: dcgm-power) do reach it, so a concurrent or future fp8 power-lane sweep through this same launcher can trip it.
Why existing code doesn't prevent it: Before this PR, no srun call in the file excluded any node, so every srun — including this validation one — was uniformly exposed to im-gb300-r01-c002. This PR is what introduces the inconsistency: it fixes the import for this same squash file just above, but the validation using the same file two lines later was apparently overlooked, likely because it isn't routed through the shared import_squash helper.
Impact: On a dcgm-power/fp8 lane, if SLURM happens to place this 30-second validation job on the excluded node, unsquashfs -l will fail against the same broken NFS mount the whole exclusion mechanism exists to avoid, and the script does exit 1 — a spurious, retriable job failure. It does not cause data loss or silently produce bad results; it just wastes a sweep run and requires a retry/re-trigger.
Step-by-step proof:
- Some fp8 power-lane job sets
USES_DCGM_POWER=1(e.g. the qwen3.5 gb300-fp8 dcgm-power recipe). import_squash "$DCGM_EXPORTER_SQSH" "$DCGM_EXPORTER_IMAGE"runs with--exclude=im-gb300-r01-c002baked in viaexclude_args, so it lands on a healthy node and imports/validates the squash fine.- The very next command, the standalone
srun --partition=$SLURM_PARTITION --exclusive --time=30 bash -c "unsquashfs -l ...", has no exclude flag. - SLURM's scheduler is free to place this exclusive 30s job on
im-gb300-r01-c002(the excluded node is still up and schedulable — just its/dataNFS mount is unhealthy). unsquashfs -l "$DCGM_EXPORTER_SQSH"on that node hits the broken mount and returns non-zero.- The
||triggersecho "Error: DCGM exporter squash invalid..."andexit 1, failing the whole launcher script even though the squash file itself is perfectly valid and was just confirmed importable one step earlier on a healthy node.
Fix: reuse the same exclude_args array that import_squash builds (hoist it to file scope, or just duplicate the --exclude=$GB300_SLURM_EXCLUDED_NODELIST construction) into this standalone srun call, e.g.:
srun --partition=$SLURM_PARTITION "${exclude_args[@]}" --exclusive --time=30 bash -c "unsquashfs -l \"$DCGM_EXPORTER_SQSH\" > /dev/null" || ...Severity justification: This does not affect this PR's own fp4 DeepSeek-V4 recipes at all (the guard makes the power path unreachable for fp4), it fails loudly and retriably rather than corrupting data, and it only manifests when SLURM happens to schedule onto one specific temporarily-excluded node on a separate fp8 lane. It's a real, PR-introduced inconsistency worth fixing, but not something that should block this PR's merge.
ivanium
added
agentx
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31133568961 |
134a47c to
2422aee
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31166635932 |
ivanium
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31217933449 |
Double the three GB300 Dynamo-vLLM AgentX MTP P/D points to concurrency 256/512/768, add the 2P1D DEP8/DEP16 nightly-vLLM point, and use 180GB of Mooncake store per GPU. Work around unhealthy shared mounts on two GB300 nodes, disable the shared AIPerf mmap cache for these concurrent jobs, and pace c768 warmup admission over 60 seconds. 中文：扩大并稳定 GB300 分离式推理 MTP 负载。将三个 Dynamo-vLLM AgentX MTP P/D 测试点的并发度翻倍至 256/512/768，新增使用 vLLM nightly 镜像的 2P1D DEP8/DEP16 测试点，并将每张 GPU 的 Mooncake 存储空间增至 180GB。 中文：临时绕过两台共享挂载异常的 GB300 节点；为这些并发任务禁用共享 AIPerf mmap 缓存，并将 c768 的预热并发准入平滑扩展到 60 秒。
Route the configured warmup concurrency ramp through the updated AIPerf submodule so high-concurrency snapshot primers do not overload Dynamo prefill ingress. Validate existing GB300 squash cache entries before opening the NFS flock, while retaining the locked miss path and applying node exclusions consistently.\n\n中文：通过更新后的 AIPerf 子模块传递预热并发爬升配置，避免高并发快照预热请求压垮 Dynamo 预填充入口。GB300 启动器会先无锁校验已有的 Squash 缓存，仅在缓存缺失或损坏时进入加锁导入流程，并统一应用故障节点排除参数。 Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Replace the previous three P/D measurements with five DEP topology points, add the aggregate TP8 c1 and TP4 c8 runs, and apply the prefill threshold and warmup safeguards. 中文：重新调优 GB300 AgentX 分离式推理扫描。用五个 DEP 拓扑测试点替换原有三个 P/D 测试点，加入聚合式 TP8 c1 与 TP4 c8，并应用预填充阈值和预热保护配置。
Drop the unpublished warmup-ramp extension and its recipe wiring, and point the submodule solely at the upstream concurrency-safe mmap release. 中文：改用上游 AIPerf v1.0.3。移除尚未发布的预热并发爬升扩展及配方接线，并将子模块仅指向包含并发安全 mmap 修复的上游版本。
1a2f9aa to
0c33d46
Compare
Stage exact Dynamo wheels for disaggregated GB300 AgentX workers and tune TCP connection growth and acknowledgement timeout for warmup bursts. 中文：为 GB300 分离式 AgentX worker 预先暂存精确版本的 Dynamo wheel，并调整 TCP 连接扩展与确认超时，避免启动解析失败和预热突发超时。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31276735958 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31302825977 |
|
/stage-results 31302825977 |
|
@ivanium staged run 31302825977: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-09~r31302825977 This run remains available across future @ivanium 已将运行 31302825977 发布到预发布环境：https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-09~r31302825977 后续的 |
Raise aggregate token budgets to 32K, add TP8 c4, exclude unhealthy GB300 nodes from TP8, and restore vLLM default long-prefill behavior for c256, c512, and c576. 中文：将聚合式配置的 Token 批处理上限提高到 32K，新增 TP8 c4，为 TP8 排除异常 GB300 节点，并在 c256、c512 和 c576 恢复 vLLM 默认的长预填充调度行为。
2 participants
Summary
mainand re-append this PR's changelog entry at the tailAIPerf dependency fix: SemiAnalysisAI/aiperf#38
Validation
bash -n runners/launch_gb300-nv.sh benchmarks/benchmark_lib.sh中文说明
flock；同时让 AIPerf 的自动生成预热阶段真正应用已配置的 60 秒并发爬升，防止快照预热请求压垮 Dynamo 预填充入口main，并按追加式变更日志规则将本 PR 条目重新放到文件末尾AIPerf 依赖修复：SemiAnalysisAI/aiperf#38