Skip to content

Commit c5e8b9b

Browse files
EricKing626chunfangamdjiejingzhangamdseungrokjclaude
authored
[AMD][AgentX] Add GLM-5.2 MXFP4 MI355X SGLang TP4/EP4 DCP4 points / 新增 GLM-5.2 MXFP4 MI355X SGLang TP4/EP4 DCP4 點位 (#4064)
* [AMD][AgentX] Add GLM-5.2 MXFP4 MI355X SGLang TP4/EP4 DCP4 points Add TP4/EP4 decode-context-parallel (DCP4) points at concurrency 16, 24, 40 and 48 to glm5.2-fp4-mi355x-sglang-agentic-mtp. The latent KV is striped across the four ranks, so these points run without HiCache, which DSA DCP rejects. Validate decode-context-parallel-size against the matrix DCP size for SGLang recipes, as for vLLM and ATOM. 中文：為 glm5.2-fp4-mi355x-sglang-agentic-mtp 新增 TP4/EP4 搭配 decode context parallel（DCP4）的 concurrency 16、24、40、48 點位。latent KV 依位置分散到四張卡上，因此這些點位不使用 HiCache（DSA DCP 不支援 HiCache）。SGLang recipe 的 decode-context-parallel-size 也會與 matrix 的 DCP size 比對驗證，與 vLLM、ATOM 一致。 * [AMD][AgentX] Point the DCP4 changelog entry at PR 4064 Record the real pull request number in the perf-changelog entry for the GLM-5.2 MXFP4 MI355X SGLang DCP4 points. 中文：把 perf-changelog 的 pr-link 換成實際的 PR 4064。 * fix changelog * fix(amd): use merged DSA DCP ROCm image for GLM-5.2 将 GLM-5.2 SGLang AgentX 配置更新到包含上游 #44322 的 20261028 ROCm 7.2 镜像，并更新对应性能变更记录。 * docs(changelog): rewrite GLM-5.2 FP4 MI355X SGLang DCP4 block from the diff - Update the image for all three arms (TP4/EP4 HiCache, TP4/EP4 DCP4, TP8/EP1) from lmsysorg/sglang:v0.5.22-rocm720-mi35x (digest-pinned release) to the lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261028 nightly, built from SGLang main e0df901d. - Add TP4/EP4 points with decode context parallelism (DCP4) at concurrency 16, 24, 40 and 48. The latent KV is striped across the four ranks, so these points keep KV on the GPU without HiCache, which SGLang rejects with DCP. - DCP4 points keep the base recipe: EAGLE with 5 steps, top-k 1 and 6 draft tokens, an FP8 KV cache and the Triton DSA prefill and decode backends. Admission (max-running-requests) is 32, 48, 80 and 96, twice the concurrency, and the decode CUDA graph limit is 32, 48, 64 and 64. - The new image adds sgl-project/sglang#44322, which enables DCP on the Triton sparse-MLA backend on ROCm, including FP8 KV writes and MTP draft sharding. It also adds DSA fixes for MTP draft decode (#42565) and GLM-5.2 CUDA-graph side-stream replay (#45349). #43735 (DSA DCP with MTP) was already in v0.5.22. - Draft data types are unchanged: the MTP layer 78 ships in BF16 and is listed in the checkpoint's Quark exclude list, so SGLang loads it unquantized at both the old and new commits. Its KV cache is FP8, shared with the target. - 三个配置组（TP4/EP4 HiCache、TP4/EP4 DCP4、TP8/EP1）的镜像从 lmsysorg/sglang:v0.5.22-rocm720-mi35x（按 digest 固定的正式版）更新为 lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261028 nightly，基于 SGLang main e0df901d 构建。 - 新增 TP4/EP4 解码上下文并行（DCP4）测试点，并发 16、24、40、48。latent KV 分散在四个 rank 上，因此这些点无需 HiCache 即可把 KV 保留在 GPU 上（SGLang 不允许 DCP 与 HiCache 同时使用）。 - DCP4 测试点沿用基础配置：EAGLE 5 步、top-k 1、6 个草稿 token，FP8 KV cache，Triton DSA prefill/decode 后端。准入上限（max-running-requests）为 32、48、80、96，即并发的两倍；decode CUDA graph 上限为 32、48、64、64。 - 新镜像包含 sgl-project/sglang#44322，在 ROCm 的 Triton sparse-MLA 后端上启用 DCP，包括 FP8 KV 写入和 MTP 草稿分片；还包含 MTP 草稿解码的 DSA 修复（#42565）和 GLM-5.2 CUDA graph 侧流重放修复（#45349）。#43735（DSA DCP 搭配 MTP）在 v0.5.22 中已存在。 - 草稿模型数据类型不变：MTP 第 78 层以 BF16 发布，并列在 checkpoint 的 Quark exclude 列表中，因此新旧两个提交下 SGLang 都以非量化方式加载它。其 KV cache 为 FP8，与目标模型共享。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Chun Fang <chun.fang@amd.com> Co-authored-by: Jiejing Zhang <280536569+jiejingzhangamd@users.noreply.github.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: seungrokj <seungrok.jung@amd.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Bryan <58582368+Oseltamivir@users.noreply.github.com>
1 parent 329d1f3 commit c5e8b9b

3 files changed

Lines changed: 81 additions & 4 deletions

File tree

‎inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/glm5.2/sglang/mi355x-fp4-mtp/agentic.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,67 @@ override_tp4_c16_hicache:
152152
CONC: '16'
153153
KV_OFFLOADING: dram
154154

155+
# DSA DCP rejects HiCache; DCP4 stripes the latent KV across the TP4 ranks instead.
156+
override_tp4_dcp4_c16:
157+
roles:
158+
agg:
159+
gpus: 4
160+
args:
161+
tensor-parallel-size: 4
162+
expert-parallel-size: 4
163+
decode-context-parallel-size: 4
164+
max-running-requests: 32
165+
cuda-graph-max-bs-decode: 32
166+
benchmark:
167+
env:
168+
CONC: '16'
169+
KV_OFFLOADING: none
170+
171+
override_tp4_dcp4_c24:
172+
roles:
173+
agg:
174+
gpus: 4
175+
args:
176+
tensor-parallel-size: 4
177+
expert-parallel-size: 4
178+
decode-context-parallel-size: 4
179+
max-running-requests: 48
180+
cuda-graph-max-bs-decode: 48
181+
benchmark:
182+
env:
183+
CONC: '24'
184+
KV_OFFLOADING: none
185+
186+
override_tp4_dcp4_c40:
187+
roles:
188+
agg:
189+
gpus: 4
190+
args:
191+
tensor-parallel-size: 4
192+
expert-parallel-size: 4
193+
decode-context-parallel-size: 4
194+
max-running-requests: 80
195+
cuda-graph-max-bs-decode: 64
196+
benchmark:
197+
env:
198+
CONC: '40'
199+
KV_OFFLOADING: none
200+
201+
override_tp4_dcp4_c48:
202+
roles:
203+
agg:
204+
gpus: 4
205+
args:
206+
tensor-parallel-size: 4
207+
expert-parallel-size: 4
208+
decode-context-parallel-size: 4
209+
max-running-requests: 96
210+
cuda-graph-max-bs-decode: 64
211+
benchmark:
212+
env:
213+
CONC: '48'
214+
KV_OFFLOADING: none
215+
155216
override_tp8_c1:
156217
roles:
157218
agg:

‎inferencex-e2e/configs/amd-master.yaml‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1088,16 +1088,19 @@ dsv41flash-fp4-mi325x-vllm-agentic-dspark:
10881088
- { tp: 2, kv-offloading: none, spec-decoding: mtp, conc-list: [1, 2, 4, 8, 16, 32], srt-recipe: agentic.yaml }
10891089

10901090
# GLM-5.2 FP4 agentic-coding benchmark on MI355X via SGLang with MTP speculative
1091-
# decoding. Two arms:
1091+
# decoding. Three arms:
10921092
# (1) TP4/EP4 with HiCache KV offloading to DRAM at concurrency [4, 8, 10, 12, 14, 16];
10931093
# serves the high-concurrency, throughput-dominant regime.
1094-
# (2) TP8/EP1 without KV offloading, concurrency sweep [1, 2, 4];
1094+
# (2) TP4/EP4 with DCP4 at concurrency [16, 24, 40, 48]; striping the latent KV
1095+
# across the four ranks holds the working set on device without HiCache,
1096+
# which DSA DCP rejects.
1097+
# (3) TP8/EP1 without KV offloading, concurrency sweep [1, 2, 4];
10951098
# EP=1 eliminates the MoE all-to-all collective, reducing ITL at low-to-mid
10961099
# concurrency; GPU-resident KV keeps latency independent of DRAM bandwidth.
1097-
# c4 intentionally overlaps both arms to compare the topology crossover.
1100+
# c4 intentionally overlaps arms (1) and (3) to compare the topology crossover.
10981101
# SA selects the Pareto-optimal arm per concurrency point.
10991102
glm5.2-fp4-mi355x-sglang-agentic-mtp:
1100-
image: lmsysorg/sglang:v0.5.22-rocm720-mi35x@sha256:c4b61360b251d62bb17da63f175d571ab22479eb46a144548d79f3ed3998ea11
1103+
image: lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261028
11011104
model: amd/GLM-5.2-MXFP4
11021105
model-prefix: glm5.2
11031106
runner: cluster:mi355x-amds
@@ -1110,6 +1113,7 @@ glm5.2-fp4-mi355x-sglang-agentic-mtp:
11101113
- dram-utilization: 0.85
11111114
search-space:
11121115
- { tp: 4, ep: 4, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [4, 8, 10, 12, 14, 16], spec-decoding: mtp, srt-recipe: agentic.yaml }
1116+
- { tp: 4, ep: 4, dcp-size: 4, kv-offloading: none, conc-list: [16, 24, 40, 48], spec-decoding: mtp, srt-recipe: agentic.yaml }
11131117
- { tp: 8, ep: 1, kv-offloading: none, conc-list: [1, 2, 4], spec-decoding: mtp, srt-recipe: agentic.yaml }
11141118

11151119
# GLM-5.2 FP8 agentic-coding benchmark on MI355X via SGLang with MTP speculative

‎inferencex-e2e/perf-changelog.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11770,6 +11770,18 @@
1177011770
- "Prefill: shortest-prefill-first, chunk 8192, mem-fraction-static 0.85, HiCache 200 GB per rank write_back."
1177111771
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4686
1177211772

11773+
- config-keys:
11774+
- glm5.2-fp4-mi355x-sglang-agentic-mtp
11775+
scenario-type:
11776+
- agentic-coding
11777+
description:
11778+
- "Update the image for all three arms (TP4/EP4 HiCache, TP4/EP4 DCP4, TP8/EP1) from lmsysorg/sglang:v0.5.22-rocm720-mi35x (digest-pinned release) to the lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261028 nightly, built from SGLang main e0df901d."
11779+
- "Add TP4/EP4 points with decode context parallelism (DCP4) at concurrency 16, 24, 40 and 48. The latent KV is striped across the four ranks, so these points keep KV on the GPU without HiCache, which SGLang rejects with DCP."
11780+
- "DCP4 points keep the base recipe: EAGLE with 5 steps, top-k 1 and 6 draft tokens, an FP8 KV cache and the Triton DSA prefill and decode backends. Admission (max-running-requests) is 32, 48, 80 and 96, twice the concurrency, and the decode CUDA graph limit is 32, 48, 64 and 64."
11781+
- "The new image adds sgl-project/sglang#44322, which enables DCP on the Triton sparse-MLA backend on ROCm, including FP8 KV writes and MTP draft sharding. It also adds DSA fixes for MTP draft decode (#42565) and GLM-5.2 CUDA-graph side-stream replay (#45349). #43735 (DSA DCP with MTP) was already in v0.5.22."
11782+
- "Draft data types are unchanged: the MTP layer 78 ships in BF16 and is listed in the checkpoint's Quark exclude list, so SGLang loads it unquantized at both the old and new commits. Its KV cache is FP8, shared with the target."
11783+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4064
11784+
1177311785
- config-keys:
1177411786
- minimaxm31flash-fp4-mi355x-sglang-agentic-dspark
1177511787
description:

0 commit comments

Comments
 (0)