Repository navigation
[DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache and concurrency / [DNM, Experimental] GLM-5.3-Flash MI325X FP8 HiCache与并发优化 - #4735
Draft
functionstackx wants to merge 3 commits into
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
functionstackx
added
the
full-sweep-fail-fast
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41352867682 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41352867682 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Draft optimization child of #4722, based on its passed head
b6c466048bda2deceb0ce81fd847053c8f9b7599. The parent remains unchanged and unmerged; this child has no GPU activation label and makes no performance or host-restore correctness claim.zai-org/GLM-5.3-Flash@46a050bec13002ad29ee795c16e61cf4fa3c4a9c, BF16 KV, as-shipped FP8/NextN, EAGLE5/1/6, 1,048,576 context, the full393-entry canonical corpus and3600s workload, and shared golden AL3.603536 for throughput only.ROCM_QUICK_REDUCE_QUANTIZATION=NONE,SGLANG_OPT_FUSE_MHC_POST_PRE=0, and both prohibited precision overrides disabled. The shared staging helper and pinned server tokenizer are unchanged; the canonical client remains online.Required official image exception
The parent remains on v0.5.21. Its actual SGLang SHA
e00930c5489053f26d86b179cee0d087f846acbblacks the hybrid INDEXER host pool, matching the reported GLM-5.3-Flash host-restore correctness bug; the .21 backport was closed unmerged (issue43997, fix40915, backport43998).This child alone uses official
lmsysorg/sglang:v0.5.22-rocm10-mi30x@sha256:5658cd0d8de0c68de10d2e349296f55ceff98053182fa87b2a7e942e5e7f3c2a, with matching cache-off controls rather than attributing a framework change to HiCache (immutable image metadata). Release source isbaa9befa381511cd4d56f83980951e5a45fffc6d; actual candidate-image runtime provenance must still be fingerprinted before GPU qualification.Host memory and pending gates
ON uses native
hicache-host-memory-fraction: 0.60,directI/O andpage_first_direct, with no explicit size/ratio or storage backend. This preserves the cgroup-aware combined auto-budget path; it does not mean that60% of static inventory is preallocated (native flag semantics).Both arms retain the same whole-node resource reservation: eight GPUs reserved, four active TP workers, exclusive allocation,384 CPUs,
mem=0. Staticdram-utilization: 1.0means100% of the existing nominal eligible GPU-share budget, not the live pool fraction or physical reservation; the generator reports1208GB for ON and0 for OFF under its existing offload metadata semantics, despite identical physical reservations. The legacy inventory cap remains unchanged and is not binding on this node.Before any sweep activation, require actual .22 source/image fingerprinting, live host/cgroup and NUMA/PCIe observations, actual pool-byte accounting, and a forced host-restore correctness probe with MTP and all synthetic-acceptance flags absent. Ordinary GSM8K alone is insufficient because it may not exercise L2 eviction and restore; the known open sidecar-budget/MTP issues also require checking matching target/draft pool rows (PR42918).
Local validation
366 focused tests passed. All10 recipes were generated and schema-validated; actual native commands were rendered for eight performance points and two eval points, with full-duration canonical replay commands, unrestricted canonical lm-eval commands, correct OFF/ON flags, no explicit host size/ratio, and SIMULATE removal in both eval configurations. The exclusive8-GPU/mem0 resource rendering is unchanged.
No GPU job has been dispatched for this child. Keep it draft with all three GPU activation labels absent until the diagnostic and runtime gates receive approval.
中文
范围与验证
本草稿直接堆叠于#4722已通过的提交
b6c466048bda2deceb0ce81fd847053c8f9b7599，不修改或合并父分支。新增并发12、16、24、32的HiCache关闭/开启配对性能点，并设置OFF16及ON32两个独立的完整GSM8K准确率任务；准确率任务清除全部模拟接受率变量。仅本子分支升级到官方固定摘要的v0.5.22 ROCm10 mi30x镜像，因为父分支v0.5.21缺少混合模型INDEXER主机恢复修复。两组控制使用相同新镜像，保留原生FP8检查点、BF16 KV、NextN、TP4/EP4、EAGLE5/1/6、完整1M上下文、393条轨迹、3600秒基准及吞吐测试共享黄金接受长度3.603536；保留NONE集合通信和gfx942 mHC回退设置。
开启组使用0.60动态主机内存比例，不设置显式大小或比例，保持内置cgroup感知的组合预算保护。两组仍独占整台8卡节点、384个CPU及全部可分配内存，实际使用4卡；静态
dram-utilization: 1.0仅表示现有名义GPU份额预算，生成器的开启组1208GB/关闭组0并不代表实际物理分配或已分配主机缓存。本地366项测试通过，10个绑定配置及实际命令完成验证。尚未启动GPU任务；仍须验证实际镜像源码指纹、主机内存及NUMA/PCIe信息，并在无模拟接受率条件下完成强制主机恢复和MTP正确性检查；草稿不添加GPU激活标签。
AI assistance disclosure
Implementation, research, and local validation were performed with AI assistance from GPT-6 Astra, including the root coordinator and this lane's engineer. GPU correctness and performance remain unverified for this child; no simulated or local-only check is presented as a GPU pass.