Repository navigation
[DNM, Experimental] feat(bench): GLM-5.3-Flash GB300 FP8 HiCache concurrency / [DNM, Experimental] feat(bench): GLM-5.3-Flash GB300 FP8 HiCache 并发 - #4732
Draft
functionstackx wants to merge 3 commits into
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
functionstackx
changed the title
functionstackx
added
the
full-sweep-fail-fast
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41352868017 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41352868017 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment
This draft is a child of PR #4715, preserving passing parent head
c611580c42f9f8975d260d1a0a567c38e188d7d8. It adds matched OFF/HiCache-ON AgentX points at concurrency 40, 48, 56, 64, and 80, plus a dedicated ON canonical-evaluation selector.Both arms keep the parent's official immutable SGLang v0.5.22-cu130 image, native FP8 checkpoint, TP4/EP1, full 1,048,576-token context, full 393-entry corpus, 3,600-second duration, FP8 E4M3 KV cache, and embedded EAGLE MTP 5/1/6. There is no max-running-requests override: the parent's observed server cap was 48, so higher client concurrency includes queued load rather than promising that many resident requests.
Only the ON recipe adds hierarchical caching with
hicache-host-memory-fraction: 0.90,write_through,kernelI/O, andpage_firstlayout. No fixed host size/ratio, engine patch, draft-precision override, or workload reduction is introduced.Source and memory qualification
Exact source
baa9befa381511cd4d56f83980951e5a45fffc6dincludes the required hybrid INDEXER fix, compressed-page ownership fix, and hybrid MLA load-back wait. Matching target/draft KV pools use the existing packed host path; this source audit is not GPU host-restore correctness proof.The builtin automatic allocator uses live available memory and cgroup room, a 10 GiB node reserve, local-rank sharing, 5% sizing slack, and a shared host allocation budget. No explicit size/ratio is set, avoiding the fixed-size INDEXER-budget problem described in open PR #42918.
dram-utilization: 1.0is the existing nominal eligible offload-budget metadata, not the allocator's 0.90 fraction, actual pinned-memory use, or an additional Slurm RAM reservation. OFF rows retain the generator's zero offload-budget metadata; both arms preserve identical parent Slurm resources. Actual resolved host ratio, pool bytes, available/cgroup memory, and NUMA placement still need runtime evidence.Validation and dispatch gate
The current workflow selects one global minimum-concurrency canary, not one canary per OFF/ON arm. The performance changelog block disables eval selection, and a separate ON eval-only block prevents same-concurrency OFF/ON eval deduplication from silently choosing OFF.
Real host-restore qualification is pending. An isolated EVAL_ONLY prelude must prove an actual host-only restore with simulation absent, compatible target/draft INDEXER and recurrent-state transfers, and deterministic parity before clearing diagnostic cache state and running the unchanged canonical evaluation. A missing host hit is inconclusive, not a pass; ordinary GSM8K alone cannot rule out the reported restore defects.
The parent also had unresolved empty-stream request errors, so subsequent frontier comparisons must preserve phase-separated warmup/profiling errors and completion counts. No error-free or improved-frontier claim is made here.
AI model disclosure
GPT-6 Astra researched the pinned SGLang host-cache paths, prepared this GB300 FP8 child configuration, and ran local validation. The source audit was coordinated with other lane owners; no GPU measurements were fabricated.
中文
本草稿直接基于 PR #4715 的已通过提交，新增并发 40、48、56、64、80 的 OFF/HiCache-ON 成对实验，并提供独立的 ON 标准评测配置。保留官方不可变镜像、原生 FP8、TP4/EP1、完整 1,048,576-token 上下文、393 条目语料、3,600 秒时长及 EAGLE MTP 5/1/6。
ON 仅启用 HiCache、0.90 自动主机内存比例、
write_through、kernel和page_first，不设置固定容量或比例，不修改引擎、推测头精度或工作负载。dram-utilization: 1.0是原有 nominal offload-budget 元数据，不代表实际 pinned memory 或新增 Slurm 内存预留；OFF 的零值表示没有 offload budget，不表示未分配物理主机内存。已生成十个性能配置和独立 ON 评测配置，并通过 265 项本地定向测试。当前没有新的 GPU 结果，保持 draft 且不添加 GPU 激活标签，待矩阵和真实 host-only restore 诊断完成审查。现有 workflow 只有一个全局最低并发 canary，不能据此声称 OFF/ON 均已通过独立 canary。
真实 restore 诊断必须在关闭模拟接受率的独立评测分配中证明 host read、INDEXER/循环状态正确传输及确定性结果一致，然后清空诊断缓存并执行原始标准评测。没有实际 host hit 属于诊断不充分，不能视为通过；当前也不声称无错误或性能前沿已经改善。