[Klaud Cold][agentic experiment][Variant I] Kimi-K3 B200 agg TP8xPP2 agentic — DSpark spec decoding @ golden AL / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——DSpark 投机解码（黄金 AL）

[functionstackx]

Summary

Agentic experiment (Variant I) — Variant D (#2359) plus DSpark speculative decoding with the committed golden AL.

Identical serving stack to the fully-green Variant D (direct vllm serve via srt-slurm PR #278 + multinode patch, GPU-resident KV, NCCL_CUMEM_ENABLE=1, no allocator changes), with:

• DSpark speculative decoding: --speculative-config '{"model":"Inferact/Kimi-K3-DSpark", "num_speculative_tokens":7, "method": "dspark", "attention_backend": "FLASHINFER_MLA", "rejection_sample_method": "synthetic", "synthetic_acceptance_length": 3.84}'
• Synthetic rejection sampling injecting synthetic_acceptance_length: 3.84 — the committed golden AL, measured under probabilistic draft + block rejection sampling (the Kimi K3 DSpark: probabilistic draft + block rejection sampling #2366 optimization) at num_speculative_tokens=7, per the golden-AL synthetic-acceptance convention, from golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml ([Klaud Cold] feat(speedbench): Kimi-K3 DSpark probabilistic+block sampling AL collector variant / 新增 Kimi-K3 DSpark probabilistic+block 采样 AL 收集器变体脚本 #2368, measured in run 30316471205)
• Concurrency curve 1/2/4/8/16/32 (adds 2 and 4 over D)
• Master entry labels spec-decoding: dspark (new schema value added alongside mtp/draft_model; downstream consumers treat it as an opaque string, so results carry spec-dspark)

Known risks called out up front: the draft head Inferact/Kimi-K3-DSpark resolves via the shared /hf_hub_cache mount — if it isn't cached and compute nodes lack egress, engine init will fail on the draft download.

Related experiments

• Variant D (base, fully green): [Klaud Cold][agentic experiment][Variant D] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve (srt-slurm PR 278) / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve（srt-slurm PR 278） #2359 · Variant F (SimpleCPU offload + retention 0): [Klaud Cold][agentic experiment][Variant F] Kimi-K3 B200 agg TP8xPP2 agentic — SimpleCPU offload + prefix-cache retention / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——SimpleCPU 卸载 + 前缀缓存留存 #2372 · Variant G (retention 0): [Klaud Cold][agentic experiment][Variant G] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve + prefix-cache retention 0 / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve + 前缀缓存留存 0 #2374 · Variant I (this PR): D + DSpark spec decoding @ golden AL · Variant J (I + retention 0): [Klaud Cold][agentic experiment][Variant J] Kimi-K3 B200 agg TP8xPP2 agentic — DSpark @ golden AL + prefix-cache retention 0 / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——DSpark 黄金 AL + 前缀缓存留存 0 #2379
• Closed: A [Klaud Cold][agentic experiment][Variant A] feat: add Kimi-K3 MXFP4 B200 aggregated TP8xPP2 Dynamo-vLLM agentic bring-up / 新增 Kimi-K3 MXFP4 B200 聚合式 TP8xPP2 Dynamo-vLLM 智能体编码基准测试（bring-up） #2355, B [Klaud Cold][agentic experiment][Variant B] Kimi-K3 B200 agg TP8xPP2 agentic — vLLM tool-call-parser flags / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——vLLM tool-call-parser 参数 #2357, C [Klaud Cold][agentic experiment][Variant C] Kimi-K3 B200 agg TP8xPP2 agentic — no parser flags / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——不带解析器参数 #2358, E [Klaud Cold][agentic experiment][Variant E] Kimi-K3 B200 agg TP8xPP2 agentic — direct vllm serve + SimpleCPU KV offload / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——直接 vllm serve + SimpleCPU KV 卸载 #2370

中文说明

智能体实验变体 I——变体 D（#2359）加 DSpark 投机解码与黄金 AL。

服务栈与全绿的变体 D 完全一致（直接 vllm serve、GPU 常驻 KV、NCCL_CUMEM_ENABLE=1、无分配器改动），另加：DSpark 投机解码（Inferact/Kimi-K3-DSpark 草稿头、num_speculative_tokens 7、FLASHINFER_MLA 注意力后端），采用合成拒绝采样注入 synthetic_acceptance_length: 3.84——该黄金 AL 在 probabilistic 草稿 + block 拒绝采样（#2366 优化）下测得（#2368，run 30316471205），符合黄金 AL 合成接受约定；并发曲线扩展为 1/2/4/8/16/32；主配置以 spec-decoding: dspark 标注（schema 新增取值，与 mtp/draft_model 并列；下游按不透明字符串处理，结果带 spec-dspark 标注）。

已预先说明的风险：草稿头需经共享 /hf_hub_cache 挂载解析，若未缓存且计算节点无外网，引擎初始化会在下载草稿模型时失败。

🤖 Generated with Claude Code

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🔴 benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-agentic.yaml:112 — The speculative-config on line 112 sets rejection_sample_method: block together with synthetic_acceptance_length: 3.84, but per golden_al_distribution/README.md and every other spec-decode recipe in the repo, synthetic_acceptance_length is only honored when rejection_sample_method is synthetic — with block, the run performs real (unsimulated) DSpark acceptance instead of forcing the committed golden AL, defeating the PR's stated 'DSpark spec decoding @ golden AL' purpose and violating the AgentX fairness guideline that agentic replay must simulate at the committed curve rather than measure real acceptance. The fix is a one-word change on line 112: "rejection_sample_method": "block" -> "rejection_sample_method": "synthetic".

  Extended reasoning...

  The recipe's speculative-config JSON (line 112) is:

  {"model":"Inferact/Kimi-K3-DSpark", "num_speculative_tokens":7, "method": "dspark", "attention_backend": "FLASHINFER_MLA", "draft_sample_method": "probabilistic", "rejection_sample_method": "block", "synthetic_acceptance_length": 3.84}
  

  It pairs rejection_sample_method: block with synthetic_acceptance_length: 3.84. synthetic_acceptance_length is vLLM's mechanism for forcing a committed golden acceptance length onto a speculative-decode run, but per golden_al_distribution/README.md this field is only wired up under rejection_sample_method: synthetic — the README's own worked example sets "rejection_sample_method": "synthetic" alongside synthetic_acceptance_length, and cites vllm-project/vllm#40662 as the PR that unified this field for that sampler. block selects a completely different code path (real block rejection sampling), under which the field is not consumed.

  This is corroborated by every other analogous config in the repo: all five DeepSeek-V4 GB300 MTP agentic recipes (disagg-gb300-1p1d-*-mtp-agentic.yaml, agg-gb300-tp4/tp8-mtp-agentic.yaml) and the dsv4/kimik2.5 single-node MTP/EAGLE3 bench scripts pair synthetic_acceptance_length with "rejection_sample_method":"synthetic" — there are zero precedents in the codebase pairing it with block. Even more tellingly, the collector script that measured the golden 3.84 value being injected here (benchmarks/single_node/speedbench/kimik3_fp4_b300_vllm_probabilistic_sample_method_block_rejection_sample_method.sh:380) invokes rejection_sample_method: block explicitly without synthetic_acceptance_length — because block is the real-measurement mode used to produce the golden curve, not the injection mode used to reproduce it. The golden YAML's own header (golden_al_distribution/kimik3_dspark_probabilistic_sample_method_block_rejection_sample_method.yaml) confirms this: # speculative-config: draft_sample_method=probabilistic | rejection_sample_method=block, with 3.84 at k=7 recorded as the measured AL from that real-decoding run, not a target for another real-decoding run to reproduce by chance.

  Concretely: with this config, the run performs real DSpark decoding with probabilistic draft sampling and real block rejection sampling against actual head quality on the agentic replay traffic — the exact same code path the collector used to measure the golden curve in the first place, just on different (agentic replay) traffic. Since agentic replay is not representative of the SPEED-Bench coding category used to calibrate the golden curve, the real AL produced here is not guaranteed to match 3.84, and the run silently produces non-comparable results if it happens to differ, with synthetic_acceptance_length: 3.84 sitting inert in the config doing nothing. This is exactly what Check 10 in .github/codeowner-signoff-verify-prompt.md (the repo's agentic spec-decode fairness check) is designed to catch: an agentic config enabling speculative decoding must pin rejection_sample_method: synthetic + synthetic_acceptance_length together, precisely because agentic replay traffic does not reproduce representative token-by-token acceptance behavior.

  One reviewer raised a plausible-sounding defense: that because the golden 3.84 value was itself measured under block, running block again here just repeats the same real measurement and "still yields meaningful golden-AL data." This does not hold up: the golden curve was measured against the SPEED-Bench coding category (the standardized calibration workload), not agentic-replay traffic, so there is no guarantee the same real AL reproduces under different input traffic — that non-reproducibility across workloads is exactly why the synthetic-forcing mechanism exists per the README's fairness guidelines ("a submission may choose any supported draft length, but it may not substitute a different acceptance target... vLLM supports this through synthetic rejection sampling"). Relying on block to coincidentally reproduce 3.84 on a different traffic distribution is precisely the "different acceptance target" substitution the fairness policy is written to prevent, whether or not it happens to land close to 3.84 in practice.

  The PR description itself flags this exact combination as a known, unresolved risk ("if this vLLM build only honors the synthetic AL under rejection_sample_method: synthetic, the first run's log will show it and it's a one-word change"), confirming it is unverified-as-written rather than a deliberate, validated design choice. The fix is a one-word change on line 112: "rejection_sample_method": "block" -> "rejection_sample_method": "synthetic".

[claude]

🔴 The new perf-changelog.yaml entry's pr-link at line 5146 reads .../pull/0 instead of .../pull/2376. This isn't just a wrong link — it will fail validate_added_pr_link() in utils/validate_perf_changelog.py, which requires the link to be either the canonical .../pull/2376 or an XXX placeholder; pull/0 matches neither, so the changelog validation gate will raise ChangelogValidationError on this PR. Fix: change pull/0 to pull/2376.

Extended reasoning...

What the bug is: The changelog entry for kimik3-fp4-b200-dynamo-vllm-agentic added at the end of perf-changelog.yaml ends with:

  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/0

instead of the correct https://github.com/SemiAnalysisAI/InferenceX/pull/2376. This was verified directly against the working tree (not just the PR diff shown in review context) — sed -n '5146p' perf-changelog.yaml confirms pull/0 is what's actually committed at HEAD.

How it happened: git blame traces this line to the merge commit ef94613 (merging origin/main into this branch), which reports a conflict in perf-changelog.yaml. The original feat commit (245e36e) had set the line to a pull/XXX placeholder (a valid, allowed value per the validator — see below), intending it to be filled in with the real PR number, 2376, before merge. The subsequent conflict resolution against main clobbered this into pull/0 instead of either keeping the XXX placeholder or filling in 2376. Notably, the PR's own diff/description show the intended value as pull/2376, and other files changed in this same PR (MODELS.md, MODELS_zh.md) correctly reference #2376 — so pull/0 is clearly an artifact of the merge, not an intentional convention. A repo-wide grep confirms pull/0 doesn't appear anywhere else in perf-changelog.yaml.

Why this breaks CI, not just metadata: utils/validate_perf_changelog.py defines validate_added_pr_link(link, pr_number) (called with pr_number=2376 during this PR's CI run) which raises ChangelogValidationError unless the link is exactly f\"https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}\" or one of PR_LINK_PLACEHOLDERS = {'XXX', '.../pull/XXX'}. I read the function directly:

def validate_added_pr_link(link: str, pr_number: int | None) -> None:
    ...
    expected = f\"https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}\"
    if link not in PR_LINK_PLACEHOLDERS and link != expected:
        raise ChangelogValidationError(
            f\"new PR entry must use {expected!r} or an XXX placeholder; \"
            f\"found {link!r}\"
        )

With pr_number=2376, expected is .../pull/2376. The committed value .../pull/0 is neither the expected link nor in PR_LINK_PLACEHOLDERS, so this function raises ChangelogValidationError and the changelog validation gate fails the PR run.

Step-by-step proof:

1. git show HEAD:perf-changelog.yaml | tail -1 → pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/0.
2. git blame on that line → attributed to merge commit ef94613.
3. git show 245e36e:perf-changelog.yaml | tail -1 (the feat commit before the merge) → pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX (a valid placeholder).
4. CI invokes validate_added_pr_link(link='https://github.com/SemiAnalysisAI/InferenceX/pull/0', pr_number=2376).
5. expected = 'https://github.com/SemiAnalysisAI/InferenceX/pull/2376'; link is neither expected nor in {'XXX', '.../pull/XXX'} → ChangelogValidationError is raised, failing the gate.

Fix: Change pull/0 to pull/2376 (or, if the PR number isn't final yet, revert to the pull/XXX placeholder, which the validator explicitly allows).

[claude]

🟡 The bullet at perf-changelog.yaml:5139 ends with '...conc 1/8/16/32', carried over verbatim from the prior Variant D changelog entry, but the actual conc-list added in configs/nvidia-master.yaml for this config is [1, 2, 4, 8, 16, 32]. This directly contradicts the last bullet in the same changelog entry, which correctly says 'Conc curve widened to 1/2/4/8/16/32' — please update the stale bullet to match.

Extended reasoning...

The kimik3-fp4-b200-dynamo-vllm-agentic entry added in this PR to perf-changelog.yaml contains two bullets that describe the same concurrency sweep but state different values. The bullet at line 5139 reads:

"...max-num-batched-tokens 8192 so a single long prefill cannot OOM a pipeline stage; conc 1/8/16/32"

while the entry's final bullet (line ~5144) reads:

"Conc curve widened to 1/2/4/8/16/32"

The conc-list actually added to configs/nvidia-master.yaml for this config is [1, 2, 4, 8, 16, 32] — matching the second bullet, not the first. The 1/8/16/32 string in the first bullet was clearly copy-pasted verbatim from the prior Variant D changelog entry (which predates the addition of the 2 and 4 concurrency points) and simply never got updated when the conc-list was widened for this variant.

Step-by-step proof:

1. configs/nvidia-master.yaml (this PR's diff) sets conc-list: [1, 2, 4, 8, 16, 32] for kimik3-fp4-b200-dynamo-vllm-agentic.
2. perf-changelog.yaml line 5139 (same PR, same changelog entry) says "...conc 1/8/16/32" — missing 2 and 4.
3. perf-changelog.yaml line ~5144 (same entry, later bullet) says "Conc curve widened to 1/2/4/8/16/32" — matches the actual conc-list.
4. Bullets 1 and 6 of the same changelog entry now contradict each other about what concurrencies were swept.

This is confined entirely to human-readable changelog prose. utils/validate_perf_changelog.py only validates structure, ordering, and the PR link — it does not cross-check bullet text against the conc-list in the master config, so there is no CI enforcement that would have caught this, and none will break as a result of leaving it as-is. The authoritative, machine-consumed value (nvidia-master.yaml's conc-list) is correct, and a reader who reaches the entry's last bullet gets the correct information anyway.

The fix is a one-word/one-string edit: change "conc 1/8/16/32" to "conc 1/2/4/8/16/32" (or simply drop the redundant conc mention from the first bullet since the last bullet already states it correctly) so the entry is internally consistent.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30329418996
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30329418996

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30334260868
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30334260868

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30336669249
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30336669249

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30338558261
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30338558261

[functionstackx]

Closing the DSpark experiment per the author's decision. State at close: the full DSpark-under-PP2 patch stack is on this branch (mamba_hybrid dtype fix, pp=1 draft verification, cross-stage aux-hidden-state transport for target_layer_ids [2, 23, 47, 71, 89], and the runner guard lift) — reusable if the experiment is revisited, e.g. once upstream lands spec-under-PP (vllm RFC #44697) or on a single-node TP8 SKU where no PP is involved. In-flight sweep cancelled.

中文：按作者决定关闭 DSpark 实验。关闭时状态：本分支已包含完整的 DSpark 流水线并行补丁栈（mamba_hybrid 类型修复、pp=1 草稿校验、跨阶段辅助隐状态传输、runner 防护解除），若日后重启实验（如上游落地 vllm RFC #44697，或改用无流水线并行的单节点 TP8 SKU）可直接复用。进行中的扫描已取消。