MTP greedy parity vs llama.cpp --spec-type draft-mtp

[pekkah]

Background

CPU MTP foundation (#25, commit a28bac4) ships a smoke test that asserts finite + non-degenerate MTP head logits, but does NOT verify the actual emitted tokens match llama.cpp's --spec-type draft-mtp --spec-draft-n-max 2 output. Without parity, we don't know if the MTP head is correctly wired (concat order, eh_proj orientation, partial-RoPE on the MTP attn block, etc.).

Scope

1. Capture reference output from llama-cli (or llama-server greedy) on the Qwen3.6-27B-MTP file with --spec-type draft-mtp --spec-draft-n-max 2, greedy, fixed seed, fixed prompt, >=60 tokens.

2. Capture our output from sharpi-cli (when CLI MTP routing lands — depends on #(cli-routing-issue), or via a test that drives InferenceEngine.GenerateAsync directly) on the same model + prompt + greedy. Confirm SHARPI_TRACE_MTP shows MTP was actually used.

3. Compare token-for-token. Expect:

   • Bit-identical when MTP is disabled (SHARPI_DISABLE_MTP=1) vs llama.cpp with --spec-draft-n-max 0.
   • Bit-identical or within Q4_K_M roundoff when MTP is enabled vs llama.cpp with --spec-draft-n-max 2 — because both use greedy correction, divergences in the MTP draft path don't affect emitted tokens.

4. If mismatched, bisect using SHARPI_TRACE_LAYERS=1 and llama.cpp's eval-callback dump. Likely culprits documented in docs/qwen35moe-plan.md Phase 5 (concat order, eh_proj orientation, partial RoPE on MTP attn).

Acceptance criteria

• Reference dump from llama.cpp captured + checked in under tests/fixtures/mtp_parity_27b.txt (or similar).
• New test in Tests.ForwardPass or a new Tests.Mtp project: MtpDecoder_GreedyParity_LlamaCpp reads the fixture and asserts >= 60 byte-identical tokens.
• Test silently skips when the 27B-MTP file isn't on disk (mirror existing HybridGdnForwardPassTests pattern).

Out of scope

• Speedup measurement — separate issue.
• Multimodal / vision parity — model has text-only fixture.

Related

• Parent: MTP / qwen3_next_mtp self-speculation for Qwen3.6 hybrid GDN #25
• Depends on: CUDA hybrid MTP — mirror CPU MTP foundation on CudaHybridGdnForwardPass #29 (CUDA hybrid MTP) only for parity of the GPU path; CPU parity is testable today.

[pekkah]

Partial progress: commit 82547e9 added `InferenceEngine_MtpGreedy_MatchesBaselineGreedy_OnCpu` — an end-to-end self-parity test that drives the real 27B-MTP model through `InferenceEngine.GenerateAsync` twice (MTP on / `SHARPI_DISABLE_MTP=1`) and asserts byte-identical emitted text. This proves the MTP code path doesn't corrupt main state and emits the same greedy sequence MTP-disabled does.

What's still missing for the full acceptance criterion:

1. External llama.cpp reference dump. Self-parity proves "MTP on == MTP off" on our side, but doesn't prove our "MTP off" matches llama.cpp's "draft-mtp off". The first-token test `CudaHybridGdnForwardPass_Qwen35Moe_FirstTokenMatchesCpuBaseline` already aligns the trunk against llama.cpp b9245 for the 35B-A3B model; need the equivalent for the 27B-MTP file + extending past step-0.

2. MTP-on vs llama.cpp `--spec-type draft-mtp` parity. This is where MTP attention KV cache is empty during decode-step 1 (prefill doesn't populate it) #33 (MTP attention KV cache not populated during prefill) likely matters most — without that fix, our MTP head sees a different attention history than llama.cpp's, so even though both emit `argmax(target_logits)`, intermediate state diverges and could trip subtle bugs in the verify path.

Recommended dependency order: land #33 first, then capture the llama.cpp fixture, then add the comparison test.

[pekkah]

Resolved earlier — MtpDecoder_GreedyParity_LlamaCpp covers this. Captures llama.cpp continuation_prefix in tests/fixtures/mtp_parity_27b.json; sharpi must match for >=min_match_bytes bytes. Test is green on master; this session's #30/#42 commits did not regress it.