feat(ds4): add default-off HIP GPU phase profiler#554
Open
cheese-cakee wants to merge 11 commits into
Open
Conversation
DFLASH_DS4_GPU_PROFILE=1 scopes HIP event timing around exact verification phases (hc_pre, attention, moe_ffn, hc_post, output_projection), the whole verification step, fused AR decode, and the explicitly approximate fused research graph. ROCTX ranges are emitted when libroctx64.so is present; it is loaded dynamically, so there is no compile- or link-time dependency. Unset and explicit 0 create no events, synchronize nothing, and emit no records.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 10 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
end(Count) while idle passed the active-phase check and indexed one past the elapsed/calls arrays. Match the sentinel guard begin() already has.
Records from deepseek4_step_layer_range now carry layers=<begin>-<end> so per-shard timings are distinguishable in layer-split runs. Single-shard and whole-graph scopes omit the field.
emit() no longer infers the exact-verification zero-call policy from the caller's scope/mode strings; the exact verify call site passes emit_zero_core directly.
The documented regeneration grep only matches literal getenv calls; flags read through wrapper helpers such as ds4_env_flag are curated manually.
begin() now runs before the per-phase Ds4TimingClock window and end() after the telemetry accumulation, so enabling DFLASH_DS4_GPU_PROFILE no longer inflates the pre-existing host telemetry counters with event synchronization. GPU event ordering is unchanged. The output-projection block keeps its bracket because tensor_get enqueues device work after compute.
DFLASH_DS4_PARITY_TRACE scans the full vocabulary for every verified row and prints one stderr line per row; document it as a short-run debug tool in the documented table and add it to the wrapper-curated inventory.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Group scope, mode, n_tokens, kv_start, layer range, and emit_zero_core into Ds4GpuProfileOptions. Call sites assign fields by name, so the growing option list can no longer be misordered positionally.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in HIP GPU phase profiler for DeepSeek4 behind
DFLASH_DS4_GPU_PROFILE(default off). rocprofv3 does not work on gfx1151, so per-phase GPU timing for the DS4 HIP path did not exist; this provides it with zero cost when disabled.Stacked on #548: the profiler instruments the exact-versus-approximate verification boundary that #548 established, so this branch carries #548's two commits until it merges. The remaining diff after #548 merges is this feature alone.
Changes
deepseek4_gpu_profiler.{h,cpp}(new): env parsing via the existing boolean helpers, reusable HIP event pairs per scope, dynamiclibroctx64.soloading for matching ROCTX ranges, stable[ds4-gpu-profile] clock=hip_eventrecords.deepseek4_graph.cpp: scopes aroundhc_pre,attention,moe_ffn,hc_post, andoutput_projectionfor exact verification forwards, plus a whole-graph scope for fused AR decode.deepseek4_dspark_spec.cpp: whole verification step scope (exact batched and sequential paths).deepseek4_fused_verify.inc: scope for the explicitly approximate fused research graph.CMakeLists.txt,docs/DS4.md,docs/ENVIRONMENT.md: build wiring and flag documentation.How it works
0are both disabled: no HIP events created, no synchronization, no ROCTX library loaded, no log lines. The implementation is compiled in on all backends but event creation is#if defined(GGML_USE_HIP)guarded; non-HIP builds are no-ops.gpu_msandcalls. Whole-graph paths (fused AR decode, approximate fused verifier) are one captured graph each, so they are honestly timed as whole graphs (fused_decode_graph,approx_fused_verify_graph).libroctx64.sois found at runtime; there is no compile-time or link-time requirement.Performance
Overhead (gfx1151, ROCm 7.2.4, fixed prompt/seed/q=4, 1 warmup + 5 measured requests per arm): profiler unset median 3397.8 ms decode, explicit
03429.1 ms (+0.92%, within run noise), enabled 3461.8 ms. All three arms produced the identical 32 generated token IDs.Example output it produces (exact q=4 verification, mean over 6 enabled requests): whole step 260.711 ms; attention 140.750 ms over 172 calls (71.5% of timed GPU subphases),
moe_ffn53.590 ms over 43 calls (27.2%),output_projection2.560 ms (1.3%); multi-token HC pre/post report 0 GPU calls because they execute on the host.Limitations
gpu_ms=0 calls=0;DFLASH_DS4_TIMINGremains the host-wall view for those phases.deepseek4_step_layer_range,ds4_try_fused_decode_step, and the dspark verify path; open PRs touching the same functions (perf(ds4): compute greedy argmax on GPU (monolithic and layer split) #531, feat(ds4): opt-in GPU F16 HC fn mirrors for ROCmFPX #546, feat(ds4): Lucebox heterogeneous expert parallel + DSpark verify #505) may need a textual rebase depending on merge order, but there is no semantic dependency.Verification
/opt/models/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf(SHA-256efc7ed607ff27076e3e501fc3fefefa33c0ed8cf1eff483a2b7fdc0c2e616668), draftds4-dspark-draft.gguf(SHA-25648883d35b8a67ecfd2858a90e12a47d04cb5ac581acef868ca0f58544816f746).GGML_CUDA_GRAPHS=ON,GGML_CUDA_FA=OFF,GGML_HIP_NO_VMM=ON,GGML_HIP_MMQ_MFMA=ON,DFLASH27B_HIP_SM80_EQUIV=OFF,GGML_HIP_ROCWMMA_FATTN=OFF);dflash_serverandtest_deepseek4_unitbuild and pass. Unset /=0/=1arms each 1 warmup + 5 measured requests: identical 32 IDs in every arm, zero[ds4-gpu-profile]lines when disabled, expected records when enabled. A sequential one-token exact forward baseline was also recorded. Local CUDA 12.6/sm_89 compile ofdflash_commonandtest_deepseek4_unitpassed.62880e2: incremental HIP rebuild passed,test_deepseek4_unitpassed, and a two-arm smoke (unset vs=1, same fixed request) reproduced identical 32 IDs[2337 344 260 2405 3417 14 305 270 3287 344 850 4190 1099 436 6179 16 2454 7367 1664 834 4031 3986 14 790 4562 260 8682 294 5131 8753 305 2883], 0 profile lines unset vs 103 enabled.git diff --checkclean.