Skip to content

feat(onnx-to-hip): fold attention score prep Mul and broadcast Add - #658

Draft
amd-mounikk wants to merge 6 commits into
mainfrom
feat/attention-score-prep-opt
Draft

feat(onnx-to-hip): fold attention score prep Mul and broadcast Add#658
amd-mounikk wants to merge 6 commits into
mainfrom
feat/attention-score-prep-opt

Conversation

@amd-mounikk

@amd-mounikk amd-mounikk commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add pre-lowering patterns for Swin-style window-attention score preparation before lowerOnnxConstants.
  • FoldMatMulScaleMul: rewrites MatMul(A,B) -> Mul(scores, scale) to Mul(A, scale) -> MatMul(A', B), avoiding a broadcast mul on large score tensors.
  • ExpandConstantBroadcastAdd: compile-time expands broadcast constant biases so Add(activation, Constant[bias]) becomes same-shape add, avoiding runtime broadcast elementwise on large attention maps.
  • Review fixes (3bfb091): MatMul scale fold guarded by K/N extent check; bias expansion capped at 1 Mi output elements.
  • Includes LIT coverage in test_attention_score_prep_opt.mlir (positive + negative cases for both patterns).

Test plan

  • pre-commit run --from-ref origin/main --to-ref HEAD (lintrunner + licenseheaders)
  • hip-mlir-opt test/lit/Conversion/onnx-to-hip/test_attention_score_prep_opt.mlir (FileCheck pass)
  • SwinV2 tiny 8x521x1024 end-to-end perf (local): avg ~274 ms/inference, matmul 27 calls / 31.8 ms GPU

Add pre-lowering patterns that run before lowerOnnxConstants:

- FoldMatMulScaleMul: MatMul(A,B)->Mul(scores,scale) becomes Mul(A,scale)->MatMul

- ExpandConstantBroadcastAdd: compile-time expand broadcast constant biases to same-shape Add

Includes LIT coverage for both optimizations.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Thanks for opening a PR!

This project follows LLVM's incremental-development and AI-tool-use
guidance. See CONTRIBUTING.md
for the project workflow.

Before requesting review, please check that:

  1. The change is focused. Substantial work links the relevant issue
    or design discussion.
  2. The PR documents relevant test results and updates affected
    documentation.
  3. If AI tools provided substantial assistance, the description
    explains what was assisted and how it was validated, and commit
    trailers identify the tool. The contributor has reviewed and
    understands the result.

Reviewers are assigned through
CODEOWNERS where ownership
is configured.

The full convert-onnx-to-hip pipeline lowers expanded onnx.Constant to arith.constant before hip.add. Update FileCheck to match actual output.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

L2 Accuracy Results (EP vs CPU)

Model Combined L2 Total Elems Skipped NaN/Inf
conv_test_hybrid 4.8668E-07 64 0
GroupQueryAttention_seq256 25.2366 2621440 0
MatMulNBits_o_seq128 259.906 368640 0
QMoE_seq128 34.9552 368640 0

Threshold: 0.01 | Run: 3998 - Commit: d3aa112

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

MorphiZen EP Performance Results

Model QPS Session (s) 1st Infer (ms) CPU% Mem (MB)
full_model_seq128 7.50 6.36 360 3 1243
GroupQueryAttention_seq128 4559.40 1.71849 11 6 313
matmul_down_seq128 495.18 2.38 71 3 354

EPContext Export Performance

Model QPS Session (s) 1st Infer (ms) CPU% Mem (MB)
full_model_seq128 7.58 45.38 361 3 15589

EPContext Import Performance

Model QPS Session (s) 1st Infer (ms) CPU% Mem (MB)
full_model_seq128 7.52 9.80 358 3 15761

OGA Benchmark Results

Model Warmup Reps Prompt Len Gen Tokens TTFT (ms) TPS Peak Mem (GB) GPU Mem (GB)
gpt-oss-20b-webgpu-int4-rtn-block-32 1 5 128 128 180.9 80.5 1.33 13.53
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml 1 5 128 128 272.3 40.2 1.22 6.43

OGA Wheel Smoke (Python benchmark_e2e.py)

Model TTFT (ms) TPS
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml 192 39.4

Run: 3998 - Commit: d3aa112

@amd-mounikk
amd-mounikk marked this pull request as ready for review August 4, 2026 15:30
Comment thread lib/Conversion/OnnxToHip/AttentionScorePrepOptimizations.cpp
Comment thread lib/Conversion/OnnxToHip/AttentionScorePrepOptimizations.cpp
amd-mounikk and others added 3 commits August 11, 2026 22:38
Tighten FoldMatMulScaleMul so scale must be constant along MatMul K and N, rejecting column-scale patterns that would change numerics if folded onto A. Cap ExpandConstantBroadcastAdd at 1 Mi output elements to avoid materializing multi-GB broadcast constants at compile time. Add lit tests for both cases.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@amd-mounikk
amd-mounikk marked this pull request as draft August 12, 2026 21:26

@fhanuman fhanuman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will approve the PR to unblock the merge. However, I am still not clear on how much this change improves performance, so in future PRs like this, it would be helpful to include an example showing where it helps and some before-and-after performance numbers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants