Dev/jax maxtext v26.6 primus cli - #979
Open
amd-fuyuajin wants to merge 6 commits into
Open
Conversation
Copied from the MI300X set, then tuned only per_device_batch_size on tus1-p3-g26 (8x MI325X, 256GB HBM vs 192GB on MI300X) so the suite uses the memory the larger cards actually have instead of inheriting MI300X limits. Each value is the largest that completed 3 training steps against the 230.4GB per-device budget. 12 of the 19 tuned configs are pinned by a real OOM one step higher; the remaining 7 stopped at the sweep's attempt cap while still passing and are annotated in-file as not OOM-pinned, so they can be pushed further. gemma4_26B bf16 in particular still has wide headroom (130.5GB peak). grok1, gemma4_26B and gemma4_31B nanoo_fp8 were excluded from the sweep and keep their MI300X batch sizes. Co-authored-by: Cursor <cursoragent@cursor.com>
MaxDiffusion's step line carries step time, TFLOP/s/device and loss but no
throughput, so benchmark runs had no sample or token rate to compare across
resolutions and clip lengths.
Patch train_utils at the before_train phase to add Samples/s/device,
Frames/s/device and Tokens/s/device to the step line and to
metrics["scalar"], so the rates reach TensorBoard and metrics_file as well.
Token counts are rebuilt from the same latent geometry
calculate_{wan,flux}_tflops bills its FLOPs for, and are omitted for families
with no token notion (SD 1.x/2.x, SDXL). Tokens/s/device follows MaxText's
naming so existing log parsers apply unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
Upstream reprints "To see full metrics 'tensorboard --logdir=...'" on every log_period, and the benchmark configs set log_period: 1, so the hint doubled the length of the step log to repeat a string that never changes. Emit it on the first step only. writer.flush() stays on log_period, so flush cadence is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
The comment claimed log_period gates the per-step "completed step: ..." line that the perf extractor parses, and that the default of 100 left a 20-step run with no perf data. That line is emitted unconditionally, so log_period never had that effect; the empty-perf symptom came from max_logging landing at DEBUG and being dropped by the console sink, fixed separately in f059e02. Describe what log_period actually controls (tensorboard flush cadence) and keep it at 1 so a run killed early still leaves a complete event file. Co-authored-by: Cursor <cursoragent@cursor.com>
amd-fuyuajin
requested review from
Xiaoming-AMD,
limou102 and
wenxie-amd
as code owners
August 12, 2026 14:40
Collaborator
Author
|
@yeandy Please help to review this PR. |
…imit Co-authored-by: Cursor <cursoragent@cursor.com>
code-lint runs `pre-commit run --all-files`, so it fails on debt anywhere in the repo, not just on the diff under review. These files were pushed straight to the branch where the PR check never ran, so the failures only surface now that a PR is open. Mechanical only: black rejoins lines that were formatted at its default 88 columns instead of the 110 configured in pyproject.toml, autoflake drops two imports that are referenced nowhere and are not re-exported (typing.Dict, ARCH_ALL), and the three maxdiffusion model YAMLs lose trailing whitespace and gain EOF newlines with no value changes. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent pieces of work. Both are confined to
examples/andprimus/backends/; nothird_party/submodule is touched.MI325X maxtext configs (22 files). Copied from the MI300X set, then
per_device_batch_sizetuned on tus1-p3-g26 (8x MI325X, 256GB HBM vs 192GB onMI300X) so the suite uses the memory the larger cards actually have instead of
inheriting MI300X limits. Each value is the largest that completed 3 training
steps against the 230.4GB per-device budget. 12 of the 19 tuned configs are
pinned by a real OOM one step higher; the remaining 7 stopped at the sweep's
attempt cap while still passing and are annotated in-file as not OOM-pinned.
grok1, gemma4_26B and gemma4_31B nanoo_fp8 were excluded from the sweep and keep
their MI300X batch sizes.
MaxDiffusion throughput metrics. The step line carried step time, FLOP rate
and loss but no throughput, so benchmark runs had no sample or token rate to
compare across resolutions and clip lengths.
Before:
After:
Samples/s/device— videos or images per second per GPU. Defined for everymodel family.
Frames/s/device—Samples/s/device * num_frames, emitted only for familieswhose config describes frames.
Tokens/s/device— patchified latent tokens per second per GPU: 79,200 for a720x1280x85 WAN video, 1,024 for a 512px FLUX image. Rebuilt from the same
latent geometry
calculate_{wan,flux}_tflopsbills its FLOPs for, and omittedfor the UNet families (SD 1.x/2.x, SDXL) which have no equivalent notion.
Named to match MaxText's step line so existing log parsers apply.
This is implemented as a
before_trainpatch underprimus/backends/maxdiffusion/patches/, the same mechanism as the existinglogger patch, rather than editing
third_party/maxdiffusion, so it survivessubmodule updates. The values are added to
metrics["scalar"]before anyconsumer runs, so they also reach TensorBoard and
metrics_filealongside thecluster-wide
perf/samples_per_second,perf/frames_per_secondandperf/tokens_per_secondtotals.Two follow-ups in the same area: the tensorboard hint is now logged once per run
instead of on every
log_period(the configs pinlog_period: 1, so it wasdoubling the length of the step log), and the
log_periodcomment in the sixmaxdiffusion configs is corrected — it claimed the value gated the
completed step:line, which it never did.Verification
count reproduces the trainer's own
Calculated TFLOPs per pass: 4085.7969to arelative error of 3.7e-9, confirming the token rate and the FLOP rate describe
the same sequence.
consecutive log lines, so 0.26% falls outside the measured window.
queue, and the double-buffered step-to-metric pairing is preserved.
per_device_batch_sizewarns once, disables, and falls back to thebyte-identical upstream line.
Test plan
wan2.1_1.3b on MI300X, 20 steps: 348.88 TFLOP/s/device, 6762.7
Tokens/s/device, 7.258 Frames/s/device
MI325X 1-node suite, steady-state means over steps 2-18:
MI355X maxdiffusion configs not re-run; only a comment changed there.
Note for whoever aggregates these numbers
Warmup must be excluded. Steps 0 and 1 are compilation (368s and 1032s in the
MI300X run), and averaging them in skews step time by +628% and the rates by
-10.3%.
tools/daily/daily_report.pycurrently averages every matched step.