diff --git a/docs/04-technical-guides/collective-operations.md b/docs/04-technical-guides/collective-operations.md index e1713f40a..afcc606b8 100644 --- a/docs/04-technical-guides/collective-operations.md +++ b/docs/04-technical-guides/collective-operations.md @@ -202,7 +202,7 @@ The following appear in ROCm / AMD deployments and partner integrations; availab |---------|--------| | **MSCCL** | Microsoft Collective Communication Library: **custom algorithms** and patterns; might be used when the stack is built and configured for them. | | **MSCCL++** | User-space collective paths aimed at **lower latency** for specific patterns and hardware. | -| **ANP (AMD Network Plugin)** | Network backend integration (e.g. **AINIC**-oriented paths). Example: `NCCL_NET_PLUGIN` might point to `librccl-anp.so` or similar when installed (see Primus `examples/run_pretrain.sh` patterns). | +| **ANP (AMD Network Plugin)** | Network backend integration (e.g. **AINIC**-oriented paths). Example: `NCCL_NET_PLUGIN` might point to `librccl-anp.so` or similar when installed (see Primus `runner/helpers/hooks/03_enable_ainic.sh`). | ### Environment variables diff --git a/docs/04-technical-guides/determinism-and-reproducibility.md b/docs/04-technical-guides/determinism-and-reproducibility.md index 44c60c244..5cc0f10fd 100644 --- a/docs/04-technical-guides/determinism-and-reproducibility.md +++ b/docs/04-technical-guides/determinism-and-reproducibility.md @@ -1,6 +1,6 @@ # Determinism and reproducibility -Reproducibility—getting bit-identical (or run-to-run stable) results—matters for debugging divergence, validating optimizations, and regression testing. This guide covers Primus's deterministic mode, the environment variables it sets, the per-backend seed/determinism knobs, and the performance trade-offs. Parameters and behavior are grounded in `examples/run_pretrain.sh`, `primus/configs/modules/megatron/trainer_base.yaml`, and `primus/configs/modules/torchtitan/pre_trainer.yaml`. +Reproducibility—getting bit-identical (or run-to-run stable) results—matters for debugging divergence, validating optimizations, and regression testing. This guide covers Primus's deterministic mode, the environment variables it sets, the per-backend seed/determinism knobs, and the performance trade-offs. Parameters and behavior are grounded in `runner/helpers/envs/base_env.sh`, `runner/helpers/hooks/05_deterministic.sh`, `primus/configs/modules/megatron/trainer_base.yaml`, and `primus/configs/modules/torchtitan/pre_trainer.yaml`. --- @@ -17,7 +17,7 @@ Full determinism also generally requires the **same world size, parallelism layo ## 2. Primus deterministic mode (`PRIMUS_DETERMINISTIC`) -Setting `PRIMUS_DETERMINISTIC=1` configures the GPU/communication stack for deterministic behavior. The CLI/runner path applies this through the hook `runner/helpers/hooks/05_deterministic.sh`; the `examples/run_pretrain.sh` script applies an equivalent inline block. The exported variables are: +Setting `PRIMUS_DETERMINISTIC=1` configures the GPU/communication stack for deterministic behavior. Every launcher mode (`direct`, `container`, `slurm`) applies this through the same hook, `runner/helpers/hooks/05_deterministic.sh`, which runs before training starts. The exported variables are: ```bash # when PRIMUS_DETERMINISTIC=1 (runner/helpers/hooks/05_deterministic.sh) @@ -28,15 +28,15 @@ export TORCH_COMPILE_DISABLE=1 # avoid torch.compile/Triton race con export PRIMUS_TURBO_AUTO_TUNE=0 # disable Primus-Turbo autotuning (stable kernel choice) ``` -> `PRIMUS_TURBO_AUTO_TUNE` also defaults to `0` in `runner/helpers/envs/base_env.sh`. The inline block in `examples/run_pretrain.sh` sets the first four variables and relies on that default for the fifth. +> `PRIMUS_TURBO_AUTO_TUNE` also defaults to `0` in `runner/helpers/envs/base_env.sh`, which the hook relies on rather than re-exporting. -Additionally, **HipBLASLt autotuning is disabled** in deterministic mode: tuning only runs when `PRIMUS_DETERMINISTIC != 1` *and* `PRIMUS_HIPBLASLT_TUNING=1` (`examples/run_pretrain.sh`). This prevents run-to-run kernel-selection differences. See [Performance tuning](./performance-tuning.md). +Additionally, **HipBLASLt autotuning is disabled** in deterministic mode: tuning only runs when `PRIMUS_DETERMINISTIC != 1` *and* `PRIMUS_HIPBLASLT_TUNING=1` (`runner/helpers/hooks/train/pretrain/prepare_experiment.sh`). This prevents run-to-run kernel-selection differences. See [Performance tuning](./performance-tuning.md). `PRIMUS_DETERMINISTIC` is on the container passthrough allowlist (`runner/.primus.yaml`), so it reaches the training container. See [Environment variables](../03-configuration-reference/environment-variables.md). ```bash export PRIMUS_DETERMINISTIC=1 -./runner/primus-cli direct -- train pretrain \ +./primus-cli direct -- train pretrain \ --config examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml ``` diff --git a/docs/04-technical-guides/diffusion-models/STRUCTURE.md b/docs/04-technical-guides/diffusion-models/STRUCTURE.md index 9bed81156..f83c67984 100644 --- a/docs/04-technical-guides/diffusion-models/STRUCTURE.md +++ b/docs/04-technical-guides/diffusion-models/STRUCTURE.md @@ -97,7 +97,7 @@ Primus/ │ │ └── ... │ └── prepare.py │ -├── examples/run_pretrain.sh # Main training script +├── runner/primus-cli # Main training launcher (direct/container/slurm) │ ├── tests/ │ ├── unit_tests/backends/megatron/diffusion/ # Unit test suite diff --git a/docs/04-technical-guides/diffusion-models/adding_new_models.md b/docs/04-technical-guides/diffusion-models/adding_new_models.md index 1c726c0cd..1e6a98273 100644 --- a/docs/04-technical-guides/diffusion-models/adding_new_models.md +++ b/docs/04-technical-guides/diffusion-models/adding_new_models.md @@ -501,7 +501,7 @@ if __name__ == "__main__": ### Step 9: Add example scripts -**File**: Use `examples/run_pretrain.sh` with appropriate config +**File**: Launch with `primus-cli` and the appropriate config ```python #!/usr/bin/env python3 @@ -520,7 +520,7 @@ from primus.backends.megatron.data.dataloader import MegatronDataloaderWrapper def main(): - # Use examples/run_pretrain.sh with config from examples/megatron/configs/MI300X/diffusion/ + # Launch via primus-cli with a config from examples/megatron/configs/MI300X/diffusion/ # MegatronDataloaderWrapper wraps an existing iterable (from dataset provider): # dataloader = MegatronDataloaderWrapper(energon_loader_or_pytorch_loader) # ... @@ -541,7 +541,7 @@ See the complete example in the step-by-step guide above. 2. `core/models/diffusion/dit/model.py` - DiT model 3. `configs/models/megatron/diffusion/dit_xl_2.yaml` - Config file 4. `tests/unit_tests/backends/megatron/diffusion/test_dit_model.py` - Tests -5. `examples/run_pretrain.sh` - Use with config from `examples/megatron/configs/MI300X/diffusion/` +5. `primus-cli` - Launch with a config from `examples/megatron/configs/MI300X/diffusion/` --- @@ -565,7 +565,8 @@ Test with actual data: ```bash # Small dataset test -./examples/run_pretrain.sh --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml ``` ### Validation diff --git a/docs/04-technical-guides/diffusion-models/api_reference.md b/docs/04-technical-guides/diffusion-models/api_reference.md index c08525eb2..5bd0d8121 100644 --- a/docs/04-technical-guides/diffusion-models/api_reference.md +++ b/docs/04-technical-guides/diffusion-models/api_reference.md @@ -1022,7 +1022,7 @@ config = FluxConfig() model = Flux(config) ``` -For more advanced examples, see `examples/run_pretrain.sh`. +For more advanced examples, see the launcher usage in `primus-cli --help`. --- diff --git a/docs/04-technical-guides/diffusion-models/fp8_training.md b/docs/04-technical-guides/diffusion-models/fp8_training.md index 9067bf201..ca016743a 100644 --- a/docs/04-technical-guides/diffusion-models/fp8_training.md +++ b/docs/04-technical-guides/diffusion-models/fp8_training.md @@ -56,25 +56,20 @@ Verify your environment has: # See primus/configs/data/megatron/diffusion/README.md # 2. Train Flux 535M with FP8 -EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml \ -GPUS_PER_NODE=1 \ -bash examples/run_pretrain.sh +GPUS_PER_NODE=1 ./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml ``` ### Production training with Flux 12B ```bash # After validating with 535M, scale to 12B (TransformerEngine FP8) -EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_te_spec_fp8.yaml \ -GPUS_PER_NODE=8 \ -NNODES=4 \ -bash examples/run_slurm_pretrain.sh +GPUS_PER_NODE=8 ./primus-cli slurm srun -N 4 -- container -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_te_spec_fp8.yaml # Or local-spec FP8 (no TransformerEngine dependency) -EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml \ -GPUS_PER_NODE=8 \ -NNODES=4 \ -bash examples/run_slurm_pretrain.sh +GPUS_PER_NODE=8 ./primus-cli slurm srun -N 4 -- container -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml ``` --- @@ -386,8 +381,8 @@ The local-spec FP8 kernels benefit from the Primus-Turbo autotuner, which picks unset PRIMUS_TURBO_GEMM_BACKEND # or scope it so it does not cover FP8 export PRIMUS_TURBO_AUTO_TUNE=1 -EXP=examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml \ - bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml ``` ### Contrast with MXFP4 @@ -448,9 +443,8 @@ export HSA_ENABLE_SDMA=0 # Disable SDMA for stability ```bash # Quick 100-step validation run -EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml \ -GPUS_PER_NODE=1 \ -bash examples/run_pretrain.sh +GPUS_PER_NODE=1 ./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml ``` ### Convergence test diff --git a/docs/04-technical-guides/diffusion-models/mxfp4_training.md b/docs/04-technical-guides/diffusion-models/mxfp4_training.md index 8711fa89e..be1e2fe67 100644 --- a/docs/04-technical-guides/diffusion-models/mxfp4_training.md +++ b/docs/04-technical-guides/diffusion-models/mxfp4_training.md @@ -52,7 +52,7 @@ export PRIMUS_TURBO_GEMM_BACKEND=FP4:AITER export AITER_CONFIG_GEMM_A4W4=$TUNED_GEMM_DIR/mi355x/flux_12b.csv export AITER_LOG_TUNED_CONFIG=1 # recommended: confirms each shape hits the CSV -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain --config "$EXP" ``` The pre-tuned CSV is distributed via an internal tuned-config source (`tuned_gemm_configs/mi355x/flux_12b.csv`). If you do not have access, omit `AITER_CONFIG_GEMM_A4W4` and AITER will fall back to its bundled `a4w4_blockscale_tuned_gemm.csv` (slower for Flux 12B shapes). diff --git a/docs/04-technical-guides/hybrid-models/README.md b/docs/04-technical-guides/hybrid-models/README.md index 0e1afae98..e3691a65e 100644 --- a/docs/04-technical-guides/hybrid-models/README.md +++ b/docs/04-technical-guides/hybrid-models/README.md @@ -272,40 +272,39 @@ Launch training inside a Docker container on a single node: ```bash # Zebra-Llama 1B with KDA (Kimi Delta Attention) -EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \ -DATA_PATH=./data \ -GPUS_PER_NODE=8 \ -HF_TOKEN=$HF_TOKEN \ -bash examples/run_local_pretrain.sh +export DATA_PATH=./data +GPUS_PER_NODE=8 HF_TOKEN=$HF_TOKEN \ +./primus-cli container --volume "$DATA_PATH:$DATA_PATH" \ + --env DATA_PATH \ + -- train pretrain --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml ``` -Other model variants: +Other model variants (same launcher, different `--config`): ```bash # Zebra-Llama 1B with Mamba SSM -EXP=examples/megatron/configs/MI300X/zebra_llama_1B-pretrain.yaml \ -bash examples/run_local_pretrain.sh +./primus-cli container -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_1B-pretrain.yaml # Zebra-Llama 1B with pure KDA (no attention layers) -EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda_pure-pretrain.yaml \ -bash examples/run_local_pretrain.sh +./primus-cli container -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_1B_kda_pure-pretrain.yaml # Zebra-Llama 1B with GDN (pure recurrent, no attention) -EXP=examples/megatron/configs/MI300X/zebra_llama_1B_gdn-pretrain.yaml \ -bash examples/run_local_pretrain.sh +./primus-cli container -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_1B_gdn-pretrain.yaml # Zebra-Llama 1B pure GDN (FLA-validated, 4-GPU) -EXP=examples/megatron/configs/MI300X/zebra_llama_1B_gdn_pure-pretrain.yaml \ -GPUS_PER_NODE=4 \ -bash examples/run_local_pretrain.sh +GPUS_PER_NODE=4 ./primus-cli container -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_1B_gdn_pure-pretrain.yaml # Zebra-Llama 3B -EXP=examples/megatron/configs/MI300X/zebra_llama_3B-pretrain.yaml \ -bash examples/run_local_pretrain.sh +./primus-cli container -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_3B-pretrain.yaml # Zebra-Llama 8B -EXP=examples/megatron/configs/MI300X/zebra_llama_8B-pretrain.yaml \ -bash examples/run_local_pretrain.sh +./primus-cli container -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_8B-pretrain.yaml ``` ### Multi-Node (Slurm) @@ -313,30 +312,31 @@ bash examples/run_local_pretrain.sh For multi-node training on a Slurm cluster: ```bash -EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \ -DATA_PATH=/shared/data \ -NNODES=2 \ -bash examples/run_slurm_pretrain.sh +export DATA_PATH=/shared/data +./primus-cli slurm srun -N 2 \ + -- container --volume "$DATA_PATH:$DATA_PATH" \ + --env DATA_PATH \ + -- train pretrain --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml ``` Ensure the `global_batch_size` in your config is divisible by `micro_batch_size * GPUS_PER_NODE * NNODES`. ### If Already Inside a Container -If you are already inside a Docker container or on a bare-metal node with the environment set up: +If you are already inside a Docker container or on a bare-metal node with the environment set up, use `direct` mode instead of `container` — it skips the container launch and runs `torchrun` in place: ```bash -EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \ -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml ``` ### Mock Data (Smoke Test) -To quickly verify the model runs without real data, the 3B and 8B configs come with `mock_data: true` by default. For the 1B configs, you can override: +To quickly verify the model runs without real data, the 3B and 8B configs come with `mock_data: true` by default. For the 1B configs, you can override on the command line — every argument after `--config` is forwarded to the Primus Python CLI: ```bash -EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \ -bash examples/run_local_pretrain.sh \ +./primus-cli container -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \ --mock_data true --train_iters 10 ``` @@ -610,9 +610,8 @@ Primus/ │ │ ├── chat_zebra_llama.py # Interactive chat │ │ └── convert_fla_to_megatron.py # FLA Arrow → Megatron binary converter │ └── docker/start_container.sh # Dev container launcher -├── examples/ -│ ├── run_local_pretrain.sh # Single-node Docker launcher -│ ├── run_slurm_pretrain.sh # Slurm launcher -│ └── run_pretrain.sh # Core training entrypoint +├── runner/ +│ ├── primus-cli # Unified launcher (direct/container/slurm) +│ └── helpers/envs/base_env.sh # NCCL/ROCm/cache env defaults └── requirements.txt # Python dependencies ``` diff --git a/docs/04-technical-guides/hybrid-models/gdn-fla-parity.md b/docs/04-technical-guides/hybrid-models/gdn-fla-parity.md index cdc307a8a..9e5ff658b 100644 --- a/docs/04-technical-guides/hybrid-models/gdn-fla-parity.md +++ b/docs/04-technical-guides/hybrid-models/gdn-fla-parity.md @@ -36,8 +36,9 @@ Inside the `rocm/primus:v26.2` container with the repo mounted at # Launch training (8 GPUs by default). The Megatron-LM behavioral patches # below are applied automatically at startup via Primus's patch system -- # no separate apply step needed. -EXP=examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml \ - bash examples/run_pretrain.sh 2>&1 | tee primus_gdn.log +./primus-cli direct --log_file primus_gdn.log \ + -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml ``` Optional toggles (all default off unless noted). Each is exposed at diff --git a/docs/04-technical-guides/hybrid-models/gdn-guide.md b/docs/04-technical-guides/hybrid-models/gdn-guide.md index 0e5b68cf8..6d4b8e4a4 100644 --- a/docs/04-technical-guides/hybrid-models/gdn-guide.md +++ b/docs/04-technical-guides/hybrid-models/gdn-guide.md @@ -256,8 +256,9 @@ The architecture-only YAML it extends from is `[primus/configs/models/megatron/z ```bash # inside the container, in /home//Primus -EXP=examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml \ - bash examples/run_pretrain.sh 2>&1 | tee primus_gdn.log +./primus-cli direct --log_file primus_gdn.log \ + -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml ``` This brings up `torchrun` with 8 ranks on the local node. Expected wall time on a healthy MI300X box: **~1h 54m** for the full 4768 iters. diff --git a/docs/04-technical-guides/hybrid-models/kda-fla-parity.md b/docs/04-technical-guides/hybrid-models/kda-fla-parity.md index d365215c1..28276cdb0 100644 --- a/docs/04-technical-guides/hybrid-models/kda-fla-parity.md +++ b/docs/04-technical-guides/hybrid-models/kda-fla-parity.md @@ -77,8 +77,9 @@ python tools/hybrid/convert_fla_kda_init_to_megatron.py # 2. Launch training (8 GPUs by default). The Megatron-LM behavioral # patches (same set as GDN) are applied automatically at startup via # Primus's patch system -- no separate apply step needed. -EXP=examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml \ - bash examples/run_pretrain.sh 2>&1 | tee primus_kda.log +./primus-cli direct --log_file primus_kda.log \ + -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml ``` ### Recommended toggle profile (YAML or env var) diff --git a/docs/04-technical-guides/hybrid-models/kda-guide.md b/docs/04-technical-guides/hybrid-models/kda-guide.md index 29a4e2298..37239f6c9 100644 --- a/docs/04-technical-guides/hybrid-models/kda-guide.md +++ b/docs/04-technical-guides/hybrid-models/kda-guide.md @@ -263,8 +263,9 @@ The architecture-only YAML it extends from is ```bash # inside the container, in /home//Primus -EXP=examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml \ - bash examples/run_pretrain.sh 2>&1 | tee primus_kda.log +./primus-cli direct --log_file primus_kda.log \ + -- train pretrain \ + --config examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml ``` Expected wall time on a healthy MI300X box: **~1h 56m** for the full 4768 diff --git a/docs/04-technical-guides/mega-moe.md b/docs/04-technical-guides/mega-moe.md index 3a20f20ee..7d01f2d33 100644 --- a/docs/04-technical-guides/mega-moe.md +++ b/docs/04-technical-guides/mega-moe.md @@ -144,7 +144,7 @@ balancing (for reproducibility) but preserves the step-to-step shape variation o The examples below use the rebuild hook (`REBUILD_PRIMUS_TURBO=1 PRIMUS_TURBO_REF=9b5d3092efcbc087657b233d8e9ae662cee6ec6b`) to build Primus-Turbo from source. -### Example 1 — single-node EP8, 4 layers (`run_pretrain_cli.sh`) +### Example 1 — single-node EP8, 4 layers (`primus-cli direct`) 1 node × 8 GPUs, `TP=1 / PP=1 / EP=8`, DeepSeek-V3 BF16, `GBS = MBS*GPUS*GA = 2*8*64 = 1024`. Minimal fused-MegaMoE run from `Primus/`: @@ -161,7 +161,7 @@ export PRIMUS_TURBO_REF=9b5d3092efcbc087657b233d8e9ae662cee6ec6b export GPU_ARCHS=gfx950 # Parallelism (EP-only) + fused MegaMoE -bash examples/run_pretrain_cli.sh \ +./primus-cli direct -- train pretrain --config "$EXP" \ --num_layers 4 \ --micro_batch_size 2 \ --global_batch_size 1024 \ @@ -198,7 +198,7 @@ export REBUILD_PRIMUS_TURBO=1 export PRIMUS_TURBO_REF=9b5d3092efcbc087657b233d8e9ae662cee6ec6b export GPU_ARCHS=gfx950 -bash examples/run_pretrain_cli.sh \ +./primus-cli direct -- train pretrain --config "$EXP" \ --num_layers 4 \ --micro_batch_size 2 \ --global_batch_size 1024 \ @@ -237,7 +237,11 @@ export USING_AINIC=1 # Toggle the fused MegaMoE layer + model config export EXP=examples/megatron/configs/MI355X/deepseek_v3-BF16-pretrain.yaml -bash examples/run_slurm_pretrain_cli.sh \ +# USING_AINIC / REBUILD_PRIMUS_TURBO / GPU_ARCHS are forwarded into the container +# by the default env whitelist in runner/.primus.yaml, and PRIMUS_TURBO_REF by the +# automatic PRIMUS_* passthrough, so none of them need an explicit --env. +./primus-cli slurm srun -N "$NNODES" -- container \ + -- train pretrain --config "$EXP" \ --train_iters 15 \ --micro_batch_size 2 \ --global_batch_size 1024 \ diff --git a/docs/04-technical-guides/multi-node-networking.md b/docs/04-technical-guides/multi-node-networking.md index 62ad1a36b..a4dba99f0 100644 --- a/docs/04-technical-guides/multi-node-networking.md +++ b/docs/04-technical-guides/multi-node-networking.md @@ -13,7 +13,7 @@ This guide summarizes how Primus configures networking, how **InfiniBand**, **Ro | Socket / interface detection | `runner/helpers/envs/get_ip_interface.sh` | | AINIC hook (container/CLI integration) | `runner/helpers/hooks/03_enable_ainic.sh` | | AINIC CLI defaults | `runner/use_ainic.yaml` | -| ANP / `NCCL_NET_PLUGIN` example | `examples/run_pretrain.sh` | +| ANP / `NCCL_NET_PLUGIN` selection | `runner/helpers/hooks/03_enable_ainic.sh` | --- @@ -91,7 +91,7 @@ Adjust **`NCCL_IB_GID_INDEX`** and **`container.options.image`** to match your c ### RCCL network plugin (ANP) -For ANP-based networking, clusters often set **`NCCL_NET_PLUGIN`** to **`librccl-anp.so`** when that library is present under `ANP_HOME_DIR`, falling back to `librccl-net.so` otherwise—see the logic in `examples/run_pretrain.sh`. This complements the library paths from `03_enable_ainic.sh`. +For ANP-based networking, `NCCL_NET_PLUGIN` is set to **`librccl-anp.so`** when that library is present under `ANP_HOME_DIR`, falling back to `librccl-net.so` otherwise. This selection and the matching library paths both live in `runner/helpers/hooks/03_enable_ainic.sh`, so they apply to every launcher mode. ### Variables commonly set for AINIC diff --git a/docs/04-technical-guides/native-sft-lora.md b/docs/04-technical-guides/native-sft-lora.md index ec160614b..7472f20f9 100644 --- a/docs/04-technical-guides/native-sft-lora.md +++ b/docs/04-technical-guides/native-sft-lora.md @@ -62,7 +62,7 @@ export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxx" # required for gated models export EXP_NAME="llama2_70b_native_$(date +%Y%m%d_%H%M%S)" ``` -Set automatically inside the container by `examples/run_pretrain.sh` (you don't need to touch these): +Set automatically inside the container by `runner/helpers/envs/base_env.sh`, which every `primus-cli` mode sources (you don't need to touch these): - `TRITON_CACHE_DIR`, `MIOPEN_USER_DB_PATH`, `PRIMUS_CACHE_ROOT`—persistent JIT cache - `NCCL_*` / `RCCL_*`—communication tuning - `HSA_*` / `GPU_MAX_HW_QUEUES`—AMD GPU performance tuning @@ -84,7 +84,7 @@ docker exec \ -e PYTORCH_HIP_ALLOC_CONF=expandable_segments:True \ -e EXP=examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-mlperf_aligned.yaml \ sft_primus_0507_native \ - bash -c 'cd /workspace/Primus && bash examples/run_pretrain.sh' \ + bash -c 'cd /workspace/Primus && ./primus-cli direct -- train pretrain --config "$EXP"' \ 2>&1 | tee /home/botahu/llama2_70b_500iter_runs/${EXP_NAME}.log ``` @@ -203,8 +203,8 @@ user_name: ${PRIMUS_USER:root} # # Recommended invocation: # export PRIMUS_EXP_NAME=native_llama2_70b_fp4_perf_$(date +%Y%m%d_%H%M%S) -# EXP=examples/megatron/configs/MI355X/llama2_70B-FP4-sft-packed-perf.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/llama2_70B-FP4-sft-packed-perf.yaml # ============================================================================= exp_name: ${PRIMUS_EXP_NAME:llama2_70B-FP4-sft-packed-perf} @@ -363,7 +363,7 @@ docker exec \ -e PYTORCH_HIP_ALLOC_CONF=expandable_segments:True \ -e EXP=examples/megatron/configs/MI355X/llama2_70B-FP4-sft-packed-perf.yaml \ sft_primus_0507_native \ - bash -c 'cd /workspace/Primus && bash examples/run_pretrain.sh' \ + bash -c 'cd /workspace/Primus && ./primus-cli direct -- train pretrain --config "$EXP"' \ 2>&1 | tee /home/botahu/llama2_70b_500iter_runs/${EXP_NAME}.log ``` diff --git a/docs/04-technical-guides/performance-tuning.md b/docs/04-technical-guides/performance-tuning.md index 4d09ea502..321ccf47d 100644 --- a/docs/04-technical-guides/performance-tuning.md +++ b/docs/04-technical-guides/performance-tuning.md @@ -6,9 +6,9 @@ This guide covers AMD-focused performance work in Primus: HipBLASLt autotuning f ## 1. HipBLASLt autotuning -Transformer Engine and GEMM-heavy training benefit from HipBLASLt kernel selection. Primus integrates a **three-stage** workflow controlled by `PRIMUS_HIPBLASLT_TUNING_STAGE` (see `examples/README.md` and `examples/run_pretrain.sh`). +Transformer Engine and GEMM-heavy training benefit from HipBLASLt kernel selection. Primus integrates a **three-stage** workflow controlled by `PRIMUS_HIPBLASLT_TUNING_STAGE` (see `examples/README.md` and `runner/helpers/hooks/train/pretrain/prepare_experiment.sh`). -> **Activate tuning first.** The stage variable is only honored when the master switch `PRIMUS_HIPBLASLT_TUNING=1` is set (and `PRIMUS_DETERMINISTIC` is not `1`). Without `PRIMUS_HIPBLASLT_TUNING=1`, both `run_pretrain.sh` and the CLI hook `runner/helpers/hooks/train/pretrain/prepare_experiment.sh` skip tuning entirely and force `TE_HIPBLASLT_TUNING_RUN_COUNT=0` / `TE_HIPBLASLT_TUNING_ALGO_COUNT=0`. Export `PRIMUS_HIPBLASLT_TUNING=1` alongside the stage in every command below. +> **Activate tuning first.** The stage variable is only honored when the master switch `PRIMUS_HIPBLASLT_TUNING=1` is set (and `PRIMUS_DETERMINISTIC` is not `1`). Without `PRIMUS_HIPBLASLT_TUNING=1`, the CLI hook `runner/helpers/hooks/train/pretrain/prepare_experiment.sh` skips tuning entirely and forces `TE_HIPBLASLT_TUNING_RUN_COUNT=0` / `TE_HIPBLASLT_TUNING_ALGO_COUNT=0`. Export `PRIMUS_HIPBLASLT_TUNING=1` alongside the stage in every command below. ### Stage 0 (default) @@ -25,7 +25,7 @@ Run a **short** training job so shapes are collected during real forward/backwar ```bash export PRIMUS_HIPBLASLT_TUNING=1 export PRIMUS_HIPBLASLT_TUNING_STAGE=1 -./runner/primus-cli direct -- train pretrain \ +./primus-cli direct -- train pretrain \ --config examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml ``` @@ -40,7 +40,7 @@ Runs offline tuning from dumped shapes (often 10–30 minutes depending on model ```bash export PRIMUS_HIPBLASLT_TUNING=1 export PRIMUS_HIPBLASLT_TUNING_STAGE=2 -./runner/primus-cli direct -- train pretrain \ +./primus-cli direct -- train pretrain \ --config examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml ``` @@ -56,7 +56,7 @@ Point the runtime at the tuned override file: export PRIMUS_HIPBLASLT_TUNING=1 export PRIMUS_HIPBLASLT_TUNING_STAGE=3 export HIPBLASLT_TUNING_OVERRIDE_FILE=/path/to/tune_hipblas_gemm_results.txt -./runner/primus-cli direct -- train pretrain \ +./primus-cli direct -- train pretrain \ --config examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml ``` @@ -64,10 +64,10 @@ export HIPBLASLT_TUNING_OVERRIDE_FILE=/path/to/tune_hipblas_gemm_results.txt | Variable | Role | |----------|------| -| `TE_HIPBLASLT_TUNING_ALGO_COUNT` | Breadth of algorithm search for TE HipBLASLt tuning (see `examples/run_pretrain.sh` defaults). | +| `TE_HIPBLASLT_TUNING_ALGO_COUNT` | Breadth of algorithm search for TE HipBLASLt tuning (see the defaults in `runner/helpers/hooks/train/pretrain/prepare_experiment.sh`). | | `TE_HIPBLASLT_TUNING_RUN_COUNT` | Number of benchmark runs per shape during TE tuning. | | `TE_HIPBLASLT_TUNING_ALGO_FILE` | Optional algorithm file for TE tuning flows. | -| `TE_HIPBLASLT_TUNING` | When set, interacts with deterministic mode; avoid conflicting settings with shape dump (see script comments in `examples/run_pretrain.sh`). | +| `TE_HIPBLASLT_TUNING` | When set, interacts with deterministic mode; avoid conflicting settings with shape dump (see the comments in `runner/helpers/hooks/train/pretrain/prepare_experiment.sh`). | | `HIPBLASLT_TUNING_OVERRIDE_FILE` | Override file for stage 3 training. | ### Standalone offline tool diff --git a/examples/README.md b/examples/README.md index 516ecd8fd..1523c659c 100644 --- a/examples/README.md +++ b/examples/README.md @@ -79,8 +79,27 @@ pre-commit install --- ### Run Pretraining -Use the `run_pretrain.sh` script to start training. +`./primus-cli` is the single entry point for all training. Pick a mode based on +where you want the process to run, then pass the Primus command after `--`: + +| Mode | When to use | Command | +| ---- | ----------- | ------- | +| `container` | Single node, launch from the host; the CLI starts the Docker/Podman container for you | `./primus-cli container -- train pretrain --config ` | +| `slurm` | Multi-node via SLURM (`srun` or `sbatch`) | `./primus-cli slurm srun -N -- container -- train pretrain --config ` | +| `direct` | You are already inside a container (or on a prepared bare-metal host) | `./primus-cli direct -- train pretrain --config ` | + +Notes on the argument shape: + +- `--` separates launcher options from the Primus Python CLI. Everything after the last + `--` (e.g. `--num_layers 4`) is forwarded to Primus as a config override. +- Use `train posttrain` instead of `train pretrain` for SFT / post-training leaf configs. +- Environment defaults (NCCL/RCCL, ROCm, persistent JIT caches, `DATA_PATH`, `HF_HOME`) + come from `runner/helpers/envs/base_env.sh` plus the per-GPU file + `runner/helpers/envs/.sh`; setup steps such as dependency installs, AINIC + enablement and dataset preparation run as hooks under `runner/helpers/hooks/`. +- `runner/.primus.yaml` holds the defaults for each mode (container image, devices, and + the list of environment variables forwarded into the container). #### 🚀 Quick Start Mode Use this mode for **rapid iteration or validation** of a model config. @@ -88,10 +107,16 @@ You do not need to enter the Docker container. Just set the config and run. ```bash # Example for megatron llama3.1_8B -EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml bash ./examples/run_local_pretrain.sh +export EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml +./primus-cli container -- train pretrain --config "$EXP" + +# Custom leaf config (SFT, mlperf, bridge_aligned, etc.) +export EXP=examples/megatron/configs/MI355X/llama3_8B-BF16-sft.yaml +./primus-cli container -- train posttrain --config "$EXP" # examples for torchtitan llama3.1_8B -EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml bash ./examples/run_local_pretrain.sh +export EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml +./primus-cli container -- train pretrain --config "$EXP" ``` --- @@ -112,10 +137,12 @@ docker exec -it dev_primus bash cd Primus && pip install -r requirements.txt # Example for megatron llama3.1_8B -EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml bash ./examples/run_pretrain.sh +export EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml +./primus-cli direct -- train pretrain --config "$EXP" # examples for torchtitan llama3.1_8B -EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml bash ./examples/run_pretrain.sh +export EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml +./primus-cli direct -- train pretrain --config "$EXP" ``` @@ -131,17 +158,24 @@ export DOCKER_IMAGE="docker.io/rocm/primus:v26.5" export NNODES=8 # Example for megatron llama3.1_8B -EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml bash ./examples/run_slurm_pretrain.sh +export EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml +./primus-cli slurm srun -N "$NNODES" -- container -- train pretrain --config "$EXP" # examples for torchtitan llama3.1_8b -EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml bash ./examples/run_slurm_pretrain.sh +export EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml +./primus-cli slurm srun -N "$NNODES" -- container -- train pretrain --config "$EXP" ``` ## 🔧 HipblasLT Auto Tuning -HipblasLT tuning is divided into three stages and controlled via the environment variable `PRIMUS_HIPBLASLT_TUNING_STAGE`: +HipblasLT tuning is divided into three stages, selected with `PRIMUS_HIPBLASLT_TUNING_STAGE`. +The stage is only honored when the master switch `PRIMUS_HIPBLASLT_TUNING=1` is also set and +deterministic mode is off (`PRIMUS_DETERMINISTIC != 1`); otherwise the +`runner/helpers/hooks/train/pretrain/prepare_experiment.sh` hook skips tuning entirely. ```bash +# master switch: tuning is off unless this is 1 +export PRIMUS_HIPBLASLT_TUNING=1 # default 0 means no tuning export PRIMUS_HIPBLASLT_TUNING_STAGE=${PRIMUS_HIPBLASLT_TUNING_STAGE:-0} ``` @@ -158,7 +192,7 @@ It is recommended to reduce `train_iters` for faster shape generation. export PRIMUS_HIPBLASLT_TUNING_STAGE=1 export EXP=examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml -NNODES=1 bash ./examples/run_slurm_pretrain.sh +./primus-cli slurm srun -N 1 -- container -- train pretrain --config "$EXP" ``` --- @@ -175,7 +209,7 @@ It typically takes 10–30 minutes depending on model size and shape complexity. export PRIMUS_HIPBLASLT_TUNING_STAGE=2 export EXP=examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml -NNODES=1 bash ./examples/run_slurm_pretrain.sh +./primus-cli slurm srun -N 1 -- container -- train pretrain --config "$EXP" ``` --- @@ -187,7 +221,7 @@ In this final stage, the tuned kernel is loaded for efficient training: ```bash export PRIMUS_HIPBLASLT_TUNING_STAGE=3 export EXP=examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml -NNODES=1 bash ./examples/run_slurm_pretrain.sh +./primus-cli slurm srun -N 1 -- container -- train pretrain --config "$EXP" ``` ## ✅ Supported Models @@ -223,15 +257,18 @@ The following models are supported out of the box via provided configuration fil Use the following command pattern to start training with a selected model configuration: ```bash -EXP=examples/megatron/configs/MI300X/ bash ./examples/run_local_pretrain.sh +export EXP=examples/megatron/configs/MI300X/ +./primus-cli container -- train pretrain --config "$EXP" ``` For example, to run the llama3.1_8B model quickly: ```bash -EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml bash ./examples/run_local_pretrain.sh +export EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml +./primus-cli container -- train pretrain --config "$EXP" -EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml bash ./examples/run_local_pretrain.sh +export EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml +./primus-cli container -- train pretrain --config "$EXP" ``` @@ -240,11 +277,13 @@ For multi-node training via SLURM, use: ```bash export NNODES=8 -#run megatron -EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml bash ./examples/run_slurm_pretrain.sh +# run megatron +export EXP=examples/megatron/configs/MI300X/llama3.1_8B-BF16-pretrain.yaml +./primus-cli slurm srun -N "$NNODES" -- container -- train pretrain --config "$EXP" # run torchtitan -EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml bash ./examples/run_slurm_pretrain.sh +export EXP=examples/torchtitan/configs/MI300X/llama3.1_8B-pretrain.yaml +./primus-cli slurm srun -N "$NNODES" -- container -- train pretrain --config "$EXP" ``` ## ☸️ Kubernetes Training Management (`run_k8s_pretrain.sh`) diff --git a/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml b/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml index 865cfe832..8bc64a7c2 100644 --- a/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml +++ b/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml @@ -7,8 +7,8 @@ # Pre-encoded mode is faster and recommended for production training. # # Usage: -# EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml work_group: ${PRIMUS_TEAM:amd} user_name: ${PRIMUS_USER:root} @@ -161,10 +161,10 @@ modules: # - examples/megatron/diffusion/README.md # # Single-node training (8 GPUs): -# EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml \ -# GPUS_PER_NODE=8 bash examples/run_pretrain.sh +# GPUS_PER_NODE=8 ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml # # Multi-node training (4 nodes, 8 GPUs each): -# EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml \ -# NNODES=4 bash examples/run_slurm_pretrain.sh +# ./primus-cli slurm srun -N 4 -- container -- train pretrain \ +# --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml # diff --git a/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml b/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml index fa4aa8da7..86c742b62 100644 --- a/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml +++ b/examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml @@ -14,8 +14,8 @@ # Requires: Transformer Engine 2.1.0+ with ROCm backend # # Usage: -# EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml \ -# GPUS_PER_NODE=1 bash examples/run_pretrain.sh +# GPUS_PER_NODE=1 ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml work_group: ${PRIMUS_TEAM:amd} user_name: ${PRIMUS_USER:root} diff --git a/examples/megatron/configs/MI300X/diffusion/flux_535m_with_guidance_embed.yaml b/examples/megatron/configs/MI300X/diffusion/flux_535m_with_guidance_embed.yaml index 38f10f88d..a3b2cfc47 100644 --- a/examples/megatron/configs/MI300X/diffusion/flux_535m_with_guidance_embed.yaml +++ b/examples/megatron/configs/MI300X/diffusion/flux_535m_with_guidance_embed.yaml @@ -11,8 +11,8 @@ # Only use this if you specifically need guidance embedding support. # # Usage: -# EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_with_guidance_embed.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI300X/diffusion/flux_535m_with_guidance_embed.yaml # Extend standard 535M config extends: diff --git a/examples/megatron/configs/MI355X/deepseek_v2_lite-BF16-sft-packed-bridge_aligned.yaml b/examples/megatron/configs/MI355X/deepseek_v2_lite-BF16-sft-packed-bridge_aligned.yaml index c858ca01f..9f3fb5f4e 100644 --- a/examples/megatron/configs/MI355X/deepseek_v2_lite-BF16-sft-packed-bridge_aligned.yaml +++ b/examples/megatron/configs/MI355X/deepseek_v2_lite-BF16-sft-packed-bridge_aligned.yaml @@ -3,8 +3,8 @@ user_name: ${PRIMUS_USER:root} # Recommended invocation:: # # export PRIMUS_EXP_NAME=deepseek_v2_lite_native_aligned_$(date +%Y%m%d_%H%M%S) -# EXP=examples/megatron/configs/MI355X/deepseek_v2_lite-BF16-sft-packed-bridge_aligned.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/deepseek_v2_lite-BF16-sft-packed-bridge_aligned.yaml exp_name: ${PRIMUS_EXP_NAME:deepseek_v2_lite-sft-packed-bridge_aligned-force_lb} workspace: ${PRIMUS_WORKSPACE:./output} diff --git a/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml b/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml index 0ec08c870..d3ede5a5b 100644 --- a/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml +++ b/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml @@ -7,8 +7,8 @@ # Pre-encoded mode is faster and recommended for production training. # # Usage: -# EXP=examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml work_group: ${PRIMUS_TEAM:amd} user_name: ${PRIMUS_USER:root} @@ -161,10 +161,10 @@ modules: # - examples/megatron/diffusion/README.md # # Single-node training (8 GPUs): -# EXP=examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml \ -# GPUS_PER_NODE=8 bash examples/run_pretrain.sh +# GPUS_PER_NODE=8 ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml # # Multi-node training (4 nodes, 8 GPUs each): -# EXP=examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml \ -# NNODES=4 bash examples/run_slurm_pretrain.sh +# ./primus-cli slurm srun -N 4 -- container -- train pretrain \ +# --config examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain.yaml # diff --git a/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain_fp8.yaml b/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain_fp8.yaml index 1ec0e6433..4a7fc1141 100644 --- a/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain_fp8.yaml +++ b/examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain_fp8.yaml @@ -14,8 +14,8 @@ # Requires: Transformer Engine 2.1.0+ with ROCm backend # # Usage: -# EXP=examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain_fp8.yaml \ -# GPUS_PER_NODE=1 bash examples/run_pretrain.sh +# GPUS_PER_NODE=1 ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/diffusion/flux_535m_pretrain_fp8.yaml work_group: ${PRIMUS_TEAM:amd} user_name: ${PRIMUS_USER:root} diff --git a/examples/megatron/configs/MI355X/diffusion/flux_535m_with_guidance_embed.yaml b/examples/megatron/configs/MI355X/diffusion/flux_535m_with_guidance_embed.yaml index 49cb41194..4a7fd5c50 100644 --- a/examples/megatron/configs/MI355X/diffusion/flux_535m_with_guidance_embed.yaml +++ b/examples/megatron/configs/MI355X/diffusion/flux_535m_with_guidance_embed.yaml @@ -11,8 +11,8 @@ # Only use this if you specifically need guidance embedding support. # # Usage: -# EXP=examples/megatron/configs/MI355X/diffusion/flux_535m_with_guidance_embed.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/diffusion/flux_535m_with_guidance_embed.yaml # Extend standard 535M config extends: diff --git a/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-bridge_aligned.yaml b/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-bridge_aligned.yaml index 21a91dbf2..7e0b762d7 100644 --- a/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-bridge_aligned.yaml +++ b/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-bridge_aligned.yaml @@ -3,8 +3,8 @@ user_name: ${PRIMUS_USER:root} # Recommended invocation:: # # export PRIMUS_EXP_NAME=native_llama2_70b_$(date +%Y%m%d_%H%M%S) -# EXP=examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-bridge_aligned.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-bridge_aligned.yaml exp_name: ${PRIMUS_EXP_NAME:llama2_70B-sft-packed-bridge_aligned} workspace: ${PRIMUS_WORKSPACE:./output} diff --git a/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-mlperf_aligned.yaml b/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-mlperf_aligned.yaml index 6ad8d0a97..8e9d32ab7 100644 --- a/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-mlperf_aligned.yaml +++ b/examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-mlperf_aligned.yaml @@ -36,8 +36,8 @@ user_name: ${PRIMUS_USER:root} # export PYTORCH_ALLOC_CONF=expandable_segments:True # export GLOO_SOCKET_IFNAME=lo # export NCCL_SOCKET_IFNAME=lo -# EXP=examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-mlperf_aligned.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/llama2_70B-BF16-sft-packed-mlperf_aligned.yaml # ' # # Apples-to-apples points (vs Bridge llama2_70b_lora_posttrain.yaml): diff --git a/examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml b/examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml index 9a1d646cd..f65a9f140 100644 --- a/examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml +++ b/examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml @@ -23,8 +23,8 @@ user_name: ${PRIMUS_USER:root} # # # (a) Native-shape FP8 perf run (seq=4096 GBS=16 squad) # export PRIMUS_EXP_NAME=native_llama2_70b_fp8_perf_$(date +%Y%m%d_%H%M%S) -# EXP=examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml # # # (b) mlperf-shape FP8 perf run (seq=8192 GBS=8) -- for like-for-like compare # export PRIMUS_SEQ_LENGTH=8192 @@ -34,10 +34,10 @@ user_name: ${PRIMUS_USER:root} # export PRIMUS_LR=4.0e-4 # export PRIMUS_TRAIN_ITERS=200 # export PRIMUS_EXP_NAME=native_llama2_70b_fp8_perf_mlperfshape_$(date +%Y%m%d_%H%M%S) -# EXP=examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/llama2_70B-FP8-sft-packed-perf.yaml # -# IMPORTANT: also export the FP8 NVTE_* env vars before run_pretrain.sh so +# IMPORTANT: also export the FP8 NVTE_* env vars before launching so # the TE FP8 kernels light up. See the README block at the bottom of this # file or the launch wrapper described in the SOP. # ============================================================================= diff --git a/examples/megatron/configs/MI355X/llama3_8B-BF16-sft-packed-bridge_aligned.yaml b/examples/megatron/configs/MI355X/llama3_8B-BF16-sft-packed-bridge_aligned.yaml index 10377d096..8a8952326 100644 --- a/examples/megatron/configs/MI355X/llama3_8B-BF16-sft-packed-bridge_aligned.yaml +++ b/examples/megatron/configs/MI355X/llama3_8B-BF16-sft-packed-bridge_aligned.yaml @@ -3,8 +3,8 @@ user_name: ${PRIMUS_USER:root} # Recommended invocation:: # # export PRIMUS_EXP_NAME=llama3_8b_native_aligned_$(date +%Y%m%d_%H%M%S) -# EXP=examples/megatron/configs/MI355X/llama3_8B-BF16-sft-packed-bridge_aligned.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/llama3_8B-BF16-sft-packed-bridge_aligned.yaml exp_name: ${PRIMUS_EXP_NAME:llama3_8B-sft-packed-bridge_aligned} workspace: ${PRIMUS_WORKSPACE:./output} diff --git a/examples/megatron/configs/MI355X/qwen3_30B_A3B-BF16-sft-packed-bridge_aligned.yaml b/examples/megatron/configs/MI355X/qwen3_30B_A3B-BF16-sft-packed-bridge_aligned.yaml index f0e6322d8..cab35a6f9 100644 --- a/examples/megatron/configs/MI355X/qwen3_30B_A3B-BF16-sft-packed-bridge_aligned.yaml +++ b/examples/megatron/configs/MI355X/qwen3_30B_A3B-BF16-sft-packed-bridge_aligned.yaml @@ -3,8 +3,8 @@ user_name: ${PRIMUS_USER:root} # Recommended invocation (each run gets its own output directory): # # export PRIMUS_EXP_NAME=qwen3_30B_A3B_native_bridge_aligned_$(date +%Y%m%d_%H%M%S) -# EXP=examples/megatron/configs/MI355X/qwen3_30B_A3B-BF16-sft-packed-bridge_aligned.yaml \ -# bash examples/run_pretrain.sh +# ./primus-cli direct -- train pretrain \ +# --config examples/megatron/configs/MI355X/qwen3_30B_A3B-BF16-sft-packed-bridge_aligned.yaml exp_name: ${PRIMUS_EXP_NAME:qwen3_30B_A3B-sft-packed-bridge_aligned} workspace: ${PRIMUS_WORKSPACE:./output} diff --git a/examples/megatron/diffusion/README.md b/examples/megatron/diffusion/README.md index bfc6ada27..efdadfcdd 100644 --- a/examples/megatron/diffusion/README.md +++ b/examples/megatron/diffusion/README.md @@ -49,10 +49,10 @@ energon prepare . --num-workers 4 2. **Launch training:** ```bash -EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml \ DATA_PATH=/tmp/flux_test_data \ GPUS_PER_NODE=1 \ -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml ``` --- @@ -111,14 +111,14 @@ uses smaller default micro/global batch sizes on the 12B DDP configs. ```bash # Flux 535M (1-8 GPUs) -EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml \ GPUS_PER_NODE=8 \ -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml # Flux 12B (FSDP2, BF16, 8 GPUs) -EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_fsdp2_energon_schnell_resample_local_spec.yaml \ GPUS_PER_NODE=8 \ -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_12b_fsdp2_energon_schnell_resample_local_spec.yaml ``` ### Multi-Node Training (SLURM) @@ -128,8 +128,8 @@ export DOCKER_IMAGE="docker.io/rocm/primus:v26.1" export NNODES=8 export GPUS_PER_NODE=8 -EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_fsdp2_energon_schnell_resample_local_spec.yaml \ -bash examples/run_slurm_pretrain.sh +./primus-cli slurm srun -N "$NNODES" -- container -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_12b_fsdp2_energon_schnell_resample_local_spec.yaml ``` ### MLPerf Benchmark Reproduction (MI355X) @@ -143,9 +143,9 @@ primus-cli direct -- data diffusion-ingest \ --config primus/configs/data/megatron/diffusion/preprocessing/mlperf_flux1.yaml # Step 2: Train (configs already set vae_latent_mode: resample, vae_scale: 0.3611, vae_shift: 0.1159) -EXP=examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8_mlperf.yaml \ GPUS_PER_NODE=8 \ -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8_mlperf.yaml ``` --- @@ -156,19 +156,19 @@ FP8 provides ~2x memory reduction and 1.5-2x training speedup on AMD MI300X/MI35 ```bash # Quick validation with 535M -EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml \ GPUS_PER_NODE=1 \ -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml # Production with 12B (TransformerEngine FP8) -EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_te_spec_fp8.yaml \ -GPUS_PER_NODE=8 NNODES=4 \ -bash examples/run_slurm_pretrain.sh +GPUS_PER_NODE=8 \ +./primus-cli slurm srun -N 4 -- container -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_te_spec_fp8.yaml # Production with 12B (local-spec FP8, no TransformerEngine dependency) -EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml \ -GPUS_PER_NODE=8 NNODES=4 \ -bash examples/run_slurm_pretrain.sh +GPUS_PER_NODE=8 \ +./primus-cli slurm srun -N 4 -- container -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml ``` | Model | Precision | Memory/GPU | Batch Size | Speed | @@ -191,11 +191,11 @@ MXFP4 (E2M1 + E8M0 block-of-32 scales) Flux 12B training on MI355X is supported # Set TUNED_GEMM_DIR to wherever you have the tuned configs available. export TUNED_GEMM_DIR=${TUNED_GEMM_DIR:-/path/to/tuned_gemm_configs} -EXP=examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_mxfp4.yaml \ PRIMUS_TURBO_GEMM_BACKEND=FP4:AITER \ AITER_CONFIG_GEMM_A4W4=$TUNED_GEMM_DIR/mi355x/flux_12b.csv \ AITER_LOG_TUNED_CONFIG=1 \ -bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_mxfp4.yaml ``` For configuration knobs, backend-selector semantics, tuned-GEMM verification, and troubleshooting, see the [MXFP4 Training Guide](../../../docs/04-technical-guides/diffusion-models/mxfp4_training.md). diff --git a/primus/configs/data/megatron/diffusion/README.md b/primus/configs/data/megatron/diffusion/README.md index 827f1420b..58bb593f6 100644 --- a/primus/configs/data/megatron/diffusion/README.md +++ b/primus/configs/data/megatron/diffusion/README.md @@ -384,8 +384,8 @@ primus-cli direct -- data diffusion-encoded \ # dataset_path: /workspace/Primus/data/encoded_pokemon # Step 3: Train -EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml \ - bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml ``` ### Raw Mode (On-the-fly Encoding) @@ -407,8 +407,8 @@ primus-cli direct -- data diffusion-raw \ # clip: { model_path: black-forest-labs/FLUX.1-dev, ... } # Step 3: Train -EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml \ - bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml ``` ### MLPerf Ingest Mode (Streaming Download + Conversion) @@ -429,8 +429,8 @@ primus-cli direct -- data diffusion-ingest \ # data_path (or the PRIMUS_DIFFUSION_DATA_PATH env var) to /workspace/Primus/data/mlperf_flux1. # Step 3: Train (MLPerf benchmark reproduction) -EXP=examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8_mlperf.yaml \ - bash examples/run_pretrain.sh +./primus-cli direct -- train pretrain \ + --config examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8_mlperf.yaml ``` **Output directory structure:** diff --git a/tests/runner/test_docs_cli_examples.py b/tests/runner/test_docs_cli_examples.py new file mode 100644 index 000000000..7c99974ae --- /dev/null +++ b/tests/runner/test_docs_cli_examples.py @@ -0,0 +1,79 @@ +############################################################################### +# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved. +# +# See LICENSE for license information. +############################################################################### + +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +HYBRID_GUIDE = ROOT / "docs" / "04-technical-guides" / "hybrid-models" / "README.md" + +# Directories that are either deprecated, vendored, or gitignored build output. +SKIPPED_DIRS = { + ".git", + "docs_deprecated", + "logs", + "node_modules", + "output", + "third_party", + "ut_out", +} + +# Only executable invocations count: the script path has to follow `bash`, `sh` +# or `source` (optionally with short flags). Narrative mentions such as +# "run_slurm_pretrain.sh is avoided on purpose" therefore do not match. The +# trade-off is that a dead script named only in prose or in a Markdown link +# goes unnoticed; catching those would require distinguishing documentation +# about a script from instructions to run one, which is not worth the false +# positives. +SCRIPT_INVOCATION = re.compile(r"(?:^|[\s(&;|`])(?:bash|sh|source)\s+(?:-\w+\s+)*([A-Za-z0-9_./-]+\.sh)\b") + +# Repo-relative references that are knowingly unresolvable. +ALLOWED_MISSING = { + # Marked "(helper; not committed)" inline: a local calibration helper that + # is intentionally kept out of the tree. + "examples/deepseek-v4/projection/script/_calibrate_flash.sh", + # Forward reference to the ODC rocSHMEM example, which lands with the PR + # that the same README says the ops are still waiting on. + "examples/llm_training/run.sh", +} + + +def _markdown_files(): + for path in sorted(ROOT.rglob("*.md")): + if SKIPPED_DIRS.isdisjoint(path.relative_to(ROOT).parts): + yield path + + +def _repo_relative_script_refs(line): + """Yield repo-relative .sh paths invoked on a documentation line.""" + for match in SCRIPT_INVOCATION.finditer(line): + ref = match.group(1) + # Absolute paths point into a container or an unrelated checkout, and a + # bare filename is relative to whatever directory the surrounding + # snippet cd'd into. Neither can be resolved against the repo root. + if ref.startswith(("/", "~")) or "/" not in ref.lstrip("./"): + continue + yield ref + + +def test_hybrid_data_path_env_stays_at_container_layer(): + guide = HYBRID_GUIDE.read_text(encoding="utf-8") + assert "-- --env DATA_PATH" not in guide + assert guide.count('--volume "$DATA_PATH:$DATA_PATH" \\\n --env DATA_PATH \\\n -- train pretrain') == 2 + + +def test_documented_script_invocations_exist(): + dead = [] + for path in _markdown_files(): + doc = path.relative_to(ROOT) + for lineno, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + for ref in _repo_relative_script_refs(line): + target = ref[2:] if ref.startswith("./") else ref + if target in ALLOWED_MISSING or (ROOT / target).is_file(): + continue + dead.append(f"{doc}:{lineno}: runs '{ref}', which does not exist") + + assert not dead, "Documentation invokes scripts that are not in the repository:\n" + "\n".join(dead)