docs: describe training launches in terms of primus-cli

docs/04-technical-guides/collective-operations.md

@@ -202,7 +202,7 @@ The following appear in ROCm / AMD deployments and partner integrations; availab
 |---------|--------|
 | **MSCCL** | Microsoft Collective Communication Library: **custom algorithms** and patterns; might be used when the stack is built and configured for them. |
 | **MSCCL++** | User-space collective paths aimed at **lower latency** for specific patterns and hardware. |
-| **ANP (AMD Network Plugin)** | Network backend integration (e.g. **AINIC**-oriented paths). Example: `NCCL_NET_PLUGIN` might point to `librccl-anp.so` or similar when installed (see Primus `examples/run_pretrain.sh` patterns). |
+| **ANP (AMD Network Plugin)** | Network backend integration (e.g. **AINIC**-oriented paths). Example: `NCCL_NET_PLUGIN` might point to `librccl-anp.so` or similar when installed (see Primus `runner/helpers/hooks/03_enable_ainic.sh`). |
 
 ### Environment variables
 

docs/04-technical-guides/determinism-and-reproducibility.md

@@ -1,6 +1,6 @@
 # Determinism and reproducibility
 
-Reproducibility—getting bit-identical (or run-to-run stable) results—matters for debugging divergence, validating optimizations, and regression testing. This guide covers Primus's deterministic mode, the environment variables it sets, the per-backend seed/determinism knobs, and the performance trade-offs. Parameters and behavior are grounded in `examples/run_pretrain.sh`, `primus/configs/modules/megatron/trainer_base.yaml`, and `primus/configs/modules/torchtitan/pre_trainer.yaml`.
+Reproducibility—getting bit-identical (or run-to-run stable) results—matters for debugging divergence, validating optimizations, and regression testing. This guide covers Primus's deterministic mode, the environment variables it sets, the per-backend seed/determinism knobs, and the performance trade-offs. Parameters and behavior are grounded in `runner/helpers/envs/base_env.sh`, `runner/helpers/hooks/05_deterministic.sh`, `primus/configs/modules/megatron/trainer_base.yaml`, and `primus/configs/modules/torchtitan/pre_trainer.yaml`.
 
 ---
 
@@ -17,7 +17,7 @@ Full determinism also generally requires the **same world size, parallelism layo
 
 ## 2. Primus deterministic mode (`PRIMUS_DETERMINISTIC`)
 
-Setting `PRIMUS_DETERMINISTIC=1` configures the GPU/communication stack for deterministic behavior. The CLI/runner path applies this through the hook `runner/helpers/hooks/05_deterministic.sh`; the `examples/run_pretrain.sh` script applies an equivalent inline block. The exported variables are:
+Setting `PRIMUS_DETERMINISTIC=1` configures the GPU/communication stack for deterministic behavior. Every launcher mode (`direct`, `container`, `slurm`) applies this through the same hook, `runner/helpers/hooks/05_deterministic.sh`, which runs before training starts. The exported variables are:
 
 ```bash
 # when PRIMUS_DETERMINISTIC=1 (runner/helpers/hooks/05_deterministic.sh)
@@ -28,15 +28,15 @@ export TORCH_COMPILE_DISABLE=1             # avoid torch.compile/Triton race con
 export PRIMUS_TURBO_AUTO_TUNE=0            # disable Primus-Turbo autotuning (stable kernel choice)
 ```
 
-> `PRIMUS_TURBO_AUTO_TUNE` also defaults to `0` in `runner/helpers/envs/base_env.sh`. The inline block in `examples/run_pretrain.sh` sets the first four variables and relies on that default for the fifth.
+> `PRIMUS_TURBO_AUTO_TUNE` also defaults to `0` in `runner/helpers/envs/base_env.sh`, which the hook relies on rather than re-exporting.
 
-Additionally, **HipBLASLt autotuning is disabled** in deterministic mode: tuning only runs when `PRIMUS_DETERMINISTIC != 1` *and* `PRIMUS_HIPBLASLT_TUNING=1` (`examples/run_pretrain.sh`). This prevents run-to-run kernel-selection differences. See [Performance tuning](./performance-tuning.md).
+Additionally, **HipBLASLt autotuning is disabled** in deterministic mode: tuning only runs when `PRIMUS_DETERMINISTIC != 1` *and* `PRIMUS_HIPBLASLT_TUNING=1` (`runner/helpers/hooks/train/pretrain/prepare_experiment.sh`). This prevents run-to-run kernel-selection differences. See [Performance tuning](./performance-tuning.md).
 
 `PRIMUS_DETERMINISTIC` is on the container passthrough allowlist (`runner/.primus.yaml`), so it reaches the training container. See [Environment variables](../03-configuration-reference/environment-variables.md).
 
 ```bash
 export PRIMUS_DETERMINISTIC=1
-./runner/primus-cli direct -- train pretrain \
+./primus-cli direct -- train pretrain \
   --config examples/megatron/configs/MI300X/llama2_7B-BF16-pretrain.yaml
 ```
 

docs/04-technical-guides/diffusion-models/STRUCTURE.md

@@ -97,7 +97,7 @@ Primus/
 │   │   └── ...
 │   └── prepare.py
 │
-├── examples/run_pretrain.sh                # Main training script
+├── runner/primus-cli                       # Main training launcher (direct/container/slurm)
 │
 ├── tests/
 │   ├── unit_tests/backends/megatron/diffusion/   # Unit test suite

docs/04-technical-guides/diffusion-models/adding_new_models.md

@@ -501,7 +501,7 @@ if __name__ == "__main__":
 
 ### Step 9: Add example scripts
 
-**File**: Use `examples/run_pretrain.sh` with appropriate config
+**File**: Launch with `primus-cli` and the appropriate config
 
 ```python
 #!/usr/bin/env python3
@@ -520,7 +520,7 @@ from primus.backends.megatron.data.dataloader import MegatronDataloaderWrapper
 
 
 def main():
-    # Use examples/run_pretrain.sh with config from examples/megatron/configs/MI300X/diffusion/
+    # Launch via primus-cli with a config from examples/megatron/configs/MI300X/diffusion/
     # MegatronDataloaderWrapper wraps an existing iterable (from dataset provider):
     #   dataloader = MegatronDataloaderWrapper(energon_loader_or_pytorch_loader)
     # ...
@@ -541,7 +541,7 @@ See the complete example in the step-by-step guide above.
 2. `core/models/diffusion/dit/model.py` - DiT model
 3. `configs/models/megatron/diffusion/dit_xl_2.yaml` - Config file
 4. `tests/unit_tests/backends/megatron/diffusion/test_dit_model.py` - Tests
-5. `examples/run_pretrain.sh` - Use with config from `examples/megatron/configs/MI300X/diffusion/`
+5. `primus-cli` - Launch with a config from `examples/megatron/configs/MI300X/diffusion/`
 
 ---
 
@@ -565,7 +565,8 @@ Test with actual data:
 
 ```bash
 # Small dataset test
-./examples/run_pretrain.sh --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml
+./primus-cli direct -- train pretrain \
+  --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain.yaml
 ```
 
 ### Validation

docs/04-technical-guides/diffusion-models/api_reference.md

@@ -1022,7 +1022,7 @@ config = FluxConfig()
 model = Flux(config)
 ```
 
-For more advanced examples, see `examples/run_pretrain.sh`.
+For more advanced examples, see the launcher usage in `primus-cli --help`.
 
 ---
 

docs/04-technical-guides/diffusion-models/fp8_training.md

@@ -56,25 +56,20 @@ Verify your environment has:
 # See primus/configs/data/megatron/diffusion/README.md
 
 # 2. Train Flux 535M with FP8
-EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml \
-GPUS_PER_NODE=1 \
-bash examples/run_pretrain.sh
+GPUS_PER_NODE=1 ./primus-cli direct -- train pretrain \
+  --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml
 ```
 
 ### Production training with Flux 12B
 
 ```bash
 # After validating with 535M, scale to 12B (TransformerEngine FP8)
-EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_te_spec_fp8.yaml \
-GPUS_PER_NODE=8 \
-NNODES=4 \
-bash examples/run_slurm_pretrain.sh
+GPUS_PER_NODE=8 ./primus-cli slurm srun -N 4 -- container -- train pretrain \
+  --config examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_te_spec_fp8.yaml
 
 # Or local-spec FP8 (no TransformerEngine dependency)
-EXP=examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml \
-GPUS_PER_NODE=8 \
-NNODES=4 \
-bash examples/run_slurm_pretrain.sh
+GPUS_PER_NODE=8 ./primus-cli slurm srun -N 4 -- container -- train pretrain \
+  --config examples/megatron/configs/MI300X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml
 ```
 
 ---
@@ -386,8 +381,8 @@ The local-spec FP8 kernels benefit from the Primus-Turbo autotuner, which picks
 unset PRIMUS_TURBO_GEMM_BACKEND          # or scope it so it does not cover FP8
 export PRIMUS_TURBO_AUTO_TUNE=1
 
-EXP=examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml \
-  bash examples/run_pretrain.sh
+./primus-cli direct -- train pretrain \
+  --config examples/megatron/configs/MI355X/diffusion/flux_12b_ddp_energon_schnell_resample_local_spec_fp8.yaml
 ```
 
 ### Contrast with MXFP4
@@ -448,9 +443,8 @@ export HSA_ENABLE_SDMA=0  # Disable SDMA for stability
 
 ```bash
 # Quick 100-step validation run
-EXP=examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml \
-GPUS_PER_NODE=1 \
-bash examples/run_pretrain.sh
+GPUS_PER_NODE=1 ./primus-cli direct -- train pretrain \
+  --config examples/megatron/configs/MI300X/diffusion/flux_535m_pretrain_fp8.yaml
 ```
 
 ### Convergence test

docs/04-technical-guides/diffusion-models/mxfp4_training.md

@@ -52,7 +52,7 @@ export PRIMUS_TURBO_GEMM_BACKEND=FP4:AITER
 export AITER_CONFIG_GEMM_A4W4=$TUNED_GEMM_DIR/mi355x/flux_12b.csv
 export AITER_LOG_TUNED_CONFIG=1   # recommended: confirms each shape hits the CSV
 
-bash examples/run_pretrain.sh
+./primus-cli direct -- train pretrain --config "$EXP"
 ```
 
 The pre-tuned CSV is distributed via an internal tuned-config source (`tuned_gemm_configs/mi355x/flux_12b.csv`). If you do not have access, omit `AITER_CONFIG_GEMM_A4W4` and AITER will fall back to its bundled `a4w4_blockscale_tuned_gemm.csv` (slower for Flux 12B shapes).

docs/04-technical-guides/hybrid-models/README.md

@@ -272,71 +272,71 @@ Launch training inside a Docker container on a single node:
 
 ```bash
 # Zebra-Llama 1B with KDA (Kimi Delta Attention)
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \
-DATA_PATH=./data \
-GPUS_PER_NODE=8 \
-HF_TOKEN=$HF_TOKEN \
-bash examples/run_local_pretrain.sh
+export DATA_PATH=./data
+GPUS_PER_NODE=8 HF_TOKEN=$HF_TOKEN \
+./primus-cli container --volume "$DATA_PATH:$DATA_PATH" \
+  --env DATA_PATH \
+  -- train pretrain --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml
 ```
 
-Other model variants:
+Other model variants (same launcher, different `--config`):
 
 ```bash
 # Zebra-Llama 1B with Mamba SSM
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B-pretrain.yaml \
-bash examples/run_local_pretrain.sh
+./primus-cli container -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_1B-pretrain.yaml
 
 # Zebra-Llama 1B with pure KDA (no attention layers)
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda_pure-pretrain.yaml \
-bash examples/run_local_pretrain.sh
+./primus-cli container -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_1B_kda_pure-pretrain.yaml
 
 # Zebra-Llama 1B with GDN (pure recurrent, no attention)
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B_gdn-pretrain.yaml \
-bash examples/run_local_pretrain.sh
+./primus-cli container -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_1B_gdn-pretrain.yaml
 
 # Zebra-Llama 1B pure GDN (FLA-validated, 4-GPU)
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B_gdn_pure-pretrain.yaml \
-GPUS_PER_NODE=4 \
-bash examples/run_local_pretrain.sh
+GPUS_PER_NODE=4 ./primus-cli container -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_1B_gdn_pure-pretrain.yaml
 
 # Zebra-Llama 3B
-EXP=examples/megatron/configs/MI300X/zebra_llama_3B-pretrain.yaml \
-bash examples/run_local_pretrain.sh
+./primus-cli container -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_3B-pretrain.yaml
 
 # Zebra-Llama 8B
-EXP=examples/megatron/configs/MI300X/zebra_llama_8B-pretrain.yaml \
-bash examples/run_local_pretrain.sh
+./primus-cli container -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_8B-pretrain.yaml
 ```
 
 ### Multi-Node (Slurm)
 
 For multi-node training on a Slurm cluster:
 
 ```bash
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \
-DATA_PATH=/shared/data \
-NNODES=2 \
-bash examples/run_slurm_pretrain.sh
+export DATA_PATH=/shared/data
+./primus-cli slurm srun -N 2 \
+  -- container --volume "$DATA_PATH:$DATA_PATH" \
+  --env DATA_PATH \
+  -- train pretrain --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml
 ```
 
 Ensure the `global_batch_size` in your config is divisible by `micro_batch_size * GPUS_PER_NODE * NNODES`.
 
 ### If Already Inside a Container
 
-If you are already inside a Docker container or on a bare-metal node with the environment set up:
+If you are already inside a Docker container or on a bare-metal node with the environment set up, use `direct` mode instead of `container` — it skips the container launch and runs `torchrun` in place:
 
 ```bash
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \
-bash examples/run_pretrain.sh
+./primus-cli direct -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml
 ```
 
 ### Mock Data (Smoke Test)
 
-To quickly verify the model runs without real data, the 3B and 8B configs come with `mock_data: true` by default. For the 1B configs, you can override:
+To quickly verify the model runs without real data, the 3B and 8B configs come with `mock_data: true` by default. For the 1B configs, you can override on the command line — every argument after `--config` is forwarded to the Primus Python CLI:
 
 ```bash
-EXP=examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \
-bash examples/run_local_pretrain.sh \
+./primus-cli container -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_1B_kda-pretrain.yaml \
   --mock_data true --train_iters 10
 ```
 
@@ -610,9 +610,8 @@ Primus/
 │   │   ├── chat_zebra_llama.py                  # Interactive chat
 │   │   └── convert_fla_to_megatron.py           # FLA Arrow → Megatron binary converter
 │   └── docker/start_container.sh                # Dev container launcher
-├── examples/
-│   ├── run_local_pretrain.sh                    # Single-node Docker launcher
-│   ├── run_slurm_pretrain.sh                    # Slurm launcher
-│   └── run_pretrain.sh                          # Core training entrypoint
+├── runner/
+│   ├── primus-cli                               # Unified launcher (direct/container/slurm)
+│   └── helpers/envs/base_env.sh                 # NCCL/ROCm/cache env defaults
 └── requirements.txt                             # Python dependencies
 ```

docs/04-technical-guides/hybrid-models/gdn-fla-parity.md

@@ -36,8 +36,9 @@ Inside the `rocm/primus:v26.2` container with the repo mounted at
 # Launch training (8 GPUs by default). The Megatron-LM behavioral patches
 # below are applied automatically at startup via Primus's patch system --
 # no separate apply step needed.
-EXP=examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml \
-  bash examples/run_pretrain.sh 2>&1 | tee primus_gdn.log
+./primus-cli direct --log_file primus_gdn.log \
+  -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml
 ```
 
 Optional toggles (all default off unless noted).  Each is exposed at

docs/04-technical-guides/hybrid-models/gdn-guide.md

@@ -256,8 +256,9 @@ The architecture-only YAML it extends from is `[primus/configs/models/megatron/z
 
 ```bash
 # inside the container, in /home/<user>/Primus
-EXP=examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml \
-  bash examples/run_pretrain.sh 2>&1 | tee primus_gdn.log
+./primus-cli direct --log_file primus_gdn.log \
+  -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_300M_gdn_pure-pretrain.yaml
 ```
 
 This brings up `torchrun` with 8 ranks on the local node. Expected wall time on a healthy MI300X box: **~1h 54m** for the full 4768 iters.

docs/04-technical-guides/hybrid-models/kda-fla-parity.md

@@ -77,8 +77,9 @@ python tools/hybrid/convert_fla_kda_init_to_megatron.py
 # 2. Launch training (8 GPUs by default). The Megatron-LM behavioral
 #    patches (same set as GDN) are applied automatically at startup via
 #    Primus's patch system -- no separate apply step needed.
-EXP=examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml \
-  bash examples/run_pretrain.sh 2>&1 | tee primus_kda.log
+./primus-cli direct --log_file primus_kda.log \
+  -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml
 ```
 
 ### Recommended toggle profile (YAML or env var)

docs/04-technical-guides/hybrid-models/kda-guide.md

@@ -263,8 +263,9 @@ The architecture-only YAML it extends from is
 
 ```bash
 # inside the container, in /home/<user>/Primus
-EXP=examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml \
-  bash examples/run_pretrain.sh 2>&1 | tee primus_kda.log
+./primus-cli direct --log_file primus_kda.log \
+  -- train pretrain \
+  --config examples/megatron/configs/MI300X/zebra_llama_300M_kda_pure-pretrain.yaml
 ```
 
 Expected wall time on a healthy MI300X box: **~1h 56m** for the full 4768

docs/04-technical-guides/mega-moe.md

@@ -144,7 +144,7 @@ balancing (for reproducibility) but preserves the step-to-step shape variation o
 The examples below use the rebuild hook (`REBUILD_PRIMUS_TURBO=1
 PRIMUS_TURBO_REF=9b5d3092efcbc087657b233d8e9ae662cee6ec6b`) to build Primus-Turbo from source.
 
-### Example 1 — single-node EP8, 4 layers (`run_pretrain_cli.sh`)
+### Example 1 — single-node EP8, 4 layers (`primus-cli direct`)
 
 1 node × 8 GPUs, `TP=1 / PP=1 / EP=8`, DeepSeek-V3 BF16, `GBS = MBS*GPUS*GA = 2*8*64 = 1024`.
 Minimal fused-MegaMoE run from `Primus/`:
@@ -161,7 +161,7 @@ export PRIMUS_TURBO_REF=9b5d3092efcbc087657b233d8e9ae662cee6ec6b
 export GPU_ARCHS=gfx950
 
 # Parallelism (EP-only) + fused MegaMoE
-bash examples/run_pretrain_cli.sh \
+./primus-cli direct -- train pretrain --config "$EXP" \
   --num_layers 4 \
   --micro_batch_size 2 \
   --global_batch_size 1024 \
@@ -198,7 +198,7 @@ export REBUILD_PRIMUS_TURBO=1
 export PRIMUS_TURBO_REF=9b5d3092efcbc087657b233d8e9ae662cee6ec6b
 export GPU_ARCHS=gfx950
 
-bash examples/run_pretrain_cli.sh \
+./primus-cli direct -- train pretrain --config "$EXP" \
   --num_layers 4 \
   --micro_batch_size 2 \
   --global_batch_size 1024 \
@@ -237,7 +237,11 @@ export USING_AINIC=1
 # Toggle the fused MegaMoE layer + model config
 export EXP=examples/megatron/configs/MI355X/deepseek_v3-BF16-pretrain.yaml
 
-bash examples/run_slurm_pretrain_cli.sh \
+# USING_AINIC / REBUILD_PRIMUS_TURBO / GPU_ARCHS are forwarded into the container
+# by the default env whitelist in runner/.primus.yaml, and PRIMUS_TURBO_REF by the
+# automatic PRIMUS_* passthrough, so none of them need an explicit --env.
+./primus-cli slurm srun -N "$NNODES" -- container \
+  -- train pretrain --config "$EXP" \
   --train_iters 15 \
   --micro_batch_size 2 \
   --global_batch_size 1024 \

docs/04-technical-guides/multi-node-networking.md

@@ -13,7 +13,7 @@ This guide summarizes how Primus configures networking, how **InfiniBand**, **Ro
 | Socket / interface detection | `runner/helpers/envs/get_ip_interface.sh` |
 | AINIC hook (container/CLI integration) | `runner/helpers/hooks/03_enable_ainic.sh` |
 | AINIC CLI defaults | `runner/use_ainic.yaml` |
-| ANP / `NCCL_NET_PLUGIN` example | `examples/run_pretrain.sh` |
+| ANP / `NCCL_NET_PLUGIN` selection | `runner/helpers/hooks/03_enable_ainic.sh` |
 
 ---
 
@@ -91,7 +91,7 @@ Adjust **`NCCL_IB_GID_INDEX`** and **`container.options.image`** to match your c
 
 ### RCCL network plugin (ANP)
 
-For ANP-based networking, clusters often set **`NCCL_NET_PLUGIN`** to **`librccl-anp.so`** when that library is present under `ANP_HOME_DIR`, falling back to `librccl-net.so` otherwise—see the logic in `examples/run_pretrain.sh`. This complements the library paths from `03_enable_ainic.sh`.
+For ANP-based networking, `NCCL_NET_PLUGIN` is set to **`librccl-anp.so`** when that library is present under `ANP_HOME_DIR`, falling back to `librccl-net.so` otherwise. This selection and the matching library paths both live in `runner/helpers/hooks/03_enable_ainic.sh`, so they apply to every launcher mode.
 
 ### Variables commonly set for AINIC