feat: configurable env vars

[mikasenghaas]

Summary

Consolidate environment-variable handling for RL and SFT: env vars are set per component from config, with one precedence model applied uniformly across single-node and multi-node (SLURM) launches.

• Per-component env_vars: [trainer.env_vars], [orchestrator.env_vars], [inference.env_vars] (RL) and [sft.env_vars] (SFT). Disaggregated inference keeps deployment.{prefill,decode}_env_vars layered on top per role.
• Single source of truth for launcher defaults in utils/process.py: DEFAULT_COMMON_ENV_VARS (every component), DEFAULT_TRAINER_ENV_VARS, DEFAULT_INFERENCE_ENV_VARS. No static defaults are hardcoded in the sbatch templates anymore.
• Precedence (low → high): os.environ < launcher defaults < [component.env_vars] < protected (CUDA_VISIBLE_DEVICES, WANDB_SHARED_*). Single-node builds the env inline at each launch site; multi-node templates loop the merged dict; standalone uv run inference applies it in inference_local.
• Protected vars are enforced, not just documented: setting CUDA_VISIBLE_DEVICES or WANDB_SHARED_* in any env_vars now fails config validation (reject_protected_env_vars / PROTECTED_ENV_VARS).
• Deep-GEMM (VLLM_USE_DEEP_GEMM/VLLM_MOE_USE_DEEP_GEMM) is driven solely by config.use_deep_gemm via setup_vllm_env — single source (removed the hardcoded template export and a redundant config validator).
• HF_HUB_OFFLINE and WANDB_SHARED_RUN_ID now read from the environment, falling back to the default if unset.
• Multi-node uses a single uv sync --all-extras on the batch node; the checkout + venv are expected on a shared filesystem (documented in scaling.md).

Changed defaults

Multi-node values are preserved (just moved into DEFAULT_*). The material changes are single-node parity + two removals:

Path	Newly applied / changed
single-node inference	now inherits DEFAULT_INFERENCE: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False, VLLM_WORKER_MULTIPROC_METHOD=spawn, VLLM_ENGINE_READY_TIMEOUT_S=4200 (vLLM 0.23 default is 600), UCX_TLS=all; plus DEFAULT_COMMON (CUDA_DEVICE_ORDER, OMP_NUM_THREADS=1, GIT_LFS_SKIP_SMUDGE). Previously single-node inference set only CUDA_VISIBLE_DEVICES.
single-node RL trainer / orchestrator / SFT	now get DEFAULT_COMMON — notably OMP_NUM_THREADS=1 (was unset single-node, i.e. all cores).
all (multi-node)	removed TRITON_CACHE_DIR and inference VLLM_CACHE_ROOT.

Breaking

• Config rename: deployment.prefill_env_overrides → deployment.prefill_env_vars and deployment.decode_env_overrides → deployment.decode_env_vars (no back-compat alias). The one in-repo config using the old names (configs/glm5.2_16node_llmd/inference.toml) is updated; update any external disaggregated configs.
• Protected keys rejected: a [component.env_vars] (or deployment.{prefill,decode}_env_vars) setting CUDA_VISIBLE_DEVICES or WANDB_SHARED_* now raises at config-parse, instead of being silently ineffective (or, on multi-node, overriding the shared W&B run id). No in-repo config does this.

Docs

• docs/configuration.md — [component.env_vars] model + precedence rules.
• docs/scaling.md — shared-filesystem venv expectation for SLURM.
• docs/inference.md — the P/D env-var rename.

---

Note

Medium Risk
Touches all launch paths (single-node, SLURM, standalone inference) and changes default process environment (e.g. single-node OMP_NUM_THREADS, inference vLLM timeouts), which can affect performance or stability; protected-var validation is a breaking config change for anyone overriding W&B or GPU visibility via TOML.

Overview
Adds configurable env_vars across RL, SFT, and inference, with launcher defaults centralized in utils/process.py (DEFAULT_COMMON_ENV_VARS, DEFAULT_TRAINER_ENV_VARS, DEFAULT_INFERENCE_ENV_VARS) instead of hardcoded exports in sbatch templates.

RL gets top-level [env_vars] plus [trainer.env_vars], [orchestrator.env_vars], and [inference.env_vars]; SFT and inference get their own [env_vars]. Single-node launchers merge defaults → shared → component env at each Popen; SLURM templates loop merged dicts (trainer_env_vars, etc.). Standalone uv run inference applies the same merge in-process before setup_vllm_env.

Disaggregated P/D renames prefill_env_overrides / decode_env_overrides → prefill_env_vars / decode_env_vars (breaking, no alias). EnvVars validation rejects CUDA_VISIBLE_DEVICES and WANDB_SHARED_* in user config.

Docs cover precedence and shared-filesystem venv for SLURM. Notable behavior shifts: single-node inference/trainer now inherit the same default env as multi-node; TRITON_CACHE_DIR and VLLM_CACHE_ROOT are removed from templates; WANDB_SHARED_RUN_ID can be preset via the environment.

^{Reviewed by Cursor Bugbot for commit 676a4fb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1944a35. Configure here.}