exp: scaleswe ablation

[S1ro1]

What

configs/debug/v1/glm45air_scaleswe.toml — a v1 RL ablation for GLM-4.5-Air (100B MoE) on the scaleswe-v1 SWE taskset (train) + swebench-verified (eval), bash_edit harness on prime sandboxes.

This PR only adds the config TOML — no source/template changes.

Config highlights

• 2 train + 4 infer nodes. Trainer: cp=8 (ulysses), muon, + the GLM-5.1 prod-run memory improvements (LM-head token chunking, activation checkpointing + offload max_inflight=1, optimizer CPU offload, skip-gather/skip-optimizer ckpt).
• Inference: 4× tp=8 replicas, no expert parallelism. No fp8 quant on the inference side (dropped to lower train/inference mismatch); prefix caching on.
• Router replay on (trainer.enable_router_replay + inference.enable_return_routed_experts): replays inference's routed-expert decisions in the trainer to cut train/inference mismatch.
• 65536 context (seq_len + max_model_len), max_inflight_rollouts = 512.
• Node-local cache dirs via [env_vars] (see 2026-06-29 below) — no SLURM template edits.
• Eval: SWE-Bench Verified at step 0 + every 20 steps (skip_first_step = false).
• Renderer (glm-4.5) and tool/reasoning parsers auto-resolve from the official zai-org/GLM-4.5-Air slug.

Changelog

2026-06-29

• Merged latest main (now includes the configurable env-vars feature, feat: configurable env vars #2863).

• Moved the node-local cache dirs out of the SLURM templates and into the config, so the PR only adds the TOML:

  • top-level [env_vars]: TRITON_CACHE_DIR (trainer + inference)
  • [inference.env_vars]: VLLM_CACHE_ROOT, FLASHINFER_WORKSPACE_BASE

  The SLURM templates are now untouched vs main. Values keep the shell-expanded per-user/per-job paths ($USER/$SLURM_JOB_ID/$(whoami)); the multi-node launcher renders env_vars as export KEY="VALUE", so the expansion still happens at runtime.

• Migrated the renderer field preserve_all_thinking = true → thinking_retention = "all" (main Update configs for renderer thinking retention #2900 removed the old bool; the config otherwise failed the config-load unit test).

2026-06-26

• Brought back router replay (trainer.enable_router_replay + inference.enable_return_routed_experts) — compatible again now that kv-cache offload and trainer fp8 (both mutually exclusive with router replay) have been dropped; replaying inference's routed-expert decisions cuts the train/inference mismatch.
• Merged main (drops removed envs; bumps verifiers + research-environments).
• Scaled inference to 4 nodes / 4 replicas (num_infer_replicas 2 → 4).
• Bumped max_inflight_rollouts 384 → 512 (saturate the 4 inference replicas).
• Fixed FlashInfer JIT cache deadlock: pin FLASHINFER_WORKSPACE_BASE to node-local /tmp (was defaulting to shared weka $HOME/.cache/flashinfer, where concurrent GLM/MoE runs deadlock on fused_moe_*.lock → inference never serves). (2026-06-29: relocated from the sbatch templates into the config.)

2026-06-25

• Merged main.
• Dropped router replay (trainer.enable_router_replay + inference.enable_return_routed_experts).
• Switched train + eval harness bash → bash_edit.
• Dropped inference fp8 quant (vllm_extra.quantization) to lower train/inference mismatch.
• Eval at step 0 + every 20 steps (skip_first_step = false).
• Kept LM-head chunking + activation checkpointing explicit at the upcoming feat: add orchestrator debug mode (no-inference, no-trainer) #2867 default values.
• Dropped the -lp length-penalty variant (glm45air_scaleswe_lp.toml).

2026-06-24

• Initial config: GLM-4.5-Air on scaleswe-v1 (train) + swebench-verified-v1 (eval), bash harness on the prime-sandbox runtime, router replay enabled, cp=8 ulysses + muon + GLM-5.1 prod-run memory knobs.
• Online fp8 inference (vllm_extra.quantization = "fp8") + 384 rollout concurrency.
• Bumped weight_broadcast NCCL timeout to 3600s for cold NFS loads.

---

Note

Low Risk
Config-only addition with no application or template changes; affects experiment launch parameters only.

Overview
Adds configs/debug/v1/glm45air_scaleswe.toml only — a v1 RL experiment wiring GLM-4.5-Air to scaleswe-v1 training and SWE-Bench Verified eval on prime sandboxes with the bash_edit harness (not the base scaleswe rlm setup).

Deployment & scale: 2 train nodes, 4 inference replicas at tp=8 (no expert parallelism), max_inflight_rollouts = 512, max_steps = 1000, 65k context. Weight broadcast NCCL timeout 3600s.

Trainer: cp=8 ulysses, muon, router replay on, plus GLM-5.1-style memory knobs (LM-head token chunking, activation checkpointing/offload, optimizer CPU offload, skip-gather / skip-optimizer checkpoints).

Inference: prefix caching, enable_return_routed_experts for replay; no fp8 quant. Eval runs at step 0 and every 20 steps.

Ops: top-level [env_vars] and [inference.env_vars] point Triton, vLLM, and FlashInfer caches at node-local /tmp to avoid shared-FS contention and FlashInfer lock deadlocks; SLURM pre_run_command deletes orphaned prime sandboxes by job label.

^{Reviewed by Cursor Bugbot for commit 2eea033. Bugbot is set up for automated code reviews on this repo. Configure here.}