Add linear length penalty as a summed GRPO advantage

[hallerite]

Stacked on #2746 (base: feat/algorithm-abstraction).

What

Brings the MAI linear length penalty (from #2702) into the algorithm abstraction as a third length-penalty arm on the GRPO-family algorithms, plus an optional length-weighted baseline. No new algorithm type — length penalties stay "just configs" under [orchestrator.advantage.length_penalty].

• type = "linear" (LinearLengthPenaltyConfig): subtracts coef * pass_rate * (completion tokens / orchestrator.seq_len) from each reward before the baseline, where pass_rate is the group's mean reward — so reliably-solved problems get the strongest concision pressure and never-solved groups get none. Optional gate_by_correctness restricts it to correct rollouts (reward == 1).
• length_weighted_baseline on GRPO: use sum(len_i · reward_i) / sum(len_i) as the baseline instead of the plain mean.

The existing tokens / turns efficiency-shaping arms are unchanged.

The penalty is a summed advantage, not a folded reward

Because centering is linear:

center(reward − penalty) = center(reward) + center(−penalty)

So the linear penalty is modeled as a standalone additive advantage that sums onto plain GRPO, rather than threading penalty/baseline knobs through one shared group-norm function. advantage.py now exposes three pure, cohesive functions:

• grpo_advantage(group, length_weighted_baseline) — plain GRPO (owns the baseline choice).
• length_penalty_advantage(group, config, max_seq_len, length_weighted_baseline) — the group-centered negative penalty −(pᵢ − baseline) (owns max_seq_len).
• efficiency_shaping_advantage(group, config) — tokens/turns (non-additive; replaces the baseline).

GRPOAlgorithm.score_group composes GRPO + linear penalty by summation. tokens/turns remain non-additive shaping. The decomposition is exact only when both terms use the same baseline operator, so length_weighted_baseline is threaded into the penalty term too (it centers by the plain or the token-length-weighted mean to match the GRPO term).

Wiring

max_seq_len (the penalty denominator) is injected onto the Algorithm by build_algorithm and threaded from the orchestrator (config.seq_len) through TrainEnvs — so it touches only the GRPO path rather than every algorithm's constructor.

Verification

Numerically validated against a faithful reimplementation of #2702's original folded default_advantage_fn over 2000 random groups per case: all four gate_by_correctness × length_weighted_baseline combinations are bit-for-bit identical. (An earlier cut centered the penalty by the plain mean unconditionally and diverged from #2702 when the linear penalty met length_weighted_baseline; the fix threads the baseline operator into the penalty term — see the second commit.)

uv run pytest tests/unit/orchestrator -q → all passing. New unit tests cover pass-rate scaling, the GRPO + penalty sum equivalence (vs. folding penalty into reward, plain and length-weighted), correctness gating, zero pass-rate no-op, and the end-to-end GRPOAlgorithm.score_group path including max_seq_len injection. Lint clean.

🤖 Generated with Claude Code