feat(orchestrator): add trace support on algorithm abstraction

[willccbb]

Summary

Adds the trace-native env/orchestrator/training base layer on top of feat/algorithm-abstraction, while intentionally preserving the existing algorithm runtime shape. This is the base branch for #2842.

Current scope:

• carries verifiers v1 traces through the orchestrator as typed rollout objects
• makes EnvConfig inherit the verifiers v1 env-server config surface while prime-rl keeps orchestration-only fields such as env name, address, ratio, and retry defaults
• keeps v0 envs behind the narrow legacy bridge through id + args
• converts traces into trainer samples from branch token/logprob/mask data
• updates sinks, filters, metrics, monitors, and batching to read trace-native rollout fields
• keeps the old algorithm runtime in place; the registry/advantage rewrite is isolated to feat(orchestrator): use trace algorithm registry #2842
• pins verifiers v1 and installs the bundled harnesses / tasksets plugin packages in the root prime-rl environment; the slim prime-rl-configs package depends only on verifiers config types

Deliberately out of scope here:

• no new demo config tree
• no deps/prime submodule path
• no inference-server monkeypatch or template rewrites
• no trace algorithm registry rewrite

Stack

This is PR 1 of 2:

1. feat(orchestrator): add trace support on algorithm abstraction #2857: trace support into feat/algorithm-abstraction
2. feat(orchestrator): use trace algorithm registry #2842: trace algorithm registry rewrite into this branch

The branch should stay focused on trace/v1 support. Algorithm API replacement belongs in #2842.

Validation

• UV_CACHE_DIR=/tmp/uv-cache uv run pytest 'tests/unit/test_configs.py::test_load_configs[configs/rlm_swe/qwen35_4b.toml]' -q -> pass
• UV_CACHE_DIR=/tmp/uv-cache uv run pytest 'tests/unit/test_configs.py::test_load_configs[examples/glm5_pd_disag/rl.toml]' -q -> pass
• UV_CACHE_DIR=/tmp/uv-cache uv run pytest 'tests/unit/test_configs.py::test_load_configs[configs/general_agent/rl_qwen3_30b_a3b.toml]' -q -> pass
• UV_CACHE_DIR=/tmp/uv-cache uv lock --check -> pass
• commit hooks: ruff check and ruff format passed on cleanup commits
• Earlier on this stack, before the cleanup/narrowing pass: targeted unit suites passed (254 passed, 1 skipped on the base branch; 69 passed for the focused algorithm/batch subset)

---

Note

High Risk
Large orchestrator/env-server boundary refactor touching rollout lifecycle, config migration, and launcher process topology; incorrect trace→sample conversion or legacy bridge behavior would silently break training data.

Overview
Replaces v0 RolloutOutput / trajectory-step plumbing with verifiers v1 Trace objects end-to-end, while keeping the existing algorithm hook shape (RolloutView, scoring stages) for the follow-up registry PR.

EnvConfig now subclasses verifiers.v1.EnvServerConfig (taskset, pool, timeouts) with prime-rl fields (address, ratio, retries) and a v0 legacy bridge via id + args; num_workers migrates to pool. Training/eval sampling uses task_idx over server-reported num_tasks instead of loading datasets in the orchestrator. Env servers run through serve_env / EnvClient; the RL launcher can spawn one env-server per env on fixed ports (env_server_base_port) and writes per-env TOMLs before subconfigs.

Rollouts unify as Rollout(vf.Trace) with train/eval metadata; the dispatcher, sinks, filters, and metrics read trace fields (has_error, num_turns, completion_len, metrics) instead of raw["trajectory"]. trace_to_samples builds TrainingSamples per trace branch (mask/logprobs on flat tokens), replacing tokenizer backfill, interleaving, and image offload in the train sink. Echo and OPSD scoring paths are updated for branch/node traces; echo custom filters are rejected on v1.

Deps: root pins verifiers>=0.1.15.dev357 plus harnesses / tasksets; slim prime-rl-configs adds verifiers as a pydantic-only dep (CI no longer treats verifiers as a forbidden heavy import). Logging/docs use flatter logs/envs/{train,eval}/{name}.log. Trainer packing gains multimodal-aware truncation when sequences exceed seq_len.

^{Reviewed by Cursor Bugbot for commit 6d0f4da. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Missing EnvConfig import

High Severity

setup_env_servers filters env fields with EnvConfig.model_fields, but EnvConfig is never imported in this module, so a local uv run rl launch raises NameError when env servers are configured.

 

^{Reviewed by Cursor Bugbot for commit 5925104. Configure here.}

[cursor]

MITO renderer lost on save

Medium Severity

Removing the wrap serializer means orchestrator.renderer = None (MITO) no longer round-trips when subconfigs are written with exclude_none=True. The renderer key disappears from TOML and reload applies the default AutoRendererConfig(), switching MITO runs back to renderer-backed tokenization after resume or control-dir dumps.

 

^{Reviewed by Cursor Bugbot for commit 9cbbb53. Configure here.}

[cursor]

Breaking config lacks changelog

Low Severity

This PR removes or renames orchestrator env worker settings (num_workers → pool), drops default train/eval env entries, and requires env-server configs to specify env explicitly, but CHANGELOG.md is not updated with migration notes for those breaking config surfaces.

Additional Locations (1)

• packages/prime-rl-configs/src/prime_rl/configs/env_server.py#L7-L9

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 9cbbb53. Configure here.}

[cursor]

Length penalty ignores harness metric

Medium Severity

Token length penalty now sums raw tool-message node tokens via RolloutView.tool_response_len, but the config still documents harness metrics like *_total_tool_response_tokens. Environments that only populate the deduped metric (not per-node tool spans) get zero tool cost and incorrect advantage shaping.

Additional Locations (1)

• src/prime_rl/orchestrator/algo/advantage.py#L43-L49

 

^{Reviewed by Cursor Bugbot for commit 9cbbb53. Configure here.}

[cursor]

Rollout timing total zeroed

Low Severity

timing_df sets total to scoring.end - timing.start only when scoring.end is truthy; otherwise it records 0.0. Rollouts without a populated scoring end timestamp report zero wall time in dashboards despite nonzero setup or generation durations.

 

^{Reviewed by Cursor Bugbot for commit 9cbbb53. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 6 total unresolved issues (including 5 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d6f7a49. Configure here.}