[Wanli-Jiang]

@coderabbitai summary

Description

trtllm-modeling-v2 is an agent-flow workflow that takes a HuggingFace checkpoint, a cluster site.yaml and a one-line goal, builds the model as a ModelingV2 target in TensorRT-LLM, proves it correct at every requested parallel configuration, and can then tune its serving performance. Agents write the code; whether something is done is decided only by Python checks (boot, accuracy, acceptance, and opt-in witnesses) on the measured tree.

This PR unifies #20145 (the gated onboarding workflow, used as the base) with the parts of #20792 (the modeling-v2-onboard planner/worker workflow) that the merge plan kept.

What a run does

• The spec: --task task.yaml, or --checkpoint, --site and --goal, which derive task.derived.yaml in one step; an open question stops the command before the run starts and is answered in that file.
• Preflight: the checkpoint against the spec (heads and experts split by every member, weight format, draft layers), host, launch substrate, KV-cache and capture-list configuration, an agent probe (every routed agent must run a shell command and write a file), and a substrate gate that runs before the planned loop.
• Planned run: a surveyor maps the model's steps onto the op catalog while the planner writes the task graph; each queue row becomes a kernel task with no planner turn. Workers (surveyor, kernel_writer, assembler, tuner) share one prompt plus per-tag sections and end their turns with a decision file. Tasks run in their own worktrees and are integrated by rebase and fast-forward. The orchestrator compiles the contract, runs its checks on read-only snapshots, adds a regression check (no agent turn) after any shared change, and wakes the planner on events.
• Delivery: <branch>-correct once every required check holds on one commit, <branch>-best, cells.json, metrics.json.
• Tune (optional): a campaign of one-hypothesis tuner turns, judged against a same-host control and the tree's own noise, gated by the contract's accuracy checks; results are listed in UNMERGED_OPTIMIZATIONS.md, never merged into the delivered target.

Running it from Claude Code or Codex

Started from a Codex session (an interactive one or a /goal), the orchestrator runs every role on Codex, on the model that Codex is set to and with the newer of its codex and the pinned runtime; the run copies its skills where Codex finds them (<workspace>/.agents/skills). In-process Codex roles now fork primers through native app-server thread/fork, with a saved-summary fallback when a fork fails. Unspecified Codex reasoning effort and context-window settings retain native defaults. Each role session asks for the most open sandbox and approval policy the site's Codex requirements allow (full access where allowed, else workspace-write with the checkout and spools writable, and on-request approvals the orchestrator grants), records what it actually ran under, and the agent probe stops the run if Codex would run the roles read-only. The Codex session starts the orchestrator detached, outside its sandbox (user guide, Starting a run). Verified on a ChatGPT-login Codex: the workflow's agent probe passes from inside a Codex session with no agents block, and a Codex role loads the entry skill.

Taken from #20792

The operator inbox (trtllm-modeling-v2 feedback), job priority classes (accuracy, then feature, then perf), primer sessions (on by default, --no-primers), the unmerged-optimizations list, and agent-flow fixes (session fork, DCO sign-off on node commits and merges, cancelling a local job's whole process group).

Changes since #20145 worth a reviewer's attention

• Roles: one worker prompt with per-tag sections; five roles (planner, surveyor, kernel_writer, assembler, tuner). The reviewer roles are gone: a verdict is a check's, and a shared change is followed by a regression check.
• Primers are on by default; a turn that cannot fork reads the primer's summary.
• The package was slimmed: dead and test-only code, duplicated helpers and most explanatory comments removed; the tests now keep only what guards a workflow decision (current CPU validation is listed below).
• Removed mechanisms: the cross-run lesson store and the improver role, gate bundles, the phase-0 geometry probe, the covered-certification shortcut, the flow's own disaggregation leg B (a disaggregated unit that owes the equivalence witness now declares contrast_command and contrast_env), the manual perf subcommand, nsys_summary.py and microbench.py, near-miss stacking, the spool's memory-pressure kill and the external driver-heartbeat check, and the external agent transport (--agent-transport external and the driver's turn hand-out): every role turn runs in process, and the agent probe proves each agent can take a turn there. A kernel writer's engine_defect report reaches the planner in the failed-receipt event; the planner plans its repair.
• Decisions are read directly from the path reserved for the current dispatch, rather than copied into progress.yaml. A missing answer cannot fall back to an earlier attempt's decision; kernel writers use the same dispatch/adoption path as other tasks.

Backend recovery and task policy

• Completed, unconsumed turns from the same task attempt survive restarts. A relaunch can integrate the original decision without repeating the worker's tools; a different attempt or explicit rebase still gets fresh work. Shared-code regression coverage now remains active after restart.
• Workers diagnose ordinary build, import and launch failures inside their existing task goal and owned paths, use existing build/test tools and record runtime evidence before returning. Planners read prior receipts and explain changed prerequisites, new evidence or a new diagnostic hypothesis before retrying. Prerequisite fixes use the existing depends_on graph; unresolved blockers with no runnable remedy use the existing escalation path.
• Codex returns child session IDs to AgentLayer; changed agent routing invalidates primers. Terminal records retain reported model/effort, elapsed time and outcome.
• Diagnostic and retry strategy stays in prompts. This adds no blocker state machine, file fingerprints, native repair runner, automatic exclusivity rule or fixed event truncation. Original numerical and delivery gates decide acceptance; prompt instructions are not a hard scheduler guarantee against redundant retries.

Codex roles under site policies, and runs that keep going

From a real Codex-driven onboarding run (25 h to bring-up, stopped by the operator), the general fixes:

• Codex role sessions no longer ask for danger-full-access/never unconditionally. On a site whose /etc/codex/requirements.toml forbids them, Codex silently ran every role read-only (105 Read-only file system failures in 91 of 483 sessions). Sessions now ask for what the requirements allow, record the effective sandbox and approval policy in each turn's .done, and the agent probe fails a session that would run read-only.
• The Codex app server is started again when it exits at startup (concurrent starts collide on Codex's shared state database).
• Operator feedback is read while only long checks run; an attempt whose worktree could not be prepared does not count toward MAX_ATTEMPTS.
• ESCALATE no longer ends the run while other work can move: the work that does not wait on the answer goes on, and the run ends escalated once nothing runs and nothing new came in.
• A state write refused for room (EDQUOT, ENOSPC) no longer kills the loop: the state stays in memory and is written again each pass, and nothing new starts until it lands.

Run-specific fixes from that run (sparse worktrees for an inode quota, per-unit check sessions for a two-node topology, fixed-length perf validation) are not included.

A leaner harness

The aim is the smallest harness that still decides everything a run needs. A third pass removes duplicated state, redundant dispatch paths and optional scheduler controls while keeping tune and the two agent-flow core changes in this PR. The third pass reduces the workflow package (including docs) from 54,255 to 49,346 lines (−4,909), and its tests from 10,974 to 10,551 (−423), relative to the preceding PR head.

• No reload or drain: a flow fix reaches a run by stopping it and running the same command again. The new process sets running tasks back to pending, adopts each turn that already answered, and runs again what the stop cut short. Ctrl-C ends the orchestrator at once, without waiting for the turns and checks in flight.
• Agents commit with plain git commit -s in their own worktree; the locking commit.sh helper and --no-task-worktrees are gone. The CLI exports the run's identity for every agent, and before a task is integrated its commits are audited (sign-off, identity, no co-author, symlink, sandbox placeholder or path outside commit_policy.allowed_paths); a commit that breaks a rule goes back to its worker like a rebase conflict.
• A session is one node, handed over by hand: managed sessions (--manage-sessions), multi-node sessions and session status/session watchdog are gone; driver status and driver wait cover the last two.
• One reading of a spool's liveness (scheduler.watcher_records) replaces six. Checks are not held in the orchestrator until a session reads as free: at most 32 are in flight on sessions, and each session's scheduler queues the rest by fit, class and age. Overflow to other sessions, lending tune dev sessions and moving queued jobs between spools are gone.
• infra/ is laid out by purpose: environment.py, git.py (git questions and the commit rules), gpu/ (sessions, spools, the scheduler and who runs a command) and driver/ (the operator loop and the orchestrator heartbeat); the turn executor and agent slots are in agents/.
• The run loop: bring-up and correctness share one milestone writer; stale checks run again in contract order, with no risk ordering or memory-margin tracking; the planner's prompt carries the newest 40 events, each clipped; the delivery files are a task the planner plans; integration keeps no attempts file, and the journal numbers its events in memory. A gate lost to its session leaves its check without a verdict, to run again; no run closes on it.
• No hand-kept registries: runtime_index.yaml, kernel_sources.yaml and runtime_map.py are gone. Workers read the code (the repository's AGENTS.md, grep), and the survey measures only what the build supplies (catalog entries and trtllm ops no entry wraps); the surveyor reads the checkpoint's demands itself.
• The spec in one step: spec/derive.py turns the checkpoint, site.yaml and the goal into task.derived.yaml beside derive.report.md, which says where every unstated value came from; a goal's <feature>@<members> writes the members it leaves out into coverage.accept. The intake package (compile, freeze and confirm, --confirm, the intake subcommand, INTAKE.md) is gone. What the checkpoint cannot meet (heads or experts that do not split, fp4 below sm_100, missing draft layers, a multimodal unit without a tower) is a preflight check, check_checkpoint_fit, for derived and written specs alike.
• One table per feature family: examples/ and the package: embargo entries are gone; which units owe acceptance or a witness is read from task_schema.SPECULATORS and witness.WITNESS_KINDS.
• Five main documents instead of twelve: the README, the user guide, the design doc with the developer guide folded in, TROUBLESHOOTING and task.example.yaml, each checked against the code.

Fewer, cheaper checks

A Codex-driven onboarding run owed 43 checks and ran 59 verdicts for them, 31 of them re-runs after a landing changed the code; its 19 full GSM8K runs (up to two hours each on a target not yet tuned) took about 21 h, three quarters of all check time. Its spec could not say what its notes asked (a multimodal smoke test; TP4/TEP4 text-only), so 15 h went to vision numerics. Now:

• Depth per unit. profile may be set per unit, with a new smoke level (the unit's boot alone); the goal says it as multimodal:smoke (or :full, the certified profile).
• One benchmark per numerics class. With a logit reference, a unit's benchmark runs at one member, the one the logit gate sees least (attention DP), else the widest; the logit gate and the boot's greedy tokens, now decoded as one batch, prove the others. A configuration variant's benchmark runs at one member too.
• Benchmarks re-run at milestones. While required work is open, a landing re-runs the cheaper checks; a stale benchmark runs again once nothing required is open, or for the bring-up before it is named.
• Disaggregation by its witness. A disagg unit and its variants owe the equivalence witness alone, always, through the spec's leg B: its boot and accuracy had run an aggregated engine. A derived spec puts it on members whose two legs fit one node (device.gpus_per_node); the units before it prove the wider members. A benchmark through the hop needs a framework leg on trtllm-serve's disaggregated serving, which is not done yet.
• Planner load. A green finish reaches the planner within 15 minutes (--event-window 900 s); failures, reports and an empty graph still wake it at once. A burst of operator messages reaches one turn. A goal the planner writes or changes is at most 2,000 characters: evidence and operator messages are cited by path.

A spec with that run's stated scope now compiles to 20 checks, 3 of them benchmarks (43 and 13 before).

Third-pass behavior changes

• Test results: no input tracing or cross-commit reuse. Regression checks run the declared ModelingV2 test list at HEAD and only failing tests at the base, one pytest process per file. Completed files survive an interrupted job; continuation is bounded. Results require an exact clean commit and a matching measured environment. Each record_at call first runs one lightweight environment-probe job, including cache hits; probe failure is incomplete, never a cached pass. No GPU timing claim is made for the full list.
• Delivery: final delivery runs missing delivered tests at the integrated HEAD. A failure closes delivery-failed; restarting lets the planner read that failure and plan a repair. Tests without recorded results do not block the earlier correctness milestone.
• Run control: the substrate gate runs before workers; the run-wide wall/device-hour budget and regression batching delay are removed. Per-turn/job timeouts and the tune campaign's own wall budget remain.
• Spools: lost allocations allow one rerun, tracked in the script. Handover is scheduler status plus SESSION_ENDING, not a separate request-file protocol. Removed host-memory admission, per-job memory limits, configurable rerun counts and manual --pick; automatic gate placement remains.
• Tuning: infrastructure/kernel/runtime blockers use BLOCKED and count as campaign misses. The tune session-staleness threshold now shares the 180-second watcher threshold.
• Comments: trim repeated descriptions and history while retaining protocol, concurrency, recovery, MPI safe-kill and measurement invariants. Comment-only edits are checked by AST comparison where pre-edit snapshots are available.

Commits

The change is split into 12 commits, one per layer, each holding its files in their final state with their tests, so each layer can be reviewed on its own. The layers import one another, so only the last commit is meant to build and pass the tests.

#	Commit
1	agent_flow: session forks; the agent CLI and sandbox a site allows
2	agent_flow: sign off worktree commits; cancel a job's process group
3	the task spec and the derived spec
4	the launch environment, git, sessions and spools
5	survey, vocabulary, promotion and delivery checks
6	gates, witnesses and their harness
7	roles, decisions, prompts and the entry skill
8	the contract, task graph, integration and records
9	preflight, the orchestrator loop and the workflow
10	the tune campaign
11	the CLI and package entry point
12	README, user guide, design and troubleshooting

Known gaps

• This unified branch has not run on a GPU yet; the first pinned GPU runs, with Codex as the main agent, are the next step, and will show what the narrower check scope saves.
• Authenticated integration of the new native Codex primer-fork path and a controlled Codex/Claude onboarding efficiency comparison remain to be run. CPU tests do not establish equal backend execution efficiency.

Test Coverage

• Full offline agent-flow/tests CPU suite at the head: 1,982 passed, 3 skipped, in 91.58 seconds; live backend checks are skipped. Both pre-commit configs are clean on the changed files. The tests keep only what guards a workflow decision; besides the layers' own, they cover the integration-time commit audit, the trial commit, the derived spec, the checkpoint-fit check and stopping on Ctrl-C. The spool scheduler and client run by path from a shell.
• A real Codex app server (0.162) honors the workspace-write request with the writable roots and on-request approvals, and the session records them; the site-policy read is covered by CPU tests, not yet exercised on the cluster whose policy motivated it.
• Both pre-commit configurations pass on all PR files. The requested pre-commit -a with the agent-flow configuration also traverses unrelated repository files and fails on existing spelling/lint issues; its unrelated automatic edits were reverted. Full repository lint/CI success is not claimed.
• The existing Codex ChatGPT-login agent probe verification remains separate from the new fork-path CPU tests. No GPU onboarding or real Codex/Claude efficiency A/B has been performed on the latest commit.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

[WeiHaocheng]

The scripts under onboard/, run/, spec/, and tune/ look like agent-generated tools from bringup. It might be hard for agents to actually reuse them later under this layout. I suggest moving them into a unified tools/ directory with a README documenting their capabilities. Some also seem quite bespoke to specific edge cases—we should review them to see what's genuinely worth keeping.