docs(tutorials): add black-box coding agent training tutorial (OpenCode + AsyncGRPO)

[sergiopaniego]

Summary

Adds a tutorial page for the black-box (loop-owning) training path: training the real OpenCode agent with TRL's experimental AsyncGRPOTrainer and the opencode_env transparent interception proxy. The recipe itself lives in TRL (docs section plus examples/scripts/openenv/opencode.py), so this page explains when to use the pattern, how the pieces fit, and links out. It also adds the row to the tutorials index and the toctree entry.

Until now this recipe was not discoverable from OpenEnv docs at all: the BrowserGym harness tutorial covers the white-box pattern where the trainer keeps sampling, and nothing pointed at the black-box path.

This is the index/discoverability piece discussed in RFC #924 (GRPO harness training recipes): recipes live in each framework's repo, OpenEnv curates the index.

Type of Change

• Documentation

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

RFC Status

• RFC exists: RFC: GRPO recipes #924 (in review, this PR is docs-only and useful independently)

Test Plan

• _toctree.yml parses (yaml.safe_load) and the new entry mirrors the existing tutorial entries.
• Docs-only change, no code paths touched.
• Links point at TRL main (docs/trl/openenv harness section, examples/scripts/openenv/opencode.py) and envs/opencode_env, all verified to exist.

Claude Code Review

N/A (docs-only)

---

Note

Low Risk
Documentation-only; no runtime, API, or training code changes.

Overview
Adds discoverability for the black-box GRPO path: training the real OpenCode agent (loop-owning) via TRL’s AsyncGRPOTrainer and opencode_env’s transparent proxy, complementing the existing Wordle (TRL-driven) and BrowserGym (white-box harness) tutorials.

New page opencode-agent-grpo.md explains when to use this pattern, the rollout flow (OpenCodeSessionFactory → trace capture → HarnessRolloutWorker → verify() → AsyncGRPOTrainer), and links to TRL’s harness docs, examples/scripts/openenv/opencode.py, and envs/opencode_env. _toctree.yml and tutorials/index.md register the tutorial in the docs nav and topic table (GPU: Yes, no Colab notebook).

^{Reviewed by Cursor Bugbot for commit e1e7a9b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Alignment Review Report

Scope: Docs-only PR — 3 files, +65 lines (docs/source/_toctree.yml, docs/source/tutorials/index.md, new docs/source/tutorials/opencode-agent-grpo.md). No Python/source changes.

Automated Checks

• Lint: N/A for this PR (no .py files changed). For reference I ran .claude/hooks/lint.sh after installing uv; it fails only on pre-existing ruff format drift in ~20 envs/* Python files (e.g. envs/opencode_env/*.py) that this PR does not touch — consistent with the baseline drift noted in AGENTS.md. (Side note, unrelated to this PR: the hook also reformats python code blocks inside envs/**/README.md but only reverts .py, so it leaves .md files dirty.)
• Debug code: CLEAN for this PR. .claude/hooks/check-debug.sh lists many print()s and 3 TODOs, but all are pre-existing in src/; none are in the changed docs files.

Doc-specific checks (all pass)

• _toctree.yml entry tutorials/opencode-agent-grpo resolves to the new file. ✓
• Internal links wordle-grpo.md and browsergym-harness.md exist. ✓
• API references are accurate: OpenCodeSessionFactory, transparent_proxy mode, and session verify() all exist in envs/opencode_env/harness.py, and the proxy description (/v1/chat/completions interception, per-turn ids + logprobs) matches envs/opencode_env/README.md. ✓
• > [!NOTE] alert syntax matches the established convention across the other tutorials. ✓

Open RFCs Context

All RFCs are In Review except RFC 010 (Draft). The directly relevant one is RFC 005 — Agentic Harness Integration (In Review, @Darktex), which is exactly the area this tutorial documents (training a real coding-agent harness).

Tier 1: Fixes Required

None. No mechanical, lint, debug, or broken-reference issues in the changed files.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. Notably the described reward path aligns with rewards inside environment (PRINCIPLES.md / RFC 004): the domain scoring lives in the session's held-out verify() inside the env, which "the agent never sees" — consistent with the reward-boundary and agents cannot reset invariants. Training the production agent as-is also supports minimize lifecycle deltas and minimize human-agent divergence.

RFC Conflicts

ALIGNMENT FLAG: Tutorial canonizes a harness-training design that diverges from the abstraction proposed in RFC 005 (In Review)

• Principle/RFC at stake: RFC 005 — Agentic Harness Integration (Status: In Review)
• The concern: RFC 005 proposes harness training via a HarnessEnvironment (extends MCPEnvironment) where each step() is one conversational turn the orchestrator drives, plus a HarnessAdapter, HarnessEvent/HarnessResponse trajectory, and a Rubric computing reward after each turn. This tutorial instead documents the shipped opencode_env + TRL path: the agent owns its entire loop, a transparent_proxy intercepts LLM calls to capture token ids/logprobs, TRL's HarnessRolloutWorker rebuilds training rows from the trace, scoring uses the session's verify(), and adaptation happens via rollout_reward_fn / train_turn_fn / agent_turn_fn. RFC 005 explicitly rejected the "each step() is the entire episode" model that this black-box path effectively uses. The divergence is not introduced by this PR (it already exists in opencode_env + TRL) — the PR just makes it canonical in user-facing docs. Per the two-tier model, flagging so the team can reconcile: most likely by updating RFC 005 to reflect the shipped Session / proxy / HarnessRolloutWorker design (or documenting both paths side by side). Non-blocking.
• Suggested reviewer: @Darktex (RFC 005 author; also authored the reward-boundary lines in PRINCIPLES.md/INVARIANTS.md). cc @burtenshaw (tutorials/docs).

Summary

• 0 mechanical issues to fix (Tier 1)
• 0 principle conflicts
• 1 RFC conflict to discuss (RFC 005 divergence — non-blocking; docs describe shipped reality, RFC likely needs updating)

Overall a clear, well-structured tutorial that accurately reflects the opencode_env implementation and is nicely positioned against the other two GRPO tutorials. The only item for humans is the RFC 005 reconciliation above.

  

_{Sent by Cursor Automation: Pre-review}

[cursor]

ALIGNMENT FLAG (Tier 2, non-blocking) — RFC 005: Agentic Harness Integration (In Review)

This "agent owns its loop" black-box model diverges from RFC 005's proposed abstraction, where harness training goes through a HarnessEnvironment with per-turn step() + HarnessAdapter + Rubric. RFC 005 explicitly rejected the "one step() = whole episode" model that this path effectively uses.

The divergence already exists in the shipped opencode_env + TRL code — this tutorial just makes it canonical in the docs. Worth reconciling: probably update RFC 005 to reflect the shipped transparent_proxy / HarnessRolloutWorker / session-verify() design (or document both paths). cc @Darktex (RFC 005 author).

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (3)

docs/source/tutorials/opencode-agent-grpo.md:14

• The NOTE block says "Three GRPO patterns, three tutorials", but this docs section now includes at least four GRPO-related tutorials (Wordle, BrowserGym harness, Pi, and this OpenCode page). This makes the statement factually incorrect and potentially confusing.

> [!NOTE]
> Three GRPO patterns, three tutorials. For a standard `reset()` / `step()`
> flow where TRL drives the episode, see the
> [Wordle GRPO tutorial](wordle-grpo.md). For harness rollouts where the
> trainer still generates each turn (white-box), see the

docs/source/tutorials/index.md:29

• The tutorial link text "Training a Real Coding Agent" is ambiguous now that there is also a separate "Training a Real Coding Agent (Pi)" tutorial. Including the agent name in the title would make the index clearer and consistent with the Pi entry.

| [Training a Real Coding Agent](opencode-agent-grpo.md) | Train the actual OpenCode agent (black-box, loop-owning) with TRL's `AsyncGRPOTrainer`: a transparent proxy captures each turn's token ids and logprobs while the agent runs its own tool loop. | Yes | — |

docs/source/_toctree.yml:56

• The toctree title "Coding Agent Training with TRL" doesn't indicate which agent this page targets, and it can be confused with the existing Pi coding-agent GRPO tutorial. Including "OpenCode" in the nav title would improve discoverability and symmetry with the Pi entry.

    title: Coding Agent Training with TRL