Add pelican_svg_env: blind SVG drawing, scored in three layers - #1019
Conversation
burtenshaw
added
environment
size: extra-large
Cursor
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| version = "0.1.2" | ||
| description = "Pelican SVG Environment for OpenEnv" | ||
| requires-python = ">=3.10" | ||
| dependencies = [ |
There was a problem hiding this comment.
Tier 1 (blocking CI): this env ships without a uv.lock — the only one of 37 envs missing it. The validate-env-locks CI job runs uv sync --frozen --all-groups --all-extras --dry-run --no-install-project here (triggered by this added pyproject.toml) and fails:
error: Unable to find lockfile at `uv.lock`, but `--frozen` was provided.
(reproduced on a clean checkout of this PR head). Fix: cd envs/pelican_svg_env && uv lock and commit the resulting uv.lock. The Docker build's uv sync --frozen also needs it to pin dependencies (reproducibility).
| "fastapi>=0.115.0", | ||
| "pydantic>=2.0.0", | ||
| "uvicorn[standard]>=0.24.0", | ||
| "requests>=2.31.0", |
There was a problem hiding this comment.
Minor: requests is declared but never imported anywhere in the env (grep finds it only here). Safe to drop unless an external consumer relies on it — the judge uses huggingface_hub and the client uses openenv.core.
| # synchronous `step()` path runs each call under its own short-lived | ||
| # loop, so a cached client fails with "Event loop is closed" on every | ||
| # request after the first. | ||
| async with AsyncInferenceClient( |
There was a problem hiding this comment.
Alignment ✓ (rewards inside the environment): this external VLM call runs server-side inside the reward path (step_async → evaluate_submission → VisionJudge.evaluate), so domain knowledge stays encapsulated per INVARIANTS "Rewards in environment" — it is not external reward computation. Nicely done too: the token comes from an env var and is never logged, building the client per-call sidesteps the "event loop is closed" reuse bug, and a judge failure returns error/scores 0 (sample marked unjudged) instead of raising — which test_judge_failure_does_not_raise_the_reward pins so "break the judge" can't inflate reward.
|
Addressed the review findings in ae7633e:
132 env tests, lint and the docs sync check all pass. |
|
Bugbot's second-pass finding was real too, fixed in 59157ba: |
|
One more edge, raised by an external review of 59157ba and fixed in the follow-up commit: the partial-pin fill could resolve to the canonical pair under Decision taken: |
|
Follow-up on the arc-flag finding, fixed in the latest commit: the path tokenizer was context-free, so compact arc flags ( Verified against the literal reported case, a negative coordinate glued to the flags ( |
Simon Willison's "generate an SVG of a pelican riding a bicycle" as an environment that can be evaluated and trained against. The model never sees its own output, so it has to hold the spatial arrangement in its head and emit coordinates for it. Scoring runs cheapest-first, so a submission that is not an honest attempt never reaches a paid call: - a source gate rejecting embedded rasters, text labels, external references and degenerate documents, before anything is rendered - seven deterministic geometry checks over the flattened shapes, weight 0.35 - a vision judge, blind caption plus a per-task feature checklist, weight 0.65 6 animals by 5 vehicles, 30 tasks. The judge's checklist, the anti-cheat terms and the expected wheel count all derive from the task rather than being hardcoded for the pelican. Also adds two examples: pelican_svg_eval.py scores any set of models through HF Inference Providers, and pelican_svg_grpo.py trains against the environment with TRL and is runnable on HF Jobs.
…mode, row ranking - _parse_checklist only counts a JSON true. bool() coerced strings like "false" or "no" from a judge that ignored the schema into approval. - sample_completions applies the chat template with the same enable_thinking the training dataset used, so the probe measures the mode the policy was trained under. - find_wheels ranks candidate rows by bottom edge, centre plus radius, which is what touches the ground. Ranking by centre let a pedal drawn below the axle displace a single expected wheel, and made the hub-versus-rim tie depend on rounding.
|
Second Bugbot pass addressed in 23e3a60, all three were real:
Two regression tests added ( |
The expected count outranked the bottom edge, so with a single expected wheel a lone pedal, which matches the count, beat two real wheels that reach the ground lower. A bicycle could pass as a unicycle. Where a row sits is a fact about the drawing and the expected count is only a preference, so the ground now votes first. Regression: two wheels plus a pedal scored as a unicycle finds two wheels and fails wheel_count.
|
One more from the external deep review, fixed in 477a1ab: the row ranking put the expected wheel count above the bottom edge, so with The key is reordered so the ground votes first: bottom edge, then expected count, then radius. Where a row sits is a fact about the drawing, the expected count is only our preference. Regression added: two wheels plus a pedal, scored as a unicycle, must find two wheels and fail |
A typo'd task_id arrives over the wire and used to surface as a bare KeyError inside an EXECUTION_ERROR envelope. The error now says which ids exist.
_frame_spans_wheels took the two leftmost wheel candidates as the hub span while wheels_apart already used the outermost pair, so a stray circle to the left of the bicycle shrank the span to a gap the frame never crosses. The span now runs between the outermost pair and only that pair is excluded from the coverage count, so a middle wheel still counts as bridging geometry.
8d591e2 to
0f15db5
Compare
|
Fixed in 0f15db5. Note on the branch: the previous head (8d591e2) accidentally swept in unrelated README formatting churn, which is what broke check-env-docs. That commit was replaced, the branch now only touches pelican files again. |
flatten_path kept one last_control for cubics and quadratics, so S reflected a quadratic control and T a cubic one. Per the SVG spec S may reflect only after C/S and T only after Q/T, otherwise the first control is the current point. The control point is now tracked per curve type.
|
Fixed in 52b19eb. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 52b19eb. Configure here.
extract_shapes walked every drawable element, so a bicycle hidden with display none, visibility hidden, opacity 0, an unpainted fill and stroke, or coordinates outside the viewBox still earned the full structural reward while a visible decoy satisfied the gate. The walk now skips hidden subtrees, tracks inherited paint, and drops shapes whose bounding box never intersects the canvas.
|
Fixed in 34e828a. |
Summary
Simon Willison's "generate an SVG of a pelican riding a bicycle" as an environment that can be evaluated and trained against. The model never sees its own output, so it has to hold the spatial arrangement in its head and emit coordinates for it. Ships with two examples: an eval script over HF Inference Providers and a GRPO training script for TRL on HF Jobs.
Type of Change
Alignment Checklist
Before submitting, verify:
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles.claude/docs/INVARIANTS.mdand no invariants are violated/pre-submit-pr(orbash .claude/hooks/lint.shand tests) and addressed all issuesRFC Status
Notes: rewards live entirely inside the environment (three layers: source gate, deterministic geometry, vision judge composed via the RFC 004 rubric tree), and the client never imports from
server/.Test Plan
PYTHONPATH=src:envs uv run pytest tests/envs/test_pelican_svg_env.py tests/envs/test_pelican_svg_training_contract.py -v— 132 tests, no network. The judge is exercised through a stub client, including the failure path where a broken judge must not inflate the reward.curl .../health, thenreset/stepover the WebSocket client.fixtures/holds 12 hand-written SVGs with verdicts fixed in advance (honest attempts, two deliberate cheats, degenerate cases) that run as regression tests for the gate.Claude Code Review
N/A
Note
Medium Risk
Large new surface area scoring untrusted model SVG (parsing, rasterization) and optional external vision API calls; reward logic is intricate but isolated to the new env package.
Overview
Adds
pelican_svg_env, a single-turn OpenEnv where the policy outputs SVG for “animal riding vehicle” tasks without seeing the render. Scoring runs in sequence: a source gate (cheats, malformed SVG, blank canvas), deterministic structure (wheels, frame, rider geometry; wheel count adapts per vehicle), then an optional HF vision judge (blind caption + per-task checklist). Rewards are composed via the RFC 004 rubric (Gate→ weighted structure/semantic).The env ships with a FastAPI server, Docker image,
openenv.yamlSpace config, client/models, 12 SVG fixtures, and docs (newpelican_svgpage plus catalog/toctree entries). Task catalog is 6×5 combinations withtask_idpinning; judge can be disabled or forced offline via env vars without renormalizing failed judge calls into higher rewards.Reviewed by Cursor Bugbot for commit 34e828a. Bugbot is set up for automated code reviews on this repo. Configure here.