docs(handoff): 2026-06-03 session handoff

docs/handoffs/HANDOFF-2026-06-03.md

@@ -0,0 +1,235 @@
+# HANDOFF — 2026-06-03
+
+> Pick-up doc for a fresh session. Read this first, then `forgeplan health`,
+> `git log --oneline -12`, and the memory note `project-executor-build-plan`.
+> Everything below is the state of `main` after a very long session that took the
+> public leaderboard from "2 cells" to a full, real, multi-chart board.
+
+---
+
+## 1. TL;DR — where we are
+
+POLLMEVALS evaluates **stacks = model × harness × scaffolding (L0–L8)**, not bare
+models. The public site (`apps/site`, Next.js static export) now shows a **real**
+leaderboard from a live executor + judge panel.
+
+- **board.json**: `21 cells, 19 scored, per_criterion on all 19`. Axes: **17 models ×
+  {raw-llm (L0), aider (L4)}**. One task scored so far: `be_01_jwt_auth`.
+- **Site charts** (all live, Playwright-verified, on `main`): transposed **compact
+  matrix** (models=rows, harness=cols), **Pareto** scatter (tier-colored + hover
+  tooltip), **RankedBars** (metric toggle + "All" 4-thin-bars + **click→right
+  drawer** with full per-criterion profile), **Radar** (per-criterion spider),
+  **ScaffoldingLift**, **master sortable/filterable table**, plus a dedicated
+  **/tasks** catalogue page.
+- **Executor (RFC-006)**: `raw-llm`→`InspectEvalCaller`, `aider`→`StackExecutorCaller`
+  (Docker sandbox bastion). `build_real_board.py` drives grids + emits board.json.
+- **BCB runner**: `PythonCorrectnessEvaluator` + `pollmevals-eval-py` image — imported
+  BigCodeBench tasks are **runnable** (10/10 gold packs pass e2e).
+- **Imports**: 30 third-party tasks catalogue-only (BCB/LCB/SWE), `ADR-014` admits
+  all 3 sources (G4 gate still required before *scored* use).
+
+This session's merges: **#45** imports, **#46** /tasks page, **#47** BCB runner +
+ADR-014, **#48** coding-ladder board, **#49** viz overhaul v2 + grok fill, **#50**
+radar + ranked + compact matrix + reasoning fill v3, **#51** stack drawer.
+
+---
+
+## 2. THE ACTIVE TASK — harness batch (user greenlit, NOT started)
+
+**User goal:** the matrix looks empty on the right because we've only run **2
+harnesses** (raw-llm, aider). The user chose **"build a batch of 2–3"**:
+**claude-code + codex + goose** → real new harness columns.
+
+This is a real Phase-5 build (per harness: Docker image + executor recipe +
+`stack.yaml` + a scored run). The branch `feat/harness-batch` was created but only
+*read* the aider pattern — **no code written yet**. (The empty branch can be reused
+or deleted.)
+
+### How to add ONE harness (the proven aider pattern)
+
+1. **Docker image** — `infra/docker/harness-<cli>/Dockerfile`, mirror
+   `infra/docker/harness-aider/Dockerfile`:
+   - slim base + install the CLI + `git` + `ca-certificates`;
+   - **non-root `useradd --uid 1000 harness`** (matches host bind owner so the
+     produced patch is writable back);
+   - git identity (`user.email/name`, `safe.directory /workspace`);
+   - disable the CLI's analytics / auto-update phone-home (no egress on the
+     internal net);
+   - `CMD` = a harmless version probe (launcher overrides `command`).
+   - Add a `make eval`-style target (mirror `harness-image-aider`) +/or
+     `make harness-image-<cli>`.
+
+2. **Recipe builder** — in `apps/eval-core-py/src/orchestrator/stack_executor.py`:
+   write `_<cli>_invocation(proxy_base_url, api_key, model_alias, prompt) ->
+   ProxyInvocation`, add it to `_PROVEN_RECIPES`, remove from `_PENDING_RECIPES`.
+   `ProxyInvocation(env, config_files, extra_args, prompt_args)`. The aider one:
+   ```python
+   env={"OPENAI_API_BASE": f"{base}/v1", "OPENAI_API_KEY": api_key},
+   config_files={}, extra_args=["--model", f"openai/{alias}"],
+   prompt_args=["--message", prompt]
+   ```
+   - **codex** (easiest, OpenAI-compatible): same env shape (`OPENAI_BASE_URL` /
+     `OPENAI_API_KEY` → proxy `/v1`); codex headless flag is `codex exec "<prompt>"`
+     (verify the non-interactive flag + `--model` for the alias). May need a
+     `config.toml` via `config_files` (provider = openai-compatible, base_url =
+     proxy). Mirror aider first; debug the headless invocation.
+   - **goose** (Block): configurable provider; point its provider base_url at the
+     proxy `/v1` via env/config; headless `goose run -t "<prompt>"` (verify).
+   - **claude-code** ⚠️ THE HARD ONE: it speaks the **Anthropic** API, but the
+     LiteLLM proxy is **OpenAI-compatible**. Options: (a) LiteLLM has an
+     `/anthropic` passthrough / `anthropic`-format support — point
+     `ANTHROPIC_BASE_URL` at the proxy and confirm it translates; (b) or add an
+     Anthropic-shaped route in `infra/litellm-config.yaml`. Do NOT assume
+     `ANTHROPIC_BASE_URL=proxy` just works. Smoke it in isolation first.
+
+3. **stack.yaml** — `stacks/<cli>/stack.yaml`, mirror `stacks/aider/stack.yaml`:
+   `agent_cli`, `layers` (L-flags — claude-code = L1+L2; codex/goose ≈ L2; the
+   board's row order is `max true L-index`), `execution.mode: repository_patch`,
+   `execution.command/args`, `input/output_contract`, `limits`, `sandbox`.
+
+4. **Run** — extend `apps/eval-core-py/scripts/build_real_board.py`: add the stack
+   to the aider-spec block (`_AIDER_MODELS` analog → make it per-stack model lists),
+   or add a third spec. Build the image, `make stack-up && make sandbox-net-up`,
+   then `build_real_board.py --confirm-spend`. The Board emitter (`board.py`
+   `_STACK_FAMILY`, `_LAYER_ORDER`) and the whole site **already** handle N
+   harnesses — new columns appear automatically.
+
+5. **Verify** the new column renders (Playwright) → PR → merge.
+
+**Smoke each CLI in isolation BEFORE wiring** (does `<cli> exec`/`run` against the
+proxy produce a patch?) — the recipe is the risky part, not the image.
+
+---
+
+## 3. HOW TO DEVELOP HERE (workflow — non-negotiable)
+
+- **Branch per slice.** Never commit to `main` directly (red-line). Fewer, larger
+  PRs — one per slice/wave, fold chore commits in.
+- **lefthook pre-commit** runs: `secret-scan`, `format-py`/`format-ts`,
+  `moon :typecheck`, `forgeplan-validate`. **Never `--no-verify`** (red-line).
+  typecheck only runs when `.tsx`/`.ts` are staged.
+- **Verify viz in a clean browser (Playwright)** before merge: navigate, `evaluate`
+  to assert structure, screenshot. Playwright has no extensions → it's the source
+  of truth for hydration.
+- **Site** = Next.js **static export**; `page.tsx` is a Server Component reading
+  `public/board.json` + `public/tasks.json` via `node:fs`. Charts that need state
+  are `"use client"`.
+- **forgeplan artifacts via MCP only** (never `Edit` the `.forgeplan/*.md`).
+- **Run python**: `uv run --project apps/eval-core-py python apps/eval-core-py/scripts/<x>.py`.
+- **Spawned agents** run in isolated worktrees + can orphan files — verify state
+  directly; do the sensitive writes (forgeplan, board) in the main thread.
+- **Spend**: real runs cost $ (judges dominate, ~$0.04/eval). The user has
+  standing authorization for autonomous runs but flag the rough cost.
+
+### Board run cheatsheet
+```
+make stack-up && make sandbox-net-up && make harness-image-aider   # prereqs
+uv run --project apps/eval-core-py python apps/eval-core-py/scripts/build_real_board.py --confirm-spend
+#   ... or targeted: --fill grok-4,glm-5     (re-run only these + merge into board.json)
+```
+- Incremental board write after the raw spec (so a slow aider/reasoning tail can't
+  lose the raw board). `max_concurrent=3` (judge eval_async is lock-guarded).
+- `--fill MODELS` re-runs only the named models on raw-llm and **merges** into the
+  existing board.json (no wasteful full re-run). Added this session.
+
+---
+
+## 4. HARD-WON GOTCHAS (do not re-learn these)
+
+- **Hydration:** NEVER `toLocaleString()` in a rendered component (server Node
+  locale ≠ browser) → use a deterministic grouping regex
+  `String(Math.round(v)).replace(/\B(?=(\d{3})+(?!\d))/g, " ")`. Browser extensions
+  (LanguageTool `data-lt-installed`) cause **false** hydration errors → already
+  handled with `suppressHydrationWarning` on `<html>`/`<body>`. A clean Playwright
+  render is the test.
+- **Build:** `eval-core-py:build` MUST be `uv sync --all-extras` (bare `uv sync`
+  prunes the dev extra → mypy can't find pydantic → 74 spurious errors that block
+  EVERY `.tsx` commit). Fixed in `apps/eval-core-py/moon.yml`. Same class: never a
+  bare `uv sync` anywhere.
+- **lefthook excludes** `evals/task-packs/**` from `format-py`/`format-ts` +
+  ruff `force-exclude` (root `pyproject.toml`) — imported `gold/` fixtures are
+  VERBATIM (ADR-007 "never edited in place"); upstream code legitimately trips
+  B006/B008.
+- **Reasoning models burn the whole `max_tokens` on reasoning_tokens** → empty
+  content (`finish_reason=length`, 0 chars) at 4–8k. Fix: **bump to 16k** — they
+  finished (qwen-3-32b/glm-4-7/glm-5/kimi-k2-5 all filled). `reasoning_effort:low`
+  does NOT tame them via OpenRouter. Per-model budget via
+  `InspectEvalCaller.max_tokens_for`; `_REASONING_HEAVY` set in build_real_board.
+- **Dead OpenRouter routes:** `x-ai/grok-4` → 404 (use `grok-4.20`);
+  `deepseek/deepseek-v3.5` → 400 (deprecated). Always check the live catalog:
+  `curl -s https://openrouter.ai/api/v1/models`.
+- **Cerebras routes 500 on full (≥4096-tok) requests** → route candidates via
+  OpenRouter instead (`qwen/qwen3-32b`, `z-ai/glm-4.7`).
+- **Judge layer crashes on empty candidate content** (`Krippendorff ValueError:
+  "There has to be more…"`, `grid_runner.py:438`) — caught → eval `FAILED`, run
+  survives. Robustness gap: should score 0, not crash. (Backlog.)
+- **Judge `eval_async` is process-global** → guarded by `_EVAL_ASYNC_LOCK` in
+  `judge_panel.py`. Safe to run evals at `max_concurrent>1`.
+- **Docker (macOS):** Desktop socket is `~/.docker/run/docker.sock`, not
+  `/var/run` → `DOCKER_HOST` auto-discovery (in `DockerHarnessLauncher` +
+  `bcb_e2e_smoke.py`). Bring up **only `make stack-up`** (NOT `obs-up`) — the full
+  observability stack + harness containers OOM-kill everything. In-sandbox the
+  proxy is `pollmevals-litellm-proxy:4000`, NOT localhost.
+- **Proxy restart:** after editing `infra/litellm-config.yaml`,
+  `docker restart pollmevals-litellm-proxy`, then poll `/health/readiness`. Right
+  after restart it returns **instant "Remote end closed connection"** until the
+  workers warm (≈30–60s) — that is NOT a route bug. A stale OTel callback error in
+  the logs (`otel-collector` unresolved) is harmless when obs is down.
+- **aider × {llama-3.3-70b, deepseek-v3.5}** time out — they don't follow aider's
+  edit format. Real harness×model-compat signal; keep aider to models that do
+  (qwen/coder/codestral/devstral).
+- **Squash-stacked PRs:** `--delete-branch` on a *base* PR closes its stacked
+  children. Merge parents first, reconcile child with `git merge origin/main`.
+
+---
+
+## 5. BACKLOG / OPEN (priority order-ish)
+
+1. **Harness batch** (THE active task) — codex → goose → claude-code (Anthropic
+   protocol caveat). Each adds a real matrix column.
+2. **Fill the last 2 empty cells**: `deepseek-v4-pro` (add to `_REASONING_HEAVY`,
+   `--fill`) and `deepseek-v3-5` (find a live route or drop). `grok-4` scored
+   **0.0** — investigate (degenerate output? judge-on-empty?).
+3. **Judge robustness** — score 0 on empty content instead of the Krippendorff
+   crash (methodology-correct).
+4. **More runners** — LCB (stdin/stdout harness) + SWE (repo-checkout executor);
+   BCB is done. Then the imported tasks can be scored (after the **G4 contamination
+   gate** per ADR-014 — a maintainer methodology call).
+5. **More tasks scored** — only `be_01` has data; per-task-category radar/leaders +
+   bump-chart unlock once ≥2 task types are scored.
+6. Viz nits: per-criterion columns could enrich the master table; the matrix is
+   honestly sparse until more harnesses land (the batch fixes this).
+
+---
+
+## 6. KEY FILES (map)
+
+- **Executor**: `apps/eval-core-py/src/orchestrator/stack_executor.py` (recipes,
+  launcher, sandbox), `stack_caller.py`, `stack_scoring.py`, `eval_caller.py`
+  (`InspectEvalCaller.max_tokens_for`), `grid_runner.py`, `judge_panel.py`.
+- **Board emitter**: `apps/eval-core-py/src/leaderboard/board.py` (`Cell`,
+  `per_criterion`, `build_board`). TS mirror: `apps/site/src/lib/board.ts`.
+- **Run scripts**: `apps/eval-core-py/scripts/build_real_board.py` (`--confirm-spend`,
+  `--fill`), `bcb_e2e_smoke.py`, `gen_tasks_json.py`, `ingest_{bigcodebench,livecodebench,swe_rebench}.py`.
+- **Evaluators**: `apps/eval-core-py/src/evaluators/python_correctness_evaluator.py`
+  (BCB), `correctness_evaluator.py` (vitest), `sandbox/runner.py`.
+- **Site charts**: `apps/site/src/components/{HarnessModelMatrix,StackParetoChart,
+  RankedBars,RadarChart,ScaffoldingLift,StackMasterTable,StackDrawer}.tsx`;
+  `app/page.tsx`, `app/tasks/page.tsx`, `app/globals.css`, `app/layout.tsx`.
+- **Infra**: `infra/litellm-config.yaml` (17 models + 3 judges), `infra/docker/
+  {harness-aider,eval-py,eval-ts}/Dockerfile`, `Makefile`, `lefthook.yml`,
+  `stacks/{raw-llm,aider}/stack.yaml`.
+- **Methodology**: `ADR-014` (admit 3 import sources), `ADR-007` (Tier-2 sourcing),
+  `evals/task-packs/IMPORTED-CATALOGUE.md`.
+- **Memory**: `project-executor-build-plan` (working snapshot — keep updated).
+
+---
+
+## 7. ONE-PARAGRAPH MISSION REMINDER
+
+Prove the thesis with numbers: *a cheap model with the right scaffolding beats an
+expensive one without it.* The board's unit is **harness × model**, scored per task
+on quality / cost / reliability with a no-self-judging median panel + Krippendorff α
+gate. Be honest with the numbers (show "—" for unscored, never fake a column), keep
+the methodology load-bearing (G4 before scored imports), and grow the board by
+running **more harnesses** and **more tasks** — that is the whole roadmap.