fix(benchmark): make the instrument honest, then cut direction's 38.7% cost share

.add/milestones/direction-velocity/MILESTONE.md

@@ -0,0 +1,109 @@
+# MILESTONE: Direction velocity — cut the pre-code phase, persist cross-task context
+
+goal: cut ADD's direction phase below 38.7% of cost and 31 minutes of wall-clock without losing requirement coverage, and make cross-task invariants inheritable through the task graph
+rationale: sub-milestone — measured, not suspected. The pay1–4 campaign (n=1, 8 runs, 2026-07-26) folded ADD's own transcripts into a call-stack flamegraph: direction is 38.7% of billed context (22.9M, ~$6.88) and 45.7% of wall-clock (31.0 of 67.8 min). Only 6.8 of those 31 minutes is reasoning; the rest is structure — 7.3m of strictly serial Reads, 4.9m building PLAN.md through 45 successive Edits, 3.9m of harness bookkeeping. Across 209 direction turns, ZERO emitted more than one tool call. The engine itself is 1.1%, so trimming verbs or output buys nothing.
+stage: mvp · status: active · created: 2026-07-27T03:27:28+00:00
+relations: relates-to: engine-output-trim, add-bench-v2
+
+> SDD living doc for this milestone. Keep it THIN: breadth, shared decisions, and
+> exit criteria only — per-task detail lives in each `.add/tasks/<slug>/PLAN.md`,
+> written just-in-time. Update this doc whenever a task reveals a milestone gap.
+
+## Scope
+In:  benchmark instrument honesty (the comparison arm + meter provenance) · direction-phase velocity (read batching, a one-shot bundle write) · cross-task context persistence (published invariants, graph inheritance, DESIGN.md at build) · edge-case rigor in §4 · an algorithm-dense workload to prove coverage
+Out: the frozen-contract semantics themselves · the security HARD-STOP · gate outcomes · `tests_weakened`/`assertions_lost` definitions · any relaxation of an existing refusal · n=3 campaigns (n=1 + mechanical proxies is the evidence tier this milestone buys)
+
+## Ground   (shared real-code context — gathered ONCE; every task's specify projects from this)
+Touches (shared files · symbols): `add-method/tooling/add.py` — `cmd_new_task` :888 (`depends_on` :912) · `cmd_freeze` :1178 (`unflagged_freeze` :1215) · `cmd_advance` `--fill` :7153 ("ONE section for ONE crossing") · `_FLAG_LABEL_RE` :5826 · `add-method/tooling/templates/PLAN.md.tmpl` §3/§4 · `benchmark/arms/*.toml` · `benchmark/score.py` `_add_tamper_metrics` · `benchmark/pilot.py` / `report.py` `--family` choices
+Anchors: the engine's four twins (`.add/tooling/add.py` · `add-method/tooling/add.py` · `add-method/.add/tooling/add.py` · `add-method/src/add_method/_bundled/tooling/add.py`) + `engine_pin.py` `ENGINE_MD5`/`ENGINE_PKG_MD5` · four template twins · three skill trees (`.claude/skills/add/` · `add-method/skill/add/` · `add-method/src/add_method/_bundled/skill/add/`)
+Honors (conventions): four-mirror-trees + engine-pin-3-mandatory-parts · additive-surface-byte-invisible · §5-scope-frozen-at-tests→build · dogfood-at-own-gate · close-gap-before-gate · never weaken a test or edit a frozen contract to pass a build
+Issues/Risks (shared): hand-mirrored twins have no parity test (the lock-reclaim class) · a conditional skip breaks `test_ci_tooling_mirror_gap`'s pinned count · `--fill`'s all-or-nothing restore is the model for `draft`, and a failed draft costs a full re-compose · engine running a test suite is a NEW responsibility and the hardest part of the all-or-nothing guarantee
+
+## Shared decisions & glossary deltas   (living — every task must honor these)
+- **Measure on mechanical proxies, not cost.** Direction turns ran 41/36/85/47 across four milestones — the within-arm spread exceeds the effect being chased, so n=1 cost deltas cannot resolve a 20% improvement. Every gate's PASS condition is a near-deterministic count (parallel turns, PLAN.md writes, direction turns); cost and wall-clock are recorded as evidence, never as the pass condition.
+- **One change measured per gate.** Landing several changes then measuring teaches nothing about which worked. Task order IS the experiment design.
+- **Instrument honesty precedes any claim.** The arm labelled `spec-kit` is `prompt_wrapper = "raw"` and produced zero `specs/` artifacts in every automated run; until that is fixed or renamed, no comparative claim is publishable.
+- **Persistence lands AFTER velocity.** Published invariants and edge rows ADD to direction. Landing them before the one-shot write would inflate the exact phase this milestone shrinks and destroy attribution.
+- **The engine never enforces what it cannot observe.** `add.py` cannot see editor writes, so a "≤N PLAN.md writes" rule is guide prose measured from the transcript — never a claimed engine guard. (`turn_ceiling` is the cautionary precedent: declared in every arm, asserted equal by `test_arms.py`, read by nothing.)
+
+## Shared / risky contracts (freeze these first)
+- `add.py draft <slug> --from <bundle> --run-red --freeze` — all-or-nothing bundle write -> owning task `direction-one-shot`
+- §3 `Invariants (published)` block + `invariant_without_proof` refusal -> owning task `invariants-publish`
+
+## Tasks (breadth-first decomposition; detail lives in each PLAN.md)
+- [x] arm-honesty        depends-on: none                — the comparison arm runs its own method or is renamed to what it is; no arm claims a method it never invokes
+- [x] meter-provenance   depends-on: none                — every scored record carries the meter version that produced it; close the last `isinstance(body, list)` oracle surface
+- [x] read-batching      depends-on: none                — direction issues independent reads/greps in ONE turn; harness bookkeeping (TaskCreate/sleep) leaves the benchmark path
+- [x] direction-one-shot depends-on: read-batching       — `add.py draft` writes §1+§3+§4 and freezes in one call, all-or-nothing; per-section `--fill` retired for direction
+- [x] invariants-publish depends-on: direction-one-shot  — §3 publishes invariants, each citing the test that proves it; freeze refuses one without a proving test
+- [x] invariant-inherit  depends-on: invariants-publish  — `new-task --depends-on` prints ancestors' invariants (a view, no new store)
+- [x] design-at-build    depends-on: invariants-publish  — DESIGN.md is written during build; a node that published invariants cannot gate without it
+- [x] edge-rigor         depends-on: invariants-publish  — §4 rows carry [GATED]/[edge]; the gate refuses an edge row that is neither green nor reasoned
+- [ ] algo-workload      depends-on: edge-rigor          — an algorithm-dense workload exists and has been run head-to-head
+
+## Exit criteria (observable; map each to the task that delivers it)
+- [x] A reader of any benchmark comparison can see what the comparison arm actually does, from the arm's own name and config        (← arm-honesty)
+- [x] A scored record states which meter version produced it, so a stale number is visible without re-reading git history            (← meter-provenance)
+- [~] A direction phase issues independent reads in a single turn — the parallel-turn count is greater than zero                     (← read-batching)
+      ← UNMEASURABLE AS WRITTEN. Census 2026-07-27: across 138 transcripts / 7560 assistant turns — every arm, every campaign ever recorded — EVERY turn carries exactly
+      one tool_use. The '0 of 209 direction turns parallelized' finding that motivated this task was an INSTRUMENT artifact, not a fact about ADD. Either the agent CLI emits
+      one tool_use per turn or the transcript writer splits them; either way this criterion cannot be satisfied by this harness. What Gate A DID show: Agent 8->0 and
+      SendMessage 7->0, matching BATCH_CLAUSE's skip-harness-bookkeeping half. Reword the criterion against a measurable proxy, or fix the transcript writer, before re-gating.   ← NOT YET MEASURED: the clause ships and is guarded, but the parallel-turn count needs a pay1-4 re-run (Gate A)
+- [x] A task's direction bundle is written by ONE engine call, and that call refuses to freeze unless the suite ran red              (← direction-one-shot)
+- [x] A task cannot freeze while publishing an invariant that no test proves                                                          (← invariants-publish)
+- [x] Creating a task with `--depends-on` shows the invariants it inherits, without copying them into a second store                 (← invariant-inherit)
+- [x] A node that published invariants cannot record a gate without its DESIGN.md                                                     (← design-at-build)
+- [x] A gate refuses an enumerated edge case that is neither green nor carries a stated reason                                        (← edge-rigor)
+- [ ] A workload whose requirements are genuinely algorithmic exists and has produced a head-to-head record                          (← algo-workload)   ← NOT STARTED: needs a paid head-to-head campaign
+
+## Strategy   (AI-drafted WITH the human — the optimized task plan; SOFT/advisory like a task's Build-strategy)
+- Approach (sequencing): **instrument-first, then cheapest-attributable-first.** arm-honesty and meter-provenance cost ~nothing and unblock every downstream claim. read-batching is a guide/wrapper change with no engine risk and no re-freeze — the highest payoff-to-risk item in the milestone, and it must be measured BEFORE direction-one-shot or the two effects are inseparable. The persistence wave lands last because it pushes direction back up.
+- Freeze-first: `add.py draft`'s signature (direction-one-shot) — invariants-publish, design-at-build and edge-rigor all write through the bundle it owns.
+- Waves (parallel): W1 = arm-honesty · meter-provenance · read-batching (independent, no shared files). W2 = direction-one-shot alone (measured in isolation). W3 = invariant-inherit · design-at-build · edge-rigor behind invariants-publish's frozen §3. W4 = algo-workload.
+- Tradeoffs weighed: (a) landing the DESIGN.md persistence first — rejected: it inflates direction and destroys the attribution the whole milestone rests on. (b) One combined "direction rewrite" task — rejected: batching and the one-shot verb have different risk profiles and one is free, so bundling them would spend engine risk to buy a wrapper-level win. (c) Gating on total cost — rejected: measured within-arm variance (41/36/85/47 direction turns) exceeds the effect. (d) n=3 campaigns — deferred: ~$65 and still under-powered; mechanical proxies resolve the same questions at n=1.
+
+## Close — ship review   (AI fills when every task is done)
+
+### Ship by domain   (what changed, per bounded context)
+- tooling : <add.py / state.json / templates — what shipped, or "untouched">
+- skill   : <SKILL.md / phases/* / guides — what shipped, or "untouched">
+- book    : <docs/* — what shipped, or "untouched">
+- benchmark : <arms / score / workload — what shipped, or "untouched">
+
+### Cross-task evidence   (one row per task)
+- <slug> : gate=<PASS|RISK-ACCEPTED> · tests=<n green> · residue=<none|note>
+
+### Goal met?
+- [ ] each Exit criterion above is satisfied by a Cross-task evidence row or a Ship-by-domain change (cite which)
+- goal: <restate the milestone goal — and the one evidence line that proves the ship meets it>
+
+## Release steps   (AI-DEFINED — the ordered steps to ship this milestone; engine records, human gate)
+- [ ] open a PR carrying the already-green meter work (syntax-agnostic `tests_weakened` · the `pay` track · the family-choice fix) — 3 commits on `fix/tamper-syntax-agnostic`, suite 493 green
+- [x] land W1, re-run pay1–4, record Gate A against the 2026-07-26 baseline
+
+### Gate A record — 2026-07-27 (`add` arm, pay1–4, n=1, runs-pay-gateA-2026-07-27, engine pin 4b6844ae)
+
+| metric | baseline 2026-07-26 | Gate A | |
+|---|---|---|---|
+| cost, 4 WMs | $22.15 | **$9.94** | −55% |
+| assistant turns | 326 | **163** | −50% |
+| requirement coverage | 17/18 | **18/18** | `R-payout-timeout-bounded` now covered |
+| oracle pass rate | — | 0.9 / 1.0 / 1.0 / 1.0 | 0 regressions |
+| Read / Edit calls | 64 / 87 | 26 / 44 | |
+| Agent / SendMessage | 8 / 7 | **0 / 0** | matches BATCH_CLAUSE's skip-bookkeeping half |
+| `tests_weakened` · `assertions_lost` | — | 0.0 · 0.0 | |
+
+**PASS on the cost and coverage evidence. NOT a causal result.** Three honest limits:
+- **`draft` was called ZERO times.** The `add-loop` wrapper names `add.py freeze` and does not
+  know the verb exists, so `direction-one-shot` — the change this milestone rests on — is still
+  UNMEASURED. Fix the wrapper before Gate B, or Gate B measures nothing either.
+- **The parallel-turn criterion is unmeasurable** (see the exit-criteria note above).
+- **Direction's share of turns is NOT reported.** The fold's seam detector flips at the first
+  `freeze` and never resets, and the two campaigns used different verbs (9 freezes vs 4, plus 3
+  `advance`), so the 18.7%→54.6% reading is a detector artifact, not a measurement.
+- n=1 against measured within-arm variance (41/36/85/47 direction turns across four milestones)
+  that already exceeds this effect. The engine also gained four gate floors in the same window.
+  Cost and coverage are real; the attribution is not earned.
+- [ ] land W2, re-run pay1–4, record Gate B
+- [ ] land W3 + W4, re-run both tracks, record Gate C — coverage ≥ 0.982 must not regress
+- [ ] the human reviews the three gate records and cuts the release

.add/specs/method.md

@@ -26,6 +26,17 @@ project: AIDD / ADD Methodology · seeded: 2026-07-17 · stage: mvp
 - The skill lean fence is a hard floor: genuinely-new doc-truth on a guide is reclaimed from the same guide's prose, not a budget rebaseline, absent an explicit human bump. [fv58 · component-worked-example]
 
 ## Deltas (newest first)
+- [open · 2026-07-27] the parallel-turn metric is UNMEASURABLE in this harness — across 138 transcripts and 7560 assistant turns, every arm and every campaign ever recorded, EVERY turn carries exactly one tool_use; the '0 of 209 direction turns parallelized' finding that motivated read-batching was an instrument artifact, not a fact about ADD (evidence: benchmark/**/transcript.jsonl census, 2026-07-27) (task:edge-rigor)
+- [open · 2026-07-27] the benchmark's add-loop wrapper names `add.py freeze` and never `add.py draft`, so the newest direction verb was called ZERO times in the Gate A campaign — the shipped skill teaches it, the headless/portable path does not; this is arm-honesty pointed inward and it means direction-one-shot's effect is still unmeasured (evidence: runs-pay-gateA-2026-07-27, 0 draft calls across 4 WMs) (task:edge-rigor)
+- [open · 2026-07-27] a new gate floor should be dogfooded against the task that ships it — edge-rigor's own §4 carries three [edge] rows, so its gate exercised the guard on real input at the moment of shipping, the same self-referential check the signal-graph milestone used (evidence: edge-rigor gated PASS through its own edge_unaccounted floor) (task:edge-rigor)
+- [open · 2026-07-27] declaring `Tests live in: <the engine dir>` makes the WHOLE directory a protected test surface, so every legitimate build edit to add.py/engine_pin.py trips tamper_detected — three of this milestone's engine tasks each paid a re-cross for it; a task whose tests sit beside its build target should expect the re-cross, or the §4 path should name the test FILES (evidence: invariant-inherit, invariants-publish, direction-one-shot all hit build_tampered on add.py) (task:invariant-inherit)
+- [open · 2026-07-27] a bare `for` inserted between an `if` body and its `else:` binds the else to the FOR — for/else fires on every loop that completes without break, so new-task printed BOTH 'linked to milestone' and the orphan nudge; valid Python, silently wrong, caught only by test_v8_1_orphan_guard (evidence: cmd_new_task inherited-invariants view; the loop now sits after the whole if/else with a comment saying why) (task:invariant-inherit)
+- [open · 2026-07-27] a new freeze floor must ship with its GRANDFATHER test — absence must never become a refusal, or every task already on disk breaks; boundary_unfilled's if-present-then-validate shape is the pattern to copy (evidence: test_no_invariants_block_is_grandfathered, which caught that 4 sibling tests were passing on contract_not_drafted rather than on the floor under test) (task:invariants-publish)
+- [open · 2026-07-27] a fixture assertion must be anchored the way the parser anchors it — the pristine PLAN.md template carries both 'FROZEN @ vN' and 'Status: FROZEN' in explanatory prose, so a substring check is satisfied by a correctly-RESTORED file; _contract_frozen anchors ^Status: at line-start and the test must too (evidence: test_run_red_refuses_a_green_suite) (task:direction-one-shot)
+- [open · 2026-07-27] an engine verb that shells out is a different kind of verb — every other add.py command touches only files it owns, so --run-red's subprocess needs an argv list (never shell=True), a bounded timeout, and NO retry (a suite run is not idempotent-cheap; a retried timeout doubles whatever side effect hung it) (evidence: _run_red_suite in add.py; test_red_run_is_bounded_by_a_timeout runs 30s under a 2s limit inside a 60s harness ceiling) (task:direction-one-shot)
+- [open · 2026-07-27] direction batched its WRITES ('ONE silent draft') years before it batched its READS — the guide mandated turn-shape on one side of the phase and said nothing about the other, costing 7.3 of 31 direction minutes to a serial Read chain (evidence: pay1-4 flamegraph fold 2026-07-26, 209 direction turns / 0 with >1 tool call; closed by the 'Batch the sweep' clause in phases/direction.md) (task:read-batching)
+- [open · 2026-07-27] a scored record must carry a content-derived stamp of the code that scored it; a hand-bumped version constant is the turn_ceiling failure mode — declared everywhere, true nowhere (evidence: benchmark/meter.py + test_meter_provenance.py; three archived campaign meters were previously distinguishable only by reading git history) (task:meter-provenance)
+- [open · 2026-07-27] an arm/config that INSTALLS a method's scaffolding must be guarded to INVOKE it — spec-kit and gsd both ran raw for months under 493 green tests because every guard checked pins and fairness fields, never whether an arm does what its name claims (evidence: zero specs/ artifacts in every automated campaign) (task:arm-honesty)
 - [open · 2026-07-26] A frozen contract can encode a false assumption about the surrounding system, and the build is where you find out: mine said the interrupt watcher polls a streaming transcript, but the runner drained stdout with communicate() and wrote the transcript only after exit. The honest move is to make the assumption TRUE where that is defensible (stream on the new path only, so the old path stays identical by construction) rather than silently redesign around a frozen clause. (task:bench-interrupt-mechanics)
 - [open · 2026-07-26] Before inventing a mechanism for a defect, grep for how the repo already cured that class: the benchmark had fixed editable-install leakage in running_app with -E -s -S months earlier, and the CLI probe was simply the one spawn path that never got the flags. I shipped a NEW provenance mechanism for an already-solved class. The cheaper and more consistent fix is to find the existing cure and apply it where it was missed — and a drift test between the two call sites is what keeps them from separating again. (task:bench-ambiguity-scoring)
 - [open · 2026-07-25] Never reach for a hard reset to move a commit off the wrong branch: it discards the uncommitted working tree too. Branching then hard-resetting main destroyed four in-flight source edits AND the engine state file, while untracked files survived — so the damage was ragged and easy to under-read. The safe move is branch, then reset --keep or reset --soft; and when the tree is dirty, verify what is uncommitted BEFORE any reset. (task:risk-proportional-skip)