Repository navigation
experiment(certify): P-045 Rust driver bridge — slices 0–1 and the validator half of slice 2 - #2737
Closed
colinmegill wants to merge 29 commits into
Closed
colinmegill wants to merge 29 commits into
colinmegill wants to merge 29 commits into
Conversation
…ngine/1 (P-026) An EXPERIMENT on a branch, not a merge candidate: a Rust coordinator that owns the poll/lease/publish loop and drives the Python science worker through the polis-engine/1 contract via a small Python adapter, with a recovery harness. Judged by the same gate as the Python poller. Reviewed (Opus, cost-reduction/04-plans/P-026-rust-coordinator-review.md): rebuilt from a clean toolchain; 10 Rust tests; clippy -D warnings on both feature sets; release+fault builds refused by build.rs; 78 pytest; three replay checkpoints match the reference writer across all published tables at zero delta; polarity pairs match with negative controls differing. Contract obligations: 5 of 7 D5 bullets pass; CO07 24/25 fault stages and the real-Node D4 read remain open and are declared. One defect to fix before a real coordinator: Fenced and lease-unavailable collapse into a single OwnershipRefused exit, so a restart inside the lease window after an unclean death exits without repairing. Eight proposed contract edits accepted, four with amendments (recorded in the review). Excluded: target/ and artifacts/. Cargo.lock committed on purpose. Authored headless by Astra (Codex) under Claude orchestration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…-026 review) Add `rust-toolchain.toml` pinning stable 1.98.1 with clippy and rustfmt so every build of this crate uses one known compiler, and declare `rust-version = "1.88"` in `Cargo.toml`. 1.88 is the true minimum: edition 2024 needs 1.85 and the highest MSRV among the locked dependencies is also 1.85, but `src/main.rs` uses a let-chain, stabilized in 1.88 for edition 2024. Documented in a new README "Toolchain" section, including that cargo must be run from `coordinator-rs/` for the pin to apply. `.gitignore` already excludes `/target/` and `/artifacts/`, and nothing under either is tracked; no change needed there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…e lease The review's one liveness defect: `Publication::Fenced` and "lease unavailable" were the same `OwnershipRefused` error, so a restart inside a dead owner's lease window crash-looped on exit 3, and a compute slower than the lease fenced itself and could never publish. Rev5's CO03 clause asks for three distinguishable states, and this implements them as `LeaseState`: * `Fenced` (exit 3) — owner or epoch superseded. Terminal: publication authority is gone and the daemon stops rather than retrying this zid. * `LeaseUnavailable` (exit 4) — another owner holds an unexpired lease. The daemon defers that conversation with the existing durable bounded backoff and keeps sweeping; it never dies and never starves its other zids. * `LeaseExpired` (exit 5) — our own lease elapsed in database time. No publication follows; a later pass reacquires and recomputes. Codes 4 and 5 are reachable only from the strict one-shot `once` and `publish-fixture` commands; `run` exits only on `FENCED`. The README carries the table. Renewal during compute (`src/lease.rs`): a heartbeat on its own connection extends `expires_at` every lease/3, conditional on `(owner_id, owner_epoch)` and on the lease still being unexpired in DB time, so it can never resurrect an expired or transferred epoch. A definite renewal failure aborts the worker operation in flight — `Worker::call` now polls rather than blocking for the full timeout — and nothing is published. An uncertain renewal is not ownership proof: after one full lease with no confirmed renewal the work stops. Ownership is then classified once more against the lease row before publication, so a heartbeat the database contradicts cannot invent a loss, and `publish` still re-checks owner, epoch and expiry inside its own transaction under `FOR UPDATE`. A process that fails cleanly now releases its own epoch (conditionally, so a transferred row is untouched) instead of holding the conversation for the rest of the window; only an unclean death leaves a live lease behind. R07 keeps its real duplicate processes: the live-owner loser is LEASE-UNAVAILABLE, and only the stale owner after a genuine transfer is FENCED. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… stage R05 no longer hands itself the answer. The 16-stage kill/restart matrix models a real unclean death: SIGKILL at the stage, then a restart while the dead owner's lease row is still there unreleased, with no `expire(db)` anywhere. The restart must stay alive, defer the conversation, and repair by itself once the lease elapses in database time; the artifact records `manual_lease_expiry: false` and the epoch it took over with. `test_r05_restart_inside_a_live_lease_defers_then_ repairs` pins the defect itself deterministically: inside an 8s window the restarted process is asserted alive and publishing nothing, then repairs, logs LEASE-UNAVAILABLE and never FENCED, and takes over at epoch 2. The explicit DB-time lease-loss fixture is kept as its own control (`test_r05_expired_lease_restart_repairs`), and the restore stages lost their `expire()` the same way. The tests that fail cleanly now assert the process released its own lease instead of expiring it by hand. CO07's 25th stage is no longer unreachable. The coordinator gets a bounded warm *bundle* cache (`P026_CACHE_CAP`, default 16, 0 disables) holding the last coherent published bundle of an unchanged conversation, so a quiet pass does not re-read the three results tables. It is not a warm worker cache: a changed source is still rebuilt from the full authoritative prefix. Two properties make the R06 bug class unrepresentable rather than merely untested — lookup and LRU touch are one `&mut self` operation, so an eviction cannot interleave between them (the Python poller's `KeyError`), and an entry is usable only while its `math_tick` is still the conversation's current generation, so any other writer's update is a miss and not a lost update. Unit tests cover both. `test_r06_cache_eviction_contends_with_same_zid_update` stages the race the way the Python matrix does: cap=1, the barrier at the eviction boundary, the ack asserted to name the evicted zid so an unstaged race fails rather than passes quietly, a vote committed to that same zid while the eviction is blocked, and then no lost input — the evicted conversation is republished coherently, the independent fold is exact, and the warm result equals a cold-rebuild reference computed with the cache disabled in another namespace. `audit_stages.py` now separates `stage_inventory_gate` (25/25) from `full_contract_gate`, which stays FAIL with its open conditions named: CO08/D4 still need the actual Node Bundle route, and CO01's metrics are still absent. Reaching every fault stage is not a certificate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ract term `coordinator.rs` ordered its source extraction by `tid,pid,created,(vote*$2),weight_x_32767 NULLS FIRST` with `$2 = storage_agree_value` written straight into the query — the one place the coordinator interprets vote polarity, and the review's point was that it must be a declared normalization rather than an implementation accident, or the next implementer uses the raw sign and silently fails the polarity property. `src/ordering.rs` now owns the declaration (`polis-order/1`): the ordered terms `(tid,pid,created_ms,semantic_vote,weight_x_32767 NULLS FIRST)`, their NULL placement, the named parameter that carries the agree convention, and `semantic_vote = raw_vote * storage_agree_value`. The coordinator builds its ORDER BY from those terms instead of a literal, and the same declaration — with an `algorithm_digest` pinning the ordering algorithm — is the worker manifest's `ordering` value and is recorded in the published checkpoint. `engine_adapter` rejects a live declaration that does not match its own `storage_agree_value` or does not declare the semantic term, while a frozen replay may still name its pinned order as a string; Rev5 keeps those two profiles distinct. The declaration is part of the source fingerprint, so changing a term or the polarity constant discards warm state. Rev5 also asks for the `equal_time_census` this prototype did not record. `ordering::census` counts the groups of rows sharing `(tid,pid,created_ms)` whose order is decided only by the trailing tie terms, and ships the count with the declaration. `test_semantic_tie_key_is_a_declared_contract_term` is the discriminating case: two vote rows share `(tid,pid,created)` and differ only in the raw sign, so the published cell is decided by the tie term. The mirrored conversation (every raw vote negated, the convention flipped) publishes an identical generation, the census reports the one tied group, and the two conventions do not share an ordering digest. A raw-sign tie key would reverse that pair and break the mirror. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The live-lease R05 test asserted the takeover epoch was exactly 2; the `run` daemon keeps cycling after it repairs, so it can legitimately be 3 by the time the row is read. Only monotonicity is the property, so assert that. Evidence for the full re-run: 83 Python tests pass, 14 Rust tests, clippy clean under both feature sets, stage inventory 25/25 with the full contract gate still FAIL and its open conditions named. The three-checkpoint replay comparison keeps 0/0/0 deltas with the Rust and reference hashes equal and byte-identical to the previous evidence, and every polarity artifact is byte-identical too — the ownership, cache and ordering work changed no published payload. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…rumented Step 2 of the P-026 review's next steps. Every pass still visits every admitted conversation independently of timestamps; what changed is what a quiet visit costs, and that the pass now says out loud how it is doing. Incremental discovery. Before the full vote/comment/participant read, src/probe.rs takes one aggregate statement and compares it with the probe recorded in the new coordinator_reconciliation row — the per-zid cursor holding when the authoritative snapshot was last taken and the probe observed immediately *before* it, so a row committing between probe and snapshot changes the next probe rather than being swallowed. Rev5 is explicit that count/max are hints, so the probe can only skip a read: a differing probe always forces the full snapshot, a matching probe skips one only while that conversation's last reconciliation is younger than P026_RECONCILE_SECONDS, and P026_INCREMENTAL=0 restores CO05's unconditional sweep. The probe carries ordering::algorithm_digest, so the declared normalization and its agree-convention constant are part of the change token. test_the_aggregate_probe_is_weak_but_the_reconciliation_ceiling_repairs_it stages a change that leaves every aggregate identical, asserts that the fast path really does miss it, and then that the bounded reconciliation publishes it anyway. The weakness is in the suite, not only in the prose. Metrics. src/metrics.rs emits CloudWatch Embedded Metric Format records in namespace Polis/Math with P-031's two fixed dimensions and no per-conversation, per-run or per-instance dimension. No AWS client and no AWS dependency: a Sink takes finished records, with a JSON-lines sink and a null sink here. The default sink is off, because a run whose stderr is an undrained pipe blocks once the pipe fills and a coordinator must not stall on its own telemetry. Named by the contract and previously absent: OldestReconciliationAgeSeconds (CO01 scan age, and what makes the fast path auditable), ReconciliationBacklogConversations, FailureBacklogConversations, OldestUnrepairedAgeSeconds (CO06), plus per-pass counts, per-zid latency, and the typed lease and publication outcomes. `polis-coordinator metrics` prints the catalog, generated from the emitting code and checked in as evidence; two Rust tests keep the two in step. PollHealthy says the pass completed and nothing more: stuck work is the failure backlog and unrepaired age, as P-031 splits A01 from the lag signals. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…publication deadline Astra's step-1 review at 910ba8a (board [256]): F1, F2, and the two Rev6 contract amendments that came with them. F1 — a resident bundle is no longer evidence about the durable store. The review deleted a math_bidtopid row under a warm cache and watched six further passes leave it missing: the cached bundle satisfied the cache-hit check, the source fingerprint kept agreeing, and CO06's promise of repair without future input was quietly broken. PgStore::resident_is_intact now re-verifies, on every cache hit, that math_ticks exists at the cached generation with a publisher epoch, that all three companions exist at that same tick, that caching_tick matches, and that the committed input_checkpoint is byte-identical to the cached one. It selects metadata and that small JSONB only, so no payload is detoasted, and any disagreement evicts the entry and falls through to the ordinary rebuild. The review's reproduction is committed with its assertion inverted: delete the companion under a warm cache and the running daemon repairs it, with no new input and no restart, and the generation advances rather than being reinstated. Its companion test checks the fix did not simply disable the cache. F2 — publish-fixture propagates its typed refusal. It printed Refused(Expired) and exited 0, contradicting the exit-5 contract that `once` honours on the same seam. It now matches the result after cleanup and returns the same LeaseState error. Conflict deliberately keeps its distinct handling: a stale expected tick overwrote nothing and is a normal outcome, not a refusal. Rev6 CO04, remaining-lease final authorization under lock. The check immediately before COMMIT re-reads owner, epoch and the *remaining* interval from the still FOR UPDATE-locked lease row and refuses below P026_COMMIT_MARGIN_SECONDS, rather than accepting any expires_at in the future and gambling that the COMMIT round trip beats the clock. Its limit is stated rather than hidden: this does not prove the round trip finishes in time, so an uncertain COMMIT is still resolved by checkpoint identity and never by a wall-clock deadline. A test drives a refusal while the lease is still unexpired, so the margin is what is under test. Both of the review's ownership positives — renewal across a pending worker call, and a lease transfer aborting one — are kept as they were written. Documentation nits from the review: the README no longer calls this an uncommitted prototype, the absent-cache and no-renewal sentences are gone, and the "only an unclean death leaves a live lease" claim now names the setup paths that also leave the lease to genuine DB-time expiry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…h the real Node server Step 3, and D5's remaining reader bullet: "a coherent-Bundle read served through the real Node route for at least the getPca + getBidIndexToPidMapping join". Method, stated plainly, because it is not the compose stack and it is not HTTP. coordinator-rs/tools/node_reader.cjs loads the actual server modules in one Node process against the same test PostgreSQL — server/src/utils/pca.ts (getPca, its LruCache keyed by [math_env, zid] since #2703, and updatePcaCache's asJSON and gzip buffer) and server/src/utils/participants.ts (getBidIndexToPidMapping, getPidsForGid) — over ts-node with no mocks and no reimplementation, reassigning Config.mathEnv between namespaces exactly as the existing math-env isolation integration test does. One process reads both namespaces, so the [math_env, zid] cache scoping is exercised rather than assumed. There is no loadBundle in the server. #2703 scoped the existing reader; it did not introduce CO04's Bundle. So this certifies the consumer that exists today, and CO04's Node cache-unit rewrite stays open. The Rust coordinator publishes to rustproto and the pinned #2704 Python poller to python, at the same visibility checkpoints in one database. The real reader then serves both, and there are zero differences: asJSON (what participationInit embeds), asBufferOfGzippedJson (the literal pca2 body), the _.pick projection the ?keys= branch serves, the ETag math_tick, getBidIndexToPidMapping, the getPidsForGid join per group, and tids/n/repness/consensus. Two findings the real reader produced, pinned as tests rather than left as prose: A committed generation of 0 is not served at all. Rev5 permits first generation zero and #2704/Clojure start there, but getPca guards the column override with a falsy 0 and then drops the row on math_tick <= (math_tick || 0), so pca2 answers 304 for a conversation whose only generation is 0. That is characterised, not endorsed, and it is why the equality checks use a second generation. C7, measured side by side. The pinned Python poller publishes nothing at all for a conversation with approved comments and no votes, so there is no two-writer equality in that shape; what the reader does instead is serve the coordinator's published empty generation for one namespace and synthesize createEmptyPcaStructure for the other. The published blob lists no tids while the server lists the approved comments from `comments` — exactly C7's ruling, with the full difference list left in evidence for polis-empty-served/1 to decide. audit_stages.py now names what step 3 did not close: the loadBundle rewrite, HTTP routes and ETag handling, the private 2,884-case corpus, generation zero, and the C7 difference. full_contract_gate remains FAIL and it still exits 1. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ction The Delphi pytest job copies delphi/tests into the delphi image at /app/tests, where there is no coordinator-rs/ and no repository checkout, so this conftest's `git show` of the pinned reference fold and R09 mapping assets raised FileNotFoundError while pytest was still importing conftests and the whole directory errored at collection. Same shape as #2702 and #2720: locate the checkout by walking up for coordinator-rs/Cargo.toml, override with POLIS_CHECKOUT_DIR, and when it cannot be found — or when neither cargo nor an already-built fault binary exists, or git/.git is unavailable — set collect_ignore_glob and print the reason at collection finish. An explicit POLIS_CHECKOUT_DIR that does not resolve raises pytest.UsageError from pytest_configure: a hard failure rather than a silent skip, and still not an exception during import or collection. Nothing is weakened when the checkout is present, because on the skip path no test module is imported at all. A missing binary with cargo present stays a loud failure, as before. Reproduced against a copy of delphi/tests with no checkout above it: before, ERROR collecting; after, "coordinator tests not collected: ...", no tests ran, exit 0. With POLIS_CHECKOUT_DIR set the same copied tree collects 98 tests, identical to collection inside the checkout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…producible Following the C7 comparison in the previous commit: two runs of the same D4 empty-shape test produced the same hash for the coordinator's published empty generation and a different hash each time for the namespace the server synthesizes. pca.ts's createEmptyPcaStructure stamps `lastVoteTimestamp: Date.now()`, so the legacy empty presentation differs between two requests and between two servers answering the same request for the same conversation, and its ETag cannot mean what an ETag means. The harness now reports lastVoteTimestamp per namespace and the test asserts it: the synthesized value is wall-clock-now, the published one is the engine's. This is recorded for polis-empty-served/1 alongside the tids difference, not fixed here — the empty serializer is separately owned. Evidence regenerated for the full re-run: 98 Python tests pass, 25/25 stages reached, full_contract_gate still FAIL. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ests/conftest The Delphi CI job on #2727 showed collection ImportErrors in four modules that have nothing to do with this work — test_dataset_selection, test_discrepancy_fixes, test_legacy_clojure_regression, test_legacy_repness_comparison, test_pipeline_integrity — all of the form `cannot import name '_get_requested_datasets' from 'conftest' (/app/tests/coordinator/conftest.py)`. That is this branch's doing. Under pytest's prepend import mode and with no package markers anywhere under delphi/tests, every file is imported as a top-level module named by its basename, so `tests/coordinator/conftest.py` and `tests/conftest.py` both want the module name `conftest`. `coordinator` sorts before `test_*`, so once the coordinator conftest imported cleanly — which is exactly what the previous commit fixed — it claimed the name and the sibling modules that do `from conftest import ...` got the wrong module. Before that fix the coordinator conftest raised while importing, so it never registered and the collision never showed. Adding __init__.py makes these modules `coordinator.*`, leaves `conftest` to the parent, and the imports here now say which conftest they mean. No test logic changes. Reproduced in the CI layout (delphi/tests copied to a shallow app/tests, no checkout above it, PYTHONPATH at the delphi package): 16 collection errors and 1048 tests collected before; 13 errors and 1075 collected after, byte-identical to the same tree with tests/coordinator deleted entirely. The 13 that remain are pre-existing and environment-related, not from this branch. Inside a checkout the coordinator suite still collects 103, and so does the copied tree with POLIS_CHECKOUT_DIR set. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…gauge scope, P-031 labels Astra's round-2 review at ef0679d (board [276]) and contract Rev7. R2-F1, persisted payload corruption. The metadata check closed the deleted-row case but cannot see a mutated payload, and a resident bundle was still trusted as the prior generation when the reconciliation ceiling expired, so corruption in any of the three data columns survived indefinitely while a cold read rejected it. The authoritative path now always re-reads and re-hashes the persisted generation with load_current and never uses the resident bundle as evidence, so corruption is repaired within one ceiling interval. Rev7 is explicit that the cheap metadata check is not complete integrity reconciliation, and the code and README now say so rather than implying otherwise. Astra's three witnesses, one per table, are committed with their assertions inverted; the cold-read control is retained. Quiet repair is a precondition of skipping, not a casualty of it: the fast path requires a metadata-complete persisted generation, so an absent companion or missing checkpoint provenance is still repaired without future input and without waiting for the ceiling. The R05/R09/R11 and missing-checkpoint tests caught the first attempt, which had lost that. R2-F2, source age. reconciled_at was clock_timestamp() at the end of the pass, so it was completion age: a compute outlasting the ceiling reset the advertised source age and bought another fast-path interval. The probe now takes database time in its own statement, before the source read, and that is what is stored. The witness asserts both the stored timestamp and the refusal to credit a stale snapshot. R2-F3, generation zero. The report's conclusion was drawn from the wrong caller. The route passes math_tick = -1 and the bigint arrives as the string "0", so the real pca2 serves a committed generation of zero with 200 and ETag "0", and a matching conditional gets 304; only the cold getPca(zid, undefined) helper misses it, and it stops missing it once the route warms the cache. The withdrawn characterisation test is replaced by Astra's actual-route probe, committed as tools/node_route_probe.cjs and driven over loopback HTTP against the real route, reader, cache and PostgreSQL. The evidence file and the audit's open conditions now describe the helper asymmetry as a server-side reader defect, which a separate investigator owns; no server code is changed here. R2-F4, scope and labels. Failure gauges were counted across the whole math environment while the source-age gauges respected shard and allowlist; they are now scoped identically, with a witness both ways. And no catalog row claims a P-031 alarm any more: PollHealthy is renamed SourcePassHealthy because page-loop liveness is not A01, the A02 labels are removed because neither age gauge is PublishLagSeconds, the catalog carries a p031_status block naming A01/A02/A03 as not implemented, and a Rust test plus audit_stages enforce it. Rev7 also asks for a distinguishable foreign-namespace control, since two namespaces agreeing proves nothing on its own: the D4 test now publishes the same conversation under the opposite agree convention and requires the reader to serve it differently, and requires a namespace with no rows to fail the mapping lookup. The bidToPid mapping is legitimately identical under the mirror, and that is recorded rather than asserted away. Full gate: cargo test 19 per feature set, both clippy clean, release+fault refused, 103 Python tests pass, 25/25 stages, replay 0/0/0 with equal hashes, 230 observer observations and no errors, all polarity pairs equal with negative controls differing, D4 zero differences. full_contract_gate stays FAIL with eight named open conditions. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…its ruling Astra's round-3 review (board [291]) accepted the work with two wording nits. The reconciliation ceiling is an eligibility threshold, not a deadline. The README's earlier "repaired within one ceiling interval" sentence, the matching comment in coordinator.rs and the audit's open conditions now say that a conversation becomes *eligible* for reconciliation when the ceiling elapses, and that page traversal, backoff, lease waits and compute sit between eligibility and the repair. The later, correct qualification was already in the README; this makes the earlier sentences agree with it instead of promising a measured bound. And C7 is not an open question. G already carries the comment and clock preservation ruling for the empty presentation, so the audit's open condition and the README point at that obligation rather than asking for a decision that has been taken. What this branch contributes there is measurement against the ruling, not a request to reopen it. Documentation only; no behaviour changes. `cargo test --locked`: 19 passed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…l bytes, epoch-bound readback Slice S1 of the step-4 plan: a distinct candidate schema id with strict five-key adapter admission (id, engine version, input digest, …; the reserved polis-input/1 id is not reused); the original candidate bytes and their digests are written in the same statements as the JSONB and verified on read; publication and the uncertain-COMMIT readback carry and compare the operation id and publisher_epoch, so a competing epoch's publication at the same tick classifies as lost — witnessed by a real Postgres wire proxy that swallows CommandComplete(COMMIT) after the backend committed. Reviewed (Opus, cost-reduction/04-plans/P-026-S1-review.md): rebuilt from a clean copy — cargo 31/31 both feature sets, clippy clean, Python 131/0/0/0, replay 0/0/0 with Rust=Python hashes, polarity byte-identical, 25/25 stages, D4 real-Node read 0 differences; migration idempotent over three re-runs. Corrections to follow in the next commit: O8 is PARTIAL, not closed (audit_stages.py pops the still-true G01–G16 disclaimer), and the S1 closure record must not pin gitignored target/ and artifacts/ paths. The identity scheme is self-certifying, not tamper-evident, and says so. Rust ≡ Clojure is neither tested nor claimed. Authored headless by Astra (Codex) under Claude orchestration. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ecord must hold off this machine Applies C1-C5 of the Opus S1 review to Astra's S1 commit. The code is unchanged except for one provenance label; everything else is the evidence and gate layer. C1: audit_stages.py no longer pops the standing "not a G01-G16 certificate" disclaimer to close O8. That sentence is still true after S1 - only the id changed - and O8's obligation is larger than S1 delivers. The condition survives, reworded to polis-candidate-input/1 and carrying the rest of the obligation from the plan: the G01-G16 case set with negative controls, an immutable manifest, the three output schedules, and the P-023 pair with raw C9 validation. O8 now reads PARTIAL, closed_conditions is empty, a new partial_conditions entry names what was recorded and what remains, and open_conditions is eight again. C2: the closure hash-gated gitignored target/ binaries and artifacts/ logs, so a clean rebuild printed "S1 closure refused" and rewrote the committed inventory to O8 OPEN. It now pins the 92 committed inputs only; binary/JUnit/log hashes move to an informational run_pins block flagged verifiable_on_producing_host_only, and the audit refuses any pin under target/ or artifacts/ so this cannot come back quietly. Proven on an rsync copy with no target/ and no artifacts/ logs: the rebuilt binary hashes differently, the audit says nothing on stderr, and the inventory summary is byte-identical to the producing host's. Five negative controls refuse and return O8 to OPEN. C3: record_s1.py derives its counts from the run instead of asserting 131/31/15, and reads the compose project and Postgres port from the environment instead of carrying p026s1x-c84e/55864 as literals. C4: the README states the resident Bundle now holds parsed JSONB and original bytes both, roughly twice the per-entry payload cost at the same P026_CACHE_CAP, and that halving the cap holds the previous ceiling. No new measurement taken. C5: publish-fixture announces which originals it synthesized by re-encoding the fixture's parsed values, on stderr so no stdout parser moves; validate_originals is tautological for those rows and they are not engine originals. Also states in the README and the record that this identity scheme is self-certifying, not tamper-evident: it catches an incoherent generation and catches nothing about a coherent forgery by anyone who can write the four math tables. Gate re-run: cargo 31/31 both feature sets, clippy clean both, Python 131 passed 0 skipped, replay 0/0/0, polarity unchanged, 25/25 stages, D4 0 differences, full_contract_gate still FAIL with eight open conditions. Rust equals Clojure is neither tested nor claimed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The step-4 plan checker literal_eval'd the eight-entry open_conditions list out of audit_stages.py and compared it straight to the emitted open_conditions. The audit removes a condition from that list when it is genuinely discharged, so the checker would report source=8, evidence=7 forever from the first real closure. It passes today only because C1 put open_conditions back to eight with closed_conditions empty: latent, not absent. Replayed against the pre-C1 shape at c50f762 (7 open + 1 closed) the old comparison does FAIL and the new accounting passes it. The parsed list is now the declaration, and each of the eight must be accounted for exactly once: still open, or removed from open and recorded in closed_conditions with its prior text, so len(open) + len(closed) == 8. A partial_conditions entry discharges nothing and must still be listed open, which is what lets O8 be PARTIAL without disappearing from the open list. Order and exact wording are still compared, over declaration-minus-closed. Three controls run on every invocation: a condition closed and recorded is accepted; a condition dropped entirely is refused; a partial record pointing at a quietly reworded condition no longer listed open is refused. The checker lives in the git-excluded cost-reduction/ folder, so what lands here is the control results, under plan_checker_controls in evidence/s1-audit-controls.json. Python test set re-run: 131 passed, 0 failed/errors/skipped; record_s1 and audit_stages agree (25/25, replay 0/0/0, 8 open, O8 PARTIAL, gate FAIL). No cargo rebuild; the fix touches no Rust. Source pins are byte-identical across the two runs - only run_pins, the replay observation count and the request-clock empty artifact moved. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…e process says so Astra's P2 on C3: with the hardcoded == 31 gone, the recorder summed only `test result: ok` lines, so a passing suite followed by a failed one - 12 passed, then 18 passed; 1 failed and error: test failed - was accepted and reported as a clean 12. The recorded 31/31 run did not fail; the point is the recorder could not have told. rust_test_counts() is now a module-level function that refuses before it counts. Each cargo invocation writes its own $? beside its log (six .status files; the clippy and build checks use the same helper), and a missing status file is malformed evidence, not a pass. Every `test result:` line in the transcript must then be ok with 0 failed and 0 ignored - ignored tests are not passes - and a malformed result line or a leading error:/error[ line is refused. Only then are the passes summed. No fixed total comes back: the two profiles must still agree and be non-empty. Eight controls land as delphi/tests/coordinator/test_recorder_controls.py, inside the Python set, on synthetic transcripts only - no cargo, no database, no engine. Multi-suite success is accepted at 31/31; pass-then-fail, an ignored test, a non-zero exit status, a missing status file, a malformed result line, no result line and disagreeing profiles are all refused. The outcomes are recorded under recorder_cargo_controls in evidence/s1-audit-controls.json, and the README gate commands now show the status capture. Python set and record script re-run: 139 passed (131 + the 8 controls), 0 failed/errors/skipped; 25/25, replay 0/0/0, 8 open conditions, O8 PARTIAL, gate FAIL. No cargo rebuild for the change, which touches no Rust - but the six cargo invocations were re-run to capture the exit status no earlier artifact carried, because writing a 0 next to a run nobody observed is the thing this correction exists to prevent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
colinmegill
force-pushed
the
experiment/rust-coordinator
branch
from
September 9, 2026 03:35
0b1eeaa to
0798f71
Compare
The driver bridge as a THIRD certify producer beside clj/py, additive by construction and honest about the slice-3 blockers. - delphi/polismath/replay/coordinator_driver.py (new, HARNESS-ONLY): DriverSpec registry (clj/py/rust), DriverReceipt, parse_drivers validation (unknown/repeated/empty rejected; rust requires both references), the closed p045-policy/1 loader (never derived from errors), the separate-namespace bridge recording-cache key binding every field the brief §2 lists, and the three-producer inventory helper. The rust producer records an honest UNSUPPORTED_PROFILE receipt (the once --replay-plan forced-compute path and the p045-replay-plan/1 schema are slice-3, BLOCKED); it is NEVER implemented by calling ensure_py_recording and relabelling. - delphi/polismath/replay/certify.py: add "replay/coordinator_driver.py" to _ENGINE_TREE_EXCLUDE (review round-2/3 correction that overrides the brief: the bridge is the fifth harness sibling of files already excluded, so editing it never re-keys the py recording cache nor moves run_provenance. engine_tree_sha256 for a legacy two-driver command). No other change; the legacy clj/py path is untouched byte-for-byte. - delphi/tests/replay_harness/test_certify_rust_cli.py: the engine-tree-hash exclusion control (a bridge edit does not move the hash; a real engine edit does), py-recording-key and provenance additivity controls, inventory additivity (legacy inventory() unchanged, rust appended), registry/policy/ cache-key units, and the gated legacy two-driver re-record byte-identity stub. Test: tests/replay_harness/test_certify_rust_cli.py -> 37 passed, 1 skipped (the skip needs the JVM + uv to re-record the two-driver battery). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
The dispatchable-now half of slice 2, graded against snapshot-rebuild/1 (the only profile the S1 binary implements) as PURE validators — no Postgres, no Rust binary. They are the contract layer the slice-3 real-mutation N-controls will drive against an actual Rust publication. coordinator_driver.py (appended): - validate_s1_identity: the five closed admission fields exact (candidate_schema / engine_version are the pinned S1 constants; a changed engine_version must NEGOTIATE a new version, never claim S1 bytes) + retained checkpoint fields; bind_campaign_cut keeps the campaign plan hash distinct from S1's per-session schedule_digest. - validate_readback: four-row math_tick agreement, prior->next tick (first publication 0), companion presence, epoch/operation binding, per-table original-byte custody + JSONB correspondence; data::text is refused as origin; caching_tick only on math_main, never on a companion. - observe_bundle_coherence: independent post-cut coherence (one generation across four rows, well-formed bid->index->pid), with an injectable fold_check so the recovery oracle's fold can be reused without importing a pinned asset. - stage sibling: load/check_stage_context (foreign + digest-mismatch rejection), stage_binding_manifest (closed p045-stage-binding/1, producer rust / stage engine py), capture_rust_stage_sibling (ONE immutable operation dir per snapshot; never write_stage_documents' delete-then-write nor a second run_stage_dump). Tests: test_certify_rust_identity.py, test_certify_rust_publication.py, test_certify_rust_stages.py. Deviation from brief §5: the same-invocation stage capture is placed in the bridge, NOT inside engine_adapter.snapshot. Rationale: (1) the real capture requires Rust Worker::start to pass --stage-json/--stage-context and the worker to INDEPENDENTLY derive+check the semantic digest from the inputs it actually applied — a real-worker-lifecycle concern that is slice 3; (2) editing engine_adapter.py (engine surface, NOT excluded from engine_tree_hash) would re-key the Python recording cache for zero output change, which both the additivity requirement and engine_tree_hash's design counsel against. The bridge exposes capture_rust_stage_sibling + check_stage_context so the in-adapter hook, when added in the real-worker slice, is a thin call into reviewed code. The once --replay-plan/--checkpoint/--evidence-dir path and all of slice 3 are untouched. Test (slice 0+1+2 replay_harness): 254 passed, 2 skipped (env-gated: JVM/uv, RUN_CLJ_INTEGRATION). Existing certify library tests: 71 passed, 1 skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…idence Closes Astra's implementation-round-1 false accepts (board [398]); each has a fail-before/pass-after regression test (verified against 35a3a40). - Correction 3 (identity): validate_s1_identity now BINDS the versioned checkpoint schema (polis-candidate-checkpoint/1) and every retained field's expected value/type, rejects null identity fields, empty admission digests and unknown admission keys. load_policy rejects non-hex field_policy_sha256 and unknown keys and returns admitting=False (the policy body is undefined — slice 3 — so shape-valid is never admission). - Correction 4 (readback): requires publication scope (non-boolean int zid + nonempty math_env, with optional expected_zid/expected_math_env binding), non-boolean integer math ticks, REQUIRED JSONB `data`, and TYPE-AWARE JSON correspondence (a JSON bool is never equal to an int; PG's 1 vs 1.0 numeric normalization is still allowed). - Correction 5 (observer): refuses absent evidence (empty bundle, absent rows/ticks/data) and actually validates the bid->index->pid mapping (integer bids -> integer-pid lists), not a list-typed member field. - Correction 6 (stage context): keyed on the FULL identity (plan_sha256, session_id, compute_id, checkpoint_id); duplicate identities rejected before insertion (S1 reuses compute-0/checkpoint-0 across sessions); typed nonnegative cut index and known profile required; check_stage_context validates expected profile/cut when consuming. - Correction 7 (stage evidence): capture_rust_stage_sibling writes EXCLUSIVELY — an identical retry is idempotent but any differing content for the same operation/cut is refused (no truncate-and-replace); an optional run_root confines the operation directory. Tests: test_certify_rust_identity.py, test_certify_rust_publication.py, test_certify_rust_stages.py (the policy-strictness test additions ride with the CLI tests in the next commit). Verified each new assertion FAILS against 35a3a40 and PASSES here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
… additivity - Correction 1 (reachability): the rust producer can now be invoked end-to-end. coordinator_driver.run_bridge_battery runs the three producers over resolved entries and writes a TERMINAL polis-certification-run/2 report (inventory, producers, pairs, verdict) with per-entry clj/py/rust rows — rust UNSUPPORTED_PROFILE, references INCONCLUSIVE, verdict INCONCLUSIVE, exit 2, never a fabricated PASS. scripts/certify.py gains `--drivers`/`--profile`: the default `clj,py` with no profile is the legacy path BYTE-FOR-BYTE unchanged; any other driver set routes to the bridge. This uses no blocked forced-compute or warm path. NAMED, not faked: the Clojure three-table companion sink (real cm/prep-* + upload-math-* capture) is a real-engine change to math/dev/replay.clj and is not implemented here. - Correction 2 (additivity): the retained-recording comparison now RUNS instead of unconditionally skipping — it exercises the UNCHANGED legacy compare_recording_pair over retained recordings (no JVM), asserting the verdict payload bytes and raw blob digests are deterministic and the engine hash is bridge-invariant, without filtering fields. The genuine pinned pre-change full-battery report is named as the one missing artifact (operator-supplied via P045_REFERENCE_RECORDING_ROOT); only the FRESH re-record stays JVM-gated. Tests: test_certify_rust_cli.py — library-level bridge-runner tests run unconditionally (no click); CLiRunner CLI tests run when click is present and skip otherwise (the substantive reachability proof is the always-running library test, not a gated one). Also carries the correction-3 policy-strictness test additions (non-hex digest, unknown key, admitting=False). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Closes Astra's implementation-round-2 residual defects; each has a fail-before/pass-after regression test (verified against 5ef0a95). Astra's 20 controls still pass and each of the 13 defect observations now inverts. - Correction 1 (observer real shape): observe_bundle_coherence models the ACTUAL math_writer.derive_bidtopid wrapper {zid, bidToPid:[[pid,...],...], lastVoteTimestamp}, resolves bids positionally through main.base-clusters.id and requires bidToPid[i] == base-clusters.members[i]. It now ACCEPTS the real writer wrapper (including a valid empty generation) and REJECTS an unrelated invented {bid:[pid]} map, overlapping/duplicate buckets, a positional membership mismatch and a zid mismatch. A dict-of-lists shape test is gone. - Correction 2 (expected binding + typing): validate_s1_identity requires persistence EXACTLY False and accepts optional expected_admission/ expected_identity so a well-typed FOREIGN session/digest is rejected; validate_readback types publisher_epoch (a JSON bool never matches an expected integer) and binds the full input_checkpoint (a foreign checkpoint object is rejected, an expected one can be pinned). PG 1-vs-1.0 normalization kept. - Correction 3 (stage-context identity): load_stage_context validates the container, the closed key set and nonempty typed identity values BEFORE building the key (a null or list-valued identity is a graded BridgeError, not an ungraded TypeError); the key now includes run_id, and check_stage_context binds run (a foreign run with the same local names does not match; an unbound consumer fails closed). - Correction 4 (report/selection agreement): the bridge runner honors a driver subset (a py-only run emits ONLY py, no clj/rust rows or pairs) via a _plan_ profile helper and drivers-scoped inventory/rows/pairs, and reference receipts now carry the selected profile instead of "". - Additivity (round-2 correction 2, completed): the additivity control now runs the UNCHANGED legacy comparison over the REAL retained clj<->py recording on disk (generated here for vw/single-cut via the real drivers) and asserts its behavior + raw blob digests are deterministic and the engine hash is bridge-invariant, on real bytes. It skips (naming the path) only when no real recording is present. A pinned pre-change full-battery verdict remains the one operator-supplied artifact for a complete byte-identity proof; the fresh re-record stays JVM-gated. scripts/certify.py is unchanged this round (legacy CLI body verified intact). Test (bridged venv with click, seven modules): 212 passed, 2 skipped (JVM-gated). Astra's env (no click): 210 passed, 4 skipped. Slice-0 oracles: 175 passed, 1 skipped. Round-3 probe vs new code: 20 controls pass, first defect inverts; a scratch inversion confirmed all 13 defects closed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…int custody Closes Astra's implementation-round-3 residual defects (board [431]); each has a fail-before/pass-after regression test (verified against 6359d68). Astra's controls still pass and each of the 10 defect observations now inverts. - Correction 1 (total observer invariants): observe_bundle_coherence now checks, on the REAL derive_bidtopid wrapper, that base-cluster ids are typed and unique, base counts equal each cluster's membership size, participant memberships are a PARTITION (unique within a bucket and disjoint across buckets), and every group-cluster bid references an existing base cluster id — in addition to the positional bidToPid==members equality. The duplicate/overlapping, wrong-count, duplicate-base-id and unknown-group-bid witnesses (all with matching original-byte custody) are now rejected; the valid distinct-id and empty-generation cases still pass. The duplicate-bucket negative control now uses the real wrapper. - Correction 2 (full worker envelope + persisted-checkpoint custody): validate_s1_identity binds the COMPLETE closed 15-field polis-candidate-checkpoint/1 envelope — every field mandatory, no unknown top-level keys, files (four {path,bytes,sha256} descriptors) and both cursor maps ({votes,moderation} -> {integer-non-bool slot, sha256}) required — and fixture_id is now a bound expected-identity field (a checkpoint for fixture 999 is rejected when fixture 1 is expected). The positive fixture is the full emitted shape. validate_readback treats the STORE checkpoint (polis- coordinator/1) as a separate REQUIRED envelope: absent, operation-only and wrong-schema checkpoints are rejected, cursor slots are typed integer-non-bool, original_digests must agree with ticks, and an expected checkpoint binds by TYPE-AWARE equality (a boolean cursor slot never equals an expected integer; PG 1-vs-1.0 spelling still normalizes). Test (bridged venv, seven modules): 232 passed, 2 skipped (JVM-gated). Astra's env (no click): 230 passed, 4 skipped. Slice-0 oracles: 175 passed, 1 skipped. Round-4 probe vs new code: 17 controls pass, first defect (observer) inverts; a scratch inversion confirmed all 10 defects closed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
Closes Astra's implementation-round-4 residual defects (board [440]); all 13 observations (11 false accepts + 2 ungraded TypeErrors) now invert to graded rejections, verified fail-before/pass-after against 9ff0dad. Following the G12 round-5 method, admission is now derived from ONE typed schema rather than per-case checks. - New schema engine (_admit) + declarative specs: every bundle/checkpoint/ envelope structure is described once (kind, rank, element kind, closedness, nonnegative bounds, hex digests). _admit walks any value TOTALLY and returns GRADED BridgeError-style failures — it never raises on a wrong type, so the two prior ungraded TypeErrors (scalar main membership `7`; unhashable group bid `[[]]`) are gone. - Correction 1 (total membership admission): validate the companion AND main sides via _BIDTOPID_SPEC/_MAIN_SPEC before any coercion, iteration or hashing; boolean main pids (`[true,2]`), float group bids (`2.0`), a non-list group container (`{}`) and a scalar group row (`[7]`) are all rejected with names. The relational partition/count/positional/group-reference checks then run on already-typed data. - Correction 2 (worker cursor/file custody by VALUE): the envelope schema binds nonnegative descriptor `bytes`, nonnegative cursor `slot`s and 64-hex `sha256`s; the two cursor maps must AGREE with each other (both bind the same source cursors at the snapshot seam), and optional expected_cursors/expected_files bind them explicitly. Shape-only acceptance is gone. - Correction 3 (persisted store checkpoint): match store.rs — require and bind `publisher_epoch` to the row/expected epoch, require and verify CANONICAL `payload_digests` (numeric-normalized JSONB digest, distinct from original-byte hashes) against each table's data, keep original_digests binding, and admit the actual versioned votes/moderation cursor shape. Missing-epoch/payload, foreign-epoch, wrong-payload-digest and unknown-cursor-stream checkpoints reject even when supplied as the expected checkpoint. Positive fixtures updated to the full emitted shapes (`_manifest`, `_bundle` now carry the complete envelope + store checkpoint with epoch and canonical payload digests). scripts/certify.py, test_certify_rust_cli.py and _stages.py unchanged. Test (bridged venv, seven modules): 248 passed, 2 skipped (JVM-gated). Astra's env (no click): 246 passed, 4 skipped. Slice-0 oracles: 175 passed, 1 skipped. A scratch inversion confirmed all 13 defects close graded (no TypeError) and the positive controls hold. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…ssion Closes Astra's implementation-round-5 residuals (board [445]); all 17 observations (6 exceptions, 9 digest mismatch/false-rejection, 2 cursor false accepts) now invert, verified fail-before/pass-after against 6a542f9. - Correction 1 (real Rust digest, byte-for-byte). Replaced the compact-JSON hash with a literal port of coordinator-rs/src/store.rs storage_digest (store.rs:21-89): the number normalizer (store.rs:23-51 — expand exponent, trim insignificant fractional zeroes, drop negative-zero sign), the encoder (store.rs:52-83 — serde string encoding with Unicode written through, no ASCII escaping; `/` unescaped; BTreeMap-sorted object keys since serde_json has no preserve_order; bool before int) and sha256. _canonical_payload_digest now matches the production Rust digest on 1/1.0, -0.0, 1e-7, "é" (which compact JSON got wrong) and on the REAL retained vw/single-cut Python output (Rust+bridge both b80bb10831a8de09558a9cdd4cef2c665368d674d13366f77874981bf30e6623). New test_certify_rust_digest_parity.py builds a STANDALONE probe from the unchanged store.rs digest source (extracted at test time — cannot drift, no full-crate build) and asserts parity on Astra's inputs and the real blobs. - Correction 2 (graded outer admission). validate_s1_identity, validate_readback and observe_bundle_coherence now return a graded BridgeError-style failure for a non-object outer container (null/list/scalar) and a non-object row, instead of raising AttributeError/TypeError — the schema's verdict controls execution at every public boundary (a _row_data accessor guards nested dereferences). - Correction 3 (stored cursor hashes). The store checkpoint cursor map now reuses the typed {slot, sha256:hex64} cursor spec (engine.rs:374,393-396), so a missing or malformed cursor sha256 is rejected even when supplied as the expected checkpoint; the _bundle fixture carries both cursor hashes. The test _canon helper now delegates to the single ported implementation (no duplicate). scripts/certify.py, test_certify_rust_cli.py and test_certify_rust_stages.py unchanged; coordinator-rs untouched (the parity probe is a scratch build). Test (bridged venv, eight modules incl. Rust parity): 292 passed, 2 skipped (JVM-gated). Astra's env (no click): 290 passed, 4 skipped. Slice-0: 175 passed, 1 skipped. Rust digest parity: exact on all probe inputs and both real vw blobs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…xture rigor Closes Astra's implementation-round-6 residuals (board [449]); all 9 observations (4 row exceptions, 4 integer-range digest differences, 1 compiler-error skip) invert, verified fail-before/pass-after against 370d1b6. The 9,997-sample f64 agreement and both real vw digests are unaffected. - Correction 1 (finish row admission). validate_readback no longer dereferences a truthy non-object row: `ticks`, the companion caching-tick loop and the main caching-tick check are all guarded with isinstance, so ticks=[1], bidtopid=[1], ptptstats=[1] and main=7 now return NAMED failures instead of raising AttributeError/TypeError. The observer already graded these. - Correction 2 (serde numeric representation at integer boundaries). _rust_encode now mirrors serde_json::Value's parse dispatch (no arbitrary_precision): an integer token in [i64::MIN, u64::MAX] is written EXACTLY (full i64/u64 precision, no float round-trip); anything outside takes the serde f64 representation (store.rs:56 hashes number(n.to_string()) of the Number, which is f64 out of range) — Python repr(float(n)) shares the unique shortest-round-trip digits that number() normalizes. So {"x": 2**64} now digests to eb369f44996329cb7252a3fad311df14e1e46bbc28f56aa15a9baa381ae16f8f (Astra's cited Rust value), and 2**64+1 / -2**63-1 / 10**25+1 match, while u64::MAX and i64::MIN stay exact. A value beyond f64 range is a graded BridgeError (serde rejects non-finite). - Correction 3 (parity fixture rigor). test_certify_rust_digest_parity SKIPS only when cargo or store.rs is genuinely ABSENT; a missing source marker, a nonzero cargo build or a missing binary now FAIL (a broken digest oracle is not an unavailable toolchain). The probe's serde_json/anyhow/sha2 are pinned to the coordinator's EXACT Cargo.lock entries (serde 1.0.229 / serde_json 1.0.151 / zmij 1.0.23), asserted against the lock, so the float formatter cannot drift. Boundary cases (2^53+1, i64::MIN, u64::MAX, 2^64(+1), -2^63-1, 10^25+1, 1e21, 1e-7, -0.0) are added to the parity comparison against the real Rust function. scripts/certify.py, test_certify_rust_cli.py, test_certify_rust_stages.py and coordinator-rs/ unchanged. Test (bridged venv, eight modules incl. Rust parity): 327 passed, 2 skipped (JVM-gated). Astra's env (no click): 325 passed, 4 skipped. Slice-0: 175 passed, 1 skipped. Rust digest parity exact on all boundary cases and both real vw blobs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…n sweep Closes Astra's implementation-round-7 residual (board [454]); all four exception witnesses invert, verified fail-before/pass-after against 91bee70. - Correction 1 (admit custody values before use; grade digest errors). In validate_readback: `ticks.original_digests` is read through a graded-safe `_dict_or_empty` accessor (a non-object digest map no longer raises AttributeError on `.get`); `original_bytes` is admitted as str OR bytes BEFORE conversion (a nested array `[[]]` or object `{"x":1}` is a named failure, not a `bytes()` TypeError); and the canonical digester's `BridgeError` (a payload integer beyond serde's f64 range, e.g. `10**400`) is CAUGHT at the readback boundary and appended as a graded entry instead of escaping the failure-list API. The low-level `_canonical_payload_digest` still raises the named BridgeError; every public failure-list function now grades it. Genuine str/bytes evidence, numeric JSONB correspondence and all valid controls keep working. - Property-style admission sweep (new test_certify_rust_admission_sweep.py): hypothesis is not in the venv, so a seeded (deterministic) recursive generator feeds malformed JSON shapes — including out-of-f64-range integers, inf/nan, control-char strings and mis-typed rows/checkpoints — into every public failure-list entry point (validate_s1_identity, validate_readback, observe_bundle_coherence) both as the outer container and substituted into each row/field of valid factory evidence, asserting each returns a list[str] with no exception. Sweep count: 33,748 public-entry calls, no exception escaped. scripts/certify.py, test_certify_rust_cli.py, test_certify_rust_stages.py, test_certify_rust_digest_parity.py and coordinator-rs/ unchanged. Test (bridged venv, nine modules incl. Rust parity + sweep): 335 passed, 2 skipped (JVM-gated). Astra's env (no click): 333 passed, 4 skipped. Slice-0: 175 passed, 1 skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
…n sweep Closes Astra's implementation-round-8 residual (board [459]); all three surrogate witnesses invert, verified fail-before/pass-after against 0771af0. - Correction (unpaired surrogates). A lone surrogate (U+D800..U+DFFF) cannot exist in a Rust String, so a JSON text carrying one fails to parse into serde_json::Value; the bridge mirrors that as a GRADED refusal instead of a UnicodeEncodeError: * _serde_json_string raises a named BridgeError on any unpaired surrogate (key or value), which _canonical_payload_digest propagates and validate_readback already catches at the readback boundary — so a payload value {"x":"\ud800"} or key {"\udfff":1} is a graded failure, not an escape; * validate_readback's original-byte encode is wrapped so a str original_bytes with an unpaired surrogate is a named failure, not UnicodeEncodeError. Valid non-BMP scalars (a proper surrogate PAIR, which Python's JSON parser has already combined into one codepoint) encode and hash EXACTLY as serde does — digest parity confirmed against the real Rust probe for é, U+1F600, the pair spelling, and an astral KEY. - Extended the property-style admission sweep: unpaired/paired surrogates and other non-UTF-8-encodable strings are injected into every string position (payload keys AND values, original_bytes, and string custody fields such as the checkpoint operation_id/schema), across every public failure-list entry point. Sweep count: 35,080 public-entry calls, no exception escaped (was 33,748). scripts/certify.py, test_certify_rust_cli.py, test_certify_rust_stages.py and coordinator-rs/ unchanged. Test (bridged venv, nine modules incl. Rust parity + sweep): 347 passed, 2 skipped (JVM-gated). Astra's env (no click): 345 passed, 4 skipped. Slice-0: 175 passed, 1 skipped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s
colinmegill
force-pushed
the
experiment/p045-certify-rust-driver
branch
from
September 9, 2026 03:36
c863fdf to
bacbe6b
Compare
colinmegill
force-pushed
the
experiment/rust-coordinator
branch
5 times, most recently
from
September 15, 2026 01:16
04ba2bf to
7538568
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
experiment/rust-coordinator(#2727). Experiment-only; nothing served.feat/engine-stage-oracles(feat(replay): per-stage engine oracles for Clojure and Python (diagnostics) #2721, head 5094568) cherry-picked linearly sostages.py/stagecompare.py/--stage-jsonexist on the Rust branch. They fall out on rebase once feat(replay): per-stage engine oracles for Clojure and Python (diagnostics) #2721 merges to edge.c3d70ae66) — additiverustcertify producer + bridge registry (polismath/replay/coordinator_driver.py), onep045-policy/1name with a validating loader.replay/coordinator_driver.pyadded to_ENGINE_TREE_EXCLUDEso the bridge never entersengine_tree_hash; control test edits the bridge (hash unchanged) then a real engine file (hash changes). Existing py/clj producers untouched; recording cache keys and provenance proven bridge-invariant.35a3a4033) — identity / readback / observer / stage-sibling validators graded againstsnapshot-rebuild/1; closed namep045-stage-context/1added for the stage-context map.Not here: the
once --replay-plan/--checkpoint/--evidence-dirforced-compute path and all of slice 3 (blocked on the warm-vs-rebuild decision, Q26). Therustproducer therefore recordsUNSUPPORTED_PROFILEper entry — an honest non-pass, no passing Rust row can exist yet.Tests: stage oracles 175 passed / 1 skipped (JVM e2e); slice 1 37 passed / 1 skipped (JVM+uv); slice 2 42 passed; legacy harness regression 71 passed / 1 skipped. Notes:
cost-reduction/04-plans/P-045-implementation-notes.md(not in repo). Second review pending.🤖 Generated with Claude Code
https://claude.ai/code/session_018NVGBuYCk4EZmiz4csUv9s