Skip to content

Add captured two-run qualification evidence - #71

Merged
Taz33m merged 1 commit into
mainfrom
codex/qualification-evidence-scaffold
Aug 3, 2026
Merged

Add captured two-run qualification evidence#71
Taz33m merged 1 commit into
mainfrom
codex/qualification-evidence-scaffold

Conversation

@Taz33m

@Taz33m Taz33m commented Aug 3, 2026

Copy link
Copy Markdown
Owner

What changed

  • add evidence-init to pin candidate name, revision, and an origin-stripped source snapshot before adapter work, then prepare two independent candidate/adapter/build/cache roots
  • add optional protocol-v1 engine revision and snapshot_id metadata plus all-or-none CLI identity enforcement before the first event
  • add evidence-verify to reject changed trees, incomplete or noncanonical bundles, mismatched metadata, invalid deterministic IDs, count or coverage drift, and byte differences
  • emit one compact, content-addressed evidence-manifest.json for a valid pair
  • document the exact workflow and its honest non-sandbox boundary

Why

The held-out agent-qualification study showed that compatible-engine release evidence still failed when task identity, fresh roots, canonical bundle capture, and equality gates lived only in prose. This moves those gates into the public tool so an agent or maintainer receives a deterministic pass/fail contract and a reviewable manifest.

This remains separate from #70 and does not modify the frozen experiment cohort, intervention, or outcomes.

User impact

Ordinary adapters remain wire-compatible: the new engine metadata fields are optional unless all three --candidate-* pins are supplied. Release and benchmark claims can require two clean, byte-identical qualifications bound to one captured candidate snapshot.

The workflow verifies captured source and artifact integrity. It does not sandbox compilers or prove that an adapter avoided undeclared external caches; disposable CI or a clean host remains the strongest execution boundary.

Controlled external-engine dogfood

The workflow was exercised after review against the real external gocronx/matcher engine at pinned revision b8d48356c8a2677e0d8a1965d754e3c4884bb947.

  • origin-stripped candidate snapshot: sha256:1055904eb895d9768f9084907701e3569917126fcbe4d1fe96be3544e507b645
  • two candidate copies, two path-bound Rust adapter builds, two Cargo homes, and two target directories
  • both builds ran on the M3 clean roots; both qualifications ran through fresh adapter processes
  • each run: 3/3 fixed cases, 25/25 generated traces, 5,000 events, 10/10 capabilities
  • both canonical bundles were byte-identical
  • manifest: sha256:4f5d936af8cd1f1a85bd306129db95f9820cc19d16bef5248ef4bda52935876d
  • qualification: sha256:1b7b8b17c73488816216e3132670385470049592a774ea0229f34859a0b502e3

This is operator-run implementation evidence, not independent adoption or an untouched-defect result.

Validation

  • 407 tests pass; 81.34% coverage against a 75% gate
  • Black: 100 files clean
  • Flake8: clean
  • mypy: 54 source files clean
  • Bandit: clean
  • all seven GitHub CI and maintained-integration checks green
  • isolated built-wheel smoke with no NumPy or psutil:
    • two independent 3/3 fixed-case qualifications
    • two independent 25/25 generated campaigns, 10,000 events total
    • 10/10 capabilities in both runs
    • final evidence manifest PASS with identical artifact hashes

@Taz33m
Taz33m marked this pull request as ready for review August 3, 2026 16:04
@Taz33m
Taz33m merged commit a0b5ca9 into main Aug 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant