Skip to content

Fix stochastic sweep seed integrity and add actual demo - #2

Open
jadenfix wants to merge 3 commits into
mainfrom
codex/reproducible-sweep-seeds
Open

jadenfix wants to merge 3 commits into
mainfrom
codex/reproducible-sweep-seeds

Conversation

@jadenfix

Copy link
Copy Markdown
Owner

What changed

  • propagate each sweep seed into both the manifest environment RNG and prompted/table policy fallback
  • make create_env honor env.config.seed for all graph-manifest environments
  • add regression coverage proving same-seed determinism, cross-seed independence, and trained-evaluation propagation
  • add a source-pinned 12-second GIF/MP4 of the actual stochastic/v1 manifest path plus a full-decode/hash verifier
  • add Python 3.11/3.12/3.13 CI and keep the demo receipt under CI

Why

Previously the sweep seed reached the prompted policy but the manifest environment and trained table-policy fallback silently used seed 0. Multi-seed stochastic evaluation therefore did not fully vary the intended random sources.

Verification

  • PYTHONDONTWRITEBYTECODE=1 python -m pytest -p no:cacheprovider → 13 passed
  • python scripts/verify_seed_integrity_demo.py → verified
  • GIF and MP4 both fully decoded with ffmpeg; each is 12.0 seconds
  • captured source commit: b39ab74c07cc56f30e96fd8d955170e012167356
  • captured source tree: ffc8450ecf5b4a271e3ca3fc04ae1f45f86aeeb9

Scope

The demo uses the checked-in provider-free stochastic fixture. It does not exercise a language model, live provider, or validate the paper's reported performance results.

Copy link
Copy Markdown
Owner Author

Exact-head verification receipt (2026-10-13 UTC)

  • final head: 7cd7086d22cff7265f23cf88d19f25aca5de1920
  • final tree: 4b2b80d075fe5f5dfbd8bb117513923546fb4a8a
  • captured source: b39ab74c07cc56f30e96fd8d955170e012167356
  • captured source tree: ffc8450ecf5b4a271e3ca3fc04ae1f45f86aeeb9
  • local: 13 tests passed; source/blob receipt verified; GIF and MP4 fully decoded; representative frames visually inspected
  • hosted run 38431595127: Python 3.11, 3.12, and 3.13 all passed tests and the intact ffmpeg-backed media verifier

The first hosted run correctly failed closed because the refreshed Ubuntu image lacked ffmpeg. Head 7cd7086 only adds an explicit ffmpeg installation to CI; it does not change captured runtime or media inputs. The provider-free demo uses the checked-in stochastic fixture and makes no model, live-provider, or paper-performance claim.

Held for the designated portfolio merger; no merge performed here.

Copy link
Copy Markdown
Owner Author

Independent exact-head clearance for 7cd7086d22cff7265f23cf88d19f25aca5de1920:

  • I previously reproduced the runtime/demo commit (7ab787b4f43bd5e14f217f5e712d7290d8b97f38) locally: 13 tests passed, the source-integrity verifier passed, GIF/MP4 fully decoded, the checkout stayed clean, and inspected frames accurately showed the deterministic seed workflow.
  • The only change from that reviewed head to this final head is .github/workflows/ci.yml adding an explicit ffmpeg installation before the existing verifier. No runtime, test, capture, or media blob changed.
  • Hosted run 38431595127 is success; the Python 3.11/3.12/3.13 matrix and intact media verifier all pass.
  • The captured source remains b39ab74c07cc56f30e96fd8d955170e012167356. Demo claims remain limited to fixture-backed reproducibility/seed propagation, with no accuracy, scale, or performance claim.

Review hold is cleared for this exact head. This is evidence for the designated merger; I did not merge.

Copy link
Copy Markdown
Owner Author

Independent exact-head re-review after the CI repair: READY (no merge performed).

Validated identities:

  • PR head 7cd7086d22cff7265f23cf88d19f25aca5de1920
  • head tree 4b2b80d075fe5f5dfbd8bb117513923546fb4a8a
  • current base 11d4afa156f3ffae05f7a81792ff4cbe03cb4740
  • captured runtime source b39ab74c07cc56f30e96fd8d955170e012167356
  • captured source tree ffc8450ecf5b4a271e3ca3fc04ae1f45f86aeeb9

The final commit is narrowly scoped: compared with prior reviewed head 7ab787b4f43bd5e14f217f5e712d7290d8b97f38, only .github/workflows/ci.yml changed, adding the ffmpeg install. The capture receipt, GIF/MP4, verifier, and runtime source blobs are byte-identical to that reviewed head.

Independent local verification from the exact remote head:

  • python -m pytest → 13 passed
  • python scripts/verify_seed_integrity_demo.py → seed integrity demo: verified
  • fresh generator replay with .[demo], source SHA b39ab74..., and a temporary output directory reproduced the GIF, MP4, and full receipt byte-for-byte
  • MP4: H.264, 1280×720, 180 frames, 12.0s, 164,354 bytes
  • GIF: 960×540, 120 frames, 12.0s, 252,820 bytes
  • published SHA-256 values match the receipt:
    • GIF d3eb157c762bb3c4677ea37fcb66916e21837ed5b48c9a95fe51cf5aafbf00d3
    • MP4 45044b69f3d3d0650361d4c1e515d882bc2273106fb03ce7b9b4aa9ab05e2988
  • contact-sheet inspection confirms the video visibly labels the provider-free fixture, shows the two actual stochastic traces, and limits claims to same-seed replay/cross-seed distinction
  • checkout remained clean

Remote blob verification from refs/pull/2/head:

  • capture.json b8c4849232070bfc6091063d973fd3641e3e3bb7
  • GIF e6fbeeed83b405d539b6c381551a86bb2cdc0802
  • MP4 777ed38b6753fb7c4af9c97af7ab123e3d0b26c4
  • verifier 316868dce7085cd1fee5501fd322cd7ac1102218

Hosted run 38431595127 is green on the integrated PR merge ref for Python 3.11, 3.12, and 3.13. Each lane reports 13 passed, successfully installs ffmpeg, and reports seed integrity demo: verified.

Scope remains truthful: provider-free stochastic/v1 fixture behavior only; this does not validate LLM/provider integrations, research accuracy, or paper-level performance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant