Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: CI

on:
push:
pull_request:

jobs:
test:
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
- run: python -m pip install -e ".[dev]"
- run: python -m pytest
- name: Install media verifier
run: sudo apt-get update && sudo apt-get install -y ffmpeg
- name: Verify source-pinned demo
run: python scripts/verify_seed_integrity_demo.py
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@ We identify *loop-action bias* as a mechanistic failure mode of outcome-only cre
| Real ALFWorld, Haiku (scorer diff.) | 13.3% | **34.8%** | = branch-aware |
| Real ALFWorld, Qwen-7B (scorer diff.) | 3.4% | **31.8%** | = branch-aware |

## Actual workflow demo

![CreditLab stochastic seed integrity demo](media/demo/seed-integrity.gif)

[Watch the MP4](media/demo/seed-integrity.mp4) · [Inspect the capture receipt](media/demo/capture.json)

This provider-free demo executes the checked-in `stochastic/v1` manifest through the real environment adapter. It shows that one seed is repeatable while seeds 0 and 1 produce distinct stochastic paths. The capture is pinned to its exact runtime source commit and guarded by `scripts/verify_seed_integrity_demo.py`. It does not exercise a language model or validate the paper's performance claims.

## Reproducing the Results

### Prerequisites
Expand Down
133 changes: 133 additions & 0 deletions media/demo/capture.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
{
"artifacts": {
"seed-integrity.gif": {
"bytes": 252820,
"sha256": "d3eb157c762bb3c4677ea37fcb66916e21837ed5b48c9a95fe51cf5aafbf00d3"
},
"seed-integrity.mp4": {
"bytes": 164354,
"sha256": "45044b69f3d3d0650361d4c1e515d882bc2273106fb03ce7b9b4aa9ab05e2988"
}
},
"benchmark": "stochastic/v1",
"cross_seed_paths_distinct": true,
"same_seed_replay_matches": true,
"schema_version": 1,
"scope": "provider-free stochastic manifest fixture; no model or paper-performance claim",
"seeds": [
0,
1
],
"source_blobs": {
"benchmarks/stochastic/v1/tasks.yaml": "e2826242df4cc340b67eba5c05b700059e5a806c",
"scripts/generate_seed_integrity_demo.py": "81e1cefd3b0d5f0f0f4da68ce6d0e31cc5d18a8a",
"src/creditlab/analysis/sweep.py": "1bf34600f9c4dd140702bc3370ac903dd4383a34",
"src/creditlab/core/registry.py": "0bd254dd96460beea41035fb2c0cf9eb8d0f3b72",
"src/creditlab/envs/manifest_graph.py": "6d7f4b70422001e97ab84869df060d4e24f32b24"
},
"source_commit": "b39ab74c07cc56f30e96fd8d955170e012167356",
"source_tree": "ffc8450ecf5b4a271e3ca3fc04ae1f45f86aeeb9",
"task_id": "stoch_gate_train_01",
"traces": {
"0": [
{
"action": "unlock_gate",
"from": "start",
"observation": "The lock resists. The gate stays shut.",
"terminal": false,
"to": "start",
"turn": 1
},
{
"action": "unlock_gate",
"from": "start",
"observation": "The lock resists. The gate stays shut.",
"terminal": false,
"to": "start",
"turn": 2
},
{
"action": "unlock_gate",
"from": "start",
"observation": "The first gate opens.",
"terminal": false,
"to": "gate_2",
"turn": 3
},
{
"action": "unlock_gate",
"from": "gate_2",
"observation": "The second gate opens.",
"terminal": false,
"to": "gate_3",
"turn": 4
},
{
"action": "unlock_gate",
"from": "gate_3",
"observation": "The third gate opens.",
"terminal": false,
"to": "gate_4",
"turn": 5
},
{
"action": "unlock_gate",
"from": "gate_4",
"observation": "The final gate opens. Task complete.",
"terminal": true,
"to": "success",
"turn": 6
}
],
"1": [
{
"action": "unlock_gate",
"from": "start",
"observation": "The first gate opens.",
"terminal": false,
"to": "gate_2",
"turn": 1
},
{
"action": "unlock_gate",
"from": "gate_2",
"observation": "The lock resists again.",
"terminal": false,
"to": "gate_2",
"turn": 2
},
{
"action": "unlock_gate",
"from": "gate_2",
"observation": "The lock resists again.",
"terminal": false,
"to": "gate_2",
"turn": 3
},
{
"action": "unlock_gate",
"from": "gate_2",
"observation": "The second gate opens.",
"terminal": false,
"to": "gate_3",
"turn": 4
},
{
"action": "unlock_gate",
"from": "gate_3",
"observation": "The third gate opens.",
"terminal": false,
"to": "gate_4",
"turn": 5
},
{
"action": "unlock_gate",
"from": "gate_4",
"observation": "The final gate opens. Task complete.",
"terminal": true,
"to": "success",
"turn": 6
}
]
}
}
Binary file added media/demo/seed-integrity.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added media/demo/seed-integrity.mp4
Binary file not shown.
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ wandb = [
paper = [
"matplotlib>=3.8,<4",
]
demo = [
"pillow>=10,<13",
]
alfworld = [
"alfworld==0.4.2",
]
Expand Down
Loading
Loading