Skip to content

Evals: per-PR gate is structurally near-circular and pass_rate_strict has no regression gate #68

Description

@JumpTechCode

Summary

The per-PR eval gate replays committed cassettes against a baseline that was itself generated by replaying those same cassettes, so per-case diffs are ~0 by construction — the paired bootstrap mainly fires on scoring/runner code changes, not model-quality drift (drift detection is consciously offloaded to the nightly live-API job). Compounding it, the README headline number pass_rate_strict has no PR-level regression gate at all.

Evidence

  • Makefileevals-baseline (:96) builds evals/baselines/main.json by replaying sentinel/evals/cassettes; evals (:88) + evals-compare (:99, compare-to-baseline) replay the same cassettes against that baseline.
  • .github/workflows/ci.yml:112-199 (evals-gate, SENTINEL_EVAL_CASSETTE_MODE: replay) vs nightly-evals.yml (--live, the actual drift sensor).
  • sentinel/evals/cli.py:973 — the regression verdict iterates only category_match, hypothesis_cosine, action_coverage, evidence_quality. pass_rate_strict lives under headline (report.py:78) and is never read by _compute_regression_verdict (cli.py:951-1050).

Notes

Cassette-replay determinism is a deliberate, documented design choice — this is not "replay is wrong." It's: (a) make the README's "blocks regressions in CI" honest about what the PR gate vs nightly job each catch, and (b) give pass_rate_strict an explicit regression gate.

Fix

  • Add a pass_rate_strict regression gate (nightly at minimum).
  • Tighten README wording to distinguish per-PR (scoring-code / structural) from nightly (model drift).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTest coverage gaps or improvements

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions