Skip to content

The replay that judges every precision fix now runs daily, prints in report and status, and raises its own regression finding - #62

Open
opencdlee-dotcom wants to merge 2 commits into
agent/fable-precision/integrationfrom
agent/precision-s8/integration
Open

opencdlee-dotcom wants to merge 2 commits into
agent/fable-precision/integrationfrom
agent/precision-s8/integration

Conversation

@opencdlee-dotcom

@opencdlee-dotcom opencdlee-dotcom commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Step S8 (Phase 4) of the precision plan: the operator sees the number the precision fixes are judged on, and a regression raises a finding without anyone asking.

What changed

  • Cached snapshot. _precision_measure runs _backtest_replay(30, reobserve=True) and reduces the dict it already returns (no text scraping) to noise_reopened/noise_total, interrupts (findings the replay routes to interrupt), open_cases, assay_routable_interrupting/_total, assay_predicate_passing/_total, and teaching_evaporation. Rule 17: the replay's own problems travel with the snapshot, and the assay statuses have to partition the lanes. It writes precision.json under STATE_DIR, with the path resolved when it is called. It only reads the store.
  • Teaching evaporation. This counts signal incidents opened in the window whose _incident_identity (or the exact correlation key, when the identity can't generalize) matches an incident that got a benign-positive/false-positive dismissals row earlier. Incidents the machine closed in the same pass that opened them (auto-tolerated, learning-period, allowlisted) are teaching that held, so they don't count. It matches on identity only because _finding_classes isn't on this base; it lands with S6.
  • When it runs. In cmd_scan after the scan lock is released, under its own .precision.lock (_scan_lock now takes a name), which no scan contends on. A scan started during the replay runs normally. It runs at most once per 24 h, using the same cadence rule as _integrity_scan_due, and checks again whether it is due once it holds the lock. A failed attempt gets a timestamp, so it's retried the next day and not on every tick. Any exception costs the snapshot and one run-log line, never the scan. Setting PRECISION_SNAPSHOT_EVERY_SECS <= 0 turns off the scan path; conftest pins it to 0 so ordinary scan tests never run a replay.
  • It checks its own inputs (rule 17). Before the replay, _ledger_rows_rejected counts the in-window custody/intent ledger rows that don't verify under the HMAC key on disk. It asks the way the lookups ask, and never mints a key. The count goes into precision.json (hmac_key, ledger_rows_rejected), and the line starts with inputs unverified: N ledger rows rejected. Why: a copy of ~/.aegis without hmac.key read 143/214 noise re-opened, 165 interrupts and 167 open cases; the same copy with the real key read 127/214, 130 and 132. The earlier line printed both with the same confidence.
  • Regression alarms. check_precision is registered in gather_all next to check_assay. It compares the snapshot with its stored baseline, which is the previous snapshot. If the number of routable lanes that interrupt, or predicate lanes that pass, drops at all, it raises a HIGH self-protection finding, "A detector positive control stopped firing", which interrupts. If noise_reopened or teaching evaporation rises by 3 or more, it raises a MEDIUM "Precision regressed", which goes to the digest. Both are built with finding(). A first snapshot raises nothing.
  • Where it shows. report (every rendering) and status print Precision (30d replay, <age>): a/b judged-noise would re-alert · n interrupts · assay x/y routable + p/q predicate · teaching evaporation e.
  • aegis.py precision [--refresh]. It prints the cached line, the open-case count, the baseline and any regression against it. --refresh measures now.

Not in this PR

  • The plan's weekly-digest lines. This step only covered report and status.
  • Matching evaporation by class. That needs _finding_classes from S6.
  • Recall alarms compare against the previous snapshot, as the step specifies. A drop that persists raises its incident once; the next snapshot does not raise it again, and the incident stays open until someone closes it or it ages out. self:precision:* isn't a state-fingerprint prefix, so the cleared-state closer doesn't resolve it.
  • In watch mode, re-observed signature verdicts stay in the in-memory sigcache and are flushed by the next scan. They are the same verdicts a scan would compute.
  • Watch mode still has a daily gap. Other scanners no longer contend on the lock, but the replay runs inside the cmd_watch process (52–129 s live). For that long the watch loop can't arm kqueue or start an event scan. A change in that window is caught by the next full scan, and the 600 s floor is counted from when the last full scan started, so the replay doesn't push it back. Closing the gap means running the replay in a child process. That is platform-shaped: in install scan mode, launchd and systemd kill children when the job exits. Left for review.
  • The live custody ledger has 2 rows (2026-09-19T07:45:48, homebrew package-managed) that don't verify under the live key. On this Mac the line will read inputs unverified: 2 ledger rows rejected until they age out (180 d). This PR does not touch the live ledger.

🤖 Generated with Claude Code

opencdlee-dotcom and others added 2 commits September 23, 2026 14:53
…mbered to, so report and status now carry its number and a regression raises its own finding

`backtest replay` is the number the precision plan is judged on, and at 3-6
minutes on the live store nothing ran it unattended. This caches it.

- `_precision_measure` runs `_backtest_replay(30, reobserve=True)` and reduces
  its returned dict (no text scraping) to noise_reopened/noise_total,
  interrupts, open_cases, assay routable-interrupting/total and
  predicate-passing/total, plus teaching evaporation. Rule 17: the replay's own
  problems ride along, and the assay statuses must partition the lanes.
- `_teaching_evaporation`: signal incidents opened in the window whose
  `_incident_identity` (exact correlation key where it cannot generalize)
  matches one with an earlier benign-positive/false-positive dismissals row.
  Machine closes at birth (auto-tolerated, learning-period, allowlisted) are
  teaching that held and do not count. Identity only: `_finding_classes` is
  not on this base (it lands with S6).
- `precision.json` in STATE_DIR, resolved at call time. The scan's tail
  refreshes it at most once per 24 h (`_integrity_scan_due`'s rule; a failed
  attempt is stamped so it is retried tomorrow, not every tick). Every failure
  is caught and logged to the run log; the scan's result and heartbeat are
  untouched. `PRECISION_SNAPSHOT_EVERY_SECS <= 0` turns the scan path off, and
  conftest pins it to 0 so ordinary scan tests never run a replay.
- `check_precision` (gather_all, next to the assay sensor) reads the snapshot
  and its stored baseline: any drop in routable-interrupting or
  predicate-passing is HIGH "A detector positive control stopped firing"; a
  rise of 3 or more in noise_reopened or teaching evaporation is MEDIUM
  "Precision regressed" (digest). Both are `self-protection` findings built
  with `finding()`. A first snapshot has no baseline and raises nothing.
- `report` (every rendering) and `status` print one line:
  `Precision (30d replay, <age>): a/b judged-noise would re-alert · n
  interrupts · assay x/y routable + p/q predicate · teaching evaporation e`.
- `aegis.py precision [--refresh]`: prints the cached snapshot, its age and
  its baseline; --refresh measures now.

Tests: tests/test_precision_snapshot.py (throttle, failing replay vs scan,
the regression rules, line format, evaporation on a synthetic store);
"precision" added to the sensor roster in test_sensor_invariants.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s as re-alerting and nothing said so; the snapshot now counts the ledger rows it cannot verify and runs outside the scan lock

The first dry run of `precision --refresh` measured a copy of ~/.aegis that
left hmac.key behind. _hmac_key() minted a fresh key in its place, every
custody and intent row failed its MAC, and every rung built on them was
gone. The same copy, run twice back to back, 2026-09-23:

  no key on disk     143/214 noise re-opened, 165 interrupts, 167 open cases
  the real key       127/214 noise re-opened, 130 interrupts, 132 open cases

The line printed both with the same confidence.

- `_ledger_rows_rejected` counts the in-window custody.jsonl and
  intent.jsonl rows that do not verify under the key on disk, asked the way
  the lookups ask (_custody_carried, _intent_attested: same byte bound, same
  age cutoff, same MAC). It runs BEFORE the replay, because the replay's
  custody lookups mint a key when there is none. It never mints one itself:
  with no key, every in-window row counts as rejected.
- The snapshot records `hmac_key` and `ledger_rows_rejected`, and the line
  leads with `inputs unverified: N ledger rows rejected` (plus "(no HMAC key
  on disk)") ahead of the numbers. The no-key copy now reads "3793 ledger
  rows rejected (no HMAC key on disk)". With the real key it reads "2 ledger
  rows rejected": two custody rows from 2026-09-19T07:45:48 that do not
  verify under the live key either.
- The daily replay no longer holds the one-writer scan lock. cmd_scan runs
  `_precision_tail` after `_scan_lock` is released, and the tail takes its
  own `.precision.lock` (`_scan_lock` gained a `name` argument), which no scan
  contends on. It asks "due" again under that lock, so two scans that finish
  together do not both replay.

Tests: a scan started from inside the replay runs to its own log line and
does not replay again; a held snapshot lock skips; no key or a foreign key
rejects the rows and leads the line; rows past retention are not counted.
The new tests fail on dc49d69 (6 failed) and pass here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant