chore(0204): complete and archive — four ops alarms live and verified on prod

[karczuRF]

Closed

All four gaps from the 2026-08-13 outage are deployed on production, and the acceptance criterion that could not be met before a deploy — alarms verified by inducing the condition, not by reading the CDK — is now met for all of them.

gap	verification
1 — CH free space	Bound raised to 30% against a measured 24.43% free. Fired 13:10 with real values in its StateReason; restored to 20, OK at 13:18
2 — DLQ ladder	1 → 10 → 50 dummy messages. Rungs fired 12:17 / 12:27 / 12:34; rungs 1 and 2 never re-notified. All three cleared 13:14 after a purge
3 — MV drift	Running on schedule, reading clean on prod (0/0, 38 visible objects, "all in sync") — the first time 0142's check has ever run
4 — USD correctness	Induced against live ClickHouse pre-deploy; publishing on prod

Gap 2's verification is the interesting one: rung 1 watched the queue go 1 → 10 → 50 and said nothing after its first message. That is exactly what it did on 2026-08-13 while the queue climbed to 91, and why nobody reading Slack could tell one dropped ledger from an outage. The defect was reproduced deliberately, next to its fix.

Two deploys failed first, and both are recorded

1. CloudWatch caps AlarmDescription at 1024 characters, and synth cannot catch it. Valid CloudFormation, green CI, cdk diff clean — rejected by the API mid-deploy after some alarms had already been created. Now guarded by assertAlarmDescriptionsFitCloudWatch() at synth. ⚠️ The text that blew the limit was text the same day's investigation had falsified.

2. 🔴 "Deploy Observability only" shipped the alarms without the probe that feeds them. Ten of thirteen went live blind, and seven settled to OK on no data at all — the console read "disk is watched" while nothing watched. That is this task's founding failure, produced by its own deploy plan.

⚠️ The three MvDrift* alarms held at INSUFFICIENT_DATA for 43 minutes instead, because the earlier code review moved them to treatMissingData: MISSING. Same deploy, same absent data — an accidental controlled experiment for a change argued on paper the day before.

Fixing that required deploying eventbridge-stack, so CleanupRule is now enabled: false in code (it asserted ENABLED while live was DISABLED, meaning every deploy silently re-enabled it) backed by assertCleanupRuleStaysDisabled(), a synth-time throw. All ten Lambda assets were also rebuilt — every local bootstrap was a 2026-08-13 build predating the 0172 oracle deploy, so deploying as-is would have rolled back live fixes.

Measured along the way

• Probe runs in 653 ms of a 60 s budget, 48 MB / 256 MB, with four checks instead of one — closes the open "1-minute config never revisited" question, with no eventbridge-stack change needed.
• An emptied DLQ takes ~25 minutes to clear its alarms. get-queue-attributes reads 0 immediately; the CloudWatch metric kept publishing 50. On a real redrive the ladder keeps firing well after the operator has fixed things.
• The disk bound stays at 20%. 15% was proposed and reversed once measured: the ~150 GiB incident sits between the two thresholds (20% fires at 78 GiB consumed, 15% needs 166), so five percentage points is the entire margin between catching that incident and missing it.

Spawns 0214

prices-production-enrichment-errors has been latched in ALARM since 2026-07-27 — 24 days — and was found only because verification listed every alarm rather than only this task's own.

⚠️ That prices gap 3's design decision 1, which accepted a latched drift alarm because latching costs "somebody may forget" rather than "we are blind to an escalation". Forgetting cost 24 days here, and gap 3 ships two alarms designed to latch.

Still open

0209 (the USDT pivot has never priced a _1m row), 0212 (1.56M peg-valued rows), 0213 (the peg check reads the repaired tier), 0214 (the latched alarm). None block this task — 0204's scope is that the alarms exist and are verified, and that the data is currently wrong is what they are for.