fix(0213): read the USD peg check on _1m, not the repaired tier - #235
Conversation
The peg-applied direction read price_ohlcv_1h, which 0182's repair wrote directly, while the peg values live in price_ohlcv_1m, which that repair never touched. It would have published a confident 0 over 1,564,045 wrong rows, indefinitely — the gap-4 alarm reproducing the very failure it was built to catch. Split rather than repoint. The peg direction gets price_ohlcv_1m over its own 48 h window; the stranded direction stays on price_ohlcv_1h over 7 days, where a zero rolls up faithfully and the 48 h grace still means BE's loss window. The two now carry separate guards and refuse independently, because a _1m scan that matched nothing says nothing about _1h. The peg window is 48 h, not 7 days: _1m is retention-managed at 7 days, so an inherited window would sit on the deletion frontier and move for reasons that are not defects. This is a re-introduction guard, not a historical audit — frozen history is 0212's population. Not deployable yet. The ladder reads that 1.5 M population immediately, which is above every rung, so it would ship permanently breached and get muted. Chain: 0111 -> 0209 -> 0212 -> this. Task moved to blocked.
Four defects in the change itself, and two pre-deploy measurements that cannot be settled from a local machine. Two assertions could not fail. `both_queries_read_final` checked for " FINAL " anywhere, which the shared `FROM assets FINAL` CTE already satisfies, so dropping FINAL from either candle table would have left it green; it now names each table. A sibling assertion looked for the Rust identifier text "STRANDED_GRACE" in generated SQL. One IT tested the window instead of the tier: the _1h-only half of each_direction_only_scans_its_own_tier seeded a 3-day-old row, which the 48 h peg window excludes whichever table is read, so it passed with PEG_TABLE reverted. Seeded at 3 HOUR now. That is the third instance of this task's own founding failure inside one change. "Reads 1,564,045 — above every rung" was the all-history population, not what the shipped 48 h query reads. Corrected in both the module docs and the CDK comment: the deploy block stands on "breached until 0209 stops the writer", not on a magnitude someone would re-size the ladder against. Broken intra-doc links to the deleted LOOKBACK_SECONDS / SanityCounts / SANITY_TABLE, and a "prunes by partition" claim that is false for a window on a non-prefix column. Crate unresolved links 9 -> 6. Recorded as blocking acceptance criteria, not fixed: the probe's 1-minute timeout was never re-measured for a second FINAL scan of the 735M-row _1m, and EmptyScan may fire on a legitimately quiet 48 h window on a leg measured at ~16 rows/day.
Review round — all six findings verified, four fixed hereEach was checked against the code before acting; none was a false positive. Fixed in
Non-vacuity re-verified after the test fixes — reverting Not fixed — promoted to blocking acceptance criteriaBoth are real, and neither can be settled from a local machine. They are now checkboxes on task 0213 alongside the 0212 dependency:
Neither blocks merge — the ladder is already undeployable until 0209/0212 — but both block deploy, which is the state this task is in anyway. |
Two measurements came back and one of them overturned this task's central claim. The ladder does NOT ship permanently breached. With 0212 unlanded it reads scanned 684 / peg_applied 0, because the 1,564,045 peg rows all sit at timestamps <= 2026-08-13 and the 48 h window does not reach them. The "must not deploy before 0212 and 0209" claim was carried over from the unbounded-repoint argument and never re-checked against the query that actually ships — the same class of error as the defect this task fixes: reasoning about a surface other than the one in use. Corrected in the module docs, the CDK comment and the alarm text. What the zero means matters more than the zero. All 684 rows in the window are close_usd = 0 — none peg-valued, none correctly priced — so the leg has been dark since 2026-08-13 and this direction has nothing to judge. A green usd-peg-applied is NOT evidence of correct USD valuation while 0209 stands, and it stays vacuous until enrichment drains far enough forward to reach the window, which is gated on 0111. The alarm text now says so, and points the operator at usd-stranded. Empty-scan concern closed by measurement: the longest stretch with no USDT-quoted _1m row in 30 days is 6.5 h against the 48 h window, 7.4x margin, so the refusal will not page on a healthy leg. Trimmed the peg description back under the 1024-char cap — the synth-time guard from 0204 caught it at 1043 before it could fail mid-deploy. Still open: the peg scan's read cost on prod, which decides whether the probe's 1-minute timeout holds.
EXPLAIN ESTIMATE on prod: the 48 h _1m peg scan reads 1,250,223 rows / 6 parts / 156 marks, against 997,376 / 5 / 125 for the 7-day _1h stranded scan that runs in 41-50 ms. 1.25x, so the probe's 1-minute timeout holds with large margin and needs no change. The concern was well-founded — timestamp is not a primary-key prefix on either table, so a window cannot be ASSUMED to prune — but toYYYYMM partition pruning plus per-part min/max does the work the key order does not, and 156 marks x 8192 corroborates the row estimate. Every acceptance criterion the code can meet is now met and measured, so the task moves blocked -> active. The remaining step is the deploy, which needs BOTH stacks: the probe binary ships from eventbridge-stack and the alarm text from observability-stack, so an Observability-only deploy leaves the ladder blind — 0204's own mistake.
✅ Last measurement in — read cost is a non-issue, task unblocked
1.25×, not orders of magnitude. The probe's 1-minute timeout holds with large margin and needs no change. The concern was well-founded — StatusAll six acceptance criteria on task 0213 are now met and measured. Task moved
Ready to merge. |
Summary
price_ohlcv_1h, which 0182's repair wrote directly, while the peg values live inprice_ohlcv_1m, which that repair never touched. It would have published a confident 0 over 1,564,045 wrong rows, indefinitely — the gap-4 alarm reproducing the failure it was built to catchpeg_appliednow readsprice_ohlcv_1mover its own 48 h window;strandedstays onprice_ohlcv_1hover 7 days, where a zero rolls up faithfully and the 48 h grace still means BE's loss window. Repointing one shared table name would have shipped a permanently-breached ladder, re-inherited the wrong retention reasoning, and degraded the direction that was already correct_1mscan that matched nothing says nothing about_1h, and suppressing a working signal because an unrelated tier failed is the muting failure from the other side_1mis retention-managed at 7 days, so an inherited window would sit on the deletion frontier and move for reasons that are not defects. This is a re-introduction guard, not a historical audit — frozen history is 0212's population_1mand 48 h, and carries the "verify on_1m, never a coarse tier" instruction; all six USD alarms render 856–937 chars against the 1024 cap🔴 Correction — the deploy block in the original PR body was wrong
This PR originally said the ladder must not deploy before 0209/0212 because it would ship permanently breached. Prod measurement on 2026-08-20 falsified that, with 0212 unlanded:
The 1,564,045 peg rows all sit at timestamps ≤ 2026-08-13, entirely outside the 48 h window. The "permanently breached" claim was carried over from the unbounded repoint argument and never re-checked against the query that actually ships — the same class of error as the defect this PR fixes: reasoning about a surface other than the one in use. Corrected in
usd_sanity.rs,observability-stack.tsand the alarm text in589d503.So this is deployable ahead of 0209/0212 — but for the right reason: it arms a guard for when the leg is priced again, and proves nothing about the leg until then.
🔴 A green
usd-peg-appliedis NOT evidence of a healthy legAll 684 rows in the window are
close_usd = 0— none peg-valued, none correctly priced:The leg has been dark since 2026-08-13 (0209), so this direction has nothing to judge and reads 0 for want of input, not want of defects. That is correct behaviour for a re-introduction guard, but it means the metric must never be read alone —
usd-strandedis what carries the darkness, and it is latched. It also stays vacuous longer than 0209 alone: enrichment drains oldest-first behind 0111, so it will price history first and nothing enters this window until the backlog reaches the present. The alarm description now says this and points at the stranded ladder.✅ Empty-scan concern closed by measurement
The review flagged that
scanned == 0on a 48 h window over a sparse leg would fail the invocation and page ops every 15 min on a healthy system. Longest stretch with no USDT-quoted_1mrow in the preceding 30 days: 6.5 hours against 48 h — 7.4× margin. No code change needed; the window must not shrink below ~24 h without re-taking it.⏳ One measurement still open
The probe's 1-minute timeout was sized on the
_1hscan alone (~1.37M rows / ~70 MiB / 41-50 ms) and was never revisited for this secondFINALscan of the 735M-row_1m. The 48 h window does not bound the read —timestampis not a primary-key prefix, so the scan prunes to a whole monthly partition. Needs theProcessed N rowsfigure from prod. Recorded as the one open acceptance criterion on task 0213.Found only by reverting the fix. The fixture was
insert_candle_into(c, PEG_TABLE, …)— expressed in terms of the constant under test, so flippingPEG_TABLEback to_1hmoved the writes too and the tier assertions kept passing. Both tables are now spelled literally in the fixtures.This is the task's founding failure a third time — after the check that read the repaired tier, and 0182 being verified against its own output: a test that cannot distinguish the thing it asserts. The non-vacuity check is the acceptance evidence, not a nicety:
PEG_TABLEprice_ohlcv_1m(fixed)price_ohlcv_1h(reverted)Verified
cargo fmt --all --check·cargo clippy -p rollup-freshness-probe --all-targets --features lambda(0 warnings) ·cargo test --workspace(92 suites, 0 failures) · 19 integration tests against a local ClickHouse pinned to 26.3.10.60 ·nx format:check --all·nx run-many -t lint typecheck·verify:push·nx build infra·make -C infra synth-production