Skip to content

fix(0213): read the USD peg check on _1m, not the repaired tier - #235

Merged
karczuRF merged 4 commits into
developfrom
fix/0213_gap4-peg-check-reads-the-repaired-tier-not-the-source
Aug 20, 2026
Merged

fix(0213): read the USD peg check on _1m, not the repaired tier#235
karczuRF merged 4 commits into
developfrom
fix/0213_gap4-peg-check-reads-the-repaired-tier-not-the-source

Conversation

@karczuRF

@karczuRF karczuRF commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • The peg-applied direction was structurally blind. It read price_ohlcv_1h, which 0182's repair wrote directly, while the peg values live in price_ohlcv_1m, which that repair never touched. It would have published a confident 0 over 1,564,045 wrong rows, indefinitely — the gap-4 alarm reproducing the failure it was built to catch
  • Split, not repoint. peg_applied now reads price_ohlcv_1m over its own 48 h window; stranded stays on price_ohlcv_1h over 7 days, where a zero rolls up faithfully and the 48 h grace still means BE's loss window. Repointing one shared table name would have shipped a permanently-breached ladder, re-inherited the wrong retention reasoning, and degraded the direction that was already correct
  • The two directions now guard, publish and refuse independently — a _1m scan that matched nothing says nothing about _1h, and suppressing a working signal because an unrelated tier failed is the muting failure from the other side
  • The peg window is 48 h, deliberately not the stranded direction's 7 days. _1m is retention-managed at 7 days, so an inherited window would sit on the deletion frontier and move for reasons that are not defects. This is a re-introduction guard, not a historical audit — frozen history is 0212's population
  • Peg alarm description now names _1m and 48 h, and carries the "verify on _1m, never a coarse tier" instruction; all six USD alarms render 856–937 chars against the 1024 cap

🔴 Correction — the deploy block in the original PR body was wrong

This PR originally said the ladder must not deploy before 0209/0212 because it would ship permanently breached. Prod measurement on 2026-08-20 falsified that, with 0212 unlanded:

resolved_legs │ peg_applied │ scanned
            1 │           0 │     684

The 1,564,045 peg rows all sit at timestamps ≤ 2026-08-13, entirely outside the 48 h window. The "permanently breached" claim was carried over from the unbounded repoint argument and never re-checked against the query that actually ships — the same class of error as the defect this PR fixes: reasoning about a surface other than the one in use. Corrected in usd_sanity.rs, observability-stack.ts and the alarm text in 589d503.

So this is deployable ahead of 0209/0212 — but for the right reason: it arms a guard for when the leg is priced again, and proves nothing about the leg until then.

🔴 A green usd-peg-applied is NOT evidence of a healthy leg

All 684 rows in the window are close_usd = 0 — none peg-valued, none correctly priced:

unpriced │ peg_valued │ properly_priced │ total
     684 │          0 │               0 │   684

The leg has been dark since 2026-08-13 (0209), so this direction has nothing to judge and reads 0 for want of input, not want of defects. That is correct behaviour for a re-introduction guard, but it means the metric must never be read alone — usd-stranded is what carries the darkness, and it is latched. It also stays vacuous longer than 0209 alone: enrichment drains oldest-first behind 0111, so it will price history first and nothing enters this window until the backlog reaches the present. The alarm description now says this and points at the stranded ladder.

✅ Empty-scan concern closed by measurement

The review flagged that scanned == 0 on a 48 h window over a sparse leg would fail the invocation and page ops every 15 min on a healthy system. Longest stretch with no USDT-quoted _1m row in the preceding 30 days: 6.5 hours against 48 h — 7.4× margin. No code change needed; the window must not shrink below ~24 h without re-taking it.

⏳ One measurement still open

The probe's 1-minute timeout was sized on the _1h scan alone (~1.37M rows / ~70 MiB / 41-50 ms) and was never revisited for this second FINAL scan of the 735M-row _1m. The 48 h window does not bound the readtimestamp is not a primary-key prefix, so the scan prunes to a whole monthly partition. Needs the Processed N rows figure from prod. Recorded as the one open acceptance criterion on task 0213.

⚠️ The first version of these tests was vacuous

Found only by reverting the fix. The fixture was insert_candle_into(c, PEG_TABLE, …) — expressed in terms of the constant under test, so flipping PEG_TABLE back to _1h moved the writes too and the tier assertions kept passing. Both tables are now spelled literally in the fixtures.

This is the task's founding failure a third time — after the check that read the repaired tier, and 0182 being verified against its own output: a test that cannot distinguish the thing it asserts. The non-vacuity check is the acceptance evidence, not a nicety:

PEG_TABLE unit integration
price_ohlcv_1m (fixed) 51 pass 19 pass
price_ohlcv_1h (reverted) 2 fail 6 fail

Verified

cargo fmt --all --check · cargo clippy -p rollup-freshness-probe --all-targets --features lambda (0 warnings) · cargo test --workspace (92 suites, 0 failures) · 19 integration tests against a local ClickHouse pinned to 26.3.10.60 · nx format:check --all · nx run-many -t lint typecheck · verify:push · nx build infra · make -C infra synth-production

The peg-applied direction read price_ohlcv_1h, which 0182's repair wrote
directly, while the peg values live in price_ohlcv_1m, which that repair
never touched. It would have published a confident 0 over 1,564,045 wrong
rows, indefinitely — the gap-4 alarm reproducing the very failure it was
built to catch.

Split rather than repoint. The peg direction gets price_ohlcv_1m over its
own 48 h window; the stranded direction stays on price_ohlcv_1h over 7
days, where a zero rolls up faithfully and the 48 h grace still means BE's
loss window. The two now carry separate guards and refuse independently,
because a _1m scan that matched nothing says nothing about _1h.

The peg window is 48 h, not 7 days: _1m is retention-managed at 7 days, so
an inherited window would sit on the deletion frontier and move for reasons
that are not defects. This is a re-introduction guard, not a historical
audit — frozen history is 0212's population.

Not deployable yet. The ladder reads that 1.5 M population immediately,
which is above every rung, so it would ship permanently breached and get
muted. Chain: 0111 -> 0209 -> 0212 -> this. Task moved to blocked.
Four defects in the change itself, and two pre-deploy measurements that
cannot be settled from a local machine.

Two assertions could not fail. `both_queries_read_final` checked for
" FINAL " anywhere, which the shared `FROM assets FINAL` CTE already
satisfies, so dropping FINAL from either candle table would have left it
green; it now names each table. A sibling assertion looked for the Rust
identifier text "STRANDED_GRACE" in generated SQL.

One IT tested the window instead of the tier: the _1h-only half of
each_direction_only_scans_its_own_tier seeded a 3-day-old row, which the
48 h peg window excludes whichever table is read, so it passed with
PEG_TABLE reverted. Seeded at 3 HOUR now. That is the third instance of
this task's own founding failure inside one change.

"Reads 1,564,045 — above every rung" was the all-history population, not
what the shipped 48 h query reads. Corrected in both the module docs and
the CDK comment: the deploy block stands on "breached until 0209 stops the
writer", not on a magnitude someone would re-size the ladder against.

Broken intra-doc links to the deleted LOOKBACK_SECONDS / SanityCounts /
SANITY_TABLE, and a "prunes by partition" claim that is false for a window
on a non-prefix column. Crate unresolved links 9 -> 6.

Recorded as blocking acceptance criteria, not fixed: the probe's 1-minute
timeout was never re-measured for a second FINAL scan of the 735M-row _1m,
and EmptyScan may fire on a legitimately quiet 48 h window on a leg
measured at ~16 rows/day.
@karczuRF

Copy link
Copy Markdown
Collaborator Author

Review round — all six findings verified, four fixed here

Each was checked against the code before acting; none was a false positive.

Fixed in f1935ea:

  • Two assertions could not fail. both_queries_read_final checked contains(" FINAL "), which the shared FROM assets FINAL CTE already satisfies — deleting FINAL from either candle table would have left it green. It now names each table. A sibling assertion searched generated SQL for the Rust identifier text "STRANDED_GRACE".
  • One IT tested the window, not the tier. The _1h-only half of each_direction_only_scans_its_own_tier seeded a 3-day-old row, which the 48 h peg window excludes whichever table is read — so it passed with PEG_TABLE reverted. Seeded at 3 HOUR now. ⚠️ Together with the fixture coupling already documented, that is the third instance of this task's own founding failure inside one change: a test that cannot distinguish the thing it asserts.
  • "Reads 1,564,045 — above every rung" was wrong for the query that ships. That is the all-history population; bounded to 48 h the reading is the recent arrival rate — tens of rows, clearing rung 1 and nothing above it. Corrected in the module docs and the CDK comment, because it is precisely the number someone would use to re-size the escalation ladder.
  • Broken intra-doc links to the deleted LOOKBACK_SECONDS / SanityCounts / SANITY_TABLE, plus a "prunes by partition" claim that is false for a window on a non-prefix column. Crate unresolved-link count 9 → 6.

Non-vacuity re-verified after the test fixes — reverting PEG_TABLE to price_ohlcv_1h still fails 2 unit + 6 integration tests.

Not fixed — promoted to blocking acceptance criteria

Both are real, and neither can be settled from a local machine. They are now checkboxes on task 0213 alongside the 0212 dependency:

  1. The probe's 1-minute timeout was never re-measured for this second scan. It was sized on _1h alone (~1.37M rows / ~70 MiB / 41-50 ms); this adds a FINAL scan of the 735M-row _1m. The 48 h window does not bound the read — both tables are PARTITION BY toYYYYMM(timestamp) with ORDER BY (asset_id, quote_asset_id, source, timestamp), so timestamp is not a primary-key prefix and the scan prunes to a whole monthly partition. A timeout is not a Rust Err: it kills the invocation with nothing published, and step 4 (MV drift) never runs.
  2. EmptyScan may fire on a legitimately quiet window. The guard was calibrated for 7 days on _1h; it now applies the same scanned == 0 refusal to 48 h on a leg measured at ~16 _1m rows/day. A quiet spell makes zero legitimate, and the refusal then pages ops via the probe's own -errors alarm every 15 minutes on a healthy system — the muting failure again. Needs the daily minimum over ≥30 days; if zero is reachable, the candidate discriminator is refusing only when the stranded direction also scanned nothing, which separates a quiet window from a renumbered identity without inventing a threshold.

Neither blocks merge — the ladder is already undeployable until 0209/0212 — but both block deploy, which is the state this task is in anyway.

Two measurements came back and one of them overturned this task's central
claim.

The ladder does NOT ship permanently breached. With 0212 unlanded it reads
scanned 684 / peg_applied 0, because the 1,564,045 peg rows all sit at
timestamps <= 2026-08-13 and the 48 h window does not reach them. The
"must not deploy before 0212 and 0209" claim was carried over from the
unbounded-repoint argument and never re-checked against the query that
actually ships — the same class of error as the defect this task fixes:
reasoning about a surface other than the one in use. Corrected in the
module docs, the CDK comment and the alarm text.

What the zero means matters more than the zero. All 684 rows in the window
are close_usd = 0 — none peg-valued, none correctly priced — so the leg has
been dark since 2026-08-13 and this direction has nothing to judge. A green
usd-peg-applied is NOT evidence of correct USD valuation while 0209 stands,
and it stays vacuous until enrichment drains far enough forward to reach
the window, which is gated on 0111. The alarm text now says so, and points
the operator at usd-stranded.

Empty-scan concern closed by measurement: the longest stretch with no
USDT-quoted _1m row in 30 days is 6.5 h against the 48 h window, 7.4x
margin, so the refusal will not page on a healthy leg.

Trimmed the peg description back under the 1024-char cap — the synth-time
guard from 0204 caught it at 1043 before it could fail mid-deploy.

Still open: the peg scan's read cost on prod, which decides whether the
probe's 1-minute timeout holds.
EXPLAIN ESTIMATE on prod: the 48 h _1m peg scan reads 1,250,223 rows /
6 parts / 156 marks, against 997,376 / 5 / 125 for the 7-day _1h stranded
scan that runs in 41-50 ms. 1.25x, so the probe's 1-minute timeout holds
with large margin and needs no change.

The concern was well-founded — timestamp is not a primary-key prefix on
either table, so a window cannot be ASSUMED to prune — but toYYYYMM
partition pruning plus per-part min/max does the work the key order does
not, and 156 marks x 8192 corroborates the row estimate.

Every acceptance criterion the code can meet is now met and measured, so
the task moves blocked -> active. The remaining step is the deploy, which
needs BOTH stacks: the probe binary ships from eventbridge-stack and the
alarm text from observability-stack, so an Observability-only deploy leaves
the ladder blind — 0204's own mistake.
@karczuRF

Copy link
Copy Markdown
Collaborator Author

✅ Last measurement in — read cost is a non-issue, task unblocked

EXPLAIN ESTIMATE on prod (no system.* grant needed, which prices_writer doesn't have):

scan parts rows marks
new — 48 h _1m peg 6 1,250,223 156
baseline — 7 d _1h stranded (41-50 ms) 5 997,376 125

1.25×, not orders of magnitude. The probe's 1-minute timeout holds with large margin and needs no change.

The concern was well-founded — timestamp is not a primary-key prefix on either table, so a window cannot be assumed to prune — but toYYYYMM partition pruning plus per-part min/max does the work the key order doesn't, and 156 marks × 8192 ≈ 1.28M corroborates the row estimate.

Status

All six acceptance criteria on task 0213 are now met and measured. Task moved blocked/active/; the only remaining step is the deploy, which is an operator action rather than a code gate.

⚠️ When it does deploy, it needs BOTH stacks. The probe binary ships from eventbridge-stack and the alarm text from observability-stack — an Observability-only deploy leaves the ladder blind, which is 0204's own documented mistake. Rebuild and strings the lambda assets first (0141).

Ready to merge.

@karczuRF
karczuRF merged commit 8915c11 into develop Aug 20, 2026
3 checks passed
@karczuRF
karczuRF deleted the fix/0213_gap4-peg-check-reads-the-repaired-tier-not-the-source branch August 20, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant