Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions infra/src/lib/stacks/observability-stack.ts
Original file line number Diff line number Diff line change
Expand Up @@ -828,11 +828,33 @@ export class ObservabilityStack extends cdk.Stack {
// path back to OK latches permanently on first breach and is then silent
// for every later incident.
//
// The probe scopes both counts to the USDT quote leg and bounds them to a
// rolling 7-day window. Both are load-bearing — exotic-quoted zeros are by
// The probe scopes both counts to the USDT quote leg and bounds each to a
// rolling window. Both are load-bearing — exotic-quoted zeros are by
// design (~74M rows), and an unbounded scan every 15 min is task 0111's
// outage wearing a health check's clothes. See
// packages/rollup-freshness-probe/src/usd_sanity.rs.
//
// ⚠️ THE TWO DIRECTIONS READ DIFFERENT TIERS AND DIFFERENT WINDOWS (task
// 0213). `peg_applied` reads `price_ohlcv_1m` over 48 h — the tier
// enrichment WRITES; `stranded` reads `price_ohlcv_1h` over 7 days, where a
// zero rolls up faithfully and the 48 h grace matches BE's loss window.
// Reading one tier for both is what made the peg direction publish a
// confident 0 over 1,564,045 wrong rows.
//
// 🔴 A GREEN `usd-peg-applied` IS NOT EVIDENCE THE USDT LEG IS HEALTHY.
// Measured on prod 2026-08-20 inside the 48 h window: 684 rows scanned, ALL
// 684 at close_usd = 0, zero peg-valued and zero correctly priced. The leg
// has been unpriced since 2026-08-13 (task 0209), so this direction has
// nothing to judge and reads 0 for want of input. `usd-stranded` is what
// carries that condition, and it is latched. Never read one without the
// other.
//
// ⚠️ An earlier version of this comment said the peg ladder must not be
// deployed before 0212/0209 because it would ship permanently breached.
// The measurement above falsifies that — the 1.5 M peg population sits at
// timestamps <= 2026-08-13, entirely outside the window. That figure
// applies to an unbounded repoint, not to the query that ships; do not
// re-size these rungs against it.
const usdSanityRungs = (
metricName: string,
idPrefix: string,
Expand Down Expand Up @@ -869,7 +891,7 @@ export class ObservabilityStack extends cdk.Stack {
'UsdPegAppliedAlarmCount',
'usd-peg-applied',
(count) =>
`${count} or more USDT-quoted candles written in the last 7 days carry a close_usd within 2% of their close — valued as if USDT were still pegged at $1. USDT depegged in June 2022 and trades at ~0.13-0.15 (task 0172), so these values are roughly 7.4x too high. Something is applying the peg path to the USDT leg again: check the enrichment tiers (USDT must be a PIVOT reference, never a peg member) and prices.oracle_prices for rows mis-attributed to the USDT identity — that is how tasks 0196 and 0168 reintroduced this WITHOUT touching the writer, so a green writer test proves nothing here. Task 0182 corrected 567,760 such candles; find what is writing them before re-running any repair. Rungs are operator-tunable via config.opsAlarms.usdSanityEscalationCounts.`,
`${count} or more USDT-quoted price_ohlcv_1m candles written in the last 48 h carry a close_usd within 2% of their close — valued as if USDT were still pegged at $1. USDT depegged in June 2022 and trades at ~0.13-0.15 (task 0172), so these values are roughly 7.4x too high. Something is applying the peg path to the USDT leg again: check the enrichment tiers (USDT must be a PIVOT reference, never a peg member) and prices.oracle_prices for rows mis-attributed to the USDT identity — that is how tasks 0196 and 0168 reintroduced this WITHOUT touching the writer. VERIFY ON _1m, NEVER a coarse tier: task 0182 repaired the coarse tables directly, so _1h reads clean over a broken _1m (tasks 0212, 0213). Find what is writing them before re-running any repair. NOTE this metric reads 0 whenever the leg is unpriced, which it has been since 2026-08-13 - a green ladder is NOT proof of correct USD valuation, check prices-{env}-usd-stranded too. Rungs tunable via config.opsAlarms.usdSanityEscalationCounts.`,
);

this.usdStrandedAlarms = usdSanityRungs(
Expand Down
Loading
Loading