docs(0182): record route (c), and settle the snapshot count at two - #222
Open
karczuRF wants to merge 3 commits into
Open
docs(0182): record route (c), and settle the snapshot count at two#222karczuRF wants to merge 3 commits into
karczuRF wants to merge 3 commits into
Conversation
The completion order agreed for the repair chain already settled this — 0201, then 0182, then 0172 close, sequenced — but the task file still said routes a/b/c were undecided under "this now blocks the run". A fresh session reads the task file, not the session note, so on paper the run was still blocked. Route (a) was rejected because a combined run swamps the rows_reset ~= rows_enriched check, ~32M against ~357k, and that check is the only signal that catches values zeroed and never recomputed. Route (b) is the better tool but costs a PR before any prod run and leaves 0201 outstanding regardless. Records what each of the two passes runs, that the FREEZE span is 67 months x 5 tables, and the one operational question left for the admin: one snapshot before pass 1, or a second between the passes so they are independently revertible.
Two snapshots, so pass 2 is revertible without discarding pass 1's ~32M recovered rows. Three consequences the decision does not state on its own. The names must differ. The freeze script's `already-frozen` branch keeps a pre-existing snapshot rather than overwriting it — correct, but it makes a second freeze under the same name a silent no-op that reports success while leaving one rollback point. `repair_0182_pre_` then `repair_0182_mid_`. The second snapshot sits between the passes, so interleaving them per table would need the admin back five times across 10-15 h, and `prices_writer` cannot FREEZE. Ordering is now the hybrid — `_1h` alone as the review gate, then the other four batched — three admin windows, tabled. Disk measured rather than estimated: 16.82 GiB in span across the five tables, 799 active parts, `_1h` alone 9.56 GiB. A FREEZE hardlinks and so costs nothing when taken; the cost accrues as the run's version+1 inserts merge and supersede the pinned parts, ceilinged at what was frozen. Both snapshots peak ~33.6 GiB plus ~4-6 GiB of new parts — ~9% of the 430.6 GiB free, but a ~68% increase on our own 58.93 GiB footprint. Adds the cleanup step and an acceptance criterion for it. Nothing expires a snapshot and neither ClickHouse nor the tool removes one, so ~33.6 GiB stays pinned on a volume BE own 96% of until an admin unfreezes it — the same shape as the incident this whole chain started from. Still no FREEZE taken and no partition written.
…says Measured during 0182's pass 1. The 2026-08-13 classification is right that ~32M rows are XLM-quoted and fillable and wrong about when: it attributes them to 2021-02..2024-01, and almost none are below 2022-04. Months 202110-202203 hold 0-13 XLM-quoted candidates each and every one ran `enriched 0` with the "no USD reference (exotic quotes)" warning. 202204 enriched 761,735 of 1,531,768 in 79 batches, ~67 s. The boundary is sharp and belongs to the data, not the tool — those earlier candidates are exotic-quoted and are the permanent no_reference floor rather than missing data. Records the per-month breakdown by quote leg, and flags that 202204+ in it is post-drain so the small numbers there are not read as the pre-run population. The count may still be near 32M; this settles the span only, and the honest count comes from pass 1's summary. Also records the two explanations falsified on the way, because both are plausible enough to be re-derived: the XLM/USDC reference DOES reach back to 2021-02-01, and its early candles do NOT have zero volume — every month 202102-202207 has ~720 candles, all with volume, non-NULL usd_ref. That last check settles 0182's own epoch as a side effect. USDT/USDC is dense and non-NULL from 202102, first candle 2021-02-07 19:00, so 1612656000 cannot zero rows into a reference hole. The strand risk the task warns about is closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The completion order agreed for the repair chain already settled this — 0201 → 0182 → 0172 close, sequenced — but the task file still listed routes a/b/c as undecided under "⛔ This now blocks the run … nothing below can be scheduled until it is settled". A fresh session reads the task file, not the session note, so on paper the run was still blocked. This closes that gap.
rows_reset ≈ rows_enrichedcheck ~32M against ~357k, and that check is the only signal that catches values zeroed and never recomputed — the worst outcome this task can produce. Route (b) is the better tool but costs a PR before any prod run and leaves 0201 outstanding anyway.coarse-repairwithout--reset-*, draining the ~32M fillable pre-Soroban zeros until the table is at its floor; pass 2 = 0182, with--reset-quote-asset-id 111 --reset-not-before 1612656000, at which pointrows_reset ≈ rows_enrichedis meaningful again. Also notes this keeps the "run reset mode once per table" rule intact, since pass 1 is not a reset invocation.Still blocking the run (unchanged by this PR)
prices_writercannot and cannot be granted it.Note: 0201 is the operator's own task — this records the sequencing only, it does not claim or start that work.
Added 2026-08-18 — the snapshot count, settled at two
The one operational question this PR originally left open ("one snapshot or
two") is now answered: two, so pass 2 is revertible without discarding pass
1's ~32M recovered rows. Three things follow that the decision does not state on
its own.
already-frozenbranch keeps apre-existing snapshot rather than overwriting it — correct behaviour, but it
makes a second freeze under the same name a silent no-op: it reports
success while leaving you one rollback point.
repair_0182_pre_…thenrepair_0182_mid_….between the passes, so a per-table interleave needs the admin back five times
across 10-15 h — and
prices_writercannotFREEZE. Ordering is now thehybrid:
_1halone as the review gate, then the other four batched. Threeadmin windows, tabled in the task file.
five tables (799 active parts),
_1halone 9.56 GiB. AFREEZEhardlinks, soit costs nothing when taken; the cost accrues as the run's
version + 1inserts merge and supersede the pinned parts, ceilinged at what was frozen.
Both snapshots peak at ~33.6 GiB plus ~4-6 GiB of new parts — ~9% of the 430.6
GiB free, though a ~68% increase on our own 58.93 GiB footprint.
Also adds a cleanup section and an acceptance criterion for it. Nothing
expires a snapshot, and neither ClickHouse nor the tool removes one, so ~33.6
GiB stays pinned on a volume BE own 96% of until an admin unfreezes it — the
same shape as the incident this whole chain started from.
Still no FREEZE taken and no partition written.