Skip to content

docs(0182): record route (c), and settle the snapshot count at two - #222

Open
karczuRF wants to merge 3 commits into
developfrom
docs/0182_record-route-c-mechanism
Open

docs(0182): record route (c), and settle the snapshot count at two#222
karczuRF wants to merge 3 commits into
developfrom
docs/0182_record-route-c-mechanism

Conversation

@karczuRF

@karczuRF karczuRF commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

The completion order agreed for the repair chain already settled this — 0201 → 0182 → 0172 close, sequenced — but the task file still listed routes a/b/c as undecided under "⛔ This now blocks the run … nothing below can be scheduled until it is settled". A fresh session reads the task file, not the session note, so on paper the run was still blocked. This closes that gap.

  • Route (c) recorded as decided, with why it beat (a) and (b): a combined run swamps the rows_reset ≈ rows_enriched check ~32M against ~357k, and that check is the only signal that catches values zeroed and never recomputed — the worst outcome this task can produce. Route (b) is the better tool but costs a PR before any prod run and leaves 0201 outstanding anyway.
  • What each pass actually runs: pass 1 = 0201, coarse-repair without --reset-*, draining the ~32M fillable pre-Soroban zeros until the table is at its floor; pass 2 = 0182, with --reset-quote-asset-id 111 --reset-not-before 1612656000, at which point rows_reset ≈ rows_enriched is meaningful again. Also notes this keeps the "run reset mode once per table" rule intact, since pass 1 is not a reset invocation.
  • FREEZE span pinned at 67 months × 5 tables, which is what the admin needs in order to take the snapshot.
  • Checkboxes, the "remaining, in order" list and the acceptance criterion all updated to match; ⛔ markers removed from the mechanism item only.

Still blocking the run (unchanged by this PR)

  • A ClickHouse admin must take the FREEZEprices_writer cannot and cannot be granted it.
  • ⛔ One operational question left open deliberately, for the operator to settle with the admin: one snapshot before pass 1, or two (a second between the passes). One is cheaper; rolling pass 2 back then also discards pass 1's ~32M recovered rows. Two makes the passes independently revertible.

Note: 0201 is the operator's own task — this records the sequencing only, it does not claim or start that work.


Added 2026-08-18 — the snapshot count, settled at two

The one operational question this PR originally left open ("one snapshot or
two") is now answered: two, so pass 2 is revertible without discarding pass
1's ~32M recovered rows. Three things follow that the decision does not state on
its own.

  • The names must differ. The freeze script's already-frozen branch keeps a
    pre-existing snapshot rather than overwriting it — correct behaviour, but it
    makes a second freeze under the same name a silent no-op: it reports
    success while leaving you one rollback point. repair_0182_pre_… then
    repair_0182_mid_….
  • It rules out interleaving the passes per table. The second snapshot sits
    between the passes, so a per-table interleave needs the admin back five times
    across 10-15 h — and prices_writer cannot FREEZE. Ordering is now the
    hybrid: _1h alone as the review gate, then the other four batched. Three
    admin windows, tabled in the task file.
  • Disk, measured on prod rather than estimated. 16.82 GiB in span across the
    five tables (799 active parts), _1h alone 9.56 GiB. A FREEZE hardlinks, so
    it costs nothing when taken; the cost accrues as the run's version + 1
    inserts merge and supersede the pinned parts, ceilinged at what was frozen.
    Both snapshots peak at ~33.6 GiB plus ~4-6 GiB of new parts — ~9% of the 430.6
    GiB free, though a ~68% increase on our own 58.93 GiB footprint.

Also adds a cleanup section and an acceptance criterion for it. Nothing
expires a snapshot, and neither ClickHouse nor the tool removes one, so ~33.6
GiB stays pinned on a volume BE own 96% of until an admin unfreezes it — the
same shape as the incident this whole chain started from.

Still no FREEZE taken and no partition written.

The completion order agreed for the repair chain already settled this —
0201, then 0182, then 0172 close, sequenced — but the task file still
said routes a/b/c were undecided under "this now blocks the run".
A fresh session reads the task file, not the session note, so on paper
the run was still blocked.

Route (a) was rejected because a combined run swamps the
rows_reset ~= rows_enriched check, ~32M against ~357k, and that check is
the only signal that catches values zeroed and never recomputed. Route
(b) is the better tool but costs a PR before any prod run and leaves
0201 outstanding regardless.

Records what each of the two passes runs, that the FREEZE span is 67
months x 5 tables, and the one operational question left for the admin:
one snapshot before pass 1, or a second between the passes so they are
independently revertible.
Two snapshots, so pass 2 is revertible without discarding pass 1's ~32M
recovered rows. Three consequences the decision does not state on its own.

The names must differ. The freeze script's `already-frozen` branch keeps a
pre-existing snapshot rather than overwriting it — correct, but it makes a
second freeze under the same name a silent no-op that reports success while
leaving one rollback point. `repair_0182_pre_` then `repair_0182_mid_`.

The second snapshot sits between the passes, so interleaving them per table
would need the admin back five times across 10-15 h, and `prices_writer`
cannot FREEZE. Ordering is now the hybrid — `_1h` alone as the review gate,
then the other four batched — three admin windows, tabled.

Disk measured rather than estimated: 16.82 GiB in span across the five
tables, 799 active parts, `_1h` alone 9.56 GiB. A FREEZE hardlinks and so
costs nothing when taken; the cost accrues as the run's version+1 inserts
merge and supersede the pinned parts, ceilinged at what was frozen. Both
snapshots peak ~33.6 GiB plus ~4-6 GiB of new parts — ~9% of the 430.6 GiB
free, but a ~68% increase on our own 58.93 GiB footprint.

Adds the cleanup step and an acceptance criterion for it. Nothing expires a
snapshot and neither ClickHouse nor the tool removes one, so ~33.6 GiB stays
pinned on a volume BE own 96% of until an admin unfreezes it — the same
shape as the incident this whole chain started from.

Still no FREEZE taken and no partition written.
@karczuRF karczuRF changed the title docs(0182): record route (c) as the decided mechanism docs(0182): record route (c), and settle the snapshot count at two Aug 18, 2026
…says

Measured during 0182's pass 1. The 2026-08-13 classification is right that
~32M rows are XLM-quoted and fillable and wrong about when: it attributes
them to 2021-02..2024-01, and almost none are below 2022-04.

Months 202110-202203 hold 0-13 XLM-quoted candidates each and every one ran
`enriched 0` with the "no USD reference (exotic quotes)" warning. 202204
enriched 761,735 of 1,531,768 in 79 batches, ~67 s. The boundary is sharp and
belongs to the data, not the tool — those earlier candidates are exotic-quoted
and are the permanent no_reference floor rather than missing data.

Records the per-month breakdown by quote leg, and flags that 202204+ in it is
post-drain so the small numbers there are not read as the pre-run population.
The count may still be near 32M; this settles the span only, and the honest
count comes from pass 1's summary.

Also records the two explanations falsified on the way, because both are
plausible enough to be re-derived: the XLM/USDC reference DOES reach back to
2021-02-01, and its early candles do NOT have zero volume — every month
202102-202207 has ~720 candles, all with volume, non-NULL usd_ref.

That last check settles 0182's own epoch as a side effect. USDT/USDC is dense
and non-NULL from 202102, first candle 2021-02-07 19:00, so 1612656000 cannot
zero rows into a reference hole. The strand risk the task warns about is closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant