Fix the spool's critical bugs before it ships

[sduchesneau]

Fixes for the spool / write-mode / constraints work in #869, on top of that branch so the
review stays separate from the 12k-line feature diff.

Twelve bugs, each in its own commit with a regression test that was checked to fail
without its fix. Whole-repo go test ./... and go vet clean, including the
container-backed integration tests against real PostgreSQL and ClickHouse.

Data loss, corruption, or a hung process

Commit	Problem
Keep the spool within the budget it was given	Row-log segments record their size in Manifest.LogBytes, which Seal never added in, so row-insert reported every segment as 0 bytes and --spool-max-size was never reached. A segment larger than the whole quota could also never be queued, leaving awaitQuota spinning forever.
Close the COPY stream's file when a segment is sealed	pgcopy.Writer.Close leaves the file to whoever opened it, and a segment is sealed and forgotten: every sealed segment leaked one descriptor per table, and the applier's os.RemoveAll freed no space while they were held. The default format.
Let a cyclic schema undo a reorg	HandleBlocksUndo propagated TableApplyOrder's cycle error, so the one schema row-insert exists to support could neither undo a reorg nor start — Run undoes from the stored cursor before any block arrives.
Write the ClickHouse cursor to its file when a segment lands	The applier ended Apply with StoreCursor, which routes the cursor back into the spool whenever one is open. The cursor file was never written for the whole backfill, and the segment just applied stamped its cursor over the newer one held by the segment still being written.
Route ClickHouse block rows through the spool	InsertBlock appended straight to the accumulator, which with a spool open belongs to the applier's goroutine — an unsynchronised write to the map the applier swaps on every flush. It also meant RecordBlock was never reached, leaving every manifest with an empty block range.
Stop doubling backslashes in rendered SQL literals	standard_conforming_strings has been on by default since PostgreSQL 9.1, so escaping the backslash stored two where the value had one — and only batch-insert and row-insert came through quoteLiteral, putting the write mode in the data.
Encode an empty repeated field in COPY mode	Binary COPY resolves the encode plan from the Go type, so retyping an empty array to []string reached no plan at all on numeric[], bigint[], bytea[], bool[] or timestamp[]: the first block carrying an empty list refused the row and took the segment with it.
Encode a repeated enum field in COPY mode	normalizeSlice had no case for the walk's EnumValue, so a TEXT[] enum column failed on the default write mode.
Stop dialing the ClickHouse database before creating it	VerifySchemaCompatibility runs ahead of CreateDatabase and read the columns over a connection naming the schema, which on a first run does not exist. ClickHouse refuses that with UNKNOWN_DATABASE and newClient retries a failed dial forever, so setup never returned.
Match a recovered segment on both ends of its range	Recovery answered "already applied?" from the first block alone, so a record left behind by an earlier run could answer for a different segment starting on the same block. Narrowing to the whole range is not sufficient alone — see the next row.
Seal a segment only once its own cursor is recorded	A flush writes its rows and then its cursor, but the seal ran between the two, so a segment committed carrying the previous flush's cursor and claiming a range that cursor stops short of. The next run resumes there, the startup undo deletes the segment's own rows, and the record stays behind to answer AlreadyApplied for the segment carrying exactly those rows — which recovery then discards while replaying the segments behind it, moving the cursor over the gap.

Breaks a default-configuration run

Commit	Problem
Size the next segment from the throughput actually measured	Scaling the target by target/elapsed ignored how much the commit moved, so an idle or drain seal — fast because it was small — doubled the target while the database may well have been slow.
Have setup create the constraints it is asked for	setup built its policy from --apply-constraints, a flag only the run command declared, so it always created a bare schema and the --disable-* flags it does carry had nothing to act on. The flag stays off setup — there the question is whether, not when — and --disable-all-constraints is the new shorthand for turning all three off, replacing the deprecated --no-constraints.
Seal the spool on every way out of the run	The database was closed only when a bounded range completed, so Ctrl-C during a backfill dropped the open segment — up to --db-write-max-size of already-paid-for blocks, re-streamed.
Write a proto that parses from extract-proto --sql	Nested enums and map entries were referenced but never declared, and the per-field annotation hints were emitted where a field option cannot go, so the file the command tells you to feed back through --proto-file-override did not parse.

| Clear the applied-segment records at the chain head | Every segment applied recorded a row in _segments_ and nothing removed them, so a sink that stays live grew the table for as long as it ran — each restart backfills the little it is behind and seals a segment or two on the way. Reaching the head drains the spool and closes it for good, which leaves no directory for a record to answer for. |

| Bound a framed record by its file rather than a constant | The reader refused any record over 64MiB while the writer had no limit, so a large rendered tuple was written, counted into the manifest and verified intact, then failed only at apply — and recovery replayed that same segment on every start, leaving the sink unable to come up until the directory was removed by hand. |

Verification worth calling out

The three that were proven against a real server rather than argued from reading:

• Backslashes — on PostgreSQL 16 with the default standard_conforming_strings=on,
  delta\path 'quoted' (19 chars) stored as 20 under batch-insert and row-insert, and
  correctly under copy. The write-modes integration test now carries that value, which is
  what pins the three modes to the same bytes.
• Empty arrays — []string{} resolves no binary encode plan on 8 of the 10 array types
  tested. normalize_test.go now covers 14 array types against the real encoder.
• ClickHouse setup — measured: never returns on a fresh server; ~4s with the fix. The
  new integration test hangs 24s without it.
• Seal ordering — the new test drives four blocks through one-byte segments and reads
  every _segments_ row back: unfixed, the first reports "the segment covering blocks 1-2
  committed with a cursor at block 1, which does not cover it".

Not addressed

Left alone deliberately, all documented: constraint-name folding under identifier folding
(MissingConstraints / DropConstraints on a mixed-case table annotation); a rendered
tuple over 64 MiB, which writes fine, fails at apply and is then replayed by recover() on
every restart — needs a decision on what to do instead of failing; --live-block-time-delta
being excluded for DatabaseChanges runs too; the _segments_ table never being pruned;
and the transaction left open when Flush() or StoreCursor() fails in flushHolding.

[sduchesneau]

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:7c3cb9a

📦 Image Reference ghcr.io/streamingfast/substreams:7c3cb9a

digest	sha256:864174bcadc16d18dad9ea12dc60bf15f6aeb5b662a6e7b090a243f6632a3083
vulnerabilities
platform	linux/amd64
size	123 MB
packages	382

📦 Base Image oisupport/staging-amd64:24.04

also known as	a215e986b44aae6f10795ded1e39ce93d9c236d8163d21a522ffd0ab3659f546 noble noble-20260730.1
digest	sha256:019e8eb29a85e74d64925745884f2ec79aa27e3feab36353d24656f4d6b89467
vulnerabilities