Parse block payloads with hyperpb instead of dynamicpb

[sduchesneau]

Replaces #873, which GitHub closed when this branch was rebased off the local-buffer PR
and onto develop — it can now merge on its own, ahead of #869.

The from-proto sink only ever has the module's descriptor at runtime, never a generated Go type, so every block payload goes through a dynamic parser. It used protobuf-go's dynamicpb. This switches it to hyperpb, which compiles the descriptor into a parser once and then parses into an arena.

The descriptor walk only ever read the message — Descriptor, Get, IsValid and nothing else — so it now takes a protoreflect.Message and the decoder chooses the parser. That is the whole structural change; the walk's behaviour is untouched.

Numbers

M4 Max, 200 entities per payload. BenchmarkUnmarshal, parse in isolation:

shape	parser	ns/op	MB/s	allocs/op
narrow	dynamicpb	122,703	64	3,218
narrow	hyperpb	7,029	1,122	1
wide	dynamicpb	1,064,019	86	16,374
wide	hyperpb	58,955	1,544	1

TestClientDecodeScaling, the real per-block work (parse + walk + buffer), wide entity, 16 cores:

workers	dynamicpb	hyperpb	ratio
1	117k	309k	2.64x
4	338k	929k	2.75x
8	517k	1.40M	2.71x
15	534k	1.56M	2.92x

18x on the parse becomes 1.9x on the full decode path and 2.7x across the pool — the parse was about half the wide-entity cost and is now about a twentieth, so the walk is what is left (~90%).

Two conclusions in benchmarks/README.md changed and have been rewritten rather than appended to:

• PostgreSQL is now the bottleneck for wide entities, where the sink used to be. Eight workers produce 1.40M rows/s against binary COPY's ~799k.
• The eight-worker cap is still right, for a different reason. dynamicpb was flat past eight because it was allocator-bound. hyperpb still gains there (4.52x → 5.06x from 8 to 15 workers), so the arena did lift that ceiling — but seven extra cores buy 11%, so the default stands.

Constraints, and how they are handled

hyperpb is not a drop-in in every respect, and each of these is a panic rather than an error, so each is pinned by a test in benchmarks/parser_test.go:

• Messages are read-only. Mutators panic. The walk qualifies; TestHyperpbRejectsMutation fails loudly if a write is ever added to that path.
• An arena carries exactly one parse until freed. Parsing again panics with attempted to parse message using in-use Context. This is why the decoder keeps one arena per block slot, not the one-per-worker arrangement that looks natural.
• Free on an arena nothing was allocated in panics on a negative slice bound. A block whose module produced no output never parses, so a slot really can reach its first free untouched — decoder.arena.used covers it.
• Values alias the arena. Strings and bytes point into it, and the rows sit in a BufferedInserter until apply replays them. So an arena is freed at the top of the next decodeAll, by which point the previous round has been applied and flushed.
• amd64/arm64 only. Every release target builds (verified linux/darwin/windows × amd64/arm64); 32-bit was already unbuildable for unrelated reasons (govalidator overflows on linux/386 on develop today).

Breaking

sql.Database.WalkMessageDescriptorAndInsert, WalkMessageDescriptorAndInsertInto, BaseDatabase.WalkMessageDescriptorAndInsertWithDialect and sql.Dialect.AppendInlineFieldValues take a protoreflect.Message where they took a *dynamicpb.Message.

tui/print.go and pb/sf/substreams/v1/package.go still use dynamicpb and should: they are jsonpb.AnyResolver implementations that hand an empty message to a JSON decoder to unmarshal into, which is mutation. hyperpb cannot serve that.

Measured on this branch alone

sink/sql/db_proto/parser_test.go compares the two parsers with no database and no
container: the parse needs only a descriptor, and the walk only a dialect and an in-memory
inserter. go test ./sink/sql/db_proto/ -bench . -benchmem reproduces it.

BenchmarkUnmarshal, the parse in isolation, 200 entities per payload, M4 Max:

shape	parser	ns/op	MB/s	allocs/op	gain
narrow	dynamicpb	114,087	69	3,218
narrow	hyperpb	6,442	1,225	1	17.7x
wide	dynamicpb	987,670	92	16,374
wide	hyperpb	54,388	1,673	1	18.2x

BenchmarkDecodeBlock, the per-block work the decoder parallelises — parse and walk,
which is the ratio an operator actually sees:

shape	dynamicpb	hyperpb	gain
narrow	225,569 ns	102,783 ns	2.19x
wide	1,620,986 ns	618,942 ns	2.62x

The parse stops being the expensive half of decoding: the walk is what is left.

Testing

• Full sink/sql suite green, Postgres and ClickHouse. The integration tests no longer need an environment variable to run — see below.

Also on this branch

Two commits that are not about the parser:

• Stop the sink/sql integration containers dying mid-suite in CI — go test ./... builds several packages that each start their own containers and runs them in parallel, so more than one test binary was connected to the shared Ryuk reaper at a time. Runs failed with every container in one package going unreachable part-way through the suite — tests that had already passed, then connection refused on both the Postgres and the ClickHouse mapped port — right as a different package's binary exited. TESTCONTAINERS_RYUK_DISABLED is now set in CI: Ryuk buys nothing there, since the runner is destroyed when the job ends.

• Always run the sql integration tests — TestMain no longer checks SF_SINK_SQL_INTEGRATION_TESTS and os.Exit(0)s when it is unset. The suite needs a container runtime anyway, and a variable that quietly skips half of it only hides a broken environment behind a green run. Same reasoning as the container correctness test in Sink from-proto through a disk spool, and load without constraints #869. The variable now has no readers left in the repo and its export is gone from bin/test.sh; the workflow runs the tests on ubuntu only, since the macOS runner has no container runtime and keeps the build step.

🤖 Generated with Claude Code

[maoueh]

Outside of failing CI

[sduchesneau]

@maoueh yeah I'm tweaking test runs and benchmarks to get this clean.
It will make the next big PR easier to get in.

[sduchesneau]

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:486df7d

📦 Image Reference ghcr.io/streamingfast/substreams:486df7d

digest	sha256:122fbf9bd56dd99dcbb9a7e3e98b116c35b75a620b9b3452fc11fcd944e6c6ad
vulnerabilities
platform	linux/amd64
size	123 MB
packages	382

📦 Base Image oisupport/staging-amd64:24.04

also known as	a215e986b44aae6f10795ded1e39ce93d9c236d8163d21a522ffd0ab3659f546 noble noble-20260730.1
digest	sha256:019e8eb29a85e74d64925745884f2ec79aa27e3feab36353d24656f4d6b89467
vulnerabilities