Parse block payloads with hyperpb instead of dynamicpb

[sduchesneau]

Stacked on #869.

The from-proto sink only ever has the module's descriptor at runtime, never a generated Go type, so every block payload goes through a dynamic parser. It used protobuf-go's dynamicpb. This switches it to hyperpb, which compiles the descriptor into a parser once and then parses into an arena.

The descriptor walk only ever read the message — Descriptor, Get, IsValid and nothing else — so it now takes a protoreflect.Message and the decoder chooses the parser. That is the whole structural change; the walk's behaviour is untouched.

Numbers

M4 Max, 200 entities per payload. BenchmarkUnmarshal, parse in isolation:

shape	parser	ns/op	MB/s	allocs/op
narrow	dynamicpb	122,703	64	3,218
narrow	hyperpb	7,029	1,122	1
wide	dynamicpb	1,064,019	86	16,374
wide	hyperpb	58,955	1,544	1

TestClientDecodeScaling, the real per-block work (parse + walk + buffer), wide entity, 16 cores:

workers	dynamicpb	hyperpb	ratio
1	117k	309k	2.64x
4	338k	929k	2.75x
8	517k	1.40M	2.71x
15	534k	1.56M	2.92x

18x on the parse becomes 1.9x on the full decode path and 2.7x across the pool — the parse was about half the wide-entity cost and is now about a twentieth, so the walk is what is left (~90%).

Two conclusions in benchmarks/README.md changed and have been rewritten rather than appended to:

• PostgreSQL is now the bottleneck for wide entities, where the sink used to be. Eight workers produce 1.40M rows/s against binary COPY's ~799k.
• The eight-worker cap is still right, for a different reason. dynamicpb was flat past eight because it was allocator-bound. hyperpb still gains there (4.52x → 5.06x from 8 to 15 workers), so the arena did lift that ceiling — but seven extra cores buy 11%, so the default stands.

Constraints, and how they are handled

hyperpb is not a drop-in in every respect, and each of these is a panic rather than an error, so each is pinned by a test in benchmarks/parser_test.go:

• Messages are read-only. Mutators panic. The walk qualifies; TestHyperpbRejectsMutation fails loudly if a write is ever added to that path.
• An arena carries exactly one parse until freed. Parsing again panics with attempted to parse message using in-use Context. This is why the decoder keeps one arena per block slot, not the one-per-worker arrangement that looks natural.
• Free on an arena nothing was allocated in panics on a negative slice bound. A block whose module produced no output never parses, so a slot really can reach its first free untouched — decoder.arena.used covers it.
• Values alias the arena. Strings and bytes point into it, and the rows sit in a BufferedInserter until apply replays them. So an arena is freed at the top of the next decodeAll, by which point the previous round has been applied and flushed.
• amd64/arm64 only. Every release target builds (verified linux/darwin/windows × amd64/arm64); 32-bit was already unbuildable for unrelated reasons (govalidator overflows on linux/386 on develop today).

Breaking

sql.Database.WalkMessageDescriptorAndInsert, WalkMessageDescriptorAndInsertInto, BaseDatabase.WalkMessageDescriptorAndInsertWithDialect and sql.Dialect.AppendInlineFieldValues take a protoreflect.Message where they took a *dynamicpb.Message.

tui/print.go and pb/sf/substreams/v1/package.go still use dynamicpb and should: they are jsonpb.AnyResolver implementations that hand an empty message to a JSON decoder to unmarshal into, which is mutation. hyperpb cannot serve that.

Testing

• TestParsersAgree — dynamicpb and hyperpb produce byte-identical protojson output for both payload shapes, read purely through protoreflect, the same interface the walk uses. Ungated: milliseconds, no container.
• TestHyperpbArenaContract — pins the three arena rules above.
• Full sink/sql suite green with SF_SINK_SQL_INTEGRATION_TESTS=true, Postgres and ClickHouse, including TestDbProtoPostgresDecodeWorkers (1 worker and 8 workers must produce identical tables, in block order).

Also on this branch

Two commits that are not about the parser:

• Stop the sink/sql integration containers dying mid-suite in CI — go test ./... builds several packages that each start their own containers and runs them in parallel, so more than one test binary was connected to the shared Ryuk reaper at a time. Runs failed with every container in one package going unreachable part-way through the suite — tests that had already passed, then connection refused on both the Postgres and the ClickHouse mapped port — right as a different package's binary exited. TESTCONTAINERS_RYUK_DISABLED is now set in CI: Ryuk buys nothing there, since the runner is destroyed when the job ends.

• Always run the sql integration tests — TestMain no longer checks SF_SINK_SQL_INTEGRATION_TESTS and os.Exit(0)s when it is unset. The suite needs a container runtime anyway, and a variable that quietly skips half of it only hides a broken environment behind a green run. Same reasoning as the container correctness test in Sink from-proto through a disk spool, and load without constraints #869. The variable now has no readers left in the repo and its export is gone from bin/test.sh; the workflow runs the tests on ubuntu only, since the macOS runner has no container runtime and keeps the build step.

🤖 Generated with Claude Code

[sduchesneau]

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:17024c7

📦 Image Reference ghcr.io/streamingfast/substreams:17024c7

digest	sha256:b38d5f0f2adae3b93b902549fdf57e2652f18f8d98817a0e037fed884255acac
vulnerabilities
platform	linux/amd64
size	123 MB
packages	382

📦 Base Image oisupport/staging-amd64:24.04

also known as	a215e986b44aae6f10795ded1e39ce93d9c236d8163d21a522ffd0ab3659f546 noble noble-20260730.1
digest	sha256:019e8eb29a85e74d64925745884f2ec79aa27e3feab36353d24656f4d6b89467
vulnerabilities

[sduchesneau]

Superseded by #876: this branch is now based on develop rather than on #869, so the parser change can merge on its own. GitHub closed this PR automatically when the head stopped descending from its base and would not let it reopen, hence the new number.