persist: fix multipart blob reads and add blob store benchmarks

[jubrad]

Issue

S3Blob::get reads a multipart-uploaded object part by part. It learns how many parts to fetch from the x-amz-mp-parts-count header on its first request, and treats a missing header as "this object has one part". A store that does not send that header therefore returns only the object's first 8 MiB, and persist hands the truncated bytes to the parquet decoder, which panics with Invalid Parquet file. Corrupt footer.

rustfs is such a store. Every cold read of a blob above the multipart threshold, which is every large compaction output, crashed environmentd. The blob cache masks it for freshly written blobs, so it only surfaces on a cold read, for example after a restart or once the cache is under pressure. minio, garage and S3 all send the header.

I found this while benchmarking blob stores, and could not tell the two stores apart on the existing instruments, which is the other half of this PR.

Solution

The read path. get now also takes the object's total size from Content-Range, which every store returns on a request that names a part. When the part count is missing and the first part is short of the total, the remainder is fetched by byte range, in chunks the size of the first part. The reassembled length is then checked against the total, so a store returning a short object fails the read, which persist retries, rather than corrupting data downstream. Where the header is present, nothing changes.

The length check is the part worth keeping regardless of rustfs: it turns silent truncation into a loud failure for any S3-compatible store.

Measuring a blob store. Two instruments, because the existing ones cannot separate the store from Materialize's own work:

• persistcli bench blob drives one store through persist's own blob client at a given object size and concurrency. It writes, lists, reads at random and deletes, reporting throughput, latency percentiles, and the retries persist's retry loop needed, so a throttling store shows up as latency and a retry count instead of an error. Reads verify object length.
• test/blob-store-benchmark sweeps that across object sizes, concurrency levels and stored volumes for several stores, samples each store container's CPU and memory, and writes a CSV. Manual-only, so it is excluded from the CI reference lint.

More blob stores in the compositions. garage and rustfs join minio and azurite as external_blob_store targets for Materialized and Testdrive, resolved through a new blob_store module. The parallel benchmark takes --blob-store and --other-blob-store in place of --azurite (nothing in CI passed that flag to this composition), so two stores can be A/B'd directly, and it prints persist's blob operation counts and latencies after each scenario. The new BlobStoreReadsWrites scenario disables the blob cache so reads actually reach the store.

garage ships an image with no shell, so it gets an mzbuild image that adds one for the cluster setup a fresh node needs before it serves S3.

What the benchmarks found

Measured on one laptop under Docker, one store at a time, no resource limits, fsync off everywhere. Treat these as relative and provisional.

A control run matters here: minio's configuration was unchanged between two rounds, yet its per-cell throughput moved by a median of -6% with a 10th-to-90th range of -44% to +43%. A single-run cell in this harness is not interpretable below roughly 50% difference. Only the large, consistent gaps below clear that bar.

operation	garage vs minio	rustfs vs minio
reads, all sizes	4.0x to 4.7x slower	0.9x to 1.6x
writes	0.6x to 3.9x	0.4x to 1.5x
deletes	1.2x to 2.6x slower	2.6x to 3.9x slower

• garage is roughly 4x behind minio on reads at every size and saturates by 32 operations in flight, but does it on under 60 MiB of RSS for small objects against minio's multiple GiB.
• garage's defaults were badly wrong for persist's object sizes. With 10 MiB blocks and a raised read-concurrency cap, its 8 MiB reads went from 70 to 470 ops/s and its p50 from 454 ms to 66 ms. The config now lives in test/garage/garage.toml with the reasoning.
• rustfs tracks minio on reads and writes, is 3 to 4x behind on deletes, and refuses concurrent 64 MiB uploads with SlowDown from an internal write admission limit. Persist rides that out with retries, but those cells lose most of their throughput.
• rustfs's RUSTFS_HEAL_ENABLED=false and RUSTFS_SCANNER_ENABLED=false appear inert in 1.0.0-rc.5: the heal manager and scanner log identical startup and cycle messages with and without them. So rustfs carries background work that minio has switched off, with no evident way to disable it.

I have a fuller write-up with the full matrices, scaling curves and caveats that I did not commit, since the numbers are machine-specific. Happy to add it to the PR or paste it in a comment.

Testing

Unit tests for the Content-Range parser and the byte-range chunking. The behavioral fix was verified end to end: the scenario that reproducibly crashed environmentd on rustfs (blob cache off, so every scan fetches from the store) now completes with about 20 GiB of cold reads and no failures, twice.

Gotchas for the reviewer

• This is really two changes, the persist fix and the benchmark tooling, and I am happy to split them if you would rather review the production change on its own. The persist change is about 130 lines of the diff.
• --azurite is gone from the parallel benchmark in favour of --blob-store azurite. No CI pipeline passed that flag to this composition, so nothing in CI changes.
• CI will build the new test/garage mzbuild image on this PR.
• x-amz-mp-parts-count is part of the S3 GetObject contract and rustfs already defines the constant internally without sending it. Worth an upstream report, which I have not filed yet.
• Lint and formatting were not run against the final state of this branch.

🤖 Generated with Claude Code