feat(module-postgres): concurrent table snapshots and pipelined snapshot flushes

[henriquekraemer]

Background

This comes out of the discussion in #448. We run a schema-per-tenant deployment (200+ schemas, same table structure per schema, wildcard schema sync rules), and the initial snapshot replicates tables one at a time. After profiling, the bottleneck in our case was not reading from the source: per 10k-row chunk, the storage flush accounted for roughly two thirds of the time, with the source read under 10%. Since chunk reads and flushes were fully serialized, both sides spent most of the time waiting on each other.

What this does

Two changes to the initial snapshot in WalStream:

1. Pipelined chunk flushes. Each table snapshot alternates between two storage writers, so one chunk's flush runs while the next chunk is read and evaluated. Progress (lastKey / replicated count) is only recorded once that chunk's flush and all earlier flushes have completed, so resumability works the same as before: a crash never resumes past unflushed rows. Chunks cover disjoint primary key ranges, so the two writers never touch the same rows.

2. A snapshot_concurrency connection option to snapshot multiple tables in parallel. Workers pull tables from a shared queue; each worker gets its own source connection and its own storage writers, so flushes from different workers run concurrently. Defaults to 1, which keeps table snapshots sequential.

replication:
  connections:
    - type: postgresql
      # ...
      snapshot_concurrency: 2

The per-table consistency model is unchanged: each table still gets its own markTableSnapshotDone with the LSN captured after that table's snapshot, and the final snapshot commit still happens once at the end.

One detail to review carefully: the secondary writers commit at ZERO_LSN when they finish. Ops flushed by a writer are only folded into the checkpoint (via keepalive_op) when that writer commits, and the final snapshot commit only covers the main batch's own persisted ops. Without those commits, ops written by the extra writers could end up above last_checkpoint and stay invisible until unrelated streamed changes advance the checkpoint. The new test caught this with a small chunk size.

Results

On our staging environment (source on db.t3.medium, Postgres bucket storage on db.r6i.xlarge, ~230 tenant schemas), the initial snapshot went from ~11.5k rows/s to ~18k rows/s with snapshot_concurrency: 2. Beyond 2 workers I didn't see further gains: at that point the service saturates a single Node core doing row evaluation, so more workers just queue on the same CPU. The interleaved op-ids across tables did not cause issues on Postgres storage, as discussed in #448.

For the pipelining to help, the storage batch_limits.max_record_count needs to be larger than the snapshot chunk size (10k rows), otherwise the auto-flush inside save() serializes the writes again.

Running with concurrent writers is also what surfaced the serialization conflicts fixed in #726.

Tests

Added parallel_snapshots.test.ts: multiple tables with 2 workers (multiple chunks per table with a small chunk size, plus a single-chunk table and an empty table), and a run with more workers than tables. The existing chunked/resumable snapshot tests cover the pipelined path, since it is active with the default concurrency as well.

[changeset-bot]

🦋 Changeset detected

Latest commit: fe1c810

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-postgres	Minor
@powersync/service-schema	Patch
@powersync/service-image	Patch
@powersync/service-core	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR