improvement(postgres-storage): Improve bucket data query performance

[stevensJourney]

Context

Bucket data is read while a client drains a checkpoint, making this query a potentially hot part of SQL-backed sync storage. Recent performance measurements from the storage benchmark PR made the cost particularly visible as the number of rows and resolved buckets increased. This PR focuses on the Postgres query improvement discovered through that work; the benchmark harness itself is covered separately.

These results are for storage v1 and come entirely from changing how existing bucket rows are read. They do not yet rely on storage v3 batching changes or potential checksum improvements. This makes the improvement comparatively cheap and low risk: it does not require a storage migration, change persisted data, or alter the sync protocol, and it leaves further gains from v3 batching and checksum work available separately.

Existing query

The previous Postgres implementation built one statement containing an OR branch for every requested bucket. Each branch paired a bucket name with that bucket's starting operation ID, after which the complete result was ordered by bucket name and operation ID and limited to the requested batch size.

SELECT *
FROM bucket_data
WHERE group_id = $1
  AND op_id <= $2
  AND (
    (bucket_name = $4 AND op_id > $5)
    OR (bucket_name = $6 AND op_id > $7)
    OR ...
  )
ORDER BY bucket_name, op_id
LIMIT $3;

This keeps the read to one database round trip, but the statement grows with the bucket count and asks PostgreSQL to plan a large disjunction containing different range bounds. Its performance also became nonlinear as buckets grew denser. The original 10k-todos-1k-buckets measurement took approximately 3 seconds to drain, while 100k-todos-1k-buckets took approximately 86 seconds. Increasing the operations by 10× therefore increased the drain time by roughly 29×.

The benchmark's initial drain starts every bucket at operation zero, which can make the original query look healthier than it is.PostgreSQL can collapse those equivalent range conditions into a single bucket_name = ANY (...) index condition. Once buckets have advanced to different operation IDs, however, their range conditions are no longer interchangeable. An explanation of the tenth batch in the 10k/200 scenario showed PostgreSQL scanning all 20,000 operations up to the checkpoint, applying the long per-bucket OR expression as a filter, materializing 11,000 matching rows, and top-N sorting those rows to return a batch of 1,000. Of the scanned rows, 9,000 were removed by the filter before the sort.

Limit                                      actual rows=1,000
  Sort                                    actual rows=1,000
    Sort Method: top-N heapsort
    Bitmap Heap Scan on bucket_data       actual rows=11,000
      Recheck: group_id and checkpoint op_id
      Filter: per-bucket name and start-op OR branches
      Rows Removed by Filter: 9,000
      Bitmap Index Scan using unique_id   actual rows=20,000

Buffers: shared hit=972
Execution Time: 44.151 ms

The important cost is not just the 44 ms for this individual query, but that the scan, filter, and sort are repeated for every batch. As the checkpoint contains more operations and the bucket cursors diverge, the amount of work grows with the stored range rather than only with the next batch being returned.

Alternatives considered

An experimental SQLite implementation used during performance testing sorted the requested buckets and executed one indexed (group_id, bucket_name, op_id) range scan per bucket while carrying the remaining batch limit across those scans. This was useful as an alternative query shape to investigate, rather than established production behavior to copy directly. Porting the experiment to Postgres improved dense workloads, reducing the 100k/1k case from approximately 86 seconds to 27 seconds. It also exposed an important difference between an embedded database and a database server: SQLite pays almost no round-trip cost for another statement, while Postgres does. Sparse workloads regressed because hundreds or thousands of queries could return only a few rows, or no rows at all. The 1k/1k case increased from approximately 149 ms to 657 ms.

A density-based hybrid was considered, but bucket count alone is not enough to choose a strategy. The same 1,000-bucket request behaved very differently depending on whether it contained 2,000 or 200,000 operations, and the read API does not have a reliable estimate of future rows per bucket. A threshold would therefore encode a workload assumption rather than fix the underlying query shape. A JSONB request-table join had also been explored previously, but expanding and joining the JSON input was substantially slower.

The more promising approach was to preserve the per-bucket index scans while moving them back into a single database query. Bucket requests can be represented as a VALUES table and passed through a correlated LATERAL subquery. This gives each bucket its own parameterized index range scan without incurring one network round trip per bucket.

The first LATERAL query

The initial LATERAL implementation assigned each bucket a deterministic ordinal and ordered the combined output by that ordinal and op_id. A global op_id order was deliberately avoided because the storage index is ordered by (group_id, bucket_name, op_id) and rows need to remain contiguous per bucket for chunk construction.

This version performed well for sparse and medium workloads, but the million-row measurements revealed another nonlinear path. The combined output was ordered and limited only after every LATERAL branch had produced its candidates. EXPLAIN (ANALYZE, BUFFERS) showed that a request returning 1,000 rows performed all 1,000 index scans, produced 360,000 candidate rows, and then used a top-N sort to discard 359,000 of them.

Limit                                      actual rows=1,000
  Sort                                    actual rows=1,000
    Sort Method: top-N heapsort
    Nested Loop                           actual rows=360,000
      Values Scan                         actual rows=1,000
      Limit                               actual rows=360 loops=1,000
        Index Scan using unique_id        actual rows=360 loops=1,000

Buffers: shared hit=356,369 read=13,320
Execution Time: 278.281 ms

The same work was repeated for every returned batch. As buckets became denser, each batch rescanned large portions of all requested buckets before discarding nearly every candidate. This explains why this intermediate version reached approximately 755 seconds for one million todos across 1,000 buckets.

Preserving bucket order for early termination

The final query orders the bucket request table inside a non-flattened subquery before joining it to the per-bucket range scan. OFFSET 0 acts as an optimization barrier that preserves the ordered outer path for PostgreSQL's planner. The final ordering can then use bucket_order as a presorted key and incrementally sort each bucket's operations. Most importantly, the outer limit can stop the nested loop as soon as the requested number of rows has been produced.

The repeated ordering and limits are intentional. The request ordering establishes the presorted bucket path, while the LATERAL ordering keeps each bucket's index range scan in operation order. The inner limit bounds that per-bucket scan and prevents it from being flattened into a broader join; because it equals the outer limit, it cannot exclude a row that the batch could return. The final ordering guarantees contiguous bucket chunks, and the outer limit applies the batch size across all buckets and stops the nested loop early.

SELECT bucket_data.*
FROM (
  SELECT *
  FROM (
    VALUES
      ($4, $5, 0),
      ($6, $7, 1),
      ...
  ) AS bucket_requests(bucket_name, start_op_id, bucket_order)
  ORDER BY bucket_order
  OFFSET 0
) AS requested
CROSS JOIN LATERAL (
  SELECT *
  FROM bucket_data
  WHERE group_id = $1
    AND bucket_name = requested.bucket_name
    AND op_id > requested.start_op_id
    AND op_id <= $2
  ORDER BY op_id
  LIMIT $3
) AS bucket_data
ORDER BY requested.bucket_order, bucket_data.op_id
LIMIT $3;

For the same 1,000-row request, the resulting plan consumes only six bucket requests and 1,001 candidate rows. The global sort is replaced by an incremental sort, shared-buffer hits fall from more than 356,000 to approximately 1,000, and execution time for the explained query falls from roughly 278 ms to 1.1 ms.

Limit                                      actual rows=1,000
  Incremental Sort                         actual rows=1,000
    Presorted Key: bucket_order
    Nested Loop                            actual rows=1,001
      Sort                                 actual rows=6
        Values Scan                        actual rows=1,000
      Limit                                actual rows=167 loops=6
        Index Scan using unique_id         actual rows=167 loops=6

Buffers: shared hit=1,027
Execution Time: 1.145 ms

Performance

The table compares the original implementation with the final ordered LATERAL query. The million-todo baseline scenarios were intentionally skipped because the original query was already taking approximately 86 seconds at 100k todos and 1,000 buckets, making the larger baseline runs prohibitively slow. After the query fix, both million-todo scenarios complete in approximately 62–63 seconds. Lower drain time and higher read throughput are better.

Scenario	Operations	Baseline drain	Final drain	Baseline read MiB/s	Final read MiB/s
1k todos / 200 buckets	2,000	89.36 ms	77.35 ms	7.14	8.25
1k todos / 1k buckets	2,000	149 ms	99.18 ms	5.47	8.21
10k todos / 200 buckets	20,000	814 ms	536 ms	7.39	11.23
10k todos / 1k buckets	20,000	2,976 ms	616 ms	2.08	10.06
100k todos / 200 buckets	200,000	11,346 ms	5,490 ms	5.30	10.96
100k todos / 1k buckets	200,000	85,634 ms	6,051 ms	0.70	9.97
1m todos / 200 buckets	2,000,000	Skipped (slow)	61,958 ms	—	9.76
1m todos / 1k buckets	2,000,000	Skipped (slow)	63,388 ms	—	9.55

---

AI Disclosure. The original storage benchmark was generated with Codex 5.5 assistance. I guided Codex through the process of testing various query improvements to reach this end result.

[changeset-bot]

🦋 Changeset detected

Latest commit: 3e75ba7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-postgres-storage	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

This looks great! What is still needed to get this merged?

[stevensJourney]

This looks great! What is still needed to get this merged?
There isn't really anything outstanding at this point. I think we could release this fix as an incremental improvement. I can't see any downsides/risks from this change.