feat(attachments): streaming attachment transport + saveFileFromUri by khawarizmus · Pull Request #1039 · powersync-ja/powersync-js

khawarizmus · 2026-07-14T15:14:13Z

Summary

Addresses the streaming request in discussion #968. The AttachmentQueue currently round-trips every file body through an ArrayBuffer in JS memory (localStorage.readFile → remoteStorage.uploadFile, and the reverse on download), which causes memory pressure / OOM for large files on lower-end devices.

This models the remote side (transfer and delete) as a single adapter so implementations can stream file-URI to the network natively, never materializing the file in the JS heap.

What's included

AttachmentTransportAdapter (common) — owns all remote operations: upload/download/delete. Supply this to fully replace remoteStorage.
AttachmentQueue options are now a discriminated union — provide exactly one of remoteStorage or transportAdapter. Supplying both, or neither, is a compile-time type error (a runtime guard also covers plain-JS callers). When only remoteStorage is given, an internal buffered transport is composed from it automatically, so existing setups are unchanged.
saveFileFromUri — registers a file already on disk and queues it for upload without reading it into memory. Only available when the queue is configured with a StreamingLocalStorageAdapter (enforced in the type system).
StreamingLocalStorageAdapter — a subinterface of LocalStorageAdapter adding buffer-free moveFile; implemented by the Node, Expo, and RN-FS adapters.
ExpoFileSystemTransportAdapter (attachments-storage-react-native) — native transport using Expo's uploadAsync/downloadAsync (+ a deleteFile callback) for buffer-free transfer.
ReactNativeFSTransportAdapter (attachments-storage-react-native) — native transport using @dr.pogodin/react-native-fs's uploadFiles/downloadFile (+ a deleteFile callback).

Backward compatibility

Default behavior is unchanged. When no transportAdapter is supplied, a buffered transport is composed internally from localStorage/remoteStorage, and RemoteStorageAdapter is unchanged. Existing { localStorage, remoteStorage } setups still compile and behave identically.

One intended behavior change for the RemoteStorageAdapter: downloadFile now receives the record with its destination localUri populated (previously null for fresh downloads).

Generated with Claude, tested and reviewed manually.

changeset-bot · 2026-07-14T15:14:21Z

🦋 Changeset detected

Latest commit: d5ae825

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 7 packages

Name	Type
@powersync/common	Minor
@powersync/attachments-storage-react-native	Minor
@powersync/node	Minor
@powersync/react-native	Patch
@powersync/tanstack-react-query	Patch
@powersync/web	Patch
@powersync/diagnostics-app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

simolus3

I think the changes and the BufferedAttachmentTransport fallback are clean, but we can probably restructure the public interface a bit to get rid of runtime errors.

simolus3 · 2026-07-15T13:17:35Z

+ * Describes the HTTP request used to upload a file's bytes to remote storage.
+ * Typically points at a presigned URL.
+ */
+export interface ExpoUploadRequest {


Would it be possible to add an optional dependency on expo and then use expo types here?

I could not import the types from Expo. The dependency is already present.

What errors did you get? We might need to tweak some typescript configs but copying the types doesn't sound right.

simolus3 · 2026-07-16T07:32:59Z

    this.logger = logger ?? db.logger;
    this.attachmentService = new AttachmentService(db, this.logger, tableName, archivedCacheLimit);
+
+    if (!transportAdapter && !remoteStorage) {


We should also throw when both are set (or better yet make it a type error), because the remote implementation wouldn't be used in that case which sounds unintentional.

Encoded this in the type now.

Can we also remove this check then? I don't really see the point in runtime checks for users who get the types wrong.

Model attachment byte-transfer as a single operation via AttachmentTransportAdapter, with a default BufferedAttachmentTransport that composes the existing local/remote adapters. Add AttachmentQueue.saveFileFromUri and an optional LocalStorageAdapter.moveFile for buffer-free registration of files already on disk, a native Expo ExpoFileSystemTransportAdapter, and moveFile in the Node and React Native local adapters. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Tag LocatedAttachmentRecord as @Alpha, use a qualified @link to AttachmentQueue.saveFile, and update etc/common.api.md for the new attachment transport API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…pter Native streaming transport for @dr.pogodin/react-native-fs (raw binary PUT via uploadFiles + downloadFile), mirroring ExpoFileSystemTransportAdapter, so non-Expo apps can transfer large attachments without buffering them in JS memory. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…pter Enhance AttachmentTransportAdapter to support file deletion operations. This includes updates to ExpoFileSystemTransportAdapter and ReactNativeFSTransportAdapter to implement the new deleteFile method. Additionally, adjustments were made to AttachmentQueue and BufferedAttachmentTransport to integrate the delete functionality into the attachment synchronization process.

…vulnerability audit This commit updates the websocket-driver dependency to version 0.7.5 in both pnpm-lock.yaml and pnpm-workspace.yaml to address security advisories. Additionally, the README.md for attachments-storage-react-native has been enhanced to clarify the functionality of local storage and streaming transport adapters, including usage examples and API details. # Conflicts: # pnpm-lock.yaml # pnpm-workspace.yaml

…er interfaces This commit refines the AttachmentQueue and transport adapter interfaces by consolidating configuration options and enhancing type safety. The `AttachmentQueueOptions` now requires either a `remoteStorage` or a `transportAdapter`, ensuring clarity in usage. Additionally, the `saveFile` and `saveFileFromUri` methods have been updated to utilize a unified `SaveAttachmentOptions` type, improving consistency across attachment operations. Documentation in the README.md has also been updated to reflect these changes.

…port streaming This commit refines the local storage adapter interfaces by replacing the `LocalStorageAdapter` with `StreamingLocalStorageAdapter` in multiple components, including `ExpoFileSystemStorageAdapter`, `ReactNativeFileSystemStorageAdapter`, and `NodeFileSystemAdapter`. The `AttachmentQueue` and related types have been updated to leverage the new streaming capabilities, ensuring that the `saveFileFromUri` method is only available when a streaming-capable adapter is used. Documentation has been adjusted to reflect these changes.

This commit updates the type definition for the `queue` variable in the attachments test file to explicitly use `AttachmentQueue<NodeFileSystemAdapter>`. This change enhances type safety and clarity in the test setup, aligning with recent refinements in the local storage adapter interfaces.

khawarizmus · 2026-07-23T01:01:22Z

@simolus3 Tests that are outside the scope of the PR are failing. But the PR is ready.

khawarizmus · 2026-07-24T03:20:19Z

@simolus3 CI is passing now.

simolus3 · 2026-07-24T07:13:43Z

+ * Describes the HTTP request used to upload a file's bytes to remote storage.
+ * Typically points at a presigned URL.
+ */
+export interface ExpoUploadRequest {


What errors did you get? We might need to tweak some typescript configs but copying the types doesn't sound right.

simolus3 · 2026-07-24T07:33:39Z

    this.logger = logger ?? db.logger;
    this.attachmentService = new AttachmentService(db, this.logger, tableName, archivedCacheLimit);
+
+    if (!transportAdapter && !remoteStorage) {


Can we also remove this check then? I don't really see the point in runtime checks for users who get the types wrong.

…uard

…ntegration This commit refines the integration of the `ReactNativeFileSystemTransportAdapter` within the `ReactNativeFileSystemStorageAdapter`. The transport adapter is now created through a method in the storage adapter, enhancing encapsulation. Additionally, type exports for upload and download requests have been added to improve type safety and clarity in usage. Documentation in the README.md has been updated to reflect these changes.

This commit updates the `expo-file-system` dependency from version 19.0.21 to 56.0.8 in the `package.json` and `pnpm-lock.yaml` files. The README.md has been revised to reflect the new API usage for file uploads and downloads, emphasizing the requirement for Expo SDK 56+. Additionally, the `ExpoFileSystemTransportAdapter` has been modified to utilize the modern upload API, enhancing performance and memory management during file transfers.

This commit introduces a new streaming attachment transport (`AttachmentTransportAdapter`) and the `AttachmentQueue.saveFileFromUri` method, allowing large files to be transferred without being buffered in JS memory. It includes native implementations for Node, Expo (SDK 56+), and React Native. Additionally, the `NodeFileSystemAdapter` is updated to create a streaming transport adapter, and a new `NodeFileSystemTransportAdapter` class is added to handle file uploads and downloads directly between local files and remote storage. Tests for the streaming transport functionality have also been added.

simolus3

Looks good to me 👍

simolus3 · 2026-08-03T11:33:18Z

+      headers,
+      body: Readable.toWeb(createReadStream(attachment.localUri)) as ReadableStream,
+      duplex: 'half'
+    } as RequestInit & { duplex: 'half' });


Maybe add a comment pointing to node-fetch/node-fetch#1769 as an explanation.

simolus3 · 2026-08-03T11:36:56Z

+
+### With Expo File System (`File.upload` / `File.downloadFileAsync`)
+
+> The Expo streaming transport requires **Expo SDK 56+** (`expo-file-system >=56`). The `ExpoFileSystemStorageAdapter` itself still works on Expo SDK 54+.


It's odd to have this as a quote. If you want an alert, add a > [!NOTE] line above. Otherwise, just add it without the quote.

khawarizmus marked this pull request as draft July 14, 2026 15:54

khawarizmus force-pushed the attachment-transport branch from 509d298 to bf2d8cc Compare July 14, 2026 23:21

khawarizmus marked this pull request as ready for review July 16, 2026 06:20

khawarizmus requested review from joshuabrink, simolus3 and stevensJourney July 16, 2026 06:20

simolus3 reviewed Jul 16, 2026

View reviewed changes

khawarizmus and others added 8 commits July 22, 2026 21:03

fix(common): resolve api-extractor report and doc warnings

bcc9900

Tag LocatedAttachmentRecord as @Alpha, use a qualified @link to AttachmentQueue.saveFile, and update etc/common.api.md for the new attachment transport API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

khawarizmus force-pushed the attachment-transport branch from a56ffa6 to 037d1fc Compare July 23, 2026 00:47

khawarizmus requested a review from simolus3 July 23, 2026 00:51

Merge branch 'main' into attachment-transport

cfe9d22

simolus3 reviewed Jul 24, 2026

View reviewed changes

khawarizmus and others added 5 commits July 30, 2026 11:01

Rename the abbreviation of FS to FileSystem and removed the runtime g…

8aa7ab3

…uard

Merge branch 'main' into attachment-transport

d5ae825

khawarizmus requested a review from simolus3 July 30, 2026 21:10

simolus3 approved these changes Aug 3, 2026

View reviewed changes


		### With Expo File System (`File.upload` / `File.downloadFileAsync`)

		> The Expo streaming transport requires Expo SDK 56+ (`expo-file-system >=56`). The `ExpoFileSystemStorageAdapter` itself still works on Expo SDK 54+.

Uh oh!

Conversation

khawarizmus commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Backward compatibility

Uh oh!

changeset-bot Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

simolus3 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

khawarizmus commented Jul 23, 2026

Uh oh!

khawarizmus commented Jul 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simolus3 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

khawarizmus commented Jul 14, 2026 •

edited

Loading

changeset-bot Bot commented Jul 14, 2026 •

edited

Loading