Native: an isolate-unsendable error thrown from `uploadData` permanently wedges the upload queue and makes `disconnect()`/`close()` hang

[WorkGithubPerets]

Summary

On native platforms, if PowerSyncBackendConnector.uploadData throws an error
object that cannot be sent across isolates (for example dio's DioException,
which references the request's CancelToken and through it an
_AsyncCompleter), the sync client is permanently and silently wedged:

• the error reply never reaches the sync isolate, so the upload stays "in
  flight" forever while holding the crud mutex — uploadData is never called
  again for the rest of the process lifetime;
• there is no logging on the sync side (the stalled-queue warning only fires
  when uploadData returns without complete()), and SyncStatus.uploadError
  is never set;
• disconnect() and close() hang forever, because abort() awaits the
  pending upload;
• only killing the process recovers.

The docs say "If this call throws an error, it is retried periodically", with
no restriction on the error type, so this is easy to hit: any connector that
uses dio and attaches a CancelToken to its upload request (a natural thing
to do) wedges on the first failed upload while offline — which is exactly
when uploads fail. Repro is 100% for us on powersync 2.3.3 / dio 5.10.0 /
iOS device; the relevant code is unchanged on current main.

Reproduction

import 'dart:async';
import 'package:powersync/powersync.dart';

/// Stand-in for any error holding isolate-unsendable state. Real-world
/// instance: dio's DioException — RequestOptions.cancelToken holds a
/// Completer, and Completer instances cannot cross an isolate boundary.
class UnsendableException implements Exception {
  final Completer<void> completer = Completer();
  @override
  String toString() => 'upload failed';
}

class WedgingConnector extends PowerSyncBackendConnector {
  @override
  Future<PowerSyncCredentials?> fetchCredentials() async =>
      PowerSyncCredentials(endpoint: '<instance url>', token: '<token>');

  @override
  Future<void> uploadData(PowerSyncDatabase database) async {
    final tx = await database.getNextCrudTransaction();
    if (tx == null) return;
    throw UnsendableException();
  }
}

1. Open a database on iOS/Android/macOS (the sync-isolate path), connect(),
   and insert a row into any synced table so uploadData runs and throws.
2. Observe: no retry after retryDelay; further local writes never trigger
   uploadData; SyncStatus.uploadError stays null; the main isolate gets an
   unhandled ArgumentError: Illegal argument in isolate message: object is unsendable … that nothing attributes to PowerSync.
3. Call disconnect() (or close()): it never completes. The log shows
   Ending Rust sync iteration but never Sync Isolate exit.
4. Kill and restart the app: ps_crud is intact and uploads normally.

With dio the trigger is simply: go offline, write a row, let the upload POST
fail once, rethrow the DioException as the docs' examples do.

Analysis (powersync 2.3.3 sources; same on main)

1. lib/src/database/native/native_powersync_database.dart — uploadCrud
   messages are handled on the main isolate with
   await (payload as PortCompleter).handle(() => connector.uploadData(this)).
2. lib/src/isolate_completer.dart — PortCompleter.handle catches the
   error and calls completeError(error, stacktrace), which does
   sendPort.send(PortResult.error(error, stackTrace)) — the raw error
   object. For an unsendable error, send throws ArgumentError
   synchronously; the throw happens inside the catch block, so it escapes
   handle() as an unhandled async error on the main isolate, and no reply
   is ever sent to the sync isolate.
3. lib/src/sync/streaming_sync.dart — _uploadAllCrud awaits
   connector.uploadCrud() inside crudMutex.lock(...). With no reply, that
   future pends forever, _activeCrudUpload never completes, and the crud
   loop never processes another trigger.
4. streaming_sync.dart — abort() does
   await Future.wait([future, if (_activeCrudUpload case final u?) u.future]),
   so teardown blocks on the same pending future: disconnect()/close()
   hang.
5. native_powersync_database.dart, _syncIsolate.shutdown() — the one
   thing that would release the pending future, results.close() (it
   completes pending IsolateResults with a StateError), runs after
   await openedStreamingSync?.abort() — i.e. after the await that is hung
   on it, so the escape hatch is unreachable.
6. Aggravating: _RemoteMutex.lock in
   lib/src/database/native/sync_isolate_protocol.dart accepts an
   abortTrigger parameter and ignores it, so the crud mutex acquisition
   path has no timeout safety on native either.

Verified experimentally with dio 5.10.0 (Dart 3.x): a DioException whose
RequestOptions carries a CancelToken fails SendPort.send/Isolate.spawn
with Illegal argument in isolate message: object is unsendable - Library:'dart:async' Class: _AsyncCompleter (path:
_completer in CancelToken <- cancelToken in RequestOptions <- requestOptions in DioException). Without a CancelToken the same
DioException happens to be sendable — which makes the failure look
nondeterministic across apps while being fully deterministic within one.

Suggested fixes

Any one of the first two prevents the permanent wedge; all three seem worth
doing:

1. Sanitize at the port boundary. In PortCompleter.completeError, try
   the send and fall back to a stringified error (e.g. a RemoteError with
   error.toString() and the stack trace string) when send throws. The
   sync isolate then sees a normal upload failure and the documented
   retry-periodically behavior is preserved for every error type.
2. Make teardown robust. In _syncIsolate.shutdown(), complete pending
   IsolateResults (results.close()) before awaiting
   openedStreamingSync.abort() — or bound the _activeCrudUpload wait —
   so disconnect()/close() cannot hang on a lost reply.
3. Honor abortTrigger in _RemoteMutex.lock (or drop the parameter on
   that path so the API doesn't suggest a safety that isn't there).

Verification of the diagnosis

Mapping every uploadData error to a strings-only exception type on our side
(no other change) fully resolves the issue in the same on-device scenario:
failures now surface as Data upload error warnings with
SyncStatus.uploadError populated, the SDK retries as documented while
offline, and the queue drains by itself within ~0.4 s of the stream
reconnecting. disconnect() completes normally again ("Sync Isolate exit"
returns to the log).

Environment

• powersync: 2.3.3 (also inspected current main — same code paths)
• dio: 5.10.0 (real-world unsendable error; any unsendable error reproduces)
• Flutter: stable, iOS 18 physical device (native sync isolate path)
• Backend: custom (PowerSync service + REST upload endpoint)

Device log excerpt

[ps-chat] ↑ POST /api/v1/powersync/upload — 1 op(s)        ← uploadData runs (offline)
[ps-chat] ↑ upload FAILED (connectionError status=null)    ← DioException rethrown
[PowerSync] INFO: Starting Rust sync iteration             ← stream retries fine…
… network returns, stream reconnects …
[PowerSync] INFO: Could not apply checkpoint due to local data. Will retry at completed upload or next checkpoint.
… and nothing else, ever: no further uploadData calls, no warnings …
… later, disconnect() requested:
[PowerSync] INFO: Ending Rust sync iteration. Immediate restart: false
(no "Sync Isolate exit"; disconnect() never returns)

[simolus3]

Nice catch! #452 could also be caused by this. I agree with the fixes, instead of serializing the error we can also assign an opaque token to it that we then deserialize if the exception ends up in a sync status. I'll take a look at this next week.

[simolus3]

2. Make teardown robust. In _syncIsolate.shutdown(), complete pending
IsolateResults (results.close()) before awaiting
openedStreamingSync.abort() — or bound the _activeCrudUpload wait —
so disconnect()/close() cannot hang on a lost reply.

I've investigated that, but I think it's a dangerous thing to do. For example, an initial uploadData call could take a long time to upload a large batch of items before completing it. If you then call disconnect() while that's still ongoing, we'd abort the sync isolate (and return the crud upload lock) while the upload on the main isolate is still ongoing. If you then connect again, there's a chance we'd have two concurrent uploadData calls uploading the same data twice, which is not good.

A potential fix could be close results first in the background isolate, but also await the underlying call in the main isolate before marking the database as disconnected. But then we'd be unable to report errors.

With the main item fixed, I don't think we need the rest.