Skip to content

HS-1258249: fix uncatchable realtime 1008 error and endless 1-second reconnect loop - #327

Open
claudear wants to merge 2 commits into
mainfrom
fix/HS-1258249-realtime-policy-violation
Open

HS-1258249: fix uncatchable realtime 1008 error and endless 1-second reconnect loop#327
claudear wants to merge 2 commits into
mainfrom
fix/HS-1258249-realtime-policy-violation

Conversation

@claudear

Copy link
Copy Markdown

Problem

Reported in HS-1258249: a self-hosted user behind a Cloudflare tunnel gets this on Flutter web after the socket has been idle for a couple of hours, repeating once per second forever:

Reconnecting in 1 seconds.
DartError: AppwriteException: , Error: Server Error (1008)
package:appwrite/src/realtime_mixin.dart 385:7   handleError
package:appwrite/src/realtime_mixin.dart 107:13
dart-sdk/lib/async/zone.dart 891:9               runUnaryGuarded

Root cause

handleError treated a 1008 policy violation as fatal by throwing — but it runs inside the _websok.stream.listen onData callback. A throw there is forwarded to the zone's uncaught-error handler, so it:

  • never reaches the try/catch in _createSocket (which has already returned),
  • never cancels the subscription or closes the socket,
  • never reaches subscription.stream.listen(onError:), so app code cannot catch it, prompt for re-auth, or stop the client.

The fatal signal was simply discarded. The only thing that then stopped the reconnect loop was the incidental _websok?.closeCode == status.policyViolation check in _retry, and that check is unreliable — closeCode is null when the failure surfaces through the channel's onError path, and 1006/1000 when a proxy or tunnel terminates the connection without forwarding the server's 1008 close frame. In all of those cases _retry() reconnected straight back into the same rejection.

On top of that, _retries = 0 was reset immediately after the WebSocket object was created, before the application-level connected handshake. When the transport connects fine but the server rejects at the app level, the counter was wiped on every attempt, so _getTimeout() always returned 1 — matching the repeated Reconnecting in 1 seconds. in the report.

Fix

  • handleError no longer throws. It records the fatal state, stops the heartbeat, cancels the stream subscription, closes the socket, and delivers the AppwriteException to every subscriber's controller so applications can handle it.
  • _retry is gated on that recorded state instead of the racy closeCode check, and no longer resets _reconnect = true on its early return (which made the old guard one-shot).
  • _retries is reset in the 'connected' case, after the handshake succeeds, so the backoff actually escalates on post-connect failures.
  • onError and onDone no longer schedule two reconnects for a single failure (the browser channel emits error-then-done), and the stale _websocketSubscription is cancelled before being replaced.

A fresh _createSocket() — i.e. an explicit subscribeTo / upsertPresence after the app re-authenticates — clears the recorded fatal state, so the client is usable again.

Tests

New test/src/realtime_mixin_test.dart:

  • Reproduces the report: server sends the 1008 error frame and the socket then dies without the close code reaching the client (the tunnel case). Before the fix this failed with the exact handleError_createSocket.<fn> stack trace from the ticket and no error delivered to the subscriber. Now the AppwriteException reaches subscription.stream, the socket is closed, and no reconnect is scheduled.
  • Regression guard: a recoverable (non-1008) error still reconnects.

Full suite: 388 existing tests + 2 new ones pass. flutter analyze reports no new issues in the touched files.

🤖 Generated with Claude Code

…o the zone

A server-sent realtime `error` frame with code 1008 was rethrown from inside
the WebSocket stream listener. A throw there is forwarded to the zone's
uncaught-error handler, so it never reached the `try/catch` in
`_createSocket`, never cancelled the subscription, never closed the socket
and was never delivered to `subscription.stream.listen(onError:)` — app code
had no way to catch it and re-authenticate.

The only thing that stopped the reconnect loop afterwards was the incidental
`_websok?.closeCode == status.policyViolation` check in `_retry`, which is
unreliable: `closeCode` is null when the failure surfaces through the
channel's onError path, and 1006/1000 when a proxy such as a Cloudflare
tunnel terminates the connection without forwarding the server's 1008 close
frame. In those cases the client reconnected into the same rejection once a
second, forever.

- `handleError` now records the fatal state, stops the heartbeat, cancels the
  subscription, closes the socket and delivers the `AppwriteException` to
  every subscriber.
- `_retry` is gated on that recorded state rather than the racy `closeCode`
  check, and no longer resets `_reconnect = true` on its early return.
- `_retries` is reset on the application-level `connected` event instead of
  right after the transport connects, so the backoff escalates when the
  server keeps rejecting the connection.
- onError/onDone no longer schedule two reconnects for one failure, and the
  stale stream subscription is cancelled before being replaced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown

Greptile Summary

The PR makes policy-violation errors catchable, terminates rejected sockets, and prevents endless immediate reconnects.

  • Records fatal realtime errors and delivers them to subscribers.
  • Deduplicates reconnect scheduling across protocol errors and stream termination.
  • Resets backoff only after the application-level handshake.
  • Adds regression coverage for fatal and recoverable server errors.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
lib/src/realtime_mixin.dart The changes address both previous findings by forcing a fresh socket after fatal rejection and sharing retry deduplication across failure callbacks.
test/src/realtime_mixin_test.dart Tests cover subscriber-visible policy violations, socket shutdown without a propagated close code, explicit recovery, and recoverable reconnect behavior.

Reviews (2): Last reviewed commit: "fix(realtime): don't reuse a rejected so..." | Re-trigger Greptile

Comment thread lib/src/realtime_mixin.dart
Comment thread lib/src/realtime_mixin.dart
…failure

Addresses two gaps in the previous commit found in review:

- `_createSocket` reused the existing socket when the URL matched and
  `closeCode` was still null. After a policy violation that close is still in
  flight, so re-subscribing after re-authenticating pushed the pending
  subscribes into a dying connection and left `_fatalError` set, leaving the
  client permanently disconnected. The reuse branch now also requires that no
  fatal error is recorded.
- `handleError`'s recoverable branch called `_retry()` directly, so an error
  frame followed by the stream terminating scheduled two reconnects and
  counted one failure twice. Retry deduplication moved from a closure local to
  `_createSocket` into `_scheduleRetry`, which all three paths now use.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@claudear

Copy link
Copy Markdown
Author

Fix Confidence: 88/100

The reproducing test failed before the change with the exact stack frames from the customer report (realtime_mixin.dart 385:7 handleError ← 107:13) and passes after. Full suite of 388 existing tests still passes, analyze shows no new issues, and CI is green. I verified the socket-reuse guard is load-bearing by removing it and watching the test fail. Remaining uncertainty: the tests use a fake WebSocketChannel, so real-world platform close-code timing (IO vs HtmlWebSocketChannel) isn't exercised end-to-end, and the reconnect/backoff paths involve real-time delays that are covered only indirectly. The retry-dedup change alters retry counting on paths not directly asserted by a test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant