Skip to content

Sync never recovers from an application-level ServerError (e.g. "You are not logged in") — the cached client is never dropped #191

Description

@pguinet

AutoKuma: 2.1.0-rc.2 (also present in 2.0.0)
Uptime Kuma: 2.2.1 (MariaDB backend)
Sync interval: 60s

Summary

#157 fixed recovery after EngineIO errors, and it works. But the recovery path is gated on the error being classified as a connection error. When the WebSocket is re-established at transport level while the application session is gone, the resulting error is ServerError — which is not in that list — so the cached client is kept indefinitely and the sync stays dead until the container is restarted.

What we observed

The reverse proxy sitting in front of Uptime Kuma was restarted, causing a short outage (seconds).

  • Uptime Kuma was reachable again within ~2 minutes. This is not an assumption: a different AutoKuma instance was started during that window and connected successfully on its first cycle.
  • The already-running instance never synced again — 13 consecutive 60s cycles — until we recreated the container. It came back immediately after.
  • The health endpoint added in rc.2 correctly reported the problem (503 on /health, container went unhealthy). That is how we caught it. But nothing recovered on its own.

We could not capture the exact log line, as the container was replaced before we could pull its logs. What the behaviour does establish is that the error was not in the is_connection_error set, otherwise the client would have been dropped and reconnected on the following cycle.

Why (reading the code)

In Sync::run (autokuma/src/sync.rs), the client is only invalidated for connection errors:

if let Err(err) = self.do_sync().await {
    warn!("Encountered error during sync: {}", err);
    if Self::is_connection_error(&err) {
        debug!("Connection error detected, will reconnect on next cycle");
        self.client = None;
    }
}

And is_connection_error matches only:

KumaError::CommunicationError(_) | KumaError::ConnectionTimeout | KumaError::NotReady
    | KumaError::CallTimeout(_) | KumaError::NotAuthenticated | KumaError::Disconnected

get_connection reconnects only when self.client.is_none(), so anything outside that set means the dead client is reused forever.

Not covered, in particular:

  • ServerError(String) — the variant behind "Server responded with an error: You are not logged in", which is exactly the symptom of a lost application session
  • LoginError(String)
  • InvalidResponse(..) / UnsupportedResponse

Suggestion

Two options, not mutually exclusive:

  1. Treat a lost session as a connection problem — either add ServerError (possibly only when the message indicates an auth/session failure) and LoginError to is_connection_error, or re-login instead of reusing the cached client.
  2. More robustly: drop the client after N consecutive failed sync cycles, whatever the error class. Any error that persists across several cycles is, in practice, indistinguishable from a dead connection — and this also covers variants nobody has enumerated yet. The current design fails closed only for known error types, which is what let this one through after EngineIO read error after some days of uptime #157.

Happy to test a patch — we hit this reproducibly whenever the proxy in front of Uptime Kuma restarts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions