feat: scale-out phase D: Optio Local relay across API instances, replica rule lifted

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Scale-out, phase D: Optio Local across API pods, and any number of API replicas** (`docs/plans/scale-out.md` §4, "Several API instances" in `docs/optio-local.md`). A daemon is one socket on one API instance, the host's **owner**, which holds the lease `local-host:<id>` (30 s, renewed every 10 s, held per daemon connection); viewers, REST calls and workers on any other instance reach it over the **Local bus** (`services/local-bus.ts`: one Redis subscriber per instance, started after `listen` and retried with backoff; binary envelopes, so terminal bytes cross without base64; sharded pub/sub in `REDIS_MODE=cluster`, so a channel keeps publish order). Everything for a daemon goes on `optio:local:host:<id>:in`, which only its owner subscribes to, naming the lease holder its sender read; what a terminal's viewers get goes on `optio:local:term:<id>:out`, subscribed only while an instance has a viewer of it, and published only while a viewer is on another instance; answers come back on `optio:local:reply:<instance>`. A viewer on any instance gets the attach snapshot and live output in order and its input, resize and view reports reach the daemon, with the grid arbiter kept by the owner; spawn and kill are acknowledged by the owner: nobody listening counts as offline (a spawn parks, a kill settles the row, as before), while an owner that heard a frame but didn't acknowledge it in 5 s counts as unknown — a spawn stays `launching` and is never sent again (`started` settles it from `launching` or `pending`; the stuck-launching sweep fails it otherwise) — so **a kill through another instance no longer marks the terminal exited while its PTY keeps running, and a spawn no longer parks as host-offline**; dirs, limits and transcript backfill get their daemon's answer relayed back to the instance that asked, in order per host; **a Claude token never crosses Redis**: a token refresh asked on another instance runs entirely on the daemon's instance and only the outcome comes back. "The newest daemon wins" across instances: a new connection seizes the lease and publishes `host-taken`, which only makes the others re-read the lease — a connection gives the host up (without marking it offline) only once its own claim has completed and the lease names another connection, so two daemons connecting at once leave exactly one; frames for a connection that arrive before its claim completes are handled right after it; a disconnect releases only its own connection's lease (one that closed mid-claim gives it back once the claim lands, before the offline mark), and a host is marked offline only while no connection holds one. Online is the lease everywhere: `isHostOnline`, and `getHost` / `listHosts` / `pickOnlineHost` derive `state` from it in the same query (the column stays the last-known state for alerts and the sweep); what a daemon can do is stored on the host row (`local_hosts.capabilities`, new migration) and read off the rows already loaded, so the host list makes no query per host. Every 10 s (one tick at a time, one lease query for every host) each instance reports its remote viewers whose attach has gone out to their owners and checks the owner still holds the host: viewers of a daemon that moved (or whose owner died) are closed with 4503 and reattach, as the clients already do. The liveness sweep is a BullMQ job scheduler (`local-sweep.sweep`) under the `poller:local-sweep` lease, and also marks offline hosts whose lease lapsed and starts terminals parked as host-offline whose host is back; the automatic Claude token refresh window is a lease shared by every instance. **The chart no longer requires `api.replicas: 1`** or forbids autoscaling (the HPA owns the count when enabled; the Deployment omits `replicas` then); rollouts stay `Recreate`. Coverage: `local-bus.test.ts` and `local-relay-cluster.test.ts` (routing, request/reply, handover, claim backlog, ordered relayed replies, owner tasks), `local-bus.int.test.ts` and `local-cluster.int.test.ts` (real Redis and Postgres: two bus instances, lease seized on `host-taken` or at renew, a late acknowledgement sends a spawn once, a close mid-claim leaves no lease, online from the lease), `redis-cluster.int.test.ts` (1000 frames in order through each bus on the three-master test cluster), and `apps/api/e2e/scale-out-local.e2e.test.ts` on three real API servers — a viewer on B streaming a daemon on A, spawn / kill / dirs / limits / input through B, the daemon reconnecting to B, A SIGKILLed while it owns the host, a local Job run started through B, every server agreeing on online state, and two hellos at once on A and B leaving exactly one connection, 20 times over. The e2e Local test clients are shared (`test-utils/e2e/local-fakes.ts`).
+
 - **Scale-out, phase B: agent runs outlive the API process that started them** (`docs/plans/scale-out.md` §1–2, `docs/reconciliation.md` → Attachment). Repo Tasks, Job runs, persistent-agent turns and PR-review runs all start under the run protocol and are streamed through one attach loop (`services/run-attachment.ts`): the attached instance holds a lease on the run (60 s, renewed every 20 s, `attached_by` / `attach_lease_until`), reads the agent's output file from byte 0, replays what earlier attachments stored into its in-memory derivations (PR tool calls, session id, terminal event, the result and auth-failure digest, a command's exit receipt) and stores the rest as log rows **with the consumed offset in one transaction**, so a run that changes hands gets no duplicate line and no gap. No worker keeps a run's whole output in memory any more: each agent adapter reads its result line by line (`createResultParser`). A run's exit and its spend are recorded once (`settleRun`), whichever instance finishes it. When an instance goes away its runs' leases expire; an attach sweep (every 30 s) and the reconciler's new `reattach` action hand them to another instance, which claims each by CAS; a run whose starter went away before its agent started is put back by its kind's own retry (`requeueRun`), and a stall counts only while a run is attached. On SIGTERM an instance gives its runs up at once. Cancel (a task, a Job run, a review, an archived agent's turn) signals the agent's process group from whichever instance takes the request, and a mid-run message to a task is appended to its agent's stdin by the route; the Redis channel only notifies viewers. A boot no longer fails or flags runs in flight. A task or review run is `running` (an agent `running` a turn) only once its agent started. A job that started or claimed an agent and ends without finishing its run (an error while setting it up, a cancel or requeue under it) stops that agent, scoped to its supervisor's pid, while an error once the agent streams (a database or parser failure) only gives the lease up so another job attaches — after three failed attach jobs in a row the run fails and its agent is stopped; every path back to `queued` (a force-redo too) stops the last attempt's agent; a re-attach never claims a run already finished, nor one a job of the same instance holds (`attached_by` names the hold); a review cancelled while its run is set up cancels the run and writes nothing to the review; on SIGTERM a run past its stream finishes settling first (up to 20 s, `OPTIO_DETACH_SETTLE_WAIT_MS`); a fresh start stops a stale supervisor TERM → KILL and moves the old attempt's files aside; a message sent after the agent finished its turn is marked undelivered; a Claude `result` event decides a task's or review's success whatever the process's exit code (other agents keep their failure-signal rule, the exit code unused, as before); a re-attach reads a Job run or turn by what the attempt ran; and a failed, lost or silent run's error carries the tail of the agent's stderr. A pod row a server abandoned mid-creation (still `provisioning` past `OPTIO_STALE_PROVISIONING_MS`, 10 min) is taken over by whoever needs the pool — a persistent agent's pod too, which before waited on it forever — and a waiter waits only until then.
 - **Scale-out, phase C2: sessions, glance and skills on every API pod** (`docs/plans/scale-out.md` §3). A pod session's chat turn still runs on the API instance that received the prompt, but its frames now reach a viewer connected to any instance: every instance subscribes once to the **session bus** (Redis channel `optio:session-bus`, `services/session-bus.ts`; after the server listens, retried with backoff, so REST serves while Redis is down) and delivers what arrives to its own sockets, while the instance that published acts on its own message first and skips it when it comes back, so a local viewer never waits on Redis. **Interrupt, end-session and share revocation work across instances**: an interrupt is recorded on the running turn's row (`finished_at` set while the turn is still `running` — no other write produces that), nudged over the bus naming that turn, and honored by the instance running the turn on its next two-second control poll even if the bus message was lost (`watchTurnControl`); a session ended or a share revoked through one instance closes the sockets held by every other one, and the 10 s recheck now also closes a socket whose session has ended. The instance running a turn holds the lease `chat-turn:<id>` (60 s, renewed every 20 s), and recovery — at boot and on every 30 s housekeeping sweep — settles only `running` turns whose lease has lapsed, so a replica booting never interrupts a live turn and a crashed instance's turn is settled within about a minute. The **glance / push bookkeeping** that lived in one process's memory — the last attention state pushed per terminal, the last needs-you and running counts per user, hosts already alerted offline, and the two timers (the Watch's quiet-period end, a snooze's expiry) — is rows in `glance_state` (`services/glance-state-service.ts`), shared by every instance; a terminal's previous attention state is read and replaced in one locked step, so a transition taken by several instances alerts once, an outage is claimed with one insert, and a change that changes nothing writes nothing; timers are due rows fired by whichever instance's sweep claims them (`workers/glance-sweep-worker.ts`, a BullMQ job scheduler with a stable id every 30 s, run under the `sweep:glance` lease, the rows claimed atomically so a timer fires once), up to one interval late; the APNs / FCM one-second Watch coalescer stays per instance (worst case one duplicate push, folded by the collapse id). **Marketplace skills live in the database**: the sync worker writes every file of a synced skill to `installed_skill_files` with the manifest in one transaction, a run spawned on any instance reads them there (`readInstalledSkillFiles`), a `.optio-synced` sentinel row marks a skill synced at its commit (a skill with no files included), a re-sync at the same commit is a no-op, a skill synced before the upgrade (a commit but no sentinel) is due again on the pass the worker enqueues at boot, syncs of one skill collapse into one queued job (`sync-one-<id>`) and overlapping writes upsert, and the clone is scratch under `/tmp` (`OPTIO_SKILLS_SCRATCH_DIR`). The chart's `installedSkillsCache` PVC, volume and mount are gone (the value is ignored; `helm upgrade` deletes the claim — `docs/production-eks.md`).
 
@@ -27,6 +29,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **Session chat frames keep their order in `REDIS_MODE=cluster`.** A broadcast `PUBLISH` reaches subscribers through whichever node took it, so two frames on the session bus could arrive swapped; the session bus (and the new Local bus) publish and subscribe through one choice in `redis-config.ts` (`publishOrdered` / `createOrderedSubscriber`): sharded pub/sub (`SPUBLISH` / `SSUBSCRIBE`, Redis 7) in cluster mode, plain pub/sub in standalone mode (`docs/redis.md`).
+- A Local daemon that reconnected while its old connection still looked open (a laptop waking up) left the terminals' viewers on the old connection, frozen: the new connection starts with no viewers, and nothing re-attached them. Their streams are now closed with 4503 so they reconnect to the new connection, as after any host blip.
 - The setup-file writer merges a runtime's project config into the repo's own file instead of replacing it (`merge: "json"`), and marks a tracked file it wrote over `skip-worktree`, so an agent's `git add -A` can never commit Optio's MCP servers and their credentials over a repo's committed `.cursor/mcp.json` or `.mcp.json`. The image entrypoint writes `/opt/optio/…` setup files into the agent's home like the pod exec scripts do.
 - Connections and providers can no longer set a runtime's config env (`GEMINI_CLI_HOME`, `OPENCODE_CONFIG*`, `COPILOT_HOME`, `CURSOR_CONFIG_DIR`) in a pod, as they already could not set `CODEX_HOME`.
 - iOS builds again from a fresh checkout: SwiftTerm is pinned to 1.20.0 in `project.yml`, since 1.99.0 (released 2026-10-08) removed `TerminalView.getTerminal()`, which the terminal theme and the local terminal stream call.