feat(api): scale-out phase C1, coordination across API instances

[jonwiggins]

Phase C1 of docs/plans/scale-out.md: the coordination that has to hold once several API pods share one database. It builds on the phase A schema (leases, ws_upgrade_tokens, inbound_webhook_deliveries, ticket_sync_claims, the unique partial index on active PR reviews). The chart still pins one replica; that limit is lifted in phase D.

What moved where

Before (in process)	Now
claimLockChain promise mutex in the task and workflow workers	services/claim-lock.ts withClaimLock(keys, fn): one READ COMMITTED transaction, pg_advisory_xact_lock(hashtext(key)) on the global key (claim:tasks, claim:workflows; per-repo and per-Job limits are counted under it), lock_timeout 10 s (a timeout resolves null and the worker re-queues). The count and the CAS claim run on the transaction (taskService.claimTransitionIn(claim, …), claimWorkflowRunIn(claim, …)), so a lock holder never needs a second pooled connection; the announcement (events, webhooks, reconciler) is registered with claim.afterCommit and runs after the commit. transitionTask / transitionWorkflowRunCas are split into a write step and an announce step to support this; their behaviour is unchanged.
queue.add(…, { repeat }) + cleanRepeatJobs wiping every queue at boot	services/repeat-jobs.ts: upsertJobScheduler(<queue>.<tick>) with stable ids. A new scheduler first ticks one interval out, as the old repeatables did. cleanRepeatJobs and its boot calls are gone. Boot now runs removeLegacyRepeatables, which removes only the hash-keyed entries an older build registered.
Pollers overlapping across instances	services/poller-lease.ts underPollerLease('poller:<name>') for ticket sync, schedule checker, external PR review, repo cleanup, PR watcher, reconcile resync, skill sync, token validation. The config directory apply uses its own config-sync:apply lease. Each sweep is also safe when two do overlap: ticket sync claims ticket_sync_claims before creating a task; the schedule checker advances next_fire_at by CAS before firing (advanceScheduleCas); an external PR review launch names the review the sweep saw (seenVersion) and lands only while that is still the review; the unique-index conflict counts as someone else's launch.
Slack/Linear in-memory set, Redis SET NX delivery dedupe	services/inbound-delivery-service.ts: INSERT … ON CONFLICT DO NOTHING RETURNING into inbound_webhook_deliveries, used by every receiver (GitHub, GitLab, Slack, Linear, Jira, PagerDuty, Sentry).
(none)	workers/sweep-worker.ts: a housekeeping tick every 30 s under a lease, with a registerSweep(name, fn) registry. It deletes deliveries older than 24 h, expired ws_upgrade_tokens and leases expired over a day ago.
In-memory WS upgrade token map	ws_upgrade_tokens: the SHA-256 is stored, the token is consumed by DELETE … RETURNING, 30 s TTL.
Optio-chat activeConnections map	lease chat:<userId> per socket, renewed while open
GitHub user-token refresh lock (per process)	lease github-refresh:<userId>. A waiter reads the token the holder stored. The installation-token cache stays per instance (documented: GitHub mints as many tokens as are asked for).

Docs: docs/production-eks.md covers what is shared through the database and what stays per pod. The per-IP WS cap is per replica. CHANGELOG.md has an entry under [Unreleased].

Behaviour notes:

• Ticket claims and deleted tasks: a claim whose task was deleted can be taken over, so a deleted ticket task is re-created as it was before claims existed. A claim with no task (its sweep died between the claim and createTask) can be taken over after 10 minutes. Both takeovers happen in the same single statement, so two racing sweeps still get one claim.
• Legacy repeatable cleanup at boot: cleanLegacyRepeatJobs removes any repeat entry on a known queue whose key is not a scheduler id this build knows. That keeps an upgraded deployment from ticking twice. Phase B's attach-sweep scheduler must be added to REPEAT_SCHEDULERS when it lands, or every boot will remove it before the worker upserts it again.

Tests

• Integration (services/scale-out-coordination.int.test.ts):
  • two claimers under a limit of 3 over 50 iterations each, with runs finishing at random, never exceed the limit. A watcher polls the count throughout, and the test fails with the advisory lock removed.
  • the claim locks never deadlock.
  • two poller-lease holders: one runs.
  • ticket claims: one of two wins, release works, and both takeovers work.
  • two concurrent syncAllTickets() create one task.
  • two schedule CAS calls: one wins.
  • two concurrent checkDueScheduleTriggers() fire once.
  • two concurrent sweepExternalPrs() launch one review and one run, and relaunch on new commits once.
  • one inbound delivery: one of two receivers claims it.
  • one WS token: one of two upgrades consumes it.
  • schedulers: one per id, and legacy repeatables are removed.
• Pipeline e2e (e2e/scale-out-coordination.e2e.test.ts, two servers on startApiCluster, auth on):
  • a WS upgrade token minted on A is accepted on B, and A then rejects it with 4401.
  • one signed GitLab delivery posted to A and B fires one Job run.
  • a schedule trigger due now fires once with both servers sweeping every second.
  • after both boots there is exactly one scheduler per repeat job and no stray repeatables.

Review fixes

From the medium-effort review (15 items), plus one follow-up:

1. GitHub token refresh (with 15). A token is only ever refreshed under github-refresh:<userId>.
   • withLease gained wait: { timeoutMs, pollMs }. It is the one lease-wait implementation: the refresh and the config apply use it, and the housekeeping tick now uses underPollerLease.
   • Inside the lease the stored token is re-read first.
   • The GitHub call has a 15 s timeout (AbortSignal.any with the lease's signal).
   • A wait that times out returns the stored token if it still works, else null, so the caller falls back to the PAT. Nothing refreshes without the lease.
   • On bad_refresh_token the tokens are compare-and-deleted in SQL on the refresh-token row's IV (secret-service.deleteSecretsIfUnchanged).
   • Integration test: two instances refreshing one user make one GitHub call and keep the new tokens. Compare-and-delete is tested both ways.
2. getTokenForUser awaits the refresh, so a rejection falls back to the PAT. Unit test with a rejecting refresh.
3. claimKey has no length cap. Tested with a 400-character repo URL (the stress test) and a 400-character key.
4. A sync-one that finds the skill's lease held throws SkillSyncBusyError. The queue's defaultJobOptions give every job 8 attempts with exponential backoff, which works with C2's stable sync-one-<id> job id.
5. withClaimLock sets isolationLevel: "read committed" explicitly, with a comment saying why.
6. SET LOCAL lock_timeout = '10s': a timeout resolves null, which the workers already treat as re-queue with delay. Integration test: a holder that never commits makes a claimer give up after about 10 s without running.
7. Sync now answers 409 with a readable message on ConfigApplyBusyError (route test).
8. A launch without seenVersion (a person's) relaunches the current review without a CAS. Integration test: a stale sweep view is refused, then a person's launch relaunches.
9. A failed recordTicketClaimTask is logged; the task is still queued and the provider loop goes on (unit test).
10. WS upgrade token expiry is set with now() + 30 s and compared with now() in the consuming DELETE … RETURNING (integration test).
11. Optio chat lease:
    • TTL 30 s, renewed every 10 s.
    • The release on close is awaited with a 2 s cap.
    • An acquireLease error closes the socket with 1011 and a readable message.
12. The inbound delivery claim runs with SET LOCAL statement_timeout = '1s'. On timeout the delivery is accepted with a warning (unit test).
13. The claim context has afterCommit(step). claimTransitionIn / claimWorkflowRunIn register their announcement there, withClaimLock runs the steps after the commit and drops them on rollback, and no caller handles an announcement. CLAUDE.md names claimTransitionIn as the in-transaction form of transitionTask.
14. The per-repo and per-Job advisory keys are gone; a comment says per-repo and per-Job limits are counted under the global lock.

• Follow-up: skill ref CAS. recordSyncResult takes the ref and subpath the sync read. In its transaction it locks the skill row (FOR UPDATE) and writes nothing if they changed. The PATCH that changes the ref clears resolved_sha in the same statement, so the skill is due at once. Integration test: the ref moves while a sync of main is in flight; nothing of main's sync lands, the skill is due, and the next sync writes the new ref's files.

Merge: origin/main with C2 (#667) is merged in.

• skill-sync-worker.ts keeps C2's database-backed sync and its sync-one-<id> job id, with the lease and retry on top.
• The sync-due repeat job stays a scheduler (scheduleRepeat), with C2's boot pass.
• index.ts starts both the housekeeping and glance sweep workers. The glance sweep keeps its own upsertJobScheduler; its queue is not in REPEAT_SCHEDULERS, so the legacy cleanup at boot never touches it.

Results (after the review fixes and the merge with main)

• pnpm turbo test: 13/13 tasks pass (api 3056 tests).
• pnpm --filter @optio/api test:integration: 55 files, 450 tests pass.
• pnpm --filter @optio/api test:e2e: 25 files, 112 tests pass.
• pnpm format:check: clean.
• pnpm turbo typecheck: 13/13.