Skip to content

architecture.md

Latest commit

 

History

History
103 lines (91 loc) · 5.76 KB

architecture.md

File metadata and controls

103 lines (91 loc) · 5.76 KB
Raw

Architecture

Cargo workspace, 11 crates, strictly layered — lower layers never depend on higher ones:

server → {views, task} → handler → dag → {engines, state} → {models, protocol, config} → consts
Crate Layer Role
consts L0 error codes, the Protocol enum
models L1 request/response domain types, typed params, usage, cost
protocol L1 OpenAI / Anthropic wire types + cross-protocol conversions
config L1 YAML config loading (conf/gateway.yaml)
state L2 auth, account pool, quotas, rate limits, ledger, batch/file stores (in-process defaults; Postgres/Redis fleet backends)
engines L2 engine implementations behind the Transport seam, SSE + AWS EventStream decoding, usage extraction, SigV4
dag L3 4-layer pipeline executor + nodes
handler L4 online/offline orchestration, DLP/blocklist plugins
task L5 background tasks (quota reset, content purge, usage rollup, availability flush/alerts, webhook dispatch)
views L5 axum HTTP/WebSocket handlers, protocol conversion
server L6 binary entrypoint: config + state + transport wiring

Request flow

client ──► views (auth, parse, protocol normalize)
       ──► handler (pre plugins: blocklist, moderation, then DLP redact)
       ──► dag: preprocess        resolve model, quota check, cache lookup
              account_select      priority / PTU-first / cooldown-aware selection, latency-ranked tiers when enabled
              model_access        rate limits, engine call, failover on an upstream fault
              post_process        usage → billing ledger, cache store
       ──► handler (post plugins) ──► views (JSON or SSE re-emit)

A client disconnect does not cancel the pipeline: every request runs on its own task (run_pipeline / spawn_stream_pipeline), so once admitted, quota and ledger accounting run to completion even if the caller goes away. Spawned background work (offline batches) likewise outlives the submitting request.

The DAG executes four fixed layers; nodes within a layer run sequentially in declaration order. Within model_access, call_engine reselects once on an upstream 5xx or a 401/402/403 credential or billing refusal: the failed account is excluded and another of the model's accounts is tried; a PTU→paygo switch is recorded as ptu_spillover in the ledger. Around the DAG, the handler re-runs all four layers for the next entry of the model's fallback_models chain when a run ends in an upstream fault before any byte was sent.

Seams (traits)

Every boundary to the outside world is a trait with a deterministic default, so the whole pipeline is testable offline:

  • Transport — engines never own an HTTP client. The server default is DispatchTransport: accounts without an endpoint resolve to mock:// sentinel URLs served in-process by MockTransport (deterministic vendor-shaped replies); real URLs go over HttpTransport (reqwest + rustls). GW_TRANSPORT=mock forces zero egress, GW_TRANSPORT=http disables the mock. Tests inject MockTransport directly.
  • TokenEncoder — prompt token estimation capability (tiktoken cl100k_base BPE default, zero-dependency heuristic fallback). cost_calc uses it to bill an aborted stream from the delivered text when the vendor's usage frame never arrives and the account keeps no generation record (OpenRouter does: engines::generation); token-aware PTU sizing is not ported yet.
  • Store — durable records (billing ledger, uploaded files, batch jobs). MemoryStore by default; SqliteStore when storage.sqlite_path is configured; PostgresStore when storage.postgres_url is — shared across a fleet.
  • KeyStore — the live access-key table. AkAuth (in-process DashMap) by default; PostgresKeyStore with storage.postgres_url — fleet-shared behind an auth cache each write invalidates fleet-wide by NOTIFY, admin key CRUD survives restarts.
  • UserBudgetStore — per-user budget overrides inside a tenant. In-process by default; PostgresUserBudgets with storage.postgres_url, cached and invalidated the same way as keys.
  • HealthStore — account cooldown/recovery, per account and served model, so one model's refusals never bench the account for the others. In-process breaker by default; RedisHealth with storage.redis_url — a tripped account is skipped for that model by every instance. Only upstream faults count (5xx, timeouts, broken streams, a 401/402/403 credential or billing refusal), never a client-caused or rate-limit error.
  • Governance — rate/quota/TPM counters. In-process by default; RedisGovernance shares them (including pooled tenant QPS) fleet-wide.
  • Moderator — the external content review behind security.moderate; allow-all by default, AWS Bedrock Guardrails when moderation: names one.
  • config store — with storage.postgres_url, config is versioned documents in Postgres (PUT /admin/config publishes; a LISTEN/NOTIFY change feed reloads every instance).
  • metrics facade — metrics crate macros throughout; the server installs a Prometheus recorder and serves /metrics.
  • Planned: see the issue tracker.

Testing

  • Unit tests per crate; engine golden tests assert exact request wire shapes (via a recording transport) and response/usage parsing against recorded vendor fixtures.
  • crates/server/tests/e2e.rs boots the full router in-process and exercises every API surface offline.
  • live_path.rs drives the real reqwest stack against a loopback vendor to prove the go-live path end to end.