observability.md

Observability

Metrics

GET /metrics renders the Prometheus registry.

Metric	Type	Labels
gateway_requests_total	counter	route, status
gateway_request_duration_seconds	histogram	route
gateway_node_duration_seconds	histogram	node (pipeline stage)
gateway_tokens_total	counter	kind (prompt/completion)
gateway_cache_hits_total	counter	—
gateway_ledger_write_failures_total	counter	—
gateway_upstream_connect_retries_total	counter	account
gateway_upstream_status_retries_total	counter	account, status
gateway_thinking_signature_review_total	counter	result (match/mismatch/miss/no_evidence)
gateway_thinking_signature_cache_events_total	counter	event
gateway_mcp_requests_total	counter	server (configured name), method (initialize / tools/list / tools/call / resources / prompts / ping / stream / other), result (HTTP status, denied, upstream_error, reply_unreadable, masked, blocked)
gateway_model_fallbacks_total	counter	from, to

gateway_requests_total is recorded by router middleware, so every gateway response (the /metrics scrape itself excepted) — including error statuses and the realtime WebSocket upgrade — is counted, which makes error-rate dashboards possible. All labels are bounded by the config, never by a caller: route templates, status codes, protocol/stage names, configured MCP server names and configured model names (gateway_model_fallbacks_total) — no per-key or per-user cardinality.

Traces

Every request runs inside one span named by its route template (/v1/chat/completions, /v1/messages, …). It carries http.request.method, http.route, http.response.status_code and, once the pipeline has served the request, the same fields as the access log: gw.request_id, gw.surface, gw.model, gw.protocol, gw.account, gw.tenant, gw.ak_id, gw.user_id, gw.prompt_tokens, gw.completion_tokens, gw.decisions. A streamed response keeps its span open until the last frame, so the span duration is the whole turn; a 5xx marks the span as an error.

The span exports over OTLP/HTTP (protobuf) when the standard OpenTelemetry environment names a collector — nothing else to configure:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318   # or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
export OTEL_SERVICE_NAME=gw                                # default when unset
export OTEL_TRACES_SAMPLER=parentbased_traceidratio        # default parentbased_always_on
export OTEL_TRACES_SAMPLER_ARG=0.1

OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TIMEOUT apply as usual. An inbound W3C traceparent header joins the request to the caller's trace; the gateway's span is then a child of it. Without a collector the span is disabled at its call site and costs nothing per request; with one, span attributes are recorded in-process and batches leave on a background thread.

Access log

One structured line per successfully served request goes to stdout (via tracing; control level with RUST_LOG), carrying surface, request_id, ak_id (a stable SHA-256 fingerprint, never the bearer credential), product, user_id, model, protocol, account, prompt_tokens, completion_tokens, total_tokens, latency_ms, and decisions — the pipeline's routing trail for that request (quota fallback, variant pick, degrade, DLP/moderation outcomes). A streaming request whose pipeline fails answers 200 with an error frame, so it is logged too: one request failed warning on the same access target with surface, request_id, ak_id, product, tenant, latency_ms, the external code and the message (never the prompt); a client that closed the connection logs nothing. Non-streaming errors are counted by gateway_requests_total{status} rather than logged. request_id joins the access log to the ledger row and the audit events for the same request.

Billing ledger

GET /internal/ledger?limit=N (global admin token) returns the most recent N billing records, oldest-first within the page; count is always the true total, independent of the page size. Records persist when a SQLite or Postgres store is configured and can be capped with storage.ledger_max_rows, though the cap is not hard: pruning spares rows not yet folded into the usage rollup, so the table can briefly exceed it under rollup lag. Each record carries request_id, the access key, product, tenant, user_id (effective end user), the requested model and the served_model (differs after a quota fallback), protocol, account, token counts, billed_units (characters / seconds / search units on the unit-priced surfaces; also summed in /admin/usage), charged cost_micros and vendor_cost_micros, created_at_epoch_secs, the PTU-spillover flag, and an estimated flag (set when counts came from an aborted stream rather than a vendor usage payload). An aborted stream served by an OpenRouter account is billed from OpenRouter's record of the generation instead (GET /api/v1/generation, polled for about 40 s with each lookup capped at 10 s; cancelled generations included), with cache reads at their weight (the record carries no cache-write count, so a cache write bills at the prompt weight) and OpenRouter's cost as vendor_cost_micros; only when no record arrives, or the gateway starts shutting down, does it fall back to the estimate. Shutdown waits for these settles before the final ledger flush, and the access log's latency_ms stops at the end of the stream, not of the lookup. Writes are idempotent by request_id; a transient store failure is held in a bounded repair queue and retried with backoff — eight attempts, about 13 s — before the batch is dropped and counted in gateway_ledger_write_failures_total. With storage.ledger_spool_path set, the batch instead goes to that local JSON-lines file after three attempts (about 0.3 s; the directory is created on first use, and a spool that cannot be written falls back to the remaining retries), and a replay every 30 s inserts it in batches once the store accepts writes again; in the same transaction it adds the rows it inserted to any usage rollup bucket the watermark already passed, so a late row counts even after its bucket's other rows were pruned, and a repeated replay adds nothing (billing spool replayed in the log). The spool path is per-instance local disk; replicas must not share one file. A full queue applies backpressure to settlement until the repair worker makes room, so overload alone never discards an accepted row. Per-user usage additionally rolls into durable minute buckets every minute, so GET /admin/usage/users stays correct after ledger_max_rows pruning (see Governance).

Audit trails

Three operator-facing audit surfaces, all under the gated /admin prefix and covered in Governance: GET /admin/audit/events (content-safety hits, no prompt text), GET /admin/audit/ops (admin-plane mutations with source IP), and GET /admin/usage/users (per-user cost). Content retention, when a tenant enables it, is read back via GET /admin/audit/content/{request_id}.