Distillery

Distillery is an open-source multi-provider AI gateway built to route, manage, and refine your AI API traffic.

Designed for developers, it provides a unified endpoint for supported models, with provider-aware routing, centralized credentials, policy controls, observability, and an opt-in data redaction engine powered by Tonic AI's Textual service to help keep sensitive text out of supported upstream prompts.

Distillery supports native provider APIs as well as OpenAI-compatible clients. Redaction is off by default and only applies to supported request dialects when explicitly enabled. Unsupported redaction paths fail closed by default.

The current release does not provide model-response caching or generic load-balancing across arbitrary backends. Its caches are internal operational caches (for example, configuration and buffered metrics), while provider-aware routing and credential mapping are the shipped traffic-management features.

Why

Production LLM traffic crosses provider boundaries, carries sensitive content, and needs operational controls that should not be reimplemented in every client. Distillery addresses that boundary with zero client changes, so you can:

Centralize credentials and routing across OpenAI-compatible and native provider APIs.
Observe usage and cost with normalized records and persistent metrics.
Apply operational policy such as proxy-issued keys, quotas, kill switches, and supported text redaction.
Capture provider-neutral interactions for analytics, evaluation, audit, or downstream data workflows.

The proxy only ever produces interaction data. It requires no downstream database or storage service — it either writes JSONL to disk or POSTs batches to an endpoint you control (via a published OpenAPI spec). Optional SQLite files power local usage metrics, quotas, and the searchable admin index. Everything downstream — de-identification, retention, analytics, and downstream data workflows happen on your side.

What it does

OpenAI SDK compatibility — clients change one line (base_url). Works with the OpenAI SDKs, curl, or any client that speaks the supported wire formats.
Low-latency streaming — Server-Sent Events and provider-native streams are relayed as they arrive. Streaming capture retains only a bounded raw prefix and performs bounded incremental metadata observation; non-streaming responses are buffered so response hooks can safely transform them. Capture adds small per-chunk processing, and transport bytes/chunk boundaries are not promised to be byte-for-byte identical.
Structured capture — each supported call becomes a provider-neutral InteractionRecord: the raw payloads as observed by the proxy after enabled transformations (including request hooks and, when configured, redaction), subject to capture limits, plus normalized fields (messages or Responses input, tools, token usage, finish reason, timing, and the model that actually served it).
Multi-vendor routing — native dialects plus supported OpenAI-compatible surfaces. Clients can speak each vendor's native dialect (OpenAI Chat/Responses, Anthropic Messages, Gemini AI Studio generateContent, Vertex generateContent, Bedrock Converse/InvokeModel) or use a provider's supported OpenAI-compatible endpoint — the proxy preserves the request body and applies any provider-specific upstream URL/path and auth adaptation server-side. Six providers ship today: OpenAI, Anthropic, Gemini AI Studio, Azure OpenAI, Vertex AI, and Bedrock. Adding another is one ProviderEgress subclass.
Labeled proxy keys + 4-level identity — issue dist-… keys with a human label; the proxy maps ProxyKey → Tenant → VendorCredential → ProviderEgress and swaps in the real vendor key server-side. Only key hashes are stored.
Passthrough or mapping mode — run with no config (forward the client's own key to a single upstream) or with the YAML config (proxy keys map to vendor keys per tenant/use-case).
Password-gated admin UI at /admin — masked config view (Raw JSON + collapsible tree) + sortable, filterable usage-metrics tables and offline bar charts (tokens in/out, request counts) sliced by proxy-key label · tenant · vendor credential · provider · model.
Persistent usage metrics — SQLite (stdlib sqlite3, zero deps) fronted by in-memory counters; flushed at least every 60s and on shutdown.
Pluggable sinks with backpressure — capture to local disk or to an HTTP receiver. A bounded queue + drain worker decouples capture from serving; failed HTTP deliveries are retained in a durable spool for explicit replay with distillery-replay-spool, and the queue sheds load loudly rather than ever blocking a request.
Pre/post-LLM pipeline seams — formal hook interfaces let you inject request transforms (e.g. stream_options.include_usage) and post-response side effects (capture, metrics) without touching the forwarding core.

Supported provider surfaces

Provider	Native/provider-specific input	OpenAI-compatible input
OpenAI	Chat Completions, Responses	Chat Completions, Responses
Anthropic	Messages	Chat Completions passthrough
Gemini AI Studio	`generateContent`	Chat Completions
Azure OpenAI	Azure OpenAI paths using the OpenAI wire format	Chat Completions, Responses
Vertex AI	`generateContent`	—
Bedrock	Converse, InvokeModel	—

The proxy forwards unmatched catch-all routes transparently, but canonical normalization is only available for the documented ingress adapters. Unsupported shapes can still be captured as raw payloads when capture is enabled.

How it works

                         ┌──────────────────────── Distillery ──────────────────────┐
  client  ──Bearer──▶    │  auth (proxy key → tenant)                                │   ──▶  OpenAI / Azure / …
 (OpenAI SDK,            │  forward + key swap                                       │   ◀──  (streamed)
  base_url=proxy)  ◀──── │  stream response back  ──┐                                │
                         │                          └─▶ tee → InteractionRecord ─┐   │
                         └───────────────────────────────────────────────────── │ ──┘
                                                                                 ▼
                                                       bounded queue → sink:  disk (JSONL)
                                                                              or  HTTP (ingest spec)

The captured record is "bronze": raw payloads as observed by the proxy plus normalized fields. Everything past bronze — de-identification, enrichment, retention, and downstream analytics — is the receiver's job, never the proxy's.

Quickstart

Requires uv (it fetches a suitable Python 3.12+).

# Run from the standalone repository root.
uv sync
DISTILLER_HOST=127.0.0.1 uv run distillery  # local-only; api.openai.com; summary-only capture

Raw capture is opt-in. Review the capture safety section before enabling it:

DISTILLER_CAPTURE_MODE=disk uv run distillery

Point any OpenAI client at it (no-auth mode forwards your own key):

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"say hi"}]}'

When raw disk capture is enabled, the captured record lands under ./captures/:

cat captures/*/interactions-*.jsonl | tail -1 | python -m json.tool

With the SDK it's just a base-URL change:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1")  # api_key from env as usual

Configuration

All settings are environment variables (prefix DISTILLER_) or a .env file. See .env.example.

The distillery entrypoint loads .env into the process environment once at startup, without overriding variables already injected by the process. This also makes SDK-managed credentials such as Botocore's AWS credential chain available when running directly with uv run distillery.

Core

Variable	Default	Description
`DISTILLER_HOST` / `DISTILLER_PORT`	`0.0.0.0` / `8080`	Bind address.
`DISTILLER_UPSTREAM_BASE_URL`	`https://api.openai.com`	Upstream used in passthrough mode (`CONFIG_SOURCE=none`).
`DISTILLER_LOG_LEVEL` / `DISTILLER_LOG_JSON`	`INFO` / `false`	Logging.
`DISTILLER_ADMIN_PASSWORD`	(unset)	Password for `/admin` (HTTP Basic). Empty value locks `/admin` (503).
`DISTILLER_METRICS_DB_PATH`	`./metrics.sqlite3`	SQLite file for persistent usage metrics.
`DISTILLER_METRICS_FLUSH_S`	`60`	Max seconds between in-memory→SQLite flushes.
`DISTILLER_UPSTREAM_IDLE_TIMEOUT`	`120`	Maximum idle time between upstream response bytes/chunks. This is not an overall request deadline.
`DISTILLER_MAX_IN_FLIGHT_REQUESTS`	`128`	Per-process admission cap; excess requests receive 503. Set `0` only for explicit unlimited operation.
`DISTILLER_MAX_REQUEST_BODY_BYTES`	`16777216`	Maximum buffered inbound request size; oversized requests receive 413.
`DISTILLER_INTERACTION_INDEX_ENABLED`	`false`	Master on/off for the interaction index. When off, the admin Interactions tab shows the empty state and the JSON endpoints return 503.
`DISTILLER_INTERACTION_INDEX_DB_PATH`	`./interactions.sqlite3`	SQLite index file. In docker-compose point at a mounted/named volume so the index survives restarts.
`DISTILLER_INTERACTION_INDEX_MAX_AGE_DAYS`	`30`	Retention by age (days); `0` disables the age cap.
`DISTILLER_INTERACTION_INDEX_MAX_BYTES`	`0`	Retention by size (bytes); `0` disables the size cap.
`DISTILLER_INTERACTION_INDEX_PRUNE_INTERVAL_S`	`3600`	How often the pruner runs (seconds).
`DISTILLER_QUOTAS_ENABLED`	`true`	Master on/off for fixed-window quota enforcement. When off, the request path never touches the quota store; when on, enforcement additionally requires per-tenant auth (`CONFIG_SOURCE=file`/`api`) and at least one `policies:` entry on the tenant or key.
`DISTILLER_QUOTAS_DB_PATH`	`./quotas.sqlite3`	SQLite file for the fixed-window quota counters. In docker-compose point at a mounted/named volume so in-window enforcement state survives restarts.
`DISTILLER_PRICING_FILE`	(unset)	Optional YAML file of per-model prices merged over the bundled defaults at startup — for adding new models or overriding a rate without a code change. See Pricing overrides.
`DISTILLER_FINGERPRINT_SALT`	`dist-fingerprint-v1`	Salt for vendor-credential fingerprints; must be stable across restarts.
`DISTILLER_FORWARDED_ALLOW_IPS`	`127.0.0.1`	Comma-separated list of immediate peer IPs whose `X-Forwarded-` headers we trust to identify the real client. `""` trusts all peers — required when running behind ngrok or Docker (the immediate peer is the bridge/edge, not the real client). Only set `"*"` when only your own reverse proxy can reach the bind address — any peer can otherwise spoof `X-Forwarded-For`.
`DISTILLER_CAPTURE_CLIENT_IP`	`truncated`	How to record the caller's IP on each `InteractionRecord.network`. `truncated` (default; privacy-safer) zeros the last IPv4 octet (`/24`) or masks IPv6 to `/48`; `full` stores the raw IP (PII — opt in; see below); `hashed` is a salted-HMAC pseudonym using `DISTILLER_CAPTURE_IP_HASH_SALT`; `off` records `None`. Applied symmetrically to every trusted hop of `X-Forwarded-For`.
`DISTILLER_CAPTURE_IP_HASH_SALT`	(unset)	Separate deployment secret required when `DISTILLER_CAPTURE_CLIENT_IP=hashed`; use at least 16 characters and do not reuse `DISTILLER_FINGERPRINT_SALT`. Hashed IPs are pseudonyms, not irreversible anonymization.
`DISTILLER_CAPTURE_ALLOW_INSECURE_HTTP`	`false`	Allow an `http://` capture receiver. Keep `false` except for a local development receiver; production capture endpoints must use HTTPS.
`DISTILLER_REDACTION_MODE`	`off`	Global default for bidirectional PII redaction. `off` bypasses redaction entirely (zero request-path cost); `outbound` redacts supported request text before upstream forwarding and keeps capture/index data redacted; `full` additionally attempts non-streaming response restoration via a request-scoped in-memory mapping. Explicit per-tenant / per-key `redaction_mode` values in `config.yaml` OVERRIDE this default (closest-wins: key > tenant > global). Unknown values fail validation at startup.
`DISTILLER_REDACTION_ADAPTER`	`tonic`	Which redaction adapter to wire at startup. `tonic` is the v1 default (Tonic Textual).
`DISTILLER_REDACTION_FAIL_OPEN`	`false`	On adapter failure or an unsupported wire dialect, `false` (default) fails closed — the request is rejected and no raw text reaches upstream. `true` permits a byte-exact raw passthrough. GLOBAL-ONLY: tenants / keys cannot flip this.
`DISTILLER_REDACTION_TONIC_BASE_URL`	`https://textual.tonic.ai`	Tonic Textual base URL (non-secret). Override for regional / private deployments.
`DISTILLER_REDACTION_TONIC_API_KEY_REF`	(unset)	Env-var NAME (not a value) holding the Tonic Textual API key. Resolved through the same `SecretsBackend` as vendor credentials; the raw value stays in process memory and never lands in `Settings`, models, logs, errors, or captured records. Required when a non-`off` mode is active on any scope.
`DISTILLER_REDACTION_TONIC_TIMEOUT_S`	`5.0`	Per-request timeout (seconds) for the Tonic Textual adapter.

Capture (`DISTILLER_CAPTURE_MODE`)

Mode	Behavior
`log` (default)	Log a one-line summary per call. No persistence.
`disk`	Append raw durable JSONL under `DISTILLER_CAPTURE_DIR` (date-partitioned, size-rotated). Treat this directory as sensitive data.
`http`	POST raw batches to an HTTPS `DISTILLER_CAPTURE_ENDPOINT` (bearer `DISTILLER_CAPTURE_TOKEN`), spooling to `DISTILLER_CAPTURE_SPOOL_DIR` on failure. Cleartext HTTP requires the explicit insecure override above. Spool files are a durable dead-letter archive; replay them explicitly with `distillery-replay-spool`.

Tuning: DISTILLER_CAPTURE_MAX_QUEUE (10000), …_MAX_BATCH (100), …_MAX_QUEUE_BYTES (268435456 — 256 MiB approximate serialized-payload budget), …_MAX_BATCH_BYTES (67108864 — 64 MiB soft approximate serialized-payload target; a large singleton may exceed it), …_LINGER_MS (500), …_MAX_RETRIES (3), …_MAX_FILE_BYTES (67108864 — 64 MiB; rotation size for disk mode and the http-mode spool), and …_MAX_INTERACTION_BYTES (16777216 — 16 MiB retained response bytes per interaction). Streaming responses continue to the client after the capture limit, but the record is marked capture_truncated. A bounded protocol-metadata pass continues to extract final usage and finish information for metrics and quotas even when the raw response prefix is truncated. An oversized buffered response receives a sanitized 502 and is recorded as a proxy error.

Capture safety

disk and http modes may contain raw request and response payloads as observed by the proxy. Request capture reflects the final outbound body after enabled hooks and any configured redaction processing; response capture is decoded for processing and may be rechunked or truncated. These payloads may include prompts, tool data, uploaded-content references, model output, and caller metadata. The proxy does not encrypt capture files; use an encrypted volume or a trusted receiver, restrict filesystem permissions, and define an external deletion and backup policy. Size rotation is not a retention policy. HTTP spool files are not replayed automatically; use distillery-replay-spool after reviewing the receiver and retained data.

outbound redaction protects supported request text before it reaches the upstream or capture sink. It does not guarantee that newly generated sensitive text in a model response is safe. full can restore mapped values for supported buffered responses, but streaming responses remain redacted and unsupported dialects fail closed unless DISTILLER_REDACTION_FAIL_OPEN=true is explicitly chosen.

Redaction is off by default. The first supported adapter is Tonic Textual, which is optional and only becomes a request-path dependency when an operator enables outbound or full mode. This keeps the default proxy deployment independent of the redaction service while providing a concrete first-party integration for deployments that need it.

Deployment topology and forwarding boundary

The local SQLite stores for metrics, quotas, and the interaction index are process-local. They are appropriate for a single proxy instance and local or small internal deployments. Quota enforcement is not globally coordinated across multiple replicas, and metrics/index data is not automatically shared between instances. Before running multiple replicas, move these stores behind shared adapters (for example PostgreSQL, with a shared connection pool and migrations) or accept per-instance semantics explicitly. This is intentionally out of scope for the initial release.

The forwarding route is intentionally a catch-all: it preserves provider-native paths and bodies rather than maintaining a narrow list of endpoints. That means the proxy may forward operations it does not normalize—for example file uploads, model-management endpoints, or provider-specific administrative calls. Put the proxy behind authentication, rate limits, and an upstream/network policy when exposing it beyond a trusted local boundary. If the proxy's configured upstream or tenant configuration can be modified by an untrusted party, treat base-URL validation as an SSRF control as well; the catch-all route will attempt to reach whatever upstream it is given.

Auth / tenancy (`DISTILLER_CONFIG_SOURCE`)

Source	Behavior
`none` (default)	Passthrough mode. No proxy keys; the client's own key is forwarded to `UPSTREAM_BASE_URL`. The forwarded key is fingerprinted (salted HMAC) for attribution — matched to a registered `VendorCredential` if its fingerprint is known, otherwise bucketed as `unregistered:<fp8>`.
`file`	Mapping mode. 4-level config + proxy-issued `dist-…` keys from `DISTILLER_CONFIG_FILE` (YAML; JSON also accepted since it's a subset of YAML).
`api`	Same shape, fetched from `DISTILLER_CONFIG_API_URL` (bearer `…_CONFIG_API_TOKEN`). Endpoint may serve YAML or JSON (same shape as `config.yaml`).

Config is loaded once at startup and refreshed every DISTILLER_CONFIG_REFRESH_S seconds in the background — there is no per-request call to the config source, and no database access.

Caller network identity (trusted proxies + client IP capture)

Every captured InteractionRecord carries a network block — client_ip, forwarded_for, user_agent, and request_id — so a proxy key / fingerprint can be correlated to a real user or server, including in passthrough mode where an unrecognized key would otherwise only be an unregistered:<fp8> bucket. Network fields live in bronze (the per-interaction record) only; they are not added to the rollup dimensions (high cardinality).

Trusted proxies (DISTILLER_FORWARDED_ALLOW_IPS). When the proxy runs behind a reverse proxy (ngrok, Docker, an L7 load balancer, …), the immediate peer is the edge, not the caller — the real IP is in X-Forwarded-For, which is spoofable, so trust must be explicit. Uvicorn's ProxyHeadersMiddleware is mounted at the app layer with trusted_hosts=DISTILLER_FORWARDED_ALLOW_IPS (default 127.0.0.1); when the immediate peer is in that list, client_ip is taken from X-Forwarded-For, otherwise it stays the peer (spoofed XFF from an untrusted peer is ignored). Set DISTILLER_FORWARDED_ALLOW_IPS=* for ngrok/Docker dev where the peer IP is dynamic — only safe when no untrusted clients can reach the bind address directly.
Capture modes (DISTILLER_CAPTURE_CLIENT_IP). truncated (default) zeros the last IPv4 octet or masks IPv6 to /48 — coarse locality without raw PII; full stores the raw IP (opt-in); hashed is the salted-HMAC fingerprint using a separate deployment IP salt; off records None. forwarded_for is captured only when the immediate peer is trusted; spoofed X-Forwarded-For from an untrusted peer is discarded. The same transform is applied to every trusted hop. user_agent and request_id (X-Request-Id) are always captured (low sensitivity).
Privacy note. The default truncated mode is privacy-safer — raw IPs are PII in most jurisdictions. Opt into full only when you need the exact address on disk / in your ingest stream; use hashed for stable per-caller pseudonyms backed by a deployment secret or off to drop IPs entirely. HMAC hashing is not irreversible anonymization when the secret or a small candidate space is available.

Pricing overrides

Per-model prices ship with the proxy in src/distiller/metrics/pricing.yaml (loaded at import time and materialized into cost at write time — later edits never re-price already-recorded requests). To add a new model or override a rate without editing the code, point DISTILLER_PRICING_FILE at a YAML file with the same shape:

# Adds a brand-new model AND overrides a bundled rate.
openai:
  gpt-9-experimental:
    input:  "0.50"     # USD per 1M tokens (decimal string — no floats)
    output: "1.50"
  gpt-4o:              # already in the bundled defaults → REPLACES it
    input:  "1.75"
    output: "7.00"
bedrock:
  anthropic.claude-mystery-9:
    input:  "1.00"
    output: "4.00"

Merge semantics keyed by (provider, model):

User entry for a bundled model → completely replaces the bundled entry for that model.
Models absent from the override file → keep bundled defaults.
Brand-new (provider, model) pairs → added.
OpenAI / Gemini overrides propagate to the Azure / Vertex clones automatically (seed-note invariant); direct azure: / vertex: overrides are also honored and win over the clone.
All normalization (dated snapshot strip, -preview strip, Bedrock region prefix + -vN:M version) applies after the merge, so a new Bedrock family entry automatically resolves cross-region / dated ids.

input / output are USD per 1M tokens as decimal strings — parsed via Decimal for exact integer nano-USD conversion, no float drift. source and updated are optional (default "user override") so a quick override file only needs the two rates.

Fault tolerance. A missing or malformed file logs a structured pricing.override_load_failed ERROR (with error_type, pricing_file, fallback=bundled_defaults, and an actionable hint) and falls back to the bundled defaults — the proxy always keeps serving on bundled rates. Verify a change with grep pricing.overrides_applied in the proxy log at startup.

Quotas / rate limiting

Fixed-window quota enforcement is on by default (DISTILLER_QUOTAS_ENABLED) but only kicks in when per-tenant auth is on (CONFIG_SOURCE=file/api) and the matched tenant or proxy key declares at least one policy. A config with no policies: costs nothing at request time.

Author policies on tenants: and/or proxy_keys: in config.yaml. Each policy has three fields:

unit: requests · tokens · cost (cost is whole USD — the limit is an integer, no decimals; internal accounting scales to micro-USD).
limit: positive integer in the unit above.
period: 1m · 1h · 1d · 7d · 30d (fixed calendar windows).

Merge rule (tenant + key). For each (unit, period) pair, a key-scope policy overrides the tenant's same-shape policy for that key. Any tenant policy whose (unit, period) is not shadowed by a key policy still applies to the key (they union). This lets you set a tenant-wide safety net and tighten (or loosen) individual keys without duplicating the rest.

Enforcement timing.

Requests — checked pre-flight. Over the limit → 429 Too Many Requests with a Retry-After header (seconds until the current fixed window rolls over). The upstream is never called and no vendor token is minted for a request we're about to 429.
Tokens / cost — accounted best-effort post-response from the normalized usage on the InteractionRecord. A single request that overshoots is not cut off mid-stream; instead the counter increment blocks the next request that would fall in the same window. Streaming and non-streaming both contribute once the record is materialized.

The admin Config tab (/admin/config) shows each (tenant, proxy key) row's effective policies and the current in-window usage per policy — read only. Persistence is a plain SQLite file at DISTILLER_QUOTAS_DB_PATH; point it at a mounted/named volume in docker-compose so in-window enforcement state survives restarts.

Bidirectional PII redaction

Optional outbound PII redaction sits in the admission chain after kill-switch and quotas so a 403/429 short-circuits before any body parsing, adapter call, or mapping allocation happens. Three modes with closest-wins resolution (key > tenant > global):

off (default) — true zero-work fast path. No body read, no JSON decode, no dialect lookup, no adapter call, no mapping allocated. Existing deployments load unchanged.
outbound — supported request text is redacted before upstream forwarding and before the capture/index write. The capture record and the interaction index therefore hold the redacted body, not the source. The client-bound response is the raw upstream reply — outbound mode never modifies bytes the client sees.
full — outbound redaction plus best-effort client-side response restoration for non-streaming supported dialects. On success the client sees the original values back while the capture record keeps the redacted upstream bytes (trust boundary preserved). Streaming responses, non-2xx upstream replies, unsupported dialects, and any reassemble/encode failure degrade to outbound: the client receives the redacted stream/body and a single redaction.full_outbound_degradation event fires per request. An unsupported dialect is stricter: it fails closed before upstream forwarding.

Supported text dialects. Redaction currently covers OpenAI Chat Completions, Anthropic Messages, Gemini/Vertex generateContent, and Bedrock Converse text fields. OpenAI Responses and Bedrock InvokeModel have no redaction dialect yet, so requests on those ingress paths fail closed by default rather than silently forwarding raw content. Within a supported dialect, multimodal binary content, provider-specific structured tool arguments, and other fields not listed by the dialect are preserved unchanged; redaction is therefore not a complete DLP guarantee for mixed or provider-specific payloads.

Fail policy. DISTILLER_REDACTION_FAIL_OPEN is global-only — tenants and keys cannot flip it, so per-tenant policy can never widen the blast radius of an adapter incident. Default false (fail-closed) rejects the request with a sanitized 502 on any adapter/reassemble/encode failure; true returns the original still-unredacted payload with no partial rewrite.

Mapping lifecycle. The request-scoped RedactionMapping lives in process memory only, is refused for pickling / repr / iteration, and is cleared idempotently on every terminal path (success, adapter/egress error, mid-stream disconnect, kill-switch/quota short-circuit AFTER redaction entered). Mappings are never persisted, never logged, and never end up in a capture record.

Adapter secret handling. DISTILLER_REDACTION_TONIC_API_KEY_REF is the env-var NAME (not the value) holding the Tonic Textual key. The raw value is resolved through the same SecretsBackend as vendor credentials and stays in process memory only — it never lands in Settings, admin templates, logs, exceptions, records, or metrics.

Observability. The admin Metrics tab (/admin/metrics) renders a Redaction status card with the global mode, the wired adapter, fail-open posture, service availability, and an aggregate counter block:

success, success_noop, blocked_fail_closed, blocked_unsupported_dialect, bypassed_fail_open, degraded_full_to_outbound (per resolved mode).
type_counts — Tonic entity-type labels (e.g. EMAIL, PHONE_NUMBER) filtered through a compile-time allowlist; unknown, malformed, or adversarial labels reported by the adapter are aggregated into a single fixed OTHER bucket so the registry's key set is bounded regardless of upstream behaviour. Raw unknown strings never become metric labels, log fields, or admin-rendered text.
Adapter latency (count / avg / max in ms).
Full → outbound degradation reasons (bounded low-cardinality set: streaming_response, unsupported_dialect, service_unavailable, adapter_unavailable, adapter_fail_open, response-side parse / extract / restore / reassemble / encode failures, and upstream_interrupted for a buffered upstream read that raised before full-mode restoration could complete).

Counters are in-memory only (reset on restart) and strictly aggregate: no tenant, key, request id, span content, replacement, mapping entry, or API key value ever appears in a metric label or a snapshot payload.

Global entity-selection policy. An optional entity_policy block on the deployment configuration narrows which entity classes the adapter is allowed to redact. Provider-neutral by design: two actions (redact, ignore) and an opaque per-entity identifier vocabulary that each adapter translates into its own detector taxonomy.

entity_policy:
  default_action: redact          # fires for every detected type not listed below
  entities:
    ORGANIZATION: ignore          # keep organisation names visible

Two shipped examples in config.example.yaml.
- Redact all except organizations — default_action: redact + ORGANIZATION: ignore. Every other detected type is redacted; org names survive verbatim.
- Redact only names and emails — default_action: ignore + NAME_GIVEN, NAME_FAMILY, EMAIL_ADDRESS set to redact. Every other detected type passes through untouched.
Deployment config is the sole mutation surface. The block is loaded from the same source as the rest of config.yaml — file or DISTILLER_CONFIG_SOURCE=api — and the ConfigCache picks up changes on the next DISTILLER_CONFIG_REFRESH_S tick. The admin /admin/config page renders the current policy read-only inside the existing masked Config projection; there is no admin editor / write API for it.
Atomic reload + last-known-good. Malformed payloads (unknown actions, over-length identifiers, disallowed characters, oversized overrides) are rejected atomically at load time — the previously loaded policy stays live until the source is fixed. Error text carries only positional refs (entity_policy.entities[<idx>]); the offending identifier / action string never surfaces in logs, the admin banner, metrics, or captures.
Tonic Textual translation. The adapter maps redact → Redaction and ignore → Off on the documented POST /api/Redact/bulk request. default_action becomes generatorDefault; each per-entity override becomes an entry in generatorConfig keyed by the operator-persisted identifier. A default-safe policy (default_action: redact with no overrides) emits the default body byte-for-byte, so upgrading with no policy change is a no-op on the wire.
Metrics & captures unchanged. Bounded aggregate type_counts, adapter latency, per-mode outcomes, and degradation reasons still reset on restart; capture records still store only the redacted upstream bytes. ignored spans never allocate a placeholder, mapping entry, or counter — including if the adapter returned one anyway (defensive drop at the adapter boundary).

Identity model (4 levels)

ProxyKey ──N:1──▶ Tenant (use case) ──N:1──▶ VendorCredential ──N:1──▶ ProviderEgress
 label,           id/label,                   id, label, fingerprint,      openai |
 key_hash,        credential (ref)            api_key_ref, base_url,        anthropic |
 tenant (ref)                                 azure fields, aws_region      gemini | azure |
(dist-…)                                                                vertex | bedrock

ProviderEgress — code-level strategy: base-URL shape + auth-header shape. One subclass per vendor (openai, anthropic, gemini, azure, vertex, bedrock). Adding a new vendor = one new ProviderEgress subclass + registration.
VendorCredential — one real upstream API key (or, for vertex / bedrock, a routing record whose secret is resolved via google-auth ADC / botocore). id is the immutable primary attribution key (survives key rotation); label is the renamable display name; fingerprint (salted HMAC) is for passthrough recognition + duplicate-secret detection.
Tenant — a use-case bucket over a credential (not customer identity). Many tenants may point at the same VendorCredential to get separate attribution buckets while sharing one upstream key.
ProxyKey — issued dist-… key with a human label. Only the SHA-256 hash is stored; the raw key is shown once at issuance and never persisted. Many proxy keys may point at the same tenant (per-user fan-out).

Every request stamps proxy_key_label · tenant_id · credential_id · credential_fingerprint · provider · model · usage · status onto the InteractionRecord and the metrics aggregator — sliceable by any level in the admin UI.

See config.example.yaml for an annotated YAML.

Issuing proxy keys

Use the bundled CLI to mint a labeled dist-… key and a ready-to-paste config entry. The raw key is shown once on stderr; copy it immediately.

# 1) (Optional) Scaffold a vendor-credential entry, then merge under
#    `vendor_credentials:` in your config.yaml.
uv run distillery-issue-key vendor-credential \
    --id openai-prod --provider openai \
    --api-key-ref OPENAI_KEY_PROD

# 2) Add a `tenants:` entry that references that credential (edit config.yaml).
#    tenants:
#      acme-team-a: { id: acme-team-a, credential: openai-prod }

# 3) Mint a proxy key bound to that tenant. The raw `dist-…` key prints
#    to stderr (copy it now); the YAML stub prints to stdout — append under
#    `proxy_keys:` in your config.yaml.
uv run distillery-issue-key proxy-key \
    --id alice --tenant acme-team-a --label alice@acme

Notes:

Proxy keys are hashed; vendor keys are not. Vendor keys must be replayed upstream, so they're resolved at request time from api_key_ref (env var) or any SecretsBackend impl (Vault/AWS/GCP) — never written to records, logs, or the metrics DB.
Identity comes from the key, never the URL. A proxy key always resolves to its bound tenant regardless of the request path; the proxy key itself is never forwarded upstream.

Admin UI

Visit http://localhost:8080/admin in a browser. HTTP Basic — username is ignored, password is DISTILLER_ADMIN_PASSWORD (constant-time compare). The surface is locked (503) if the password is unset rather than silently open.

Four primary views. The content is server-rendered Jinja templates with a small vendored admin.css + admin.js for progressive enhancement — every page remains usable with JavaScript disabled. The Admin surface also exposes CSRF-protected operational controls for key minting, kill switches, and graceful drain.

/admin/config — the parsed ProxyConfig with every api_key masked (sk-•••abcd form) and every fingerprint truncated. Two interchangeable views toggled in-page:
- Raw JSON (the JS-enabled default) — syntax-highlighted, with a Copy button.
- Tree — a native <details> collapsible tree (so expand/collapse keeps working without JS) with Expand-all / Collapse-all controls living inside the tree panel.
With JavaScript off, the tree is rendered visible so the page still degrades to a readable, human-friendly view.

Emergency kill switch. The Config page also carries per-row block/restore toggles (one per proxy key) plus an Emergency kill switch panel for blocking a whole tenant or vendor credential. A blocked scope returns 403 on subsequent requests before any quota / token work runs; the affected rows carry a danger badge naming the blocked scope. Toggles are CSRF-protected via the same double-submit pattern as key minting, work without JavaScript (form POST + 303 redirect), and gain a confirm-on-block step when JS is enabled. Blocks are in-memory only and reset on process restart — this is deliberate (fail-open on restart for an emergency control) and is stated in the panel. The whole surface is hidden unless per-tenant auth is enabled, because there is no AuthContext for the filter to key off otherwise.
/admin/metrics — grand totals + per-(credential · tenant · provider · model · proxy-key label) rollup tables with aligned right-justified numeric columns, zebra striping, sticky headers, and click-to-sort on every column (toggles ascending/descending). On top of the tables, four categorical bar charts show the top entries by the selected metric for credential, provider, model, and proxy-key label (tenant is a filter but does not get its own chart). Charts are rendered with a vendored, offline Chart.js v4.4.7 UMD bundle committed under src/distiller/static/vendor/chart.umd.js and served from /admin/static/vendor/chart.umd.js — no CDN, no network fetch. Client-side filters (AND across credential, tenant, provider, model, plus a metric selector) re-aggregate the full row cross-product and redraw the totals, tables, and charts in place. Each chart is capped at top 20 entries with a "showing top N of M" caption when truncated. The charts are categorical, not time-series — the metrics store holds cumulative counters with no time dimension, so trends-over-time would require a backend schema change (out of scope here).

With JavaScript off, the server-rendered rollup tables remain fully usable; filters, sort, and charts are progressively layered on when JS is available. Backed by SQLite at DISTILLER_METRICS_DB_PATH; in-memory counters flush at least every DISTILLER_METRICS_FLUSH_S seconds and on shutdown.
/admin/interactions — a browsable feed of captured calls, backed by the SQLite interaction index at DISTILLER_INTERACTION_INDEX_DB_PATH. Opt-in via DISTILLER_INTERACTION_INDEX_ENABLED=true — when off, the tab shows the empty state and the JSON endpoints return 503. Two feed modes share one filter bar (provider · model · tenant · since/until):
- Grouped feed (default, no q) — one row per conversation, sorted by latest-turn time DESC, served from GET /admin/conversations (?cursor= opaque, paged). Each row expands inline into its ordered turns via GET /admin/conversations/{id}.
- Flat search (?q=…) — one row per call from GET /admin/interactions, ordered by relevance / recency. Text search is intentionally flat: keyword hits are more useful as per-turn rows than as group summaries. Each hit still carries conversation_id so the UI can link back to the parent group.
Conversation ids are derived, not client-supplied. The index hashes the full ordered message list at write time and does a longest-prefix parent lookup against previously-indexed heads; a fresh uuid4 is minted when no ancestor matches. Consequences: (a) two independent one-shot jobs that only share an opener message stay in separate groups — parents attach on longest-prefix match, not equality with a leaf; (b) the first time a DB created by an earlier version is opened, ids are backfilled in insertion order once and then never recomputed; (c) pruning a mid-thread turn leaves the surviving turns tagged with the same id, so turn_count / first_at / last_at on the grouped feed are computed from survivors and may show gaps. Full derivation lives in indexing/derive.py; see ARCHITECTURE.md for the port-level contract.

Running

Local (uv)

The application default is 0.0.0.0 for container deployments. For a local process, set DISTILLER_HOST=127.0.0.1 so the no-auth passthrough is not exposed to the local network.

# No-auth passthrough with summary-only capture (default):
uv run distillery

# No-auth passthrough with raw disk capture (review capture safety first):
DISTILLER_CAPTURE_MODE=disk uv run distillery

# Multi-tenant auth + capture to a receiver:
DISTILLER_CONFIG_SOURCE=file DISTILLER_CONFIG_FILE=./config.yaml \
DISTILLER_CAPTURE_MODE=http \
DISTILLER_CAPTURE_ENDPOINT=https://receiver.example/v1/interactions:batch \
DISTILLER_CAPTURE_TOKEN=$INGEST_TOKEN \
uv run distillery

On startup, confirm the mode in the logs — auth=file capture=http means auth and HTTP capture are active; auth=none means config is not loaded.

Docker

cp .env.example .env       # set DISTILLER_ADMIN_PASSWORD + any vendor keys you reference
docker compose up --build  # → :8080 with /admin reachable

What the bundled docker-compose.yml wires up:

:8080 exposed for the proxy + /admin UI.
DISTILLER_ADMIN_PASSWORD sourced from .env (ADMIN_PASSWORD). Empty value locks /admin (503).
YAML config mount (commented by default). Uncomment ./config.yaml:/config/config.yaml:ro + DISTILLER_CONFIG_SOURCE=file to enable proxy-issued-key auth.
Vendor key env vars (OPENAI_KEY_PROD, ANTHROPIC_KEY_PROD, GEMINI_KEY_PROD, AZURE_OPENAI_KEY, VERTEX_API_KEY) forwarded from .env; the loader only reads the ones your config.yaml references via api_key_ref:. Vertex OAuth and Bedrock do not use api_key_ref: — they pick up creds from GOOGLE_APPLICATION_CREDENTIALS / workload identity and the AWS default-provider chain (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN) respectively. When running directly rather than through Compose, referenced vendor keys are read from the local .env file as a fallback; explicitly exported process environment variables take precedence.
Named volumes distillery-captures, distillery-metrics, distillery-interactions, and distillery-quotas hold capture JSONL, usage metrics, the optional interaction index, and quota counters respectively. They survive docker compose down && up (but are removed by docker compose down -v).
The image runs as the non-root distillery user and declares a Docker health check against /healthz.

Examples

Multi-vendor mapping (proxy keys → vendor keys)

# 1) Author config.yaml from the annotated example. The 4-level model is:
#    vendor_credentials → tenants → proxy_keys.
cp config.example.yaml config.yaml
# Edit to point `api_key_ref:` at env-vars holding your real vendor keys.

# 2) Scaffold + mint with the CLI (raw key prints to STDERR, exactly once).
uv run distillery-issue-key vendor-credential --id openai-prod \
    --provider openai --api-key-ref OPENAI_KEY_PROD   # merge into vendor_credentials:
uv run distillery-issue-key proxy-key --id alice \
    --tenant acme-team-a --label alice@acme           # append under proxy_keys:

# 3) Export the vendor key(s) referenced from config.yaml.
export OPENAI_KEY_PROD="sk-...real-openai-key..."

# 4) Run with auth on.
DISTILLER_CONFIG_SOURCE=file DISTILLER_CONFIG_FILE=./config.yaml \
DISTILLER_ADMIN_PASSWORD=$(openssl rand -hex 24) \
uv run distillery

# 5) Clients use the PROXY key — never the vendor key.
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer dist-alice-..." \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

A missing/invalid/disabled proxy key returns 401. Anthropic and Gemini routes work identically — flip provider: on the VendorCredential (and set base_url: to the vendor's OpenAI-compatible endpoint).

Passthrough mode (no proxy keys)

# Default: CONFIG_SOURCE=none. The client's own key is forwarded verbatim
# to DISTILLER_UPSTREAM_BASE_URL. Useful for single-tenant or local dev.
DISTILLER_UPSTREAM_BASE_URL=https://api.openai.com uv run distillery

# Clients send their own OpenAI key — Distillery fingerprints it for
# attribution but does not store it.
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

Azure OpenAI

Set provider: azure on the VendorCredential (+ azure_deployment and azure_api_version). Clients may send plain OpenAI requests to /v1/chat/completions or use the native Azure SDK path /openai/deployments/{client-deployment}/chat/completions. The proxy rewrites either form to …/openai/deployments/{configured-deployment}/chat/completions with the configured API version and key in the api-key header. The inbound proxy key is also accepted in Azure's native api-key header.

Native-dialect transparent forwarding

Every vendor is supported as a vendor-native transparent passthrough: the caller's native SDK speaks its own dialect (Chat Completions / Responses / Messages / generateContent / Converse / InvokeModel), the proxy preserves the request semantics and forwards the body unchanged unless an enabled pre-hook (such as stream-usage enrichment or redaction) must transform it, while a structured InteractionRecord is captured off to the side. Provider egresses apply the URL/path and authentication transformations required by the upstream, such as Azure deployment paths, Gemini base prefixes, or Bedrock SigV4 signing.

Multi-carrier ingress. A single dist-… key is accepted from any carrier the calling SDK uses (auth/keys.py:extract_proxy_key):

Authorization: Bearer <key> — OpenAI, Anthropic, Vertex OAuth, Bedrock bearer.
x-api-key: <key> — Anthropic SDK.
api-key: <key> — Azure OpenAI SDK.
x-goog-api-key: <key> — Gemini / Vertex API-key SDKs.
SigV4 Authorization: AWS4-HMAC-SHA256 Credential=<key>/… — Bedrock boto3 / AWS SDKs. The access-key-id slot smuggles the proxy key; the SigV4 signature is not verified (the proxy key is the trust anchor) and the inbound Credential= header is stripped before real upstream SigV4 signing.

The carrier is recognized, the proxy key is stripped, and the real upstream auth (vendor API key, OAuth bearer, or freshly minted SigV4) is applied server-side. The proxy key is never forwarded upstream.

Per-vendor caller setup

For each vendor, point the native SDK at https://your-proxy/ and put your dist-… key in whatever slot the SDK uses for that vendor's auth.

OpenAI (Authorization: Bearer)

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="dist-…")
client.chat.completions.create(model="gpt-4o-mini", messages=[...])

Anthropic (x-api-key)

from anthropic import Anthropic
client = Anthropic(base_url="http://localhost:8080", api_key="dist-…")
client.messages.create(model="claude-3-5-sonnet-latest", max_tokens=256, messages=[...])

Gemini AI Studio (x-goog-api-key)

from google import genai
client = genai.Client(
    api_key="dist-…",
    http_options={"base_url": "http://localhost:8080"},
)
client.models.generate_content(model="gemini-1.5-flash", contents="hi")

Vertex AI — two shapes; pick the one your VendorCredential is configured for.

OAuth (ADC) mode: the proxy mints the GCP bearer itself; the caller just needs to carry the dist-… key. With the Vertex SDK in api_key mode, the simplest carrier is plain HTTPS:

# Native generateContent path forwarded verbatim — project + location in the URL.
curl http://localhost:8080/v1/projects/$PROJECT/locations/us-central1/publishers/google/models/gemini-1.5-flash:generateContent \
  -H "Authorization: Bearer dist-…" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'

API-key (x-goog-api-key) mode is identical, with the proxy key in x-goog-api-key instead. Tradeoffs: OAuth/ADC is the GCP-native path — short-lived tokens, workload identity, IAM-policy-driven — and is recommended for prod. The x-goog-api-key fallback is simpler for express / publisher endpoints but ties you to a long-lived static key.

Bedrock — boto3 SigV4 smuggle (the credential carrier; the SigV4 signature is intentionally not verified):

import boto3
session = boto3.Session(
    aws_access_key_id="dist-…",   # the proxy key rides in the AccessKeyId slot
    aws_secret_access_key="x",          # dummy; signature isn't checked by the proxy
    region_name="us-east-1",
)
bedrock = session.client("bedrock-runtime", endpoint_url="http://localhost:8080")
bedrock.converse(modelId="anthropic.claude-3-5-sonnet-20240620-v1:0",
                 messages=[{"role": "user", "content": [{"text": "hi"}]}])

Bedrock requires aws_region on the VendorCredential — it scopes the real upstream SigV4 signature and can't be derived from FIPS/PrivateLink hosts. AWS creds themselves come from botocore's default chain (env vars, shared config, or instance/ECS/EKS role) — there is no AWS-key field in the config.

A note on the OpenAPI surface

openapi/ingest.yaml describes the receiver-facing capture endpoint (POST /v1/interactions:batch) — the contract a downstream ingest service implements. The forwarding surface is deliberately not a hand-authored OpenAPI spec: it's a vendor-native transparent passthrough (a catch-all that preserves request bodies while applying any provider-specific upstream URL/path and authentication adaptation), so there is no per-vendor request schema to document on the proxy. Use each vendor's own API docs for the request/response shape; point the SDK at the proxy.

The captured data

Each call yields one InteractionRecord (one JSON object per line in disk mode):

{
  "interaction_id": "0e9c2b1a-…",
  "schema_version": "0.1",
  "provider": "openai",
  "api_type": "chat.completions",
  "tenant_id": "acme",
  "destination": "https://api.openai.com",
  "requested_model": "gpt-4o",
  "resolved_model": "gpt-4o-2024-08-06",
  "request": {
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize this ticket…"}],
    "tools": null,
    "stream": true,
    "params": {"temperature": 0.2}
  },
  "response": {
    "model": "gpt-4o-2024-08-06",
    "message": {"role": "assistant", "content": "The ticket reports…"},
    "finish_reason": "stop",
    "usage": {"prompt_tokens": 412, "completion_tokens": 88, "total_tokens": 500}
  },
  "streaming": {"was_streamed": true, "chunk_count": 90, "ttft_ms": 310.4, "assembled_ok": true, "interrupted": false},
  "operational": {"status": "success", "http_status": 200, "latency_ms": 1840.2},
  "raw_request": "{…captured outbound body…}",
  "raw_response": "data: {…}\n\n…"
}

Downstream consumers can project the canonical request/response fields into their own analytics, evaluation, or data formats without coupling those workflows to the proxy's forwarding path.

Building a receiver (HTTP capture)

In http mode the proxy POSTs batches to POST /v1/interactions:batch. The wire contract is the versioned openapi/ingest.yaml, generated from the canonical models — point your codegen at it to build a receiver in any language. Records carry interaction_id as an idempotency key, so retries dedupe cleanly.

Development

# Run from the repository root.
uv sync
uv run pytest                       # tests
uv run ruff check .                 # lint
uv run python scripts/gen_openapi.py   # regenerate openapi/ingest.yaml after model changes

CI (.github/workflows/ci.yml) runs lint + tests on Python 3.12–3.14, checks the ingest spec is in sync, and builds the Docker image.

Status

Version 0.1.0 is an initial standalone alpha release. It supports OpenAI, Anthropic, Gemini AI Studio, Azure OpenAI, Vertex AI, and Bedrock across the provider surfaces listed above, with raw payload capture plus normalized fields for supported shapes. Redaction covers OpenAI Chat, Anthropic Messages, Gemini/Vertex generateContent, and Bedrock Converse text fields; unsupported redaction dialects fail closed by default. Includes a password-gated /admin UI showing masked config + persisted SQLite usage metrics. Planned: upstream circuit breakers and Prometheus export.

0.1.0 release limitations

The initial release is intended for a single proxy instance or a small internal deployment. SQLite metrics, quotas, and interaction indexing are process-local; they are not coordinated across replicas. High availability and globally shared quota enforcement require a shared database adapter, such as PostgreSQL, which is intentionally outside this release.

Raw capture is off by default, but disk and http capture can contain raw prompts, tool data, model output, and caller metadata as observed by the proxy. Operators are responsible for encrypted storage or a trusted HTTPS receiver, access controls, retention, backup, and deletion policies. Redaction is also off by default and is not a general DLP guarantee: the first supported adapter is Tonic Textual, only the documented wire dialects are covered, and unsupported dialects fail closed by default.

The forwarding boundary is intentionally catch-all and may relay provider-native operations the proxy does not normalize, including file uploads or model-management calls. Deployments exposed beyond a trusted local boundary must provide authentication, rate limiting, and upstream/network restrictions; untrusted tenant-controlled upstream URLs also require SSRF-aware validation. See Security for the operational checklist.

License

MIT.

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.github/workflows		.github/workflows
openapi		openapi
scripts		scripts
src/distiller		src/distiller
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
ARCHITECTURE.md		ARCHITECTURE.md
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
config.example.yaml		config.example.yaml
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
requirements.lock		requirements.lock
uv.lock		uv.lock

Uh oh!

Folders and files

Latest commit

History

Repository files navigation