Skip to content

feat(k8s): add LiteLLM proxy + Ollama to the kustomize deployment (PR #72 architecture)#73

Open
jcschaff wants to merge 7 commits into
mainfrom
feat/kustomize-litellm
Open

feat(k8s): add LiteLLM proxy + Ollama to the kustomize deployment (PR #72 architecture)#73
jcschaff wants to merge 7 commits into
mainfrom
feat/kustomize-litellm

Conversation

@jcschaff

@jcschaff jcschaff commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

✅ Deployed & validated on the dev overlay (vcell-ai-rke-dev) — all workloads healthy, LiteLLM migrated its Supabase schema, backend on hosted OpenAI, Auth0 login live, trusted Let's Encrypt cert. Ready for review. Tracking: #71.

Summary

PR #72 (feat/litellm) reworked chat to route through a LiteLLM proxy with per-user virtual keys/budgets. The kustomize deployment (from #69) predates that and would deploy a backend expecting a litellm service that doesn't exist. This PR extends the deployment to match the new architecture.

What's added

  • base/litellm.yaml — LiteLLM Deployment + Service (ghcr.io/berriai/litellm:main-stable, :4000). Config mounted from a litellm-config-file ConfigMap generated from base/litellm_config.yaml; secrets from litellm-secrets.
  • base/ollama.yaml — Ollama StatefulSet + Service (:11434, 10Gi PVC) serving the local-model fallback. Pulls phi4-mini on startup (CPU-friendly, ~3.8B).
  • Backend gets LITELLM_URL (config) + LITELLM_MASTER_KEY (from backend-secrets) to provision per-user virtual keys. It still uses Azure directly for KB embeddings.
  • config/<env>/litellm.env — LOCAL_LLM_MODEL=ollama/phi4-mini, LOCAL_LLM_API_BASE=http://ollama:11434, LANGFUSE_HOST.
  • New sealed secret litellm-secrets — LITELLM_MASTER_KEY, OPENAI_API_KEY, Langfuse keys, DATABASE_URL — via scripts/sealed_secret_litellm.sh. secrets.sh + secrets.dat.template updated across all overlays; LITELLM_MASTER_KEY also added to backend-secrets.

Design decisions (per discussion)

  • DATABASE_URL → Supabase Postgres (LiteLLM persists virtual keys/budgets there; sealed).
  • local-model → in-cluster Ollama running phi4-mini on CPU.
  • Image tags bumped to 0.1.7.

Prerequisites before deploying

  1. Tag 0.1.7 must be cut so CI (build_containers.yml) publishes backend/frontend images containing the LiteLLM code — the newest existing image (0.1.6.2) predates Integrate LiteLLM: multi-model chat, per-user budgets, and automatic fallback #72.
  2. Fill real values in each overlay's secrets.dat (now includes LiteLLM master key, OpenAI key, DATABASE_URL) and the non-secret config/<env>/*.env placeholders.

.gitignore fix

PR #72 added a bare litellm.env ignore (for the docker-compose secrets file), which also matched the committed kustomize/config/*/litellm.env. Anchored it to /litellm.env (repo root only).

Validation

  • kubectl kustomize renders all three overlays (20 objects each, up from 13); ConfigMap name-hashes propagate into the litellm/backend pods.
  • All shell scripts pass bash -n.
  • Confirmed no secret material staged (secrets.dat, secret-*.yaml gitignored, including new secret-litellm.yaml).

Builds on #69 (merged). Follow-up #70 (frontend build-time API URL) still applies.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ

jcschaff and others added 3 commits July 12, 2026 13:42
@jcschaff @claude
feat(k8s): add LiteLLM proxy + Ollama for the PR #72 architecture
a4ac971 
PR #72 routes chat through a LiteLLM proxy with per-user virtual keys/budgets.
Extend the kustomize deployment to match:

- base/litellm.yaml: LiteLLM Deployment+Service (:4000), config mounted from a
  litellm-config-file ConfigMap (base/litellm_config.yaml), secrets from
  litellm-secrets
- base/ollama.yaml: Ollama StatefulSet+Service (:11434, 10Gi PVC) serving the
  local-model fallback; pulls phi4-mini on start (CPU-friendly)
- backend gets LITELLM_URL (config) + LITELLM_MASTER_KEY (backend-secrets) to
  provision per-user virtual keys; still uses Azure for KB embeddings
- config/<env>/litellm.env: LOCAL_LLM_MODEL=ollama/phi4-mini, LOCAL_LLM_API_BASE,
  LANGFUSE_HOST
- new sealed secret litellm-secrets (master key, OpenAI key, Langfuse, Supabase
  DATABASE_URL) via scripts/sealed_secret_litellm.sh; secrets.sh + templates
  updated across overlays
- bump overlay image tags to 0.1.7 (must be tagged/built before deploy)
- fix .gitignore: anchor the docker-compose `/litellm.env` so it no longer
  ignores the committed kustomize/config/*/litellm.env

Validated: kubectl kustomize renders all three overlays (20 objects each);
scripts pass bash -n.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
@jcschaff @claude
feat(k8s): back LiteLLM "openai-model" with Azure instead of hosted O…
1cc0823 
…penAI

The project uses Azure OpenAI (already used for embeddings) and has no separate
hosted-OpenAI key, so point LiteLLM's openai-model alias at Azure. The alias
name stays "openai-model" (hardcoded in the backend LLMModel Literal + frontend).

- litellm_config.yaml (kustomize base + repo root, kept in sync): openai-model
  litellm_params -> model=azure/<deployment> via os.environ/AZURE_MODEL, plus
  api_base/api_version/api_key from env
- config/<env>/litellm.env: add AZURE_MODEL, AZURE_API_BASE, AZURE_API_VERSION
- litellm-secrets: openai-api-key -> azure-api-key; base/litellm.yaml env and
  sealed_secret_litellm.sh updated; secrets.sh passes the existing AZURE_API_KEY
- secrets.dat.template: drop OPENAI_API_KEY (Azure key reused); fix DATABASE_URL
  example to the Supabase SESSION POOLER (IPv4, 5432) — the direct host is IPv6-only
- litellm.env.example: OpenAI -> Azure vars

Validated: kubectl kustomize renders all three overlays (20 objects); the
litellm ConfigMap carries the AZURE_* vars and the pod reads azure-api-key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
@jcschaff @claude
feat(k8s): LiteLLM openai-model accepts either Azure or OpenAI key
f513cf8 
Make the "openai-model" alias a group of two interchangeable deployments (Azure
OpenAI + hosted OpenAI). Provide a key for either (or both) in secrets.dat;
LiteLLM routes to whichever authenticates. Gives deploy-time flexibility without
committing to a provider now.

- litellm_config.yaml (base + root): second openai-model deployment (openai/gpt-4o)
- litellm-secrets regains openai-api-key alongside azure-api-key; base/litellm.yaml,
  sealed_secret_litellm.sh (7 args), secrets.sh and secrets.dat.template updated
- unify the endpoint var: LiteLLM now reads AZURE_ENDPOINT (the value the backend
  already uses for embeddings) instead of a separate AZURE_API_BASE, and the
  backend/litellm placeholders are aligned so the shared value can't drift
- note in configs/README that KB embeddings still require the Azure key
  specifically, and root litellm.env.example local model -> phi4-mini

Validated: kubectl kustomize renders all overlays (20 objects); the mounted
config has both openai-model deployments and the pod reads both api keys.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
@jcschaff jcschaff mentioned this pull request Jul 13, 2026
Open
jcschaff and others added 3 commits July 13, 2026 14:34
@jcschaff @claude
feat(k8s): make LiteLLM openai-model configurable; point dev at hoste…
63e3e73 
…d OpenAI

- litellm_config.yaml (base + root): openai-model deployment model is now
  os.environ/OPENAI_MODEL (default openai/gpt-4o-mini) instead of hardcoded gpt-4o
- config/<env>/litellm.env + litellm.env.example: add OPENAI_MODEL=openai/gpt-4o-mini
- config/vcell-ai-rke-dev/backend.env: switch dev to hosted OpenAI — PROVIDER=openai
  (singleton uses a plain OpenAI client with base_url=AZURE_ENDPOINT on the non-azure
  path), AZURE_ENDPOINT=https://api.openai.com/v1, gpt-4o-mini / text-embedding-3-small

Dev now runs chat + embeddings on hosted OpenAI (verified the key against
/v1/models: valid, both models accessible), so it no longer blocks on the Azure
endpoint. The Azure deployment in the openai-model group stays dormant in dev
(placeholder endpoint) and LiteLLM falls back to the OpenAI deployment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
@jcschaff @claude
fix(k8s): raise LiteLLM memory limit to 2Gi (was OOMKilled at 1Gi)
f6961cd 
The litellm pod was CrashLoopBackOff / OOMKilled (exit 137) on the vxrails
cluster: Prisma migrate + proxy startup spikes past 1Gi. Bump requests to
512Mi/250m and limits to 2Gi/1cpu. Verified: litellm reaches 1/1 Ready and
completes its DB migrations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
@jcschaff @claude
chore(k8s): switch dev ingress cert to letsencrypt-prod
96b9604 
DNS for vcell-ai-dev.cam.uchc.edu now resolves to the ingress, so switch the
dev overlay from letsencrypt-staging (browser-untrusted) to letsencrypt-prod
for a trusted cert.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
@jcschaff jcschaff requested a review from androemeda July 13, 2026 20:14
@jcschaff @claude
chore(k8s): fill dev Auth0 + Supabase config (non-secret)
b591e7f 
Real dev values for vcell-ai-rke-dev: AUTH0_DOMAIN/CLIENT_ID/AUDIENCE and
SUPABASE_URL (derived from the DB project ref). These are public OIDC/Supabase
identifiers, not secrets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XpNzobVi83p3YKL9sqtGwZ
@jcschaff jcschaff mentioned this pull request Jul 13, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@androemeda androemeda Awaiting requested review from androemeda

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jcschaff