Skip to content

OPERATIONS.md

Latest commit

 

History

History
323 lines (221 loc) · 5.83 KB

OPERATIONS.md

File metadata and controls

323 lines (221 loc) · 5.83 KB
Raw

OPERATIONS.md

Purpose

This document is the operator runbook for RAGU.

Use it to:

  • start and stop the stack
  • verify that services are healthy
  • troubleshoot common failures
  • understand which logs and endpoints matter first

For product overview and setup, see README.md. For architectural structure, see ARCHITECTURE.md. For security assumptions and risks, see SECURITY.md.

Operational scope

RAGU currently runs as a Docker Compose stack with these core services:

  • app
  • ollama
  • chroma
  • redis
  • otel-collector
  • zipkin
  • ollama-exporter
  • prometheus
  • grafana
  • cadvisor

Fast start

Start the full stack:

docker compose up --build

Stop the stack:

docker compose down

Stop and remove named volumes:

docker compose down -v

Remove orphaned containers from older compose definitions:

docker compose down --remove-orphans

Health checklist

Core app path

The core product is healthy when all of these are true:

  • Streamlit UI loads
  • PDFs can be processed
  • Chroma accepts retrieval traffic
  • Ollama returns chat and embedding responses
  • answers stream in the UI

Observability path

The observability stack is healthy when:

  • Zipkin UI loads
  • Prometheus UI loads
  • Grafana UI loads
  • Grafana dashboards show real data

Default endpoints

  • App: http://localhost:8501
  • Ollama API: http://localhost:11434
  • Chroma API: http://localhost:8000
  • Redis: localhost:6379
  • Zipkin UI: http://localhost:9411
  • Ollama exporter metrics: http://localhost:8001/metrics
  • Prometheus UI: http://localhost:9090
  • Grafana UI: http://localhost:3000
  • cAdvisor UI: http://localhost:8081

Runtime flow

flowchart TD
    User[User] --> App[Streamlit app]
    App --> Ollama[Ollama]
    App --> Chroma[Chroma]
    App -. future .-> Redis[Redis]
    Chroma --> OTel[OTEL Collector]
    OTel --> Zipkin[Zipkin]
    OTel --> Prometheus[Prometheus]
    Ollama --> Exporter[Ollama exporter]
    Exporter --> Prometheus
    CAdvisor[cAdvisor] --> Prometheus
    Prometheus --> Grafana[Grafana]
Loading

Service verification

1. Compose status

Check whether containers are up:

docker compose ps

Expected high-level result:

  • app running
  • ollama healthy
  • redis healthy
  • all remaining services running

2. App verification

Open the UI and verify:

  • the page loads
  • the logo appears
  • the question box is visible
  • TTS status is visible

3. Ollama verification

Check loaded models:

docker compose exec ollama ollama list

Expected:

  • configured chat model exists
  • configured embedding model exists

4. Chroma verification

Quick heartbeat:

curl http://localhost:8000/api/v2/heartbeat

5. Prometheus verification

Open:

  • http://localhost:9090/targets

Expected:

  • ollama-exporter is UP
  • cadvisor is UP
  • otel-collector scrape is UP

6. Grafana verification

Open Grafana and verify:

  • Prometheus datasource exists
  • Ollama Overview dashboard loads
  • CPU and RAM gauges show values
  • GPU gauge may stay 0 on hosts without exposed accelerator metrics

7. Zipkin verification

Generate traffic:

curl http://localhost:8000/api/v2/heartbeat

Then open Zipkin and run a query.

First-response troubleshooting

Use this order:

  1. docker compose ps
  2. docker compose logs <service>
  3. direct endpoint checks
  4. UI validation

Common issues

Ollama models are missing

Checks:

docker compose logs ollama
docker compose exec ollama ollama list

Likely causes:

  • first startup still downloading models
  • .env model names do not match pulled models

Chroma is reachable but the app still fails retrieval

Checks:

docker compose logs chroma
docker compose logs app
curl http://localhost:8000/api/v2/heartbeat

Likely causes:

  • Chroma started but app-side collection init failed
  • embedding requests to Ollama failed
  • collection configuration drift

Grafana shows no data

Checks:

  • Prometheus targets page
  • exporter metrics endpoint
  • Grafana datasource configuration

Commands:

docker compose logs prometheus
docker compose logs grafana
curl http://localhost:8001/metrics

Chroma OTEL errors appear in logs

Check:

docker compose logs otel-collector
docker compose logs chroma

TTS is unavailable

Expected behavior:

  • the app still works
  • the UI shows a TTS availability message
  • no audio player is rendered if Piper is unavailable

Check:

docker compose logs app

cAdvisor metrics are missing

Check:

docker compose logs cadvisor

Recovery actions

Rebuild only the app image

docker compose build --no-cache app
docker compose up

Restart a specific service

docker compose restart app
docker compose restart grafana
docker compose restart prometheus

Reset the whole environment

Use with care. This removes persisted data too.

docker compose down -v --remove-orphans
docker compose up --build

Manual acceptance checks

Use the sample PDFs from examples and validate:

  1. Upload one or more PDFs.
  2. Process them successfully.
  3. Ask a question and receive a streamed answer.
  4. Open the retrieval debug expanders.
  5. If a PDB is detected, confirm the Mol* viewer appears.
  6. Confirm the TTS status is visible.
  7. If TTS is available, confirm audio playback appears.
  8. Confirm Grafana panels show data.
  9. Confirm Zipkin receives Chroma traffic.

Operational notes for AI agents

If you are an AI agent changing runtime behavior:

  • read README.md first
  • read AGENT.md for repository guardrails
  • read ARCHITECTURE.md before refactoring service boundaries
  • update this file if ports, services, health behavior, startup order, or recovery steps change