feat(dashboard): usage & analytics page with aggregation endpoints (#303)#345
Conversation
) Add a Usage & analytics page and the aggregation API behind it. Backend (src/gateway/api/routes/usage.py): - GET /v1/usage/summary: range-bounded aggregates (totals, by model, by user, by API key, and a UTC-bucketed time series) reusing _usage_filters. - GET /v1/usage/summary.csv: uncapped CSV export with a formula-injection guard. - Portable hour/day bucketing across SQLite and PostgreSQL, pinned to UTC so buckets do not shift with the server timezone. - Coalesced sums, a conditional error count, top-N breakdowns with a reconciling "other" fold, a default 30d window and a hard span cap. Dashboard (web/): - UsagePage: stat tiles with period-over-period deltas, spend-by-model and spend-by-user tables that drill into the Activity log, a single-metric SVG bar chart with a metric toggle, and CSV export. - FilterComboBox typeahead for user and model, replacing the native select and the exact-match text box. Activity gains the same model picker, with suggestions sourced from usage so they match what the log stores. - Nav entry and route under Observability; ActivityPage seeds its filters from the drill-down query string. Tests: integration coverage for aggregation correctness and cross-dialect bucketing; Vitest coverage for the page and filters. OpenAPI regenerated and the dashboard bundle rebuilt. A seed_usage_smoke.py helper is included for local smoke testing. Closes #303. Part of #299. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This PR appears to be missing required sections from the PR template. Please edit your PR description to include the PR Type, Checklist, and AI Usage sections. The template helps maintainers review your contribution. This PR will be automatically closed in 24 hours if the template is not restored. If you're using an AI coding tool, please ensure it preserves the PR template. |
|
Warning Review limit reached
Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (12)
WalkthroughChangesUsage analytics
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
web/src/pages/ActivityPage.tsx (1)
224-229: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDerive model options directly from the query result.
modelOptionsmirrorsmodelSummary.datainto local state, which can temporarily retain options from a prior filter set. Derive the array directly instead.Suggested change
- const [modelOptions, setModelOptions] = useState<string[]>([]); - useEffect(() => { - if (modelSummary.data?.by_model) { - setModelOptions(modelSummary.data.by_model.filter((r) => r.key !== null).map((r) => r.key as string)); - } - }, [modelSummary.data]); + const modelOptions = modelSummary.data?.by_model.flatMap((row) => (row.key === null ? [] : [row.key])) ?? [];As per coding guidelines, “never mirror server state into
useState” and “derive values from props or query data instead of duplicating them in state.”🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/src/pages/ActivityPage.tsx` around lines 224 - 229, Remove the modelOptions useState and its synchronization useEffect, and derive modelOptions directly from modelSummary.data?.by_model in the component render. Preserve filtering out null keys and returning a string[] with the same empty-result behavior when query data is unavailable.Source: Coding guidelines
src/gateway/api/routes/usage.py (2)
392-400: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicated window/conditions/totals resolution across both endpoints.
usage_summary(Lines 392-400) andusage_summary_csv(Lines 471-480) call_resolve_window→_usage_filters→_totalsin an identical sequence. Pulling this into a small_summary_context()helper returning(start, end, conditions, totals)would remove the duplication and mean any future fix (like the naive-datetime handling above) only needs to land once.♻️ Sketch
+async def _summary_context( + db: AsyncSession, + *, + start_date: datetime | None, + end_date: datetime | None, + user_id: str | None, + status: str | None, + model: str | None, + endpoint: str | None, +) -> tuple[datetime, datetime, list[ColumnElement[bool]], UsageTotals]: + start, end = _resolve_window(start_date, end_date) + conditions = _usage_filters( + start_date=start, end_date=end, user_id=user_id, status=status, model=model, endpoint=endpoint, + ) + totals = await _totals(db, conditions) + return start, end, conditions, totalsAlso applies to: 471-480
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/gateway/api/routes/usage.py` around lines 392 - 400, Extract the duplicated _resolve_window → _usage_filters → _totals sequence from usage_summary and usage_summary_csv into a shared _summary_context() helper returning (start, end, conditions, totals). Update both endpoints to use this helper while preserving their existing outputs and endpoint-specific behavior.
454-500: 🚀 Performance & Scalability | 🔵 TrivialUncapped CSV export is a deliberate tradeoff — worth a sanity cap.
The docstring explains the export is intentionally uncapped for finance's full-row needs, which is a reasonable product decision. Just flagging that with no ceiling at all, a pathological window (up to the 366-day max) combined with high-cardinality
model/user_id/api_key_idvalues could still buffer an unexpectedly large CSV entirely inio.StringIObefore it's returned. A generous safety cap (e.g. 50k rows) or a streaming response would keep the "give finance everything" intent while guarding against worst-case memory spikes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/gateway/api/routes/usage.py` around lines 454 - 500, Update usage_summary_csv to prevent unbounded in-memory CSV generation for large windows and high-cardinality dimensions. Prefer a streaming response or add a generous safety cap around the breakdown rows and StringIO buffering, while preserving the finance export behavior and existing CSV format.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/gateway/api/routes/usage.py`:
- Around line 246-259: The _resolve_window function must normalize start_date
and end_date to UTC-aware datetimes before evaluating ordering or subtracting
them. Convert naive query values as UTC while preserving already-aware values,
then retain the existing clamping and maximum-span behavior.
In `@web/src/api/types.ts`:
- Around line 344-348: Update UsageGroupRow to add a kind discriminator with
"value", "missing", and "other" variants; make key present only for value rows
and undefined for missing/other rows. Normalize incoming wire null keys at the
API boundary so internal TypeScript types do not use null for absent values,
while preserving the distinction between real NULL groups and synthesized
“other” rows.
In `@web/src/components/ui.tsx`:
- Around line 186-196: Update the combobox synchronization around labelFor and
the text state to derive a selectedLabel from options and value, then sync text
from selectedLabel in the effect. Include selectedLabel’s dependencies so
asynchronously loaded option labels update the input without waiting for value
to change, and remove the suppressed dependency workaround.
In `@web/src/pages/UsagePage.tsx`:
- Around line 295-303: Update the shared usage params builder used by both
summary requests and CSV URLs to serialize the filters.end_date value into
usageParams. Preserve the previousFilters end_date assignment so prevTotals
requests are bounded at the current window’s start, and ensure both generated
request paths honor that upper bound.
---
Nitpick comments:
In `@src/gateway/api/routes/usage.py`:
- Around line 392-400: Extract the duplicated _resolve_window → _usage_filters →
_totals sequence from usage_summary and usage_summary_csv into a shared
_summary_context() helper returning (start, end, conditions, totals). Update
both endpoints to use this helper while preserving their existing outputs and
endpoint-specific behavior.
- Around line 454-500: Update usage_summary_csv to prevent unbounded in-memory
CSV generation for large windows and high-cardinality dimensions. Prefer a
streaming response or add a generous safety cap around the breakdown rows and
StringIO buffering, while preserving the finance export behavior and existing
CSV format.
In `@web/src/pages/ActivityPage.tsx`:
- Around line 224-229: Remove the modelOptions useState and its synchronization
useEffect, and derive modelOptions directly from modelSummary.data?.by_model in
the component render. Preserve filtering out null keys and returning a string[]
with the same empty-result behavior when query data is unavailable.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 58c1de88-3a1d-4945-92e1-32c40196aee6
⛔ Files ignored due to path filters (1)
docs/public/openapi.jsonis excluded by!docs/public/openapi.json
📒 Files selected for processing (18)
.gitignorescripts/seed_usage_smoke.pysrc/gateway/api/routes/usage.pysrc/gateway/static/dashboard/assets/index-Cm2ds2xC.jssrc/gateway/static/dashboard/assets/index-Cs7gh5Mq.jssrc/gateway/static/dashboard/assets/index-ssW08Y1y.csssrc/gateway/static/dashboard/index.htmltests/integration/test_usage_summary.pyweb/src/App.tsxweb/src/api/client.tsweb/src/api/hooks.tsweb/src/api/types.tsweb/src/components/AppShell.tsxweb/src/components/ui.tsxweb/src/pages/ActivityPage.test.tsxweb/src/pages/ActivityPage.tsxweb/src/pages/UsagePage.test.tsxweb/src/pages/UsagePage.tsx
| // One breakdown row (a model, a user, or an API key). `key` is null both for the | ||
| // synthesized "other" fold row and for usage whose grouping column was NULL | ||
| // (e.g. a since-deleted user). | ||
| export interface UsageGroupRow { | ||
| key: string | null; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Distinguish folded rows from missing group values.
key: null represents both a real NULL group and the synthesized “other” fold, so the dashboard cannot label or reconcile them correctly. Add an explicit discriminator such as kind: "value" | "missing" | "other"; make key optional for non-value rows and normalize wire nulls at the API boundary.
As per coding guidelines, “Use undefined rather than null for absent values in own TypeScript types.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@web/src/api/types.ts` around lines 344 - 348, Update UsageGroupRow to add a
kind discriminator with "value", "missing", and "other" variants; make key
present only for value rows and undefined for missing/other rows. Normalize
incoming wire null keys at the API boundary so internal TypeScript types do not
use null for absent values, while preserving the distinction between real NULL
groups and synthesized “other” rows.
Source: Coding guidelines
| const labelFor = (v: string): string => options.find((o) => o.value === v)?.label ?? v; | ||
| const [text, setText] = useState(() => labelFor(value)); | ||
|
|
||
| // Re-sync the input when the committed value changes from outside (a "Clear | ||
| // filters" press, or a drill-down seeding the value). Typing does not change | ||
| // `value` until a selection commits, so this does not fight the user mid-type. | ||
| useEffect(() => { | ||
| setText(labelFor(value)); | ||
| // labelFor closes over options; value is the external source of truth here. | ||
| // eslint-disable-next-line react-hooks/exhaustive-deps | ||
| }, [value]); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant files first.
ast-grep outline web/src/components/ui.tsx --view expanded
printf '\n---\n'
ast-grep outline web/src/pages/ActivityPage.tsx --view expandedRepository: mozilla-ai/otari
Length of output: 1786
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the relevant section of the component and the caller that supplies options.
sed -n '140,240p' web/src/components/ui.tsx
printf '\n--- ActivityPage ---\n'
sed -n '1,240p' web/src/pages/ActivityPage.tsxRepository: mozilla-ai/otari
Length of output: 13255
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find the FilterComboBox definition and any direct uses that might show how options are populated.
rg -n "FilterComboBox|options\s*=" web/src/components/ui.tsx web/src/pages/ActivityPage.tsx web/src -g '!**/node_modules/**'Repository: mozilla-ai/otari
Length of output: 2155
Resync the input when the matching option label arrives.
If options load after a seeded value, the combobox keeps showing the raw ID until value changes again. Derive a selectedLabel from options and sync from that instead of suppressing the dependency.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@web/src/components/ui.tsx` around lines 186 - 196, Update the combobox
synchronization around labelFor and the text state to derive a selectedLabel
from options and value, then sync text from selectedLabel in the effect. Include
selectedLabel’s dependencies so asynchronously loaded option labels update the
input without waiting for value to change, and remove the suppressed dependency
workaround.
…guard delta hint Address self-review findings on the usage analytics page: - Serialize end_date (added to UsageFilters and usageParams) so the period-over-period query uses a bounded window ending at the current window's start, instead of defaulting end to now and overlapping the current period. - Add an is_other flag to breakdown rows so usage from a since-deleted user (a real NULL key) is no longer mislabeled as the synthesized fold row; it now renders as "(unknown)". - Render the delta hints only once totals have loaded, avoiding a brief "-100% vs prev" flicker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an operator-facing Usage & analytics view to the standalone dashboard, backed by new master-key-gated aggregation endpoints over UsageLog so spend/tokens/requests can be summarized by dimension and over time (plus CSV export).
Changes:
- Backend: add
GET /v1/usage/summary(JSON aggregates + UTC bucketed series) andGET /v1/usage/summary.csv(CSV export with formula-injection guard). - Dashboard: add
UsagePage(tiles, breakdown tables, trend chart, drill-down to Activity, CSV export) and enhanceActivityPagewith drill-down seeding + model typeahead suggestions. - Tests: add integration coverage for the new usage summary endpoints and Vitest coverage for the new/updated dashboard flows.
Reviewed changes
Copilot reviewed 15 out of 19 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/gateway/api/routes/usage.py |
Adds summary + CSV aggregation endpoints and supporting SQL helpers. |
tests/integration/test_usage_summary.py |
Validates aggregation correctness, bucketing, folding reconciliation, and CSV injection guard. |
web/src/pages/UsagePage.tsx |
New dashboard Usage & analytics page (filters, tiles, breakdowns, chart, CSV export). |
web/src/pages/UsagePage.test.tsx |
UI tests for totals, filters, drill-down, chart toggle, and CSV export. |
web/src/pages/ActivityPage.tsx |
Seeds filters from drill-down query string; replaces select/text model filter with typeahead + suggestions. |
web/src/pages/ActivityPage.test.tsx |
Tests model suggestions and drill-down seeding behavior. |
web/src/components/ui.tsx |
Introduces FilterComboBox for large option sets (users/models). |
web/src/api/types.ts |
Adds types for usage summary response (totals, breakdowns, series). |
web/src/api/hooks.ts |
Adds useUsageSummary and shared filter serialization support for end_date. |
web/src/api/client.ts |
Adds apiFetchBlob for authenticated non-JSON downloads. |
web/src/components/AppShell.tsx |
Adds “Usage” nav entry under Observability. |
web/src/App.tsx |
Wires /usage route to UsagePage. |
src/gateway/static/dashboard/index.html |
Updates built dashboard asset hashes. |
docs/public/openapi.json |
Updates OpenAPI spec with new schemas and routes. |
scripts/seed_usage_smoke.py |
Adds local smoke-test seeding helper for usage logs. |
.gitignore |
Ignores local smoke DB files. |
| <ComboBox.InputGroup> | ||
| <Input placeholder={placeholder} autoComplete="off" onFocus={(event) => event.currentTarget.select()} /> | ||
| <ComboBox.Trigger /> | ||
| </ComboBox.InputGroup> |
| UsageGroupRow( | ||
| key=None, | ||
| cost=totals.cost - sum(r.cost for r in result), | ||
| tokens=totals.total_tokens - sum(r.tokens for r in result), |
| def _csv_safe(value: str) -> str: | ||
| if value and value[0] in _CSV_FORMULA_PREFIXES: | ||
| return "'" + value | ||
| return value |
| return value | ||
|
|
||
|
|
||
| @router.get("/summary.csv", dependencies=[Depends(verify_master_key)]) |
| const [rangeSeconds, setRangeSeconds] = useState<number | null>(initialStart ? null : DAY_S); | ||
| // Snapshotted when a range is picked (and on refresh), so the query key is | ||
| // stable across renders instead of recomputing "now" every render. | ||
| const [startDate, setStartDate] = useState<string | undefined>(() => isoAgo(DAY_S)); | ||
| const [statusFilter, setStatusFilter] = useState(""); | ||
| const [modelFilter, setModelFilter] = useState(""); | ||
| const [startDate, setStartDate] = useState<string | undefined>(() => initialStart ?? isoAgo(DAY_S)); | ||
| const [statusFilter, setStatusFilter] = useState(() => searchParams.get("status") ?? ""); |
khaledosman
left a comment
There was a problem hiding this comment.
Solid work. Both endpoints are master-key gated and standalone-only, SQL is parameterized (the one dynamic token is a validated Literal), the window is hard-bounded, the top-N "other" fold reconciles, the CSV formula-injection guard is in place, and cross-dialect UTC bucketing is correct. Approving to unblock; one Medium worth fixing before/right after merge, plus two Low notes inline.
Medium: _resolve_window can 500 on a naive start_date (see inline). The dashboard always sends ...Z so it is unaffected, but a direct API client passing a valid offset-less ISO datetime hits it.
Review by Claude Code (Opus 4.8), driven by @khaledosman.
| ``_DEFAULT_SUMMARY_LOOKBACK``; a span wider than ``_MAX_SUMMARY_SPAN`` has its | ||
| start pulled forward so the aggregates stay bounded by the timestamp index. | ||
| """ | ||
| end = end_date or datetime.now(UTC) |
There was a problem hiding this comment.
Medium: datetime.now(UTC) is tz-aware, but a query start_date without an offset (e.g. ?start_date=2025-01-01T00:00:00, a value the param description advertises as valid ISO-8601) parses to a naive datetime. With end_date omitted, start > end on the next line raises TypeError: can't compare offset-naive and offset-aware datetimes -> unhandled 500. The epoch form parses to aware UTC and works, so the crash is input-shape-dependent. Normalize both bounds to aware UTC at the top of _resolve_window (e.g. dt.replace(tzinfo=UTC) if dt.tzinfo is None else dt) and add a naive-input test.
| }), | ||
| [startDate, statusFilter, endpointFilter, userFilter], | ||
| ); | ||
| const modelSummary = useUsageSummary(modelSuggestFilters, "day"); |
There was a problem hiding this comment.
Low (perf): this calls useUsageSummary only to populate the model typeahead, but usage_summary runs five aggregate GROUP BY scans (totals + 3 breakdowns + series) and refires on every start_date/status/endpoint/user_id change. Only by_model keys are consumed, and by_model is the intentionally-unindexed dimension. Consider a lighter endpoint/param that returns just distinct in-window model names, or document the cost as a deliberate trade-off.
| {series.map((point, i) => { | ||
| const value = metricValue(point, metric); | ||
| const h = (value / max) * (height - pad); | ||
| const x = pad + i * slot + (slot - barW) / 2; |
There was a problem hiding this comment.
Low: bars are positioned by array index (pad + i * slot), but the backend series omits empty buckets (GROUP BY). A sparse range (e.g. usage on day 1 and day 20 of a 30-day window) renders as two adjacent bars, so the x-axis is not linear in time and the trend reads misleadingly. Zero-fill the series server-side (bounded by the span cap) or lay bars out by timestamp rather than index.
…reamble Address review feedback from @khaledosman and CodeRabbit on the usage summary: - _resolve_window normalizes offset-less (naive) start_date/end_date to UTC before comparing them to the aware "now", so an ISO datetime without an offset (advertised as valid) no longer 500s. Adds a naive-input test. - Zero-fill the time series across empty buckets (bounded by a max-points cap) so a sparse range renders with a time-linear x-axis instead of a few adjacent bars. Adds a gap-day test. - Extract the shared _resolve_window / _usage_filters / _totals preamble into _summary_context, used by both the JSON and CSV endpoints. - Derive the model typeahead options directly from query data instead of mirroring server state into useState (Usage and Activity pages); the Usage page sources them from a summary that omits the model filter so the list stays complete while a model is selected. OpenAPI regenerated and the dashboard bundle rebuilt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI follow-ups: - _dense_series returns an empty series when the window has no rows at all, instead of a full window of zero buckets, restoring the empty-range contract while still gap-filling any window that does have data. - Regenerate the Postman collection so make postman-check passes; the new summary endpoints had left it stale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What
Adds the Usage & analytics page (issue #303) and the aggregation API behind it, on the observability track of #299. Operators can now see aggregate spend, tokens, and request volume by model, by user, and over time, drill from any breakdown row into the Activity log, and export to CSV.
Backend (
src/gateway/api/routes/usage.py)GET /v1/usage/summary(master-key gated, JSON): totals,by_model/by_user/by_api_key, and a UTC-bucketed time series, all reusing the existing_usage_filters.GET /v1/usage/summary.csv: uncapped CSV export (finance wants every row), with a spreadsheet formula-injection guard, kept on its own route so/summarykeeps one clean JSON response model.timestampindex.strftime) and PostgreSQL (date_trunc), pinned to UTC withAT TIME ZONE 'UTC'so buckets do not shift with the server timezone or break across DST.modelindex would not help the group-by; range-bounding does the work).Dashboard (
web/)UsagePage: stat tiles with period-over-period deltas, spend-by-model and spend-by-user tables whose rows deep-link into the Activity log pre-filtered on that dimension and window, a single-metric SVG bar chart with a Cost/Tokens/Requests toggle (hand-rolled, no new chart dependency), and CSV export.FilterComboBox: a type-to-filter combobox replacing the native<select>(unusable at thousands of users) and the exact-match model text box. The Usage model picker is sourced from the window's ownby_model; the Activity model picker gains the same typeahead with suggestions sourced from usage so they match what the log stores, while still accepting any typed model.ActivityPageseeds its filters from the drill-down query string.Testing
tests/integration/test_usage_summary.py: aggregation correctness, cross-dialect UTC bucketing, top-N reconciliation, empty/null/ZeroDivisionguards, and CSV shape plus injection. Runs on both SQLite and PostgreSQL in CI (needs Docker; verified locally on SQLite end to end).Notes
scripts/seed_usage_smoke.pyis a local smoke-test helper; happy to drop it if you would rather not ship it.Closes #303. Part of #299.
🤖 Generated with Claude Code
Summary