Skip to content

feat: attribute model usage events to agent, model and provider - #2219

Open
Douglasymlai wants to merge 3 commits into
mainfrom
usage-event-attribution
Open

Douglasymlai wants to merge 3 commits into
mainfrom
usage-event-attribution

Conversation

@Douglasymlai

@Douglasymlai Douglasymlai commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What

The model.invocation.* Run events now carry enough to group token usage by agent, model and provider, and to see how long each call took.

  • model.invocation.dispatched gains agent_name.
  • Every terminal event (completed, failed, outcome_unknown) gains agent_name, provider, model and duration_ms. This covers all three places that close a call: the normal finish, the run-end sweep, and startup reconciliation after a restart.
  • duration_ms is completed_at - started_at for calls that finish or fail normally. It is null when the call's end time is unknown: for calls closed as outcome_unknown by the run-end sweep or by startup reconciliation (those run when the Run ends or the app restarts, not when the call ended), or if the clock went backwards.

agent_id is a random id per agent instance, so it could not be used to group calls by agent before.

It also makes prompt_tokens mean the same thing for every provider: total input, including cached input. Uncached input is then always prompt_tokens - cache_read_tokens - cache_write_tokens.

  • OpenAI Chat Completions and Responses already report total input. No change, except that Responses cached tokens are now read from input_tokens_details.
  • Anthropic reports input_tokens without cached input, plus cache_read_input_tokens and cache_creation_input_tokens. CAMEL's Anthropic backend copies the uncached count into prompt_tokens, so cached Anthropic calls were under-counted. They are now input + cache_read + cache_write.
  • When an OpenAI-compatible gateway sends both an inclusive prompt_tokens with prompt_tokens_details and the Anthropic cache fields, the details object marks the count as inclusive and nothing is added.

For Anthropic calls that hit the prompt cache, the token totals shown in the app will go up, to match how OpenAI calls were already counted.

How

  • instrument_model_backend() takes an optional agent_name, and agent_model() passes the agent's name in.
  • model_invocations gets a nullable agent_name column (RunJournal schema v44). Rows from older versions keep NULL, and their events report "agent_name": null. The migration skips the ALTER if the column already exists.
  • The three terminal payloads were built separately; they now come from one helper so they cannot drift apart.

Compatibility

  • Additive only. The nested usage object is unchanged, so runUsageReconciliation.ts and other readers work as before.
  • New keys are always present (as null when unknown) so consumers can rely on a fixed key set.
  • An older app build will refuse a journal at schema v44, as with every earlier schema bump.

Privacy

Prompt and response text still never enter Run events. New tests assert that the Cloud-projected payload of each event type has exactly the allowed key set, and that request text, response text and error messages do not appear in it.

agent_name is new data that reaches Cloud. For built-in agents it is a fixed name; for agents a user creates, it is the name the user gave that agent. It is a short label, never prompt text.

Tests

  • New: tests/app/run_journal/test_model_invocation_usage_events.py (key sets for every event type, duration, run-end sweep, startup reconciliation, missing agent name, v43 to v44 upgrade with an in-flight call).
  • New: capture-level test in test_model_capture.py that the agent name flows from instrument_model_backend() to the events.
  • New: _usage() cases for OpenAI Chat, OpenAI Responses, native Anthropic, CAMEL's Anthropic shape, a gateway sending both, and missing usage.
  • Upgrade tests that stub later migrations now also stub v44, and the tests pinned to schema 43 now expect 44.
  • Full backend suite compared with the base commit on Windows (run on the first commit; tests/app/run_journal and tests/app/model were rerun after each later commit): 445 failed and 208 errors on this branch, all of which also fail or error on the base commit (mostly workspace_runtime symlink, git and tokenizer-asset setup on Windows, plus one frontend-vocabulary check). No new failures. tests/app/run_journal and tests/app/model pass apart from that same pre-existing vocabulary check. Ruff 0.14.3 lint and format pass on the changed files.

Not verified

  • The case of a gateway such as LiteLLM in front of Anthropic models (inclusive prompt_tokens with prompt_tokens_details plus the Anthropic cache fields) is covered by a hand-written fixture, not a recorded response.

🤖 Generated with Claude Code

Douglasymlai and others added 2 commits November 20, 2026 02:04
model.invocation.dispatched now carries agent_name, and every terminal
model.invocation event (completed, failed, outcome_unknown from the
run-end sweep and from startup reconciliation) carries agent_name,
provider, model and duration_ms. agent_id is a random per-instance id,
so usage could not be grouped by agent before.

agent_name is stored on model_invocations (RunJournal schema v44,
nullable; rows from older versions keep NULL). The three terminal
payloads now come from one helper. The nested usage object is
unchanged, and tests pin the exact synced key set so prompt and
response text cannot leak into Run events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prompt_tokens on a model invocation now always includes cached input,
so uncached input is prompt - cache_read - cache_write for every
provider. Anthropic reports input_tokens without cached input, and
CAMEL's Anthropic backend copies that into prompt_tokens next to the
separate cache fields, so those calls were under-counted. OpenAI-style
usage (prompt_tokens or Responses input_tokens with *_tokens_details)
is already inclusive and is unchanged. Responses API cached tokens are
now read from input_tokens_details.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

Claude Code Review is paused for this repository because its GitHub organization is not linked to your Claude organization.

To turn it back on, one person needs both roles: owner of the GitHub organization (or the account owner, for a personal repository) and admin or owner of your Claude organization. If that is not you, pass this on to someone who is.

Open Organization settings, Git providers and click Link next to the GitHub organization's name. If it is not listed, click Enable GitHub integration or Connect your GitHub first, whichever the page shows.

New pull requests are then reviewed as before. To get a review on one that is already open, someone with write access can comment @claude review.

Tip: disable this comment in your organization's Code Review settings.

The run-end sweep closes calls when the Run ends, not when the call
ended, so sweep time minus start time is not the call's duration.
Send duration_ms as null there, as startup reconciliation already
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant