Repository navigation
feat: attribute model usage events to agent, model and provider - #2219
Douglasymlai wants to merge 3 commits into
Conversation
model.invocation.dispatched now carries agent_name, and every terminal model.invocation event (completed, failed, outcome_unknown from the run-end sweep and from startup reconciliation) carries agent_name, provider, model and duration_ms. agent_id is a random per-instance id, so usage could not be grouped by agent before. agent_name is stored on model_invocations (RunJournal schema v44, nullable; rows from older versions keep NULL). The three terminal payloads now come from one helper. The nested usage object is unchanged, and tests pin the exact synced key set so prompt and response text cannot leak into Run events. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
prompt_tokens on a model invocation now always includes cached input, so uncached input is prompt - cache_read - cache_write for every provider. Anthropic reports input_tokens without cached input, and CAMEL's Anthropic backend copies that into prompt_tokens next to the separate cache fields, so those calls were under-counted. OpenAI-style usage (prompt_tokens or Responses input_tokens with *_tokens_details) is already inclusive and is unchanged. Responses API cached tokens are now read from input_tokens_details. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Claude Code Review
Claude Code Review is paused for this repository because its GitHub organization is not linked to your Claude organization.
To turn it back on, one person needs both roles: owner of the GitHub organization (or the account owner, for a personal repository) and admin or owner of your Claude organization. If that is not you, pass this on to someone who is.
Open Organization settings, Git providers and click Link next to the GitHub organization's name. If it is not listed, click Enable GitHub integration or Connect your GitHub first, whichever the page shows.
New pull requests are then reviewed as before. To get a review on one that is already open, someone with write access can comment @claude review.
Tip: disable this comment in your organization's Code Review settings.
The run-end sweep closes calls when the Run ends, not when the call ended, so sweep time minus start time is not the call's duration. Send duration_ms as null there, as startup reconciliation already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
The
model.invocation.*Run events now carry enough to group token usage by agent, model and provider, and to see how long each call took.model.invocation.dispatchedgainsagent_name.completed,failed,outcome_unknown) gainsagent_name,provider,modelandduration_ms. This covers all three places that close a call: the normal finish, the run-end sweep, and startup reconciliation after a restart.duration_msiscompleted_at - started_atfor calls that finish or fail normally. It isnullwhen the call's end time is unknown: for calls closed asoutcome_unknownby the run-end sweep or by startup reconciliation (those run when the Run ends or the app restarts, not when the call ended), or if the clock went backwards.agent_idis a random id per agent instance, so it could not be used to group calls by agent before.It also makes
prompt_tokensmean the same thing for every provider: total input, including cached input. Uncached input is then alwaysprompt_tokens - cache_read_tokens - cache_write_tokens.input_tokens_details.input_tokenswithout cached input, pluscache_read_input_tokensandcache_creation_input_tokens. CAMEL's Anthropic backend copies the uncached count intoprompt_tokens, so cached Anthropic calls were under-counted. They are nowinput + cache_read + cache_write.prompt_tokenswithprompt_tokens_detailsand the Anthropic cache fields, the details object marks the count as inclusive and nothing is added.For Anthropic calls that hit the prompt cache, the token totals shown in the app will go up, to match how OpenAI calls were already counted.
How
instrument_model_backend()takes an optionalagent_name, andagent_model()passes the agent's name in.model_invocationsgets a nullableagent_namecolumn (RunJournal schema v44). Rows from older versions keepNULL, and their events report"agent_name": null. The migration skips theALTERif the column already exists.Compatibility
usageobject is unchanged, sorunUsageReconciliation.tsand other readers work as before.nullwhen unknown) so consumers can rely on a fixed key set.Privacy
Prompt and response text still never enter Run events. New tests assert that the Cloud-projected payload of each event type has exactly the allowed key set, and that request text, response text and error messages do not appear in it.
agent_nameis new data that reaches Cloud. For built-in agents it is a fixed name; for agents a user creates, it is the name the user gave that agent. It is a short label, never prompt text.Tests
tests/app/run_journal/test_model_invocation_usage_events.py(key sets for every event type, duration, run-end sweep, startup reconciliation, missing agent name, v43 to v44 upgrade with an in-flight call).test_model_capture.pythat the agent name flows frominstrument_model_backend()to the events._usage()cases for OpenAI Chat, OpenAI Responses, native Anthropic, CAMEL's Anthropic shape, a gateway sending both, and missing usage.tests/app/run_journalandtests/app/modelwere rerun after each later commit): 445 failed and 208 errors on this branch, all of which also fail or error on the base commit (mostlyworkspace_runtimesymlink, git and tokenizer-asset setup on Windows, plus one frontend-vocabulary check). No new failures.tests/app/run_journalandtests/app/modelpass apart from that same pre-existing vocabulary check. Ruff 0.14.3 lint and format pass on the changed files.Not verified
prompt_tokenswithprompt_tokens_detailsplus the Anthropic cache fields) is covered by a hand-written fixture, not a recorded response.🤖 Generated with Claude Code