Token widgets over-report ~1.84x: one JSONL entry per content block is counted per API call

[rborkow]

Summary

The transcript-summing path in src/utils/jsonl-metrics.ts counts several JSONL entries per API call, so tokens-cached, tokens-input, tokens-output and tokens-total over-report. Measured across 148 transcripts / 11,007 real API calls on ccstatusline 2.2.27 + Claude Code 2.1.222–2.1.228:

widget	shown	actual	over-report
tokens-cached	6,062,470,358	3,292,144,479	1.84x
tokens-output	20,390,026	9,484,311	2.15x
tokens-input	2,770,164	1,240,935	2.23x

context-length and context-percentage-* are not affected — they read a single entry rather than summing.

Root cause

The dedup filter at jsonl-metrics.ts:207-212 assumes duplicate entries are streaming partials carrying stop_reason: null:

const entriesToCount = hasStopReasonField
    ? parsedEntries.filter((entry, index) => {
        const stopReason = entry.data.message?.stop_reason;
        return Boolean(stopReason) || (stopReason === null && index === parsedEntries.length - 1);
    })
    : parsedEntries;

That holds for one transcript shape but not the dominant one. Claude Code also writes one entry per content block — thinking, text, and each tool_use — and those entries all share a single message.id and the final non-null stop_reason. Boolean(stopReason) is true for every one of them, so none are filtered.

Grouping messages by shape:

share	shape	filter outcome
57.3%	multi-entry, all entries carry a non-null stop_reason	fails — every duplicate kept
0.6%	multi-entry, several non-null	fails
12.3%	multi-entry, exactly one non-null (the streaming-partial case)	works as intended
29.8%	single entry	n/a

So the filter is doing real work for the 12.3% — this isn't a "revert it" fix.

Why prompt-side fields are safe to dedup, and output needs care

Comparing entries that share a message.id, only output_tokens differs, and it is non-decreasing — the last entry carries the complete value. input_tokens, cache_read_input_tokens and cache_creation_input_tokens are byte-identical across an id's entries. Measured: 1,390 ids disagreed on usage, and in all 1,390 the differing field set was exactly {output}, always non-decreasing.

message …tlpxvdiq   (3 JSONL entries, one message.id)
  in=2  out=1     cread=3978  ccreate=12278  stop=None       blocks=['text']
  in=2  out=1     cread=3978  ccreate=12278  stop=None       blocks=['tool_use']
  in=2  out=287   cread=3978  ccreate=12278  stop=tool_use   blocks=['tool_use']

Suggested fix

Dedup by message.id, keeping the entry with the highest output_tokens. This subsumes the streaming-partial case (the partial always has a lower output_tokens than the finalized entry), so it handles both shapes and doesn't depend on stop_reason semantics staying put.

// Claude Code writes one JSONL entry per content block (thinking / text / each
// tool_use). All entries for one API call share a message.id and repeat identical
// prompt-side usage; only output_tokens grows, and is complete on the last one.
// Dedup by id, keeping the highest output_tokens, then restore transcript order
// so the compaction / most-recent-entry logic below is unchanged.
const byId = new Map<string, { data: TranscriptLine; lineIndex: number }>();
const withoutId: typeof parsedEntries = [];
for (const entry of parsedEntries) {
    const id = entry.data.message?.id;
    if (typeof id !== 'string' || id.length === 0) {
        withoutId.push(entry);
        continue;
    }
    const prev = byId.get(id);
    const out = entry.data.message?.usage?.output_tokens ?? 0;
    const prevOut = prev?.data.message?.usage?.output_tokens ?? 0;
    if (!prev || out >= prevOut) {
        byId.set(id, entry);
    }
}
const entriesToCount = [...byId.values(), ...withoutId]
    .sort((a, b) => a.lineIndex - b.lineIndex);

Notes:

• message.id was present on all 23,168 usage-bearing entries scanned, but the withoutId passthrough keeps the old behaviour if that ever changes.
• Live/in-flight updates are preserved: the current in-flight message still has an id, so it is still counted.
• contextLength is unaffected, since it uses only prompt-side fields, which are identical across an id's entries.
• hasStopReasonField becomes unused and can be dropped.

Repro

Run against your own transcripts — prints what ccstatusline counts vs. distinct message.id:

import json, glob, os, collections
root = os.path.expanduser("~/.claude/projects")
counted = ids = 0
for f in glob.glob(os.path.join(root, "**", "*.jsonl"), recursive=True):
    entries = []
    for line in open(f, errors="replace"):
        try: d = json.loads(line)
        except Exception: continue
        if (d.get("message") or {}).get("usage"): entries.append(d)
    if not entries: continue
    has_sr = any("stop_reason" in e["message"] for e in entries)
    kept = [e for i, e in enumerate(entries)
            if (e["message"].get("stop_reason")
                or (e["message"].get("stop_reason") is None and i == len(entries) - 1))] if has_sr else entries
    counted += len(kept)
    ids += len({e["message"].get("id") for e in entries})
print(f"counted by ccstatusline : {counted:,}")
print(f"real API calls          : {ids:,}")
print(f"inflation               : {counted/ids:.2f}x")

Observed: counted 20,307 / real 11,007 / 1.84x.

Possibly related

#439 covered a different mismatch in the same equation (statusline context_window fields vs. transcript sums) and is closed. This one is entirely within the transcript-summing path, so the four widgets can be internally consistent and still all be ~1.8x high together.

Happy to send a PR if the approach looks right.

[bmihaila-bd]

Independent confirmation from a different environment (Claude Code 2.1.231).

One session showed 180 usage-bearing JSONL lines collapsing to 89 unique message.ids — about 2.02x inflation on the cumulative token widgets.

Same root cause: every line for a given id shares that id and a non-null stop_reason.

In this session's case, duplicate lines for a given id also carried byte-identical usage — not just the prompt-side fields, but output_tokens too. That's consistent with Claude Code splitting an already-completed multi-content-block message across lines, rather than writing genuine streaming partials.

The proposed fix ("dedup by message.id, keep max output_tokens") handles this case correctly as well, since all copies tie and any one gets kept.