fix(summarize): make its per-call budget configurable, and count the calls it abandons - #65
Merged
OsherElhadad merged 1 commit intoAug 13, 2026
Conversation
…calls it abandons rossoctl#64 did this for extract_llm and left summarize on a hardcoded `summarizeCallTimeout = 150 * time.Second`. That 150s was sized against an IDLE server, and the 10x asymmetry with extract_llm's old 15s hid the fact that it has the same failure mode: on a loaded self-hosted backend the deadline can expire before the model answers, and the operator has no knob and no counter to see it. Two things make summarize's budget the harder of the two to size, and they ADD: - queue wait, which is what a loaded server actually charges: p50 17.2s / p95 78.8s measured under KV-cache pressure, before the model starts work at all; - this component's own prefill, which is large by construction. Measured over one 50-task SWE-bench arm: 78,155,276 input tokens across 1,372 calls, i.e. ~57k prompt tokens per call, because it summarizes the whole middle of the transcript. The same arm spent 26,890,609 ms on those calls — a 19.6s mean, so the idle-server figure measured only the third of those three terms. So: - CONTEXT_GURU_SUMMARIZE_TIMEOUT (Go duration; bare integers are seconds) now sets the budget, defaulting to 300s. It is deliberately SEPARATE from CONTEXT_GURU_LLM_TIMEOUT: the two components send requests that differ in size by ~20x, so a single ceiling is either generous for one or tight for the other. - summarize_timeouts / summarize_errors / summarize_call_timeout_ms are served at /stats, merged by the host with the same layering as the LLM* and Frozen* counters, and registered in the /stats golden contract. - The timeout is distinguished from a transport error. summarize's fail path is louder than extract_llm's — it returns the error and the pipeline reverts the component, which shows up as a per-component `reverted` count — but `reverted` cannot say WHY, and the two causes call for opposite responses: a blown deadline means the budget is too small for this load (savings are an undercount), while a model error means the compaction route is broken (the arm is not measuring summarization at all). - The retry loop now stops once the shared deadline has expired. The ctx is built by the caller, OUTSIDE the loop, so all three attempts share one deadline: past it, each retry failed instantly and only obscured the cause. This also documents that the worst case is one budget, not three. The env parsing is extracted to resolveTimeoutEnv and shared with extract_llm rather than duplicated. A value accepted by one component and silently ignored by the other would be invisible in a run and would look like the component not firing. Fail-open behaviour is unchanged throughout: a summarize that cannot summarize must leave the request valid. What changes is that a loaded server gets time to answer, and that giving up is now countable. Tested: a slow model that always exhausts the deadline is counted as a timeout and not as an error, and the message list is left intact on the error path (summarize is the one component that changes the message COUNT, so a partial rebuild would hand the caller a transcript with no summary in it). Plus a table test for the shared parser: bare integers, Go durations, and fallback on zero/negative/garbage. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Itay-Nakash <itay.nakash@ibm.com>
OsherElhadad
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#64 gave
extract_llma configurable per-call budget and counters for the calls itabandons.
summarizewas left on a hardcodedsummarizeCallTimeout = 150 * time.Second— and the 10x asymmetry with the old 15s ceiling hid the fact that it has the same
failure mode: on a loaded self-hosted backend the deadline can expire before the model
answers, and today the operator has neither a knob nor a counter to see it happen.
Two terms make summarize's budget the harder of the two to size, and they add:
under KV-cache pressure, before the model starts work at all.
arm: 78,155,276 input tokens across 1,372 calls, i.e. ~57k prompt tokens per call,
because it summarizes the whole middle of the transcript.
The same arm spent 26,890,609 ms on those calls — a 19.6s mean on an idle server. So
150s looked like 7.6x headroom while only measuring the third of those three terms.
What
CONTEXT_GURU_SUMMARIZE_TIMEOUT(Go duration; bare integers are seconds) sets thebudget, defaulting to 300s. Deliberately separate from
CONTEXT_GURU_LLM_TIMEOUT: thetwo components send requests differing in size by ~20x, so one ceiling is either
generous for one or tight for the other.
summarize_timeouts/summarize_errors/summarize_call_timeout_msat/stats,merged by the host with the same layering as the
LLM*andFrozen*counters, andregistered in the
/statsgolden contract. Separate fromllm_timeoutsfor the samereason as the budget: a summarize-only pipeline reports
llm_timeouts0 however badlyits own deadline is being hit.
extract_llm's — it returns the error and the pipeline reverts the component, which
surfaces as a per-component
revertedcount — butrevertedcannot say why, and thetwo causes call for opposite responses: a blown deadline means the budget is too small
for this load (savings are an undercount), a model error means the compaction route is
broken (the arm is not measuring summarization at all).
caller, outside the loop, so all three attempts share one deadline; past it each retry
failed instantly and only obscured the cause. This also makes explicit that the worst
case is one budget, not three.
resolveTimeoutEnvand shared with extract_llm rather thanduplicated — a value accepted by one component and silently ignored by the other would
be invisible in a run and would look like the component not firing.
Fail-open behaviour is unchanged throughout: a summarize that cannot summarize must leave
the request valid. What changes is that a loaded server gets time to answer, and that
giving up is now countable.
Testing
make test(-race) andmake lintgreen; the suite also passes with-tags cg_skeleton.New
summarize_timeout_test.go:and the model is asserted to have actually been called (otherwise a component that
merely declined would make the assertion vacuous — the same trap called out in the
extract_llm timeout test);
changes the message count, so a partial rebuild would hand the caller a transcript with
no summary in it;
and fallback on zero / negative / garbage.
Note on the default
300s is a ceiling, not a target — on an idle server nothing changes, because the call
still returns in ~20s. It is sized so that queue wait plus a ~57k-token prefill still fit.
The cost of a large ceiling lands on the agent (the call is on the request's hot path), so
the counters matter as much as the number: past some load level the honest answer stops
being "raise the budget" and becomes "this concurrency cannot serve compaction and the
agent at once".