Skip to content

fix(summarize): make its per-call budget configurable, and count the calls it abandons - #65

Merged
OsherElhadad merged 1 commit into
rossoctl:mainfrom
itay-nakash:feat/configurable-summarize-timeout
Aug 13, 2026
Merged

fix(summarize): make its per-call budget configurable, and count the calls it abandons#65
OsherElhadad merged 1 commit into
rossoctl:mainfrom
itay-nakash:feat/configurable-summarize-timeout

Conversation

@itay-nakash

Copy link
Copy Markdown
Collaborator

Why

#64 gave extract_llm a configurable per-call budget and counters for the calls it
abandons. summarize was left on a hardcoded summarizeCallTimeout = 150 * time.Second
— and the 10x asymmetry with the old 15s ceiling hid the fact that it has the same
failure mode: on a loaded self-hosted backend the deadline can expire before the model
answers, and today the operator has neither a knob nor a counter to see it happen.

Two terms make summarize's budget the harder of the two to size, and they add:

  • Queue wait — what a loaded server actually charges: measured p50 17.2s / p95 78.8s
    under KV-cache pressure, before the model starts work at all.
  • Its own prefill, which is large by construction. Measured over one 50-task SWE-bench
    arm: 78,155,276 input tokens across 1,372 calls, i.e. ~57k prompt tokens per call,
    because it summarizes the whole middle of the transcript.

The same arm spent 26,890,609 ms on those calls — a 19.6s mean on an idle server. So
150s looked like 7.6x headroom while only measuring the third of those three terms.

What

  • CONTEXT_GURU_SUMMARIZE_TIMEOUT (Go duration; bare integers are seconds) sets the
    budget, defaulting to 300s. Deliberately separate from CONTEXT_GURU_LLM_TIMEOUT: the
    two components send requests differing in size by ~20x, so one ceiling is either
    generous for one or tight for the other.
  • summarize_timeouts / summarize_errors / summarize_call_timeout_ms at /stats,
    merged by the host with the same layering as the LLM* and Frozen* counters, and
    registered in the /stats golden contract. Separate from llm_timeouts for the same
    reason as the budget: a summarize-only pipeline reports llm_timeouts 0 however badly
    its own deadline is being hit.
  • Timeout distinguished from transport error. summarize's fail path is louder than
    extract_llm's — it returns the error and the pipeline reverts the component, which
    surfaces as a per-component reverted count — but reverted cannot say why, and the
    two causes call for opposite responses: a blown deadline means the budget is too small
    for this load (savings are an undercount), a model error means the compaction route is
    broken (the arm is not measuring summarization at all).
  • The retry loop stops once the shared deadline has expired. The ctx is built by the
    caller, outside the loop, so all three attempts share one deadline; past it each retry
    failed instantly and only obscured the cause. This also makes explicit that the worst
    case is one budget, not three.
  • Env parsing extracted to resolveTimeoutEnv and shared with extract_llm rather than
    duplicated — a value accepted by one component and silently ignored by the other would
    be invisible in a run and would look like the component not firing.

Fail-open behaviour is unchanged throughout: a summarize that cannot summarize must leave
the request valid. What changes is that a loaded server gets time to answer, and that
giving up is now countable.

Testing

make test (-race) and make lint green; the suite also passes with -tags cg_skeleton.

New summarize_timeout_test.go:

  • a slow model that always exhausts the deadline is counted as a timeout, not an error,
    and the model is asserted to have actually been called (otherwise a component that
    merely declined would make the assertion vacuous — the same trap called out in the
    extract_llm timeout test);
  • the message list is left intact on the error path. summarize is the one component that
    changes the message count, so a partial rebuild would hand the caller a transcript with
    no summary in it;
  • a table test for the shared parser: bare integers as seconds, Go durations, whitespace,
    and fallback on zero / negative / garbage.

Note on the default

300s is a ceiling, not a target — on an idle server nothing changes, because the call
still returns in ~20s. It is sized so that queue wait plus a ~57k-token prefill still fit.
The cost of a large ceiling lands on the agent (the call is on the request's hot path), so
the counters matter as much as the number: past some load level the honest answer stops
being "raise the budget" and becomes "this concurrency cannot serve compaction and the
agent at once".

…calls it abandons

rossoctl#64 did this for extract_llm and left summarize on a hardcoded
`summarizeCallTimeout = 150 * time.Second`. That 150s was sized against an IDLE
server, and the 10x asymmetry with extract_llm's old 15s hid the fact that it has
the same failure mode: on a loaded self-hosted backend the deadline can expire
before the model answers, and the operator has no knob and no counter to see it.

Two things make summarize's budget the harder of the two to size, and they ADD:

- queue wait, which is what a loaded server actually charges: p50 17.2s / p95
  78.8s measured under KV-cache pressure, before the model starts work at all;
- this component's own prefill, which is large by construction. Measured over one
  50-task SWE-bench arm: 78,155,276 input tokens across 1,372 calls, i.e. ~57k
  prompt tokens per call, because it summarizes the whole middle of the
  transcript. The same arm spent 26,890,609 ms on those calls — a 19.6s mean, so
  the idle-server figure measured only the third of those three terms.

So:

- CONTEXT_GURU_SUMMARIZE_TIMEOUT (Go duration; bare integers are seconds) now
  sets the budget, defaulting to 300s. It is deliberately SEPARATE from
  CONTEXT_GURU_LLM_TIMEOUT: the two components send requests that differ in size
  by ~20x, so a single ceiling is either generous for one or tight for the other.
- summarize_timeouts / summarize_errors / summarize_call_timeout_ms are served at
  /stats, merged by the host with the same layering as the LLM* and Frozen*
  counters, and registered in the /stats golden contract.
- The timeout is distinguished from a transport error. summarize's fail path is
  louder than extract_llm's — it returns the error and the pipeline reverts the
  component, which shows up as a per-component `reverted` count — but `reverted`
  cannot say WHY, and the two causes call for opposite responses: a blown deadline
  means the budget is too small for this load (savings are an undercount), while a
  model error means the compaction route is broken (the arm is not measuring
  summarization at all).
- The retry loop now stops once the shared deadline has expired. The ctx is built
  by the caller, OUTSIDE the loop, so all three attempts share one deadline: past
  it, each retry failed instantly and only obscured the cause. This also documents
  that the worst case is one budget, not three.

The env parsing is extracted to resolveTimeoutEnv and shared with extract_llm
rather than duplicated. A value accepted by one component and silently ignored by
the other would be invisible in a run and would look like the component not
firing.

Fail-open behaviour is unchanged throughout: a summarize that cannot summarize
must leave the request valid. What changes is that a loaded server gets time to
answer, and that giving up is now countable.

Tested: a slow model that always exhausts the deadline is counted as a timeout and
not as an error, and the message list is left intact on the error path (summarize
is the one component that changes the message COUNT, so a partial rebuild would
hand the caller a transcript with no summary in it). Plus a table test for the
shared parser: bare integers, Go durations, and fallback on zero/negative/garbage.

Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Itay-Nakash <itay.nakash@ibm.com>
@OsherElhadad
OsherElhadad merged commit 54a8738 into rossoctl:main Aug 13, 2026
4 checks passed
@github-project-automation github-project-automation Bot moved this from New/ToDo to Done in Rossoctl Issue Prioritization Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants