feat(infinity-provider-protocol,infinity-provider-bedrock,infinity-agent-core)!: provider-classified errors, provider-owned timeouts, and oversized-input recovery - #121
Open
shadaj wants to merge 1 commit into
Conversation
Deploying infinity with
|
| Latest commit: |
132dfef
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://29e41bf1.infinity-dc7.pages.dev |
| Branch Preview URL: | https://sandbox-23ca95d7-b2d9-4ea4-a.infinity-dc7.pages.dev |
…ent-core)!: provider-classified errors, provider-owned timeouts, and oversized-input recovery
## `infinity-provider-protocol`: structured, classified errors
* New `ErrorClass` enum (`Transient` / `Throttled` / `ContextOverflow` /
`Fatal`) — providers declare how callers should react; the agent runtime
never parses error message strings.
* `CompletionError::ProviderError` now carries `{ message, class }`
(breaking); added `CompletionError::provider(class, msg)` and
`CompletionError::class()` (RequestError/JsonError → Fatal,
ResponseError → Transient).
* Remote wire protocol forwards errors as `WireError { message, class }` so
classification survives the Unix-socket boundary (`Fatal` default for
older providers). Covered by `error_class_survives_the_wire`.
## `infinity-provider-bedrock`: timeouts + classification live here now
* 60s request-initiation timeout around `ConverseStream` (Bedrock
occasionally black-holes requests), reported as `Transient` so core
retries. Deliberately **no** mid-stream/inactivity timeout anywhere: once
a stream is live it is never artificially cut off by us. Tested with a
never-responding HTTP connector under paused time.
* `classify_bedrock_error` maps service codes + message heuristics (moved
out of core) to `ErrorClass`: throttling → `Throttled`, input-too-long /
context-limit → `ContextOverflow`, backend hiccups → `Transient`, rest
`Fatal`. Mid-stream recv errors are classified the same way.
* New non-default `live-tests` feature (`tests/live.rs`) hitting real
Bedrock with local credentials — overflow classification, unknown-model
fatality, happy path — instead of tautological string-table unit tests.
All pass (and caught a bad token estimate during development).
* `infinity-provider-rig` classifies rig errors from common
OpenAI-compatible phrasings.
## `infinity-agent-core`: no timeouts, class-driven retries, oversized-input fix
`run_completion` drops the 60s initiation timeout and the 120s mid-stream
stall timer; retries are driven purely by `ErrorClass` (Throttled → 30s
backoff, Transient → short retry, Fatal → give up), capped at
`MAX_COMPLETION_RETRIES`.
**Three-phase pending items** (fixes: an oversized user input / tool result
used to be persisted and permanently wedge the thread):
* `HistoryManager` inputs move through **not-known-safe**
(`unvalidated_items`) → **known-safe** (`pending_items`, promoted the
moment the model streams any output — proof the context didn't overflow;
model-produced content goes here directly) → **synced**. `sync()` only
persists known-safe items; a sequentiality assert makes committing model
output over unvalidated inputs impossible.
* Interrupts: cancelling before any model output keeps the inputs in memory
(re-sent with the interrupting message next round) but persists nothing;
after output, input + partial output persist as before.
* `safe_spawn_point()` stays before unvalidated inputs (children inherit
history from the store).
**Overflow recovery** (`recover_from_overflow`):
* Exactly one unvalidated item and it's replaceable (tool result /
subscription event): the culprit is unambiguous → replace its body with
`TOOL_RESULT_TOO_LARGE_PLACEHOLDER`, retry. (The replace helper skips
bodies already equal to the placeholder, so a second overflow falls
through to drop mode instead of retrying forever.)
* Anything else (user text present, multiple items, or the retry overflowed
again): settle replaceables with the shared `TOOL_CALL_INTERRUPTED_TEXT`
(a "too large" note could mislead the model into re-running the tool),
drop the user inputs (`drop_unvalidated_user_inputs`, dedup IDs
forgotten), and **stop** — dropping the user's words is never followed by
a silent retry. Tool calls always stay answered in memory; nothing
unvalidated ever persists. TODO left for blocking on a lower-threshold
compaction when committed history itself is too close to the limit.
## Tests (each behavior landed red-first)
* event_processor: oversized user input dropped & unpersisted; oversized
tool result / subscription event → placeholder retry; tool result + user
input → interrupted-settle + drop + **no retry**; second overflow gives up
keeping the call answered; interrupt before/after model output;
throttled/transient retry vs fatal no-retry; `sync()` skips unvalidated;
flush-with-unvalidated panics; spawn-point exclusion.
* system: interrupt-before-output persists nothing until validated; overflow
after interrupt drops both inputs and stops (thread recovers on next
message); oversized tool result never wedges the thread end-to-end;
shutdown persists validated but not unvalidated tool results; rebooted
session settles a persisted unanswered tool call via the synthetic
"interrupted" result.
`./check.bash`: fmt, clippy, and full workspace tests pass (99 core tests +
all other crates); the web-e2e step stalled on a sandboxed `npm ci`
(network), unrelated to these changes.
Co-authored-by: Infinity 🤖 <infinity@hydro.run>
PR: #121
shadaj
force-pushed
the
sandbox-23ca95d7-b2d9-4ea4-a456-3eeaeb5b1749
branch
from
September 4, 2026 20:59
8f30f19 to
132dfef
Compare
shadaj
marked this pull request as ready for review
September 4, 2026 23:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
infinity-provider-protocol: structured, classified errorsErrorClassenum (Transient/Throttled/ContextOverflow/Fatal) — providers declare how callers should react; the agent runtimenever parses error message strings.
CompletionError::ProviderErrornow carries{ message, class }(breaking); added
CompletionError::provider(class, msg)andCompletionError::class()(RequestError/JsonError → Fatal,ResponseError → Transient).
WireError { message, class }soclassification survives the Unix-socket boundary (
Fataldefault forolder providers). Covered by
error_class_survives_the_wire.infinity-provider-bedrock: timeouts + classification live here nowConverseStream(Bedrockoccasionally black-holes requests), reported as
Transientso coreretries. Deliberately no mid-stream/inactivity timeout anywhere: once
a stream is live it is never artificially cut off by us. Tested with a
never-responding HTTP connector under paused time.
classify_bedrock_errormaps service codes + message heuristics (movedout of core) to
ErrorClass: throttling →Throttled, input-too-long /context-limit →
ContextOverflow, backend hiccups →Transient, restFatal. Mid-stream recv errors are classified the same way.live-testsfeature (tests/live.rs) hitting realBedrock with local credentials — overflow classification, unknown-model
fatality, happy path — instead of tautological string-table unit tests.
All pass (and caught a bad token estimate during development).
infinity-provider-rigclassifies rig errors from commonOpenAI-compatible phrasings.
infinity-agent-core: no timeouts, class-driven retries, oversized-input fixrun_completiondrops the 60s initiation timeout and the 120s mid-streamstall timer; retries are driven purely by
ErrorClass(Throttled → 30sbackoff, Transient → short retry, Fatal → give up), capped at
MAX_COMPLETION_RETRIES.Three-phase pending items (fixes: an oversized user input / tool result
used to be persisted and permanently wedge the thread):
HistoryManagerinputs move through not-known-safe(
unvalidated_items) → known-safe (pending_items, promoted themoment the model streams any output — proof the context didn't overflow;
model-produced content goes here directly) → synced.
sync()onlypersists known-safe items; a sequentiality assert makes committing model
output over unvalidated inputs impossible.
(re-sent with the interrupting message next round) but persists nothing;
after output, input + partial output persist as before.
safe_spawn_point()stays before unvalidated inputs (children inherithistory from the store).
Overflow recovery (
recover_from_overflow):subscription event): the culprit is unambiguous → replace its body with
TOOL_RESULT_TOO_LARGE_PLACEHOLDER, retry. (The replace helper skipsbodies already equal to the placeholder, so a second overflow falls
through to drop mode instead of retrying forever.)
again): settle replaceables with the shared
TOOL_CALL_INTERRUPTED_TEXT(a "too large" note could mislead the model into re-running the tool),
drop the user inputs (
drop_unvalidated_user_inputs, dedup IDsforgotten), and stop — dropping the user's words is never followed by
a silent retry. Tool calls always stay answered in memory; nothing
unvalidated ever persists. TODO left for blocking on a lower-threshold
compaction when committed history itself is too close to the limit.
Tests (each behavior landed red-first)
tool result / subscription event → placeholder retry; tool result + user
input → interrupted-settle + drop + no retry; second overflow gives up
keeping the call answered; interrupt before/after model output;
throttled/transient retry vs fatal no-retry;
sync()skips unvalidated;flush-with-unvalidated panics; spawn-point exclusion.
after interrupt drops both inputs and stops (thread recovers on next
message); oversized tool result never wedges the thread end-to-end;
shutdown persists validated but not unvalidated tool results; rebooted
session settles a persisted unanswered tool call via the synthetic
"interrupted" result.
./check.bash: fmt, clippy, and full workspace tests pass (99 core tests +all other crates); the web-e2e step stalled on a sandboxed
npm ci(network), unrelated to these changes.