Skip to content

fix(checkpoint): delete the writer's blind failure count, classify the failure, and give the final threshold a bounded recovery - #1945

Merged
wqymi merged 6 commits into
mainfrom
fix/checkpoint-writer-timeout-followup
Jul 30, 2026
Merged

fix(checkpoint): delete the writer's blind failure count, classify the failure, and give the final threshold a bounded recovery#1945
wqymi merged 6 commits into
mainfrom
fix/checkpoint-writer-timeout-followup

Conversation

@wqymi

@wqymi wqymi commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #1938 (merged). Two halves, worth reading as separate commits: Part 1 deletes the checkpoint writer's blind failure accounting, and Part 2 replaces it by making a spawned agent's failure classifiable, then using that classification in the writer's one reporting site.

Part 1's previous revision argued for extending prune's wait and adding a cap; that argument is void and has been removed from this description. It also recorded an owed follow-on — "classify the failure instead of counting it" — which was blocked on AgentOutcome's failure arm carrying only a string. Part 2 widens that contract and discharges it, so this PR no longer reads as "deleted the accounting, replaced it with nothing".

#1938's core fix is untouched throughout: waitForWriter still returns a distinct "timeout" when its 300s bound expires with the writer still in flight.

Part 1 — delete the blind failure count

What is deleted is the failure accounting layered on top of #1938 — including the pre-existing counter that predates it.

Why: the premise the accounting was built on does not hold

The writer fires at every threshold crossing (4 @ 20% for mid-tier windows, 9 @ 10% for extended-context — see the schedule comment above prune.ts:35). It is designed fail-acceptable, and the code already implements that:

  • ensureCheckpointTemplate (checkpoint.ts:117-121) writes the template only when the file does not exist, so a failed writer does not clobber the previous checkpoint. It is a conditional write, not a scaffold-then-fill.
  • last_checkpoint_message_id advances only on a successful insert (checkpoint.ts:918-934; the sole external caller of resetThresholds is prompt.ts:428, gated on if (inserted)).

So one failed write leaves the file and the watermark mutually consistent, both pointing at the last good checkpoint. A rebuild uses that automatically. The cost of a failure is a staler checkpoint, never a missing one — and the next threshold crossing is a natural retry, with fresher context than an in-place re-fire of the same threshold would have had.

On top of that redundancy, the previous revision added machinery that did not trust it, and the failure model behind the counter was wrong: a writer failing repeatedly means the provider is broken, in which case the foreground turns are failing too and the user already knows. The writer's LLM calls go through the same retry ladder as the foreground (retry.tsisRetryableTransientError, RETRY_INITIAL_DELAY = 2000, RETRY_BACKOFF_FACTOR = 2, consumed by session/llm.ts:34 and session/processor.ts:17; the writer runs as a spawned subagent through the same prompt loop). Any failure that reaches prune is therefore already post-retry. Counting it again, three times, then falling silent, adds nothing.

What was deleted

what where it was
MAX_WRITER_WAIT_EXTENSIONS = 12 and the whole wait-extension loop prune.ts:32, loop at prune.ts:341-350
MAX_WRITER_FAILURES = 3 prune.ts:27
writerFailures map + its comment prune.ts:178-182
maxFailures config read prune.ts:280
the in-place-retry path: crossed.delete() / maxCrossed.delete() on failure, plus the failure/give-up logs prune.ts:351-373
cfg.checkpoint.max_writer_failures schema entry config/config.ts:278
its docs row and generated SDK declarations mimocode-docs/reference/config.md:126, sdk/js/src/v2/gen/types.gen.ts:2209, sdk/openapi.json:13896

grep -c "writerFailures\|MAX_WRITER_FAILURES\|MAX_WRITER_WAIT_EXTENSIONS\|max_writer_failures" over the pushed blobs of all five source files returns 0. (The only surviving mentions repo-wide are three comment lines in the rewritten tests that state, deliberately, what those tests used to assert.)

The watcher fiber survives — as observation only, and that is load-bearing

The obvious reading of "delete the accounting" is to delete the forked watcher entirely, since nothing else consumed it. That would have been wrong, and the check is worth recording: prune.ts:340 and :349 were waitForWriter's only production callers (grep -rn waitForWriter src/). Deleting the fork would have made #1938's bound-expiry log — the one thing this PR was asked to keep — unreachable in production, surviving only as a line that tests exercise.

So the fork remains, reduced to a single call whose result is discarded:

yield* checkpoint.waitForWriter(input.sessionID).pipe(Effect.forkDetach)

No loop, no counter, no retry. The wait exists purely so waitForWriter's checkpoint writer wait bound expired — writer still in flight (checkpoint.ts:1009) still fires. That log is cheap, at most once per writer, and it is the only evidence a slow writer leaves; #1938's own diagnosis came from a log line exactly like it. Terminal outcomes are already logged by the settle watcher (checkpoint.ts:920-934), so nothing is lost by discarding the return value.

The config surface: removed, not deprecated

cfg.checkpoint?.max_writer_failures was a public config surface, so removal vs. accept-and-ignore was a real decision. I expected the back-compat argument to force deprecation, and measured instead of assuming:

Config.Info.safeParse({ checkpoint: { bogus_key_xyz: 3 } })  -> success: true
Config.Info.safeParse({ bogus_top_xyz: 3 })                  -> success: false
   [{"code":"unrecognized_keys","keys":["bogus_top_xyz"],...}]

.strict() at config/config.ts:485 is applied to the top-level object only and does not recurse. An unknown key nested under checkpoint is silently accepted. So removing the field cannot break an existing config — a user who still has max_writer_failures set gets no error, exactly as before.

That removes the only real argument for keeping it. Deprecating would be strictly worse: a knob still published in the JSON schema, the docs table, and the SDK types, which provably does nothing — a user setting max_writer_failures: 10 would reasonably expect behaviour and get silence. Removed, with the docs row and both generated declarations removed alongside so no artifact advertises a knob the server dropped.

Tests — rewritten, not deleted

Six cases in prune.test.ts and T10 in checkpoint-child-session.test.ts pinned the deleted machinery. All seven are rewritten to pin the opposite requirement. Each rewrite states in-file what it used to assert and why that is no longer the requirement.

What they used to assert: that a failure below the cap re-armed its own threshold (enqueue 1→2→3); that a success cleared the counter; that the watcher kept re-entering the bounded wait so a late outcome still reached the counter; and the exact 13-call boundary of that re-entry loop. All of it encoded in-place retry plus a give-up gate as requirements. Neither is a requirement now.

What is pinned insteadprune.test.ts (3 cases, on the existing makeRetryHarness):

  • a failed writer does not re-arm its own threshold — after a failure, three more fires at the same token level stay at 1 enqueue.
  • a failed writer at the max threshold leaves maxThresholdCrossed set — the old watcher paired crossed.delete with maxCrossed.delete, which withdrew the already-raised discard+rebuild signal prompt.ts consumes. Overflow readiness must not depend on whether the writer happened to succeed.
  • the next threshold crossing still fires after a failure — the retry path the design now relies on. The exact count is what pins it: the old code also reached a 2nd enqueue, but reached a 3rd, because the failure had additionally re-armed the first threshold.

And the self-healing property asserted directlyT10 now drives the real service, seeds a known previous checkpoint (file content + watermark), fails the writer, and asserts the premise this whole change rests on:

expect(yield* Effect.promise(() => Bun.file(cpFile).text())).toBe(previousContent)
expect(yield* svc.lastBoundary(info.id)).toBe(previousBoundary)

The file still holds the previous content and lastBoundary still returns the previous boundary — mutually consistent at the last good checkpoint. These two assertions pass under the old code as well, by design: they are properties of checkpoint.ts, not of the deleted accounting. They are here to make the premise executable rather than asserted in prose.

Revert probe

Restoring only src/session/prune.ts to the pre-change blob (git show HEAD:…), keeping the new tests:

345 |         expect(harness.state.enqueueCount).toBe(1)
error: expect(received).toBe(expected)
Expected: 1
Received: 2
(fail) a failed writer does not re-arm its own threshold — no in-place retry [659.62ms]

375 |         expect(yield* svc.maxThresholdCrossed(info.id)).toBe(true)
error: expect(received).toBe(expected)
Expected: true
Received: false
(fail) a failed writer at the max threshold leaves maxThresholdCrossed set [456.32ms]

408 |         expect(harness.state.enqueueCount).toBe(2)
error: expect(received).toBe(expected)
Expected: 2
Received: 3
(fail) the next threshold crossing still fires after a failure [512.93ms]

 13 pass
 3 fail
 76 expect() calls

The probe also caught a flaw in my own T10. On that first run T10 passed under the reverted code — it should have failed. Cause: I had dropped the original T10's 50 ms pre-settle tick, so prune's fork had not yet reached writers.get(...) inside waitForWriter when the outcome settled, and it saw "no-writer" instead of the failure — the race documented in prune.ts. The test was passing by accident, not because the behaviour held. Tick restored (with a comment explaining why it is required), and T10 then discriminates:

526 |         expect(spawnLog.count).toBe(1)
error: expect(received).toBe(expected)
Expected: 1
Received: 2
(fail) T10: a failed writer is self-healing … and no in-place retry fires [378.74ms]

 5 pass
 1 fail
 30 expect() calls

prune.ts restored afterwards; grep -c MAX_WRITER_WAIT_EXTENSIONS src/session/prune.ts → 0.

Part 2 — the replacement: classify the failure instead of counting it

Part 1 deletes the blind count. On its own that leaves the writer's failure handling with nothing in its place, which is precisely what this PR's earlier revision recorded as owed: the better replacement is to classify the failure. That was not implementable then, and the obstacle was a type.

// actor/spawn.ts, before
| { status: "failure"; error: string }

AgentOutcome's failure arm carried a stringified message and nothing else — no APICallError, no status code, no retryable flag. A consumer could not tell a transient provider blip from a deterministic, will-never-work failure without matching on prose. Blind counting was the only thing the type permitted. So the count could be deleted honestly, but not replaced, until the contract was widened. That is what Part 2 does.

The shape

export type FailureKind = "transient" | "overflow" | "auth" | "aborted" | "other"

export interface FailureInfo {
  readonly kind: FailureKind
  readonly retryable: boolean
  /** The persisted NamedError name, e.g. "APIError" / "ContextOverflowError". */
  readonly name: string
}

| { status: "failure"; error: string; failure?: FailureInfo }

Additive and backward-compatible: error: string is untouched and still carries the human text, and the new field is optional, so every existing consumer and every existing hand-built test literal still typechecks unchanged.

failure is present only when the failure came from a settled assistant error. A work fiber that failed some other way — a defect during teardown, or a hand-built outcome such as waitForWriter's { status: "failure", error: "timeout" } (checkpoint.ts:986) — leaves it absent rather than asserting a class it does not have.

retryable means "this error belonged to the retryable class", not "please retry". The child's LLM calls already run through retry.ts's ladder, so any failure reaching a consumer is already post-retry — the same fact that licensed Part 1's deletion.

The alternative I rejected

Two independent flat fields on the arm: retryable?: boolean and kind?: FailureKind. Rejected because they admit incoherent states (kind: "overflow" with retryable absent versus present-and-false) and because they force every read site to write its own absence guard — precisely the hazard AGENTS.md § "Reading a nullable column" describes, since a guard written === undefined typechecks, reads correctly in review, and silently does nothing. One optional object is present-or-absent as a unit: a single truthiness check, after which the interior fields are non-optional and cannot disagree.

Also rejected: carrying provider/error.ts's ParsedAPICallError directly. It is bound to the AI-SDK APICallError shape at the provider boundary; by the time a failure reaches spawn the error has already been normalized into the persisted named-error union, so re-deriving it there would mean re-parsing — the thing this change exists to avoid.

Populated where the typed error still exists

actor/spawn.ts:354, inside runAgentLoop. That line previously threw the typed error away:

// before — the class of the error is reduced to its name inside a string
return yield* Effect.fail(new Error(`Actor assistant failed: ${info.error.name}`))

info.error is the assistant message's already-normalized named error, so no new taxonomy was added — the existing classifiers are reused at exactly this point:

  • SessionRetry.retryable (session/retry.ts) is the retryability oracle. Its input type is this data shape (Err === the return of NamedError["toObject"]), and it already folds in isRetryableTransientError plus every 429 / 5xx / quota / upstream_error special case. Reusing it means the classification cannot drift from the ladder that actually decides retries.
  • kind is read off the named-error identity that MessageV2.fromError (message-v2.ts:1159-1186) derived from ProviderError.parseAPICallError / isOverflow / isOpenAiErrorRetryable: ContextOverflowErroroverflow, ProviderAuthErrorauth, MessageAbortedErroraborted.
  • One refinement: an APIError with statusCode 401/403 also maps to auth, reading the status code parseAPICallError already extracted. ProviderAuthError only covers a missing key (LoadAPIKeyError); a rejected one arrives as an APIError, and to a consumer they are the same thing.

Carrying it the last few frames needs one small class, because forkWork's onFailure — the single place AgentOutcome is constructed (spawn.ts:737-744) — only has a Cause:

const squashed = Cause.squash(cause)   // established idiom: session/prompt.ts, tool/shell-wrap.ts
const failure = squashed instanceof AssistantSettledError ? squashed.failure : undefined

AssistantSettledError is a plain Error subclass, chosen deliberately: Cause.pretty renders it byte-identically to new Error(message) (measured before committing to it), so error's existing text is unchanged rather than gaining a class prefix.

Consumer inventory

Every consumer is an awaiter of the AgentOutcome Deferred. grep -rn "Deferred.await" src/ | grep -i outcome finds six production sites across four files:

# consumer site what it reads verdict
1 workflow runtime, shared-spawn step workflow/runtime.ts:845-852 status !== "success", then (outcome as { error?: string }).error no break — additive field ignored
2 workflow runtime, isolated/worktree step workflow/runtime.ts:971-980 same no break
3 workflow runtime, disposition + deliverable workflow/runtime.ts:999-1023 status === "success" only no break — success arm untouched
4 ordinary subagents (actor run) tool/actor.ts:788-816 status === "failure".error no break
5 peer / fork-query (session ask) tool/session.ts:149-151 status === "failure" ? .error : .status no break
6 checkpoint writer, settle watcher checkpoint.ts:914-960 status === "success" gates the watermark changed on purpose (below); watermark logic untouched
6b checkpoint writer, waitForWriter translation checkpoint.ts:984-990 maps to success/failure; also constructs { status: "failure", error: "timeout" } no break — the literal still typechecks and correctly carries no classification

Two entries from the original consumer list turn out not to be consumers, and the check is worth recording:

  • dream / distill never see an AgentOutcome. They are agent types, invoked through sp.prompt({ agent: "dream" | "distill", … }) at session/prompt.ts:3115 and :3126 — they do not go through Actor.spawn at all. (spawn.ts:217's comment lists them among spawn's callers, which is what makes this worth stating explicitly; the actual code path is a direct prompt.)
  • actor/group.ts:141-146 matches a grep for outcome but reads actor-registry status strings, not AgentOutcome.

Test literals that hand-build the failure arm — auto-overflow-writer-first.test.ts:164, checkpoint-child-session.test.ts:367 and :477, checkpoint-watermark-transactional.test.ts:131 — all still compile unchanged, which is the optional field doing its job.

AgentOutcome is an internal TypeScript type, never surfaced through the zod/OpenAPI layer: grep -n "AgentOutcome\|FailureInfo\|FailureKind" over sdk/js/src/v2/gen/types.gen.ts and sdk/openapi.json returns nothing, so Part 2 needs no artifact regeneration. Part 1's SDK edits, which remove max_writer_failures, stand as described above.

The one behaviour change: the writer's log stops lying by omission

Part 1 leaves the settle watcher emitting one undifferentiated warn for every non-success outcome (checkpoint.ts:930), so "will fix itself at the next threshold" and "will never work" read identically. That is the hole the deletion opened. With the classification available:

  • a transient failure keeps the existing line — it is the retry ladder's business, and the next crossing re-covers the delta with fresher context;
  • a deterministic one (overflow / auth / bad request) says so and names the cause, because it will recur identically at every future threshold.

Stateless on purpose. No per-session memory of previous failures, under any name. Such memory is a trend detector, and the trend belongs to the user-facing warning described under "Residual" below — accumulating it here would be re-adding, with a different name, exactly what Part 1 removed. Read as "report a deterministic failure as the distinct thing it is", not as "suppress the repeat"; suppression would need that state.

Nothing else changes. No retry policy, no give-up logic, no user-facing warning, and no consumer other than #6 reads the new field.

Tests for Part 2

Four cases in test/actor/spawn.test.ts, all driven through the real construction path: the classification is produced by runAgentLoop, carried across the failure channel, squashed back out and assembled by forkWork's onFailure. No test builds an AgentOutcome.

  • transient (APIError 429, isRetryable: true) → kind: "transient", retryable: true
  • deterministic (ContextOverflowError) → kind: "overflow", retryable: false
  • deterministic (APIError 401) → kind: "auth", retryable: false
  • a clean turn → failure == null (truthiness / == null, never === undefined), so a success can never carry an invented classification

Each also asserts error still contains Actor assistant failed: <Name>, pinning the backward-compatible string.

The upstream provider round-trip is stood in for at the SessionPrompt.prompt boundary, and that is a deliberate, disclosed choice rather than convenience: SessionRetry.policy is uncapped — its schedule stops only when retryable() returns falsy — so a genuinely retryable provider error never settles on its own. A real-server transient case would hang, not fail. This is also why consumer #6 branches on !failure.retryable rather than trying to act on transient: in the current architecture kind carries the discriminating power and retryable is largely documentary.

Revert probe

Restoring only the construction site at spawn.ts:354 to its previous new Error(...) — keeping the new tests, so the field exists but is never populated:

1440 |         expect(outcome.failure).toBeTruthy()
       ^
error: expect(received).toBeTruthy()

Received: undefined

      at toBeTruthy (unknown:1:1)
      at /…/packages/opencode/test/actor/spawn.test.ts:1440:33

1478 |         expect(outcome.failure).toBeTruthy()
       ^
error: expect(received).toBeTruthy()

Received: undefined

      at /…/packages/opencode/test/actor/spawn.test.ts:1478:33

1511 |         expect(outcome.failure?.retryable).toBe(false)
       ^
error: expect(received).toBe(expected)

Expected: false
Received: undefined

      at /…/packages/opencode/test/actor/spawn.test.ts:1511:44

 1 pass
 26 filtered out
 3 fail

The single pass is the negative case — it asserts absence, so it must keep passing under the revert, and does. Construction site restored afterwards; bun typecheck exit 0 and the classification tests green again.

Verification

bun typecheckexit 0 for packages/opencode and for packages/sdk/js (the two packages touched).

Scoped run, identical command both sides — bun test test/session/prune.test.ts test/session/prune-skip-system.test.ts test/session/checkpoint-child-session.test.ts test/session/checkpoint-writer-wait-timeout.test.ts --timeout 120000:

result
pristine baseline at 3068ed9f9, before any edit 26 pass / 0 fail, 106 expects, 10.93s
after this change 23 pass / 0 fail, 94 expects, 10.36s

Net −3 tests: six prune.test.ts cases replaced by three; T10 replaced 1:1.
Part 2, identical command both sides — bun test test/actor/spawn.test.ts test/session/checkpoint-child-session.test.ts test/session/checkpoint-writer-wait-timeout.test.ts test/session/prune.test.ts --timeout 30000, run in the same fresh worktree at the same base so the environment is a genuine control:

result
pristine baseline at 4e011eaf7 (Part 1's head), before any Part 2 edit 45 pass / 0 fail, 163 expects, 30.99s
after Part 2 49 pass / 0 fail, 178 expects, 31.76s

Net +4 tests, +15 expects, and the 45 pre-existing cases — including Part 1's rewritten prune.test.ts and T10, which pin "a failed writer does not re-arm its own threshold" — all still pass. No existing assertion was weakened or removed.

bun typecheckexit 0 for packages/opencode and for packages/sdk/js. The pre-push hook additionally ran the repo-wide typecheck: 12 of 12 tasks successful.

The full suite was deliberately not run — other work is live on this box and concurrent full runs starve each other badly enough to make pass counts meaningless.

Residual, stated plainly

A session whose last threshold fails gets no further fire. — CLOSED BY PART 3. This was written down here as an accepted consequence of Part 1 ("the retry is the next crossing, and a last threshold has no next"). A review pushed back, and on measurement the consequence was worse than recorded: the final threshold's would-be successor is the discard+rebuild's resetThresholds, which never runs when lastBoundary is unset — i.e. exactly when no writer has ever succeeded — so the session stopped checkpointing permanently rather than merely going stale. Part 3 gives the final threshold a class-gated, window-bounded recovery gate. The paragraph is kept rather than deleted so the reasoning that accepted the hole, and the measurement that overturned it, both stay on the record.

A user-visibility gap this PR deliberately does not close. When checkpointing is persistently broken, the overflow fallback is compaction — and that fallback is lossy. Correction, measured after this section was written (see #1973): compaction.create in isolation is only 3 local writes (p50 0.240 ms), which is where the earlier "unsummarized amputation" claim came from — but runLoop pairs it with compaction.process (prompt.ts:3290-3298), a real LLM summarization, so the path does summarize. The in-tree comment that asserted otherwise was false and is corrected in #1973. The user-visibility gap therefore remains, but its stake is "your context is being summarized away repeatedly and silently", not "dropped unsummarized". The user should be warned about that, but the warning belongs in its own change, and its trigger must be a trend ("the last N thresholds all failed") or the moment a rebuild finds its checkpoint too stale — never a failure count, which is exactly the mechanism deleted here for being the wrong shape. Part 2 supplies the input such a warning needs (a deterministic failure is now identifiable as such) and deliberately stops there: it adds no per-session state, so nothing here is a trend detector wearing a new name.

Relationship to #1938

Unchanged and not reverted: waitForWriter (now checkpoint.ts:1087), its 5-min bound, its distinct "timeout" outcome, the WriterOutcome type (checkpoint.ts:534), and the bound-expiry log (checkpoint.ts:1070). Part 3 moved the implementation into waitForWriterSettlement (checkpoint.ts:1040) and made waitForWriter a projection of it, so the contract is now derived from the same code path rather than sitting beside it — it cannot drift. Line numbers here are re-read from the pushed blob at 3ee914261. #1938's checkpoint-writer-wait-timeout.test.ts assertions are all retained verbatim; only two of its comments changed, where they justified the timeout/failure distinction by pointing at the counter that no longer exists. The distinction itself stands on its own: a slow-but-working writer must not read as a broken one.

checkpoint-align.ts's doc comment, which described when a rejection increments writerFailures, now describes what actually happens — the previous checkpoint and its watermark stay in place and the next crossing re-covers the same delta.

Deliberately not included

  • The user-facing persistent-checkpoint-failure warning. Specified above: trend or stale-at-rebuild, never a count. It needs state this PR deliberately does not introduce.
  • Acting on the classification anywhere but consumer SoX v14.4.2 waveaudio bug: default device recording fails with Bluetooth headset #6. The other five consumers are untouched by design; giving them retry or give-up policy is a behaviour change per consumer and belongs with whoever owns that consumer.
  • Wiring kind into the WriterCachePerf bus event. That event is the natural substrate for cross-layer writer-health accounting, which is the trend detector's business, not this PR's.
  • computeBoundary coverage pinning (checkpoint.ts:254-293) — it changes what a checkpoint covers and is a separate, riskier decision.

Part 3 — recovery: the final threshold gets a gate

Part 1's argument was "the next threshold crossing is the retry." A review correctly pointed out that this presumes a next threshold, and the last one has none. Part 3 closes that, using Part 2's classification as the discriminator.

What "no next threshold" actually costs — measured, because the naive statement understates it

The final threshold is not simply successor-less. It has a would-be successor, and that successor is conditional in exactly the wrong way:

  1. When the final threshold fires, fireCheckpoints also sets maxCrossed (prune.ts:428), and the same runLoop iteration consumes it: overflowCheck(...) || prune.maxThresholdCrossed(sessionID)rebuildFromCheckpoint (prompt.ts:3308-3352).
  2. A successful rebuild calls resetThresholds (prompt.ts:428), which clears crossedre-arming the entire ladder from 20%. So in the healthy case the final threshold does get a retry: the ladder restarts.
  3. But rebuildFromCheckpoint returns false when lastBoundary is unset (prompt.ts:413), and the watermark is unset precisely when no writer has ever succeeded.

⇒ In the one case that needs recovery most — the first checkpoint failed — there is no boundary, no reset, crossed still holds every threshold, and the session never fires another writer for the rest of its life. Every subsequent iteration falls to compaction instead. That is not "a staler checkpoint"; it is the subsystem switched off silently.

(maxCrossed is also never cleared in that path, so the compaction fallback repeats every other iteration. That is a pre-existing, separate defect — it predates this branch and is not touched here.)

The principle

A failed checkpoint write earns a retry only when the failure's class says a retry could succeed, and only where the token axis still has room for one. Both are readings of the present state, so neither needs any memory of previous failures.

The mechanism below is meant to be readable as nothing more than that sentence applied to the one threshold that lacks a successor.

The mechanism

prune.ts gains one piece of per-session state — finalRetryAt: Map<SessionID, number> (prune.ts:190), the token count at or above which the final threshold may fire a second time.

Admission (prune.ts:303-318) — the only threshold that can re-fire is the final one, and only past its gate:

if (already.has(t)) {
  const gate = finalRetryAt.get(input.sessionID)
  if (t !== maxThreshold || gate == null || currentTokens < gate) continue
  finalRetryAt.delete(input.sessionID)   // one gate is one attempt
}

Arming (prune.ts:384-418), in the forked settle watcher that Part 1 already kept for the bound-expiry log:

condition line why
if (!isFinal) return prune.ts:386 every other threshold's successor is its retry
if (settled.outcome !== "failure") return prune.ts:389 "timeout" means still in flight — the writer may yet succeed and advance the watermark
if (!settled.failure?.retryable) → log, return prune.ts:392 deterministic (overflow / auth / bad request) recurs identically; unclassified carries no evidence a retry helps, so it is treated the same
gate = Math.min(firedAt + step, maxAllowed) prune.ts:401 one ladder step above where it fired, capped at the last position a write is meaningful
if (gate <= firedAt) return prune.ts:402 no room left in the window ⇒ no attempt

step is the ladder's own spacing (prune.ts:297), and maxAllowed now comes from a shared maxAllowedFor() (prune.ts:79) that resolveThresholds also uses, so the clamp and the ceiling cannot drift apart. resetThresholds drops the gate (prune.ts:535): a rebuild re-arms the whole ladder, which supersedes it.

Why this is not MAX_WRITER_FAILURES under a new name

This was the explicit bar, so it is answered directly rather than asserted.

the deleted counter this gate
what it measures attempts — how many times an already-abandoned attempt was abandoned a position on the token axis
across attempts accumulates overwritten, and dropped when consumed
where its value comes from a stipulated constant (3) the threshold ladder + the context window
conversation stops growing still permits N retries stops retrying entirely
conversation grows fast still permits only N keeps retrying — because there is genuinely new uncovered delta each time
retry rate unbounded within N can never exceed the ladder's own firing rate

The retry budget is therefore the unused window, a physical quantity the system already measures: with the mid-tier ladder (4 @ 20%) the final threshold sits at 80% and one step overshoots maxAllowed, so it affords at most one retry; with a 1M-window ladder (18 @ 5%) it affords two. Nobody chose those numbers — geometry did.

Measurement that decided the alternatives

Rejected: "the loop is self-limiting because a broken writer means a broken provider, so the foreground dies too." This needed checking specifically for the writer, since a separately-configured writer model could be rate-limited while the foreground works. It does not have one. prune.ts:324 passes the foreground turn's own model into tryStartCheckpointWriter; checkpoint.ts forwards it to actor.spawn({ model }); spawnSubagentforkWorkrunAgentLoopsessionPrompt.prompt({ model }), and the resolution is inputModel ?? agentModel ?? lastModel(...) (prompt.ts:1836) — inputModel wins. The checkpoint-writer agent definition (agent/agent.ts:347) declares no model/modelRef at all, so there is nothing to lose the race anyway. ⇒ the shared-fate argument holds, which is why persistent failure still needs no counter — but it says nothing about a single transient failure at the last threshold, which is the only case Part 3 acts on.

Rejected: bound the retry with a count (even 1). Arbitrary, and it is the deleted mechanism with a smaller constant. Ruled out by the bar, not by measurement.

Rejected: append maxAllowed to the ladder unconditionally, so the final threshold always sits at the last meaningful position and "no successor" becomes a fact about the window rather than a defect. Elegant and needs zero new state — but it adds a fifth writer per ladder cycle for every session, failure or not. Paying that always, to fix a case that arises only on failure, is the wrong trade.

Rejected: clear the threshold whenever the failure was transient. This is the hot loop the brief warned about: fireCheckpoints runs at the start of every runLoop iteration, so an unchanged token count would spawn a writer — a dozen sequential round-trips — per iteration. The progress gate is precisely what removes it, and a test pins that (below).

Plumbing: how the class reaches prune without touching #1938's contract

Part 2 put FailureInfo on AgentOutcome, but prune only sees WriterOutcome — three flat strings. Rather than widen that union (which would break every === "failure" consumer and #1938's own test), checkpoint.ts gains a companion:

  • WriterSettlement = { outcome: WriterOutcome | "no-writer"; failure?: FailureInfo } (checkpoint.ts:546)
  • waitForWriterSettlement (checkpoint.ts:1040) holds the implementation — the same 5-min bound, the same bound-expiry log (checkpoint.ts:1070), plus the classification.
  • waitForWriter (checkpoint.ts:1087) is now a projection of it: (yield* waitForWriterSettlement(id)).outcome.

So #1938's contract is not merely "still there" — it is derived from the same code path, and the two cannot disagree about what a settled writer did. failure is absent for a cancelled writer and for a failure that was never classifiable, i.e. absent means unknown, never retryable.

Tests for Part 3 — six new cases, every one revert-probed

test/session/prune.test.ts, new nested describe final-threshold recovery gate (harness stub rewired so the settlement path — the one production calls — is the driver; a stub implementing only waitForWriter would have left every failure case in this file vacuous, since the seeded queue would never drain):

  1. a deterministic failure at the final threshold arms no gate
  2. a transient failure re-fires it, but only once growth reaches a full ladder step (fires at 31 000, gate at 41 000; asserts no re-fire at 31 000 / 35 000 / 40 999, then a re-fire at 41 000)
  3. a transient failure arms no gate when the window has no room left
  4. a transient failure at a non-final threshold arms no gate — including that it cannot leave a gate for the final threshold to consume
  5. a bound-expiry "timeout" arms no gate
  6. resetThresholds drops a pending gate

test/session/checkpoint-writer-wait-timeout.test.ts, new describe SessionCheckpoint.waitForWriterSettlement — five cases against the real service (prune's tests stub the checkpoint service, so without these the AgentOutcome → WriterSettlement hop would be asserted nowhere and every case above would rest on an assumption): classification carried through for a deterministic failure, for a retryable one, cancelled → unclassified, unclassified failure stays unclassified, and a bound expiry → "timeout" with no classification. Each enters the wait before settling the Deferred, because the settle watcher removes the writers-map entry on settlement and a later wait returns "no-writer" — which would pass "no classification" vacuously.

Revert probes — all verbatim

# reverted result
1 admission block → if (already.has(t)) continue (Part 1's code) Expected: 3 / Received: 2a TRANSIENT failure at the final threshold re-fires it… · 14 pass / 1 fail
2 drop currentTokens < gate (the progress bound) Expected: 2 / Received: 3 — same case · 14 pass / 1 fail
3 arm on any settled failure (drop the retryable check) Expected: 2 / Received: 3a DETERMINISTIC failure at the final threshold arms no gate · 14 pass / 1 fail
4 drop if (!isFinal) return Expected: 2 / Received: 3a TRANSIENT failure at a NON-final threshold arms no gate · 14 pass / 1 fail
5 drop if (gate <= firedAt) return (the window ceiling) Expected: 2 / Received: 3…no gate when the window has no room left for a retry · 14 pass / 1 fail
6 resetThresholds no longer clears the gate Expected: 4 / Received: 5resetThresholds drops a pending gate… · 14 pass / 1 fail
7 drop the classification passthrough in checkpoint.ts expect(received).toEqual(expected) - Expected - 5 / + Received + 1 and Expected: true / Received: undefinedcarries the failure classification off the writer's outcome, carries a retryable classification through unchanged · 6 pass / 2 fail

Reported because it is a negative result, not hidden: removing only if (settled.outcome !== "failure") return leaves the suite 16 pass / 0 fail. That guard is intent-stating and structurally redundant, because a "timeout" settlement also carries no classification, so the retryable check rejects it anyway. Removing both guards turns it red (Expected: 2 / Received: 3, two cases). The test comment says exactly this rather than claiming a probe it does not have, and the invariant the redundancy rests on — a timeout never carries a classification — is asserted directly against the real service.

Part 3 verification

Identical command both sides, same fresh worktree, same base — baseline sha printed and compared: e3bd071eb (Part 2's head) before and after.

bun test test/session/prune.test.ts test/session/prune-skip-system.test.ts test/session/checkpoint-writer-wait-timeout.test.ts test/session/checkpoint-child-session.test.ts test/session/checkpoint-splitover-integration.test.ts

result
pristine baseline at e3bd071eb, before any Part 3 edit 27 pass / 0 fail, 109 expects, 10.33s
after Part 3 38 pass / 0 fail, 143 expects, 15.77s

Net +11 tests, +34 expects, 0 fail either side. All 27 pre-existing cases still pass, including Part 1's three rewritten prune.test.ts cases that pin "a failed writer does not re-arm its own threshold" — see the next section for why they did not need rewriting.

bun typecheckexit 0 (12 of 12 turbo tasks). Nothing generated changed, so packages/sdk/js is untouched by Part 3. oxlint on the five changed files: 0 errors, warnings 30 → 36, all six new ones the pre-existing {} as any promptOps pattern this file already uses throughout. The full suite was deliberately not run.

No existing assertion was weakened — and why that is not luck

Part 1's cases assert that a failed writer does not re-arm its own threshold. The gate is narrower than those cases on both axes: it requires a retryable class, and it applies only to the final threshold. Every failure in those cases is pushed as { outcome: "failure" } with no failure field — an unclassified failure, which the gate refuses — and the first case's failure is at a non-final threshold anyway. So all three assertions are unchanged, and they now additionally pin that an unclassified failure is treated as "cannot know whether a retry helps" and behaves exactly as before. The in-file comment records this explicitly so a future reader does not read the new describe block as a reversal of the old one.

The only edits to those tests are scaffolding: the seeded queue now holds WriterSettlement objects and the stub implements waitForWriterSettlement (the method production calls). One test was strengthened during review — the non-final case originally stopped one fire too early and did not actually catch probe 4; it now fires once more and does.

The stale comment Part 1 left behind — treated as a defect, not a nit

test/session/checkpoint-writer-wait-timeout.test.ts still justified two #1938 cases with "this is the direction that keeps the failure cap reachable in the slow regime … a permanently broken slow writer would never be counted", and "booking happens in prune, whose counter is closure-private". The cap and the counting were deleted by Part 1 of this very PR. A stale test comment reads as intent and it caused a real misreading during this work.

Rewritten to state what the cases actually pin — #1938's success/failure/timeout contract — with the dead justification recorded as removed rather than quietly dropped. And the honest answer to "is the distinction consumed only by tests?": it was, and it no longer is. Before Part 3, prune discarded the wait's return value entirely, so nothing in production read "timeout" differently from "failure". Now prune's gate refuses to arm on "timeout" (prune.ts:389), so the distinction has a production consumer. The flat three-value waitForWriter itself, however, has no production caller left — prune uses the settlement shape — so it is now a contract these tests hold still, and that is stated in the file.

…xpires

Follow-up to #1938. That PR stopped a merely-slow writer from being booked as
a failure by returning a distinct "timeout" from waitForWriter, which prune's
`result !== "failure"` guard skips. Correct, but it left two holes.

1. Hitting the bound became completely silent. waitForWriter returned and
prune's watcher returned, neither logging — yet the +300s log line is exactly
the evidence #1938 was diagnosed from. waitForWriter now logs
"checkpoint writer wait bound expired — writer still in flight".

2. The real outcome was booked nowhere. prune's watcher fiber is the only
holder of the per-fire accounting and writerFailures is private to the prune
layer, so once the watcher returned on "timeout" a writer that genuinely
FAILED past 300s ticked no counter — making MAX_WRITER_FAILURES unreachable
for exactly the slow regime #1938 is about, so a permanently-broken-but-slow
writer retried forever with no give-up warning. Symmetrically, a writer that
SUCCEEDED past 300s never cleared a counter left at 1-2 by earlier fast
failures, so a later fast failure could trip "gave up" for a session whose
writers demonstrably work.

The watcher now extends its wait across bound expiries and accounts for the
settled result, capped at MAX_WRITER_WAIT_EXTENSIONS (~1h) so a writer that
never settles cannot pin the fiber for the life of the process. Two
microsecond-wide re-entry windows are documented rather than papered over.

Tests: prune.test.ts gains the two cases that pin the prune-side consequence
#1938 is actually about (timeouts never tick the counter and a late success
clears it; a late failure is still counted so the cap stays reachable) — the
existing test only asserted waitForWriter's return value. The timeout test now
also pins the BOUND (still pending at 4 minutes, so shrinking it to 1s fails)
and asserts the writer is still running after the expiry, replacing a dead
`not.toBe("failure")` implied by the line above it.

Also folds the stale 5-min-padding comment into the current block and corrects
checkpoint-align.ts's now-conditional claim about writerFailures.
wqymi added 2 commits July 29, 2026 17:25
…ion cap

The bounded wait's late-outcome contract and MAX_WRITER_WAIT_EXTENSIONS were
both asserted only by reading the code. Two gaps are now covered:

- waitForWriter: a writer that settles AFTER the 5-minute bound reports its
  REAL outcome (success / failure) to the re-entered wait, which is the
  only surface prune learns it from. Driven on TestClock against the real
  service, so the seam the prune-side stubs assume is now exercised.
- prune: the extension cap is a boundary, so both sides of it are pinned —
  12 extensions still books the writer's outcome, a 13th abandons the wait.
  The stub's leftover queue counts the waitForWriter calls exactly, so moving
  the constant by one fails a test.
…hold retry

The checkpoint writer is fail-acceptable and the code already implements
that redundancy:

  - ensureCheckpointTemplate (checkpoint.ts:117-121) writes the template
    only when the file is ABSENT, so a failed writer cannot clobber the
    previous checkpoint.
  - last_checkpoint_message_id advances only on a successful insert
    (checkpoint.ts:918-934; the sole external resetThresholds caller is
    prompt.ts:428, gated on `if (inserted)`).

So a failed write leaves file and watermark mutually consistent, both
pointing at the last good checkpoint, and a rebuild uses that
automatically. The cost of a failure is a STALER checkpoint, never a
missing one, and the next threshold crossing is a natural retry with
fresher context than an in-place one.

The accounting layered on top did not trust that, and its failure model
was wrong: a writer failing repeatedly means the provider is broken, in
which case the foreground turns are failing too and the user already
knows. The writer's LLM calls go through the same retry ladder as the
foreground (retry.ts), so any failure reaching prune is already
post-retry. Counting it again, three times, then falling silent, added
nothing.

Deleted:
  - MAX_WRITER_WAIT_EXTENSIONS and the wait-extension loop
  - writerFailures, MAX_WRITER_FAILURES, and the in-place-retry path
    that cleared crossed/maxCrossed on failure
  - cfg.checkpoint.max_writer_failures (schema, docs, generated SDK)

Kept: the bounded wait itself, forked purely as observation, so
waitForWriter's bound-expiry log stays reachable. That log is the only
evidence a slow writer leaves, and a line exactly like it is what
diagnosed #1938. Terminal outcomes are already logged by the settle
watcher, so the outcome is discarded here.

waitForWriter and its distinct "timeout" outcome are #1938's merged work
and are untouched.

Residual, accepted: with no in-place retry, a session whose LAST
threshold fails (mid-tier's last is 80%) gets no further fire, so its
checkpoint stays stale and an overflow rebuild uses the stale one. That
is the design's own premise, not a regression.
@wqymi wqymi changed the title fix(checkpoint): book the writer's real outcome when the wait bound expires fix(checkpoint): delete the writer failure accounting — the next threshold is the retry Jul 29, 2026
wqymi added 2 commits July 30, 2026 02:24
The failure arm was `{ status: "failure"; error: string }` — a stringified
message and nothing else. A consumer could not tell a transient provider
error from a deterministic one without matching on prose, which is why the
previous commit could only delete the checkpoint writer's blind failure
count rather than replace it.

Widen it additively: `failure?: FailureInfo` with a coarse `kind`
(transient / overflow / auth / aborted / other), a `retryable` flag and the
persisted NamedError `name`. `error: string` is unchanged and still carries
the human text.

Populated at runAgentLoop's failure site, where the typed error still
exists: `info.error` is the already-normalized named error, so
SessionRetry.retryable is reused verbatim as the retryability oracle (its
input type IS that shape) and `kind` is read off the named-error identity
that MessageV2.fromError derived from ProviderError.parseAPICallError /
isOverflow / isOpenAiErrorRetryable. No new taxonomy.

A plain Error subclass carries it across the Effect failure channel to the
single AgentOutcome construction site; Cause.pretty renders that subclass
byte-identically to `new Error(message)`, so `error` is unchanged. Failures
raised anywhere else leave the field absent rather than claiming a class
they do not have.
…ansient one

The settle watcher logged one undifferentiated warn for every non-success
outcome, so "will fix itself at the next threshold" and "will never work"
read identically. That was the gap left by deleting the failure count:
the accounting went away and nothing took its place.

Read the classification the outcome now carries. A transient failure is
already the retry ladder's business and keeps the existing line; a
deterministic one (overflow / auth / bad request) recurs identically at
every future threshold, so it says so and names the cause.

Stateless on purpose — no per-session memory of previous failures. That
memory is a trend, and the trend belongs to the user-facing persistent-
failure warning, which is a separate change. Reintroducing it here under
another name is exactly what was removed.

The watermark invariant is untouched: it still advances only on success.
@wqymi wqymi changed the title fix(checkpoint): delete the writer failure accounting — the next threshold is the retry fix(checkpoint): delete the writer's blind failure count and classify the failure instead Jul 29, 2026
…very

"The next threshold crossing is the retry" presumes a next threshold. The
last one has none, and its only would-be successor is unreliable: the
discard+rebuild that maxThresholdCrossed triggers re-arms the ladder via
resetThresholds (prompt.ts:428), but rebuildFromCheckpoint bails when
lastBoundary is unset (prompt.ts:413) — and the watermark is unset exactly
when no writer has ever succeeded. So in the one case that needs recovery
most, the session never checkpoints again until overflow.

The principle: a failed checkpoint write earns a retry only when the
failure's CLASS says a retry could succeed, and only where the token axis
still has ROOM for one. Both are readings of the present state, so neither
needs failure history.

Mechanism (prune.ts): on a settled retryable failure of the FINAL
threshold, arm a per-session gate one ladder STEP above the token count
that fired, clamped to maxAllowed. Deterministic or unclassified failures
arm nothing — they would recur identically. Non-final thresholds arm
nothing — the ladder already is their retry. The gate is consumed when it
fires and dropped by resetThresholds.

Why this is not MAX_WRITER_FAILURES renamed: a counter accumulates attempts
and permits N retries whether or not anything changed. This is a position,
overwritten not accumulated, derived from the ladder and the window. It
bounds the retry RATE to the ladder's own firing rate, bounds the retry
COUNT to whatever the unused window affords (zero once firedAt + step is
past maxAllowed), and a conversation that stops growing stops retrying by
itself.

Plumbing: waitForWriterSettlement carries the FailureInfo the outcome
already had. #1938's flat waitForWriter contract is untouched — it is now
projected off the same implementation, so the two cannot drift. A
"timeout" never carries a classification, so a merely slow writer can
never arm the gate.

Also corrects a test comment that still justified itself by "keeping the
failure cap reachable" so a broken slow writer "would never be counted" —
the cap and the counting were deleted earlier in this branch.
@wqymi wqymi changed the title fix(checkpoint): delete the writer's blind failure count and classify the failure instead fix(checkpoint): delete the writer's blind failure count, classify the failure, and give the final threshold a bounded recovery Jul 29, 2026
@wqymi
wqymi force-pushed the fix/checkpoint-writer-timeout-followup branch from 6af991d to 3ee9142 Compare July 29, 2026 22:55
@wqymi
wqymi merged commit 6c87eda into main Jul 30, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant