fix(checkpoint): delete the writer's blind failure count, classify the failure, and give the final threshold a bounded recovery - #1945
Merged
Conversation
…xpires Follow-up to #1938. That PR stopped a merely-slow writer from being booked as a failure by returning a distinct "timeout" from waitForWriter, which prune's `result !== "failure"` guard skips. Correct, but it left two holes. 1. Hitting the bound became completely silent. waitForWriter returned and prune's watcher returned, neither logging — yet the +300s log line is exactly the evidence #1938 was diagnosed from. waitForWriter now logs "checkpoint writer wait bound expired — writer still in flight". 2. The real outcome was booked nowhere. prune's watcher fiber is the only holder of the per-fire accounting and writerFailures is private to the prune layer, so once the watcher returned on "timeout" a writer that genuinely FAILED past 300s ticked no counter — making MAX_WRITER_FAILURES unreachable for exactly the slow regime #1938 is about, so a permanently-broken-but-slow writer retried forever with no give-up warning. Symmetrically, a writer that SUCCEEDED past 300s never cleared a counter left at 1-2 by earlier fast failures, so a later fast failure could trip "gave up" for a session whose writers demonstrably work. The watcher now extends its wait across bound expiries and accounts for the settled result, capped at MAX_WRITER_WAIT_EXTENSIONS (~1h) so a writer that never settles cannot pin the fiber for the life of the process. Two microsecond-wide re-entry windows are documented rather than papered over. Tests: prune.test.ts gains the two cases that pin the prune-side consequence #1938 is actually about (timeouts never tick the counter and a late success clears it; a late failure is still counted so the cap stays reachable) — the existing test only asserted waitForWriter's return value. The timeout test now also pins the BOUND (still pending at 4 minutes, so shrinking it to 1s fails) and asserts the writer is still running after the expiry, replacing a dead `not.toBe("failure")` implied by the line above it. Also folds the stale 5-min-padding comment into the current block and corrects checkpoint-align.ts's now-conditional claim about writerFailures.
…ion cap The bounded wait's late-outcome contract and MAX_WRITER_WAIT_EXTENSIONS were both asserted only by reading the code. Two gaps are now covered: - waitForWriter: a writer that settles AFTER the 5-minute bound reports its REAL outcome (success / failure) to the re-entered wait, which is the only surface prune learns it from. Driven on TestClock against the real service, so the seam the prune-side stubs assume is now exercised. - prune: the extension cap is a boundary, so both sides of it are pinned — 12 extensions still books the writer's outcome, a 13th abandons the wait. The stub's leftover queue counts the waitForWriter calls exactly, so moving the constant by one fails a test.
…hold retry
The checkpoint writer is fail-acceptable and the code already implements
that redundancy:
- ensureCheckpointTemplate (checkpoint.ts:117-121) writes the template
only when the file is ABSENT, so a failed writer cannot clobber the
previous checkpoint.
- last_checkpoint_message_id advances only on a successful insert
(checkpoint.ts:918-934; the sole external resetThresholds caller is
prompt.ts:428, gated on `if (inserted)`).
So a failed write leaves file and watermark mutually consistent, both
pointing at the last good checkpoint, and a rebuild uses that
automatically. The cost of a failure is a STALER checkpoint, never a
missing one, and the next threshold crossing is a natural retry with
fresher context than an in-place one.
The accounting layered on top did not trust that, and its failure model
was wrong: a writer failing repeatedly means the provider is broken, in
which case the foreground turns are failing too and the user already
knows. The writer's LLM calls go through the same retry ladder as the
foreground (retry.ts), so any failure reaching prune is already
post-retry. Counting it again, three times, then falling silent, added
nothing.
Deleted:
- MAX_WRITER_WAIT_EXTENSIONS and the wait-extension loop
- writerFailures, MAX_WRITER_FAILURES, and the in-place-retry path
that cleared crossed/maxCrossed on failure
- cfg.checkpoint.max_writer_failures (schema, docs, generated SDK)
Kept: the bounded wait itself, forked purely as observation, so
waitForWriter's bound-expiry log stays reachable. That log is the only
evidence a slow writer leaves, and a line exactly like it is what
diagnosed #1938. Terminal outcomes are already logged by the settle
watcher, so the outcome is discarded here.
waitForWriter and its distinct "timeout" outcome are #1938's merged work
and are untouched.
Residual, accepted: with no in-place retry, a session whose LAST
threshold fails (mid-tier's last is 80%) gets no further fire, so its
checkpoint stays stale and an overflow rebuild uses the stale one. That
is the design's own premise, not a regression.
The failure arm was `{ status: "failure"; error: string }` — a stringified
message and nothing else. A consumer could not tell a transient provider
error from a deterministic one without matching on prose, which is why the
previous commit could only delete the checkpoint writer's blind failure
count rather than replace it.
Widen it additively: `failure?: FailureInfo` with a coarse `kind`
(transient / overflow / auth / aborted / other), a `retryable` flag and the
persisted NamedError `name`. `error: string` is unchanged and still carries
the human text.
Populated at runAgentLoop's failure site, where the typed error still
exists: `info.error` is the already-normalized named error, so
SessionRetry.retryable is reused verbatim as the retryability oracle (its
input type IS that shape) and `kind` is read off the named-error identity
that MessageV2.fromError derived from ProviderError.parseAPICallError /
isOverflow / isOpenAiErrorRetryable. No new taxonomy.
A plain Error subclass carries it across the Effect failure channel to the
single AgentOutcome construction site; Cause.pretty renders that subclass
byte-identically to `new Error(message)`, so `error` is unchanged. Failures
raised anywhere else leave the field absent rather than claiming a class
they do not have.
…ansient one The settle watcher logged one undifferentiated warn for every non-success outcome, so "will fix itself at the next threshold" and "will never work" read identically. That was the gap left by deleting the failure count: the accounting went away and nothing took its place. Read the classification the outcome now carries. A transient failure is already the retry ladder's business and keeps the existing line; a deterministic one (overflow / auth / bad request) recurs identically at every future threshold, so it says so and names the cause. Stateless on purpose — no per-session memory of previous failures. That memory is a trend, and the trend belongs to the user-facing persistent- failure warning, which is a separate change. Reintroducing it here under another name is exactly what was removed. The watermark invariant is untouched: it still advances only on success.
…very "The next threshold crossing is the retry" presumes a next threshold. The last one has none, and its only would-be successor is unreliable: the discard+rebuild that maxThresholdCrossed triggers re-arms the ladder via resetThresholds (prompt.ts:428), but rebuildFromCheckpoint bails when lastBoundary is unset (prompt.ts:413) — and the watermark is unset exactly when no writer has ever succeeded. So in the one case that needs recovery most, the session never checkpoints again until overflow. The principle: a failed checkpoint write earns a retry only when the failure's CLASS says a retry could succeed, and only where the token axis still has ROOM for one. Both are readings of the present state, so neither needs failure history. Mechanism (prune.ts): on a settled retryable failure of the FINAL threshold, arm a per-session gate one ladder STEP above the token count that fired, clamped to maxAllowed. Deterministic or unclassified failures arm nothing — they would recur identically. Non-final thresholds arm nothing — the ladder already is their retry. The gate is consumed when it fires and dropped by resetThresholds. Why this is not MAX_WRITER_FAILURES renamed: a counter accumulates attempts and permits N retries whether or not anything changed. This is a position, overwritten not accumulated, derived from the ladder and the window. It bounds the retry RATE to the ladder's own firing rate, bounds the retry COUNT to whatever the unused window affords (zero once firedAt + step is past maxAllowed), and a conversation that stops growing stops retrying by itself. Plumbing: waitForWriterSettlement carries the FailureInfo the outcome already had. #1938's flat waitForWriter contract is untouched — it is now projected off the same implementation, so the two cannot drift. A "timeout" never carries a classification, so a merely slow writer can never arm the gate. Also corrects a test comment that still justified itself by "keeping the failure cap reachable" so a broken slow writer "would never be counted" — the cap and the counting were deleted earlier in this branch.
wqymi
force-pushed
the
fix/checkpoint-writer-timeout-followup
branch
from
July 29, 2026 22:55
6af991d to
3ee9142
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1938 (merged). Two halves, worth reading as separate commits: Part 1 deletes the checkpoint writer's blind failure accounting, and Part 2 replaces it by making a spawned agent's failure classifiable, then using that classification in the writer's one reporting site.
Part 1's previous revision argued for extending prune's wait and adding a cap; that argument is void and has been removed from this description. It also recorded an owed follow-on — "classify the failure instead of counting it" — which was blocked on
AgentOutcome's failure arm carrying only a string. Part 2 widens that contract and discharges it, so this PR no longer reads as "deleted the accounting, replaced it with nothing".#1938's core fix is untouched throughout:
waitForWriterstill returns a distinct"timeout"when its 300s bound expires with the writer still in flight.Part 1 — delete the blind failure count
What is deleted is the failure accounting layered on top of #1938 — including the pre-existing counter that predates it.
Why: the premise the accounting was built on does not hold
The writer fires at every threshold crossing (4 @ 20% for mid-tier windows, 9 @ 10% for extended-context — see the schedule comment above
prune.ts:35). It is designed fail-acceptable, and the code already implements that:ensureCheckpointTemplate(checkpoint.ts:117-121) writes the template only when the file does not exist, so a failed writer does not clobber the previous checkpoint. It is a conditional write, not a scaffold-then-fill.last_checkpoint_message_idadvances only on a successful insert (checkpoint.ts:918-934; the sole external caller ofresetThresholdsisprompt.ts:428, gated onif (inserted)).So one failed write leaves the file and the watermark mutually consistent, both pointing at the last good checkpoint. A rebuild uses that automatically. The cost of a failure is a staler checkpoint, never a missing one — and the next threshold crossing is a natural retry, with fresher context than an in-place re-fire of the same threshold would have had.
On top of that redundancy, the previous revision added machinery that did not trust it, and the failure model behind the counter was wrong: a writer failing repeatedly means the provider is broken, in which case the foreground turns are failing too and the user already knows. The writer's LLM calls go through the same retry ladder as the foreground (
retry.ts—isRetryableTransientError,RETRY_INITIAL_DELAY = 2000,RETRY_BACKOFF_FACTOR = 2, consumed bysession/llm.ts:34andsession/processor.ts:17; the writer runs as a spawned subagent through the same prompt loop). Any failure that reaches prune is therefore already post-retry. Counting it again, three times, then falling silent, adds nothing.What was deleted
MAX_WRITER_WAIT_EXTENSIONS = 12and the whole wait-extension loopprune.ts:32, loop atprune.ts:341-350MAX_WRITER_FAILURES = 3prune.ts:27writerFailuresmap + its commentprune.ts:178-182maxFailuresconfig readprune.ts:280crossed.delete()/maxCrossed.delete()on failure, plus the failure/give-up logsprune.ts:351-373cfg.checkpoint.max_writer_failuresschema entryconfig/config.ts:278mimocode-docs/reference/config.md:126,sdk/js/src/v2/gen/types.gen.ts:2209,sdk/openapi.json:13896grep -c "writerFailures\|MAX_WRITER_FAILURES\|MAX_WRITER_WAIT_EXTENSIONS\|max_writer_failures"over the pushed blobs of all five source files returns 0. (The only surviving mentions repo-wide are three comment lines in the rewritten tests that state, deliberately, what those tests used to assert.)The watcher fiber survives — as observation only, and that is load-bearing
The obvious reading of "delete the accounting" is to delete the forked watcher entirely, since nothing else consumed it. That would have been wrong, and the check is worth recording:
prune.ts:340and:349werewaitForWriter's only production callers (grep -rn waitForWriter src/). Deleting the fork would have made #1938's bound-expiry log — the one thing this PR was asked to keep — unreachable in production, surviving only as a line that tests exercise.So the fork remains, reduced to a single call whose result is discarded:
No loop, no counter, no retry. The wait exists purely so
waitForWriter'scheckpoint writer wait bound expired — writer still in flight(checkpoint.ts:1009) still fires. That log is cheap, at most once per writer, and it is the only evidence a slow writer leaves; #1938's own diagnosis came from a log line exactly like it. Terminal outcomes are already logged by the settle watcher (checkpoint.ts:920-934), so nothing is lost by discarding the return value.The config surface: removed, not deprecated
cfg.checkpoint?.max_writer_failureswas a public config surface, so removal vs. accept-and-ignore was a real decision. I expected the back-compat argument to force deprecation, and measured instead of assuming:.strict()atconfig/config.ts:485is applied to the top-level object only and does not recurse. An unknown key nested undercheckpointis silently accepted. So removing the field cannot break an existing config — a user who still hasmax_writer_failuresset gets no error, exactly as before.That removes the only real argument for keeping it. Deprecating would be strictly worse: a knob still published in the JSON schema, the docs table, and the SDK types, which provably does nothing — a user setting
max_writer_failures: 10would reasonably expect behaviour and get silence. Removed, with the docs row and both generated declarations removed alongside so no artifact advertises a knob the server dropped.Tests — rewritten, not deleted
Six cases in
prune.test.tsandT10incheckpoint-child-session.test.tspinned the deleted machinery. All seven are rewritten to pin the opposite requirement. Each rewrite states in-file what it used to assert and why that is no longer the requirement.What they used to assert: that a failure below the cap re-armed its own threshold (enqueue 1→2→3); that a success cleared the counter; that the watcher kept re-entering the bounded wait so a late outcome still reached the counter; and the exact 13-call boundary of that re-entry loop. All of it encoded in-place retry plus a give-up gate as requirements. Neither is a requirement now.
What is pinned instead —
prune.test.ts(3 cases, on the existingmakeRetryHarness):maxThresholdCrossedset — the old watcher pairedcrossed.deletewithmaxCrossed.delete, which withdrew the already-raised discard+rebuild signalprompt.tsconsumes. Overflow readiness must not depend on whether the writer happened to succeed.And the self-healing property asserted directly —
T10now drives the real service, seeds a known previous checkpoint (file content + watermark), fails the writer, and asserts the premise this whole change rests on:The file still holds the previous content and
lastBoundarystill returns the previous boundary — mutually consistent at the last good checkpoint. These two assertions pass under the old code as well, by design: they are properties ofcheckpoint.ts, not of the deleted accounting. They are here to make the premise executable rather than asserted in prose.Revert probe
Restoring only
src/session/prune.tsto the pre-change blob (git show HEAD:…), keeping the new tests:The probe also caught a flaw in my own T10. On that first run T10 passed under the reverted code — it should have failed. Cause: I had dropped the original T10's 50 ms pre-settle tick, so prune's fork had not yet reached
writers.get(...)insidewaitForWriterwhen the outcome settled, and it saw"no-writer"instead of the failure — the race documented inprune.ts. The test was passing by accident, not because the behaviour held. Tick restored (with a comment explaining why it is required), and T10 then discriminates:prune.tsrestored afterwards;grep -c MAX_WRITER_WAIT_EXTENSIONS src/session/prune.ts→ 0.Part 2 — the replacement: classify the failure instead of counting it
Part 1 deletes the blind count. On its own that leaves the writer's failure handling with nothing in its place, which is precisely what this PR's earlier revision recorded as owed: the better replacement is to classify the failure. That was not implementable then, and the obstacle was a type.
AgentOutcome's failure arm carried a stringified message and nothing else — noAPICallError, no status code, no retryable flag. A consumer could not tell a transient provider blip from a deterministic, will-never-work failure without matching on prose. Blind counting was the only thing the type permitted. So the count could be deleted honestly, but not replaced, until the contract was widened. That is what Part 2 does.The shape
Additive and backward-compatible:
error: stringis untouched and still carries the human text, and the new field is optional, so every existing consumer and every existing hand-built test literal still typechecks unchanged.failureis present only when the failure came from a settled assistant error. A work fiber that failed some other way — a defect during teardown, or a hand-built outcome such aswaitForWriter's{ status: "failure", error: "timeout" }(checkpoint.ts:986) — leaves it absent rather than asserting a class it does not have.retryablemeans "this error belonged to the retryable class", not "please retry". The child's LLM calls already run throughretry.ts's ladder, so any failure reaching a consumer is already post-retry — the same fact that licensed Part 1's deletion.The alternative I rejected
Two independent flat fields on the arm:
retryable?: booleanandkind?: FailureKind. Rejected because they admit incoherent states (kind: "overflow"withretryableabsent versus present-and-false) and because they force every read site to write its own absence guard — precisely the hazard AGENTS.md § "Reading a nullable column" describes, since a guard written=== undefinedtypechecks, reads correctly in review, and silently does nothing. One optional object is present-or-absent as a unit: a single truthiness check, after which the interior fields are non-optional and cannot disagree.Also rejected: carrying
provider/error.ts'sParsedAPICallErrordirectly. It is bound to the AI-SDKAPICallErrorshape at the provider boundary; by the time a failure reaches spawn the error has already been normalized into the persisted named-error union, so re-deriving it there would mean re-parsing — the thing this change exists to avoid.Populated where the typed error still exists
actor/spawn.ts:354, insiderunAgentLoop. That line previously threw the typed error away:info.erroris the assistant message's already-normalized named error, so no new taxonomy was added — the existing classifiers are reused at exactly this point:SessionRetry.retryable(session/retry.ts) is the retryability oracle. Its input type is this data shape (Err=== the return ofNamedError["toObject"]), and it already folds inisRetryableTransientErrorplus every 429 / 5xx / quota /upstream_errorspecial case. Reusing it means the classification cannot drift from the ladder that actually decides retries.kindis read off the named-error identity thatMessageV2.fromError(message-v2.ts:1159-1186) derived fromProviderError.parseAPICallError/isOverflow/isOpenAiErrorRetryable:ContextOverflowError→overflow,ProviderAuthError→auth,MessageAbortedError→aborted.APIErrorwithstatusCode401/403 also maps toauth, reading the status codeparseAPICallErroralready extracted.ProviderAuthErroronly covers a missing key (LoadAPIKeyError); a rejected one arrives as anAPIError, and to a consumer they are the same thing.Carrying it the last few frames needs one small class, because
forkWork'sonFailure— the single placeAgentOutcomeis constructed (spawn.ts:737-744) — only has aCause:AssistantSettledErroris a plainErrorsubclass, chosen deliberately:Cause.prettyrenders it byte-identically tonew Error(message)(measured before committing to it), soerror's existing text is unchanged rather than gaining a class prefix.Consumer inventory
Every consumer is an awaiter of the
AgentOutcomeDeferred.grep -rn "Deferred.await" src/ | grep -i outcomefinds six production sites across four files:workflow/runtime.ts:845-852status !== "success", then(outcome as { error?: string }).errorworkflow/runtime.ts:971-980workflow/runtime.ts:999-1023status === "success"onlyactor run)tool/actor.ts:788-816status === "failure"→.errorsession ask)tool/session.ts:149-151status === "failure" ? .error : .statuscheckpoint.ts:914-960status === "success"gates the watermarkwaitForWritertranslationcheckpoint.ts:984-990success/failure; also constructs{ status: "failure", error: "timeout" }Two entries from the original consumer list turn out not to be consumers, and the check is worth recording:
dream/distillnever see anAgentOutcome. They are agent types, invoked throughsp.prompt({ agent: "dream" | "distill", … })atsession/prompt.ts:3115and:3126— they do not go throughActor.spawnat all. (spawn.ts:217's comment lists them among spawn's callers, which is what makes this worth stating explicitly; the actual code path is a direct prompt.)actor/group.ts:141-146matches a grep foroutcomebut reads actor-registry status strings, notAgentOutcome.Test literals that hand-build the failure arm —
auto-overflow-writer-first.test.ts:164,checkpoint-child-session.test.ts:367and:477,checkpoint-watermark-transactional.test.ts:131— all still compile unchanged, which is the optional field doing its job.AgentOutcomeis an internal TypeScript type, never surfaced through the zod/OpenAPI layer:grep -n "AgentOutcome\|FailureInfo\|FailureKind"oversdk/js/src/v2/gen/types.gen.tsandsdk/openapi.jsonreturns nothing, so Part 2 needs no artifact regeneration. Part 1's SDK edits, which removemax_writer_failures, stand as described above.The one behaviour change: the writer's log stops lying by omission
Part 1 leaves the settle watcher emitting one undifferentiated warn for every non-success outcome (
checkpoint.ts:930), so "will fix itself at the next threshold" and "will never work" read identically. That is the hole the deletion opened. With the classification available:Stateless on purpose. No per-session memory of previous failures, under any name. Such memory is a trend detector, and the trend belongs to the user-facing warning described under "Residual" below — accumulating it here would be re-adding, with a different name, exactly what Part 1 removed. Read as "report a deterministic failure as the distinct thing it is", not as "suppress the repeat"; suppression would need that state.
Nothing else changes. No retry policy, no give-up logic, no user-facing warning, and no consumer other than #6 reads the new field.
Tests for Part 2
Four cases in
test/actor/spawn.test.ts, all driven through the real construction path: the classification is produced byrunAgentLoop, carried across the failure channel, squashed back out and assembled byforkWork'sonFailure. No test builds anAgentOutcome.APIError429,isRetryable: true) →kind: "transient",retryable: trueContextOverflowError) →kind: "overflow",retryable: falseAPIError401) →kind: "auth",retryable: falsefailure == null(truthiness /== null, never=== undefined), so a success can never carry an invented classificationEach also asserts
errorstill containsActor assistant failed: <Name>, pinning the backward-compatible string.The upstream provider round-trip is stood in for at the
SessionPrompt.promptboundary, and that is a deliberate, disclosed choice rather than convenience:SessionRetry.policyis uncapped — its schedule stops only whenretryable()returns falsy — so a genuinely retryable provider error never settles on its own. A real-server transient case would hang, not fail. This is also why consumer #6 branches on!failure.retryablerather than trying to act on transient: in the current architecturekindcarries the discriminating power andretryableis largely documentary.Revert probe
Restoring only the construction site at
spawn.ts:354to its previousnew Error(...)— keeping the new tests, so the field exists but is never populated:The single pass is the negative case — it asserts absence, so it must keep passing under the revert, and does. Construction site restored afterwards;
bun typecheckexit 0 and the classification tests green again.Verification
bun typecheck→ exit 0 forpackages/opencodeand forpackages/sdk/js(the two packages touched).Scoped run, identical command both sides —
bun test test/session/prune.test.ts test/session/prune-skip-system.test.ts test/session/checkpoint-child-session.test.ts test/session/checkpoint-writer-wait-timeout.test.ts --timeout 120000:3068ed9f9, before any editNet −3 tests: six
prune.test.tscases replaced by three;T10replaced 1:1.Part 2, identical command both sides —
bun test test/actor/spawn.test.ts test/session/checkpoint-child-session.test.ts test/session/checkpoint-writer-wait-timeout.test.ts test/session/prune.test.ts --timeout 30000, run in the same fresh worktree at the same base so the environment is a genuine control:4e011eaf7(Part 1's head), before any Part 2 editNet +4 tests, +15 expects, and the 45 pre-existing cases — including Part 1's rewritten
prune.test.tsandT10, which pin "a failed writer does not re-arm its own threshold" — all still pass. No existing assertion was weakened or removed.bun typecheck→ exit 0 forpackages/opencodeand forpackages/sdk/js. The pre-push hook additionally ran the repo-wide typecheck: 12 of 12 tasks successful.The full suite was deliberately not run — other work is live on this box and concurrent full runs starve each other badly enough to make pass counts meaningless.
Residual, stated plainly
A session whose last threshold fails gets no further fire.— CLOSED BY PART 3. This was written down here as an accepted consequence of Part 1 ("the retry is the next crossing, and a last threshold has no next"). A review pushed back, and on measurement the consequence was worse than recorded: the final threshold's would-be successor is the discard+rebuild'sresetThresholds, which never runs whenlastBoundaryis unset — i.e. exactly when no writer has ever succeeded — so the session stopped checkpointing permanently rather than merely going stale. Part 3 gives the final threshold a class-gated, window-bounded recovery gate. The paragraph is kept rather than deleted so the reasoning that accepted the hole, and the measurement that overturned it, both stay on the record.A user-visibility gap this PR deliberately does not close. When checkpointing is persistently broken, the overflow fallback is compaction — and that fallback is lossy. Correction, measured after this section was written (see #1973):
compaction.createin isolation is only 3 local writes (p50 0.240 ms), which is where the earlier "unsummarized amputation" claim came from — butrunLooppairs it withcompaction.process(prompt.ts:3290-3298), a real LLM summarization, so the path does summarize. The in-tree comment that asserted otherwise was false and is corrected in #1973. The user-visibility gap therefore remains, but its stake is "your context is being summarized away repeatedly and silently", not "dropped unsummarized". The user should be warned about that, but the warning belongs in its own change, and its trigger must be a trend ("the last N thresholds all failed") or the moment a rebuild finds its checkpoint too stale — never a failure count, which is exactly the mechanism deleted here for being the wrong shape. Part 2 supplies the input such a warning needs (a deterministic failure is now identifiable as such) and deliberately stops there: it adds no per-session state, so nothing here is a trend detector wearing a new name.Relationship to #1938
Unchanged and not reverted:
waitForWriter(nowcheckpoint.ts:1087), its 5-min bound, its distinct"timeout"outcome, theWriterOutcometype (checkpoint.ts:534), and the bound-expiry log (checkpoint.ts:1070). Part 3 moved the implementation intowaitForWriterSettlement(checkpoint.ts:1040) and madewaitForWritera projection of it, so the contract is now derived from the same code path rather than sitting beside it — it cannot drift. Line numbers here are re-read from the pushed blob at3ee914261. #1938'scheckpoint-writer-wait-timeout.test.tsassertions are all retained verbatim; only two of its comments changed, where they justified thetimeout/failuredistinction by pointing at the counter that no longer exists. The distinction itself stands on its own: a slow-but-working writer must not read as a broken one.checkpoint-align.ts's doc comment, which described when a rejection incrementswriterFailures, now describes what actually happens — the previous checkpoint and its watermark stay in place and the next crossing re-covers the same delta.Deliberately not included
kindinto theWriterCachePerfbus event. That event is the natural substrate for cross-layer writer-health accounting, which is the trend detector's business, not this PR's.computeBoundarycoverage pinning (checkpoint.ts:254-293) — it changes what a checkpoint covers and is a separate, riskier decision.Part 3 — recovery: the final threshold gets a gate
Part 1's argument was "the next threshold crossing is the retry." A review correctly pointed out that this presumes a next threshold, and the last one has none. Part 3 closes that, using Part 2's classification as the discriminator.
What "no next threshold" actually costs — measured, because the naive statement understates it
The final threshold is not simply successor-less. It has a would-be successor, and that successor is conditional in exactly the wrong way:
fireCheckpointsalso setsmaxCrossed(prune.ts:428), and the same runLoop iteration consumes it:overflowCheck(...) || prune.maxThresholdCrossed(sessionID)→rebuildFromCheckpoint(prompt.ts:3308-3352).resetThresholds(prompt.ts:428), which clearscrossed— re-arming the entire ladder from 20%. So in the healthy case the final threshold does get a retry: the ladder restarts.rebuildFromCheckpointreturnsfalsewhenlastBoundaryis unset (prompt.ts:413), and the watermark is unset precisely when no writer has ever succeeded.⇒ In the one case that needs recovery most — the first checkpoint failed — there is no boundary, no reset,
crossedstill holds every threshold, and the session never fires another writer for the rest of its life. Every subsequent iteration falls to compaction instead. That is not "a staler checkpoint"; it is the subsystem switched off silently.(
maxCrossedis also never cleared in that path, so the compaction fallback repeats every other iteration. That is a pre-existing, separate defect — it predates this branch and is not touched here.)The principle
The mechanism below is meant to be readable as nothing more than that sentence applied to the one threshold that lacks a successor.
The mechanism
prune.tsgains one piece of per-session state —finalRetryAt: Map<SessionID, number>(prune.ts:190), the token count at or above which the final threshold may fire a second time.Admission (
prune.ts:303-318) — the only threshold that can re-fire is the final one, and only past its gate:Arming (
prune.ts:384-418), in the forked settle watcher that Part 1 already kept for the bound-expiry log:if (!isFinal) returnprune.ts:386if (settled.outcome !== "failure") returnprune.ts:389"timeout"means still in flight — the writer may yet succeed and advance the watermarkif (!settled.failure?.retryable)→ log, returnprune.ts:392gate = Math.min(firedAt + step, maxAllowed)prune.ts:401if (gate <= firedAt) returnprune.ts:402stepis the ladder's own spacing (prune.ts:297), andmaxAllowednow comes from a sharedmaxAllowedFor()(prune.ts:79) thatresolveThresholdsalso uses, so the clamp and the ceiling cannot drift apart.resetThresholdsdrops the gate (prune.ts:535): a rebuild re-arms the whole ladder, which supersedes it.Why this is not
MAX_WRITER_FAILURESunder a new nameThis was the explicit bar, so it is answered directly rather than asserted.
3)The retry budget is therefore the unused window, a physical quantity the system already measures: with the mid-tier ladder (4 @ 20%) the final threshold sits at 80% and one step overshoots
maxAllowed, so it affords at most one retry; with a 1M-window ladder (18 @ 5%) it affords two. Nobody chose those numbers — geometry did.Measurement that decided the alternatives
Rejected: "the loop is self-limiting because a broken writer means a broken provider, so the foreground dies too." This needed checking specifically for the writer, since a separately-configured writer model could be rate-limited while the foreground works. It does not have one.
prune.ts:324passes the foreground turn's own model intotryStartCheckpointWriter;checkpoint.tsforwards it toactor.spawn({ model });spawnSubagent→forkWork→runAgentLoop→sessionPrompt.prompt({ model }), and the resolution isinputModel ?? agentModel ?? lastModel(...)(prompt.ts:1836) —inputModelwins. Thecheckpoint-writeragent definition (agent/agent.ts:347) declares nomodel/modelRefat all, so there is nothing to lose the race anyway. ⇒ the shared-fate argument holds, which is why persistent failure still needs no counter — but it says nothing about a single transient failure at the last threshold, which is the only case Part 3 acts on.Rejected: bound the retry with a count (even
1). Arbitrary, and it is the deleted mechanism with a smaller constant. Ruled out by the bar, not by measurement.Rejected: append
maxAllowedto the ladder unconditionally, so the final threshold always sits at the last meaningful position and "no successor" becomes a fact about the window rather than a defect. Elegant and needs zero new state — but it adds a fifth writer per ladder cycle for every session, failure or not. Paying that always, to fix a case that arises only on failure, is the wrong trade.Rejected: clear the threshold whenever the failure was transient. This is the hot loop the brief warned about:
fireCheckpointsruns at the start of every runLoop iteration, so an unchanged token count would spawn a writer — a dozen sequential round-trips — per iteration. The progress gate is precisely what removes it, and a test pins that (below).Plumbing: how the class reaches prune without touching #1938's contract
Part 2 put
FailureInfoonAgentOutcome, but prune only seesWriterOutcome— three flat strings. Rather than widen that union (which would break every=== "failure"consumer and #1938's own test),checkpoint.tsgains a companion:WriterSettlement = { outcome: WriterOutcome | "no-writer"; failure?: FailureInfo }(checkpoint.ts:546)waitForWriterSettlement(checkpoint.ts:1040) holds the implementation — the same 5-min bound, the same bound-expiry log (checkpoint.ts:1070), plus the classification.waitForWriter(checkpoint.ts:1087) is now a projection of it:(yield* waitForWriterSettlement(id)).outcome.So #1938's contract is not merely "still there" — it is derived from the same code path, and the two cannot disagree about what a settled writer did.
failureis absent for a cancelled writer and for a failure that was never classifiable, i.e. absent means unknown, never retryable.Tests for Part 3 — six new cases, every one revert-probed
test/session/prune.test.ts, new nested describefinal-threshold recovery gate(harness stub rewired so the settlement path — the one production calls — is the driver; a stub implementing onlywaitForWriterwould have left every failure case in this file vacuous, since the seeded queue would never drain):"timeout"arms no gateresetThresholdsdrops a pending gatetest/session/checkpoint-writer-wait-timeout.test.ts, new describeSessionCheckpoint.waitForWriterSettlement— five cases against the real service (prune's tests stub the checkpoint service, so without these the AgentOutcome → WriterSettlement hop would be asserted nowhere and every case above would rest on an assumption): classification carried through for a deterministic failure, for a retryable one,cancelled→ unclassified, unclassified failure stays unclassified, and a bound expiry →"timeout"with no classification. Each enters the wait before settling the Deferred, because the settle watcher removes the writers-map entry on settlement and a later wait returns"no-writer"— which would pass "no classification" vacuously.Revert probes — all verbatim
if (already.has(t)) continue(Part 1's code)Expected: 3 / Received: 2— a TRANSIENT failure at the final threshold re-fires it… · 14 pass / 1 failcurrentTokens < gate(the progress bound)Expected: 2 / Received: 3— same case · 14 pass / 1 failretryablecheck)Expected: 2 / Received: 3— a DETERMINISTIC failure at the final threshold arms no gate · 14 pass / 1 failif (!isFinal) returnExpected: 2 / Received: 3— a TRANSIENT failure at a NON-final threshold arms no gate · 14 pass / 1 failif (gate <= firedAt) return(the window ceiling)Expected: 2 / Received: 3— …no gate when the window has no room left for a retry · 14 pass / 1 failresetThresholdsno longer clears the gateExpected: 4 / Received: 5— resetThresholds drops a pending gate… · 14 pass / 1 failcheckpoint.tsexpect(received).toEqual(expected)- Expected - 5 / + Received + 1andExpected: true / Received: undefined— carries the failure classification off the writer's outcome, carries a retryable classification through unchanged · 6 pass / 2 failReported because it is a negative result, not hidden: removing only
if (settled.outcome !== "failure") returnleaves the suite 16 pass / 0 fail. That guard is intent-stating and structurally redundant, because a"timeout"settlement also carries no classification, so theretryablecheck rejects it anyway. Removing both guards turns it red (Expected: 2 / Received: 3, two cases). The test comment says exactly this rather than claiming a probe it does not have, and the invariant the redundancy rests on — a timeout never carries a classification — is asserted directly against the real service.Part 3 verification
Identical command both sides, same fresh worktree, same base — baseline sha printed and compared:
e3bd071eb(Part 2's head) before and after.bun test test/session/prune.test.ts test/session/prune-skip-system.test.ts test/session/checkpoint-writer-wait-timeout.test.ts test/session/checkpoint-child-session.test.ts test/session/checkpoint-splitover-integration.test.tse3bd071eb, before any Part 3 editNet +11 tests, +34 expects, 0 fail either side. All 27 pre-existing cases still pass, including Part 1's three rewritten
prune.test.tscases that pin "a failed writer does not re-arm its own threshold" — see the next section for why they did not need rewriting.bun typecheck→ exit 0 (12 of 12 turbo tasks). Nothing generated changed, sopackages/sdk/jsis untouched by Part 3.oxlinton the five changed files: 0 errors, warnings 30 → 36, all six new ones the pre-existing{} as anypromptOps pattern this file already uses throughout. The full suite was deliberately not run.No existing assertion was weakened — and why that is not luck
Part 1's cases assert that a failed writer does not re-arm its own threshold. The gate is narrower than those cases on both axes: it requires a
retryableclass, and it applies only to the final threshold. Every failure in those cases is pushed as{ outcome: "failure" }with nofailurefield — an unclassified failure, which the gate refuses — and the first case's failure is at a non-final threshold anyway. So all three assertions are unchanged, and they now additionally pin that an unclassified failure is treated as "cannot know whether a retry helps" and behaves exactly as before. The in-file comment records this explicitly so a future reader does not read the new describe block as a reversal of the old one.The only edits to those tests are scaffolding: the seeded queue now holds
WriterSettlementobjects and the stub implementswaitForWriterSettlement(the method production calls). One test was strengthened during review — the non-final case originally stopped one fire too early and did not actually catch probe 4; it now fires once more and does.The stale comment Part 1 left behind — treated as a defect, not a nit
test/session/checkpoint-writer-wait-timeout.test.tsstill justified two #1938 cases with "this is the direction that keeps the failure cap reachable in the slow regime … a permanently broken slow writer would never be counted", and "booking happens in prune, whose counter is closure-private". The cap and the counting were deleted by Part 1 of this very PR. A stale test comment reads as intent and it caused a real misreading during this work.Rewritten to state what the cases actually pin — #1938's success/failure/timeout contract — with the dead justification recorded as removed rather than quietly dropped. And the honest answer to "is the distinction consumed only by tests?": it was, and it no longer is. Before Part 3, prune discarded the wait's return value entirely, so nothing in production read
"timeout"differently from"failure". Now prune's gate refuses to arm on"timeout"(prune.ts:389), so the distinction has a production consumer. The flat three-valuewaitForWriteritself, however, has no production caller left — prune uses the settlement shape — so it is now a contract these tests hold still, and that is stated in the file.