The release sweep has never once closed a retained child's slot: 0 released records in 6,916 runs, and 154 of 601 children never settle

[drewstone]

Summary

The released-terminal-record path added in 0.233.0 for #1247 has never executed once in production. Across 6,916 run journals on this operator host: 7,589 settled events, 0 with retainedExecution: 'released', and 0 environment-teardown receipts on any child id. Every retained-pending child still ends as reconciled with an open cursor slot, which fleetYield reports as neverSettled.

The rate is not improving. On the 108 runs that carry a fleetYield (0.233.0+): 154 of 601 spawned children never settled (25.6%). Today, verified-agency-20260919a (0.240.0, provider-tangle 1.6.0) ran 10.0 hours, spawned 4 children and lost 4 of 4.

I reproduced the gate locally on v0.240.0 (63c8f5a4) through the real providerAsExecutor retained path. Test output below.

The mechanism

A retained child that cannot be observed settles down with recoveryPending: true. scope.ts:3210-3218 writes a reconciled floor instead of a terminal record and defers the real one to the release sweep. That is correct and deliberate.

The sweep (scope.ts:2008-2110) only writes the deferred record behind child.cleanupConfirmed (scope.ts:2085), and cleanupConfirmed is set only when every teardown receipt reports destroyed: true and confirmTeardown() then succeeds (scope.ts:2037-2041).

For the failure that actually kills these children, that condition is unsatisfiable by construction. The child died because its sandbox stopped; environment.destroy() on a stopped sandbox answers HTTP 409, so providerAsExecutor.releaseRetained (environment-provider.ts:851-905) returns [receipt(false, …)], the gate stays shut, and the slot never closes.

The same fact — the environment is gone — is both the reason the child failed and the reason its settlement is withheld.

Downstream: spawn-journal.ts:501 puts the node in inDoubt, supervisor.ts:1313 counts it as fleetYield.neverSettled, spendGaps marks it never-settled, its tokens and USD are unattributed, and its transcript is unreachable.

Local reproduction (v0.240.0, 63c8f5a4)

A retained provider whose event stream and result() both raise HTTP 409: Sandbox is not running (status: stopped) — the production error verbatim — driven through createScope + providerAsExecutor + releaseRetainedEnvironments. Only environment.destroy() differs between the two arms.

[ok]    settled fields = {"retainedExecution":"pending","retainedPendingCause":"unobservable","reason":"retained provider execution requires reconciliation before replacement: caused by AggregateError: retained event observa…"}
[ok]    journal BEFORE sweep = {"spawned":1,"execution-input":1,"execution-admitted":3,"reconciled":1}
[ok]    executor.releaseRetained bound = function
[ok]    journal AFTER sweep  = {…,"reconciled":1,"environment-teardown":1,"settled":1}
[ok]      environment-teardown: destroyed=true
[ok]      settled: status=down, retainedExecution=released          <-- slot closes

[throw] settled fields = {"retainedExecution":"pending","retainedPendingCause":"unobservable","reason":"retained provider execution requires reconciliation before replacement: caused by AggregateError: retained event observa…"}
[throw] journal BEFORE sweep = {"spawned":1,"execution-input":1,"execution-admitted":3,"reconciled":1}
[throw] executor.releaseRetained bound = function
[throw] journal AFTER sweep  = {…,"reconciled":1,"environment-teardown":1}
[throw]   environment-teardown: destroyed=false,
            detail="providerAsExecutor(stopped-sandbox): environment.destroy() failed — HTTP 409: Sandbox is not running (status: stopped)"
[throw]   (no settled record)                                        <-- slot stays open forever

The [throw] arm is the production shape. It is deterministic: the child is neverSettled for the life of the run and of every reader after it.

A second red test shows the gate also shuts on a contract-violating executor that answers destroyed: true from releaseRetained but still destroyed: false from teardown — because confirmTeardown is teardownExecutor(…), which throws on a destroyed: false receipt (deadline.ts:116-120) and the throw is swallowed at scope.ts:2041. That contract is stated at types.ts:205-213 but nothing enforces or tests it.

Why this shipped untested

releaseRetained appears in exactly five source files and zero test files:

src/runtime/environment-provider.ts
src/runtime/supervise/scope.ts
src/runtime/supervise/types.ts
src/runtime/supervise/supervisor.ts
src/runtime/supervise/driver-executor.ts

No test drives releaseRetainedEnvironments through an executor that implements releaseRetained, so the release-written terminal record has no end-to-end coverage at all. observer-projection.test.ts asserts the projection of a released record that nothing produces.

A second, latent instance of the same class

Two wrappers rebuild Executor from an explicit allowlist and silently drop optional methods, releaseRetained among them:

• gateOnDeliverable — completion-gate.ts:78-141
• retryPreSpawnRefusals — worker-retry.ts:255-273, whose own doc at worker-retry.ts:179-180 claims it "delegates every other surface to it"

Both forward runtime, budgetExempt, deliver, progress, traceSource, metered, harnessTranscript, execute, teardown, resultArtifact and drop releaseRetained, recover, cancel, accounting, interactiveReady, teardownTimeoutMs. A worker behind either wrapper can never be released (so never settles), never recovered on resume, and never cancelled with provider acknowledgement — silently, because undefined is the legitimate "not implemented" answer. Neither wrapper is on the path the runs above took (no deliverable, no workerRetry), so this is a latent defect rather than the measured one, but it is the same omission and it produces exactly the zero-receipt shape.

Population

154 of 601 spawned children never settled across the 108 runs carrying a fleetYield. All 154 have the identical journal shape: reconciled present, environment-teardown absent, teardown-unconfirmed present, node status failed (147) or cancelled (7) with a settledAt already stamped.

run	date	runtime	provider	spawned	done	down	neverSettled
diu-proj00716-20260916-glm-v1-wide-f	09-16	0.236.0	1.5.0	7	0	0	7
diu-proj00716-20260916-kimi-v1-wide-f	09-16	0.236.0	1.5.0	7	0	0	7
mech-interp-foundations-sandbox-a-20260917d	09-17	0.237.0	1.5.0	12	4	3	5
mech-interp-foundations-sandbox-a-20260917g	09-17	0.237.0	1.5.0	14	7	2	5
diu-proj00716-20260916-long-horizon-j	09-18	0.238.0	1.5.0	25	7	0	18
diu-proj00716-20260916-release-manager-j	09-18	0.238.0	1.5.0	26	4	1	21
diu-proj00716-20260916-model-steward-j	09-18	0.238.0	1.5.0	15	6	0	9
zkp-frontier-prover-a-20260918a	09-18	0.239.0	1.6.0	8	2	0	6
verified-agency-20260918e	09-18	0.240.0	1.6.0	107	86	11	10
continual-capability-20260918e	09-18	0.240.0	1.6.0	118	99	8	11
verified-agency-20260919a (withdrawn: 81% host sleep)	09-19	0.240.0	1.6.0	4	0	0	4

(18 further runs with 1–4 each.) By version: 0.236.0 19/22, 0.237.0 12/60, 0.238.0 90/246, 0.239.0 8/40, 0.240.0 25/229.

Host sleep is controlled for, and it does not explain the population. I built the sleep timeline from pmset -g log on this host (Sleep -> next DarkWake/Wake; the log covers 2026-09-13T05:54Z..2026-09-19T23:32Z UTC). verified-agency-20260919a is sleep-confounded and withdrawn as an exhibit: the host was asleep 486.7 of its 600.6 minutes (81.0%) across 69 windows, all four children lived through 75-86% host sleep, and three of the four had their last observation land while the host was asleep. A laptop-resident supervisor cannot observe a retained run while the laptop is in Deep Idle, so that run explains itself.

The rest of the population does not. Within pmset coverage there are 154 never-settled and 447 settled children from the same runs in the same window. 133 of the 154 never-settled children (86%), across 26 of 31 runs, had ZERO host sleep over their entire spawn-to-last-observation span. Cohort comparison: host asleep over the child's lifetime, mean 10.6% never-settled vs 2.0% settled, median 0.0% for both; spawn instant inside a sleep window 0/154 and 0/447. The sleep signal is real but confined to a minority.

The distinction that matters: host sleep can explain why a child enters the reconciled state. It cannot explain why it can never leave it. The reproduction below runs offline, with no network and no provider, and shows the promotion gate failing on its own.

Ask

1. A retained-pending child whose environment cannot be destroyed must still reach a terminal state. An unreachable environment is stronger evidence that no later process will recover it, not weaker. scope.ts:2085 currently reads "we proved we destroyed it"; on the no-resume path (supervisePursuit forces retainedAtSettlement: 'release' and refuses re-entry) the correct condition is "no process will resume this run", which is already known at the call site. The unconfirmed environment is separately reported as teardown-unconfirmed, so closing the slot loses no leak evidence. If the distinction between "gone" and "alive but undeletable" matters, the receipt's detail (or a typed field) can carry it, rather than conflating both into "stay open forever".

2. Cover the release sweep end to end. One test with an executor that implements releaseRetained, on each of: destroyed: true, destroyed: false, [], and a throwing confirmTeardown. Today all four behave the same in production, and three of them are silent.

3. Stop rebuilding Executor from an allowlist. gateOnDeliverable and retryPreSpawnRefusals should delegate every unlisted surface (or spread the inner executor and override only what they change), so a new optional method cannot be silently dropped by a wrapper that claims to delegate it.

Open

Production is one step worse than the reproduction and I could not separate the last branch from records alone: the runs above have no environment-teardown receipt at all for any child, whereas the reproduced [throw] arm writes a destroyed: false one. releaseRetained returns [] only at environment-provider.ts:855 (!pending) and :860 (environmentId === undefined), or the sweep's own catch at scope.ts:2046. The records cannot tell these apart.

The measurement that separates them: log which of those three the sweep took, per child, at scope.ts:2037-2048 — the branch taken, typeof child.releaseRetained, and receipts.length — then press one short pursuit. One line, one run.

---

Filed from the discovery program; reproduction sources are src/runtime/supervise/never-settled-e2e.test.ts and never-settled-repro.test.ts in a worktree at v0.240.0, available on request.

[drewstone]

Correction (posted by the filer, same day). The original text used verified-agency-20260919a as the headline exhibit and asserted the run's coordination bad-gateway window did not explain its four lost children. That was wrong and I have withdrawn that run from the evidence.

This operator host was asleep for most of that run. Parsed from pmset -g log (Sleep -> next DarkWake/Wake): 486.7 of 600.6 minutes, 81.0%, across 69 windows, in ~45 s DarkWake cycles per ~10 min. All four children lived through 75-86% host sleep and three of four had their last observation land inside a sleep window. A laptop-resident supervisor cannot observe a retained provider run while the laptop is in Deep Idle, and a Tailscale funnel on a sleeping host has no upstream, so the bad gateways and the unobservable children are the same event. That run is not evidence of anything upstream.

What survives, and why the issue stands:

1. The population is not sleep-explained. Same pmset timeline, same runs, same window: of 154 never-settled children inside log coverage, 133 (86%) had zero host sleep over their entire spawn-to-last-observation span, across 26 of 31 runs. Control cohort of 447 settled children from the same runs: mean sleep exposure 2.0% vs 10.6% for never-settled, median 0.0% for both, spawn instant inside a sleep window 0/154 and 0/447.

2. The two questions are different. Host sleep can explain why a child enters reconciled. It cannot explain why a child can never leave it. The reproduction in the body runs offline — in-memory journal, fake provider, no network, no sandbox — and the gate fails there on its own.

3. The journal counts are unaffected by sleep. 0 released terminal records and 0 child environment-teardown receipts in 6,916 run journals are facts about what the runtime wrote, not about reachability.

Known limit I cannot close: host CPU load is not retained anywhere I can read retroactively, so for the 133 zero-sleep children I cannot rule out load-induced unreachability of the laptop-resident acknowledger as the reason they entered reconciled. That would change the cause of entry, not the gate that prevents exit.

[drewstone]

The same slot leak reaches a ROOT, and there it ends the run

Measured on discovery-lab's DIU run 3, 2026-09-20: 39 lanes pressed as two arms, each lane a root in its own Tangle sandbox under a cloud driver, agent-runtime 0.240.0 with provider-tangle 1.6.0, opencode with deepseek/deepseek-v4.1-flash, 8 GB roots.

31 settled: 9 winners, 19 driver-failed, 3 all-children-down.

All 19 carry one signature, and 11 of them end on a root-level teardown receipt that says the environment was not destroyed:

environment-teardown  destroyed: false  "retained owner environment cleanup was not confirmed"
teardown-unconfirmed  label: "scope owner"  runtime: "tangle-sandbox"  status: "done"

The sequence before it, from spawn-journal.jsonl:

1. opencode exits while a model step is in progress, killed by SIGABRT (this is the trigger and it is a real crash).
2. Every driver retry then meters nothing: iterations: 0, tokens {input: 0, output: 0}, tokensKnown: false, ms: 0, providerModel.status: "unknown".
3. RetainedExecutionPendingError: retained provider execution requires reconciliation before replacement: caused by Error: provider "tangle-sandbox" cannot control a retry-safe retained run.
4. The supervisor stops after 12 or 13 attempts on no progress, and the teardown above is the last event.

So the crash is survivable and the unconfirmed teardown is what makes it terminal: the root's retained execution cannot be reconciled or replaced, so there is nothing to retry into. This is your neverSettled child slot one level up, where the cost is the whole run rather than one worker.

One detail that argues for a platform moment rather than a per-lane fault: six of the 19 died within 11 seconds of each other, at 01:10:41Z to 01:10:52Z, across both arms and four different driver sandboxes. The other 13 are spread from 01:10Z to 07:51Z and interleave with winners.

Two things ruled out. The operator host was in Low Power Sleep from 00:45Z for nine hours while these lanes went on settling winners until 05:10Z, so the host is not involved; these records declare no placement.coordination and serve coordination from inside their own boxes. And the harness is not the variable: a scan of 2,238 driver-failed runs on the same host finds this signature in 27, of which 15 are pi with glm-5.3 from one unrelated run.

Evidence lives in pursuits/diu-proj00716-20260916-*-run3-{control,enabled}/spawn-journal.jsonl in discovery-lab, and the wave's reading is reports/diu-proj00716-portfolio/20260920-run3-products/README.md (discovery-lab#860, corrected in #861).

[drewstone]

Fresh measurement, and the transcript loss it causes.

Over the Discovery fleet records of 2026-09-23/24 on drew-gtr-pro (243 closed runs, agent-runtime 0.249.1 to 0.261.0), 181 spawned nodes in 71 runs never settled.
Runtime's own fleetYield.neverSettled agrees with that count on every run that has a result.json.
The last non-progress event of each is teardown-unconfirmed (143) or reconciled (38).

What those nodes kept of their decisions:

Retained turn records	Nodes
No turn dispatched	49
Every dispatched turn has an execution-result output blob	15
Some dispatched turns retained	9
None retained	108

So for 117 nodes the only record of what they did is progress counts: the in-flight turn's provider events were held by the controller and dropped when the run settled, and no native session was captured either.
Example: ft2-frontier-focused-v1-hadamard-open-order-lead-http-on-no-res-reflect-restart2, children :s0 and :s1: one dispatched turn each, 269 and 226 progress events, then reconciled and teardown-unconfirmed, no execution-result.

A fix that closes this needs the release path to persist what the child did before it gives the box up: the in-flight turn's provider events as that turn's output, and the harness session capture, the same two things a settled leaf records.
Coverage report with per-wave numbers: tangle-network/discovery-lab reports/fleet-observability/coverage.md.

[drewstone]

State at 23:30Z: #1376 is merged (f2dced1). It is not released: npm latest is 0.266.0, and main declares 0.268.0.

What #1376 closes:

• Under retainedAtSettlement: 'release', the barrier now closes every retained slot after the retry window, confirmed or not.
• An unconfirmed slot closes marked retainedExecution: 'release-unconfirmed'. Its teardown-unconfirmed record still names what a sweeper must delete.
• FleetYield.releaseUnconfirmed counts these slots, and the spend gap is the committed floor.
• In the Lab fleet records of 2026-09-23/24, this covers all 338 never-settled agents: 179 never had an environment named, 9 were never dispatched, and 150 were dispatched.

What stays open here: the release path still destroys a live retained box without reading the child's harness session first. The in-flight turn's provider events are not persisted either. For the 150 dispatched children, these two records are what is lost.

No fleet run has used #1376 yet, because the router refuses flash.