Nothing but the client process ever deletes a sandbox, so any client death leaks boxes permanently

[drewstone]

Two sandboxes billed for 103 and 86 hours on our account before a human noticed and deleted them by hand. They are not an accident of one run; they are what this lifecycle design produces whenever a client dies, and the runtime's own source says so.

The design, from the source

supervisor-CFSPj8rx.js around DEFAULT_SANDBOX_IDLE_TIMEOUT_SECONDS = 1800:

Idle means inactivity to Sandbox, which suspends the sandbox (the container stops; the workspace is kept) rather than deleting it. [...] It is a backstop for a process that dies holding an environment; the settlement barrier releases retained environments itself (Executor.releaseRetained). It does nothing on a driver with a create/delete-only lifecycle, which the SDK says skips suspension.

So there are three things that could reclaim a box, and for a fleet driver all three are off:

1. The client's settlement barrier is the only thing that deletes. It runs in the process that created the box.
2. The idle timeout suspends rather than deletes, and the comment states it does nothing for the create/delete-only lifecycle our drivers use.
3. Nothing server-side deletes on a schedule. A 103-hour box is the proof.

The result is a single point of failure with no backstop: the only thing that can delete a sandbox is the process that created it. Every way that process can die is a permanent leak.

The ways it dies, each measured here

• Teardown is never confirmed. Across 8,710 run journals on this host: 1,389 teardown-unconfirmed events against 147 environment-teardown events, touching 392 runs. Nine unconfirmed for every one confirmed. A child whose teardown is unconfirmed never settles and its box is never released; destroying an already-stopped sandbox returns 409, so the receipt says not-destroyed (this is the mechanism in The release sweep has never once closed a retained child's slot: 0 released records in 6,916 runs, and 154 of 601 children never settle #1301).
• The client process is killed. Our supervisors have been killed by a host low-memory sweep (39 in one night), by the laptop sleeping, and most recently by a spend cap: a campaign key hit its $60 budget, every driver settled budget_exhausted, and 34 worker boxes kept running with no supervisor. A budget cap ends supervision, not spending.
• The host is simply elsewhere. Nothing requires the creating process to outlive the box, and nothing notices when it does not.

What would fix it, in order of how much it buys

1. Give a sandbox a server-side maximum lifetime at create, independent of the client. An absolute deadline, not an idle timer, since idle only suspends. A box outliving its declared lifetime is deleted by the platform. This alone makes every leak self-healing.
2. Make the idle backstop delete rather than suspend for the create/delete-only lifecycle, or say plainly in the SDK that no reclamation exists for it so callers stop believing there is one.
3. Stop treating a 409 on an already-stopped sandbox as a failed teardown. A box that is already gone is the outcome teardown wanted; that one mapping turns 1,389 unconfirmed teardowns into confirmations and lets those children settle.
4. Tear down a budget-exhausted run's children before settling its drivers. Exhausting a cap currently removes the only thing that would have cleaned up.

The first is the design fix; the rest are the ones that stop the bleeding without it. Right now every operator of this runtime is one process death away from paying for a box forever, and the only defence anyone has is a human noticing a number.

[drewstone]

Correcting my own number in the issue body before anyone plans against it.

I wrote "1,389 teardown-unconfirmed events against 147 environment-teardown events, touching 392 runs". That counted the same pursuit directory more than once: this host keeps copies of a run under several git worktrees, so a journal with 16 unconfirmed teardowns was counted once per copy.

Deduplicated by runId, keeping the largest copy of each journal, over 654 unique runs on this host:

	events	runs
teardown-unconfirmed	562	101
environment-teardown	147	147

So the ratio is 3.8 to 1, not 9 to 1, and it affects 101 of 654 runs rather than 392.

Two things this sharpens rather than weakens.

It is concentrated, not uniform. The top runs carry 35, 21, 21, 19 and 18 unconfirmed teardowns each, and they are long fleet runs with many children. A run that spawns a handful of children and settles cleanly does not hit it at all: the run I have live right now, six children including two that died, produced zero teardown events of either kind, and every one of its child boxes is gone from the platform. So the healthy path works, and the defect bites hardest exactly where the most boxes are at stake.

The design point in the issue body is unchanged and does not rest on that ratio. The only thing that deletes a sandbox is the process that created it; the idle backstop suspends rather than deletes and the source says it does nothing for the create/delete-only lifecycle; and nothing server-side reclaims on a schedule, which is why two boxes on this account reached 103 and 86 hours. A server-side maximum lifetime at create still fixes the class regardless of how often the client-side path fails.