Make agent state durable without live bucket writes

[lvwerra]

Why

The mounted /data bucket is an object-backed FUSE filesystem, not a POSIX disk. The current implementation still exposes actively mutated state to failure modes the mount cannot safely support:

• Codex holds a rollout file descriptor open for the lifetime of a resumed session. The mount's streaming writer can keep that entire open epoch in memory until close, so a container restart can lose much more than the nominal sync interval.
• opencode and Hermes were copied as a live SQLite database plus WAL/SHM files. Those files can be observed at different transaction boundaries, so an rsync copy is not necessarily a valid database checkpoint.
• The harnesses use several separate background rsync loops with different layouts and behavior, making shutdown and migration difficult to reason about.

Design

This PR gives every local CLI the same persistence contract:

1. Live state is always local POSIX state. Agent processes mutate only paths under $AM_LOCAL.
2. The bucket contains closed checkpoints. A single checkpoint process publishes state every 15 seconds by default and once more after a graceful shutdown.
3. File trees are incremental. Codex, Claude Code, Gemini CLI, and OpenClaw walk only their local trees, select paths newer than a per-adapter checkpoint marker, and send those paths through rsync --files-from --delay-updates. The hot loop does not recursively enumerate the bucket.
4. SQLite is checkpointed through SQLite. opencode and Hermes use .backup into a local staging database, require PRAGMA quick_check = ok, then upload the closed database. They publish only when the DB or WAL changed and never publish live WAL/SHM companions.
5. Shutdown is ordered. PID 1 forwards termination to the server; the server gives PTYs a bounded chance to exit and close their transcripts; the final checkpoint waits for any periodic checkpoint lock before reading quiet state.
6. Restore fails closed. A fresh container restores durable state before agents start. On an in-container restart, newer valid local state wins. An incomplete restore prevents startup rather than starting a new lineage over partial state.

Harness layout

Harness	Live state	Durable checkpoint	Adapter
Codex	$AM_LOCAL/codex-home	/data/state/codex	Incremental file tree; local SQLite caches excluded
Claude Code	$AM_LOCAL/agent-state/claude	/data/state/claude	Incremental file tree
Gemini CLI	$AM_LOCAL/agent-state/gemini-home/.gemini	/data/state/gemini	Incremental file tree
OpenClaw	$AM_LOCAL/oc-home/.openclaw	/data/state/openclaw-backup	Incremental file tree
opencode	$AM_LOCAL/opencode-share	/data/state/opencode	File tree plus verified online opencode.db backup
Hermes	$AM_LOCAL/hermes	/data/state/hermes	File tree plus verified online state.db backup

The implementation lives in scripts/agent-state.sh; the full rationale and lifecycle are documented in docs/agent-state-checkpoints.md.

Migration and rollback

• The old Codex sessions symlink is removed only at the known local link path; the durable target is never deleted.
• Existing Gemini state is copied into its durable checkpoint and retained as rollback state.
• Existing real opencode and Hermes directories are copied, then renamed to *.pre-agent-state (with a timestamped fallback if that name already exists).
• Existing Codex SQLite remnants remain quarantined. They are disposable caches; rollouts, history, auth, and config remain the recovery sources.
• Migration copies and restore failures stop startup instead of silently continuing with empty state.

Rollback therefore does not require reversing destructive migration steps.

Durability semantics

• Graceful termination takes a final checkpoint.
• An ungraceful container loss is bounded by the checkpoint interval (15 seconds by default), rather than the lifetime of an open rollout.
• This does not claim synchronous per-token durability. Immutable complete-line segments or a manager-side input journal would be the next step if that guarantee is required.
• Deletions are deliberately not propagated by the hot loop. Retention should use an explicit manifest/tombstone process; rsync --delete against a transient local view is too risky.
• Exact Gemini conversation pinning remains separate. This PR makes Gemini state durable but does not use --resume latest, which could attach the wrong pane in a shared folder.

First production deployment

Before restarting the current Space onto this branch, copy and verify the currently open Codex rollout as a closed migration object through the bucket API. Restarting first could repeat the exact open-writer loss mode this PR addresses.

Verification

npm test passes, including the existing libghostty migration/resize suite and new recovery tests that:

• restore every file-backed harness onto local storage;
• preserve newer local files and SQLite databases across hot restarts;
• checkpoint a Codex rollout while another process keeps its FD open;
• kill that writer and reconstruct the rollout from the checkpoint;
• snapshot committed opencode data while a WAL-mode writer remains alive;
• verify idle SQLite databases do not generate bucket writes;
• validate opencode and Hermes snapshots with quick_check;
• prove corrupt live SQLite cannot replace the previous durable snapshot; and
• restore SQLite without stale WAL/SHM state.