[fctl-sh]

In #46 a host stopped serving a session for good: the transcript froze on a spinning tool call while the agent kept working, and ten minutes later the fleet stopped the session. Whatever set it off, the host had no way out once Docker stopped answering. Every docker … call was awaited without a deadline, several of them on the control connection itself (hello() probed every driver and ran docker info; the hourly sweep and the cleanup of sandboxes the control plane asked to stop ran inside the serve loop), and nothing noticed an agent link that went quiet or a docker exec that never reached the sandbox.

This is the host's side of #46, items 2 to 5 of its checklist.

Driver commands time out. Every command a driver runs is killed after a minute, or after 30 minutes for image pulls, docker run (which may pull) and Lima's clone, start and stop, and the host logs which one: WARN `docker ps -a …` did not finish within 60 seconds, so the host stopped it. Probes report "the docker daemon is not answering". A container whose state can't be read now fails the session after a minute, instead of the host going on to wait 30 minutes for docker run.

The control connection never waits on a driver. The hello reports driver probes and sandbox limits that the host measures at start and every five minutes in the background. A Docker daemon that doesn't answer leaves the limits as they were. The hourly sweep and the stopping of leftover sandboxes run as chores in tasks of their own, and a session that starts in a sandbox a chore holds waits for it. Heartbeats and pongs keep flowing, and a message that can't be sent waits for the next connection instead of being dropped.

Agent links are checked (crates/fctl-host/src/liveness.rs). The agent must say hello within 30 s of attaching, and the host pings it every 15 s (every fctl-agent answers pings). A link with no frame for a minute is dropped, which kills its exec. The host then attaches again from a task of its own, waiting 1 s, then twice as long each time up to 30 s, and starting fctl-agent again first. A runtime that hangs is retried for as long as it takes. Attaches that fail outright (the exec exits, the sandbox is gone) fail the session after two minutes, as five failed attaches did before. Deliveries the agent hasn't confirmed go over the new link again (the agent ignores ones it has), so a message sent during a stall isn't lost until the uplink happens to reconnect.

Both sides log the link. The host logs each attach with the agent's journal position, each new attempt with its reason, and the exec's stderr. fctl-agent logs host links attaching, replacing each other and detaching, and what it replays from which agent_seq.

Also:

• The host listens for shutdown signals from the start. Before, it listened only inside its select loops, so a SIGTERM (fctl host restart, fctl host uninstall) that arrived while the daemon awaited something else was lost, and launchd killed the host 20 s later.
• A host that starts while its driver doesn't answer waits to connect until it can list its sandboxes. Connecting with an empty list would make the fleet stop sessions whose agents may still run.
• A stop now cancels a sandbox that is still being prepared ("Stopped before it started").
• fctl-agent announces host connections in the order it accepts them. Each connection used to announce itself from its own task, so a short-lived probe (fctl-agent serve checking whether a supervisor already runs, which the host now does before attaching again) could replace the link the host had just opened. Older agents still recover: the host attaches again when the link closes.

Not here: item 1 (the trigger; what the code tells about it is in #46), item 6 (showing a stall on the session page, a control-plane and web change), and item 7 and the disk questions, which wait on decisions. Draining a host is #56.

Compatibility

Hosts and agents only, with no protocol change. A new host works with existing sandboxes, since every fctl-agent answers Ping. The supervisor's logging and connection order come with a new fctl-agent, which a sandbox gets when its agent next starts.

#39 also changes ensure() in driver/container.rs and hello() in daemon.rs; whichever merges second needs a small rebase.

How I verified it

• Unit tests: a hanging command is killed and its error says which (driver::tests); link health and the waits between attempts (liveness::tests).
• cargo xtask check --quick.
• Against the control plane of cargo xtask dev, with two hosts of this build:
  • Docker hangs. FCTL_DOCKER_BIN pointed at a stand-in that answers like an empty Docker until a flag file makes every command hang.
    • The runner stayed online, last_seen_at advancing every 15 s, while a session's docker inspect hung until the host killed it after 60 s.
    • Dropping the control connection (disabling and re-enabling the runner) got the host registered again 12 ms after it reconnected. The old hello() would have waited on docker version indefinitely.
    • SIGTERM ended the host in 0.1 s while a session's docker run hung.
    • A host started while Docker hung logged the commands that timed out, and connected 10 s after Docker answered again.
    • A new session failed after a minute, with the docker inspect timeout as its reason. One stopped during a hung prepare ended "Stopped before it started".
  • A frozen agent. On the process driver, I stopped a session's fctl-agent with SIGSTOP and sent the session a message. The host dropped the link after a minute of silence, and the next attach timed out on its hello. Once the agent resumed, the host attached again, the journal replayed from the last acknowledged event, and the message was answered once.

Refs #46

Session: https://app.fctl.sh/s/ses_01M56G8AYJ4ECHDXFSXZW31XVP