A stalled host freezes transcripts and stops sessions whose agents are still working

What is wrong

In fctl session #12 (https://app.fctl.sh/s/ses_01M56BTAZM8R6TT2QC6HXQV6WX), on runner now (Docker driver, OrbStack), the transcript on app.fctl.sh froze at 02:25:58 UTC on 2026-10-18, on a tool call that kept spinning as if it were running. The agent noticed nothing. It worked for another half hour, opened #44, hyperstructure/design#4 and hyperstructure/fctl-site#5 and pushed to all three, and none of it reached the app. Later the session said "Runner now has been offline for 10 minutes. Resume the session to continue elsewhere." and was Stopped. The person's next message still reached the agent, but not through fctl: the host was not attached, so it most likely came over Remote Control.

Until the runner was expired, nothing told the person that the transcript had stopped.

What happened

Seen from inside the sandbox; times are UTC on 2026-10-18.

Time	What
02:00:30	fctl-agent starts (`~/.fctl/agent.log`).
02:25:26–02:26:27	The agent links fctl's debug binaries with 4 build jobs (`cargo build -p fctl -p fctl-agent -p fake-claude` for `cargo xtask dev`), writing about 250 MB.
02:25:54	Event 1343, the last one the host acknowledged: `~/.fctl/state/journal.ack` holds 1343 and was last written at 02:25:54.
02:25:58	Event 1344 (a 428-byte `task_started`) is never acknowledged, nor is anything after it. The events right after it are ordinary (300 B–3 KB).
before 02:51	The fleet has stopped the session: "Runner now has been offline for 10 minutes".
02:58	Still nothing acknowledged (the journal is at event 2735), and there is no `fctl-agent attach` process in the sandbox: the host never attached again.

Also checked in the sandbox:

The container, fctl-agent and claude ran without interruption (container up since 02:00, same processes throughout).
Nothing ran out of memory: the cgroup's memory.events shows oom_kill 0 (peak 7.67 of 8 GiB). The OrbStack VM has 16 GB of RAM and 17 GB of swap, with practically no swap in use at 02:52.
Not an oversized event: events of up to 1.3 MB (screenshots) went through before the stall, and MAX_FRAME_BYTES is 64 MiB.
Not the local stack the agent was starting: the fctl host that cargo xtask dev runs uses its own sockets and state (/tmp/fctl-fctl/…, .fctl-dev/) and only started at about 02:37.
Disk: the Docker VM's shared disk went from 15 of 43 GB used (35%) at 02:05 to 52 of 57 GB (93%, 4.3 GB free) at 02:52. This sandbox accounts for about 8 GB of it (5.7 GB is Rust's target/); other sandboxes wrote the rest. Another session (ses_01M56CE8ANDXBDMB930CSXN186, started 02:11, the one behind cargo xtask dev fails where worker-build isn't installed #41) ran cargo xtask dev and Chromium at the same time, probably on the same runner.

The session's agent link, its acknowledgements (from its uplink) and the runner's control connection all failed at about the same time. So the host process as a whole stalled, crashed or lost Docker; no single session event broke the link.

The sandbox can't show what the host did. ~/Library/Logs/fctl/host.log on the runner, 02:20–02:50 UTC should, by its lines:

fctl host 0.2.0 on now using the docker driver …: the host started again, so it crashed or was killed;
control connection: … or cannot reach …: it lost the control plane;
listing sandboxes: … or attaching to fctl-ses-01m56btazm8r6tt2qc6hxqv6wx: …: Docker calls failed;
removed … stopped for more than … hours: the hourly sweep ran. Its first run is an hour after the host starts (interval_at(now + SWEEP_EVERY)), so a host restarted around the v0.2.0 release (01:26) swept at about 02:26.

Worth recording with it: the Mac's free disk (df -h /), docker system df, and whether docker ps answered promptly at the time.

How it works now

Why a stalled host turns into a frozen transcript and a stopped session (permalinks at 3b70a94):

Docker calls have no timeout. run_cmd awaits docker … with no deadline, and so does the docker exec -i child behind each agent link (attach, spawn). One hung Docker call hangs whatever awaits it.
The control connection waits on Docker. Daemon::serve awaits sweep_stopped() inside its select! loop (L244). That runs docker ps -a, then docker inspect and docker rm -f for each expired sandbox, one after another. Every (re)connect first awaits hello(), which runs Driver::probe_all() and refresh_sandbox_limits(). While those run, the host sends no heartbeats and answers no pings, and if they hang, the runner never comes back. recover() lists sandboxes the same way at startup.
Nothing checks that the agent link is alive. ContainerDriver::attach returns as soon as docker exec … fctl-agent attach is spawned. The session task (session.rs#L365, AgentLink::open) then waits for frames with no deadline. The protocol has HostToAgent::Ping, and the supervisor answers it (supervisor.rs#L254), but the host never sends one. A link that goes quiet, or an exec that never connects, is never replaced.
Nothing records or shows the stall. The supervisor doesn't log host attaches and detaches (supervisor.rs#L234; agent.log has two lines for this whole session). The host doesn't tell the control plane that a session's agent link is down, and the web app keeps the last tool call spinning.
After 10 minutes the fleet stops the sessions, even when their agents are fine. expire_offline_runners (RUNNER_GRACE_MS, L43) marks every session of a runner away that long as Stopped. When the runner comes back and reports the session as running, reconcile finds it Stopped, which is terminal. It then sends StopSession { reason: Deleted } (L668), ending an agent that kept working throughout. Its unacknowledged events (here 1344 to 2735 and counting) exist only in the sandbox's journal.
Disk isn't watched. Docker sandboxes share the VM's disk with no quota. Heartbeats report disk_available_mb for the host's data directory (sysinfo.rs#L87), not the Docker VM's disk, and scheduling ignores disk. Sandbox size for heavy builds: host limits, per-session sizes and disk #26 described this and was closed without it.
The agent loses GitHub an hour in. The session's GitHub token (~/.fctl/state/github-token, written at 02:00:30) is renewed through the host. With the host away it expired at about 03:00, after which gh and git push failed with "Bad credentials", and the agent couldn't file this issue itself.

To check a stuck session from inside its sandbox: compare cat ~/.fctl/state/journal.ack and its modification time with the last agent_seq in ~/.fctl/state/journal.jsonl, and look for an fctl-agent attach process (ps -eo pid,etime,args).

What should happen

Find the trigger in the runner's host.log (above), add it here, and adjust these items if it shows something else.
Every driver command has a timeout (about a minute; longer for image builds and pulls). When it expires, the command is killed and the host logs which one. (fix: keep runners online and agents reachable when Docker hangs #57, released in v0.4.0)
The control connection never waits on Docker: the sweep runs in its own task, hello() uses driver info and sandbox limits refreshed in the background, and heartbeats and pongs keep flowing whatever the sessions or the sweep are doing. (fix: keep runners online and agents reachable when Docker hangs #57, released in v0.4.0)
The host checks each agent link. It expects the agent's Hello within about 30 s of attaching and sends HostToAgent::Ping every 15 s. A minute without any frame counts as a dead link: the host kills the docker exec child and attaches again, with backoff. (fix: keep runners online and agents reachable when Docker hangs #57, released in v0.4.0)
Both sides log the link at info level: the supervisor on attach and detach (connection id, the agent_seq it replays from), and the host whenever it attaches again and why. (fix: keep runners online and agents reachable when Docker hangs #57, released in v0.4.0)
People can see a stall. While a session's runner is away or its agent link is down, the session page says so ("Runner now hasn't been heard from since 02:26; the agent's work shows up when it's back") instead of a spinning tool call. (feat: show when a session's runner is away or can't reach its agent #69, released in v0.5.0)
A session stopped only because its runner was away isn't killed when the runner comes back with its agent still running (see the first decision below).

Compatibility

Timeouts, the background sweep, link pings and the host's logging need only a host release. The supervisor already answers Ping (fctl-agent 0.2.0), so a new host works with existing sandboxes.
The supervisor's logging needs a new fctl-agent, which a sandbox gets when its agent next starts.
Showing the stall and keeping sessions alive are control-plane and web changes. Hosts already report their sessions and agent_seq in the hello and in heartbeats, so older hosts keep working; any new field, such as "agent link down since", must be optional.

Decisions for a person

When a runner comes back with a session the fleet stopped for its absence, and nobody resumed it elsewhere meanwhile: should the session carry on where it is, or stay stopped and say why? Today it gets StopSession with reason Deleted.
Is 10 minutes the right grace period before a runner's sessions are stopped?
Disk, left over from Sandbox size for heavy builds: host limits, per-session sizes and disk #26: a quota per sandbox, a minimum of free space below which a host takes no new sessions, or both?

In ~/Library/Logs/fctl/host.log on the host there is a line:

2026-10-18T02:25:05.514833Z  WARN control connection: WebSocket protocol error: Connection reset without closing handshake: Connection reset without closing handshake

The fleet marks a runner connected the moment its WebSocket opens, and the 10-minute offline gap points to a stalled Docker-side reconnect command with no timeout — likely in hello(), which probes all three drivers unnecessarily. Might be relevant. This might have happened because the host had fctl host uninstall run to not receive any more runners on the host. Would be good to be able to disable workers from being scheduled on the host, but still serve the available runners, if that was the case.

Session: https://app.fctl.sh/s/ses_01M56BTAZM8R6TT2QC6HXQV6WX

Where the trigger points, from the code

The fleet stops a runner's sessions only after its control WebSocket has been closed for 10 minutes. on_close sets connected = 0, and expire_offline_runners skips any runner with connected = 1. An open socket keeps the runner connected, even one that never sent a hello. So from about 02:26 until after 02:36 the host held no control connection, and it never attached to the sandbox again.
Connection reset without closing handshake is tungstenite's error for a connection that ended without a WebSocket close frame. No deploy of app.fctl.sh ran then: the nearest ran 02:18:57–02:20:08 and 02:50:13–02:51:32. Cloudflare closes WebSockets that carry nothing in either direction for a while (its docs don't say how long; other sources say about 100 s). The host sent a heartbeat every 15 s from its serve loop, but awaited sweep_stopped() and the cleanup in StopSession inside that same loop. So one way to get this error is a loop stuck on a Docker call for that long.
Daemon::run reconnects about a second after that line. Two readings fit what the sandbox showed, and host.log tells them apart:
1. The host stalled on Docker. The log shows connected to https://app.fctl.sh, then no registered as now (…) and nothing after it. hello() hung in Driver::probe_all() or docker info before it sent anything, and the idle socket was closed, so the runner went offline. The session's docker exec died at about 02:25:58, and its replacement never reached the sandbox.
2. The host was stopped. fctl host uninstall boots the launchd job out (SIGTERM, then SIGKILL after 20 s) and removes it, so nothing starts the host again. The exec dies with it, which fits everything seen in the sandbox: nothing acknowledged after 02:25:54, no fctl-agent attach, the runner offline, the session stopped ten minutes later. The log ends with shutting down; sessions keep running in their sandboxes. Or it ends with nothing, if the SIGTERM came while the daemon awaited a Docker call: it listened for signals only inside its select loops, and missed them otherwise.

Could you paste host.log's lines from 02:25:05 to about 02:40? With them, item 1 can say which it was.

Done in #57

#57 is open, with CI green. It covers items 2 to 5, which reach hosts with the next release:

Driver commands time out after a minute (30 for image pulls, docker run and VM boots), and the host logs which one.
The hello uses driver probes and sandbox limits refreshed in the background. The sweep and the cleanups of sandboxes run in tasks of their own, so heartbeats and pongs keep flowing.
Agent links must say hello within 30 s, are pinged every 15 s, and are replaced after a minute of silence. The host attaches again, waiting longer after each attempt, and messages the agent hasn't confirmed go over the new link again.
The host and fctl-agent log every attach, new attempt and replay.
Also: shutdown signals are no longer lost, a host whose driver doesn't answer at start waits to connect rather than report its sessions as gone, and a stop cancels a sandbox that is still being prepared.

I checked these against cargo xtask dev with a stand-in docker that hangs, and with a session's fctl-agent frozen with SIGSTOP. The details are in #57.

What's left

Item 1: the log lines above.
Item 6: today the session page shows a stall only as a dot beside the runner's name in the side panel. The dot is hidden at phone widths, and goes grey only when the session's uplink closes, so in reading 1 it stayed green. The page needs to know when the agent link is down, as an optional field from the host (in HostSessionReport or the session state it sends up), and to say both that and "the runner hasn't been heard from since …" in the transcript. That is the next step; it needs a protocol field, the control plane and the web app.
Item 7, the grace period and disk wait on the decisions in the issue.
Keeping a host's sessions running while it takes no new ones, from the last paragraph, is now Hosts can't stop taking new sessions without dropping the ones they run #56, with its own decisions.

Session: https://app.fctl.sh/s/ses_01M56G8AYJ4ECHDXFSXZW31XVP

fix: keep runners online and agents reachable when Docker hangs (#57)

feat: show when a session's runner is away or can't reach its agent (#69

Items 2 to 5: done

#57 merged and shipped in v0.4.0; I ticked those boxes.

Item 6: #69

#69 is open, with CI green. The session page says when the agent's work stops coming through, and why, at the end of the transcript and in the header, and nothing looks in progress meanwhile (no spinning tool call, no "Working"):

"Runner now hasn't been heard from since 02:26. The agent's work shows up here when it's back, and messages you send reach the agent then." when the session's uplink closed, or sent nothing for a minute: a frozen host, or a laptop that slept, can leave its socket open.
"Runner now hasn't been able to reach the agent in its sandbox since 02:26: nothing came from fctl-agent for a minute. It keeps trying. …" when the host says so. Hosts report it once they have known for 10 seconds that no link to the agent works, in a new optional uplink message.

fctl attach prints the same. A runner back within 10 seconds, such as a host restarting, goes unmentioned. The runner-away notice is all control plane, so it covers released hosts too; the agent-link one comes with the next host release. I checked it against cargo xtask dev with a frozen fctl-agent, a killed host and a frozen host; the details and screenshots are in #69.

Found on the way: #70

Testing #69, resuming the frozen agent lost a tool's result from the transcript for good. After a reconnect, the agent sends new events over its new link before the host's replay of the ones lost with the old link, and SessionDO stored the newer ones, then skipped the replayed older one as already stored. The same can happen when an uplink reconnects during a turn, as in every deploy. #70 (open, CI green) makes SessionDO wait for the replay, for at most a minute.

What's left

Item 1 needs host.log from the runner, 02:25:05 to about 02:40, to tell the two readings in my earlier comment apart.
Item 7, the grace period and disk wait on the decisions in the issue. Suggestions, for whoever decides:
1. A runner that comes back with a session the fleet stopped for its absence: carry on where it is. Since feat: stop idle sessions after ten minutes and resume them for a message #67 a stopped session stays with its machine and a message resumes it there. One stopped only because its runner was away could come back the same way when the runner does, with a notice saying so, instead of getting StopSession with reason Deleted. One resumed elsewhere meanwhile would still be stopped, with a reason that says why.
2. The grace period: 10 minutes seems right once the first is in. The page now says at once that the runner is away, and a session stopped for it comes back with its runner.
3. Disk: first a minimum of free space below which a host takes no new sessions. It works with every driver, and heartbeats already carry free disk, although they need to measure the Docker VM's disk rather than the host's data directory. Per-sandbox quotas depend on the driver (Docker's need overlay2 on xfs), so they can come later where a driver supports them.

Session: https://app.fctl.sh/s/ses_01M57HG0XSAWK0CZXYRTVFZBV4

fix(control): don't tell viewers a runner is back when the fleet give…

fix(control): keep a session running elsewhere when its old run ends

Item 6: done

#69 shipped in v0.5.0, and app.fctl.sh runs it. I checked it again on today's main with cargo xtask dev: with the host killed mid-turn, fctl attach said ◌ runner dev-host is away (last heard from just now); the agent's work shows up here when it's back ten seconds later. I ticked it.

Item 7 and the decisions: #94

#94 is open. Since you asked for what was left here to be done, it takes the answers suggested above, and ADR 0020 records them. Say if you want any of them otherwise:

A session stopped only because its runner was away carries on when the runner comes back with it, unless it has run elsewhere since, or someone stopped or archived it, or its project was deleted. FleetDO remembers the sessions it stops this way. When the runner's hello reports one running, FleetDO takes it back, the host's uplink replays what the agent did meanwhile, and the transcript says the runner is back. A session resumed meanwhile that no runner took carries on too, and the agent gets the message.
The grace period stays at ten minutes.
Disk was settled meanwhile by feat(host): make room when the disk runs low #93 (ADR 0019): a floor of free disk, measured where the sandboxes are (in Docker's VM on macOS, as here), below which a host takes no new sessions. Quotas per sandbox are left out.

Found on the way, fixed in the same pull request:

When a runner came back after its session had moved to another runner, the end of its old run marked the session stopped there.
fctl attach said "runner … is back" just as the fleet gave up on the runner.

What's left

Item 1 still needs host.log from now, 02:25:05 to about 02:40, to say which of the two readings in my first comment it was. Both are handled now: fix: keep runners online and agents reachable when Docker hangs #57 for a host stalled on Docker, and feat: drain runners so they take no new sessions #74 and feat: ask before fctl host uninstall stops the sessions on a host #75 for fctl host uninstall, which now asks before it stops a host's sessions, or drains the host first with --wait. So it would only settle the record. If you're fine to drop it, A stalled host freezes transcripts and stops sessions whose agents are still working #46 can close once fix(control): let sessions carry on when their runner comes back #94 is merged.
Item 7 gets its tick once fix(control): let sessions carry on when their runner comes back #94 is merged and checked.

Session: https://app.fctl.sh/s/ses_01M585PP4R26Y6BKA64816JD1W

fix(control): let sessions carry on when their runner comes back (#94)

Navigation Menu

Repository navigation

A stalled host freezes transcripts and stops sessions whose agents are still working #46

What is wrong

What happened

How it works now

What should happen

Compatibility

Decisions for a person

Where the trigger points, from the code

Done in #57

What's left

Items 2 to 5: done

Item 6: #69

Found on the way: #70

What's left

Item 6: done

Item 7 and the decisions: #94

What's left

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A stalled host freezes transcripts and stops sessions whose agents are still working #46

Description

What is wrong

What happened

How it works now

What should happen

Compatibility

Decisions for a person

Activity

fctl-sh commented on Oct 18, 2026

Where the trigger points, from the code

Done in #57

What's left

fctl-sh commented on Oct 18, 2026

Items 2 to 5: done

Item 6: #69

Found on the way: #70

What's left

fctl-sh commented on Oct 18, 2026

Item 6: done

Item 7 and the decisions: #94

What's left

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions