Skip to content

feat(host): run sessions in Firecracker microVMs on Linux hosts - #344

Merged
kalaspuff merged 9 commits into
mainfrom
fctl/536-linux-hosts-with-firecracker-microvms-su
Oct 29, 2026
Merged

kalaspuff merged 9 commits into
mainfrom
fctl/536-linux-hosts-with-firecracker-microvms-su

Conversation

@fctl-sh

@fctl-sh fctl-sh Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Linux hosts can now run each session in a Firecracker microVM with a kernel of its own (fctl host install --driver firecracker), from the same images as Docker and with nothing running as root: the host's user needs /dev/kvm (the kvm group), passt and e2fsprogs, and the host downloads the rest itself. ADR 0070 records the design and docs/architecture.md#firecracker the details:

  • Images: the host pulls them from their registries (digests checked; anonymous or with docker login's credentials) and unpacks the layers in a short-lived VM, as root there, so owners and setuid bits survive and no image file touches the host's filesystem. Sandboxes share their image's filesystem read only, each with its own sparse ext4 disk laid over it; stopping keeps the disk, resuming boots from it.
  • fctl-agent is the VM's init (crates/fctl-agent/src/vm): it switches to that root, sets the system up, reaps, and runs what the host asks over the vsock as docker exec would (fctl_proto::vm).
  • Network: a TAP device inside the VM whose frames fctl-agent carries over the vsock to a passt per VM on the host, with no bridge, TAP or firewall rule on the host.
  • Assets: Firecracker 1.17.0 (pinned SHA-256) from its GitHub release, and the kernel image ghcr.io/hyperstructure/sandbox-kernel:6.18.59 from the linked sandbox pull request. Firecracker and passt run in their own process groups, so restarting or updating the host leaves sessions running.

Driver::spawn now returns pipes and an exit code (Running) rather than a child process, since a VM's programs aren't the host's children; Claude sign-in works as before. What the driver leaves for later (package caches, builds, the jailer, snapshots, CI with KVM) is in #345.

Unit tests cover the protocol, the init (whiteouts, users, netlink), the registry client, the image store and the VM configuration. CI has no KVM, so I verified on Firecracker nested in an aarch64 VM under QEMU here: the opt-in cargo test -p fctl-host boots_a_sandbox -- --ignored (boot, user and hostname, DNS and HTTP out, piped programs, fctl-agent, stop and restart from disk), and sessions end to end from cargo xtask dev on a Firecracker runner: start, follow-ups, stop, resume from the disk, and a host restart with the VM running, which it recovered. I pulled images from Docker Hub, ghcr.io and a local registry, and booted the real default image (ghcr.io/hyperstructure/sandbox:latest): Claude Code, gh and Node run as agent, and nproc and free show the VM's size. I booted both Firecracker CI's kernel and the sandbox pull request's 6.18.59, whose Kernel workflow also booted the x86_64 build on a GitHub runner with KVM. Not tried: the driver on an x86_64 host, an Ubuntu host (its AppArmor profile for passt allows these socket paths, by reading it), and Claude sign-in.

Merge order

Merge the sandbox pull request first: its Kernel workflow publishes the kernel this driver pulls by default. The fctl-site pull request shows the driver on fctl.sh's landing and sandboxes pages; merge it with or after this one.

Worth confirming

  • Decision: --driver auto never picks Firecracker: a host's owner chooses it, so updating fctl changes no host's driver.
  • Decision: Firecracker runs as the host's user, without its jailer (which needs root), so a VM escape would land as that user, as a container escape can with Docker. The security page says so.
  • Decision: no package caches and no image builds (allowed_builds) with this driver for now; projects' own images run.
  • Decision: each sandbox gets a sparse 64 GiB disk ([firecracker] disk_gb in host.toml), which takes what the session writes and gives back what it deletes.
  • Decision: VMs reach the internet and the host's network as Docker containers do, but not each other or the host's loopback.
  • Assumption: joining kvm and installing passt and e2fsprogs is setup enough for "seamless"; the host downloads Firecracker and the kernel itself.

fctl session


Note

Medium risk. It adds a whole new sandbox driver that downloads and runs binaries and a kernel, unpacks images as root inside VMs, and changes the shared Driver::spawn interface the Docker driver also uses; it is opt-in, so existing hosts keep working.

What it does. Adds an opt-in Firecracker driver so Linux hosts can run each session in its own microVM instead of a Docker container, from the same images. fctl-agent doubles as the VM's init and runs programs over a vsock, with networking carried through passt and nothing running as root on the host. Driver::spawn now returns pipes and an exit code rather than a child process, and the docs, an ADR and the fctl host commands are updated to match.

Merge order and untested paths. The host pulls the kernel image ghcr.io/hyperstructure/sandbox-kernel:6.18.59, which hyperstructure/sandbox#12 publishes, so that pull request must merge first. CI has no KVM, so the VM paths were only tried by hand on aarch64.

fctl review's first look at 32be81c · 37 files, +6005 −82 · Review agent settings

fctl-sh Bot added 4 commits October 29, 2026 21:51
fctl-agent becomes the VM's init (crates/fctl-agent/src/vm): it lays the
image's filesystem under the sandbox's own disk, runs what the host asks
over the vsock as docker exec would, and bridges the VM's network to the
host's passt. The host pulls images from their registries and unpacks
them in a short-lived VM, boots each sandbox from Firecracker 1.17.0,
which it downloads, and keeps VMs running across its restarts.

Spawned programs come back as a Running, which the VMs' programs, not
child processes of the host's, can be too.
…lours out of VM consoles

Also refuse to build images with the firecracker driver before downloading
the image's sources, and list the driver in fctl host install's help.
…18.59

ADR 0069 records why Linux hosts get Firecracker microVMs, and how: no
root, the same images as Docker, unpacked in a VM, a disk per sandbox
over its image's filesystem, fctl-agent as the VM's init, passt for the
network, and a kernel that hyperstructure/sandbox publishes.
@fctl-review

fctl-review Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Partly addressed as of 4a7fed9. Later commits fixed 6 of the 7 issues this review found.


fctl reviewed 501c1b6 and found 7 issues.

Large, well-documented change. The findings below are from reading the code; the VM paths can't be exercised in CI. Also worth knowing before merge: the default kernel image ghcr.io/hyperstructure/sandbox-kernel:6.18.59 is published only once hyperstructure/sandbox#12 is merged, and it is pinned by a mutable tag while Firecracker itself is pinned by SHA-256.

On lines the diff doesn't show:

  • host status reports builds as allowed on a Firecracker host · Low severity · crates/fctl/src/cmd/host.rs line 418: uses_images() is now true for Firecracker, so cfg.images(d.uses_images()) here shows builds as permitted, while the daemon uses builds_images() (session.rs:89) and allows none. The same mixup lets session.rs:658 (ImageWant::Build(_) if ctx.driver.uses_images()) take the build path, which ends in a bail. Use builds_images() where the meaning is builds.

Reviewed by fctl · Review agent settings

Comment thread crates/fctl/src/cmd/host.rs Resolved
- fctl host install's auto never picks Firecracker, as the daemon's
  doesn't; status, doctor and sessions ask whether a driver builds images
  rather than whether it uses them.
- A VM's start stops what an earlier boot left (a passt whose VM powered
  off) and what it started when it fails.
- Requests to a VM's init are time-limited, so a stuck VM holds up no
  stop or remove of its sandbox.
- Sandbox records are written whole, and a disk without one is never
  made again over.
- A file lock guards the image store's names and removals across
  processes (the host, and fctl host image pull).
- Layer downloads sign in again when their token runs out.
- The VM's init answers only the host on its vsock, and the host tells a
  kernel thread that took a VM's process id from a VM that is starting.
@fctl-sh

fctl-sh Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

On the review's 7th finding (host status and the session path using uses_images() where builds are meant): fixed in 4a7fed9. Status, install's summary, doctor's allowed_builds note and the session's build path now ask builds_images(), which only Docker and Apple's container answer yes to.

@fctl-review

This comment has been minimized.

A make that stopped halfway, the host stopping or the session's start
called off, left a disk without a record, which blocked the sandbox for
good. The record now comes first, the disk is made under another name and
then put in place, and a recorded sandbox without a disk gets one.
@fctl-review

This comment has been minimized.

main took ADR 0069, so the Firecracker ADR is 0070.
@fctl-review

This comment has been minimized.

… unpacking failed

Two pulls of one image, by two names or from two processes, shared a
directory and could record a half-unpacked filesystem; now each has its
own, the second keeps the image the first brought, and what pulls that
stopped halfway left goes after two days. When the VM fails to unpack a
layer, its reply is the error rather than the write that failed after it,
and the whole upload has a time limit.
kalaspuff pushed a commit to hyperstructure/sandbox that referenced this pull request Oct 29, 2026
## Summary

fctl's new `firecracker` driver (the fctl pull request below) boots each
sandbox VM with a kernel published from here: Linux 6.18.59, the
long-term release, configured with Firecracker 1.17's guest
configuration for 6.18 plus `/proc/config.gz`, everything built in.
`kernel/build.sh` builds it for the machine it runs on, and
`kernel/check.sh` boots it in Firecracker and checks it reaches
userspace with ext4, overlayfs, vsock and TAP devices, which
fctl-agent's init needs. The new Kernel workflow builds it for amd64 and
arm64 on their own runners, boots it where the runner has KVM, and on
`main` publishes it as `ghcr.io/hyperstructure/sandbox-kernel:6.18.59`
and `:6.18`, an image whose `/vmlinux` is the kernel. Pull requests
build and check it without publishing. The sandbox image itself doesn't
change.

I built 6.18.59 for arm64 with `kernel/build.sh` and booted it in
Firecracker nested in a VM (an fctl session has no KVM of its own):
`kernel/check.sh` passed, and so did a whole fctl sandbox on it
(network, programs, fctl-agent, stop and restart). This pull request's
Kernel workflow built both architectures with the Dockerfile and booted
the amd64 build in Firecracker on its runner's KVM; the arm64 runner has
none. `kernel/check.sh` mounts a drive as ext4, lays an overlay on it
and checks vsock's transport and TAP devices, and fails when one is
missing (tried with a device no kernel has). A published version is
never built again: a change to the build takes a new version
(`6.18.59-2`). ShellCheck and actionlint pass.

## Merge order

Merge this before the fctl pull request: the driver pulls
`sandbox-kernel:6.18.59` by default, which this publishes.

- hyperstructure/fctl#344
- hyperstructure/fctl-site#95

## Worth confirming

- **Decision:** the kernel is an image of its own, tagged by Linux
version and pinned by each fctl release, rather than a file in the
sandbox image or a release download; a new kernel takes a pull request
here and one in fctl.
- **Decision:** upstream Linux from kernel.org with Firecracker's
configuration, rather than Firecracker's CI kernels (Amazon Linux
builds, which its S3 bucket keeps for less than a year).
- **Assumption:** GitHub's amd64 runners have `/dev/kvm`, which the
workflow opens to the runner's user; the arm64 runners likely don't, and
the check says so and boots nothing there.

[fctl session](https://app.fctl.sh/s/ses_01M64PV6RKNWY1BK8M26YC1TPP)

<!-- FCTL_REVIEW_SUMMARY -->
---

> [!NOTE]
> **Medium risk.** It adds a workflow that publishes a kernel image
hosts will boot, and a published tag never moves, so a bad build or
wrong tag stays; the sandbox image itself doesn't change.
>
> **What it does.** Adds a `kernel/` directory that builds the Linux
6.18.59 kernel fctl's Firecracker sandboxes boot, using Firecracker's
guest configuration, and a `check.sh` that boots it in Firecracker with
a drive, an overlay, vsock and a TAP device. A new Kernel workflow
builds it for amd64 and arm64, boots it where the runner has KVM, and
publishes it from `main` as `ghcr.io/hyperstructure/sandbox-kernel`. A
published version is never built again, so a change to the build takes a
new version in `kernel/VERSION`.
>
> **Tags never move.** The first merge to `main` publishes `6.18.59` for
good, and fctl should pin it only after it exists. The arm64 build is
not booted in CI, as its runners have no KVM.
>
> <sub>[fctl
review](https://app.fctl.sh/s/ses_01M64Y4AF63NTT9P12ACQEKN8Z)'s first
look at
[`62c4034`](62c4034)
· 14 files, +8009 −8 · [Review agent
settings](https://app.fctl.sh/p/fctl?tab=reviews)</sub>

<!-- /FCTL_REVIEW_SUMMARY -->

---------

Co-authored-by: fctl-sh[bot] <fctl-sh[bot]@users.noreply.github.com>
@kalaspuff
kalaspuff merged commit 70fb5ff into main Oct 29, 2026
11 checks passed
@kalaspuff
kalaspuff deleted the fctl/536-linux-hosts-with-firecracker-microvms-su branch October 29, 2026 23:11
kalaspuff pushed a commit to hyperstructure/fctl-site that referenced this pull request Oct 29, 2026
…rs (#95)

## Summary

fctl's new `firecracker` driver (the fctl pull request below) runs each
session in a Firecracker microVM with a kernel of its own on Linux hosts
with KVM. The landing page now names Firecracker in the strip of what
fctl works with, in step 01, in the runners card and in "How the pieces
fit" (a Linux box: Firecracker or Docker), and the features overview's
Sandboxes card says it too. The sandboxes page names it in its
description, hero and points, and gets a Firecracker card beside Apple
container and Docker, which says the driver is experimental, builds no
images and has no package caches; the page's sentences on builds and
caches now say they are Apple container's and Docker's. The driver cards
sit three in a row where they fit; at two to a row, the odd one out
takes its row.

I checked with `npm run check` and a full build against fctl's branch
(`FCTL_DOCS_DIR`), whose metadata and link checks pass. No heading or
scene changed, so the link preview images stay as they are.

## Merge order

Merge with or after fctl's pull request: until then, this describes a
driver that no release has.

- hyperstructure/fctl#344
- hyperstructure/sandbox#12

## Screenshots

At 02177d5 for the sandboxes page and a6556e6 for the landing page, 1440
px wide unless it says otherwise.

The sandboxes page's drivers, light:

| Before | After |
|---|---|
| ![Before: two driver cards, Apple container and
Docker](https://app.fctl.sh/api/v1/published/42uCc6JrOl8mAM22GP5GikfLU4h6OA7EvRdgBzEl5Wj/before-drivers-light.png)
| ![After: three driver cards, Apple container, Firecracker and
Docker](https://app.fctl.sh/api/v1/published/JN7S6XSyLePAcmv0ZirkKYNEJEyKUu3N0y218vGo105/after2-drivers-light.png)
|

At 1000 px, where Docker's card takes its row, and at 390 px, dark:

| 1000 px | 390 px |
|---|---|
| ![After at 1000 px: two cards, then Docker across the
row](https://app.fctl.sh/api/v1/published/NTZA23dSem1WSv2vdVSzk0H2aHseMMFuCj8rinMK45P/after2-drivers-1000-light.png)
| ![After at 390 px: the three cards one under another,
dark](https://app.fctl.sh/api/v1/published/uA8f2Pt93NLWOa9pyLQHaWxlDxu79duUb1ABRGLElqx/after2-drivers-390-dark.png)
|

The landing page's step 01, dark:

| Before | After |
|---|---|
| ![Before: Apple's container runtime on a Mac, Docker on
Linux](https://app.fctl.sh/api/v1/published/ByfxR2FWr6bCVq25BSUX9AwewsM9JbYXACFj5ULZNkv/before-steps-dark.png)
| ![After: a Firecracker microVM or a Docker container on
Linux](https://app.fctl.sh/api/v1/published/ppB4aeDqbnC5lLnkCwR6VWKFp4Yb9m3QR4opDJGqM1h/after-steps-dark.png)
|

"How the pieces fit", the runners card and the strip under the hero,
light:

| Before | After |
|---|---|
| ![Before: a Linux box with
Docker](https://app.fctl.sh/api/v1/published/kgN3BzT6KGNLiQPGWW2cKZVBS8a6pKP3RnzlfKqCCDs/before-map-light.png)
| ![After: a Linux box with Firecracker or
Docker](https://app.fctl.sh/api/v1/published/isRTw7A275nwjM1IBKop56c8zbiapoTLNHfGhpa8oJX/after-map-light.png)
|
| ![Before: the runners
card](https://app.fctl.sh/api/v1/published/8PMZ52h9nFVQkv1t5Ob2bCVmT1TRhqyhljPIpCsev4N/before-card-light.png)
| ![After: the runners card names a Firecracker
microVM](https://app.fctl.sh/api/v1/published/xOOLPvzGXCmfC4T2gqodWnTLSXZbCymT2C195sPhFJ2/after-card-light.png)
|
| ![Before: the strip of what fctl works
with](https://app.fctl.sh/api/v1/published/V1UTdA7XCf3H086SynKHJVkrBh1znUeU6jnMWUB8lqt/before-works-light.png)
| ![After: Firecracker between Apple container and
Docker](https://app.fctl.sh/api/v1/published/KrEODzbcbXC2PDkS9LDdR3dKsGHdGoAWZZvQwpLDGQc/after-works-light.png)
|

## Worth confirming

- **Decision:** the Firecracker card calls the driver experimental, as
fctl's roadmap does, and says Firecracker is "the virtual machine
monitor AWS Lambda runs on" and that "nothing on the host runs as root".
The landing page lists it without that word.
- **Decision:** Lima stays unmentioned on the site, as before; the pages
name Apple container, Firecracker and Docker.

[fctl session](https://app.fctl.sh/s/ses_01M64PV6RKNWY1BK8M26YC1TPP)

<!-- FCTL_REVIEW_SUMMARY -->
---

> [!NOTE]
> **Low risk.** Copy changes to the landing page and the sandboxes page,
plus a small CSS rule for a third card, with no logic, routes or stored
data touched.
>
> **What it does.** Adds Firecracker microVMs to the pages that describe
sandboxes: a new driver card on the sandboxes page marked experimental,
and mentions on the landing page, the features index and the page's meta
descriptions. Claims about Dockerfile builds and package caches are now
limited to Apple container and Docker runners, since Firecracker builds
no images and keeps no caches. A CSS rule lays the driver cards out
three to a row on wide screens.
>
> **Depends on fctl#344.** The copy describes the Firecracker driver
from hyperstructure/fctl#344, so it should merge with or after that
change.
>
> <sub>[fctl
review](https://app.fctl.sh/s/ses_01M64XAS341Z5ZMJB7PPC47AB2)'s first
look at
[`02177d5`](02177d5)
· 4 files, +45 −20 · [Review agent
settings](https://app.fctl.sh/p/fctl?tab=reviews)</sub>

<!-- /FCTL_REVIEW_SUMMARY -->

---------

Co-authored-by: fctl-sh[bot] <fctl-sh[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant