Repository navigation
feat(host): run sessions in Firecracker microVMs on Linux hosts - #344
Conversation
fctl-agent becomes the VM's init (crates/fctl-agent/src/vm): it lays the image's filesystem under the sandbox's own disk, runs what the host asks over the vsock as docker exec would, and bridges the VM's network to the host's passt. The host pulls images from their registries and unpacks them in a short-lived VM, boots each sandbox from Firecracker 1.17.0, which it downloads, and keeps VMs running across its restarts. Spawned programs come back as a Running, which the VMs' programs, not child processes of the host's, can be too.
…with-firecracker-microvms-su
…lours out of VM consoles Also refuse to build images with the firecracker driver before downloading the image's sources, and list the driver in fctl host install's help.
…18.59 ADR 0069 records why Linux hosts get Firecracker microVMs, and how: no root, the same images as Docker, unpacked in a VM, a disk per sandbox over its image's filesystem, fctl-agent as the VM's init, passt for the network, and a kernel that hyperstructure/sandbox publishes.
|
Partly addressed as of fctl reviewed Large, well-documented change. The findings below are from reading the code; the VM paths can't be exercised in CI. Also worth knowing before merge: the default kernel image On lines the diff doesn't show:
Reviewed by fctl · Review agent settings |
- fctl host install's auto never picks Firecracker, as the daemon's doesn't; status, doctor and sessions ask whether a driver builds images rather than whether it uses them. - A VM's start stops what an earlier boot left (a passt whose VM powered off) and what it started when it fails. - Requests to a VM's init are time-limited, so a stuck VM holds up no stop or remove of its sandbox. - Sandbox records are written whole, and a disk without one is never made again over. - A file lock guards the image store's names and removals across processes (the host, and fctl host image pull). - Layer downloads sign in again when their token runs out. - The VM's init answers only the host on its vsock, and the host tells a kernel thread that took a VM's process id from a VM that is starting.
|
On the review's 7th finding ( |
This comment has been minimized.
This comment has been minimized.
A make that stopped halfway, the host stopping or the session's start called off, left a disk without a record, which blocked the sandbox for good. The record now comes first, the disk is made under another name and then put in place, and a recorded sandbox without a disk gets one.
This comment has been minimized.
This comment has been minimized.
main took ADR 0069, so the Firecracker ADR is 0070.
This comment has been minimized.
This comment has been minimized.
… unpacking failed Two pulls of one image, by two names or from two processes, shared a directory and could record a half-unpacked filesystem; now each has its own, the second keeps the image the first brought, and what pulls that stopped halfway left goes after two days. When the VM fails to unpack a layer, its reply is the error rather than the write that failed after it, and the whole upload has a time limit.
## Summary fctl's new `firecracker` driver (the fctl pull request below) boots each sandbox VM with a kernel published from here: Linux 6.18.59, the long-term release, configured with Firecracker 1.17's guest configuration for 6.18 plus `/proc/config.gz`, everything built in. `kernel/build.sh` builds it for the machine it runs on, and `kernel/check.sh` boots it in Firecracker and checks it reaches userspace with ext4, overlayfs, vsock and TAP devices, which fctl-agent's init needs. The new Kernel workflow builds it for amd64 and arm64 on their own runners, boots it where the runner has KVM, and on `main` publishes it as `ghcr.io/hyperstructure/sandbox-kernel:6.18.59` and `:6.18`, an image whose `/vmlinux` is the kernel. Pull requests build and check it without publishing. The sandbox image itself doesn't change. I built 6.18.59 for arm64 with `kernel/build.sh` and booted it in Firecracker nested in a VM (an fctl session has no KVM of its own): `kernel/check.sh` passed, and so did a whole fctl sandbox on it (network, programs, fctl-agent, stop and restart). This pull request's Kernel workflow built both architectures with the Dockerfile and booted the amd64 build in Firecracker on its runner's KVM; the arm64 runner has none. `kernel/check.sh` mounts a drive as ext4, lays an overlay on it and checks vsock's transport and TAP devices, and fails when one is missing (tried with a device no kernel has). A published version is never built again: a change to the build takes a new version (`6.18.59-2`). ShellCheck and actionlint pass. ## Merge order Merge this before the fctl pull request: the driver pulls `sandbox-kernel:6.18.59` by default, which this publishes. - hyperstructure/fctl#344 - hyperstructure/fctl-site#95 ## Worth confirming - **Decision:** the kernel is an image of its own, tagged by Linux version and pinned by each fctl release, rather than a file in the sandbox image or a release download; a new kernel takes a pull request here and one in fctl. - **Decision:** upstream Linux from kernel.org with Firecracker's configuration, rather than Firecracker's CI kernels (Amazon Linux builds, which its S3 bucket keeps for less than a year). - **Assumption:** GitHub's amd64 runners have `/dev/kvm`, which the workflow opens to the runner's user; the arm64 runners likely don't, and the check says so and boots nothing there. [fctl session](https://app.fctl.sh/s/ses_01M64PV6RKNWY1BK8M26YC1TPP) <!-- FCTL_REVIEW_SUMMARY --> --- > [!NOTE] > **Medium risk.** It adds a workflow that publishes a kernel image hosts will boot, and a published tag never moves, so a bad build or wrong tag stays; the sandbox image itself doesn't change. > > **What it does.** Adds a `kernel/` directory that builds the Linux 6.18.59 kernel fctl's Firecracker sandboxes boot, using Firecracker's guest configuration, and a `check.sh` that boots it in Firecracker with a drive, an overlay, vsock and a TAP device. A new Kernel workflow builds it for amd64 and arm64, boots it where the runner has KVM, and publishes it from `main` as `ghcr.io/hyperstructure/sandbox-kernel`. A published version is never built again, so a change to the build takes a new version in `kernel/VERSION`. > > **Tags never move.** The first merge to `main` publishes `6.18.59` for good, and fctl should pin it only after it exists. The arm64 build is not booted in CI, as its runners have no KVM. > > <sub>[fctl review](https://app.fctl.sh/s/ses_01M64Y4AF63NTT9P12ACQEKN8Z)'s first look at [`62c4034`](62c4034) · 14 files, +8009 −8 · [Review agent settings](https://app.fctl.sh/p/fctl?tab=reviews)</sub> <!-- /FCTL_REVIEW_SUMMARY --> --------- Co-authored-by: fctl-sh[bot] <fctl-sh[bot]@users.noreply.github.com>
…rs (#95) ## Summary fctl's new `firecracker` driver (the fctl pull request below) runs each session in a Firecracker microVM with a kernel of its own on Linux hosts with KVM. The landing page now names Firecracker in the strip of what fctl works with, in step 01, in the runners card and in "How the pieces fit" (a Linux box: Firecracker or Docker), and the features overview's Sandboxes card says it too. The sandboxes page names it in its description, hero and points, and gets a Firecracker card beside Apple container and Docker, which says the driver is experimental, builds no images and has no package caches; the page's sentences on builds and caches now say they are Apple container's and Docker's. The driver cards sit three in a row where they fit; at two to a row, the odd one out takes its row. I checked with `npm run check` and a full build against fctl's branch (`FCTL_DOCS_DIR`), whose metadata and link checks pass. No heading or scene changed, so the link preview images stay as they are. ## Merge order Merge with or after fctl's pull request: until then, this describes a driver that no release has. - hyperstructure/fctl#344 - hyperstructure/sandbox#12 ## Screenshots At 02177d5 for the sandboxes page and a6556e6 for the landing page, 1440 px wide unless it says otherwise. The sandboxes page's drivers, light: | Before | After | |---|---| |  |  | At 1000 px, where Docker's card takes its row, and at 390 px, dark: | 1000 px | 390 px | |---|---| |  |  | The landing page's step 01, dark: | Before | After | |---|---| |  |  | "How the pieces fit", the runners card and the strip under the hero, light: | Before | After | |---|---| |  |  | |  |  | |  |  | ## Worth confirming - **Decision:** the Firecracker card calls the driver experimental, as fctl's roadmap does, and says Firecracker is "the virtual machine monitor AWS Lambda runs on" and that "nothing on the host runs as root". The landing page lists it without that word. - **Decision:** Lima stays unmentioned on the site, as before; the pages name Apple container, Firecracker and Docker. [fctl session](https://app.fctl.sh/s/ses_01M64PV6RKNWY1BK8M26YC1TPP) <!-- FCTL_REVIEW_SUMMARY --> --- > [!NOTE] > **Low risk.** Copy changes to the landing page and the sandboxes page, plus a small CSS rule for a third card, with no logic, routes or stored data touched. > > **What it does.** Adds Firecracker microVMs to the pages that describe sandboxes: a new driver card on the sandboxes page marked experimental, and mentions on the landing page, the features index and the page's meta descriptions. Claims about Dockerfile builds and package caches are now limited to Apple container and Docker runners, since Firecracker builds no images and keeps no caches. A CSS rule lays the driver cards out three to a row on wide screens. > > **Depends on fctl#344.** The copy describes the Firecracker driver from hyperstructure/fctl#344, so it should merge with or after that change. > > <sub>[fctl review](https://app.fctl.sh/s/ses_01M64XAS341Z5ZMJB7PPC47AB2)'s first look at [`02177d5`](02177d5) · 4 files, +45 −20 · [Review agent settings](https://app.fctl.sh/p/fctl?tab=reviews)</sub> <!-- /FCTL_REVIEW_SUMMARY --> --------- Co-authored-by: fctl-sh[bot] <fctl-sh[bot]@users.noreply.github.com>
Summary
Linux hosts can now run each session in a Firecracker microVM with a kernel of its own (
fctl host install --driver firecracker), from the same images as Docker and with nothing running as root: the host's user needs/dev/kvm(thekvmgroup),passtand e2fsprogs, and the host downloads the rest itself. ADR 0070 records the design anddocs/architecture.md#firecrackerthe details:docker login's credentials) and unpacks the layers in a short-lived VM, as root there, so owners and setuid bits survive and no image file touches the host's filesystem. Sandboxes share their image's filesystem read only, each with its own sparse ext4 disk laid over it; stopping keeps the disk, resuming boots from it.crates/fctl-agent/src/vm): it switches to that root, sets the system up, reaps, and runs what the host asks over the vsock asdocker execwould (fctl_proto::vm).passtper VM on the host, with no bridge, TAP or firewall rule on the host.ghcr.io/hyperstructure/sandbox-kernel:6.18.59from the linked sandbox pull request. Firecracker and passt run in their own process groups, so restarting or updating the host leaves sessions running.Driver::spawnnow returns pipes and an exit code (Running) rather than a child process, since a VM's programs aren't the host's children; Claude sign-in works as before. What the driver leaves for later (package caches, builds, the jailer, snapshots, CI with KVM) is in #345.Unit tests cover the protocol, the init (whiteouts, users, netlink), the registry client, the image store and the VM configuration. CI has no KVM, so I verified on Firecracker nested in an aarch64 VM under QEMU here: the opt-in
cargo test -p fctl-host boots_a_sandbox -- --ignored(boot, user and hostname, DNS and HTTP out, piped programs, fctl-agent, stop and restart from disk), and sessions end to end fromcargo xtask devon a Firecracker runner: start, follow-ups, stop, resume from the disk, and a host restart with the VM running, which it recovered. I pulled images from Docker Hub, ghcr.io and a local registry, and booted the real default image (ghcr.io/hyperstructure/sandbox:latest): Claude Code,ghand Node run asagent, andnprocandfreeshow the VM's size. I booted both Firecracker CI's kernel and the sandbox pull request's 6.18.59, whose Kernel workflow also booted the x86_64 build on a GitHub runner with KVM. Not tried: the driver on an x86_64 host, an Ubuntu host (its AppArmor profile for passt allows these socket paths, by reading it), and Claude sign-in.Merge order
Merge the sandbox pull request first: its Kernel workflow publishes the kernel this driver pulls by default. The fctl-site pull request shows the driver on fctl.sh's landing and sandboxes pages; merge it with or after this one.
Worth confirming
--driver autonever picks Firecracker: a host's owner chooses it, so updating fctl changes no host's driver.allowed_builds) with this driver for now; projects' own images run.[firecracker] disk_gbinhost.toml), which takes what the session writes and gives back what it deletes.kvmand installingpasstand e2fsprogs is setup enough for "seamless"; the host downloads Firecracker and the kernel itself.fctl session
Note
Medium risk. It adds a whole new sandbox driver that downloads and runs binaries and a kernel, unpacks images as root inside VMs, and changes the shared
Driver::spawninterface the Docker driver also uses; it is opt-in, so existing hosts keep working.What it does. Adds an opt-in Firecracker driver so Linux hosts can run each session in its own microVM instead of a Docker container, from the same images.
fctl-agentdoubles as the VM's init and runs programs over a vsock, with networking carried throughpasstand nothing running as root on the host.Driver::spawnnow returns pipes and an exit code rather than a child process, and the docs, an ADR and thefctl hostcommands are updated to match.Merge order and untested paths. The host pulls the kernel image
ghcr.io/hyperstructure/sandbox-kernel:6.18.59, which hyperstructure/sandbox#12 publishes, so that pull request must merge first. CI has no KVM, so the VM paths were only tried by hand on aarch64.fctl review's first look at
32be81c· 37 files, +6005 −82 · Review agent settings