Repository navigation
feat: build and publish the kernel of fctl's Firecracker sandboxes - #12
Merged
kalaspuff merged 3 commits intoOct 29, 2026
Merged
Conversation
Linux 6.18.59 with Firecracker's guest configuration for 6.18, built for amd64 and arm64 by the Kernel workflow, booted in Firecracker where the runner has KVM, and published as ghcr.io/hyperstructure/sandbox-kernel, an image whose /vmlinux is the kernel. fctl's firecracker driver pulls the version each release pins.
|
Partly addressed as of fctl reviewed The kernel build and the publish-by-digest flow look sound. The two problems that matter are in the workflow: the boot check cannot fail the job, and a later edit under On lines the diff doesn't show:
Reviewed by fctl · Review agent settings |
![fctl-review[bot]](https://avatars.githubusercontent.com/in/5527664?s=60&v=4){kind=small}
…y and vsock - The boot check fails its job: the step runs with pipefail. - A published version is never built again; a change to the build takes a new version (6.18.59-2), and a registry error other than a missing package stops the run. - kernel/check.sh mounts a drive as ext4, lays an overlay on it and checks vsock's virtio transport, not only that the kernel has them. - The README says what fctl.config adds, and the skills know the Kernel workflow and lint its scripts.
Contributor
Author
|
On the review's 5th finding (the skills): fixed in 5584b0b. The commit skill's ShellCheck command now includes |
AGENTS.md keeps both new entries of what fctl relies on: main's on the Image workflow's file name and version input, then the kernel image.
kalaspuff
approved these changes
Oct 29, 2026
kalaspuff
deleted the
fctl/536-linux-hosts-with-firecracker-microvms-su
branch
October 29, 2026 23:10
kalaspuff
pushed a commit
to hyperstructure/fctl
that referenced
this pull request
Oct 29, 2026
## Summary Linux hosts can now run each session in a [Firecracker](https://firecracker-microvm.github.io) microVM with a kernel of its own (`fctl host install --driver firecracker`), from the same images as Docker and with nothing running as root: the host's user needs `/dev/kvm` (the `kvm` group), `passt` and e2fsprogs, and the host downloads the rest itself. ADR 0070 records the design and `docs/architecture.md#firecracker` the details: - **Images:** the host pulls them from their registries (digests checked; anonymous or with `docker login`'s credentials) and unpacks the layers in a short-lived VM, as root there, so owners and setuid bits survive and no image file touches the host's filesystem. Sandboxes share their image's filesystem read only, each with its own sparse ext4 disk laid over it; stopping keeps the disk, resuming boots from it. - **fctl-agent is the VM's init** (`crates/fctl-agent/src/vm`): it switches to that root, sets the system up, reaps, and runs what the host asks over the vsock as `docker exec` would (`fctl_proto::vm`). - **Network:** a TAP device inside the VM whose frames fctl-agent carries over the vsock to a `passt` per VM on the host, with no bridge, TAP or firewall rule on the host. - **Assets:** Firecracker 1.17.0 (pinned SHA-256) from its GitHub release, and the kernel image `ghcr.io/hyperstructure/sandbox-kernel:6.18.59` from the linked sandbox pull request. Firecracker and passt run in their own process groups, so restarting or updating the host leaves sessions running. `Driver::spawn` now returns pipes and an exit code (`Running`) rather than a child process, since a VM's programs aren't the host's children; Claude sign-in works as before. What the driver leaves for later (package caches, builds, the jailer, snapshots, CI with KVM) is in #345. Unit tests cover the protocol, the init (whiteouts, users, netlink), the registry client, the image store and the VM configuration. CI has no KVM, so I verified on Firecracker nested in an aarch64 VM under QEMU here: the opt-in `cargo test -p fctl-host boots_a_sandbox -- --ignored` (boot, user and hostname, DNS and HTTP out, piped programs, fctl-agent, stop and restart from disk), and sessions end to end from `cargo xtask dev` on a Firecracker runner: start, follow-ups, stop, resume from the disk, and a host restart with the VM running, which it recovered. I pulled images from Docker Hub, ghcr.io and a local registry, and booted the real default image (`ghcr.io/hyperstructure/sandbox:latest`): Claude Code, `gh` and Node run as `agent`, and `nproc` and `free` show the VM's size. I booted both Firecracker CI's kernel and the sandbox pull request's 6.18.59, whose Kernel workflow also booted the x86_64 build on a GitHub runner with KVM. Not tried: the driver on an x86_64 host, an Ubuntu host (its AppArmor profile for passt allows these socket paths, by reading it), and Claude sign-in. ## Merge order Merge the sandbox pull request first: its Kernel workflow publishes the kernel this driver pulls by default. The fctl-site pull request shows the driver on fctl.sh's landing and sandboxes pages; merge it with or after this one. - hyperstructure/sandbox#12 - hyperstructure/fctl-site#95 ## Worth confirming - **Decision:** `--driver auto` never picks Firecracker: a host's owner chooses it, so updating fctl changes no host's driver. - **Decision:** Firecracker runs as the host's user, without its jailer (which needs root), so a VM escape would land as that user, as a container escape can with Docker. The security page says so. - **Decision:** no package caches and no image builds (`allowed_builds`) with this driver for now; projects' own images run. - **Decision:** each sandbox gets a sparse 64 GiB disk (`[firecracker] disk_gb` in `host.toml`), which takes what the session writes and gives back what it deletes. - **Decision:** VMs reach the internet and the host's network as Docker containers do, but not each other or the host's loopback. - **Assumption:** joining `kvm` and installing `passt` and e2fsprogs is setup enough for "seamless"; the host downloads Firecracker and the kernel itself. [fctl session](https://app.fctl.sh/s/ses_01M64PV6RKNWY1BK8M26YC1TPP) <!-- FCTL_REVIEW_SUMMARY --> --- > [!NOTE] > **Medium risk.** It adds a whole new sandbox driver that downloads and runs binaries and a kernel, unpacks images as root inside VMs, and changes the shared `Driver::spawn` interface the Docker driver also uses; it is opt-in, so existing hosts keep working. > > **What it does.** Adds an opt-in Firecracker driver so Linux hosts can run each session in its own microVM instead of a Docker container, from the same images. `fctl-agent` doubles as the VM's init and runs programs over a vsock, with networking carried through `passt` and nothing running as root on the host. `Driver::spawn` now returns pipes and an exit code rather than a child process, and the docs, an ADR and the `fctl host` commands are updated to match. > > **Merge order and untested paths.** The host pulls the kernel image `ghcr.io/hyperstructure/sandbox-kernel:6.18.59`, which hyperstructure/sandbox#12 publishes, so that pull request must merge first. CI has no KVM, so the VM paths were only tried by hand on aarch64. > > <sub>[fctl review](https://app.fctl.sh/s/ses_01M64XWMY126EQVVWH6ZK7YBYD)'s first look at [`32be81c`](32be81c) · 37 files, +6005 −82 · [Review agent settings](https://app.fctl.sh/p/fctl?tab=reviews)</sub> <!-- /FCTL_REVIEW_SUMMARY --> --------- Co-authored-by: fctl-sh[bot] <fctl-sh[bot]@users.noreply.github.com>
kalaspuff
pushed a commit
to hyperstructure/fctl-site
that referenced
this pull request
Oct 29, 2026
…rs (#95) ## Summary fctl's new `firecracker` driver (the fctl pull request below) runs each session in a Firecracker microVM with a kernel of its own on Linux hosts with KVM. The landing page now names Firecracker in the strip of what fctl works with, in step 01, in the runners card and in "How the pieces fit" (a Linux box: Firecracker or Docker), and the features overview's Sandboxes card says it too. The sandboxes page names it in its description, hero and points, and gets a Firecracker card beside Apple container and Docker, which says the driver is experimental, builds no images and has no package caches; the page's sentences on builds and caches now say they are Apple container's and Docker's. The driver cards sit three in a row where they fit; at two to a row, the odd one out takes its row. I checked with `npm run check` and a full build against fctl's branch (`FCTL_DOCS_DIR`), whose metadata and link checks pass. No heading or scene changed, so the link preview images stay as they are. ## Merge order Merge with or after fctl's pull request: until then, this describes a driver that no release has. - hyperstructure/fctl#344 - hyperstructure/sandbox#12 ## Screenshots At 02177d5 for the sandboxes page and a6556e6 for the landing page, 1440 px wide unless it says otherwise. The sandboxes page's drivers, light: | Before | After | |---|---| |  |  | At 1000 px, where Docker's card takes its row, and at 390 px, dark: | 1000 px | 390 px | |---|---| |  |  | The landing page's step 01, dark: | Before | After | |---|---| |  |  | "How the pieces fit", the runners card and the strip under the hero, light: | Before | After | |---|---| |  |  | |  |  | |  |  | ## Worth confirming - **Decision:** the Firecracker card calls the driver experimental, as fctl's roadmap does, and says Firecracker is "the virtual machine monitor AWS Lambda runs on" and that "nothing on the host runs as root". The landing page lists it without that word. - **Decision:** Lima stays unmentioned on the site, as before; the pages name Apple container, Firecracker and Docker. [fctl session](https://app.fctl.sh/s/ses_01M64PV6RKNWY1BK8M26YC1TPP) <!-- FCTL_REVIEW_SUMMARY --> --- > [!NOTE] > **Low risk.** Copy changes to the landing page and the sandboxes page, plus a small CSS rule for a third card, with no logic, routes or stored data touched. > > **What it does.** Adds Firecracker microVMs to the pages that describe sandboxes: a new driver card on the sandboxes page marked experimental, and mentions on the landing page, the features index and the page's meta descriptions. Claims about Dockerfile builds and package caches are now limited to Apple container and Docker runners, since Firecracker builds no images and keeps no caches. A CSS rule lays the driver cards out three to a row on wide screens. > > **Depends on fctl#344.** The copy describes the Firecracker driver from hyperstructure/fctl#344, so it should merge with or after that change. > > <sub>[fctl review](https://app.fctl.sh/s/ses_01M64XAS341Z5ZMJB7PPC47AB2)'s first look at [`02177d5`](02177d5) · 4 files, +45 −20 · [Review agent settings](https://app.fctl.sh/p/fctl?tab=reviews)</sub> <!-- /FCTL_REVIEW_SUMMARY --> --------- Co-authored-by: fctl-sh[bot] <fctl-sh[bot]@users.noreply.github.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fctl's new
firecrackerdriver (the fctl pull request below) boots each sandbox VM with a kernel published from here: Linux 6.18.59, the long-term release, configured with Firecracker 1.17's guest configuration for 6.18 plus/proc/config.gz, everything built in.kernel/build.shbuilds it for the machine it runs on, andkernel/check.shboots it in Firecracker and checks it reaches userspace with ext4, overlayfs, vsock and TAP devices, which fctl-agent's init needs. The new Kernel workflow builds it for amd64 and arm64 on their own runners, boots it where the runner has KVM, and onmainpublishes it asghcr.io/hyperstructure/sandbox-kernel:6.18.59and:6.18, an image whose/vmlinuxis the kernel. Pull requests build and check it without publishing. The sandbox image itself doesn't change.I built 6.18.59 for arm64 with
kernel/build.shand booted it in Firecracker nested in a VM (an fctl session has no KVM of its own):kernel/check.shpassed, and so did a whole fctl sandbox on it (network, programs, fctl-agent, stop and restart). This pull request's Kernel workflow built both architectures with the Dockerfile and booted the amd64 build in Firecracker on its runner's KVM; the arm64 runner has none.kernel/check.shmounts a drive as ext4, lays an overlay on it and checks vsock's transport and TAP devices, and fails when one is missing (tried with a device no kernel has). A published version is never built again: a change to the build takes a new version (6.18.59-2). ShellCheck and actionlint pass.Merge order
Merge this before the fctl pull request: the driver pulls
sandbox-kernel:6.18.59by default, which this publishes.Worth confirming
/dev/kvm, which the workflow opens to the runner's user; the arm64 runners likely don't, and the check says so and boots nothing there.fctl session
Note
Medium risk. It adds a workflow that publishes a kernel image hosts will boot, and a published tag never moves, so a bad build or wrong tag stays; the sandbox image itself doesn't change.
What it does. Adds a
kernel/directory that builds the Linux 6.18.59 kernel fctl's Firecracker sandboxes boot, using Firecracker's guest configuration, and acheck.shthat boots it in Firecracker with a drive, an overlay, vsock and a TAP device. A new Kernel workflow builds it for amd64 and arm64, boots it where the runner has KVM, and publishes it frommainasghcr.io/hyperstructure/sandbox-kernel. A published version is never built again, so a change to the build takes a new version inkernel/VERSION.Tags never move. The first merge to
mainpublishes6.18.59for good, and fctl should pin it only after it exists. The arm64 build is not booted in CI, as its runners have no KVM.fctl review's first look at
62c4034· 14 files, +8009 −8 · Review agent settings