Skip to content

feat: let hosts allow and build projects' sandbox images - #88

Merged
kalaspuff merged 8 commits into
mainfrom
fctl/39-can-you-check-if-something-is-left-to-do
Oct 18, 2026
Merged

kalaspuff merged 8 commits into
mainfrom
fctl/39-can-you-check-if-something-is-left-to-do

Conversation

@fctl-sh

@fctl-sh fctl-sh Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Finishes #16. A project starts its sessions from an image of its own, or from one that hosts build out of a Dockerfile or devcontainer.json in its repository. Host owners decide which images their machines run and which repositories they build from, and sessions go only to runners that allow their image. The docs now say how to do all of it.

Fixes #16

What changed, by the issue's points

  1. The contract (docs/sandbox-images.md). "What an image needs" now lists gh, a non-root user, both architectures and the oldest Claude Code fctl works with. That is 2.1.101, the first release that asks its host for fresh OAuth tokens (oauth_token_refresh; remote_control came in 2.1.63). I found both by checking published Claude Code builds, because the changelog doesn't say. fctl-agent now runs claude --version before it starts Claude, and fails the session with a clear message when Claude Code is missing or older. "What fctl writes into a sandbox" is new. The worked example adds .NET rather than Rust, since the default image has had Rust since feat: add Python 3.15, Rust, Bun and TypeScript to the sandbox image #45.
  2. CLI. fctl projects set gets --image, --build / --build-context, --setup-script (the script, @FILE, or - for stdin), --env / --unset-env, and --cpus / --memory (default for the runner's own), with --json like the rest. fctl projects show prints them.
  3. The image's Claude settings stay. fctl-agent writes its own keys over ~/.claude/settings.json instead of replacing the file. It owns the permission mode, attribution, disableClaudeAiConnectors and the auto-updater. It drops apiKeyHelper, ANTHROPIC_* and provider variables, and permissions.disableBypassPermissionsMode. Plugins, marketplaces, hooks and permission rules stay.
  4. Builds from the repository (crates/fctl-host/src/build.rs). The new project setting image_build names a Dockerfile (plus the directory it builds in) or a devcontainer.json.
    • When it builds: a host that allows it reads the file at the tip of the default branch through GitHub's API, using the session's token. It hashes the file, the Dockerfile and the git tree of the build directory, and builds only when no image has that hash.
    • How it builds: it downloads the repository archive (up to 2 GiB) and unpacks only the build directory and the Dockerfile. It sends the token only to GitHub's API origin. Then it runs docker build or container build, one build at a time.
    • Limits: build_timeout_minutes (60) and max_build_mb (20 GiB). It keeps each project's newest build, and starts from the last build while GitHub is unreachable.
    • Dev containers: fctl reads image, build.dockerfile/context/args/target. It leaves out features and lifecycle commands, and says so in the session's log.
  5. Host owners decide what runs.
    • host.toml gets allowed_images (references, /* prefixes, digest pins) and allowed_builds (owner/name, owner/*). fctl host install gets --allow-image and --allow-build for them.
    • Hosts send both as optional hello fields. FleetDO assigns a session only to a runner that runs or builds its image (fctl_domain::schedule::pick). A session no runner takes waits and says why ("No runner allows this project's image, …" / "No runner builds this project's image from …").
    • The host checks again before it accepts a session. docs/security.md says that whoever sets the image, the build or the setup script can read the credentials of the people who start sessions.
  6. The app. Settings → Sandbox → Image picks each runner's own image, an image from a registry, or a build from the repository. Its hint says which drivers use images. Admin → Runners, fctl admin runners and fctl host status show each runner's image and what else it runs and builds.
  7. Private registries. Hosts pull with their own login (docker login, container registry login), and the docs say so. fctl stores no registry credentials.

ADR 0017 records the decisions. One worth a look: allowed_images unset still means any image, so upgrading no host stops a project's sessions.

Compatibility

  • New fields: all optional. ProjectSettings.image_build is always serialized, as null when there is none. HostHello.allowed_images / allowed_builds and Runner.image / allowed_* are skipped when empty.
  • Older hosts send no lists: they run any image and build none, as before. Older agents ignore image_build.
  • Older clients (an old web tab, the previous CLI) that send settings without image_build keep a project's build, unless they choose an image instead (settings::keep_unsent).
  • Validation: image references are now validated with Docker's grammar.
  • Images with Claude Code older than 2.1.101 now fail at start with a message. Before, they failed once an OAuth token expired.

How I verified it

  • Rust: cargo xtask check passes (fmt, generated types, Biome and TypeScript, Clippy native and wasm, all tests, the web build). New unit tests cover:
    • reference parsing and pattern matching, and scheduling by image;
    • keep_unsent and build validation;
    • JSONC and devcontainer parsing, the build key, tar unpacking (hard links skipped, nothing escapes through links);
    • the settings merge and the version check.
    • One host test builds end to end against a fake GitHub (API, plus a second origin for the archive) and a fake docker. It covers: build once, reuse, no rebuild after an unrelated commit, rebuild after a change in the build directory or the Dockerfile, removal of the old build, fallback while GitHub is down, a failing build, the size cap, and a devcontainer image. It also caught the token going to the archive's redirect origin, which is fixed.
  • Web: Vitest covers the image chooser's state and the runner descriptions.
  • Dev stack (cargo xtask dev --no-host, two hosts I registered with the new flags: a process-driver host allowing ghcr.io/acme/*, and a Docker-driver host behind a stand-in docker that builds from acme/*):
    • a project image no runner allows waited with the reason above; after switching to an allowed image, the waiting session started on the right host and finished its turn;
    • a build from other/tool waited ("No runner builds this project's image from other/tool"); a build from acme/app went to the building host, and stopped there only because the dev stack has no GitHub App;
    • stopping and resuming a session merged seeded "image" settings: plugins, hooks and RUST_LOG stayed, apiKeyHelper and ANTHROPIC_BASE_URL were gone;
    • the CLI output, fctl admin runners and fctl host status read as described.
  • The .NET example's steps ran in this sandbox, which runs the default image: SDK 8.0.426 installed, a console app built and ran, and the profile.d snippet set the PATH for login shells.
  • Docs: hyperstructure/fctl-site built with these docs, and every link and anchor between pages resolves.
  • Independent review: a separate agent reviewed the whole diff, and b59b708 fixes what it found, with tests for each:
    • a tag in allowed_images allowed that tag pinned to any digest;
    • reading a large .json build file was quadratic;
    • the host fell back to the last build after any error, not only when GitHub was unreachable;
    • a Dockerfile that is a symlink would have been read on the host;
    • Docker Hub prefixes written with docker.io matched nothing;
    • smaller ones: a stale build directory in the CLI, leftover build files after a stop, a signed download link in error messages, a few gaps in reference parsing, and the chooser forgetting typed values.

Not verified: a real docker build or Apple container build. This sandbox has neither runtime, so the builder commands, container image inspect sizes and container image delete are untested on real runtimes. A first build on a Docker host and on a Mac host is worth doing before relying on it.

Not in this change

For a person to decide; I can open issues for them:

  • Hosts that run only their own image unless their owner says otherwise.
  • Registry credentials from project secrets.
  • Dev container features, which need the Dev Container CLI.

Other repositories

  • hyperstructure/fctl-site: docs(legal): say what hosts download for projects' images fctl-site#16 updates the privacy policy. Hosts now pull the images projects choose, download repositories from GitHub to build them, and report the images and repositories they allow.
  • hyperstructure/sandbox needs no change: the image's layout is the same, and it ships the newest Claude Code.

Screenshots

Taken at 809b7b2 against cargo xtask dev with two dev hosts, 1440 px wide unless noted. Later commits don't change these screens.

Settings → Sandbox, a project that builds its image. Before, the form has no way to show a build; after, the Image choice, the file and the directory it builds in.

Before After
Before: the Sandbox section has only an image field After: Image set to "Built from the repository", with File and Directory

The three choices, an image from a registry with its hint, and the dark theme at 390 px:

Choices An image Dark, 390 px
The Image select open: each runner's own, an image from a registry, built from the repository An image from a registry, with the reference ghcr.io/acme/agent:1 The build fields stacked at phone width, dark theme

Admin → Runners. Each runner now says its image, what else it runs and what it builds.

Before After
Before: runners without their images After: "Image ghcr.io/hyperstructure/sandbox:0.5.0 · runs its own image and ghcr.io/acme/* · builds no images"

After, dark theme

Session: https://app.fctl.sh/s/ses_01M57YPG1ATK9ARAXTZG43G51J

fctl-sh Bot and others added 8 commits October 18, 2026 17:22
…ositories

Hosts list the images they run for projects that choose their own
(allowed_images in host.toml: references, or prefixes in /*, optionally
pinned to digests) and the repositories whose Dockerfile or
devcontainer.json they build images from (allowed_builds). Both reach the
fleet as optional fields in the hello; FleetDO schedules sessions only on
runners that run or build their image, and a session that no runner takes
says why. Hosts refuse sessions they don't allow, for control planes that
send one anyway.

A project's image_build setting names a Dockerfile (with a context) or a
dev container configuration in its repository. A host that allows it
reads the file on the default branch, keys the build by the files it
reads, downloads only what the build needs, and builds it once, one build
at a time, within build_timeout_minutes; an image over max_build_mb is
removed. The next build of a project removes its earlier one, and while
GitHub can't be reached sessions start from the last build.

Refs #16
fctl-agent merged nothing into ~/.claude/settings.json: it overwrote it,
so an image's plugins, marketplaces, hooks and permission rules were
lost. It now writes its own keys over the image's settings and drops
what could move Claude off the subscription or out of bypass mode
(apiKeyHelper, ANTHROPIC_* and provider variables in env,
permissions.disableBypassPermissionsMode).

Before starting Claude it runs claude --version and fails the session
with a clear message when Claude Code is missing or older than 2.1.101,
the first release that asks its host for fresh OAuth tokens. The agent's
hello reports the version.

Refs #16
fctl projects set gets --image, --build and --build-context, --setup-script
(the script, @file or - for stdin), --env and --unset-env, and --cpus and
--memory (default for each runner's own), and fctl projects show says
where sessions start from, their size, setup script and variables.

fctl host install takes --allow-image and --allow-build for
allowed_images and allowed_builds, refusing entries that match nothing;
fctl host status and doctor, and fctl admin runners, say which images a
host runs and which repositories it builds images from.

Refs #16
Settings → Sandbox → Image picks each runner's own image, an image from a
registry, or one built from the repository (a Dockerfile and the
directory it builds in, or a devcontainer.json), and says which drivers
use images and which runners run or build them. Admin → Runners shows
each runner's image and what else it runs and builds.

The docs describe projects' own images with a worked example, builds
from the repository, which images a host runs, private registries, what
an image needs and what fctl writes into a sandbox; ADR 0017 records the
decisions.

Refs #16
"A project's own image" made a different anchor on fctl.sh than on
GitHub; "Your own image" makes the same. The .NET example's profile.d
line fits its code block.
Two sessions that finished getting their images at once could each save
builds.json without the other's project. The docs now say that a build
whose directory is the repository's root runs again after any change,
and how to avoid that.
- A tag in allowed_images no longer allows a reference that pins a
  digest to it: a pull by digest ignores the tag.
- References follow Docker's grammar: uppercase makes the first part a
  registry, repositories are lowercase, separators and sha256 digests are
  checked. Docker Hub prefixes written with docker.io or index.docker.io
  match its images.
- Hosts read dev container configurations in one pass, and no file over
  1 MiB.
- Only an unreachable GitHub starts sessions from the last build, and
  only one built for the same setting; a missing file fails the session.
- Builds use the Dockerfile's text as GitHub sends it, never a file a
  link in the repository points to on the host, and unpack nothing
  through a link.
- Sessions that need the same build wait for the one that builds it, and
  each downloads before waiting for another build, so its token can't
  expire in the queue. A stopped build removes its files, and hosts clear
  leftovers when they start.
- Download errors leave out signed links' tokens; a dev container's image
  must be a valid reference.
- fctl projects set --build x.json drops the Dockerfile's directory; the
  web form brings back what was typed when switching back; an image and a
  build together fail on the image's field, which older forms show.

Refs #16
@kalaspuff
kalaspuff merged commit fc72054 into main Oct 18, 2026
4 of 5 checks passed
@kalaspuff
kalaspuff deleted the fctl/39-can-you-check-if-something-is-left-to-do branch October 18, 2026 18:23
kalaspuff pushed a commit to hyperstructure/fctl-site that referenced this pull request Oct 18, 2026
The privacy policy says what `fctl host` downloads and what app.fctl.sh
keeps about hosts. hyperstructure/fctl#88 changes both, for
hyperstructure/fctl#16:

- **What hosts download.** Hosts pull the images projects choose, from
those images' registries. Where their owners allow it, they also
download a project's repository from GitHub to build the image the
project builds from it.
- **What app.fctl.sh keeps.** Hosts report the sandbox image they use,
and the images and repositories they allow. app.fctl.sh keeps these with
the rest of a host's details.

The policy's `updated` date is already today's.

Refs hyperstructure/fctl#16
Pairs with hyperstructure/fctl#88.

## How I verified it

`npm run check` passes, with the same two warnings as `main`.
`FCTL_DOCS_DIR=<that branch> npm run build` renders the policy and the
docs from the fctl branch. Every link and anchor between the docs pages
resolves, including the new sections of "Sandbox images".

Session: https://app.fctl.sh/s/ses_01M57YPG1ATK9ARAXTZG43G51J

Co-authored-by: fctl-sh[bot] <343971779+fctl-sh[bot]@users.noreply.github.com>
fctl-sh Bot added a commit that referenced this pull request Oct 18, 2026
Resolves the conflicts with #88. Its build tests and the image refresh's
share one stand-in CLI, driver/fake.rs, which gains #88's builds, image
sizes and failing builds; the two remove_image methods it and this branch
added were the same, so one stays. docs/sandbox-images.md keeps #88's
wording for project images, and says that a host on another image removes
none of the default image's: it does remove old project builds.
kalaspuff pushed a commit that referenced this pull request Oct 18, 2026
Hosts on the default sandbox image pulled their tag once and kept it.
hyperstructure/sandbox's Image workflow rebuilds the newest release's
image when Claude Code has a newer release, since fctl turns off Claude
Code's auto-updater. But a host that already had the tag kept its first
pull until it was upgraded or someone ran `fctl host image pull`, so its
sessions' Claude Code aged between upgrades anyway. And every upgrade
left the previous version's image on the host, as did the `latest` a
host pulls while its release's image isn't published yet. Nothing
removed them.

The two decisions #47 left to a person were made in this session, and
are recorded in [the
issue](#47 (comment)).
The host tells a local build from the published image by the
`org.opencontainers.image.revision` label, which only the Image workflow
sets. It pulls again at start and every 6 hours.

## What changed

**Refreshing** (`crates/fctl-host/src/refresh.rs`, `daemon.rs`). The
host's background pull used to run once, at start, and only for a
missing image. Now it runs at start and every 6 hours:

- An image that isn't here is pulled as before, falling back to `latest`
while the version's image can't be pulled.
- On the default image, the host pulls its tag again, so new sessions
start from the newest build. Sessions never wait for it: they start from
the build that is there. When the tag hasn't changed, the pull costs one
manifest request.
- It never pulls over an image without the revision label. That covers
builds from `fctl host image build` or by hand: the sandbox's
`Dockerfile` sets no labels, and its base, `debian:bookworm-slim`, has
none. An image whose description can't be read counts as a local build,
so the host leaves it alone, as it did before.
- A host whose `host.toml` sets another image pulls it only when it
isn't there, as before.

**Removing** (`refresh::clean_up`). At start, after each pull and after
sweeps that remove sandboxes, the host removes these, without force:

- builds that a newer one replaced;
- other versions' images (`X.Y.Z` tags);
- `latest`, once the version's image is here. Until then, sessions start
from it.

Images without the revision label stay, and so do tags hosts don't pull,
like `0.5`, which projects may use. A host on another image removes
nothing.

**How a replaced build stays findable.** A pull that moves a tag leaves
the old build without a name. Docker keeps it as `<none>`, and with the
containerd image store nothing finds that again by name. Apple's
`container` keeps its blobs and snapshot orphaned until an `image
delete` runs its cleanup. So before a pull (or `fctl host image build`)
replaces a published build, the host gives that build a second name,
`<tag>-replaced-<the start of its ID>`, and removes the name right
after:

- If the build is unchanged, removing the name only untags it.
- If it was replaced and nothing uses it, the build goes.
- With Docker, if it was replaced and a sandbox still uses it, Docker
refuses. The name stays until a cleanup after the sweep removes that
sandbox (`keep_stopped_hours`).

This needs no state file, and a host that stops halfway leaves a name
the cleanup finds. A name whose build is still the one sessions start
from is left alone, so a cleanup that runs during a pull can't strand
the build.

**Apple.** `container image delete` doesn't check whether a container
uses the image. It's still safe: each container gets a clone of the
image's filesystem when it is created (`Bundle.cloneContainerRootFs` in
apple/container), and starting it again uses only that clone. So on
Apple the host removes these right away, and stopped sandboxes still
start.

**Driver** (`driver/container.rs`, `driver/mod.rs`):

- `image`: ID and labels from `image inspect`, in Docker's, Podman's and
both of Apple's shapes (its JSON changed at 1.0).
- `tag_image`.
- `remove_image`: `image rm` or `image delete`, never forced.
- `images`: `image ls --format …` or `image list --quiet`.

**CLI** (`crates/fctl/src/cmd/host.rs`):

- `fctl host image pull` also removes what sessions no longer start
from, and says what it did:

  ```
✓ Pulled ghcr.io/hyperstructure/sandbox:0.5.0, a newer build (Claude
Code 2.1.310)
    Removed ghcr.io/hyperstructure/sandbox:0.4.0
The build it replaced stays until no sandbox uses it; the host removes
it then.
  ```

`--json` adds `newer` (whether the pull brought a newer build) and
`removed` (the names it removed) to `image` and `instead_of`.

- `fctl host doctor` shows the image's Claude Code version, or that the
host doesn't pull over a local build:

  ```
✓ sandbox image ghcr.io/hyperstructure/sandbox:0.5.0 (Claude Code
2.1.310)
✓ sandbox image ghcr.io/hyperstructure/sandbox:0.5.0 (built here: the
host doesn't pull over it)
  ```

**Docs:**

- `docs/sandbox-images.md` says when the host pulls again, how it tells
a local build, and what it removes and when, with Docker and with Apple.
- `docs/architecture.md` and `docs/cli.md` say the same briefly.
- AGENTS.md lists the revision label among what the host relies on in
hyperstructure/sandbox. hyperstructure/sandbox#3 does the same there.

**Tests.** The driver tests' stand-in for the `docker` and `container`
CLIs moved to `driver/fake.rs`, shared by the driver tests, these tests
and #88's build tests. It gained image IDs, labels, tags, listing, and
Docker's refusal to remove an image's last name while a container uses
it. It keeps #88's builds, image sizes and failing builds.

**With #88.** `main` gained #88 (projects' own images and builds) while
this was open, and is merged in. Both added the same `remove_image` to
the driver, so one stays. Project images and builds aren't the default
image, so the host doesn't pull them again here or remove them in this
cleanup: #88 keeps each project's newest build itself.

## Compatibility

This changes only the host: no protocol or control-plane change, and it
reaches hosts with the next release. `fctl host image pull --json` gains
two fields.

- At its first start after the upgrade, a host removes the previous
version's image, unless a stopped sandbox uses it (Docker).
- Builds that earlier pulls left as `<none>` stay: nothing tells them
from other unnamed images.
- Two hosts sharing one Docker daemon on different versions would each
remove the other's image when no sandbox uses it, and pull it again
later. That costs extra downloads; nothing breaks.

## Verification

- **Unit tests:** 14 new; fctl-host has 57 with #88's.
- `removable`: what goes and what stays, including `latest` before and
after the version's image arrives, local builds, `X.Y` tags, and a
replaced name whose build is still current.
- Version tags, replaced names, and inspect output from Docker, Podman,
Apple 0.x and Apple 1.x.
  - Flows against the stand-in:
- a newer build replaces one a stopped sandbox uses, which goes once the
sandbox is gone;
    - an unchanged build is only tagged and untagged;
- the host never pulls over a local build, and `fctl host image pull`
does;
    - a build over the published image (and a failed one);
    - `latest` going once the version's image is published;
    - other versions going after an upgrade, unless in use;
    - Apple removing a replaced build that a sandbox was made from;
    - a host on another image doing nothing.
- **The published image** (read from ghcr.io anonymously): the config of
`ghcr.io/hyperstructure/sandbox:0.5.0` has
`org.opencontainers.image.revision` (`12685ad…`) and
`sh.fctl.claude-code.version` (`2.1.302`). `debian:bookworm-slim`, the
base of local builds, has no labels.
- **With real Podman** (rootless, `FCTL_DOCKER_BIN=podman`): I imported
images of a few KB under the default image's names. The host's `0.5.0`
was a local build, and a stopped `fctl-ses-…` container used `0.3.0`.
`fctl host run`:
  - logged that it doesn't pull over the local build;
- removed `0.4.0` (only untagged, since `0.4` names the same build),
`latest`, and a `0.5.0-replaced-…` build;
- kept `0.3.0` because Podman refused ("image is in use by a
container"), and kept the local `0.2.0` and the `0.4` tag.

  `fctl host doctor` said "built here: the host doesn't pull over it".
- **With the stand-in CLI** (`FCTL_DOCKER_BIN`), the real binary:
  - `fctl host image pull` printed the output above.
- Once the sandbox was gone, `fctl host image pull --json` printed
`{"image": "ghcr.io/hyperstructure/sandbox:0.5.0", "instead_of": null,
"newer": false, "removed":
["ghcr.io/hyperstructure/sandbox:0.5.0-replaced-0a1d00000000"]}`.
- `fctl host run` logged `pulled a newer build of
ghcr.io/hyperstructure/sandbox:0.5.0 (Claude Code 2.1.310)`, kept the
replaced build while a sandbox used it, and removed `0.4.0`.
  - `doctor` showed the Claude Code version.
- `cargo xtask check --quick` passes, before and after merging `main`. I
reran the Podman check on the merged binary, with the same result.
- **Not run here:** Docker itself (this sandbox can't run a Docker
daemon) and Apple's `container` (that needs a Mac). The commands are the
documented ones. I read the behaviors this relies on in their sources:
- Docker refuses, without force, to remove an image's last name while a
container uses it, in both image stores, and moby's containerd store
counts its `moby-dangling@…` records as that same name
(`daemon/images/image_delete.go`, `daemon/containerd/image_delete.go`).
- In apple/container, a container keeps its own clone of the image's
filesystem, a pull leaves the old build orphaned until `image delete`
cleans up, and `image tag` overwrites an existing name.

Trying the first release on a Docker host and on a Mac is the check
that's left.

Fixes #47

Session: https://app.fctl.sh/s/ses_01M5813Y3P1VY7K0ZSS6PKRS9V

---------

Co-authored-by: fctl-sh[bot] <fctl-sh[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Custom sandbox images for projects and hosts

1 participant