Skip to content

Run every agent in an NVX microVM of its own - #975

Merged
lavaman131 merged 11 commits into
mainfrom
nvx-sandbox
Nov 21, 2026
Merged

lavaman131 merged 11 commits into
mainfrom
nvx-sandbox

Conversation

@lavaman131

@lavaman131 lavaman131 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Every agent the connector starts now runs in an NVX microVM of its own: each coding agent of a Build, Run, Retry, investigation or prototype, each Planner backend (cold or spare), the agent checks, and the builds of agent images. This replaces the process sandbox (MXC with Seatbelt, bubblewrap and ProcessContainer) completely. There is no fallback and no unsandboxed mode, and a machine that can't run NVX is refused at pairing and connect.

The design, and what was built, is in docs/nvx-sandbox.md. The guide for people who run chopin is updated in docs/chopin-cli.md, and the AGENTS.md sandbox rule is rewritten.

What changes for people

  • Which machines can run work. Linux on x86_64 with read and write access to /dev/kvm (or /dev/mshv), or Windows on x86_64 with the Windows Hypervisor Platform. On a Mac, an Arm machine, or a machine whose user can't use KVM, chopin pair and chopin connect refuse before they pair or spend a key. The refusal says what the machine needs and, where there is one, the fix (sudo usermod -aG kvm $USER, …).
  • No agent install on the machine. chopin builds an image for the agent once, in a microVM, with the version this release pins (Copilot CLI 1.0.114; Atomic 0.10.2 with atomic-acp 0.0.18). Another ACP agent needs --install '<command>', and a Planner command needs --planner-install. The first connect downloads NVX (about 275 MB, checked against a pinned SHA-256) and builds the image (about 40 s).
  • No worktree folder is left after a run. The agent works in its own clone inside its microVM and pushes what it keeps. runs/ keeps each run's record and its agent's state.
  • Branches appear only once pushed. A run's or Planner's branches are never in the person's own repository. They show up in git branch only after a git fetch.
  • A Planner keeps a disk of its own (planners/<owner>/<repo>/<document>.disk, sparse, up to 32 GiB) with its clone. The disk lasts between the Planner's starts. It goes when Chopin says the document is archived or gone, after 30 days unused, or when the machine is unpaired or revoked, never while a launch holds it. Commits on it that were never pushed go with it, and the connector says so.
  • --require-sandbox and --no-sandbox are gone. Saved settings that had them keep working without them. --sandbox-host, --sandbox-write and --sandbox-hide stay.
  • The Machines page says what a machine needs in Add machine. While a paired machine is away and can't run work (it can't run microVMs, or its chopin is too old), its status reads "Can't run work", with the reason and the fix.
  • A run against a Chopin without AI launches (which answers 404) no longer gets the machine owner's own models. None of the owner's setup is in a microVM, so the run has only what --env gives it.

How it works

  • One seam. Sandbox.spawn(launch) in apps/connector/src/sandbox.ts is the only way to start an agent. vmSpec in apps/connector/src/nvx.ts is a pure function that decides everything a guest gets: its image and user, its shares and their modes, its proxy, its environment and its command. nvxArguments turns that into one nvx sandbox run. The environment goes in a 0600 file, never on a command line.
  • Repositories in the guest. At start, the guest clones the repository into its own file system (PREPARE). A run uses git clone --shared from a read-only share of the checkout's git directory, into memory. A Planner uses --reference --dissociate onto its disk. S1's measurements drove this, and the supervisor approved it: a worktree shared over virtio-fs took 1.7 to 11 s for git status, while an in-guest clone took 32 ms.
  • Images. Each image is an EROFS layer built in a microVM from the NVX release's Ubuntu layer, with Node 24 and pinned git packages (each checked by SHA-256). It also carries an account for the connector's uid and gid, so shared files belong to the guest's user, plus safe.directory=*. Images are rebuilt only when their recipe changes.
  • Network. --network-egress=deny --host-loopback=deny --network-proxy=10.0.0.1:<port>: the connector's allow-list proxy is the only way out. The AI relay stays on host loopback and is reached through the proxy.
  • MCP bridge in the guest. A run's bridge to Chopin's tools runs beside the agent on the image's Node, from a bundle the chopin binary carries (guest-bridge.ts). An investigation's upload_image_file reads the image inside the guest's repository.
  • Teardown. On Linux each nvx runs in its own process group, which is ended with its launch. Launch records under <base>/nvx/launches hold the start IDs of the launch and its connector. The next connector kills only a process that still has its recorded start, never one that took over its number, and never a live chopin's launch. On Windows each nvx.exe goes in a kill-on-close job object, so a connector crash ends its VMs too. A stop first ends the workload's stdin and waits; only then does it kill the job. A VM still running 10 s later is reported as wedged rather than taken for gone.
  • What a guest left on the host is never followed. A Planner's agent state is shared writable and lasts. Before the host writes there, it removes any link the guest left (realDirectory, writeOwn, readOwn). A writable share never holds anything the launch may only read, and --sandbox-write can't name a folder inside the state directory.
  • Planner disks. A disk is held by its launch (<disk>.held, linked into place atomically and counted per process), and a sweep takes the same hold before it removes one. A disk is removed only for the server its document is on (<slot>.server). A Planner reset in Chopin removes the document's disk: the reset is recorded (channels.planner_reset_at, migration 052, expand-only), and each Planner credential names it. The connector stamps each disk with the reset it was made after (<slot>.reset) and removes an older disk before the next launch, or at the hourly planner/kept sweep, saying that unpushed commits went with it.
  • Server. Connector protocol 8. A connection, a Planner credential, a GitHub token, an MCP token, an AI launch and a machine's database login go only to a connector that says on that request that its agents run in microVMs (microvmsOnly). An older connector is told to update (connector-update-required) whatever it says of its agents, and a refused connection is kept on the machine as one that can't run work. A protocol 8 connector that doesn't say so gets microvm-required, and a machine's words about why are kept. The connector says so on every request once chopin pair or connect has checked the machine. POST /api/connector/capability records why a paired machine can't run them (machines.vm_problem, migration 051, expand-only), and the Machines listing shows it while the machine is away. POST /api/connector/planner/kept tells a machine which Planner disks it may forget: each archived, gone, or reset since the disk was made.

What read a run's worktree on the host, and where it went

Feature Now
Investigation and prototype screenshots (upload_image_file) Read in the guest's repository by the bridge in the guest. 1 MiB at most, never outside the repository
Chopin's tools for a run The same bridge in the guest, to Chopin through the proxy
Investigation and prototype results Unchanged: JSON over MCP
A prototype's throwaway branch Only ever in the guest, and refused if pushed
Extra worktrees for concurrent PRs git worktree add under /work in the guest
Servers an agent left running End with its microVM
Retention (--keep-runs) Keeps records and agent state, and removes an earlier release's worktrees
Build, Retry and read_rebuild Unchanged: Chopin reads GitHub
A spare Planner whose checkout's HEAD moved Let go, so the document starts cold. The host runs no git in a guest's repository

Invariants and their tests

Invariant Tests
No agent starts outside a microVM nvx.test.ts (the agent is nvx sandbox run's workload); nvx-vm.test.ts (ending a launch ends OpenVMM)
A guest's shares are exactly its launch's allowed set, and it writes nothing of the person's nvx.test.ts; nvx-vm.test.ts (writing into the person's .git fails read-only)
A clone reads the person's objects only through a read-only path nvx.test.ts; run.test.ts; planner.test.ts; nvx-vm.test.ts
The only way out is the proxy's allow-list nvx.test.ts; nvx-vm.test.ts (no direct connection, no DNS); sandbox-proxy.test.ts
No real credential in a guest nvx.test.ts; nvx-vm.test.ts (no secret on any host command line); launch-handoff.test.ts
A Planner's disk is never removed while a launch holds it, nor by another server planner-disks.test.ts (including a sweep racing a launch in the same process, and two servers)
A writable share never holds what a launch may only read nvx.test.ts (each shape, plus a property check)
The host never writes or reads through a link a guest left machine-store.test.ts; nvx-vm.test.ts (the guest plants the link)
A dead connector's leftovers are ended, and nothing else is sandbox.test.ts (a reused pid survives, a matching orphan is killed)
A machine that can't run NVX pairs and starts nothing, says why, and Chopin keeps it nvx.test.ts, main.test.ts, messages.test.ts, routes.test.ts, machine-contract.ts
No credential or work goes to a connector that would run agents outside a microVM routes.test.ts (connections, GitHub token, MCP token, AI launch, database login); planner-dispatch.test.ts (Planner credential); each refusal comes before anything is minted. machine.test.ts (the connector says so on each request)
A launch that never starts leaves no tokens on disk sandbox.test.ts
A Planner reset in Chopin removes the machine's disk from before it, and none made after planner-disks.test.ts (sweep and launch); planner.test.ts (a launch after a reset starts on a new disk); routes.test.ts (planner/kept); planner-dispatch.test.ts (the credential names the reset); channels/routes.test.ts; the storage contract on memory and PostgreSQL
Snapshots hold nothing of a launch Holds trivially: this release takes no snapshots

Validation

On the final head:

  • bun run fix made no changes. bun run ci and bun run types passed.
  • Unit tests: bun test apps/connector apps/server/src/{experiments,tasks,harness,auth,ai,machine-store,mcp,channels,storage/memory} apps/web/src/experiments packages/protocol gave 2190 tests: 0 failed, 28 skipped.
  • PostgreSQL: TEST_DATABASE_URL=… bun test apps/server/src/storage/postgres (a throwaway postgres:17-alpine) gave 222 passed. apps/server/src/machine-store/server.test.ts against the same database gave 18 passed.
  • Real microVMs on KVM: CHOPIN_NVX_ROOT=<nvx-0.1.0-linux-kvm> bun test apps/connector/src/nvx-vm.test.ts gave 8 passed. Those tests cover:
    • the run clone and the read-only .git;
    • the Planner's disk across starts and its read of uncommitted files;
    • a link the guest left in agent state, which is never followed;
    • proxy-only networking;
    • the MCP bridge in the guest;
    • an orphaned server ending with its VM;
    • teardown, and the environment never on a command line.
  • Browser: bun run e2e for setup-agents, machines, implementation-launcher, planner-spare, experiments, planner-picker, handover and build-restart, under flock with an own port base and database. 119 passed. One machines test, "the sidebar's Machines menu sets the default machine and leads to the page", failed once under load and then passed 3 times out of 3 on its own.
  • Each new gate and reset test was seen failing on the code before its fix.

Windows

The three gaps found while measuring on WHP are closed, and windows-host.test.ts passed on a Windows Server 2025 host with WHP:

  • Planner disks are made sparse on the host (FSCTL_SET_SPARSE and SetEndOfFile) and formatted with nodiscard.
  • A Planner checkout's hidden .git is shown again.
  • Each nvx.exe runs in a kill-on-close job, as described under Teardown. A machine without the Windows Hypervisor Platform is refused with the command that turns it on.

On that host, the Windows binary ran --version and status. Pair, connect and a Build or Planner end to end on Windows are the next check.

Not done, or not verified

  • NVX is a 0.1.0 development release. The pinned package is downloaded from NVX's own GitHub release. Carrying it in Chopin's releases, and fetching it from install.sh and install.ps1, is the next step.
  • OpenVMM sometimes fails to start under heavy host load ("driver does not support io-uring"). The launch fails with its reason and can be started again; an automatic retry isn't built.
  • nvx sandbox run leaves its temporary folder when a launch is ended by force; sweeping it isn't built.
  • Snapshots aren't used. A restore can't take a Planner's own disk, and a spare already keeps a backend warm.

The connector no longer starts agents in the system's process sandbox
(MXC with Seatbelt, bubblewrap or ProcessContainer), and has no
unsandboxed mode. Every coding agent, Planner backend (cold or spare),
agent check and image build runs in an NVX microVM, through one seam
(Sandbox.spawn), whose guest is decided by one pure function (vmSpec).

- A guest boots its agent's image: an EROFS layer built once per recipe
  in a microVM, with Node, pinned git packages, the agent at the version
  this release pins (or the owner's --install commands) and an account
  for the connector's own user. The host needs neither the agent nor
  Node.
- An agent works in a repository of its own inside its guest: a run
  clones with --shared into memory, a Planner with --reference
  --dissociate onto a sparse disk that lasts between its starts. Shares
  of the person's checkout and git directory are read-only; the host
  runs no git in a tree an agent wrote, and a run leaves no worktree.
- The only way out is the connector's allow-list proxy, through NVX's
  proxy-only network. The environment goes in a 0600 file, never on a
  command line.
- The run's MCP bridge runs in the guest on the image's Node; the binary
  carries it.
- A Planner's disk is held while a launch uses it, and goes when Chopin
  says its document is archived or gone, after 30 days unused, or when
  the machine is unpaired or revoked.
- A machine that can't run NVX is refused by pair and connect before
  anything is paired, with what it needs and the fix. One already paired
  tells Chopin, which keeps it (migration 051) and shows it on the
  Machines page. Chopin takes connections only from connector protocol 8
  saying its agents run in microVMs, and tells older connectors to
  update.

Tests of what the connector does around an agent run it on the host
through a stand-in (testing/host-sandbox.ts, testing/chopin-on-host.ts),
never built into chopin. Real-microVM tests run where KVM and an NVX
package are available, including a new CI job.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
Co-authored-by: Alex Lavaee <lavaman131@github.com>
- vmSpec refuses a folder shared writable that holds what a launch may
  only read (the checkout, its git directory, its tokens), and refuses
  checkouts at /work or /opt/chopin.
- Agent state a guest writes is never written or read through a link it
  left there.
- A dead connector's microVM is ended only while it is still the process
  recorded; on Windows each nvx.exe is in a kill-on-close job, stopped by
  ending its workload first, and a VM that outlives its stop is reported.
- Planner disks are made sparse on the host, held per process against
  this process's own sweep, and only the server a disk's document is on
  may remove it; unpair and revoke take unassigned spares' disks too.
- A Planner checkout's hidden .git is shown again on Windows; Windows
  without the Hypervisor Platform, and root, are refused with the fix.
- connector.js carries the bridge its microVMs run.
- An older connector is kept on the machine as one that can't run work,
  and the Machines table says why and the fix while a machine is away.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
- A disk's lock is written whole and linked into place, so no process
  takes one half written.
- --sandbox-write can't name a folder inside chopin's state directory.
- On Windows a launch whose job object can't be made never runs.
- A service on a machine that can't run microVMs says so once and waits;
  service install checks the machine even with --skip-check.
- The Machines page's "Can't run work" has browser tests, and setup
  agents show what the machine needs above their own steps.
- Docs for operators, setup agents, what a guest can read, leftovers,
  and the disk locks; comments no longer describe the process sandbox.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
Chopin now refuses a Planner credential, a GitHub or MCP token, an AI
launch and a machine's database login, as well as a connection, to a
connector that would start agents outside a microVM. A connector
speaking a protocol before 8 is told to update whatever it says of its
agents, and a current one must say on the request that its agents run
in NVX. The connector says so on every request while a `chopin pair`
or `connect` that checked the machine runs.

Tests that drove older protocols through these routes now speak
protocol 8; tests of what an older connection did are gone, since none
connects. Each route has a test that an older connector, or one that
doesn't say, is refused before anything is minted.

Also from review: a launch that never starts (as on Windows without its
job object) removes its environment file at once, and a setup agent's
dialog no longer says chopin installs it.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
Both create and drop databases, which wait on a checkpoint that took
over four seconds on a busy CI runner, so they timed out at five and
raced the test after them.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
A reset in Chopin now starts a machine's Planner afresh, as it does in
Chopin. The reset route records when it happened
(`channels.planner_reset_at`, expand-only migration 052), before it
ends the Planner's link, and each reset is later than the one before.
Each Planner credential names that time. The connector stamps a disk
with the reset it was made after (`<slot>.reset`), and removes a disk
from before the latest reset before the Planner's next launch. It also
removes one at the hourly `planner/kept` sweep, which now names each
disk's reset and is answered with the documents reset since. Either
way, it says that commits never pushed went with the disk.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
The credential helper named the token file by its host path. A Windows
guest has that file under /mnt/<drive>, so the helper's `cat` found
nothing, and a Build's or Planner's push to GitHub got an empty
password. The helper now reads `$CHOPIN_GITHUB_TOKEN_FILE`, which every
caller of `gitCredentials` sets. A microVM's environment gives that
variable the path its guest sees, and host git reads it as before.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
A Windows connector passed its own profile variables to every agent's
Linux guest: APPDATA, USERPROFILE, PSModulePath, ComSpec, Path, a
PowerShell SHELL and the like. Those name nothing the guest has except
the host user's profile paths. The allow-list for agents on Chopin's AI
accounts now names only Linux's variables. Whatever the host, a microVM
is given no variable only Windows sets, from that list or from a
launch's own environment, and on a Windows host it is given neither its
`Path` nor its `SHELL`.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
`mkfs.ext4` left the inode tables for the guest's kernel to zero after
the first mount. On a 32 GiB disk that is about 512 MiB of zeros, and
both NTFS and Linux allocate zeros written into a sparse file, so every
new Planner disk grew to about 520 MiB on the host within a minute with
nothing written. A new disk is a fresh sparse file, which reads as
zeros, so `mkfs.ext4` is now told that (`assume_storage_prezeroed=1`)
and no lazy init runs. A KVM test boots a new disk for 40 seconds and
checks that it takes under 64 MiB on the host. Before this change it
took 519 MiB.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
A second `chopin connect` on a base dir that a running connector holds
opened its microVMs before it found the lock. Opening them built its
agent's image and pruned the running connector's image, which broke
that connector's next launch. Then the second connect never exited,
because its proxy and AI relay sockets were never closed.

`connect` now takes the base dir's lock before it opens the microVMs.
It lets go of both the microVMs and the lock when opening them or its
agent's start check fails. The service install check, which holds no
base dir, removes no images, because the running service may use one
another agent's recipe doesn't name.

Co-authored-by: Alex Lavaee <lavaman131@github.com>
@lavaman131
lavaman131 merged commit f6ed3a0 into main Nov 21, 2026
26 checks passed
@lavaman131
lavaman131 deleted the nvx-sandbox branch November 21, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant